Developer Documentation

Reference for the CyberGIS-Compute Python SDK

Attention

These are auto-generated documentation which describe the inner workings of the Python SDK. They are helpful for those who want to contribute to the project, but not designed to help users run jobs. If you are just trying to use the SDK, check out the Help page.

cybergis_compute_client.Client module

This module exposes Client class which creates a Client object that serves as a tool to make requests to different servers

Example

client = Client()

class cybergis_compute_client.Client.Client(url='cgjobsup.cigi.illinois.edu', port=443, protocol='HTTPS', suffix='v2')

An inteface that handles requests made to different servers

Parameters:

• url (str) – url that needs to be accessed

• port (str) – port of the Jupyter or Python interface

• protocol (str) – Typically HTTP or HTTPS

• suffix (str) – specify version. For e.g v2

url

url that needs to be accessed

Type:

str

port

port of the Jupyter or Python interface

Type:

str

suffix

specify version. For e.g v2

Type:

str

request(method, uri, body={})

Returns data from a request made to the specified uri

Parameters:

• methods (str) – type of request that needs to be made. For e.g “POST”

• uri (str) – uri of the server

• body (str) – data that needs to be sent

Returns:

output thats returned by the server

Return type:

JSON

cybergis_compute_client.CyberGISCompute module

This module exposes CyberGISCompute class which creates a CyberGISCompute object that serves as an entry point to the CyberGISX environment from a Python/Jupyter notebook. All interactions with the High Performance Computing (HPC) backend are performed using this object.

Example

cybergis = CyberGISCompute(url=’localhost’, port=’3030’, protocol=’HTTP’, isJupyter=False)

class cybergis_compute_client.CyberGISCompute.CyberGISCompute(url='cgjobsup.cigi.illinois.edu', port=443, protocol='HTTPS', suffix='v2', isJupyter=True)

CyberGISCompute class An inteface that handles all interactions with the HPC backend

Parameters:

• url (str) – url that needs to be accessed

• port (str) – port of the Jupyter or Python interface

• protocol (str) – Typically HTTP or HTTPS

• suffix (str) – specify version. For e.g v2

• isJupyter (bool) – set to True if you are using Jupyter environment

client

Initialized using url(str), protocol(str), port(str) and suffix(str)

Type:

Client object

jupyterhubApiToken

jupyterhub’s REST API token that can be used to authenticate the user

Type:

string

username

username

Type:

string

isJupyter

set to True if you are working in a jupyter environment else set it to False

Type:

bool

ui

Serves as entry point to UI functionality

Type:

UI

job

Serves as entry point to access job interactions

Type:

Job

recentDownloadPath

Gets the most recent download path from globus

Type:

str

jupyterhubHost

static variable that stores the path to jupyterhubHost

Type:

str

Initializes instance CyberGISCompute using inputs from the client

Parameters:

• url (str) – url that needs to be accessed

• port (str) – port of the Jupyter or Python interface

• protocol (str) – Typically HTTP or HTTPS

• suffix (str) – specify version. For e.g v2

• isJupyter (bool) – set to True if you are using Jupyter environment

Returns:

this CyberGISCompute

Return type:

CyberGISCompute

create_job(maintainer='community_contribution', hpc=None, hpcUsername=None, hpcPassword=None, verbose=True)

Creates a job object Initializes instance CyberGISCompute using inputs from the client

Parameters:

• maintainer (str) – Pre-packaged programs which can be configured and controlled remotely

• behave as a bridge between user and HPC backends (and) –

• hpc (str) – HPC backend that is being accessed. For e.g ‘keeling_community’

• hpcUsername (str) – username for HPC backend

• hpcPassword (str) – password for HPC backend

• printJob (str) – prints the Job infortmation if set to True

Returns:

The new job instance that was initialized

Return type:

Job

create_job_by_ui(input_params=None, defaultJob='hello_world', defaultDataFolder='./', defaultRemoteResultFolder=None)

Displays the job submission UI.

Parameters:

• defaultJob (str) – Stores the default job that shows up on the UI

• defaultDataFolder (str) – Stores the default input folder that shows up on the UI

• defaultRemoteResultFolder (str) – Stores the default output folder that shows up on the UI

enable_jupyter()

Sets up jupyter environment in jupyterhubHost

encrypt_token(token)

Encrypts the token using host variable.

Parameters:

token (str) – User/Environment provided token.

get_job_by_id(id=None, verbose=True)

Returns Job object with the specified id

Parameters:

id (int) – Job id

Returns:

Job object with the specified id otherwise None

Return type:

Job

get_jupyterhubHost()

Gets the jupyterhub host(str) from the user.

get_latest_created_job()

Return the current job instance

Returns:

Latest Job object instance

Return type:

Job

get_slurm_usage(raw=False)

Prints slurm usage

Parameters:

raw (bool) – set to True if you want the raw output

Returns:

Raw output if raw=True otherwise its printed or displayed directly into the interface

Return type:

JSON

get_user_jupyter_globus()

Return the current job instance

Returns:

Latest Job object instance

Return type:

Job

host_token_login(token)

Gets the host(str), encrypts the token(str) and calls login.

Parameters:

token (str) – User/Environment provided token.

is_login()

Checks whether jupyterhubApi token exists or not

Returns:

jupyterhubAPI existence check

Return type:

bool

list_container(raw=False)

Prints a list of containers that the server supports

Parameters:

raw (bool) – set to True if you want the raw output

Returns:

Raw output if raw=True otherwise its printed or displayed directly into the interface

Return type:

JSON

list_git(raw=False)

Prints a list of Git projects that the server supports

Parameters:

raw (bool) – set to True if you want the raw output

Returns:

Raw output if raw=True otherwise its printed or displayed directly into the interface

Return type:

JSON

list_hpc(raw=False)

Prints a list of hpc resources that the server supports

Parameters:

raw (bool) – set to True if you want the raw output

Returns:

Raw output if raw=True otherwise its printed or displayed directly into the interface

Return type:

JSON

list_info(list_maintainer=False, list_container=False)

Calls cybergis_compute_client.CyberGISCompute.CyberGISCompute.list_git(), cybergis_compute_client.CyberGISCompute.CyberGISCompute.list_hpc(), cybergis_compute_client.CyberGISCompute.CyberGISCompute.list_job() with options to call cybergis_compute_client.CyberGISCompute.CyberGISCompute.list_maintainer() and cybergis_compute_client.CyberGISCompute.CyberGISCompute.list_container().

Parameters:

• list_maintainer (bool) – set to True if you want to call list_maintainer

• list_container (bool) – set to True of you want to call list

list_job(raw=False)

Prints a list of jobs that were submitted

Parameters:

raw (bool) – set to True if you want the raw output

Returns:

Raw output if raw=True otherwise its printed or displayed into the interface

Return type:

JSON

list_jupyter_host(raw=False)

Prints a list of jupyter hosts that the server supports

Parameters:

raw (bool) – set to True if you want the raw output

Returns:

Raw output if raw=True otherwise its printed or displayed directly into the interface

Return type:

JSON

list_maintainer(raw=False)

prints a list of maintainers that the server supports

Parameters:

raw (bool) – set to True if you want the raw output

Returns:

Raw output if raw=True otherwise its printed or displayed directly into the interface

Return type:

JSON

login(manualLogin=False, manualHost=None, verbose=True)

Authenticates the client’s jupyterhubApiToken and gives them access to CyberGISCompute features

Parameters:

manualLogin (bool) – set to True if env variable and file login modes are not available

Todo

Document exceptions/errors raised.

login_json()

Checks for json file and calls login_token function.

login_manual()

Asks for token and host from user and calls login_token function.

login_token()

Saves username(str) and token(str).

save_token()

Writes token(str) to json file.

set_username()

Authenticates the token(str) and saves the username(str).

show_ui(simple=False, input_params=None, defaultJob='hello_world', defaultDataFolder='./', defaultRemoteResultFolder=None, jupyterhubApiToken=None)

Displays the job submission UI

Parameters:

• defaultJob (str) – Stores the default job that shows up on the UI

• defaultDataFolder (str) – Stores the default input folder that shows up on the UI

• defaultRemoteResultFolder (str) – Stores the default output folder that shows up on the UI

Returns:

None

cybergis_compute_client.Job module

class cybergis_compute_client.Job.Job(maintainer=None, hpc=None, id=None, hpcUsername=None, hpcPassword=None, client=None, isJupyter=None, jupyterhubApiToken=None, printJob=True)

Job class

client

Client that this job requests information from

Type:

obj

maintainer

Maintainer pool that this job is in

Type:

obj

isJupyter

Whether or not this is running in Jupyter

Type:

bool

jupyterhubApiToken

API token needed to send requests using the JupyterHub API

Type:

str

id

Id assigned to this job by the client

Type:

str

hpc

HPC that this job will be submitted to

Type:

str

basicEventTypes = ['JOB_QUEUED', 'JOB_REGISTERED', 'JOB_INIT', 'GLOBUS_TRANSFER_INIT_SUCCESS', 'JOB_ENDED', 'JOB_FAILED']

download_result_folder_by_globus(localPath=None, localEndpoint=None, remotePath=None, raw=False)

Downloads the folder with results from the job using Globus

Parameters:

• remotePath (string) – Path to the remote result folder

• raw (bool) – If the function should return the

• from the client (output) –

Returns:

Output from the client when downloading the results using globus. Only returned when raw is true.

Return type:

dict

Raises:

• Exception – If the job ID is None

• Exception – If the key ‘resultFolder’ is not returned with status

• Exception – If the result folder is formatted improperly

events(raw=False, basic=True, refreshRateInSeconds=10)

While the job is running, display the events generated by the client

Parameters:

• raw (bool) – If true, return a list of the events

• by status (generated) –

• liveOutput (bool) –

• basic (bool) – If true, exclude non-basicEventType events

• RefreshRateInSeconds (int) – Number of seconds to wait before

• status (refreshing) –

Todo

Modify function to include liveOutput or remove it from the arguments

logs(raw=False, liveOutput=True, refreshRateInSeconds=15)

While the job is running, display the logs generated by the client.

Parameters:

• raw (bool) – If true, return a list of the events

• by status (generated) –

• liveOutput (bool) –

• RefreshRateInSeconds (int) – Number of seconds to wait

• refreshing status (before) –

Returns:

List of logs generated by the client. Only returned if raw is true.

Return type:

list

Todo

Modify function to include liveOutput or remove it from the arguments

result_folder_content()

Returns the results from the job

Returns:

Results from running the job

Return type:

dict

Raises:

Exception – If the id is None

set(localExecutableFolder=None, localDataFolder=None, localResultFolder=None, param=None, env=None, slurm=None, printJob=True)

PUT requests information about this job to the client so it can be submitted to the hpc. Displays information about this job unless specified otherwise.

Parameters:

• executableFolder (str) – Path of the executable folder

• dataFolder (str) – Path of the data folder

• resultFolder (str) – Path of the result folder

• param (dict) – Rules for input data

• env (dict) – Enviorment variables required by the appliation

• slurm (dict) – Slurm input rules

• printJob (bool) – If the status of the job should be printed

status(raw=False)

Displays the status of this job, and returns it if specified.

Parameters:

raw (bool) – If information about this job should be returned

Returns:

Infomation about this job returned by the client. This includes the job’s ‘id’, ‘hpc’, ‘executableFolder’, ‘dataFolder’, ‘resultFolder’, ‘param’, ‘slurm’, ‘userId’, ‘maintainer’, ‘createdAt’, and ‘events’

Return type:

dict

Raises:

Exception – If the ‘id’ attribute is None

submit()

Submits this job to the client, and prints the output

Returns:

This job

Return type:

Job

cybergis_compute_client.MarkdownTable module

class cybergis_compute_client.MarkdownTable.MarkdownTable

static render(data, headers)

cybergis_compute_client.UI module

class cybergis_compute_client.UI.UI(compute, defaultJobName='hello_world', defaultDataFolder='./', defaultRemoteResultFolder=None)

UI class. .. note:

Many UI elements use an internal `on_change`
function or `on_click` function. If you click the `[source]`
next to the function, it will give that information.

compute

Instance of CyberGISCompute

style

Style of each widget (specifically width)

Type:

dict

layout

Widget layout

Type:

obj

jobs

Jobs being managed currently

Type:

list

hpcs

HPCs the jobs are being submitted to

Type:

list

defaultJobName

Name that jobs are given by default

Type:

string

defaultRemoteResultFolder

Default remote location that results are saved to.

Type:

string

defaultDataFolder

Default folder that data will be saved to

Type:

string

slurm_configs

Default configurations for slurm

Type:

list

slurm_integer_configs

Slurm configurations that can be stored as integers

Type:

list

slurm_integer_storage_unit_config

Slurm configurations related to storage

Type:

list

slurm_integer_time_unit_config

Slurm configurations related to time units

Type:

list

slurm_integer_none_unit_config

Slurm configurations related to units other than time

Type:

list

slurm_string_option_configs

Slurm configurations for string operations

Type:

list

globus_filename

Output filename submitted to Globus (set when entered by the user)

Type:

string

jupyter_globus

Information about where the output data will be stored (container_home_path, endpoint, root_path)

Type:

dict

get_data()

Get data about the job submitted (template, computing resource used, slurm rules, param rules, user email)

Returns:

Information about the job submitted (template, computing resource used, slurm rules, param rules, user email)

Return type:

dict

init()

Initialization helper function that sets default arguments. Runs when the UI is rendered.

makeNameSafe(text)

onComputingResourceDropdownChange()

onDownloadButtonClick()

onFolderDownloadButtonClick(folder)

onJobDropdownChange()

onJobEntryButtonClick(job_id)

onLoadMoreClick()

onNextPageButton(totalJobs)

onPrevPageButton()

onRenameJobButton(folder, wdgt)

onSubmitButtonClick()

onSubmitNewButtonClick()

render()

Render main UI by initializing, rendering, and displaying each component

renderAnnouncements()

Displays announcements if the announcement route exists and there are any

renderAutoDownload()

Automatically downloading results once a job is finished

renderComponents()

Render each section of the UI

renderComputingResource()

Display computing resources in a dropdown for the user to select

renderDescription()

Display information about the job (job name, job description, HPC name, HPC description, estimated runtime)

renderDownload()

Creates the components of the download section

renderEmail()

Displays a checkbox that lets the user receive an email on job status and input their email.

renderFolders()

Display a user’s folders with ability to download and rename them

renderJobTemplate()

Display a dropdown of jobs to run. Update jobTemplate when the dropdown changes.

renderLoadMore()

Renders a button to load more recently submitted jobs.

renderNaming()

Displays a box to toggle naming the job being submitted and provides a text entry for the user to input the name

renderParam()

Displays input areas for the job parameters

renderRecentlySubmittedJobs()

Display the jobs most recently submitted by the logged in user, allows user to restore these jobs.

renderResultCancel()

renderResultEvents()

Display any events that occured while the job was being processed.

renderResultLogs()

Display when the job is finished and rerender the download section when it is.

renderResultStatus()

Display the status of the job.

renderSlurm()

Configures Slurm input rules (default value, min, m), allows the user to input custom input rules if they want.

renderSubmit()

Render submit button. If the job has been submitted, display that, otherwise display the submit button.

renderSubmitNew()

Render submit new button, which allows the user to return the SDK to a pre-submission state so they can submit successive jobs.

renderUploadData()

Lets the user select the upload data location from a file chooser.

rerender(components=[])

Clears and renders the specified components :param components: components to be rerendered :type components: list

secondsToTime(seconds)

Helper function that turns seconds into minutes, days, hours format

unitTimeToSecond(unit, time)

Helper function that turns time in a specific unit into seconds

Parameters:

• unit (string) – The unit of the time being passed (Minutes, Hours, or Days)

• time (int) – The time in that specific unit

Returns:

the amount of time in the given unit

Return type:

int

cybergis_compute_client.Zip module

This module exposes Zip class which creates an in-memory zip object to avoid disk access

Example

zip = Zip()

class cybergis_compute_client.Zip.Zip

Zip class

An interface that creates an in-memory zip object to avoid disk access

in_memory_zip

A BytesIO in-memory file

Inits Zip with in_memory_zip

append(filename_in_zip, file_contents)

Appends a file with name filename_in_zip and contents of file_contents to the in-memory zip.

Parameters:

• filename_in_zip (str) – Name of the zip_file

• file_contents (str) – Contents that need to be written to the zip_file

Returns:

this Zip

Return type:

Zip

mkdir(filedir_in_zip)

Creates a directory with the name filedir_in_zip

Parameters:

fildir_in_zip (str) – Name of the zip_directory

Returns:

this Zip

Return type:

Zip

read()

Reads the contents of the in-memory zip.

Returns:

contents of the in-memory zip

Return type:

str

write(filename)

Writes the in-memory zip to a file

Parameters:

filename (str) – Name of the file that needs to be written