Library Reference

gridengineapp.qstat(effective_user=None, job_list=None)

Get status of all jobs in the job_list belonging to the given user:

import getpass
user = getpass.getuser()
job_info = qstat(user, "dm_38044_*")

Parameters

• effective_user (str) – The user ID.

• job_list (str) – Can be model version IDs, or a job name, or a job name with a wildcard. See man sge_types.

Returns

Information about the jobs.

Return type

List[FlyWeightJob]

gridengineapp.qstat_short(effective_user=None)

Calling qstat without -j gets a much smaller result that just has job information.

Parameters

effective_user (str) – Request qstat for this user’s jobs. Default is to use the current user.

Returns

A list of jobs with less information than what qstat -j shows.

Return type

List[MiteWeightJob]

gridengineapp.qsub(template, command)

Runs a qsub command with a template. By using the template, as described below, this function makes it easier to create a default set of qsub settings and overwrite them, job by job, without doing string manipulation.

We can either try to put a super-thoughtful interface on qsub, or we let the user manage its arguments. This focuses on making it a little easier to manage arguments with the template.

Parameters

• template – Suitable for template_to_args.

• command (List[str]) – A list of strings to pass to qsub.

Returns

The model version ID. It’s a str because it isn’t an int. Can you add 37 to it? No. Is it ordered? That’s not guaranteed. Does it sometimes have a “.1” at the end? Yes. That makes it a string.

Return type

str

The template argument is a dictionary where each entry corresponds to an argument to qsub. Here are the rules:

• If the argument is a flag with no argument, set template[flag] = None.

• If the argument is a flag with a true or false, set template[flag] = True, or False.

• If the argument is a comma-separated list, set the value to a list, template["dc"] = ["LD_LIBRARY_PATH", "CC"].

• If the argument is a set of key-value pairs, set the value to a dictionary, template["l"] = dict(m_mem_free="16G", fthreads=16).

gridengineapp.qsub_template()

Basic template for qsub. This means that any flags that can have multiple copies are already included in the data structure. So you can do template["l"]["intel"] without having to check that “l” exists.

template = qsub_template()
template["q"] = "all.q"
template["P"] = "proj_forecasting"
template["l"]["h_rt"] = "12:00:00"
args = template_to_args()
assert "-q all.q" in " ".join(args)

class gridengineapp.FlyWeightJob(job_jsonlike)

Sits on top of the parsed XML to answer job questions. The for_each_member creates a reasonable Pythonic data structure. What’s missing at that point is knowing what tag correspondes to what human information. We layer that here and will add what we need when we need it.

job_dict = None

Dictionary containing all information from qstat.

property status

Set of strings like: idle, running, as a set. This can be in more than one state at a time, such as {"queued", "waiting"}, which we know as qw.

property tasks

FlyWeightTasks for tasks in the job. Can be none.

property name

As given by the -N qsub option.

property job_id

The job ID, as in 2349272.

property task_cnt

How many tasks are associated with this job. Jobs contain tasks, and it’s the tasks that run, have statuses, and have CPU times.

class gridengineapp.FlyWeightTask(task_jsonlike)

Responsible for presenting task-specific information from qstat. Every job contains at least one task. Task arrays have one or more tasks.

task_dict = None

Dictionary containing all information from qstat.

property number

Tasks within a job are numbered from 1.

property status

Status is a set of strings.

property restarted

Bool: Whether this task did restart.

property hostname

Hostname where this task will run, is running, or has run.

class gridengineapp.MiteWeightJob(job_jsonlike)

Like the FlyWeightJob, this represents a Job. This one includes everything in the simplified version of qstat.

job_dict = None

Dictionary with all information from qstat.

property status

Set of strings like: idle, running, as a set. This can be in more than one state at a time, such as {"queued", "waiting"}, which we know as qw.

property tasks

FlyWeightTasks for tasks in the job. Can be none.

property name

As given by the -N qsub option.

property job_id

The job ID, as in 2349272.

property task_cnt

How many tasks are associated with this job. Jobs contain tasks, and it’s the tasks that run, have statuses, and have CPU times.

class gridengineapp.GridParser(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True)

error(message)

Override the base class because it calls sys.exit. This library uses the parser as an internal tool, not just for user interaction. For instance, it’s used in testing, where an exception is more appropriate.

If the status is 0, that means nothing is wrong, and the user requested --help, so, yes, exit.

Otherwise, print a message to standard error and raise an exception.

exception gridengineapp.ArgumentError

An error for command-line arguments.

class gridengineapp.FileEntity(file_path)

Responsible for making a path that is writable for a file.

Parameters

relative_path (Path|str) – Path to the file, relative to a root.

property path

Return a full file path to the file, given the current context.

validate()

Validate by checking file exists.

Returns

None, on success, or a string on error.

mock()

Touch the file into existence.

remove()

Delete, unlink, remove the file. No error if it doesn’t exist.

class gridengineapp.PandasFile(file_path, required_frames=None)

Responsible for validating a Pandas file.

Parameters

• file_path (Path|str) – Path to the file.

• required_frames (Dict[str,set]) – Map from the name of the dataset, as specified by the Pandas key argument, to a list of columns that should be in that dataset.

validate()

Returns

None, on success, or a string on error.

mock()

Touch the file into existence.

class gridengineapp.ShelfFile(file_path, required_keys=None)

Responsible for validating a Python shelf file.

Parameters

• file_path (Path|str) – Path to the file.

• required_keys (Set[str]) – String names of variables to find in the file.

validate()

Validates that there are variables named after the required keys. :returns: None, on success, or a string on error.

mock()

Touch the file into existence.

remove()

Delete, unlink, remove the file. No error if it doesn’t exist.

gridengineapp.check_complete(identify_job, check_done, timeout=3600)

Submit a job and check that it ran. If the job never shows up in the queue, and it didn’t run, that’s a failure. If it shows up in the queue and goes over the timeout, we abandon it, because these are tests.

Parameters

• identify_job (function) – True if it’s this job.

• check_done (function) – True if job is done.

• timeout (float) – How many seconds to wait until calling the job lost.

Returns

None

gridengineapp.entry(app, arg_list=None)

This starts the application. Use it with:

if __name__ == "__main__":
    application = MyApplication()
    entry(application)

Parameters

• app (application.Application) – The main application to run.

• arg_list (Namespace|SimpleNamespace) – Arguments to the command line. This is usually None and is used for testing. Pass this around instead of using sys.argv because pytest makes it hard to set sys.argv.

gridengineapp.execution_ordered(graph)

This iterator orders the nodes such that they go depth-first. This is chosen so that the data has the most locality during computation. It’s not strictly depth-first, but depth-first, given that all predecessors must be complete before a node executes.

exception gridengineapp.NodeMisconfigurationError

Failures whose faults may be due to node misconfiguration.