lwr.managers Module

Job Managers

lwr.managers.base Module

Base Classes and Infrastructure Supporting Concret Manager Implementations.

lwr.managers.base.base_drmaa Module

class lwr.managers.base.base_drmaa.BaseDrmaaManager(name, app, **kwds)

Bases: lwr.managers.base.external.ExternalBaseManager

shutdown()

lwr.managers.base.directory Module

class lwr.managers.base.directory.DirectoryBaseManager(name, app, **kwds)

Bases: lwr.managers.base.BaseManager

return_code(job_id)

stderr_contents(job_id)

stdout_contents(job_id)

lwr.managers.base.external Module

class lwr.managers.base.external.ExternalBaseManager(name, app, **kwds)

Bases: lwr.managers.base.directory.DirectoryBaseManager

Base class for managers that interact with external distributed resource managers.

clean(job_id)

get_status(job_id)

kill(job_id)

setup_job(input_job_id, tool_id, tool_version)

class lwr.managers.base.BaseManager(name, app, **kwds)

Bases: lwr.managers.ManagerInterface

clean(job_id)

job_directory(job_id)

system_properties()

class lwr.managers.base.JobDirectory(staging_directory, job_id, lock_manager=None)

Bases: lwr.lwr_client.job_directory.RemoteJobDirectory

calculate_path(remote_path, input_type)

Verify remote_path is in directory for input_type inputs and create directory if needed.

contains_file(name)

delete()

exists()

has_metadata(metadata_name)

load_metadata(metadata_name, default=None)

lock(name='.state')

make_directory(name)

open_file(name, mode='wb')

outputs_directory_contents()

read_file(name, default=None)

remove_file(name)

Quietly remove a job file.

remove_metadata(metadata_name)

setup()

store_metadata(metadata_name, metadata_value)

working_directory_contents()

write_file(name, contents)

lwr.managers.base.get_id_assigner(assign_ids)

lwr.managers.base.get_mapped_file(directory, remote_path, allow_nested_files=False, local_path_module=<module 'posixpath' from '/home/docs/checkouts/readthedocs.org/user_builds/lwr/envs/latest/lib/python2.7/posixpath.pyc'>, mkdir=True)

>>> import ntpath
>>> get_mapped_file(r'C:\lwr\staging\101', 'dataset_1_files/moo/cow', allow_nested_files=True, local_path_module=ntpath, mkdir=False)
'C:\\lwr\\staging\\101\\dataset_1_files\\moo\\cow'
>>> get_mapped_file(r'C:\lwr\staging\101', 'dataset_1_files/moo/cow', allow_nested_files=False, local_path_module=ntpath)
'C:\\lwr\\staging\\101\\cow'
>>> get_mapped_file(r'C:\lwr\staging\101', '../cow', allow_nested_files=True, local_path_module=ntpath, mkdir=False)
Traceback (most recent call last):
Exception: Attempt to read or write file outside an authorized directory.

lwr.managers.queued Module

class lwr.managers.queued.QueueManager(name, app, **kwds)

Bases: lwr.managers.unqueued.Manager

A job manager that queues up jobs directly (i.e. does not use an external queuing software such PBS, SGE, etc...).

launch(job_id, command_line, submit_params={}, dependencies_description=None, env=[])

manager_type = 'queued_python'

run_next()

Run the next item in the queue (a job waiting to run).

shutdown()

lwr.managers.queued_drmaa Module

class lwr.managers.queued_drmaa.DrmaaQueueManager(name, app, **kwds)

Bases: lwr.managers.base.base_drmaa.BaseDrmaaManager

DRMAA backed queue manager.

launch(job_id, command_line, submit_params={}, dependencies_description=None, env=[])

manager_type = 'queued_drmaa'

lwr.managers.queued_external_drmaa Module

class lwr.managers.queued_external_drmaa.ExternalDrmaaQueueManager(name, app, **kwds)

Bases: lwr.managers.base.base_drmaa.BaseDrmaaManager

DRMAA backed queue manager.

get_status(job_id)

launch(job_id, command_line, submit_params={}, dependencies_description=None, env=[])

manager_type = 'queued_external_drmaa'

lwr.managers.queued_condor Module

class lwr.managers.queued_condor.CondorQueueManager(name, app, **kwds)

Bases: lwr.managers.base.external.ExternalBaseManager

Job manager backend that plugs into Condor.

get_status(job_id)

launch(job_id, command_line, submit_params={}, dependencies_description=None, env=[])

manager_type = 'queued_condor'

lwr.managers.queued_pbs Module

class lwr.managers.queued_pbs.PbsQueueManager(name, app, **kwds)

Bases: lwr.managers.base.BaseManager

Placeholder for PBS-python backed queue manager. Not yet implemented, for many situations this would be used the DRMAA or CLI+Torque managers may be better choices or at least stop gaps.

manager_type = 'queued_pbs'

lwr.managers.unqueued Module

class lwr.managers.unqueued.Manager(name, app, **kwds)

Bases: lwr.managers.base.directory.DirectoryBaseManager

A simple job manager that just directly runs jobs as given (no queueing). Preserved for compatibilty with older versions of LWR client code where Galaxy is used to maintain queue (like Galaxy’s local job runner).

get_status(job_id)

kill(job_id)

launch(job_id, command_line, submit_params={}, dependencies_description=None, env=[])

manager_type = 'unqueued'

setup_job(input_job_id, tool_id, tool_version)

lwr.managers.stateful Module

class lwr.managers.stateful.ActiveJobs(manager)

Bases: object

Keeps track of active jobs (those that are not yet “complete”). Current implementation is file based, but could easily be made database-based instead.

TODO: Keep active jobs in memory after initial load so don’t need to repeatedly hit disk to recover this information.

activate_job(job_id)

active_job_ids()

deactivate_job(job_id)

class lwr.managers.stateful.ManagerMonitor(stateful_manager)

Bases: object

Monitors active jobs of a StatefulManagerProxy.

shutdown()

class lwr.managers.stateful.StatefulManagerProxy(manager, **manager_options)

Bases: lwr.managers.ManagerProxy

get_status(job_id)

Compute status used proxied manager and handle state transitions and track additional state information needed.

handle_remote_staging(job_id, staging_config)

launch(job_id, *args, **kwargs)

name

set_state_change_callback(state_change_callback)

setup_job(*args, **kwargs)

shutdown()

lwr.managers.stateful.new_thread_for_manager(manager, name, target, daemon)

lwr.managers.status Module

lwr.managers.status.is_job_done(status)

Does the supplied status correspond to a finished job (done processing).

lwr.managers.util Module

This module and its submodules contains utilities for running external processes and interfacing with job managers. This module should contain functionality shared between Galaxy and the LWR.

lwr.managers.staging Module

This module contains the code that allows the LWR to stage file’s during preprocessing (currently this means downloading or copying files) and then unstage or send results back to client during postprocessing.

lwr.managers.staging.preprocess Module

lwr.managers.staging.preprocess.preprocess(job_directory, setup_actions)

lwr.managers.staging.postprocess Module

lwr.managers.staging.postprocess.postprocess(job_directory)

class lwr.managers.ManagerInterface

Bases: object

Defines the interface to various job managers.

clean(job_id)

Delete job directory and clean up resources associated with job with id job_id.

get_status(job_id)

Return status of job as string, currently supported statuses include ‘cancelled’, ‘running’, ‘queued’, and ‘complete’.

job_directory(job_id)

Return a JobDirectory abstraction describing the state of the job working directory.

kill(job_id)

End or cancel execution of the specified job.

launch(job_id, command_line, submit_params={}, dependencies_description=None, env=[])

Called to indicate that the client is ready for this job with specified job id and command line to be executed (i.e. run or queue this job depending on implementation).

return_code(job_id)

Return integer indicating return code of specified execution or LWR_UNKNOWN_RETURN_CODE.

setup_job(input_job_id, tool_id, tool_version)

Setup a job directory for specified input (galaxy) job id, tool id, and tool version.

stderr_contents(job_id)

After completion, return contents of stderr associated with specified job.

stdout_contents(job_id)

After completion, return contents of stdout associated with specified job.

class lwr.managers.ManagerProxy(manager)

Bases: object

Subclass to build override proxy a manager and override specific functionality.

clean(*args, **kwargs)

get_status(*args, **kwargs)

job_directory(*args, **kwargs)

kill(*args, **kwargs)

launch(*args, **kwargs)

return_code(*args, **kwargs)

setup_job(*args, **kwargs)

shutdown()

Optional.

stderr_contents(*args, **kwargs)

stdout_contents(*args, **kwargs)

system_properties()

lwr.core Module

class lwr.core.LwrApp(**conf)

Bases: object

shutdown()

lwr.daemon Module

class lwr.daemon.ArgumentParser(**kwargs)

Bases: optparse.OptionParser

add_argument(*args, **kwargs)

parse_args()

class lwr.daemon.LwrConfigBuilder(args=None, **kwds)

Bases: object

Generate paste-like configuration from supplied command-line arguments.

load()

classmethod populate_options(clazz, arg_parser)

setup_logging()

to_dict()

class lwr.daemon.LwrManagerConfigBuilder(args=None, **kwds)

Bases: lwr.daemon.LwrConfigBuilder

classmethod populate_options(clazz, arg_parser)

to_dict()

lwr.daemon.app_loop(args)

lwr.daemon.load_lwr_app(config_builder, config_env=False, log=None, **kwds)

lwr.daemon.main()

lwr.scripts Module

This module contains entry points into various LWR scripts. Corresponding shell scripts that setup the deployment specific environments and then delegate to these Python scripts can be found in LWR_ROOT/scripts.

lwr.scripts.chown_working_directory Module

lwr.scripts.chown_working_directory.main()

lwr.scripts.drmaa_kill Module

lwr.scripts.drmaa_kill.main()

lwr.scripts.drmaa_launch Module

lwr.scripts.drmaa_launch.main()

lwr.scripts.lwr_submit Module

lwr.scripts.lwr_submit.main()

lwr.scripts.lwr_submit.manager_from_args(config_builder)

lwr.scripts.lwr_submit.wait_for_job(manager, job_config, poll_time=2)

lwr.scripts.mesos_executor Module

class lwr.scripts.mesos_executor.LwrExecutor

Bases: object

frameworkMessage(driver, message)

launchTask(driver, task)

lwr.scripts.mesos_executor.run_executor()

lwr.scripts.mesos_framework Module

lwr.scripts.mesos_framework.main()

lwr.web Module

The code explicitly related to the LWR web server can be found in this module and its submodules.

lwr.web.framework Module

Tiny framework used to power LWR application, nothing in here is specific to running or staging jobs. Mostly deals with routing web traffic and parsing parameters.

class lwr.web.framework.Controller(response_type='OK')

Bases: object

Wraps python functions into controller methods.

body(result)

class lwr.web.framework.FileIterator(path)

Bases: six.Iterator

class lwr.web.framework.RoutingApp

Bases: object

Abstract definition for a python web application.

add_route(route, controller, **args)

lwr.web.framework.build_func_args(func, *arg_dicts)

lwr.web.framework.file_response(path)

lwr.web.routes Module

class lwr.web.routes.LwrController(**kwargs)

Bases: lwr.web.framework.Controller

class lwr.web.routes.LwrDataset(id)

Bases: object

Intermediary between lwr and objectstore.

lwr.web.wsgi Module

class lwr.web.wsgi.LwrWebApp(lwr_app)

Bases: lwr.web.framework.RoutingApp

Web application for LWR web server.

lwr.web.wsgi.app_factory(global_conf, **local_conf)

Returns the LWR WSGI application.

lwr.messaging Module

This module contains the server-side only code for interfacing with message queues. Code shared between client and server can be found in submodules of lwr.lwr_client.

lwr.messaging.bind_amqp Module

lwr.messaging.bind_amqp.bind_manager_to_queue(manager, queue_state, connection_string, conf)

lwr.messaging.bind_amqp.get_exchange(connection_string, manager_name, conf)

class lwr.messaging.QueueState

Bases: object

Passed through to event loops, should be “non-zero” while queues should be active.

deactivate()

lwr.messaging.bind_app(app, queue_id, connect_ssl=None)

lwr.mesos Module

This module and submodules contain code for interfacing the Apache Mesos framework.

lwr.mesos.framework Module

class lwr.mesos.framework.LwrScheduler(executor, manager_options, mesos_url)

Bases: object

frameworkMessage(driver, executorId, slaveId, message)

handle_setup_message(body, message)

registered(driver, frameworkId, masterInfo)

resourceOffers(driver, offers)

statusUpdate(driver, update)

lwr.mesos.framework.run(master, manager_options, config)

lwr.mesos.ensure_mesos_libs()

Raise import error if mesos is not actually available. Original import errors above supressed because mesos is meant as an optional dependency for the LWR.

lwr.app Module

Deprecated module for wsgi app factory. LWR servers should transition to lwr.web.wsgi:app_factory.

lwr.app.app_factory(global_conf, **local_conf)

Returns the LWR WSGI application.

lwr.manager_endpoint_util Module

Composite actions over managers shared between HTTP endpoint (routes.py) and message queue.

lwr.manager_endpoint_util.full_status(manager, job_status, job_id)

lwr.manager_endpoint_util.setup_job(manager, job_id, tool_id, tool_version)

Setup new job from these inputs and return dict summarizing state (used to configure command line).

lwr.manager_endpoint_util.submit_job(manager, job_config)

Launch new job from specified config. May have been previously ‘setup’ if ‘setup_params’ in job_config is empty.

lwr.manager_factory Module

lwr.manager_factory.build_managers(app, conf)

Takes in a config file as outlined in job_managers.ini.sample and builds a dictionary of job manager objects from them.

lwr.locks Module

class lwr.locks.LockManager(lockfile=None)

free_lock(path)

get_lock(path)

Get a job lock corresponding to the path - assumes parent directory exists but the file itself does not.

lwr.tools Module

Tools

lwr.tools.toolbox Module

class lwr.tools.toolbox.InputsValidator(command_validator, config_validators)

Bases: object

validate_command(job_directory, command)

validate_config(job_directory, name, path)

class lwr.tools.toolbox.SimpleToolConfig(tool_el, tool_path)

Bases: lwr.tools.toolbox.ToolConfig

Abstract description of a Galaxy tool loaded from a toolbox with the tool tag not containing a guid, i.e. one not from the toolshed.

class lwr.tools.toolbox.ToolBox(path_string)

Bases: object

Abstraction over a tool config file largely modelled after Galaxy’s shed_tool_conf.xml. Hopefully over time this toolbox schema will be a direct superset of Galaxy’s with extensions to support simple, non-toolshed based tool setups.

get_tool(id)

class lwr.tools.toolbox.ToolConfig

Bases: object

Abstract description of a Galaxy tool.

get_tool_dir()

inputs_validator

class lwr.tools.toolbox.ToolShedToolConfig(tool_el, tool_path)

Bases: lwr.tools.toolbox.SimpleToolConfig

Abstract description of a Galaxy tool loaded from a toolbox with the tool tag, i.e. one from the toolshed.

<tool file=”../shed_tools/gvk.bx.psu.edu/repos/test/column_maker/f06aa1bf1e8a/column_maker/column_maker.xml” guid=”gvk.bx.psu.edu:9009/repos/test/column_maker/Add_a_column1/1.1.0”>

<tool_shed>gvk.bx.psu.edu:9009</tool_shed> <repository_name>column_maker</repository_name> <repository_owner>test</repository_owner> <installed_changeset_revision>f06aa1bf1e8a</installed_changeset_revision <id>gvk.bx.psu.edu:9009/repos/test/column_maker/Add_a_column1/1.1.0</id> <version>1.1.0</version>

</tool>

lwr.tools.authorization Module

class lwr.tools.authorization.AllowAnyAuthorization

Bases: object

authorize_config_file(job_directory, name, path)

authorize_execution(job_directory, command_line)

authorize_setup()

authorize_tool_file(name, contents)

class lwr.tools.authorization.AllowAnyAuthorizer

Bases: object

Allow any, by default LWR is assumed to be secured using a firewall or private_token.

ALLOW_ANY_AUTHORIZATION = <lwr.tools.authorization.AllowAnyAuthorization object>

get_authorization(tool_id)

class lwr.tools.authorization.ToolBasedAuthorization(tool)

Bases: lwr.tools.authorization.AllowAnyAuthorization

authorize_config_file(job_directory, name, path)

authorize_execution(job_directory, command_line)

authorize_setup()

authorize_tool_file(name, contents)

class lwr.tools.authorization.ToolBasedAuthorizer(toolbox)

Bases: object

Work In Progress: Implement tool based white-listing of what jobs can run and what those jobs can do.

get_authorization(tool_id)

lwr.tools.authorization.get_authorizer(toolbox)

lwr.tools.validator Module

class lwr.tools.validator.ExpressionValidator(xml_el)

Bases: object

validate(job_directory, string)

class lwr.tools.ToolBox(path_string)

Bases: object

Abstraction over a tool config file largely modelled after Galaxy’s shed_tool_conf.xml. Hopefully over time this toolbox schema will be a direct superset of Galaxy’s with extensions to support simple, non-toolshed based tool setups.

get_tool(id)