Package iceprod :: Package server :: Package plugins :: Module dag :: Class TaskQ

Class TaskQ

grid.iGrid --+
             |
            TaskQ

Known Subclasses:

This class represents a job that executes in multiple parts using a DAG.

Instance Methods

[hide private]

__init__(self)

source code

SubmitTask(self, job)
Write submit file to a file.

source code

GetFiles(self, job, td, args, idx=False, iter=False, catalog=False)

source code

FindFile(self, steering, td, catalog, file)

source code

WriteFileManifest(self, job, filename, input, output, notes)

source code

QueueJobs(self, db, maxjobs, grid_id, jobs_at_once=20, fifo=True, debug=0)

source code

jobfinalize(self, dataset_id, job_id, job, status='OK', clear_errors=True)
Update status for job source code

reset_old_jobs(self, grid_id, maxidletime, maxruntime, maxsubmittime, maxcopytime, maxfailures, maxevicttime, keepalive)
reset status of jobs that where queued but who's status has not changed in more that maxtime minutes

source code

GetResetJobs(self, grid_id, max_reset=50)

source code

GetFinishedJobs(self, grid_id, max_reset=50)

source code

GetActiveJobs(self, grid_id)

source code

GetProcessingJobs(self, grid_id, max_reset=50)

source code

GetQueuedJobs(self, grid_id, max_reset=50)

source code

EnumerateParentNodes(self, steering, td)

source code

GetArguments(self, job, td, idx=False, iter=False, output='str') source code

SetArguments(self, job, td, idx=False, iter=False)
Add DAG arguments for job

source code

ArgStrToDict(self, argstr)

source code

getAdditionalOptions(self, job)

source code

CleanQ(self, jobs=None)
Interface: Check status of job/cluster in queuing system.

source code

Clean(self, jobdict, force=False)
Interface: clean submit directory

source code

Inherited from grid.iGrid: AddArgOption, AddEnv, AddParam, CheckJobStatus, CheckQ, CleanDir, FileName, GetArgOpt, GetArgOptions, GetClusterId, GetEnv, GetHost, GetInitialdir, GetJob, GetJobIds, GetJobList, GetLogs, GetMetadataPath, GetMonitorDB, GetParam, GetParamKeys, GetPort, GetProduction, GetQueue, GetRootDir, GetSimDBKey, GetSteering, GetSubmitHost, GetURL, IsUrl, PopJob, PostCopy, PushJob, QRemove, SelfPickle, SetCopyClass, SetHost, SetInitialdir, SetMetadataPath, SetMonitorDB, SetPort, SetProduction, SetQueue, SetRootDir, SetSimDBKey, SetStatus, SetSteering, SetSubmitHost, SetURL, Submit, Suffix, UpdateMetadata, format_arg, urlcopy, wait

Inherited from grid.iGrid (private): _submit

Class Variables

[hide private]

Inherited from grid.iGrid: CopyStatusEnum

Method Details

[hide private]

init(self)
(Constructor)

source code

Overrides: grid.iGrid.__init__

SubmitTask(self, job)

source code

Write submit file to a file.

Parameters:

job - i3Job object
config_file - path to file where submit file will be written

QueueJobs(self, db, maxjobs, grid_id, jobs_at_once=20, fifo=True, debug=0)

source code

Overrides: grid.iGrid.QueueJobs

jobfinalize(self, dataset_id, job_id, job, status=`'OK'`, clear_errors=True)

source code

Update status for job

Parameters:

dataset_id - dataset index
job_id - process number within dataset

Overrides: grid.iGrid.jobfinalize

reset_old_jobs(self, grid_id, maxidletime, maxruntime, maxsubmittime, maxcopytime, maxfailures, maxevicttime, keepalive)

source code

reset status of jobs that where queued but who's status has not changed in more that maxtime minutes

Parameters:

grid_id - id of current cluster
maxruntime - maximum run time for jobs
maxsubmittime - maximum submit time for jobs
maxcopytime - maximum time for jobs to be in 'copying' state
maxfailures - maximum number of time a job is allowd to fail
keepalive - how often should server expect to hear from jobs

Overrides: grid.iGrid.reset_old_jobs

Interface: Check status of job/cluster in queuing system.

Overrides: grid.iGrid.CleanQ: (inherited documentation)

Clean(self, jobdict, force=False)

source code

Interface: clean submit directory

Overrides: grid.iGrid.Clean

Class TaskQ

__init__(self) (Constructor)

SubmitTask(self, job)

QueueJobs(self, db, maxjobs, grid_id, jobs_at_once=20, fifo=True, debug=0)

jobfinalize(self, dataset_id, job_id, job, status='OK', clear_errors=True)

reset_old_jobs(self, grid_id, maxidletime, maxruntime, maxsubmittime, maxcopytime, maxfailures, maxevicttime, keepalive)

GetResetJobs(self, grid_id, max_reset=50)

GetFinishedJobs(self, grid_id, max_reset=50)

GetActiveJobs(self, grid_id)

GetProcessingJobs(self, grid_id, max_reset=50)

GetQueuedJobs(self, grid_id, max_reset=50)

CleanQ(self, jobs=None)

Clean(self, jobdict, force=False)

init(self)
(Constructor)

jobfinalize(self, dataset_id, job_id, job, status=`'OK'`, clear_errors=True)