5.3) Job Scheduling On ARC2

On ARC2 the batch system installed is Son of Grid Engine (SGE), with some locally developed and implemented features. We will limit discussion to this batch scheduler as it is configured on ARC2, but the principles will apply to other batch systems.

SGE is a sophisticated scheduler that is used to define usage policies, control maximum resource request limits, ensure distribution of resources according to a desired policy and can produce detailed usage and accounting information.

On ARC2 a “fair share” scheduling policy is adopted. It attempts to allocate resources fairly between projects (in this case projects==Faculties) and between users within these projects. The scheduler takes into account the current and past usage of a Faculty and prioritises the Faculty with the lower usage. The same applies when comparing users in the same Faculty. The usage is calculated via a combination of CPU hours and memory usage and has a half-life of a 7 days.

With the current configuration, SGE then reserves resources for the top 128 jobs, i.e. it ensures that these jobs have their start time booked in the future. The command,

$ qsched –a

can be used to generate a list of the anticipated start times of these jobs. Note, that qsched is a locally developed command and is not a standard SGE command.