Overview of Job Scheduling
The scheduler is a process running on the master node which determines when and where jobs will run. It is what is responsible for seeing that your job gets the resources it requested so it can run, and for doing so in a manner which tries to get everyone's jobs scheduled and running in a reasonable amount of time. The following is a simplified overview of the scheduling process. There is a lot of complexity to the problem, but the following should give you a basic understanding and help you to understand why specifying realistic requirements for your job will help reduce the amount of time it spends in the queue waiting to be scheduled.
Jobs submitted by sbatch, etc. get placed into a queue, and the scheduler periodically checks the list of jobs in the queue trying to find resources for them so they can run. Even if the cluster is lightly loaded and there are no other jobs in the queue, this might take a minute or two, but since jobs on the HPC typically run for hours this is a minor overhead. If the cluster is heavily loaded, the jobs might spend hours or even days in the queued state, waiting for the resources needed for the job to become available.
The scheduler basically goes through the list in a FIFO (first in, first out) fashion, that is, jobs are more or less processed in the order in which they are submitted. But this is only a first approximation. Jobs will have differing priorities; on the current UMD HPC clusters, all jobs (except for those in the scavenger partition) start with the same priority, and the priority slowly increases over the time the the job is in the queue and eligible to run.
Jobs also need varying amounts of resources. The scheduler does not know what resources your job actually requires; it only knows what you requested, and it assumes that you know what your job requires. If you do not request enough of a resource, the job might start but will likely fail. E.g., if you did not request enough memory for your job, it very likely will start to run, but will fail at some point (perhaps immediately, perhaps only after many hours) because the scheduler only gave the job the amount of memory that was requested (of the default amount).
On the other hand, you should not request signfificantly more resources than you need, as doing such is inefficient and wasteful of valuable cluster resources and your allocation's SUs. It also will delay the scheduling of your job, as it will in general the more resources you request the longer it will take for the scheduler to find the resources; while this is generally true, requesting unneeded resources will introduce an unneccessary dealy. Usually, you will want to specify a little more than is required, i.e. you do not want your job to fail shortly before it finishes due being a bit short on memory. But neither should your jobs typically be requesting 50% more memory than they need.
Generally, your jobs are charged for resources allocated to the job (which is based on what the job requested), not what the job actually used, since the allcoated resources are reserved for the exclusive use of your job as long as it runs. Therefore it is important that you request an appropriate amount of resources, see:
Walltime is handled slightly differently, as jobs are only charged for the allocated resources for the time for which the job actually ran (as once the job terminates, the scheduler can allocated the resources it used to other job(s)). Again, you will want to request slightly more walltime than you expect your job to require, as you do not wish for a job to be terminated for running out of walltime shortly before the calculation is finished. However, you still should not request excessive amounts of walltime --- doing so may prevent the job from being backfilled thereby causing significant delays in scheduling.
As previously stated, the scheduler schedules jobs in a primarily FIFO within each partition. If there are sufficient idle resources in the partition to meet the requirements of the first job in the queue for that partition, the job will be scheduled and the scheduler will process the next job in that queue. If there are not sufficient idle resources in the partition to meet the requirements of the first job in the queue, the job will remain queued with a reason of "Resources". Any resources which could be used by this top job will be reserved for it until sufficient resources are available for the job to run; otherwise large jobs would effectively be "starved" of resources by smaller jobs "stealing" all idle resources before a sufficient number of idle resources are found to run the large job.
Example
Consider a job which requires 2000 CPU cores.
When that job eventually comes to the top of the queue, lets assume
there are only 1000 idle cores on the cluster; therefore there are
insufficient resources available to run the job, so it will wait
in the queue. As other jobs finish (and release their resources),
number of idle cores will increase. The scheduler will reserve
all of the idle cores for the 2000 core job, until sufficient
resources are available to start the 2000 core job.
This is required because otherwise a constant stream of smaller
jobs, only needing a few cores, could consume all of the idle cores
as they get freed up, and keeping the 2000 core job from ever running.
There is an exception to the above called backfilling. As the scheduler knows the walltime requested for all jobs, it can (assuming these walltimes are somewhat accurate) make estimates about when a can start to run. It therefore can allow some smaller jobs to run using resources "reserved" for the larger job, as long as the smaller job is short enough to complete before the larger job is expected to run.
Example
Returning to the 2000 core job example above, when it reaches the top of the queue there are only 1000 idle cores. The scheduler knows the walltime limits for all of the running jobs, and so can estimate when the remaining resources (i.e. the additional 1000 cores) required to start the 2000 core job will be freed due to the jobs consuming them terminating. Lets assume that the scheduler predicts that the remaining 1000 cores will only become idle in 8 hours; rather than keeping the 1000 cores currently idle to remain idle for the full 8 hours until the 2000 core job, it can allow other jobs to use those resources as long as they will finish in 8 hours or less (based on the walltimes they requested). Lets assume that the next 100 jobs in the queue have identical configurations, needed 10 cores for 6 hours; the scheduler can allow all of those 100 jobs to run immediately. This benefits everyone, as the 100 10-core jobs get to run without any more delay (on resources that would have otherwise just sat idle), and the 2000 core job should not be delayed.
This of course only works well if the requested walltimes for jobs are reasonably accurate. If e.g. the jobs that the scheduler expected to finish in 8 hours (to release the additional 1000 CPU cores for the 2000 core job) actually all finished in only 4 hours, then running the 100 6-hour jobs will actually cause some delay to the 2000 core job.
So making reasonable walltime estimates for your jobs is important for best performance of the backfill algorithm. Regardless, we allow the scheduler to backfill as even with less than ideal walltime estimates we still see reasonable gains in cluster efficiency.