JUSTUS2/Jobscripts: Running Your Calculations - Revision history

J Schäfer: /* Job Priorities */

2026-08-25T09:37:31Z

Job Priorities

← Older revision		Revision as of 11:37, 25 August 2026
Line 330:		Line 330:
	* Step 4: Can it start now, without delaying the start time of any job before it in the queue?		* Step 4: Can it start now, without delaying the start time of any job before it in the queue?
	* Step 5: If it can, remove it from the queue, start it, recalculate what nodes are free, look at next job and continue with step 4.		* Step 5: If it can, remove it from the queue, start it, recalculate what nodes are free, look at next job and continue with step 4.
	* Step 6: If it ~~can not~~, look at next job, and continue with step 4.		* Step 6: If it cannot, look at next job, and continue with step 4.

	As soon as a new job is submitted and as soon as a job finishes, Slurm restarts its main scheduling cycle with step 1.		As soon as a new job is submitted and as soon as a job finishes, Slurm restarts its main scheduling cycle with step 1.

J Schäfer: /* Partitions */

2026-08-25T09:30:35Z

Partitions

← Older revision		Revision as of 11:30, 25 August 2026
Line 308:		Line 308:

	= Partitions =		= Partitions =
	Job allocations at JUSTUS 2 are routed automatically to the most suitable compute node(s) that can provide the requested resources for the job (e.g. amount of cores, memory, local scratch space). This is to prevent fragmentation of the cluster system and to ensure most efficient usage of available compute resources. Thus, there is no point in requesting a partition in batch job scripts, i.e. users '''should not''' specify any partition "-p, --partition=<partition_name>" on job submission. This is of particular importance if you adapt job scripts from other cluster systems (e.g. bwUniCluster 2.0) to JUSTUS 2.		Job allocations at JUSTUS 2 are routed automatically to the most suitable compute node(s) that can provide the requested resources for the job (e.g. amount of cores, memory, local scratch space). This is to prevent fragmentation of the cluster system and to ensure most efficient usage of available compute resources. Thus, there is no point in requesting a partition in batch job scripts, i.e. users '''should not''' specify any partition "-p, --partition=<partition_name>" on job submission. This is of particular importance if you adapt job scripts from other cluster systems (e.g., bwUniCluster 3.0) to JUSTUS 2.

	= Job Priorities =		= Job Priorities =

J Schäfer: /* "squeue" */

2026-08-25T09:07:15Z

"squeue"

← Older revision		Revision as of 11:07, 25 August 2026
Line 249:		Line 249:
	Output shows:		Output shows:
	* JOBID: the jobid is a unique number your job gets		* JOBID: the jobid is a unique number your job gets
	* PARTITION: the cluster can be divided in different types of nodes.		* PARTITION: the cluster can be divided into different types of nodes.
	* NAME: the name you gave your job with the --job-name= option		* NAME: the name you gave your job with the --job-name= option
	* USER: your username		* USER: your username

J Schäfer: /* "squeue" */

2026-08-25T09:04:19Z

"squeue"

← Older revision		Revision as of 11:04, 25 August 2026
Line 248:		Line 248:

	Output shows:		Output shows:
	* JOBID: the jobid is an unique number your job gets		* JOBID: the jobid is a unique number your job gets
	* PARTITION: the cluster can be divided in different types of nodes.		* PARTITION: the cluster can be divided in different types of nodes.
	* NAME: the name you gave your job with the --job-name= option		* NAME: the name you gave your job with the --job-name= option

J Schäfer: /* Process Placement on CPU Cores */

2026-08-25T08:15:37Z

Process Placement on CPU Cores

← Older revision		Revision as of 10:15, 25 August 2026
Line 198:		Line 198:

	The Justus 2 nodes are two socket machines, with each processor having 24 physical CPU cores and its own NUMA domain, i.e. it can access its half of the memory faster than the other processor's part.		The Justus 2 nodes are two socket machines, with each processor having 24 physical CPU cores and its own NUMA domain, i.e. it can access its half of the memory faster than the other processor's part.
	Depending on the memory access scheme of your program, it may be beneficial to keep all processes/threads on one socket, i.e. NUMA domain, or spread it to both: As a rule of thumb, the restriction to a single domain is preferable if your memory requirements fit well inside one NUMA domain and your processes do inter process communication and/or access shared memory. In order to effectively use all memory, achieve maximal memory bandwidth or if your processes work independently, using both sockets is required.		Depending on the memory access scheme of your program, it may be beneficial to keep all processes/threads on one socket, i.e., NUMA domain, or spread it to both: As a rule of thumb, the restriction to a single domain is preferable if your memory requirements fit well inside one NUMA domain and your processes do inter process communication and/or access shared memory. In order to effectively use all memory, achieve maximal memory bandwidth or if your processes work independently, using both sockets is required.

	To force the distribution of your tasks to both sockets, we recommend the Slurm option <code>--ntasks-per-socket</code>.		To force the distribution of your tasks to both sockets, we recommend the Slurm option <code>--ntasks-per-socket</code>.

J Schäfer: /* File Access */

2026-08-25T07:46:22Z

File Access

← Older revision		Revision as of 09:46, 25 August 2026
Line 62:		Line 62:
	Use local storage /tmp in the ramdisk for small files or /scratch on disk (see [[BwForCluster_JUSTUS_2_Slurm_HOWTO#How_to_request_local_scratch_.28SSD.2FNVMe.29_at_job_submission.3F\|How to request NVME]]) for this purpose.		Use local storage /tmp in the ramdisk for small files or /scratch on disk (see [[BwForCluster_JUSTUS_2_Slurm_HOWTO#How_to_request_local_scratch_.28SSD.2FNVMe.29_at_job_submission.3F\|How to request NVME]]) for this purpose.

	Often, you must configure ~~the~~ the program you are using to write temporary files not to use the global file systems.		Often, you must configure the program you are using to write temporary files not to use the global file systems.
	If the program uses the current directory to look for files, you must copy these files to a temporary directory, start the program there and copy/save the results of the calculation in the end. The contents of /tmp and /scratch are deleted by the automated cleanup happening after the job.		If the program uses the current directory to look for files, you must copy these files to a temporary directory, start the program there and copy/save the results of the calculation in the end. The contents of /tmp and /scratch are deleted by the automated cleanup happening after the job.

J Schäfer: /* Submitting Jobs on the bwForCluster JUSTUS 2 */

2026-08-25T07:45:41Z

Submitting Jobs on the bwForCluster JUSTUS 2

← Older revision		Revision as of 09:45, 25 August 2026
Line 53:		Line 53:
	{\| style=" background:#deffee; width:100%;"		{\| style=" background:#deffee; width:100%;"
	\| style="padding:12px; background:#cef2e0; text-align:left" \|		\| style="padding:12px; background:#cef2e0; text-align:left" \|
	Software examples: Most [[Environment Modules\|installed software]] comes with example job scripts. <br> To find it e.g. for lammps: <code> module load chem/lammps; cd $LAMMPS_EXA_DIR; ls -la</code>.		Software examples: Most [[Environment Modules\|installed software]] comes with example job scripts. <br> To find it, e.g., for lammps: <code> module load chem/lammps; cd $LAMMPS_EXA_DIR; ls -la</code>.
	\|}		\|}

Line 210:		Line 210:
	#SBATCH --ntasks-per-socket=12		#SBATCH --ntasks-per-socket=12
	</pre>		</pre>


	= Testing Your Jobs =		= Testing Your Jobs =

M Carmesin at 08:57, 27 May 2026

2026-05-27T08:57:24Z

← Older revision		Revision as of 10:57, 27 May 2026
Line 194:		Line 194:

	Also note that allocated memory is factored into resource usage accounting for fair share. This means over-requesting memory may have a negative impact on the priority of subsequent jobs.		Also note that allocated memory is factored into resource usage accounting for fair share. This means over-requesting memory may have a negative impact on the priority of subsequent jobs.

			== Process Placement on CPU Cores ==

			The Justus 2 nodes are two socket machines, with each processor having 24 physical CPU cores and its own NUMA domain, i.e. it can access its half of the memory faster than the other processor's part.
			Depending on the memory access scheme of your program, it may be beneficial to keep all processes/threads on one socket, i.e. NUMA domain, or spread it to both: As a rule of thumb, the restriction to a single domain is preferable if your memory requirements fit well inside one NUMA domain and your processes do inter process communication and/or access shared memory. In order to effectively use all memory, achieve maximal memory bandwidth or if your processes work independently, using both sockets is required.

			To force the distribution of your tasks to both sockets, we recommend the Slurm option <code>--ntasks-per-socket</code>.

			If you have for example a job with 24 tasks that need about 90GB memory in total, i. e. half of the smallest nodes RAM, you should specify in you job script

			<pre>
			#SBATCH --mem=90GB
			#SBATCH --nodes=1
			#SBATCH --ntasks=24
			#SBATCH --ntasks-per-socket=12
			</pre>


	= Testing Your Jobs =		= Testing Your Jobs =

	Justus2 has three compute nodes reserved for jobs with a walltime under 15 minutes. You can test if your jobs start properly just by specifying a short walltime, e.g. --time=00:14:00 and your job should start very quickly.		Justus2 has three compute nodes reserved for jobs with a walltime under 15 minutes. You can test if your jobs start properly just by specifying a short walltime, e.g. <code>--time=00:14:00</code> and your job should start very quickly.

	= Monitoring Your Jobs =		= Monitoring Your Jobs =

K Siegmund: /* Monitoring Your Jobs */

2026-03-12T14:26:33Z

Monitoring Your Jobs

← Older revision		Revision as of 16:26, 12 March 2026
Line 212:		Line 212:
	* do my jobs scale reasonably well → [[Scaling]]		* do my jobs scale reasonably well → [[Scaling]]

	== squeue ==		== "squeue" ==

	{\|style="background:#deffee; width:100%;"		{\|style="background:#deffee; width:100%;"
Line 241:		Line 241:
	* NODELIST(REASON): either show the node(s) the job is running on, or a reason why it hasn't started		* NODELIST(REASON): either show the node(s) the job is running on, or a reason why it hasn't started

	== scontrol ==		== "scontrol" ==

	You can then show more info on one specific running job using the <code>scontrol</code> command, e.g for the job with ID 6260301 listed above:		You can then show more info on one specific running job using the <code>scontrol</code> command, e.g for the job with ID 6260301 listed above:
Line 281:		Line 281:
	</code>		</code>

	== jobreport ==		== "jobreport" - Get Usage Data on Running or Finished Jobs ==

	Further, we provide the tool jobreport (only on the login nodes), that generates plots for the resource usage over time of a given job.		Further, we provide the tool jobreport (only on the login nodes), that generates plots for the resource usage over time of a given job.

K Siegmund: /* Monitoring Your Jobs */

2026-03-12T14:24:41Z

Monitoring Your Jobs

← Older revision		Revision as of 16:24, 12 March 2026
Line 241:		Line 241:
	* NODELIST(REASON): either show the node(s) the job is running on, or a reason why it hasn't started		* NODELIST(REASON): either show the node(s) the job is running on, or a reason why it hasn't started

	==scontrol==		== scontrol ==

	You can then show more info on one specific running job using the <code>scontrol</code> command, e.g for the job with ID 6260301 listed above:		You can then show more info on one specific running job using the <code>scontrol</code> command, e.g for the job with ID 6260301 listed above:
Line 264:		Line 264:
	This does not work for already completed jobs. When enabled in slurm, one can see those job scripts with <code>sacct -B -j 6260301</code>		This does not work for already completed jobs. When enabled in slurm, one can see those job scripts with <code>sacct -B -j 6260301</code>

	== Monitoring a Started Job ==		== Live Monitoring a Started Job ==

	After a job has started, you can ssh from a login node to the node(s) the job is running on, using the node name from NODELIST, e.g. if your job runs on n0603:		After a job has started, you can ssh from a login node to the node(s) the job is running on, using the node name from NODELIST, e.g. if your job runs on n0603:
Line 280:		Line 280:
	<code>nvtop		<code>nvtop
	</code>		</code>

			== jobreport ==

	Further, we provide the tool jobreport (only on the login nodes), that generates plots for the resource usage over time of a given job.		Further, we provide the tool jobreport (only on the login nodes), that generates plots for the resource usage over time of a given job.