JUSTUS2/Jobscripts: Running Your Calculations - Revision history

K Siegmund: /* Monitoring Your Jobs */

2026-03-12T14:26:33Z

Monitoring Your Jobs

← Older revision		Revision as of 16:26, 12 March 2026
Line 212:		Line 212:
	* do my jobs scale reasonably well → [[Scaling]]		* do my jobs scale reasonably well → [[Scaling]]

	== squeue ==		== "squeue" ==

	{\|style="background:#deffee; width:100%;"		{\|style="background:#deffee; width:100%;"
Line 241:		Line 241:
	* NODELIST(REASON): either show the node(s) the job is running on, or a reason why it hasn't started		* NODELIST(REASON): either show the node(s) the job is running on, or a reason why it hasn't started

	== scontrol ==		== "scontrol" ==

	You can then show more info on one specific running job using the <code>scontrol</code> command, e.g for the job with ID 6260301 listed above:		You can then show more info on one specific running job using the <code>scontrol</code> command, e.g for the job with ID 6260301 listed above:
Line 281:		Line 281:
	</code>		</code>

	== jobreport ==		== "jobreport" - Get Usage Data on Running or Finished Jobs ==

	Further, we provide the tool jobreport (only on the login nodes), that generates plots for the resource usage over time of a given job.		Further, we provide the tool jobreport (only on the login nodes), that generates plots for the resource usage over time of a given job.

K Siegmund: /* Monitoring Your Jobs */

2026-03-12T14:24:41Z

Monitoring Your Jobs

← Older revision		Revision as of 16:24, 12 March 2026
Line 241:		Line 241:
	* NODELIST(REASON): either show the node(s) the job is running on, or a reason why it hasn't started		* NODELIST(REASON): either show the node(s) the job is running on, or a reason why it hasn't started

	==scontrol==		== scontrol ==

	You can then show more info on one specific running job using the <code>scontrol</code> command, e.g for the job with ID 6260301 listed above:		You can then show more info on one specific running job using the <code>scontrol</code> command, e.g for the job with ID 6260301 listed above:
Line 264:		Line 264:
	This does not work for already completed jobs. When enabled in slurm, one can see those job scripts with <code>sacct -B -j 6260301</code>		This does not work for already completed jobs. When enabled in slurm, one can see those job scripts with <code>sacct -B -j 6260301</code>

	== Monitoring a Started Job ==		== Live Monitoring a Started Job ==

	After a job has started, you can ssh from a login node to the node(s) the job is running on, using the node name from NODELIST, e.g. if your job runs on n0603:		After a job has started, you can ssh from a login node to the node(s) the job is running on, using the node name from NODELIST, e.g. if your job runs on n0603:
Line 280:		Line 280:
	<code>nvtop		<code>nvtop
	</code>		</code>

			== jobreport ==

	Further, we provide the tool jobreport (only on the login nodes), that generates plots for the resource usage over time of a given job.		Further, we provide the tool jobreport (only on the login nodes), that generates plots for the resource usage over time of a given job.

M Carmesin: /* Monitoring a Started Job */

2025-11-06T07:41:00Z

Monitoring a Started Job

← Older revision		Revision as of 09:41, 6 November 2025
Line 281:		Line 281:
	</code>		</code>

	Further, we provide the tool jobreport, that generates plots for the resource usage over time of a given job.		Further, we provide the tool jobreport (only on the login nodes), that generates plots for the resource usage over time of a given job.

	<code>jobreport 6260301 </code>		<code>jobreport 6260301 </code>

K Siegmund: /* Bash Loop */

2025-07-14T13:56:25Z

Bash Loop

← Older revision		Revision as of 15:56, 14 July 2025
Line 425:		Line 425:
	running_jobs+=($!) # Track PID		running_jobs+=($!) # Track PID

	while [ "${#running_jobs[@]}" -ge 8 ]; do		while [ "${#running_jobs[@]}" -ge 48 ]; do
	sleep 2 # adjust duration depending on your runtime		sleep 5 # adjust duration depending on your runtime
	echo running_jobs: ${running_jobs[@]}		echo running_jobs: ${running_jobs[@]}
	echo pid-out: $(ps -o pid= -p "${running_jobs[@]}" 2>/dev/null \| xargs)		echo pid-out: $(ps -o pid= -p "${running_jobs[@]}" 2>/dev/null \| xargs)

K Siegmund: /* Bash Loop */

2025-07-14T13:54:53Z

Bash Loop

← Older revision		Revision as of 15:54, 14 July 2025
Line 409:		Line 409:
	echo starting up $i		echo starting up $i
	bash my_calculation $i &		bash my_calculation $i &
	while [ $(pgrep -c -f my_calculation) -gt 48 ] ; do echo sleeping; sleep 5; done		while [ $(pgrep -c -f my_calculation) -ge 48 ] ; do echo sleeping; sleep 5; done
	done		done
	wait		wait

M Carmesin: /* File Access */

2025-06-23T13:22:50Z

File Access

← Older revision		Revision as of 15:22, 23 June 2025
Line 63:		Line 63:

	Often, you must configure the the program you are using to write temporary files not to use the global file systems.		Often, you must configure the the program you are using to write temporary files not to use the global file systems.
	If the program uses the current directory to look for files, you must copy these files to a temporary directory, start the program there and copy/save the results of the calculation in the end. The contents ~~of the~~ of /tmp and /scratch are deleted by the automated cleanup happening after the job.		If the program uses the current directory to look for files, you must copy these files to a temporary directory, start the program there and copy/save the results of the calculation in the end. The contents of /tmp and /scratch are deleted by the automated cleanup happening after the job.

	Each node has a file system in memory (“ram disk”), that can have a maximum of half the size of the total RAM. Note that files created plus the memory requirements of your job need to fit into the total memory.		Each node has a file system in memory (“ram disk”), that can have a maximum of half the size of the total RAM. Note that files created plus the memory requirements of your job need to fit into the total memory.
Line 98:		Line 98:
	</syntaxhighlight>		</syntaxhighlight>

	$SLURM_JOB_ID contains the jobid in slurm during the run on the node and so makes sure the filename is unique.		$SLURM_JOB_ID contains the jobid in slurm during the run on the node and so makes sure the filename is unique.

	== Resource Requests ==		== Resource Requests ==

M Carmesin: /* Submitting Jobs on the bwForCluster JUSTUS 2 */

2025-06-23T13:13:33Z

Submitting Jobs on the bwForCluster JUSTUS 2

← Older revision		Revision as of 15:13, 23 June 2025
Line 50:		Line 50:

	Note: --time=00:14:00 should start your job very quickly. see [[#Testing Your Jobs]]		Note: --time=00:14:00 should start your job very quickly. see [[#Testing Your Jobs]]

		⚫	{\| style=" background:#deffee; width:100%;"
		⚫	\| style="padding:12px; background:#cef2e0; text-align:left" \|
		⚫	Software examples: Most [[Environment Modules\|installed software]] comes with example job scripts. <br> To find it e.g. for lammps: <code> module load chem/lammps; cd $LAMMPS_EXA_DIR; ls -la</code>.
		⚫	\|}

	== File Access ==		== File Access ==
Line 94:		Line 99:

	$SLURM_JOB_ID contains the jobid in slurm during the run on the node and so makes sure the filename is unique.		$SLURM_JOB_ID contains the jobid in slurm during the run on the node and so makes sure the filename is unique.

⚫	{\| style=" background:#deffee; width:100%;"
⚫	\| style="padding:12px; background:#cef2e0; text-align:left" \|
⚫	Software examples: Most [[Environment Modules\|installed software]] comes with example job scripts. <br> To find it e.g. for lammps: <code> module load chem/lammps; cd $LAMMPS_EXA_DIR; ls -la</code>.
⚫	\|}
	<br/>

	== Resource Requests ==		== Resource Requests ==

M Carmesin: /* Submitting Jobs on the bwForCluster JUSTUS 2 */

2025-06-23T13:02:33Z

Submitting Jobs on the bwForCluster JUSTUS 2

← Older revision		Revision as of 15:02, 23 June 2025
Line 51:		Line 51:
	Note: --time=00:14:00 should start your job very quickly. see [[#Testing Your Jobs]]		Note: --time=00:14:00 should start your job very quickly. see [[#Testing Your Jobs]]

			== File Access ==
⚫	Note: <font color="red"> Compute jobs must not write/read from the global file systems (HOME and WORK) as a calculation swap ~~file~~. </font>

		⚫	Note: <font color="red"> Compute jobs must not write/read temporary files from the [[JUSTUS2/Hardware#Storage_Architecture\|global file systems]] (HOME and WORK) such as a calculation swap files. </font>
⚫	Use local storage /tmp in the ramdisk for small files or /scratch (see [[BwForCluster_JUSTUS_2_Slurm_HOWTO#How_to_request_local_scratch_.28SSD.2FNVMe.29_at_job_submission.3F\|How to request NVME]]) for this purpose

		⚫	Use local storage /tmp in the ramdisk for small files or /scratch on disk (see [[BwForCluster_JUSTUS_2_Slurm_HOWTO#How_to_request_local_scratch_.28SSD.2FNVMe.29_at_job_submission.3F\|How to request NVME]]) for this purpose.
⚫	~~To not use the central file system for calculation~~, you must ~~often~~ configure the the program you are using to write temporary files ~~elsewhere.~~

		⚫	Often, you must configure the the program you are using to write temporary files not to use the global file systems.
	If the program uses the current directory to look for files, you must copy files to a temporary directory - and copy/save the results of the calculation in the end, ~~else~~ ~~your~~ ~~results~~ ~~get~~ deleted by automated cleanup happening after the job.		If the program uses the current directory to look for files, you must copy these files to a temporary directory, start the program there and copy/save the results of the calculation in the end. The contents of the of /tmp and /scratch are deleted by the automated cleanup happening after the job.

	~~There~~ ~~diskless~~ ~~nodes~~ ~~have~~ a ~~disk~~ in ~~RAM~~ memory, that can have a maximum of half the size of the total RAM. Note that files created plus memory ~~requirement~~ of your job need to fit into the total memory.		Each node has a file system in memory (“ram disk”), that can have a maximum of half the size of the total RAM. Note that files created plus the memory requirements of your job need to fit into the total memory.

	There are more diskless nodes than nodes with disks, so if your job can run on a diskless node, you should choose this option.		There are more diskless nodes than nodes with scratch disks, so if your job can run on a diskless node, you should choose this option.

	Example job script with requesting 700GB disk space and copying files:		Example job script with requesting 700GB disk space and copying files:
Line 86:		Line 87:
	</syntaxhighlight>		</syntaxhighlight>

	If there are many files or you don't know exactly how the output files are called, you can just tar the whole directory instead of using cp:		If there are many files or you don't know exactly how the output files are called, you can just create a tar archive of the whole hole directory (in HOME) instead of using cp:

	<syntaxhighlight lang='bash'>		<syntaxhighlight lang='bash'>

K Siegmund: /* Monitoring Your Jobs */

2025-06-06T11:18:10Z

Monitoring Your Jobs

← Older revision		Revision as of 13:18, 6 June 2025
Line 201:		Line 201:
	= Monitoring Your Jobs =		= Monitoring Your Jobs =

	~~Test~~ things first with few jobs before you roll out hundreds of jobs!		Always test things first with few jobs before you roll out hundreds of jobs!

	Please ensure at minimum:		Please ensure at minimum:
Line 207:		Line 207:
	* is my job using near to the amount of memory I requested		* is my job using near to the amount of memory I requested

	If you are ~~only~~ running more than 1-5 jobs:		If you are running more than 1-10 jobs:

	* are my jobs running at the very least over 10 minutes		* are my jobs running at the very least over 10 minutes

K Siegmund: /* squeue */

2025-06-06T09:02:30Z

squeue

← Older revision		Revision as of 11:02, 6 June 2025
Line 213:		Line 213:

	== squeue ==		== squeue ==

			{\|style="background:#deffee; width:100%;"
			\|style="padding:5px; background:#cef2e0; text-align:left"\|
			[[Image:Attention.svg\|center\|25px]]
			\|style="padding:5px; background:#cef2e0; text-align:left"\| Do not run squeue and other slurm commands in loops or "watch" as not to saturate up the slurm daemon with rpc requests
			\|}

	After you submitted the job, you can see it waiting using the <code>squeue</code> command:		After you submitted the job, you can see it waiting using the <code>squeue</code> command: