Justus3/Getting Started: Difference between revisions
< Justus3
Jump to navigation
Jump to search
K Siegmund (talk | contribs) (→Basics) |
K Siegmund (talk | contribs) No edit summary |
||
| Line 8: | Line 8: | ||
== Basics == |
== Basics == |
||
* know the cluster rules: → '''[[Justus3/Policy_Agreement]]''' |
* know the cluster rules: → '''[[Justus3/Policy_Agreement]]''' |
||
* log in to the cluster (and first set a 2FA secret): '''[[ |
* log in to the cluster (and first set a 2FA secret): '''[[Justus3/Login]]''' |
||
* get accustomed with the linux commandline: |
* get accustomed with the linux commandline: |
||
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]''' |
** HPC Wiki (external site) → '''[https://hpc-wiki.info/hpc/Introduction_to_Linux_in_HPC/The_Command_Line Introduction to Linux Commandline]''' |
||
| Line 18: | Line 18: | ||
* scientific software: learn how to load '''[[Software Modules]]''' |
* scientific software: learn how to load '''[[Software Modules]]''' |
||
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]''' |
* continue reading until you found that there are example job scripts: '''[[Environment_Modules#Software_job_examples]]''' |
||
* submit a sample job from a software as mentioned in the job example. Also see: '''[[ |
* submit a sample job from a software as mentioned in the job example. Also see: '''[[Justus3/Slurm]]''' |
||
* monitor your job: '''[[ |
* monitor your job: '''[[Justus3/Monitoring Your Jobs]]''' |
||
== Running Your Own Calculations in a Batch Job == |
== Running Your Own Calculations in a Batch Job == |
||
| Line 25: | Line 25: | ||
* transfer your own data to the cluster → '''[[Data Transfer]]''' |
* transfer your own data to the cluster → '''[[Data Transfer]]''' |
||
** Learn about available Hardware: |
** Learn about available Hardware: |
||
*** → '''[[ |
*** → '''[[Justus3/File Systems]]''' |
||
*** → '''[[ |
*** → '''[[Justus3/Compute Node Types]]''' |
||
* adapt the sample job script to run your own job |
* adapt the sample job script to run your own job |
||
<!-- TODO: review and put script on own page and link here [[BwForCluster_JUSTUS_2_Slurm_HOWTO#How_to_clean-up_or_save_files_before_a_job_times_out.3F| Slurm Howto]] shows how to copy and clean up your data from /tmp or /scratch at the end of the job --> |
|||
* Submit and monitor your jobs with Slurm commands. |
* Submit and monitor your jobs with Slurm commands. |
||
** → '''[[ |
** → '''[[Justus3/Jobscripts: Running Your Calculations|Jobscripts: Running Your Calculations]]''' |
||
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]''' |
** → '''[[BwForCluster_JUSTUS_2_Slurm_HOWTO| extensive Slurm HOWTO on specific tasks]]''' |
||
* Remember to mention the cluster in your publications. → '''[[bwForCluster |
* Remember to mention the cluster in your publications. → '''[[bwForCluster Justus 3 Acknowledgement|Acknowledgement]]''' |
||
== Do Production Runs with Hundreds of Jobs == |
== Do Production Runs with Hundreds of Jobs == |
||
| Line 41: | Line 40: | ||
** Make especially sure your compute jobs are running efficiently → '''[[Energy Efficient Cluster Usage]]''' |
** Make especially sure your compute jobs are running efficiently → '''[[Energy Efficient Cluster Usage]]''' |
||
* Specially remember to mention the cluster in your publications. → '''[[bwForCluster |
* Specially remember to mention the cluster in your publications. → '''[[bwForCluster Justus 3 Acknowledgement|Acknowledgement]]''' |
||
Latest revision as of 15:58, 26 June 2026
Here is a short list of things you may need to do first when you get onto the cluster
Pre-Requisites
- Know how a HPC compute Cluster works: In short: you do not simply run log in and start your software, but you write a "job script" that contains all commands to run and process your job and send this into a waiting queue to be run on one of several hundred computers. → Running Calculations
- Get Access: Follow the registration process for the bwForCluster. → How to Register for a bwForCluster
Basics
- know the cluster rules: → Justus3/Policy_Agreement
- log in to the cluster (and first set a 2FA secret): Justus3/Login
- get accustomed with the linux commandline:
- HPC Wiki (external site) → Introduction to Linux Commandline
- Have commands ready → HPC User's Linux Cheat Sheet (PDF)
- configure your shell → .bashrc Do's and Don'ts
- linux course at training.bwhpc.de
Running an Example with Preinstalled Software
- scientific software: learn how to load Software Modules
- continue reading until you found that there are example job scripts: Environment_Modules#Software_job_examples
- submit a sample job from a software as mentioned in the job example. Also see: Justus3/Slurm
- monitor your job: Justus3/Monitoring Your Jobs
Running Your Own Calculations in a Batch Job
 |
Compute jobs on nodes must not write temporary I/O intensive data to central file systems. |
- transfer your own data to the cluster → Data Transfer
- Learn about available Hardware:
- adapt the sample job script to run your own job
- Submit and monitor your jobs with Slurm commands.
- Remember to mention the cluster in your publications. → Acknowledgement
Do Production Runs with Hundreds of Jobs
- Learn about Scaling your Job
- Determine how many compute-cores the job can use? → Scaling
- Make especially sure your compute jobs are running efficiently → Energy Efficient Cluster Usage
- Specially remember to mention the cluster in your publications. → Acknowledgement