BwHPC BPG Data Management
1 Local File Systems
In addition to computing capacity, each bwHPC cluster is equipped with a parallel file system. For local data management it is important to differentiate if data is frequently used and persistent or quick access during a job's lifetime is desicive.
For each registered user a $HOME directory is provided in the parallel file system. A regular backup secures the files stored in this directory. But quick access from compute nodes is not possible. For data that is read or written during a job's lifetime additional storage without backup is temporarily placed at the disposal. Since implementation varies between the bwHPC clusters, please visit the sites of bwUniCluster 2.0 or bwForCluster JUSTUS 2 for details.
Directory | Characteristics | Kind of Data |
---|---|---|
$HOME | with backup, limited, global file system | software packages, configuration files, important results |
Workspaces, $WORK | quick access, limited, temporary, global file system | input/output files |
$TMPDIR, $TMP | local file system, temporarily limited to batch job's lifetime | intermediate results |
As a rule of thumb: Do not compute in $HOME!
Disk space is a limited resource on all HPC systems. If disk space is not sufficient, external storage services such as SDS@hd should be used.
2 Data Transfer
The transfer of a single large file usually achieves a higher throughput than transferring many files of small size. Therefore, it is recommended to collect files to a compressed archive with tools such as zip, tar, xz or others before transferring them to a target system.
2.1 Transfer Tools
Type | Software | Remarks | Executable on | Transfer from/to | |||||
---|---|---|---|---|---|---|---|---|---|
Local° | bwUniCluster | bwForCluster | www | bwHPC cluster | SDS@hd | ||||
Command-line tool | scp | Throughput < 150 MB/s (depending on cipher) | + | + | + | + | |||
sftp | + | + | + | + | + | ||||
rsync | + | + | + | + | |||||
rdata | Throughput of 350-400 MB/s | + | + | ||||||
wget | Download only | + | + | + | + | ||||
Client | WinSCP | based on SCP/SFTP, Windows only | + | + | + | ||||
FileZilla | based on SFTP | + | + | + |
° Depending on the installed operating system (OS).
2.2 Hosts
System | Host |
---|---|
bwUniCluster | uc1.scc.kit.edu |
bwForCluster JUSTUS 2 | justus2.uni-ulm.de |
bwForCluster MLS&WISO Production | bwfor.cluster.uni-mannheim.de |
bwForCluster MLS&WISO Production | bwforcluster.bwservices.uni-heidelberg.de |