BwHPC BPG Data Management

From bwHPC Wiki
Jump to: navigation, search

1 Local File Systems

In addition to computing capacity, each bwHPC cluster is equipped with a parallel file system. For local data management it is important to differentiate if data is frequently used and persistent or quick access during a job's lifetime is desicive.

For each registered user a $HOME directory is provided in the parallel file system. A regular backup secures the files stored in this directory. But quick access from compute nodes is not possible. For data that is read or written during a job's lifetime additional storage without backup is temporarily placed at the disposal. Since implementation varies between the bwHPC clusters, please visit the sites of bwUniCluster 2.0 or bwForCluster JUSTUS 2 for details.

Directory Characteristics Kind of Data
$HOME with backup, limited, global file system software packages, configuration files, important results
Workspaces, $WORK quick access, limited, temporary, global file system input/output files
$TMPDIR, $TMP local file system, temporarily limited to batch job's lifetime intermediate results

As a rule of thumb: Do not compute in $HOME!

Disk space is a limited resource on all HPC systems. If disk space is not sufficient, external storage services such as SDS@hd should be used.

2 Data Transfer

The transfer of a single large file usually achieves a higher throughput than transferring many files of small size. Therefore, it is recommended to collect files to a compressed archive with tools such as zip, tar, xz or others before transferring them to a target system.

2.1 Transfer Tools

Type Software Remarks Executable on Transfer from/to
Local° bwUniCluster bwForCluster www bwHPC cluster SDS@hd
Command-line tool scp Throughput < 150 MB/s (depending on cipher) + + + +
sftp + + + + +
rsync + + + +
rdata Throughput of 350-400 MB/s + +
wget Download only + + + +
Client WinSCP based on SCP/SFTP, Windows only + + +
FileZilla based on SFTP + + +

° Depending on the installed operating system (OS).

2.2 Hosts

System Host
bwForCluster JUSTUS 2
bwForCluster MLS&WISO Production
bwForCluster MLS&WISO Production