BwHPC BPG Data Management

1 Local File Systems

In addition to computing capacity, each bwHPC cluster is equipped with a parallel file system. For local data management it is important to differentiate if data is frequently used and persistent or quick access during a job's lifetime is desicive.

For each registered user a $HOME directory is provided in the parallel file system. A regular backup secures the files stored in this directory. But quick access from compute nodes is not possible. For data that is read or written during a job's lifetime additional storage without backup is temporarily placed at the disposal. Since implementation varies between the bwHPC clusters, please visit the sites of bwUniCluster 2.0 or bwForCluster JUSTUS 2 for details.

Directory	Characteristics	Kind of Data
$HOME	with backup, limited, global file system	software packages, configuration files, important results
Workspaces, $WORK	quick access, limited, temporary, global file system	input/output files
$TMPDIR, $TMP	local file system, temporarily limited to batch job's lifetime	intermediate results

As a rule of thumb: Do not compute in $HOME!

Disk space is a limited resource on all HPC systems. If disk space is not sufficient, external storage services such as SDS@hd should be used.

2 Data Transfer

The transfer of a single large file usually achieves a higher throughput than transferring many files of small size. Therefore, it is recommended to collect files to a compressed archive with tools such as zip, tar, xz or others before transferring them to a target system.

2.1 Transfer Tools

Type	Software	Remarks	Executable on				Transfer from/to
			Local°	bwUniCluster	bwForCluster	www	bwHPC cluster	SDS@hd
Command-line tool	scp	Throughput < 150 MB/s (depending on cipher)	+	+	+		+
	sftp		+	+	+		+	+
	rsync		+	+	+		+
	rdata	Throughput of 350-400 MB/s		+				+
	wget	Download only	+	+	+	+
Client	WinSCP	based on SCP/SFTP, Windows only	+				+	+
	FileZilla	based on SFTP	+				+	+

° Depending on the installed operating system (OS).

2.2 Hosts

System	Host
bwUniCluster	uc1.scc.kit.edu
bwForCluster JUSTUS 2	justus2.uni-ulm.de
bwForCluster MLS&WISO Production	bwfor.cluster.uni-mannheim.de
bwForCluster MLS&WISO Production	bwforcluster.bwservices.uni-heidelberg.de