SDS@hd/Access: Difference between revisions

From bwHPC Wiki
Jump to navigation Jump to search
m (reordered and corrected table, added ssh, switched position of two chapters)
m (Update regarding Rclone sections)
 
Line 14: Line 14:
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de''
* Port: 22


== Recommended Setup ==
== Recommended Setup ==
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages.<br />
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages.<br />
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it and it works not just for SDS@hd but also for the bwHPC clusters.<br />
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data.<br />
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]].


Line 33: Line 32:
! Possible Bandwith
! Possible Bandwith
! Firewall Ports
! Firewall Ports
|-
| [[Data_Transfer/Rclone|Rclone]] + <protocol>
| copy, sync and mount, multithreading
| ✓
| ✓
| ✓
| depends on used protocol
| depends on used protocol
|-
|-
| [[SDS@hd/Access/SMB|SMB]]
| [[SDS@hd/Access/SMB|SMB]]
| mount as network drive in file explorer
| mount as network drive in file explorer or usage via Rclone
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Windows|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
| [[SDS@hd/Access/SMB#Mac|✓]]
Line 41: Line 48:
| up to 40 Gbit/sec
| up to 40 Gbit/sec
| 139 (netbios), 135 (rpc), 445 (smb)
| 139 (netbios), 135 (rpc), 445 (smb)
|-
|- style="vertical-align:middle;"
| [[Data_Transfer/Rclone|Rclone]]
| [[Data_Transfer/WebDAV|WebDAV]]
| go to solution for restricted networks
| copy, sync and mount with WebDAV
| ✓
| []
| ✓
| ✓
| ✓
| ✓
| up to 100GBit/sec
| up to 100GBit/sec
| 80 (http), 443 (https)
| 80 (http), 443 (https)
|- style="vertical-align:middle;"
|-
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]]
| Graphical User Interface (GUI)
| Graphical User Interface (GUI)
Line 57: Line 64:
| see sftp
| see sftp
| see sftp
| see sftp
|- style="vertical-align:middle;"
|-
| [[SDS@hd/Access/NFS|NFS]]
| [[SDS@hd/Access/NFS|NFS]]
| mount for multi-user environments
| mount for multi-user environments
Line 65: Line 72:
| up to 40 Gbit/sec
| up to 40 Gbit/sec
| -
| -
|- style="vertical-align:middle;"
|-
| [[Data_Transfer/SSHFS|SSHFS]]
| [[Data_Transfer/SSHFS|SSHFS]]
| mount, needs stable internet connection
| mount
| ☓
| ☓
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]]
Line 75: Line 82:
|- style="vertical-align:middle;"
|- style="vertical-align:middle;"
| [[Data_Transfer/SFTP|SFTP]]
| [[Data_Transfer/SFTP|SFTP]]
| interactive shell, better usability when used together with Rclone
| not recommended
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#Windows|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]]

Latest revision as of 13:36, 13 March 2025

This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see data transfer.

Prerequisites

  • You need to be registered.
  • You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).

Needed Information, independent of the chosen tool

Recommended Setup

The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages.
If you have various use cases, it is recommended to use Rclone. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data.
For an overview of all connection possibilities, please have a look at all data transfer routes.

Data transfer diagram simple.jpg

Figure 1: SDS@hd main transfer routes

Use Case Windows Mac Linux Possible Bandwith Firewall Ports
Rclone + <protocol> copy, sync and mount, multithreading depends on used protocol depends on used protocol
SMB mount as network drive in file explorer or usage via Rclone up to 40 Gbit/sec 139 (netbios), 135 (rpc), 445 (smb)
WebDAV go to solution for restricted networks [✓] up to 100GBit/sec 80 (http), 443 (https)
MobaXterm Graphical User Interface (GUI) see sftp see sftp
NFS mount for multi-user environments up to 40 Gbit/sec -
SSHFS mount, needs stable internet connection see sftp see sftp
SFTP interactive shell, better usability when used together with Rclone up to 40 Gbit/sec 22 (ssh)

Table 1: SDS@hd transfer routes

Access from a bwHPC Cluster

bwForCluster Helix:
You can directly access your storage space under /mnt/sds-hd/ on all login and compute nodes.

bwForCluster BinAC
You can directly access your storage space on all login nodes.

Access via Webbrowser (read-only)

Visit lsdf02-webdav.urz.uni-heidelberg.de and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.

Best Practices

  • Managing access rights with ACLs
    -> Please set ACLs either via the Windows terminal server or via bwForCluster Helix. ACL changes won't work when used locally on a mounted directory.
  • Multiuser environment -
    -> Use NFS