SDS@hd/Access: Difference between revisions
S Richling (talk | contribs) |
H Schumacher (talk | contribs) m (Update regarding Rclone sections) |
||
(10 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see [[Data_Transfer|data transfer]]. |
|||
= Overview = |
|||
== Prerequisites == |
|||
Picture |
|||
* You need to be [[SDS@hd/Registration|registered]]. |
|||
= Access Protocols = |
|||
* You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook). |
|||
* [[SDS@hd/Access/SFTP|SFTP/SSHFS (Win,Mac,Linux)]] |
|||
* [[SDS@hd/Access/CIFS|SMB/CIFS (Win,Mac,Linux)]] |
|||
* [[SDS@hd/Access/NFS|NFSv4 (Linux)]] |
|||
== Needed Information, independent of the chosen tool == |
|||
= Access from bwForCluster Helix = |
|||
* [[Registration/Login/Username| Username]]: Same as for the bwHPC Clusters |
|||
On bwForCluster Helix is possible to directly access your storage space under /mnt/sds-hd/ on all login and compute nodes. |
|||
* Password: The Service Password that you set at bwServices in the [[SDS@hd/Registration|registration step]]. |
|||
* Hostname: The hostname depends on the chosen network protocol: |
|||
** For [[Data_Transfer/SSHFS|SSHFS]] and [[Data_Transfer/SFTP|SFTP]]: ''lsdf02-sshfs.urz.uni-heidelberg.de'' |
|||
** For [[SDS@hd/Access/SMB|SMB]] and [[SDS@hd/Access/NFS|NFS]]: ''lsdf02.urz.uni-heidelberg.de'' |
|||
** For [[Data_Transfer/WebDAV|WebDAV]] the url is: ''https://lsdf02-webdav.urz.uni-heidelberg.de'' |
|||
== Recommended Setup == |
|||
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages.<br /> |
|||
If you have various use cases, it is recommended to use [[Data_Transfer/Rclone|Rclone]]. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data.<br /> |
|||
For an overview of all connection possibilities, please have a look at [[Data_Transfer/All_Data_Transfer_Routes|all data transfer routes]]. |
|||
[[File:Data_transfer_diagram_simple.jpg|center|500px]] |
|||
<p style="text-align: center; font-size: small; margin-top: 10px">Figure 1: SDS@hd main transfer routes</p> |
|||
{| class="wikitable" |
|||
|- style="font-weight:bold; text-align:center; vertical-align:middle;" |
|||
! |
|||
! Use Case |
|||
! Windows |
|||
! Mac |
|||
! Linux |
|||
! Possible Bandwith |
|||
! Firewall Ports |
|||
|- |
|||
| [[Data_Transfer/Rclone|Rclone]] + <protocol> |
|||
| copy, sync and mount, multithreading |
|||
| ✓ |
|||
| ✓ |
|||
| ✓ |
|||
| depends on used protocol |
|||
| depends on used protocol |
|||
|- |
|||
| [[SDS@hd/Access/SMB|SMB]] |
|||
| mount as network drive in file explorer or usage via Rclone |
|||
| [[SDS@hd/Access/SMB#Windows|✓]] |
|||
| [[SDS@hd/Access/SMB#Mac|✓]] |
|||
| [[SDS@hd/Access/SMB#Linux|✓]] |
|||
| up to 40 Gbit/sec |
|||
| 139 (netbios), 135 (rpc), 445 (smb) |
|||
|- |
|||
| [[Data_Transfer/WebDAV|WebDAV]] |
|||
| go to solution for restricted networks |
|||
| [✓] |
|||
| ✓ |
|||
| ✓ |
|||
| up to 100GBit/sec |
|||
| 80 (http), 443 (https) |
|||
|- style="vertical-align:middle;" |
|||
| [[Data_Transfer/Graphical_Clients#MobaXterm|MobaXterm]] |
|||
| Graphical User Interface (GUI) |
|||
| [[Data_Transfer/Graphical_Clients#MobaXterm|✓]] |
|||
| ☓ |
|||
| ☓ |
|||
| see sftp |
|||
| see sftp |
|||
|- style="vertical-align:middle;" |
|||
| [[SDS@hd/Access/NFS|NFS]] |
|||
| mount for multi-user environments |
|||
| ☓ |
|||
| ☓ |
|||
| [[SDS@hd/Access/NFS|✓]] |
|||
| up to 40 Gbit/sec |
|||
| - |
|||
|- style="vertical-align:middle;" |
|||
| [[Data_Transfer/SSHFS|SSHFS]] |
|||
| mount, needs stable internet connection |
|||
| ☓ |
|||
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]] |
|||
| [[Data_Transfer/SSHFS#MacOS_&_Linux|✓]] |
|||
| see sftp |
|||
| see sftp |
|||
|- style="vertical-align:middle;" |
|||
| [[Data_Transfer/SFTP|SFTP]] |
|||
| interactive shell, better usability when used together with Rclone |
|||
| [[Data_Transfer/SFTP#Windows|✓]] |
|||
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]] |
|||
| [[Data_Transfer/SFTP#MacOS_&_Linux|✓]] |
|||
| up to 40 Gbit/sec |
|||
| 22 (ssh) |
|||
|} |
|||
<p style="text-align: center; font-size: small; margin-top: 10px">Table 1: SDS@hd transfer routes</p> |
|||
=== Access from a bwHPC Cluster === |
|||
'''bwForCluster Helix:'''<br /> |
|||
You can directly access your storage space under ''/mnt/sds-hd/'' on all login and compute nodes. |
|||
'''bwForCluster BinAC'''<br /> |
|||
You can directly access your storage space on all login nodes. |
|||
=== Access via Webbrowser (read-only) === |
|||
Visit [https://lsdf02-webdav.urz.uni-heidelberg.de/ lsdf02-webdav.urz.uni-heidelberg.de] and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above. |
|||
== Best Practices == |
|||
* '''Managing access rights with ACLs''' <br /> -> Please set ACLs either via the [https://www.urz.uni-heidelberg.de/de/service-katalog/desktop-und-arbeitsplatz/windows-terminalserver Windows terminal server] or via bwForCluster Helix. ACL changes won't work when used locally on a mounted directory. |
|||
* '''Multiuser environment''' -<br /> -> Use [[SDS@hd/Access/NFS|NFS]] |
Latest revision as of 13:36, 13 March 2025
This page provides an overview on how to access data served by SDS@hd. To get an introduction to data transfer in general, see data transfer.
Prerequisites
- You need to be registered.
- You need to be in the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal notebook).
Needed Information, independent of the chosen tool
- Username: Same as for the bwHPC Clusters
- Password: The Service Password that you set at bwServices in the registration step.
- Hostname: The hostname depends on the chosen network protocol:
Recommended Setup
The following graphic shows the recommended way for accessing SDS@hd via Windows/Mac/Linux. The table provides an overview of the most important access options and links to the related pages.
If you have various use cases, it is recommended to use Rclone. You can copy, sync and mount with it. Thanks to its multithreading capability Rclone is a good fit for transferring big data.
For an overview of all connection possibilities, please have a look at all data transfer routes.
Figure 1: SDS@hd main transfer routes
Use Case | Windows | Mac | Linux | Possible Bandwith | Firewall Ports | |
---|---|---|---|---|---|---|
Rclone + <protocol> | copy, sync and mount, multithreading | ✓ | ✓ | ✓ | depends on used protocol | depends on used protocol |
SMB | mount as network drive in file explorer or usage via Rclone | ✓ | ✓ | ✓ | up to 40 Gbit/sec | 139 (netbios), 135 (rpc), 445 (smb) |
WebDAV | go to solution for restricted networks | [✓] | ✓ | ✓ | up to 100GBit/sec | 80 (http), 443 (https) |
MobaXterm | Graphical User Interface (GUI) | ✓ | ☓ | ☓ | see sftp | see sftp |
NFS | mount for multi-user environments | ☓ | ☓ | ✓ | up to 40 Gbit/sec | - |
SSHFS | mount, needs stable internet connection | ☓ | ✓ | ✓ | see sftp | see sftp |
SFTP | interactive shell, better usability when used together with Rclone | ✓ | ✓ | ✓ | up to 40 Gbit/sec | 22 (ssh) |
Table 1: SDS@hd transfer routes
Access from a bwHPC Cluster
bwForCluster Helix:
You can directly access your storage space under /mnt/sds-hd/ on all login and compute nodes.
bwForCluster BinAC
You can directly access your storage space on all login nodes.
Access via Webbrowser (read-only)
Visit lsdf02-webdav.urz.uni-heidelberg.de and login with your SDS@hd username and service password. Here you can get an overview of the data in your "Speichervorhaben" and download single files. To be able to do more, like moving data, uploading new files, or downloading complete folders, a suitable client is needed as described above.
Best Practices
- Managing access rights with ACLs
-> Please set ACLs either via the Windows terminal server or via bwForCluster Helix. ACL changes won't work when used locally on a mounted directory. - Multiuser environment -
-> Use NFS