SDS@hd/Access/NFS

Prerequisites

• Attention: To access data served by SDS@hd, you need a Service Password. See details SDS@hd/Registration.

• Additionally the access to SDS@hd is currently only available inside the belwue-Network. This means you have to use the VPN Service of your HomeOrganization, if you want to access SDS@hd from outside the bwHPC-Clusters (e.g. via eduroam or from your personal Laptop)

• The access via nfs protocol is machine-based, which means new nfs-Clients have to be registered on SDS@hd. During this registration each machine receives a keytab file from the SDS@hd team, which allows mounting SDS@hd. In order the keytab file to be created, please send an email to SDS@hd team for Clientregistration with the following information:
  • hostname of the new nfs-Client
  • IP address
  • short description
  • location
  • acronym of the Speichervorhaben which should be available on this machine

Using NFSv4 for UNIX client

The authentication for data access via NFSv4 is performed using Kerberostickets. This requires a functioning Kerberos environment on the client!

Kerberos environment for SDS@hd

• For Kerberos authentication to work, a correctly synchronized system time must be set on each nfs client (e.g. via ntpdate ntp01.urz.uni-heidelberg.de or chrony)

The following parameters of kerberos tickets are set on server side:

• max. Lifetime of a Serviceticket: 10 hours
• max. Lifetime of a Userticket: 24 hours
• max. Renewaltime for Usertickets: 10 days

The properties (e.g. lifetimes, encryption, ...) of the kerberos tickets can be changed on client site with different kinit parameters (see manpages of kinit) or via /etc/krb5.conf.

First you have to install kerberos packages in your system to provide a working kerberos environment. The exact names of the packages depending on you linux distribution (see examples below).

Example RedHat/CentOS

yum install krb5-workstation

Example debian/ubuntu

apt install krb5-user

On ubuntu server: nfs-kernel-server

After installing the packages you have to use the following kerberos parameters for connecting to SDS@hd:

• Default Realm = BWSERVICES.UNI-HEIDELBERG.DE
• KDC = bwservices.uni-heidelberg.de

So your kerberos configuration file (/etc/krb5.conf) should contain the following entries:

[libdefaults]
     default_realm = BWSERVICES.UNI-HEIDELBERG.DE

     [realms]
     BWSERVICES.UNI-HEIDELBERG.DE= {
          kdc = bwservices.uni-heidelberg.de
          admin_server = bwservices.uni-heidelberg.de
      }
      [domain_realm]
         .uni-heidelberg.de = BWSERVICES.UNI-HEIDELBERG.DE
         uni-heidelberg.de = BWSERVICES.UNI-HEIDELBERG.DE

The keytab file of the machine, which you get from the SDS@hd Team, has to be stored as /etc/krb5.keytab in the system.

Because of caching issue with the kerberos ticket cache, you have to disable gssproxy service:

systemctl stop gssproxy.service
systemctl mask gssproxy.service

After configuring kerberos, you have to install nfs packages in your system, and enable kerberized NFSv4. The exact names of the packages depending on you linux distribution (see examples below).

Example RedHat/CentOS

> yum install nfs-utils nfs4-acl-tools

/etc/sysconfig/nfs:
NEED_IDMAPD=yes
NEED_GSSD=yes

Example debian/ubuntu

> apt install nfs-common nfs4-acl-tools nfs-server

/etc/default/nfs-common:
NEED_IDMAPD=yes
NEED_GSSD=yes

On ubuntu server: nfs-kernel-server

SDS@hd ID-Mapping

ID-Mapping allows you to map the uidNumbers/gidNumbers of SDS@hd accounts to more descriptive usernames.

If ID-Mapping is not or not correct configured, the ownerships and permissions of files/folders you see in the filesystem, will be incorrect. This could be confusing for users, but nevertheless the permission checking is done correctly on serversite.

Because SSSD is one of the standard tools and it supports more than one ldap/identity provider on a system, we are showing here an example configuration for this tool.

But of course you can use any other mechanism/tool to do the LDAP queries for ID Mapping if you want. The connection to SDS@hd LDAP Server is authenticated with the kerberos keytab of your machine. You can use any tool which supports GSSAPI with kerberos for authentication with the following parameters:

• uri: ldap://bwservices.uni-heidelberg.de
• search_base: dc=bwservices,dc=uni-heidelberg.de,dc=de
• sasl mech: GSSAPI Authentication
• krb5 Realm: BWSERVICES.UNI-HEIDELBERG.DE

If you don't need a machine keytab, but you still need ID Mapping (e.g. for CIFS Mounts on linux), you can use your Servicepassword to create a user keytab for LDAP authentication.

Example configuration of SSSD

The authentication to SDS@hd is done via kerberos.

If you have setup a working kerberos environment, you have to install the needed packages for kerberos and SSSD, e.g:

• RedHat/CentOS:

$ yum install sssd-client sssd-krb5 sssd-ldap

• debian/ubuntu:

$ apt install sssd sssd-krb5 sssd-ldap sssd-tools libnss-sss libsasl2-modules-gssapi-mit

If not existing, create a SSSD configuration file (/etc/sssd/sssd.conf) like this:

[sssd]
            domains = BWSERVICESAD
            config_file_version = 2
            services = nss

[domain/BWSERVICESAD]
            id_provider = ldap
            ldap_uri = ldap://bwservices.uni-heidelberg.de
            ldap_search_base = dc=bwservices,dc=uni-heidelberg,dc=de
            ldap_referrals = false

            ldap_schema = ad
            ldap_id_mapping = true
            min_id = 2000

            ldap_sasl_mech = GSSAPI
            krb5_realm = BWSERVICES.UNI-HEIDELBERG.DE
            ldap_sasl_authid = <HOSTNAME$  or Username>
            ldap_krb5_keytab = <path_to_your_keytab>
            krb5_server = bwservices.uni-heidelberg.de
            ldap_sasl_canonicalize = false
            krb5_canonicalize = false

            use_fully_qualified_names = true
            full_name_format = %3$s\%1$s

            re_expression = (((?P<domain>[^\\]+)\\(?P<name>.+$))|((?P<name>[^@]+)@(?P<domain>.+$))|(?P<name>.+$))
            enumerate = false

Attention: Don't forget to change "ldap_sasl_authid" to the Hostname/Username corresponding to your keytab file and the path to your keytab (e.g. /etc/krb5.keytab)!

If you are allready using another service for authentication or name resolution on the machine, an additional domain block can be set up for this and has to be added to the line "domains".

To enable SSSD for ID-Mapping in your system the lines "passwd" and "group" in file "/etc/nsswitch.conf" have to be extended by "sss", e.g.:

$ cat /etc/nsswitch.conf
        [...]
        passwd: compat sss
        group: compat sss

Note: If you are using sssd you should not use "nscd" in parallel! Otherwise this could lead to undefined behaviour due to double caching passwd and group entries.

After configuring SSSD you should enable and restart the service, e.g.:

$ systemctl enable sssd.service 
$ systemctl restart sssd.service

To enable the ID-Mapping for NFSv4 mounts change the file /etc/idmapd.conf with the following lines:

in /etc/idmapd.conf:
        [General]
        Domain = urz.uni-heidelberg.de
        Local-Realms = BWSERVICES.UNI-HEIDELBERG.DE

mount a nfs share

The usual restrictions for mounting drives under Linux apply. Usually this can only be done by the superuser "root". For detailed information, please contact the system administrator of your system.

After successfull configuration (s. 2.1) you can mount your SDS@hd share with the following commands:

> mkdir <mountpoint>
> mount -t nfs4 -o sec=krb5,vers=4.1 lsdf02.urz.uni-heidelberg.de:/gpfs/lsdf02/ <mountpoint>

To enable the mounting after a restart, you have to add the following line to the file "/etc/fstab"

   lsdf02.urz.uni-heidelberg.de:/gpfs/lsdf02/   <mountpoint>   nfs4     sec=krb5,vers=4.1     0 0

AutoFS Setup

Instead of the fstab-entry you can also use the automounter "autofs".

• RedHat/CentOS:

$ yum install autofs
$ systemctl enable autofs 
$ systemctl start autofs 

• debian/ubuntu:

$ apt install autofs
$ systemctl enable autofs 
$ systemctl start autofs 

Afterwards you configure the SDS@hd Speichervorhaben in a new map file:

$ cat /etc/auto.sds-hd
sds-hd -fstype=nfs4,rw,sec=krb5,vers=4.1,nosuid,nodev   lsdf02.urz.uni-heidelberg.de:/gpfs/lsdf02
....

You have to include the new map into the auto.master file, e.g.:

$ cat /etc/auto.master
[...]
/mnt   /etc/auto.sds-hd
[...]

To display all available SDS@hd shares on this machine to the users, you should enable "browser_mode":

$ cat /etc/autofs.conf
[...]
# to display all available SDS-hd shares on this to the users
browse_mode=yes
[...]

otherwise each share-folder will only be visible after a user has mounted.

After changing the configuration, you should restart the autofs daemon, e.g.:

$ systemctl restart autofs

Of course you can adopt all other autofs options, like timeouts, etc. to the specific needs of your environment or use any other method for dynamically mounting the shares.

access your data

Attention! The access can not be done as root user, because root uses the Kerberosticket of the machine, which does not have data access!

To access your data on SDS@hd you have to fetch a valid kerberos ticket with your SDS@hd user and Servicepassword:

> kinit hd_xy123
Password for hd_xy123@BWSERVICES.UNI-HEIDELBERG.DE: 

You can check afterwards your kerberos ticket with:

> klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: hd_xy123@BWSERVICES.UNI-HEIDELBERG.DE

Valid starting       Expires              Service principal
20.09.2017 04:00:01  21.09.2017 04:00:01  krbtgt/BWSERVICES.UNI-HEIDELBERG.DE@BWSERVICES.UNI-HEIDELBERG.DE
        renew until 29.09.2017 13:38:49

Afterwards you should be able to access the mountpoint, which contain all Speichervorhaben exported to your machine:

> ls <mountpoint>
sd16j007  sd17c010  sd17d005

renew a kerberos ticket

Because a kerberos ticket has a limited lifetime (default: 10 hours, maximum 24 hours) for security reasons, you have to renew your ticket before it expires to prevent access loss.

> kinit -R

This renewal could only be done for maximum time of 10 Days and as long as the current kerberos ticket is still valid. For renewal of an expired ticket, you have to use again your Servicepassword.

destroy kerberos ticket

Even if kerberos tickets are only valid for a limited period of time, a ticket should be destroyed as soon as access is no longer needed to prevent misuse on multi-user systems:

kdestroy

automated kerberos tickets

Attention! Keep this generated Keytab safe and use it only in trusted environments!

If your workflow needs a permanent access to SDS@hd for longer than 10 Days, you can use ktutil to encrypt your Service Password into a keytab file:

interactive way:

ktutil
ktutil: addent -password -p hd_xy123@BWSERVICES.UNI-HEIDELBERG.DE -k 1 -e rc4-hmac
    Password for hd_xy123@BWSERVICES.UNI-HEIDELBERG.DE:
ktutil:  addent -password -p hd_xy123@BWSERVICES.UNI-HEIDELBERG.DE -k 1 -e aes256-cts
    Password for hd_xy123@BWSERVICES.UNI-HEIDELBERG.DE:
ktutil:  wkt xy123.keytab
ktuitl: quit

non-interactive way:

echo -e "addent -password -p hd_xy123@BWSERVICES.UNI-HEIDELBERG.DE -k 1 -e rc4-hmac\n<your_servicepasword>\n
addent -password -p hd_xy123@BWSERVICES.UNI-HEIDELBERG.DE -k 1 -e aes256-cts\n<your_servicepasword>\nwkt xy123.keytab" | ktutil

With this keytab, you can fetch a kerberos ticket without an interactive password:

kinit -k -t xy123.keytab hd_xy123 

Troubleshooting

incorrect mount option

When you try to mount SDS@hd you get the following error message:

mount.nfs4: mount(2): Invalid argument
mount.nfs4: an incorrect mount option was specified

This means the service rpc-gssd.service is not yet running. Please try to start the service via

systemctl restart rpc-gssd.service

permission denied while mounting when SELinux is enabled

When trying to mount, you get the following error message:

mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5,vers=4.1,addr=x.x.x.x,clientaddr=a.b.c.d'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting lsdf02.urz.uni-heidelberg.de:/gpfs/lsdf02

You should check your systemlogs for entries like:

Aug 07 13:09:53 systemname setroubleshoot[510523]: SELinux is preventing /usr/sbin/rpc.gssd from open access on the file /etc/krb5.keytab. For complete SELinux messages run: sealert -l XXXXXX	

or similar. To solve the issue you have to add a SELinux rule to allow access to /etc/krb5.keytab (or disable SElinux completely).