2011-08-26

NFSv4+KRB5+Ubuntu+FreeIPA

(Apologies to my non-tech friends, but this was such a pain in the ass to get working that I decided to tell the Internet how it's done. This was my entire day, and much of yesterday as well. I did this work for Opin Kerfi and MR.)


NFSv4 Howto Brain Dump

These are my notes from setting up Kerberized NFSv4 on Ubuntu 11.04 clients and server, with a RHEL 6.1 FreeIPA Kerberos authority.

Process overview

  1. Configure NFS server
    1. Generate host and nfs principles on FreeIPA server
    2. Install principles in /etc/krb5.keytab on NFS server
    3. Install software on NFS server
    4. Create the /exports filesystem with bind mounds, add to fstab
    5. Edit lots of configs in /etc/
    6. Start exporting.
  2. Configure each client
    1. Generate host and nfs principles on FreeIPA server
    2. Install only the NFS pricinple on the NFS server
    3. Install both principles on the client
    4. Install software
    5. Edit lots of configs in /etc/
    6. Mount, test.

Things I learned the hard way

  • This stuff seems rather brittle (or secure by default), small misconfigurations result in access being denied or broken.

  • The default kerberos configuration only allows proper new ciphers, but the kernel NFS can only handle the ancient DES cipher. So ALL machines need the [libdefaults] section of /etc/krb5.conf updated with: allow_weak_crypto = true

  • The file /etc/krb5.keytab is a key storage file. It tends to accumulate crud, because the ktutil tool on Ubuntu will by default only add things, never remove them. This causes problems, because the NFS stuff will absolutely not work if there are any NFS principles in place using fancy new ciphers. I ended up deleting /etc/krb5.keytab many times. Also, the contents need to be kept in sync across multiple machines, so delete anywhere means delete everywhere.

  • Kerberos uses /tmp/krb* files as caches. If you delete these files by hand, be sure to restart daemons that use them, such as rpc.gssd.

  • You will also need to restart daemons after editing /etc/krb5.conf.

  • The services we care about are rpc.gssd, rpc.sysgssd, rpc.idmapd, anything to do with FreeIPA, anything to do with NFS, ...

  • Most of the rpc servers can be run from the command-line like so, for easy debugging: rpc.gssd -f -vvv (do service gssd stop first).

  • More debugging: echo 32767 > /proc/sys/sunrpc/rpc_debug

  • More debugging: echo 65535 > /proc/sys/sunrpc/nfs_debug

  • Echo 0 to make those two shut up.

  • The Domain line in /etc/idmapd.conf has to be the same on all machines.

  • /etc/default/nfs-common needs NEED_IDMAPD=yes and NEED_GSSD=yes on all machines.

  • Permissions will be loco even after mounting succeeds, without the word krb5 in sec=krb5i:krb5 in /etc/exports on the NFS server. This may be your problem if mounting works, but all users appear to be squashed to nobody/nogroup.

  • If you are using puppet for system config management, you'll want to service puppet stop first, and then make sure all your changes end up in puppet at some point.


NFS server configuration commands

(These comands are mostly OK to copy-paste - that is how I avoid typos while working. Just make sure you set HOST to match your environment. Actual exports will of course differ.)

On the FreeIPA server

HOST=nfsserver.yourdomain
IPA=ipaserver.youdomain

# You may need to kinit as admin first
ipa service-add nfs/$HOST
ipa-getkeytab -s $IPA -p nfs/$HOST -k $HOST.nfs.keytab \
              -e des-cbc-crc:normal
scp $HOST.nfs.keytab $HOST:/tmp

# Here we assumed you already have a host key generated for this machine,
# see the client config if not, ipa host add etc.

On the NFS server

HOST=nfsserver.yourdomain

# Add server service key to /etc/krb5.keytab
(  echo rkt /tmp/$HOST.nfs.keytab
   echo wkt /etc/krb5.keytab
) |ktutil
rm -f /tmp/$HOST.*.keytab

# Note: Repeat above for host key if necessary

# Verify...
klist -ke |grep $HOST

# Install software
apt-get install nfs-common nfs-kernel-server

# Probe modules, make persistant
modprobe rpcsec_gss_krb5
modprobe nfs
modprobe nfsd
echo rpcsec_gss_krb5 >>/etc/modules
echo nfs >>/etc/modules
echo nfsd >>/etc/modules

# Edit /etc/krb5.conf: allow_weak_crypto = true
vim /etc/krb5.conf

# NEED_IMAPD, NEED_GSSD, NEED_SVCGSSD = yes
vim /etc/default/nfs-common
vim /etc/default/nfs-kernel-server

# Configure your domain
vim /etc/idmapd.conf

# Update /etc/fstab to bind mount things into /exports
mkdir /exports /exports/this /exports/that
echo '/var/data/this  /export/this  none  bind  0 0' >>/etc/fstab 
echo '/var/data/that  /export/that  none  bind  0 0' >>/etc/fstab 
mount -av

# Enable exports!
echo '/exports *(sec=krb5i:krb5,rw,async,fsid=0,no_subtree_check)' \
  >>/etc/exports
echo '/exports/this *(sec=krb5i:krb5,rw,async,no_subtree_check)' \
  >>/etc/exports
echo '/exports/that *(sec=krb5i:krb5,rw,async,no_subtree_check)' \
  >>/etc/exports
service nfs-kernel-server restart
# or:
exportfs -va

Client configuration commands

(Again, these comands are mostly OK to copy-paste, just make sure you set HOST and NFSSERVER to match your environment. Actual mount commands will of course differ.)

On the FreeIPA server

HOST=yourclient.yourdomain
NFSSERVER=nfsserver.yourdomain
IPA=youripaserver.yourdomain

# You may need to kinit as admin first
ipa host-add $HOST
ipa service-add nfs/$HOST

ipa-getkeytab -s $IPA -p host/$HOST -k $HOST.host.keytab
ipa-getkeytab -s $IPA -p nfs/$HOST -k $HOST.nfs.keytab \
              -e des-cbc-crc:normal

scp $HOST.*.keytab $HOST:/tmp/
scp $HOST.nfs.keytab $NFSSERVER:/tmp/

On the NFS server

HOST=yourclient.yourdomain

# Add client service key to /etc/krb5.keytab
(  echo rkt /tmp/$HOST.nfs.keytab
   echo wkt /etc/krb5.keytab
) |ktutil
rm -f /tmp/$HOST.*.keytab

# Verify...
klist -ke |grep $HOST

On the client itself

NFSSERVER=nfsserver.yourdomain

# Install software
apt-get install nfs-common

# Load modules, make persistant
modprobe rpcsec_gss_krb5
modprobe nfs  
echo rpcsec_gss_krb5 >>/etc/modules
echo nfs >>/etc/modules

# Add client host key to /etc/krb5.keytab
(  echo rkt /tmp/$(hostname -f).host.keytab
   echo wkt /etc/krb5.keytab
) |ktutil

# Add client service key to /etc/krb5.keytab
(  echo rkt /tmp/$(hostname -f).nfs.keytab
   echo wkt /etc/krb5.keytab
) |ktutil

# Cleanup
rm -f /tmp/$(hostname -f).*.keytab

# Verify...
klist -ke

# Edit /etc/krb5.conf: allow_weak_crypto = true
vim /etc/krb5.conf

# NEED_IMAPD, NEED_GSSD
vim /etc/default/nfs-common

# Make sure this file is EXACTLY like the one on the NFS server.
vim /etc/idmapd.conf

# REBOOT or hope this is enough:
service idmapd start
service gssd start

# Mount!
echo "$NFSSERVER:/this /mnt/this nfs4 sec=krb5i,rw,proto=tcp,port=2049" \
  >>/etc/fstab
mount -av

# Test whether you can read...
# Test whether you can write!

# REBOOT EVERYTHING to make sure it STILL WORKS.

If you want to download these instructions and edit for your own environment, here is the source. The format is markdown.

Tags: tech


Recent posts

...