Archive for December, 2014

Slackware, luks, root encryption and remote booting through ssh

December 7th, 2014 -- Posted in Tech | No Comments »

It’s quite a mouth full, and while writing, I find myself amused about what crazy project I pulled off this time.

I got myself into quite a bit of trouble with failing hard disks (of course always one too many for the raid array to recover from), and a complete lack of backups. I know, I know, shame on me. So I decided to better myself. A NAS unit is a bit above my budget, and besides, I have pretty high standards. I’d rather whip something up myself that meets my needs completely. So I decided to repurpose an old pc I got as a gift a while back.

The problem I ran into was as follows: This is going to be a headless machine, hooked up in the meter closet (or whatever it’s called in English). I want to encrypt the root filesystem, it’s not much use encrypting anything if your backup leaks data. But… ans here comes the trouble: I’m bedridden. I can’t get up and go to the machine to provide it with a password in the event it needs rebooting. One option is to use a key to decrypt, and put it on a usb stick. But it’s not much use if the usb stick is always in the pc.

So I started looking into providing a password through ssh, during boot. There are several tutorials and scripts available, but only for ubuntu/debian style initrd. Slackware has it’s own particular initrd system, and I couldn’t find anyone who had documented doing this before. Seeing as I got it to work, I though it’d share it here.

Disclaimer

This is not for the faint-hearted! It will involve getting to know the boot process, debugging it, editing scripts, working with the kernel and a lot of tinkering.

Aside from that, I wrote this guide from memory, after 3 days of debugging and mucking about and finally getting everything in working order. I did not do a spell check, so don’t copy commands blindly. I might have missed a few steps, my memory isn’t great. Feel free to contact me if you’re having trouble, you can leave a comment, but you have a better chance of getting an answer if you email me at info_AT_nihlaeth_DOT_nl

Root encryption in Slackware 14.1

This needs a bit of explanation, as the documentation provided with slackware 14.1 won’t get you a working system. The error in README_CRYPT is that you’re told you can name your crypt devices anything. This is false. If you use an initrd, you have to name your cryptdevice the standard: luks[device] – so, for example: lukssda3, which would be the third partition on /dev/sda.

Another error is in the instructions for lilo.conf. If you use the huge kernel, you need the “root=” line, but this will conflict with options set in initrd. So make sure /boot/vmlinuz points to the generic kernel, and not the huge kernel. Or even better, just point it directly to the correct kernel and prevent confusion. And don’t forget to comment out the root line in /etc/lilo.conf.

I advise you to use the script in /usr/share/mkinitrd/mkinitrd_command_generator.sh to create your initrd. I just echo it to /etc/mkinitrd.conf because we’ll be running it A LOT (and editing a bit).

With the above in mind, complete the installation and make sure you can boot the normal way. If not, get back in with the installation cd.

The process of getting back in goes as follows:
decrypt your root filesystem:

cryptsetup luksOpen /dev/[device#partition] luks[device#partition]

Now mount it:

mount /dev/mapper/luks[device#partition] /mnt

If you have incorrectly shut down the machine before, the filesystem might need checking before you mount:

fsck.[fstype] /dev/mapper/luks[device#partition]

If you have more than one encrypted partitions, go ahead and decrypt and mount them. Don’t forget to mount your boot partition (vital!):

mount /dev/[device#partition] /mnt/boot

Now you need to provide the dev, proc and sys filesystems:

mount -o bind /proc /mnt/proc
mount -o bind /sys /mnt/sys
mount -o bind /dev /mnt/dev

Now you’re ready to get back in and fix whatever went wrong:

chroot /mnt

Now you can make the necessary changes, run mkinitrd and lilo again, and rinse and repeat until it works.

By the way, if you get a screen saying “L99 99 99 99 99…(etc)”, you might try telling lilo to overwrite the MBR:

lilo -M /dev/[device] mbr

Booting with an initrd

First of all, I’ll explain some of what I learned about the boot process with an initrd. I might be wrong about some of it, all I know I got from the initrd scripts because there isn’t much to be found about the workings of initrd.

First, your bios loads. This will look for either a boot flag, on some partition, or a bootloader (for example in the master boot record). In our case, this will be lilo. Grub is also an option, but that’s beyond the scope of this tutorial.

Secondly, lilo loads the kernel. We told it where to find that in /etc/lilo.conf, but as our devices are not decrypted yet, it doesn’t know about that. That’s why you have to run lilo every time you change something (even if all you changed was a single line in the initrd init script). When you run lilo, it writes the exact disk locations for the initrd and kernel so they can be loaded with minimal knowledge of the filesystem.

After the kernel is loaded, lilo extracts the initrd image, mounts it as read only temporary root and starts the init script inside.

The init script then starts loading the kernel modules needed to use raid, cryptsetup, lvm, and pretty much anything else you tell it to, so they are available without the root filesystem being mounted.

After that, it starts udev, and checks what needs to be done (for example, starting raid arrays, activating lvm groups, decrypting drives). When all the needed filesystems are available, it mounts them, and passes the torch to the normal booting process.

What we want to do here, is tell init to load the necessary kernel modules to start the network, and have udev make the network interfaces available. When that’s done, we’ll actually start it, and start a minimal ssh daemon (dropbear in our case).

After that’s loaded, we’re going to halt the boot process. If we didn’t do that, your timing in logging in would have to be very good, and even then you wouldn’t have any time to type the passphrase. We want to still be able to boot with a connected keyboard, so we have to be able to continue via that way, and also of course, to be able to continue via ssh.

The solution for this is simple. We start a script that waits for an input of ‘ok’. If you’re done decrypting via ssh, you can kill the script, and init will continue like normal. At the terminal, you can type ok, and boot the regular way.

It’s actually a bit dirty, because what you do with an ssh login during boot, is decrypt the needed harddisks and then have init continue like nothing happened. Which means, init will try to decrypt a couple of already decrypted devices. It won’t do any damage, and it’ll work, but it does not deserve a beauty award.

Remote boot via ssh

First of all, we need to correct the command that generates initrd. Initrd first checks if /boot/initrd-tree exists, and if it has the -c option. If it doesn’t exist, or you used -c, it will write a new source tree to /boot/initrd. So the first thing we’ll do after the source tree is present, is remove the -c option from /etc/mkinitrd.conf (if you don’t have that, start reading at the top). If you screwed up and want to start over, just remove /boot/initrd-tree and run /etc/mkinitrd.conf again, it will provide you with a fresh tree. And because it’s easy to overwrite the tree by accident, I advise you to back up the customized tree.

From that tree, the initrd image will be made, so we’ll be doing all adaptions in /boot/initrd-tree.

The second step is to ensure that the right device is indicated. -C should point to the physical device. This should be ONE device, unless you use raid, and you need multiple devices to make up the root filesystem. Any other devices you want to decrypt through ssh, you should hardcode into the init script.

The mkinitrd command generator script automatically includes the needed modules to use a usb keyboard, now all we need to add is the correct module for your network card. In my case, this was e1000e. To find out which driver your network card uses:

dmesg | grep "eth"

Or, if you use a different interface name, grep for that of course. That will tell you which kernel module is used by your network card. Include this in the modules list. My mkinitrd command looks like this:


#
# mkinitrd_command_generator.sh revision 1.45
#
# This script will now make a recommendation about the command to use
# in case you require an initrd image to boot a kernel that does not
# have support for your storage or root filesystem built in
# (such as the Slackware 'generic' kernels').
# A suitable 'mkinitrd' command will be:


#added e1000e module to be able to start the network during boot
mkinitrd -k 3.10.17 -f jfs -r /dev/mapper/lukssda3 -m usbhid:hid_generic:uhci-hcd:jfs:e1000e -C /dev/sda3 -u -o /boot/initrd.gz -L

The -L option is also vital, it adds /sbin/dmsetup. It wasn’t automatically added in my case, and I didn’t think I’d need it as I don’t use LVM. But I needed it, I suspect you will too. If your boot process dies with the message it can’t find /sbin/dmsetup after asking you for a password, chances are you need the -L option with mkinitrd.

Anyway. That’s mkinitrd. Before we dive into the init script, we’ll need to add some files to the initrd tree. First of all, don’t be fooled into adding/replacing libraries in the tree. It will most likely result in kernel panics if you do. Other distros need this, initrd in slackware doesn’t. Aside from a boot script, initrd in slackware also provides a rescue console, so most essentials are already there.

All we need to add is the dropbear binary, and some config files in the /boot/initrd-tree/etc directory.

You can compile dropbear with this slackbuild(link). After compilation, I would advise against installing it, as it conflicts with the openssh package. Just dump the package inside an empty directory and untar it to get to the binaries we need.

First of all, create the usr/bin dir inside the initrd tree:

mkdir /boot/initrd-tree/usr/bin

Now, copy dropbearmulti and make a symlink to it that can start the server:

cp /[pkgdir]/usr/bin/dropbearmulti /boot/initrd-tree/usr/bin/.
ln -s dropbearmulti /boot/initrd-tree/usr/bin/dropbear

Now we need to arrange the configuration for dropbear. It doesn’t need much, the defaults will be fine in most cases, but it does need a set of host keys. For this, we need to create a symlink to dropbearmulti capable of creating keys:

ln -s dropbearmulti /[pkgdir]/usr/bin/dropbearkey
mkdir /boot/initrd-tree/etc/dropbear
/[pkgdir]/usr/bin/dropbearkey -t dss -f /boot/initrd-tree/etc/dropbear/dropbear_dss_host_key
/[pkgdir]/usr/bin/dropbearkey -t rsa -t /boot/initrd-tree/etc/dropbear/dropbear_rsa_host_key

Now we’re making a banner file. It’s just some text that will be displayed when you ssh in during boot. I recommend explaining to yourself what to do from there, because you probably won’t remember when you have to reboot this headless backup server six months from now:

vim /boot/initrd-tree/etc/dropbear/banner

My banner file says something along the lines of just run “unlock”, and that if I need to do anything other than provide passwords, not to run unlock, as it will advance the boot process.

That’s all the configuration dropbear needs. Now you need to edit the /boot/initrd-tree/etc/passwd file, as the root user has a default shell of bash, which is not installed in the initrd. After changing it, it should look like this:

root:x:0:0::/root:/bin/sh

It’s best to delete the rest, you won’t be needing other users anyway.

One file that’s not included by default is the shadow file. If you’re going to use a password to log in, you’ll need it. It’s a bit of a security risk, as you’ll be putting it on an unencrypted filesystem, but it’s one I can live with personally. You can also use public/private key authentication, and it shouldn’t be hard to set up. I just didn’t bother to do it.

cp /etc/shadow /boot/initrd-tree/etc/.

It’s best to delete all lines that are not root here as well.

In order for udev to correctly setup the network interface(s), it needs an udev rules file. The default one is fine:

cp /etc/udev/rules.d/70-persistent-net.rules /boot/initrd-tree/etc/udev/rules.d/.

Now we’ll need a lock and unlock script.

vim /boot/initrd-tree/bin/unlock

This is my unlock script, obviously you should adjust the devices and crypt device names accordingly. No mounting is needed, init takes care of that. I do recommend you put the kill lock and exit on the same line, my session wouldn’t terminate otherwise.

#!/bin/sh
PATH="/sbin:/bin:/usr/sbin:/usr/bin"
# unlock needed devices
cryptsetup luksOpen /dev/sda3 lukssda3
cryptsetup luksOpen /dev/sdb1 lukssdb1

echo "#############################"
echo "###        TNXBYE         ###"
echo "#############################"

# terminate locking script
killall bootlock; exit 0

I named my lock script bootlock for no good reason. If you decide to call it something else, don’t forget to adjust the ‘killall’ command above.

vim /boot/initrd-tree/bin/bootlock

My bootlock script (this shouldn’t need any adjustments):

#!/bin/sh

#lock the booting process to allow a network user to login first

echo "###########################################"
echo "###        Wait for user input          ###"
echo "###########################################"
echo "Type ok and press enter to continue booting:"
INPUT='wait'
while [ $INPUT != 'ok' ] ; do 
  read INPUT
done

Now we make them both executable:

chmod +x /boot/initrd-tree/bin/bootlock
chmod +x /boot/initrd-tree/bin/unlock

That’s all the files and configuration that’s needed. All that’s left now is the init script. I’ll paste mine here. Don’t copy this without editing, there are several spots where you need to customize. Besides, it’s way more fun if you understand what it all does. I’ll put my adaptions in red, so it will be easier to adapt future versions of the init script.


#!/bin/ash
#
# /init:  init script to load kernel modules from an initramfs
#         This requires that your kernel supports initramfs!!!
#
# Copyright 2004  Slackware Linux, Inc., Concord, CA, USA
# Copyright 2007, 2008, 2009, 2010, 2012  Patrick J. Volkerding, Sebeka, MN, USA
# All rights reserved.
#
# Redistribution and use of this script, with or without modification, is
# permitted provided that the following conditions are met:
#
# 1. Redistributions of this script must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#
#  THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED
#  WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
#  MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO
#  EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
#  PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
#  OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
#  WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
#  OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
#  ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
##################################################################################
# With a generic kernel, you need to load the modules needed to mount the
# root partition.  This might mean a SCSI, RAID, or other drive controller
# module, as well as the module to support the root filesystem.  Once the
# root partition is mounted all the other modules will be available so you
# don't need to load them here.
#
# Config files used by this script:
#
# /rootdev        Contains the name of the root device, such as: /dev/hda1 
#
# /rootfs         Contains the root filesystem type, such as: xfs
#
# /initrd-name    Contains the name of the initrd file.
#
# /resumedev      Contains the name of the device to resume from hibernation.
#
# /luksdev        Contains colon separated list of luks encrypted devices to
#                 be unlocked.
#
# /lukskey        Contains the path to a LUKS key-file for automatic unlock
#                 Format: LABEL=:/path/to/file
#                         UUID=:/path/to/file
#
# /wait-for-root  Contains a number - the init script will wait this amount
#                 of seconds before creating device nodes.
#
# /keymap         Contains the name for a custom keyboard map 
#
# Optional:
#
# /load_kernel_modules
#                 A script that uses modprobe to load the desired modules.
#
#                 There's an example in here.  To actually use it, you'll
#                 need to make it executable:  
#
#                     chmod 755 load_kernel_modules
##################################################################################
# Changelog
# 10-Dec-2012 
#  * Added support for the official Kernel parameters to select root filesystem
#    type ('rootfstype') and pause before attempting to mount the root filesystem
#    ('rootdelay').  The original parameters may continue to be used.
##################################################################################

INITRD=$(cat /initrd-name)
ROOTDEV=$(cat /rootdev)
ROOTFS=$(cat /rootfs)
LUKSDEV=$(cat /luksdev)
LUKSKEY=$(cat /lukskey)
RESUMEDEV=$(cat /resumedev)
WAIT=$(cat /wait-for-root)
KEYMAP=$(cat /keymap)
INIT=/sbin/init

PATH="/sbin:/bin:/usr/sbin:/usr/bin"

# Mount /proc and /sys:
mount -n proc /proc -t proc
mount -n sysfs /sys -t sysfs
mount -n tmpfs /run -t tmpfs -o mode=0755

if grep devtmpfs /proc/filesystems 1>/dev/null 2>/dev/null ; then
  DEVTMPFS=1
  mount -n devtmpfs /dev -t devtmpfs
fi	

# Parse command line
for ARG in $(cat /proc/cmdline); do
  case $ARG in
    0|1|2|3|4|5|6|S|s|single)
      RUNLEVEL=$ARG
    ;;
    init=*)
      INIT=$(echo $ARG | cut -f2 -d=)
    ;;
    luksdev=/dev/*)
      LUKSDEV=$(echo $ARG | cut -f2 -d=)
    ;;
    lukskey=*)
      LUKSKEY=$(echo $ARG | cut -f2- -d=)
    ;;
    rescue)
      RESCUE=1
    ;;
    resume=*)
      RESUMEDEV=$(echo $ARG | cut -f2 -d=)
    ;;
    root=/dev/*)
      ROOTDEV=$(echo $ARG | cut -f2 -d=)
    ;;
    root=LABEL=*)
      ROOTDEV=$(echo $ARG | cut -f2- -d=)
    ;;
    root=UUID=*)
      ROOTDEV=$(echo $ARG | cut -f2- -d=)
    ;;
    rootfs=*|rootfstype=*)
      ROOTFS=$(echo $ARG | cut -f2 -d=)
    ;;
    waitforroot=*|rootdelay=*)
      WAIT=$(echo $ARG | cut -f2 -d=)
    ;;
  esac
done

# If udevd is available, use it to generate block devices
# else use mdev to read sysfs and generate the needed devices 
if [ -x /sbin/udevd -a -x /sbin/udevadm ]; then
  /sbin/udevd --daemon --resolve-names=never
  /sbin/udevadm trigger --subsystem-match=block --action=add
  /sbin/udevadm settle --timeout=10
  # initialize network interfaces for ssh boot
  /sbin/udevadm trigger --action='add' --subsystem-match='net'
  /sbin/udevadm settle --timeout=10
else
  [ "$DEVTMPFS" != "1" ] && mdev -s
fi

# Load kernel modules (ideally this was already done by udev):
if [ ! -d /lib/modules/$(uname -r) ]; then
  echo "No kernel modules found for Linux $(uname -r)."
elif [ -x ./load_kernel_modules ]; then # use load_kernel_modules script:
  echo "${INITRD}:  Loading kernel modules from initrd image:"
  . ./load_kernel_modules
else # load modules (if any) in order:
  if ls /lib/modules/$(uname -r)/*.*o 1> /dev/null 2> /dev/null ; then
    echo "${INITRD}:  Loading kernel modules from initrd image:"
    for module in /lib/modules/$(uname -r)/*.*o ; do
      /sbin/modprobe $module
    done
    unset module
  fi
fi

# Sometimes the devices need extra time to be available.
# A root filesystem on USB is a good example of that.
sleep $WAIT

# Load a custom keyboard mapping:
if [ -n "$KEYMAP" ]; then
  echo "${INITRD}:  Loading '$KEYMAP' keyboard mapping:"
  tar xzOf /etc/keymaps.tar.gz ${KEYMAP}.bmap | loadkmap
fi

if [ "$RESCUE" = "" ]; then 
  # Initialize RAID:
  if [ -x /sbin/mdadm ]; then
    # If /etc/mdadm.conf is present, udev should DTRT on its own;
    # If not, we'll make one and go from there:
    if [ ! -r /etc/mdadm.conf ]; then
      /sbin/mdadm -E -s >/etc/mdadm.conf
      /sbin/mdadm -S -s
      /sbin/mdadm -A -s
      # This seems to make the kernel see partitions more reliably:
      fdisk -l /dev/md* 1> /dev/null 2> /dev/null
    fi
  fi

  # Unlock any encrypted partitions necessary to access the
  # root filesystem, such as encrypted LVM Physical volumes, disk
  # partitions or mdadm arrays.
  # Unavailable devices such as LVM Logical Volumes will need to be
  # deferred until they become available after the vgscan.

  if [ -x /sbin/cryptsetup ]; then
    
    # I assume this functionality will only be needed if one uses encryption
    # if you need it for something else, put the code below outside this if.

    # See if network card gets loaded properly.
    # Uncomment this section if you're having trouble.
    # The sleeps are there so you have the time to examine
    # the command output, as nothing is logged at this point
    # and there's no scrolling.
    #echo "###########################################"
    #echo "###         Debug network               ###"
    #echo "###########################################"
    #sleep 5
    #echo "-------- ls /dev/net"
    #ls /dev/net
    #sleep 5
    #echo "-------- cat /proc/net/dev"
    #cat /dev/net/dev
    #sleep 5
    #echo "-------- ifconfig -a"
    #ifconfig -a
    #sleep 5

    # init network
    # Obviously, you should put in the correct interfaces
    # here, and edit the ip addresses. It's easiest to 
    # use a different ip address than the pc usually
    # has, because the host keys are different and ssh
    # will die over that, unless you explicitly tell it
    # not to EVERY SINGLE TIME. This way, ssh will not
    # see this as the same host and there's no problem.
    # Right now, it's configured for a static ip. Dhcp 
    # is within the possibilities, I just don't use it.
    # It will most likely be difficult though, as it's
    # not automatically included in the initrd. Do
    # yourself a favor and just configure it statically.
    # ;)
    echo "###########################################"
    echo "###      Initializing network           ###"
    echo "###########################################"
    echo "-------- Bringing up lo on 127.0.0.1"
    ifconfig lo 127.0.0.1
    echo "-------- Bringing up eth0 on 192.168.1.34"
    ifconfig eth0 up 192.168.1.34
    echo "-------- Adding 192.168.1.1 as default gw"
    route add -net 127.0.0.0 netmask 255.0.0.0 lo
    route add default gw 192.168.1.1

    # Test if internet connection is ok
    # Uncomment this if the network isn't coming up.
    # Be sure the ip is actually reachable, and don't
    # use a domain name, as there's no nameserver
    # capability at this point.
    #echo "###########################################"
    #echo "###       Test network                  ###"
    #echo "###########################################"
    #sleep 5
    #ping 192.168.1.1 -c 5
    #sleep 5

    # start ssh
    # This kinda speaks for itself. You could set a few
    # options for dropbear if you wanted (-s to disable
    # password login for example).
    echo "###########################################"
    echo "###    Starting ssh daemon              ###"
    echo "###########################################"
    echo "-------- Starting daemon on port 22"
    dropbear -b /etc/dropbear/banner

    # lock system until user provides input, or this script is killed (via ssh)
    bootlock

    # We don't need ssh anymore - kill off dropbear
    echo "###############################################"
    echo "###            Shut down network            ###"
    echo "###############################################"
    killall dropbear

    # Now shut down the network so it doesn't interfere with the normal booting process
    # I'm not sure this is needed, but I've read reports of network issues because the
    # regular boot process couldn't overwrite the initial configuration. Seemed the safe
    # thing to do. Also, I don't like the idea of leaving the network open unprotected
    # (no firewall) any longer than I have to. 
    ifconfig eth0 down
    ifconfig lo down
    

    # Determine if we have to use a LUKS keyfile:
    if [ ! -z "$LUKSKEY" ]; then
      mkdir  /mountkey
      KEYPART=$(echo $LUKSKEY |cut -f1 -d:)
      LUKSPATH="/mountkey$(echo $LUKSKEY |cut -f2 -d:)"
      # Catch possible mount failure:
      if blkid -t TYPE=vfat $KEYPART 1>/dev/null 2>&1 ; then
        MOUNTOPTS="-t vfat -o shortname=mixed"
      else
        MOUNTOPTS="-t auto"
      fi
      mount $MOUNTOPTS $(findfs $KEYPART) /mountkey 2>/dev/null
      # Check if we can actually use this file:
      if [ ! -f $LUKSPATH ]; then
        LUKSKEY=""
      else
        echo ">>> Using LUKS key file: '$LUKSKEY'"
        LUKSKEY="-d $LUKSPATH"
      fi
    fi

    LUKSLIST_DEFERRED=""
    LUKSLIST=$(echo $LUKSDEV | tr -s ':' ' ')
    for LUKSDEV in $LUKSLIST ; do
      if /sbin/cryptsetup isLuks ${LUKSDEV} 1>/dev/null 2>/dev/null ; then
        if echo $ROOTDEV | grep -q "LABEL=" || echo $ROOTDEV | grep -q "UUID=" ; then
          CRYPTDEV="luks$(basename $LUKSDEV)"
        elif [ "x$ROOTDEV" = "x$(basename $ROOTDEV)" ]; then
          CRYPTDEV="$ROOTDEV"
        else
          CRYPTDEV="luks$(basename $LUKSDEV)"
        fi
        echo "Unlocking LUKS encrypted device '${LUKSDEV}' as luks mapped device '$CRYPTDEV':"
        /sbin/cryptsetup ${LUKSKEY} luksOpen ${LUKSDEV} ${CRYPTDEV} /dev/tty0 2>&1
        if [ "$ROOTDEV" = "$LUKSDEV" -o "$ROOTDEV" = "$CRYPTDEV" ] ; then
          ROOTDEV="/dev/mapper/$CRYPTDEV"
        fi
      else
        LUKSLIST_DEFERRED="${LUKSLIST_DEFERRED} ${LUKSDEV}"
      fi
    done
  fi

  # Initialize LVM:
  if [ -x /sbin/vgchange ]; then
    mkdir -p /var/lock/lvm	# this avoids useless warnings
    /sbin/vgchange -ay --ignorelockingfailure 2>/dev/null
    /sbin/udevadm settle --timeout=10
  fi
  
  # Unlock any LUKS encrypted devices that were deferred above but have now
  # become available due to the vgscan (i.e. filesystems on LVM Logical Volumes)

  if [ -x /sbin/cryptsetup -a -n "${LUKSLIST_DEFERRED}" ]; then
    for LUKSDEV in ${LUKSLIST_DEFERRED} ; do
      if /sbin/cryptsetup isLuks ${LUKSDEV} 1>/dev/null 2>/dev/null ; then
        if echo $ROOTDEV | grep -q "LABEL=" || echo $ROOTDEV | grep -q "UUID=" ; then
          CRYPTDEV="luks$(basename $LUKSDEV)"
        elif [ "x$ROOTDEV" = "x$(basename $ROOTDEV)" ]; then
          CRYPTDEV="$ROOTDEV"
        else
          CRYPTDEV="luks$(basename $LUKSDEV)"
        fi
        echo "Unlocking LUKS encrypted device '${LUKSDEV}' as luks mapped device '$CRYPTDEV':"
        /sbin/cryptsetup ${LUKSKEY} luksOpen ${LUKSDEV} ${CRYPTDEV} /dev/tty0 2>&1
        if [ "$ROOTDEV" = "$LUKSDEV" -o "$ROOTDEV" = "$CRYPTDEV" ] ; then
          ROOTDEV="/dev/mapper/$CRYPTDEV"
        fi
      else
        echo "LUKS device '${LUKSDEV}' unavailable for unlocking!"
      fi
    done
    /sbin/udevadm settle --timeout=10
  fi

  # Scan for btrfs multi-device filesystems:
  if [ -x /sbin/btrfs ]; then
    /sbin/btrfs device scan
  fi
  
  # Find root device if a label or UUID was given:
  if echo $ROOTDEV | grep -q "LABEL=" || 
     echo $ROOTDEV | grep -q "UUID=" ; then
    ROOTDEV=$(findfs $ROOTDEV)
  fi

  # Clean up after LUKS unlock using a keyfile:
  if grep -q mountkey /proc/mounts 2>/dev/null ; then
    umount -l /mountkey
    rmdir /mountkey 2>/dev/null
  fi
  
  # Resume state from swap
  if [ "$RESUMEDEV" != "" ]; then
    if ls -l $RESUMEDEV | grep -q "^l" ; then
      #RESUMEDEV=$(ls -l $RESUMEDEV | awk '{ print $NF }')
      RESUMEDEV=$(readlink -f $RESUMEDEV)
    fi
    echo "Trying to resume from $RESUMEDEV"
    RESMAJMIN=$(ls -l $RESUMEDEV | tr , : | awk '{ print $5$6 }')
    echo $RESMAJMIN > /sys/power/resume
  fi

  # Switch to real root partition:
  /sbin/udevadm settle --timeout=10
  echo 0x0100 > /proc/sys/kernel/real-root-dev
  mount -o ro -t $ROOTFS $ROOTDEV /mnt
  
  if [ ! -r /mnt/sbin/init ]; then
    echo "ERROR:  No /sbin/init found on rootdev (or not mounted).  Trouble ahead."
    echo "        You can try to fix it. Type 'exit' when things are done." 
    echo
    /bin/sh
  fi
else
  echo
  echo "RESCUE mode"
  echo
  echo "        You can try to fix or rescue your system now. If you want"
  echo "        to boot into your fixed system, mount your root filesystem"
  echo "        read-only under /mnt:"
  echo
  echo "            # mount -o ro -t filesystem root_device /mnt"
  echo
  echo "        Type 'exit' when things are done."
  echo
  /bin/sh
fi

# Need to make sure OPTIONS+="db_persist" exists for all dm devices
# That should be handled in /sbin/mkinitrd now
/sbin/udevadm info --cleanup-db
/sbin/udevadm control --exit

unset ERR
mount -o move /proc /mnt/proc
mount -o move /sys /mnt/sys
mount -o move /run /mnt/run

[ "$DEVTMPFS" = "1" ] && mount -o move /dev /mnt/dev
echo "${INITRD}:  exiting"
exec switch_root /mnt $INIT $RUNLEVEL


Well, that was all. Don’t forget to run /etc/mkinitrd.conf and then lilo after every alteration. I hope this guide saves you a few days of constant rebooting 😉 But most of all I hope you’ll have fun rooting around in the boot process, how cool is this?