April 14, 2012

Remote Wipe of Debian Linux Server

Filed under: Technical — Tags: , — James Bunton @ 12:05 am

I recently retired an old server of mine, cerberus. It was hosted in a remote data centre to which I have no physical access. My goal was to zero the drives before powering it off for the last time.

To wipe the machine properly the kernel must not be using the hard drives anymore. To do this pivot_root into a tmpfs mount and then kill all remaining processes. Here are the steps I used. Note these may or may not work for you. If you don’t understand what something does you should probably not continue! This is more of a set of rough notes in case I need to do this again than any real guide.

  1. Shut down as much stuff as possible, you should only have init, random kernel junk, sshd and your shells running
  2. Run these commands:
    swapoff -a
    mkdir /scrubber
    mount -t tmpfs scrubber /scrubber
    cd /scrubber
    zcat /boot/initrd.img-"$(uname -r)" | cpio --extract
    for i in dev proc sys; do
        mkdir "$i"
        mount --bind /"$i" "$i"
    done
    mount --bind /dev/pts dev/pts
    cp -H /bin/{bash,rm,ps,pivot_root,nc} bin/
    cp -H /lib/libproc*.so lib/
    cp -H /lib/libncurses*.so* lib/
    echo -e '#!/bin/sh\nsleep 99999' > sbin/init
    chmod 0755 sbin/init
    mkdir old_root
    
  3. Now chroot into /scrubber and make sure all the tools you need are there and working. Depending on your distro and version you may need to copy different libraries or copy them from different directories. Use ldd to find them.
  4. Now start a shell with remote access from inside the root:
    echo 'mkfifo inp; nc -lp 9998 < <(bash <inp 2>&1) >inp &' | chroot . /bin/bash
  5. Connect to this shell from another PC with nc yourhost 9998. You should have a usable shell, no autocomplete or fancies, but usable. Make sure it works before continuing.
  6. Now pivot_root to the tmpfs and tell init to reload from the dummy version. Run these from the SSH shell:
    pivot_root . old_root
    LD_LIBRARY_PATH=/old_root/lib/ /old_root/sbin/telinit u
    
  7. Close your SSH shell. Go back to your netcat shell and run ps aux. Kill sshd, now there should be no processes running in old_root anymore.
  8. Now unmount stuff. cat /proc/mounts in order to find out what stuff is still mounted in the old root. You can lazy unmount virtual filesystems like umount -l /dev/pts for example. But any filesystems on disk should be properly unmounted to ensure the kernel won’t try to use them again.
  9. If you’ve reached this point, you should have cat /proc/mounts /proc/swaps showing nothing on any hard drives. Now you can just use dd if=/dev/zero of=/dev/sda bs=1048576 to erase stuff. Run this for each drive.