If you are interested in booting your Raspberry Pi 4 without local storage, this guide can help you accomplish it. While most tutorials cover how to do this with NFS, this one uses iSCSI. Additionally, with the latest firmware update for the Pi 4 (as of 2020-04-16), setting up network booting is much simpler. Other guides require modifying the local DHCP server or spinning up a proxy DHCP server to get this working.
Context
I decided that I wanted to get the Raspberry Pi systems I have scattered about into a more maintainable state. I have read countless tales of SD card failures with Pis, and if I’m being honest, I don’t have a good backup strategy for them. I do, however, have a FreeNAS machine, with plenty of space. That got me thinking about how I could utilize that to solve the problem.
Thanks
This post by XLAB helped point me in the right direction, but I found the Pi 4 is sufficiently different from the Pi 3 that it did not work verbatim. My friend Sharon W. helped out a bunch by copy-editing this post.
Requirements
- Raspberry Pi 4
- SD card for the Pi
- FreeNAS (screenshots from 11.3, but older versions probably work)
- Linux machine (I spun up an Ubuntu Server 20.04 VM for this on Hyper-V)
I found it valuable to have a monitor attached to the Pi when debugging why something was not working correctly. You can get through this guide without one, however.
Assumptions
This guide has some assumptions baked in. These things are true in my environment, but may not be in yours. These steps may work without these assumptions, but I have not tested them. Most of these are easy to work around by changing netboot-pi-config.json
, which will be introduced below.
- You will use a wired connection (via
eth0
) to connect to the Pi - You will statically assign an IP address to the Pi
- You have IPv6 set up on the local network
- Your local timezone is US/Pacific
- You have a local NTP server
- You are comfortable with public key authentication set up for sshing into the
root
account on the Pi
Setting up FreeNAS
TFTP Server
I created a new user and group, tftp
, and set the permissions on the directory I planned to share accordingly (owned by root
, with the group being tftp
). The FreeNAS documentation may be useful when setting this up. Here is how I configured this:
NFS Server
In order to share the TFTP folder for the Pi, which contains the contents of the /boot
folder, NFS sharing will also need to be set up. Using NFS like this allows the Pi to mount it under /boot
, such that any updates that happen in the folder are updated on the TFTP server as well for the next boot.
This step does require that the MAC address of the Pi is known. If it is not already known, skip this for now and come back to it when it is collected off of the Pi later in the guide.
The FreeNAS documentation may be useful, especially if you plan to deviate from this guide. Here is how I configured this:
Note: after everything is setup, you can add the hostname to restrict access to this folder to just the Pi. While doing the setup, however, the Ubuntu server will also need access to the NFS share.
iSCSI Server
FreeNAS now has a wizard that makes this easy to set up. I entered this information into the wizard, and it set everything up correctly:
Building a Custom Image of Raspberry Pi OS (32-bit)
I plan to do this with a bunch of Pis, so I spent the time setting up Packer, along with a plugin to support arm images to make generating an image much easier and repeatable. In my Ubuntu Server VM, I created this config file:
{ | |
"variables": { | |
"eth0_ipv4": "", | |
"eth0_ipv6": "", | |
"eth0_gateway": "", | |
"eth0_dns": "", | |
"hostname": "", | |
"iscsi_initiator_iqn": "", | |
"iscsi_target_iqn": "", | |
"iscsi_target_ip": "", | |
"ntpd_servers": "", | |
"root_pub_key": "" | |
}, | |
"builders": [{ | |
"type": "arm-image", | |
"image_type": "raspberrypi", | |
"iso_url": "http://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2020-05-28/2020-05-27-raspios-buster-lite-armhf.zip", | |
"iso_checksum_type": "sha256", | |
"iso_checksum": "f5786604be4b41e292c5b3c711e2efa64b25a5b51869ea8313d58da0b46afc64" | |
}], | |
"provisioners": [ | |
{ | |
"type": "shell", | |
"environment_vars": [ | |
"DEBIAN_FRONTEND=noninteractive", | |
"DEBCONF_NONINTERACTIVE_SEEN=true" | |
], | |
"inline": [ | |
"echo 'tzdata tzdata/Areas select US' | debconf-set-selections", | |
"echo 'tzdata tzdata/Zones/US select Pacific' | debconf-set-selections", | |
"rm /etc/timezone", | |
"rm /etc/localtime", | |
"dpkg-reconfigure -f noninteractive tzdata" | |
] | |
}, | |
{ | |
"type": "shell", | |
"inline": [ | |
"apt update", | |
"apt full-upgrade -y" | |
] | |
}, | |
{ | |
"type": "shell", | |
"inline": [ | |
"echo 'Setting up network...'", | |
"echo interface eth0 >> /etc/dhcpcd.conf", | |
"echo static ip_address={{user `eth0_ipv4`}} >> /etc/dhcpcd.conf", | |
"echo static ip6_address={{user `eth0_ipv6`}} >> /etc/dhcpcd.conf", | |
"echo static routers={{user `eth0_gateway`}} >> /etc/dhcpcd.conf", | |
"echo static domain_name_servers={{user `eth0_dns`}} >> /etc/dhcpcd.conf" | |
] | |
}, | |
{ | |
"type": "shell", | |
"inline": [ | |
"echo 'Setting up hostname...'", | |
"echo {{user `hostname`}} > /etc/hostname", | |
"sed -i -r -e 's/(.*)raspberrypi(.*?)$/\\1{{user `hostname`}}\\2/g' /etc/hosts" | |
] | |
}, | |
{ | |
"type": "shell", | |
"inline": [ | |
"echo 'Setting up NTP...'", | |
"sed -i -r -e 's/#?NTP.*?$/NTP={{user `ntpd_servers`}}/g' /etc/systemd/timesyncd.conf" | |
] | |
}, | |
{ | |
"type": "shell", | |
"inline": [ | |
"echo 'Setting up sshd...'", | |
"touch /boot/ssh", | |
"sed -i -r -e 's/#?.*?PermitRootLogin.*?$/PermitRootLogin without-password/g' /etc/ssh/sshd_config", | |
"sed -i -r -e 's/#?.*?PasswordAuthentication.*?$/PasswordAuthentication no/g' /etc/ssh/sshd_config", | |
"mkdir -p /root/.ssh/", | |
"chmod 700 /root/.ssh", | |
"echo {{user `root_pub_key`}} >> /root/.ssh/authorized_keys", | |
"chmod 644 /root/.ssh/authorized_keys" | |
] | |
}, | |
{ | |
"type": "shell", | |
"inline": [ | |
"echo 'Disabling wifi...'", | |
"echo 'dtoverlay=disable-wifi' >> /boot/config.txt" | |
] | |
}, | |
{ | |
"type": "shell", | |
"inline": [ | |
"echo 'Disabling bluetooth...'", | |
"echo 'dtoverlay=disable-bt' >> /boot/config.txt" | |
] | |
}, | |
{ | |
"type": "shell", | |
"inline": [ | |
"echo 'Installing additional packages...'", | |
"apt install -y initramfs-tools open-iscsi vim" | |
] | |
}, | |
{ | |
"type": "shell", | |
"inline": [ | |
"echo 'Setting up /etc/iscsi/iscsi.initramfs...'", | |
"echo ISCSI_INITIATOR={{user `iscsi_initiator_iqn`}} > /etc/iscsi/iscsi.initramfs", | |
"echo ISCSI_TARGET={{user `iscsi_target_iqn`}} >> /etc/iscsi/iscsi.initramfs", | |
"echo ISCSI_TARGET_IP={{user `iscsi_target_ip`}} >> /etc/iscsi/iscsi.initramfs", | |
"echo 'Setting up /etc/iscsi/initiatorname.iscsi...'", | |
"echo InitiatorName={{user `iscsi_initiator_iqn`}} > /etc/iscsi/initiatorname.iscsi" | |
] | |
} | |
] | |
} |
Then, I installed the required packages, Packer, and the arm image builder plugin.
# Fetch and Install Dependencies | |
sudo apt install \ | |
cloud-guest-utils \ | |
golang-go \ | |
kpartx \ | |
nfs-common \ | |
open-iscsi \ | |
qemu-user-static | |
# Install Packer | |
https://releases.hashicorp.com/packer/1.5.6/packer_1.5.6_linux_amd64.zip | |
shasum -a256 packer_1.5.6_linux_amd64.zip | |
unzip packer_1.5.6_linux_amd64.zip | |
rm packer_1.5.6_linux_amd64.zip | |
sudo mv packer /usr/local/bin/ | |
sudo chown root:root /usr/local/bin/packer | |
# Get and Build Packer Plugin | |
git clone https://github.com/solo-io/packer-builder-arm-image | |
cd packer-builder-arm-image | |
go mod download | |
go build |
There are many environment variables that will be used throughout this guide on the Ubuntu Server. Be sure to update them to reflect the local environment.
ETH0_IPV4="10.117.0.40/23" | |
ETH0_IPV6="fd36:3eb3:43b0:75::28/64" | |
ETH0_GATEWAY="10.117.0.1" | |
ETH0_DNS="10.117.0.3 fd36:3eb3:43b0:75::3 10.117.0.4 fd36:3eb3:43b0:75::4" | |
HOSTNAME="monitor01.hogs.tswn.us" | |
ISCSI_INITIATOR_IQN="iqn.2019-09.us.tswn.us:monitor01" | |
ISCSI_TARGET_IQN="iqn.2019-09.net.shawnwilsher.firstlightweaveslivingsong.ctl:monitor01" | |
ISCSI_TARGET_IP="10.117.0.10" | |
NTPD_SERVERS="10.117.0.10" | |
ROOT_PUB_KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOLhuqsROt8cbpGzED6l47JXPTTsPyxAOl9Kji3ezjr8 ed25519-key-20200521" | |
# Build the Image | |
sudo packer build \ | |
-var eth0_ipv4="${ETH0_IPV4}" \ | |
-var eth0_ipv6="${ETH0_IPV6}" \ | |
-var eth0_gateway="${ETH0_GATEWAY}" \ | |
-var eth0_dns="${ETH0_DNS}" \ | |
-var hostname="${HOSTNAME}" \ | |
-var iscsi_initiator_iqn="${ISCSI_INITIATOR_IQN}" \ | |
-var iscsi_target_iqn="${ISCSI_TARGET_IQN}" \ | |
-var iscsi_target_ip="${ISCSI_TARGET_IP}" \ | |
-var ntpd_servers="${NTPD_SERVERS}" \ | |
-var root_pub_key="${ROOT_PUB_KEY}" \ | |
../netboot-pi-config.json |
Setting up the Pi 4
Installing
I grabbed the image (located in output-arm-image/image
) with WinSCP, flashed it to an SD card with BalenaEtcher, placed the card in the Pi, and powered it up. If a monitor is attached, ssh in once the Pi enables sshd
. Otherwise, just wait a few minutes and then connect.
Before moving on, collect the MAC address from the Pi.
echo "PI_MAC=$(ifconfig eth0 | awk '/ether/{ print $2;}' | tr ':' '-')" |
This information will be needed back on the imaging server. If the NFS step was skipped above because it was not yet known, go back and set the NFS share up as well.
Updating the EEPROM
The Pi 4 uses an SPI-attached EEPROM to boot the system instead of bootcode.bin
that older models of the Pi used (more can be read about it in the EEPROM documentation). The firmware and configuration on the Pi will need to be updated to set up network booting. The documentation for these bootloader settings covers a lot more options that are not utilized here. In order to see the current configuration, run:
vcgencmd bootloader_config |
When I wrote this, pieeprom-2020-04-16.bin
is the current stable release. Be sure to check for newer stable releases, and then reference the EEPROM documentation to see if any additional settings should be set. This guide relies on features that only became available in pieeprom-2020-04-16.bin
.
cp /lib/firmware/raspberrypi/bootloader/stable/pieeprom-2020-04-16.bin pieeprom.bin | |
rpi-eeprom-config pieeprom.bin > bootconf.txt | |
# Enable pxe booting, and copy over additions. | |
sed -i -r -e 's/BOOT_ORDER.*?$/BOOT_ORDER=0x21/g' bootconf.txt | |
# Set prefix to be the mac address, like normal PXE things. | |
TFTP_IP=$(cat /etc/iscsi/iscsi.initramfs | grep 'ISCSI_TARGET_IP=' | cut -d '=' -f 2) | |
sed -i -r -e 's/TFTP_PREFIX.*?$/TFTP_PREFIX=2/g' bootconf.txt | |
# Set the TFTP IP address (no need to mess with DHCP settings). | |
sed -i -r -e "s/TFTP_IP.*?$/TFTP_IP=${TFTP_IP}/g" bootconf.txt | |
# Set this device's IP address. | |
PI_IP=$(ifconfig eth0 | awk '/inet /{ print $2;}') | |
PI_NETMASK=$(ifconfig eth0 | awk '/netmask/{ print $4;}') | |
sed -i -r -e "s/^\[none\]$/CLIENT_IP=${PI_IP}\nSUBNET=${PI_NETMASK}\n[none]/g" bootconf.txt | |
# Save the new configuration to the firmware, and install it (requires a reboot). | |
rpi-eeprom-config --out pieeprom-netboot.bin --config bootconf.txt pieeprom.bin | |
rpi-eeprom-update -d -f ./pieeprom-netboot.bin | |
shutdown -r now |
Setting up the TFTP Environment
The initramfs
has to be created, and then the entire /boot
folder copied over to the TFTP server. Be sure to update the TFTP_ROOT
environment variable to be the path the TFTP server serves files from on the FreeNAS machine.
TFTP_ROOT="/mnt/rust0/tftp/" | |
PI_HOSTNAME=$(hostname) | |
PI_IP=$(ifconfig eth0 | awk '/inet /{ print $2;}') | |
PI_MAC=$(ifconfig eth0 | awk '/ether/{ print $2;}' | tr ':' '-') | |
PI_NETMASK=$(ifconfig eth0 | awk '/netmask/{ print $4;}') | |
ISCSI_INITIATOR_IQN=$(cat /etc/iscsi/iscsi.initramfs | grep 'ISCSI_INITIATOR=' | cut -d '=' -f 2) | |
ISCSI_TARGET_IP=$(cat /etc/iscsi/iscsi.initramfs | grep 'ISCSI_TARGET_IP=' | cut -d '=' -f 2) | |
ISCSI_TARGET_IQN=$(cat /etc/iscsi/iscsi.initramfs | grep 'ISCSI_TARGET=' | cut -d '=' -f 2) | |
NFS_IP=${ISCSI_TARGET_IP} | |
NFS_ROOT_PATH="${TFTP_ROOT}${PI_MAC}" | |
# Copy over to the TFTP server, via the NFS mount. | |
mkdir -p /nfs/$(hostname -s)-boot | |
mount ${NFS_IP}:${NFS_ROOT_PATH} /nfs/$(hostname -s)-boot | |
rsync -a --delete /boot/ /nfs/$(hostname -s)-boot | |
update-initramfs -v -k "$(uname -r)" -c -b /nfs/$(hostname -s)-boot | |
# Update the cmdline.txt file that the pi boots with | |
# https://www.raspberrypi.org/documentation/configuration/cmdline-txt.md | |
# ip= docs can be found at | |
# https://git.kernel.org/pub/scm/libs/klibc/klibc.git/tree/usr/kinit/ipconfig/README.ipconfig | |
# However, don't put anything after static in the line below, because it completely breaks | |
# everything in a very non-obvious way. | |
sed -i -r -e \ | |
"s/$/ ip=${PI_IP}:::${PI_NETMASK}:${PI_HOSTNAME}:eth0:static ISCSI_INITIATOR=${ISCSI_INITIATOR_IQN} ISCSI_TARGET_NAME=${ISCSI_TARGET_IQN} ISCSI_TARGET_IP=${ISCSI_TARGET_IP} rw/g" \ | |
/nfs/$(hostname -s)-boot/cmdline.txt | |
# Update /boot/config.txt to use our new initramfs. | |
sed -i -r -e \ | |
"s@\[pi4\]@[pi4]\ninitramfs initrd.img-$(uname -r) followkernel@" \ | |
/nfs/$(hostname -s)-boot/config.txt | |
umount /nfs/$(hostname -s)-boot |
It is worth noting that the call to update-initramfs
is tied to the currently running kernel version. As a result, future updates to the kernel will not be reflected in the initramfs
without running this command again. I have not had to handle that yet, and it appears that this will get easier to manage in the future. It is an exercise for the reader to tackle this problem, and this Stack Exchange thread has some solutions.
Power down the Pi, and remove the SD card.
Setting up the iSCSI Device
Back on the Ubuntu Server where image was built for the Pi, the iSCSI device the Pi will use as its root device can now be setup.
Connecting to the iSCSI Device
This code relies on environment variables that were set in previous steps on this machine, so if this is a new shell, be sure to copy and paste those in as well.
sudo iscsiadm \ | |
--mode discovery \ | |
--type sendtargets \ | |
--portal ${ISCSI_TARGET_IP} | |
sudo iscsiadm \ | |
--mode node \ | |
--targetname ${ISCSI_TARGET_IQN} \ | |
--portal ${ISCSI_TARGET_IP} \ | |
--login |
I am unaware of a programmatic way to determine what device maps to the iSCSI connection that was just made. It might be /dev/sdb
or something under /dev/mapper/
. Use lsblk --output NAME,KNAME,TYPE,SIZE,MOUNTPOINT
to help figure out which device actually represents the iSCSI device.
Creating and Populating the Root Partition
Creating the new partition for the Pi is fairly straightforward. This will create a single partition taking up the entire device. Be sure to update the ISCSI_DEVICE
environment variable with the proper device from the previous step.
ISCSI_DEVICE="/dev/sdb" | |
sudo parted ${ISCSI_DEVICE} mklabel gpt | |
sudo parted --align optimal ${ISCSI_DEVICE} mkpart primary ext4 0% 100% |
A new device will be created for the partition. Depending on the original device, it could be /dev/sdb1
or something like /dev/mapper/mpathb-part1
. Be sure to update the ISCSI_ROOT_PARTITION
environment variable with the proper device for the partition.
ISCSI_ROOT_PARTITION="/dev/sdb1" | |
# Create the ext4 Filesystem | |
sudo mkfs.ext4 /${ISCSI_ROOT_PARTITION} | |
ISCSI_ROOT_DIR="$(mktemp -d)" | |
sudo mount "${ISCSI_ROOT_PARTITION}" ${ISCSI_ROOT_DIR} | |
# Mount the Root Partion in the Generated Image | |
CREATION_OUTPUT=$(sudo partx --add -v output-arm-image/image) | |
LOOP_DEVICE=$(echo ${CREATION_OUTPUT} | sed -E 's@.*(/dev/loop[0-9]+).*@\1@') | |
IMAGE_ROOT_DIR="$(mktemp -d)" | |
sudo mount "${LOOP_DEVICE}p2" ${IMAGE_ROOT_DIR} | |
# Begin the Copy | |
sudo rsync -a --info=progress2 "${IMAGE_ROOT_DIR}/" ${ISCSI_ROOT_DIR} | |
# The generated image is no longer needed, so we can cleanup. | |
sudo umount ${IMAGE_ROOT_DIR} | |
sudo partx --delete -v ${LOOP_DEVICE} |
Updating /etc/fstab
/etc/fstab
needs to be updated to properly mount /boot
and /
when using the iSCSI device. Be sure to update the PI_MAC
environmental variable (that was taken from the Pi earlier) as well as the TFTP_ROOT
environment variable.
PI_MAC="dc-a6-32-7a-b4-02" | |
TFTP_ROOT="/mnt/rust0/tftp/" | |
# Update /boot to point to the NFS mount of the TFTP folder for the Pi. | |
NFS_IP=$(cat ${ISCSI_ROOT_DIR}/etc/iscsi/iscsi.initramfs | grep 'ISCSI_TARGET_IP=' | cut -d '=' -f 2) | |
NFS_ROOT_PATH="${TFTP_ROOT}${PI_MAC}" | |
sudo sed -i -r -E \ | |
"s@.*/boot +.*@${NFS_IP}:${NFS_ROOT_PATH} /boot nfs defaults,vers=4.1,proto=tcp 0 0@" \ | |
${ISCSI_ROOT_DIR}/etc/fstab | |
# Update / to point to the iSCSI drive. | |
ISCSI_ROOT_PARTUUID=$(sudo blkid | grep ${ISCSI_ROOT_PARTITION} | tr ' ' '\n' | grep PARTUUID | cut -d '=' -f 2 | tr -d '"') | |
sudo sed -i -r -E \ | |
"s@.*/ +.*@PARTUUID=${ISCSI_ROOT_PARTUUID} / ext4 _netdev,noatime 0 1@" \ | |
${ISCSI_ROOT_DIR}/etc/fstab | |
# The connection to the iSCSI device is no longer needed, so clean it up. | |
sudo umount ${ISCSI_ROOT_DIR} | |
sudo iscsiadm --m node -T ${ISCSI_TARGET_IQN} --portal ${ISCSI_TARGET_IP}:3260 -u | |
sudo iscsiadm -m node -o delete -T ${ISCSI_TARGET_IQN} --portal ${ISCSI_TARGET_IP}:3260 |
Update PARTUUID
on the TFTP Server
The last step is updating the PARTUUID
that is in the cmdline.txt
file that the Pi boots with (from the TFTP server) to match the one in the new partition that was created on the iSCSI device.
# Mount the NFS Folder | |
NFS_DIR="$(mktemp -d)" | |
sudo mount ${NFS_IP}:${NFS_ROOT_PATH} ${NFS_DIR} | |
# Update the PARTUUID | |
sudo sed -i -r -E \ | |
"s@(.*root=PARTUUID=)[A-Za-z0-9-]+(.*)@\1${ISCSI_ROOT_PARTUUID}\2@" \ | |
${NFS_DIR}/cmdline.txt | |
# Unmount the NFS Folder | |
sudo umount ${NFS_DIR} |
Booting
After making sure the SD card has been removed from the Pi, it should boot from the network once power is turned on!
Debugging
If things are not working, I strongly suggest hooking up a monitor to the Pi to verify configuration files or what step of the boot process is failing.
TFTP Connections
To see if the Pi is even talking to the TFTP server, check the request logs by running tail -f /var/log/xferlog
on the FreeNAS machine. To see the raw traffic to the TFTP server, run tcpdump -vv -i igb0 port 69
(updating igb0
with the network interface used by the TFTP server).
iSCSI Connections
To see if the Pi is even talking to the iSCSI server, check the raw network traffic on the port by running tcpdump -vv -i igb0 port 3260
. If there is more than one device connecting to the server, add a host
filter.