If you are interested in booting your Raspberry Pi 4 without local storage, this guide can help you accomplish it. While most tutorials cover how to do this with NFS, this one uses iSCSI. Additionally, with the latest firmware update for the Pi 4 (as of 2020-04-16), setting up network booting is much simpler. Other guides require modifying the local DHCP server or spinning up a proxy DHCP server to get this working.
Context
I decided that I wanted to get the Raspberry Pi systems I have scattered about into a more maintainable state. I have read countless tales of SD card failures with Pis, and if I’m being honest, I don’t have a good backup strategy for them. I do, however, have a FreeNAS machine, with plenty of space. That got me thinking about how I could utilize that to solve the problem.
Thanks
This post by XLAB helped point me in the right direction, but I found the Pi 4 is sufficiently different from the Pi 3 that it did not work verbatim. My friend Sharon W. helped out a bunch by copy-editing this post.
Requirements
- Raspberry Pi 4
- SD card for the Pi
- FreeNAS (screenshots from 11.3, but older versions probably work)
- Linux machine (I spun up an Ubuntu Server 20.04 VM for this on Hyper-V)
I found it valuable to have a monitor attached to the Pi when debugging why something was not working correctly. You can get through this guide without one, however.
Assumptions
This guide has some assumptions baked in. These things are true in my environment, but may not be in yours. These steps may work without these assumptions, but I have not tested them. Most of these are easy to work around by changing netboot-pi-config.json, which will be introduced below.
- You will use a wired connection (via
eth0) to connect to the Pi - You will statically assign an IP address to the Pi
- You have IPv6 set up on the local network
- Your local timezone is US/Pacific
- You have a local NTP server
- You are comfortable with public key authentication set up for sshing into the
rootaccount on the Pi
Setting up FreeNAS
TFTP Server
I created a new user and group, tftp, and set the permissions on the directory I planned to share accordingly (owned by root, with the group being tftp). The FreeNAS documentation may be useful when setting this up. Here is how I configured this:
NFS Server
In order to share the TFTP folder for the Pi, which contains the contents of the /boot folder, NFS sharing will also need to be set up. Using NFS like this allows the Pi to mount it under /boot, such that any updates that happen in the folder are updated on the TFTP server as well for the next boot.
This step does require that the MAC address of the Pi is known. If it is not already known, skip this for now and come back to it when it is collected off of the Pi later in the guide.
The FreeNAS documentation may be useful, especially if you plan to deviate from this guide. Here is how I configured this:
Note: after everything is setup, you can add the hostname to restrict access to this folder to just the Pi. While doing the setup, however, the Ubuntu server will also need access to the NFS share.
iSCSI Server
FreeNAS now has a wizard that makes this easy to set up. I entered this information into the wizard, and it set everything up correctly:
Building a Custom Image of Raspberry Pi OS (32-bit)
I plan to do this with a bunch of Pis, so I spent the time setting up Packer, along with a plugin to support arm images to make generating an image much easier and repeatable. In my Ubuntu Server VM, I created this config file:
Then, I installed the required packages, Packer, and the arm image builder plugin.
There are many environment variables that will be used throughout this guide on the Ubuntu Server. Be sure to update them to reflect the local environment.
Setting up the Pi 4
Installing
I grabbed the image (located in output-arm-image/image) with WinSCP, flashed it to an SD card with BalenaEtcher, placed the card in the Pi, and powered it up. If a monitor is attached, ssh in once the Pi enables sshd. Otherwise, just wait a few minutes and then connect.
Before moving on, collect the MAC address from the Pi.
This information will be needed back on the imaging server. If the NFS step was skipped above because it was not yet known, go back and set the NFS share up as well.
Updating the EEPROM
The Pi 4 uses an SPI-attached EEPROM to boot the system instead of bootcode.bin that older models of the Pi used (more can be read about it in the EEPROM documentation). The firmware and configuration on the Pi will need to be updated to set up network booting. The documentation for these bootloader settings covers a lot more options that are not utilized here. In order to see the current configuration, run:
When I wrote this, pieeprom-2020-04-16.bin is the current stable release. Be sure to check for newer stable releases, and then reference the EEPROM documentation to see if any additional settings should be set. This guide relies on features that only became available in pieeprom-2020-04-16.bin.
Setting up the TFTP Environment
The initramfs has to be created, and then the entire /boot folder copied over to the TFTP server. Be sure to update the TFTP_ROOT environment variable to be the path the TFTP server serves files from on the FreeNAS machine.
It is worth noting that the call to update-initramfs is tied to the currently running kernel version. As a result, future updates to the kernel will not be reflected in the initramfs without running this command again. I have not had to handle that yet, and it appears that this will get easier to manage in the future. It is an exercise for the reader to tackle this problem, and this Stack Exchange thread has some solutions.
Power down the Pi, and remove the SD card.
Setting up the iSCSI Device
Back on the Ubuntu Server where image was built for the Pi, the iSCSI device the Pi will use as its root device can now be setup.
Connecting to the iSCSI Device
This code relies on environment variables that were set in previous steps on this machine, so if this is a new shell, be sure to copy and paste those in as well.
I am unaware of a programmatic way to determine what device maps to the iSCSI connection that was just made. It might be /dev/sdb or something under /dev/mapper/. Use lsblk --output NAME,KNAME,TYPE,SIZE,MOUNTPOINT to help figure out which device actually represents the iSCSI device.
Creating and Populating the Root Partition
Creating the new partition for the Pi is fairly straightforward. This will create a single partition taking up the entire device. Be sure to update the ISCSI_DEVICE environment variable with the proper device from the previous step.
A new device will be created for the partition. Depending on the original device, it could be /dev/sdb1 or something like /dev/mapper/mpathb-part1. Be sure to update the ISCSI_ROOT_PARTITION environment variable with the proper device for the partition.
Updating /etc/fstab
/etc/fstab needs to be updated to properly mount /boot and / when using the iSCSI device. Be sure to update the PI_MAC environmental variable (that was taken from the Pi earlier) as well as the TFTP_ROOT environment variable.
Update PARTUUID on the TFTP Server
The last step is updating the PARTUUID that is in the cmdline.txt file that the Pi boots with (from the TFTP server) to match the one in the new partition that was created on the iSCSI device.
Booting
After making sure the SD card has been removed from the Pi, it should boot from the network once power is turned on!
Debugging
If things are not working, I strongly suggest hooking up a monitor to the Pi to verify configuration files or what step of the boot process is failing.
TFTP Connections
To see if the Pi is even talking to the TFTP server, check the request logs by running tail -f /var/log/xferlog on the FreeNAS machine. To see the raw traffic to the TFTP server, run tcpdump -vv -i igb0 port 69 (updating igb0 with the network interface used by the TFTP server).
iSCSI Connections
To see if the Pi is even talking to the iSCSI server, check the raw network traffic on the port by running tcpdump -vv -i igb0 port 3260. If there is more than one device connecting to the server, add a host filter.