Categories
Technology

Network Booting a Raspberry Pi 4 with an iSCSI Root via FreeNAS

If you are interested in booting your Raspberry Pi 4 without local storage, this guide can help you accomplish it. While most tutorials cover how to do this with NFS, this one uses iSCSI. Additionally, with the latest firmware update for the Pi 4 (as of 2020-04-16), setting up network booting is much simpler. Other guides require modifying the local DHCP server or spinning up a proxy DHCP server to get this working.

Context

I decided that I wanted to get the Raspberry Pi systems I have scattered about into a more maintainable state. I have read countless tales of SD card failures with Pis, and if I’m being honest, I don’t have a good backup strategy for them. I do, however, have a FreeNAS machine, with plenty of space. That got me thinking about how I could utilize that to solve the problem.

Thanks

This post by XLAB helped point me in the right direction, but I found the Pi 4 is sufficiently different from the Pi 3 that it did not work verbatim. My friend Sharon W. helped out a bunch by copy-editing this post.

Requirements

  • Raspberry Pi 4
  • SD card for the Pi
  • FreeNAS (screenshots from 11.3, but older versions probably work)
  • Linux machine (I spun up an Ubuntu Server 20.04 VM for this on Hyper-V)

I found it valuable to have a monitor attached to the Pi when debugging why something was not working correctly. You can get through this guide without one, however.

Assumptions

This guide has some assumptions baked in. These things are true in my environment, but may not be in yours. These steps may work without these assumptions, but I have not tested them. Most of these are easy to work around by changing netboot-pi-config.json, which will be introduced below.

  • You will use a wired connection (via eth0) to connect to the Pi
  • You will statically assign an IP address to the Pi
  • You have IPv6 set up on the local network
  • Your local timezone is US/Pacific
  • You have a local NTP server
  • You are comfortable with public key authentication set up for sshing into the root account on the Pi

Setting up FreeNAS

TFTP Server

I created a new user and group, tftp, and set the permissions on the directory I planned to share accordingly (owned by root, with the group being tftp). The FreeNAS documentation may be useful when setting this up. Here is how I configured this:

NFS Server

In order to share the TFTP folder for the Pi, which contains the contents of the /boot folder, NFS sharing will also need to be set up. Using NFS like this allows the Pi to mount it under /boot, such that any updates that happen in the folder are updated on the TFTP server as well for the next boot.

This step does require that the MAC address of the Pi is known. If it is not already known, skip this for now and come back to it when it is collected off of the Pi later in the guide.

The FreeNAS documentation may be useful, especially if you plan to deviate from this guide. Here is how I configured this:


Note: after everything is setup, you can add the hostname to restrict access to this folder to just the Pi. While doing the setup, however, the Ubuntu server will also need access to the NFS share.

iSCSI Server

FreeNAS now has a wizard that makes this easy to set up. I entered this information into the wizard, and it set everything up correctly:

Building a Custom Image of Raspberry Pi OS (32-bit)

I plan to do this with a bunch of Pis, so I spent the time setting up Packer, along with a plugin to support arm images to make generating an image much easier and repeatable. In my Ubuntu Server VM, I created this config file:

Then, I installed the required packages, Packer, and the arm image builder plugin.

There are many environment variables that will be used throughout this guide on the Ubuntu Server. Be sure to update them to reflect the local environment.

Setting up the Pi 4

Installing

I grabbed the image (located in output-arm-image/image) with WinSCP, flashed it to an SD card with BalenaEtcher, placed the card in the Pi, and powered it up. If a monitor is attached, ssh in once the Pi enables sshd. Otherwise, just wait a few minutes and then connect.

Before moving on, collect the MAC address from the Pi.

This information will be needed back on the imaging server. If the NFS step was skipped above because it was not yet known, go back and set the NFS share up as well.

Updating the EEPROM

The Pi 4 uses an SPI-attached EEPROM to boot the system instead of bootcode.bin that older models of the Pi used (more can be read about it in the EEPROM documentation). The firmware and configuration on the Pi will need to be updated to set up network booting. The documentation for these bootloader settings covers a lot more options that are not utilized here. In order to see the current configuration, run:

When I wrote this, pieeprom-2020-04-16.bin is the current stable release. Be sure to check for newer stable releases, and then reference the EEPROM documentation to see if any additional settings should be set. This guide relies on features that only became available in pieeprom-2020-04-16.bin.

Setting up the TFTP Environment

The initramfs has to be created, and then the entire /boot folder copied over to the TFTP server. Be sure to update the TFTP_ROOT environment variable to be the path the TFTP server serves files from on the FreeNAS machine.

It is worth noting that the call to update-initramfs is tied to the currently running kernel version. As a result, future updates to the kernel will not be reflected in the initramfs without running this command again. I have not had to handle that yet, and it appears that this will get easier to manage in the future. It is an exercise for the reader to tackle this problem, and this Stack Exchange thread has some solutions.

Power down the Pi, and remove the SD card.

Setting up the iSCSI Device

Back on the Ubuntu Server where image was built for the Pi, the iSCSI device the Pi will use as its root device can now be setup.

Connecting to the iSCSI Device

This code relies on environment variables that were set in previous steps on this machine, so if this is a new shell, be sure to copy and paste those in as well.

I am unaware of a programmatic way to determine what device maps to the iSCSI connection that was just made. It might be /dev/sdb or something under /dev/mapper/. Use lsblk --output NAME,KNAME,TYPE,SIZE,MOUNTPOINT to help figure out which device actually represents the iSCSI device.

Creating and Populating the Root Partition

Creating the new partition for the Pi is fairly straightforward. This will create a single partition taking up the entire device. Be sure to update the ISCSI_DEVICE environment variable with the proper device from the previous step.

A new device will be created for the partition. Depending on the original device, it could be /dev/sdb1 or something like /dev/mapper/mpathb-part1. Be sure to update the ISCSI_ROOT_PARTITION environment variable with the proper device for the partition.

Updating /etc/fstab

/etc/fstab needs to be updated to properly mount /boot and / when using the iSCSI device. Be sure to update the PI_MAC environmental variable (that was taken from the Pi earlier) as well as the TFTP_ROOT environment variable.

Update PARTUUID on the TFTP Server

The last step is updating the PARTUUID that is in the cmdline.txt file that the Pi boots with (from the TFTP server) to match the one in the new partition that was created on the iSCSI device.

Booting

After making sure the SD card has been removed from the Pi, it should boot from the network once power is turned on!

Debugging

If things are not working, I strongly suggest hooking up a monitor to the Pi to verify configuration files or what step of the boot process is failing.

TFTP Connections

To see if the Pi is even talking to the TFTP server, check the request logs by running tail -f /var/log/xferlog on the FreeNAS machine. To see the raw traffic to the TFTP server, run tcpdump -vv -i igb0 port 69 (updating igb0 with the network interface used by the TFTP server).

iSCSI Connections

To see if the Pi is even talking to the iSCSI server, check the raw network traffic on the port by running tcpdump -vv -i igb0 port 3260. If there is more than one device connecting to the server, add a host filter.

Categories
Technology

UniFi Controller on DreamHost VPS

I recently purchased a UniFi UAP-PRO for my home wireless. I choose it because it is commercial grade hardware with good management software for a low price (comparatively). It then occurred to me that I could take advantage of my DreamHost VPS that I barely use to host the controller software so I don’t need to bother having it on any of my local computers. The EdgeRouter Lite makes it trivial to automatically point your access points to a place in the cloud with a given IP address, so the hardest part was going to be getting the software running on my VPS.

Once I got on a newer version of DreamHost’s VPS offering (I was on something running Debian 5 before I switched to one running Ubuntu 12.04), I had a bit of a rocky start. Some instructions I found online were outdated and had me install a very old version of the controller software. I was trying to import the settings I had done on my local controller so I didn’t have to set everything up again, and that import process wasn’t going to work out with that old controller software. I’ve got it working now, so I wanted to share the steps that worked for me so hopefully nobody else has to go through the pains I did.

Step One: Get a newer version of MongoDB

We’ll want to get a newer version than what is installed by default, so simply follow the instructions from MongoDB (version 2.4).

Step Two: Follow the release instructions to install the controller

As of this writing, 4.6.6 is the latest version. In the announcement thread for that version, search for “UniFi Controller APT howto”, and follow those instructions (skipping step two since we did that in step one from this blog post).

Step Three: Load our controller and import our config

I exported my local controller’s config (Settings -> Maintenance -> Download Backup Settings) before doing this next step. When we navigate to our server’s address (over https on port 8443), we’re given the option to import a config. Once we’ve imported it, the service will restart, and then we’ll be able to point our access points to our controller. Note: we can also create a completely new config.

Step Four: Set the Controller Hostname/IP

The last step is to open the Settings pane, clicking the Controller tab and entering the hostname or IP address of our controller.

Categories
Mozilla Technology

Using Xperf to Track I/O

Back in December when I was working on the Places branch, I was using Xperf quite a bit to try and figure out why we were regressing certain performance tests. Xperf is an incredibly powerful tool, but it’s really hard to get it to do what you want sometimes. This is largely due to the fact that the documentation isn’t great, and there appears to be wrong information on the Internet (from blogs, which tend to be more useful than the documentation).

I cared about getting information about hard faults, file I/O, and disk I/O to see if patterns changed with my work. In order to accomplish this, I started Xpef like this (from an admin console):
xperf -start "NT Kernel Logger" -on PROC_THREAD+LOADER+HARD_FAULTS+FILE_IO+FILE_IO_INIT+DISK_IO -stackwalk FileWrite+FileRead+FileFlush -MaxBuffers 1024 -BufferSize 1024 -f output.etl
The first list options after -on is a list of providers. To see the list of installed kernel providers on your system and what they do, open up a command prompt, and run xperf -providers K.
The options after -stackwalk tells Xperf to get call stacks for certain events. The full list of events supported can be found here. I found it very difficult to get stacks, and if you are on a 64-bit version of windows, you have an extra hoop to jump through. I was actually never able to get stacks from an optimized build with symbols, so I ended up just using debug builds when I needed stacks, and opt builds when I wanted good numbers for everything else.
The -MaxBuffers and -BufferSize arguments were useful to prevent events from being dropped (if Xperf doesn’t have enough memory set aside to record an event, it just drops it). You can tweak those values to your needs, but the values I used should be fine.

Once I ran that command in my console, I’d perform the test I wanted to get data on. Upon completion, Xperf needs to be told to stop, and then have it merge (merging may be unnecessary when you only use kernel providers, but I never tested this theory):
xperf -stop "NT Kernel Logger"
xperf -merge output.etl output_final.etl

You can now open output_final.etl to examine the data you just gathered!

Categories
Mozilla Technology

Highlight Warnings in Make

Curtis just gave me this incredibly handy piece of code that higlights errors and warnings in make output. Now, when I’m building, all the warnings are highlighted in yellow, and the errors in red. Just put the following in your bash profile script:

make()
{
  pathpat="(/[^/]*)+:[0-9]+"
  ccred=$(echo -e "\033[0;31m")
  ccyellow=$(echo -e "\033[0;33m")
  ccend=$(echo -e "\033[0m")
  /usr/bin/make "$@" 2>&1 | sed -E -e "/[Ee]rror[: ]/ s%$pathpat%$ccred&$ccend%g" -e "/[Ww]arning[: ]/ s%$pathpat%$ccyellow&$ccend%g"
  return ${PIPESTATUS[0]}
}

Of course, improvements and more ideas welcome! Thanks goes to Curtis for this!

Categories
Mozilla Personal Technology

I’m in a podcast!

A little while ago I got interviewed by Anthony Bryan from the metalink project. I feel sorry for him because it took me several long months to actually get the time to sit down and talk to him. Anyway, you can check out the podcast here. They used an old facebook photo of mine, so, uh, pardon the odd image of me. I figured it could be worse though.

It’s worth a listen though. I talk about some of the features of the new download manager (old news now, but he ask for this interview a while ago…), how I got involved with the Mozilla project, and a few other interesting tidbits including what I’ve currently been working on. There is other interesting things in there too!