Ubuntu ARM64 on Raspberry Pi CM4 with NVMe SSD

As you can imagine from the number of acronyms in the title, this took a huge amount of fiddling and was very confusing, so documenting it here for my own benefit and others’. 🙂

Those blue LEDs >_<

I wanted to run Ubuntu Server ARM64 on a Raspberry Pi CM4 dev board using a NVMe SSD for at least the root filesystem (and ideally booting from it)

Step 1: Update the CM4’s bootloader firmware

The CM4 probably has an older bootloader firmware installed from the factory that doesn’t support NVMe. This step is best done from a Linux PC: I used an old Acer laptop running Zorin 16. The below commands are to be entered in a terminal window:

First you need the latest version of the Raspberry Pi USB boot tool. This has to be built from source, and you may need to install some dependencies first (-1000 geek points if you didn’t already have git and build-essential 🙂 )

sudo apt install git
sudo apt install build-essential
sudo apt install libusb-1.0-0-dev

Now it’s time to build the USB boot tool from source…

git clone --depth=1 https://github.com/raspberrypi/usbboot
cd usbboot

Now you need to edit the boot order settings and get the latest EEPROM image. This part is adapted from this tutorial which doesn’t work verbatim any more (the “nvme” folder doesn’t exist in the usbboot repository, but substituting the “recovery” folder works)

cd recovery
sed -i -e '/^BOOT_ORDER=/ s/=.*$/=0xf25416/' boot.conf

NVMe on Raspberry Pi is still kind of bleeding edge, so it’s best to use the newest versions of everything. pieeprom-2021-07-06.bin was still the latest image at the time of writing:

rm -f pieeprom.original.bin
curl -L -o pieeprom.original.bin https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/beta/pieeprom-2021-07-06.bin

Note: At this point the update script crashed complaining that it couldn’t find Python. I guess Zorin 16 removed Python 2.x and forgot to make Python 3 the default? The python-is-python3 package fixed that for me. Hopefully your Linux distro has a functioning “python” already, in which case don’t execute this command, it’ll probably break something!

sudo apt install python-is-python3

Finally it’s time to burn your changes to the CM4… The “disable eMMC Boot” jumper must be fitted on J2 for this to work (Unless you have a CM4 Lite I guess…)

cd ..
sudo ./rpiboot -d recovery

Now connect the CM4 Dev board J11 “USB Slave” to a USB port of your Linux PC and apply power to the 12V DC input.

The rpiboot tool should boot up the CM4 over USB and flash the new EEPROM image and boot order settings. Once the command line spew stops and the power LED on the dev board starts to blink rapidly, the process is complete and you can disconnect power.

Step 2: Try to boot from NVMe, fail

At this point I already had an ARM64 Ubuntu 20.04 LTS image downloaded from here and uncompressed to my Gigabyte GP-GSM2NE3128GNTD SSD. So I removed the SD card from my CM4 dev board and tried to boot from the NVMe.

I got a surprisingly informative error message from the Raspberry Pi bootloader (on a monitor connected to HDMI0)

start4.elf is not compatible. nvme boot requires newer software

A bit of Googling got me to this forum thread:

So my first attempt at a fix was to update a 32-bit Raspbian SD card image with rpi-update and then copy the start4.elf and fixup4.dat files from that image to the NVMe boot partition. This got me to the next problem: The Ubuntu 20.04 image uses a version of U-Boot that doesn’t support NVMe, so it just sticks waiting for a boot device.

Next I tried downloading the Ubuntu Server 21.04 image and burning it to a SD card. I planned to go through the full process documented in the forum thread, starting with upgrading the Ubuntu image, so I inserted the SD card and powered up.

Step 3: Unexpected success

Imagine my surprise when it booted into Ubuntu 20.04 from the NVMe instead of 21.04 from the SD card as I was expecting. 😀

I guess what must have happened is that the boot partition on the 21.04 SD card had a version of UBoot that recognised the NVMe and treated it as a higher priority than the SD.

Confirming that the SD card is unused and root filesystem is on the NVMe SSD. It’s fast 8)

The next gambit would be to copy the boot partition from the SD card to the NVMe and see if it can boot directly from NVMe. But it works well enough as is, I don’t want to break it! This is left as an exercise for the reader 😉

Step 5: Try to make it boot directly from NVMe

Unfortunately on subsequent boots it flipflopped between NVMe and SD card as the root partition. So I thought since it’s Friday afternoon I should try to get it to boot directly from NVMe.

I started by deleting all files in the boot partition on the NVMe and replacing with the ones from the Ubuntu 21.04 SD card. The kernel is in /boot/vmlinuz, while the modules are stored on the root filesystem, so this would normally cause the modules to fail to load, except I noticed that I somehow managed to update the Ubuntu 20.04 installation on the NVMe so it was running the same kernel and modules as 21.04.

This still didn’t work, so I went back to that forum thread and replaced start4.elf and fixup4.dat with the ones from the updated Raspbian image.

Finally I got a direct boot from NVMe, with what I guess is Ubuntu 21.04 kernel and 20.04 userland. That works fine for me, I’d rather it ran a LTS version of Ubuntu.

Wolfson Pi Audio Card – first impressions

Ever since the Raspberry Pi came out, I’ve been experimenting with its audio capabilities. The latest audio gizmo available for it is the Wolfson Pi Audio Card, which promises 24 bit, 192kHz recording and playback, with analog and digital I/O, for a very reasonable price. So of course I ordered one straight away. 🙂

After waiting a month I finally got my hands on it. The software installation is somewhat unclear so I will document what I did here. I didn’t want to use the Wolfson official image as it was a massive 8GB download. I started with a copy of the image that I developed for PiTunes, and applied this patch to it, which adds the Wolfson kernel and the support files for the audio card. I then changed mpd.conf to use audio output device hw0,0 (it was previously 1,0 for the USB audio device) and added a call to SPDIF_playback.sh in my .bash_login file, to set the card up for digital output.

I also removed the invocation of pikeyd from /etc/rc.local, as the keypad and encoder were not present. They can’t be used anyway, since the Wolfson audio card hogs all of the GPIO pins. It doesn’t really matter, as MPD can always be controlled remotely.

On firing this up, I was surprised to find that it worked first time! 🙂 I verified the output to be bit perfect at 24 bit, 96kHz. This is possibly the best value for money HD audio source you can get anywhere: you should be able to pick up a Raspberry Pi, a Wolfson Audio Card, a wifi dongle and a hard disk for under £100.

I <3 Soma FM

Lately I’ve been listening to Soma FM a lot. I especially like their Groove Salad channel. They play Boards of Canada all the time, but that’s not necessarily a bad thing.

Anyway, their 128k MP3 stream has pretty decent quality – I guess it’s technically the same as a DAB radio station – and it plays on just about anything. Winamp on my PC, with its Delta 1010 hooked up to the stereo, no problem. Linosaur, my Linux server, was a little trickier, but once I figured out the stream URL, I pasted it into phpMP and saved it as a playlist, and off it went. I have a LCD and keypad hooked up to linosaur using LCDproc and a modified version of lcd-stuff, and I can play stuff from that.

Perhaps the coolest thing is that it plays on my Pocket PC (an old Ipaq hx4700) using GSPlayer. When you visit Soma FM’s page, it downloads a Winamp playlist onto your machine to start playback. That contains several streams, I guess as backup if some of them are down. I just got this groovesalad.pls file and copied it onto my PDA’s memory card. (I prefer to keep stuff on a card, since my hx4700 crashes and wipes its RAM with monotonous regularity.) When I run GSPlayer (which I also keep on the card to save reinstalling every 2 weeks) the playlist can be opened and the fat beats begin.

So, if I do this, and dangle my DSL modem/router out of the window, I can listen in the back yard while washing my bike. I’m off to do that now 🙂

Nix command of the week: tail

If you’ve messed around with Linux, you might well have used the “tail” command. All it does is print the last few lines of a file. For example,

tail -n 10 /var/log/apache2/access.log

will show you the last 10 pages that the webserver served. (Assuming you’re running the 2.x version of Apache.)

But for extra geeky thrills, the -f option will make tail watch the file and print out any new lines as they appear. So:

tail -f /var/log/apache2/access.log

will print out whatever Apache is putting into the log file as it is putting it in. Handy for keeping an eye on log files when you’re debugging stuff.

You can crank the geeky thrills up another level by pipelining it into something else, like our old friend from Unix school, grep. For instance,

tail -f /var/log/apache2/access.log |grep sheep

will alert you whenever anyone tries to access a page with “sheep” in the filename. How useful!

PS: press Ctrl-C to exit.