Intro

It's been quite a long while since the 64-bit CPUs took over the world. It seems that most of the ecosystem moved on, and people stopped paying close attention to the 32-bit compatibility issues. I know I did. It's all fine - compatibility can be quite a pain for very negligible gain. However, some notable software projects are still stuck in the 32-bit-only rut. Furthermore, it seems to be the case even when their primary target platforms have been 64-bit capable for quite a long time now.

One of the projects in the first category is Bitcoin. The Bitcoin developers made their stable release without noticing that the crypto unit tests fail on 32-bit platforms. On the other side of the spectrum, there is the RaspberryPi software stack. It could not run well in 64-bit mode up until a couple of days ago. The good people at Balena Linux distro did the necessary porting work and submitted it to the upstream RPi kernel. This work is so fresh and has been done mainly by a third-party, even though the boards from RaspberryPi 2B 1.2 onwards have an ARMv8 CPU. That's early 2016. I guess that this situation is the result of not pushing the changes to the upstream kernel that seems to be very common in the embedded world and the lack of documentation.

The reason I am writing about this is that I have just hit both of these issues when trying to run a full Bitcoin node on my fresh and new RaspberryPi 4B. I ended up having to install stock Debian on it. It was not exactly hard, but there seemed to have been no instruction on the Internet, so I write down what I did here.

Prerequisites

Except for a RaspberryPi board, you will need a microSD card and a Linux system that can write to this card. Following what Raspbian does, I partitioned my card in the following way:

]==> fdisk /dev/sdb

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/sdb: 7.41 GiB, 7948206080 bytes, 15523840 sectors
Disk model: USB SD Reader   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x8c7e33d1

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sdb1         8192   532480   524289  256M  c W95 FAT32 (LBA)
/dev/sdb2       540672 15523839 14983168  7.1G 83 Linux

The first partition is the boot partition holding the VideoCore GPU firmware necessary to boot a kernel as well as the Linux kernel image. I will discuss how to get both later. The second partition is for the root filesystem. I mounted them in my host system as described below and the rest of this document follows this convention.

]==> mount /dev/sdb2 /mnt
]==> mkdir /mnt/boot
]==> mount /dev/sdb1 /mnt/boot/

Furthermore, you will need to have the following packages installed in the host system:

]==> apt-get install debootstrap qemu-user-static
]==> apt-get install gcc-aarch64-linux-gnu bison flex python-pyelftools device-tree-compiler

The packages in the first group are necessary to bootstrap the root filesystem, while the ones in the second group are needed to cross-compile the kernel image.

System installation

You'll need the arm64 flavor of Debian. I use the testing distribution for pretty much every system I have, but you probably can successfully run any other. I also like using Midnight Commander, so I include that package in the target installation. The include specification is a comma-separated list, and you can declare any package you want there. My network operator is Init7, so I use their mirror. You should select one that is close to you. Finally, the target for the installation is /mnt.

]==> qemu-debootstrap --include=mc --arch=arm64 testing /mnt/ http://mirror.init7.net/debian/

After the bootstrapping process completes, you need to build the kernel image that can run on the board. The official website describes the process quite well, but I like building proper Debian packages instead. The commit with hash db690083a4 worked fine for me.

]==> cd /tmp
]==> git clone https://github.com/raspberrypi/linux.git
]==> cd linux
]==> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
]==> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- KBUILD_IMAGE=arch/arm64/boot/Image -j12 bindeb-pkg
]==> make ARCH=arm64 CROSS_COMPILE=/usr/bin/aarch64-linux-gnu- dtbs
]==> cp /tmp/linux/arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb /mnt/boot/
]==> mkdir /mnt/boot/overlays
]==> cp /tmp/linux/arch/arm64/boot/dts/overlays/vc4-fkms-v3d.dtbo /mnt/boot/overlays/
]==> cp /tmp/linux*deb /mnt

The first make command configures the kernel for this particular board. The second one builds the Debian packages with the kernel, the kernel headers, and the kernel userspace headers. The third one compiles the Device Tree files. The last four commands install the necessary Device Tree files in the boot partition and copy the kernel packages to the root filesystem.

You can now chroot into the root filesystem, install the kernel, create a user account, and change passwords. We also need to prevent the linux-libc-dev package from being updated using apt-mark.

]==> chroot /mnt /usr/bin/qemu-aarch64-static /bin/bash
]==> dpkg -i linux-*deb
]==> rm linux*deb
]==> apt-mark hold linux-libc-dev
]==> useradd -m youruser
]==> passwd youruser
]==> passwd root

You then need to edit the files that play a role in the boot process. The relevant listings are presented below. Make sure that you got the partition setup right. You can get the PARTUUIDs by listing: ls -l /dev/disk/by-partuuid.

]==> cat /etc/hostname 
cryptopi
]==> cat /etc/fstab   
PARTUUID=8c7e33d1-01  /boot           vfat    defaults          0       2
PARTUUID=8c7e33d1-02  /               ext4    defaults,noatime  0       1
]==> cat /etc/apt/sources.list
deb http://mirror.init7.net/debian/ testing main contrib non-free
deb-src http://mirror.init7.net/debian/ testing main contrib non-free

deb http://security.debian.org testing-security main contrib non-free
deb-src http://security.debian.org testing-security main contrib non-free

Finally, all that's left is the installation of the VideoCore GPU firmware files and the configuration of the bootloader. Again, make sure that the partition IDs are right and that you name the kernel image name correctly.

]==> cd /mnt/boot
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/start4.elf
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/start4cd.elf
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/start4db.elf
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/start4x.elf
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/fixup4.dat
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/fixup4cd.dat
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/fixup4db.dat
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/fixup4x.dat
]==> cat config.txt 
enable_uart=1
dtparam=audio=on
kernel=vmlinuz-4.19.73-v8+
disable_overscan=1

[pi4]
dtoverlay=vc4-fkms-v3d
max_framebuffers=2
arm_64bit=1
enable_gic=1

]==> cat cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=8c7e33d1-02 rootfstype=ext4 elevator=deadline fsck.repair=yes quiet rootwait

That's it. You can now unmount the filesystems from the host, insert the microSD card into your RaspberryPi, and the Debian system should boot.

Post-installation setup

Generally, it's nice to have the network interface start up at the boot time so that you can download stuff from the Internet without a hassle. NetworkManager does that job well. The SSH daemon is also useful for other reasons.

]==> dhclient eth0
]==> apt-get install network-manager
]==> systemctl enable NetworkManager
]==> apt-get install openssh-server
]==> systemctl enable ssh

You may also want to suppress the annoying Perl warnings about missing locale files and set up your favorite time zone.

]==> apt-get install locales tzdata
]==> dpkg-reconfigure locales
]==> dpkg-reconfigure tzdata

Conclusion

My Bitcoin node runs fine with all its >250GB of blockchain data, but I have not checked if anything else works at all. In particular, I have not tested the display drivers nor any camera setup. However, random people on the Internet claim that the GPU drivers are now in the kernel, so things should be fine.

Have fun!