Content tagged howto

Intro

It starts to feel like this blog has become mostly about installing Debian on more or less esoteric pieces of hardware. One of the reasons for this is that I have lately been getting more and more into building embedded systems of various kinds and probably the largest annoyance in all this is the issue of the operating system that controls the hardware. If the device is powerful enough to run Linux, more often than not, it comes with some more or less useless vendor-specific and horrendously outdated IoT distribution. The only reasonable argument that I have heard for this being the state of affairs is that these embedded devices become obsolete pretty quickly, and the vendors are not keen on providing proper support. That's, in fact, an argument for making things more open, if you ask me, and for enabling the community to provide adequate support. I would have gladly put some time into it if getting the documentation was not close to impossible.

Even disregarding the fact that running this outdated software usually is a huge security risk, it ends up being an enormous drag when you want to build something that the vendor has not foreseen. The good people at Armbian do a pretty great job of bringing mainstream Linux to these small devices. However, I'd still instead run a vanilla distribution whenever possible. In most cases, making it work is just a matter of building the bootloader and tweaking the kernel.

So what's the deal this time around? Well, I need a file server and a media player for my home network. I found a suitable device with a bunch of SATA controllers attached over PCIe, a relatively powerful GPU, and a hardware video decoder. But then again, making vanilla Linux distribution run on it is somewhat of a challenge, so here's a howto.

NanoPi M4 with a SATA shield
NanoPi M4 with a SATA shield

Partitioning

You need to get a MicroSD card and partition it. You only need one partition because the preloader will look for the SPL, U-boot, and ARM Trusted Firmware directly in the block device. U-boot can boot from ext4 just fine, so there is no need for any FAT. That said, you may consider adding some swap if your use case calls for it. I created a GPT disklabel with the first partition starting at the 65536th block to have enough room for all the bootloader payload.

]==> fdisk /dev/sdc

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): g
Created a new GPT disklabel (GUID: 3124F0FC-9528-A44C-BD06-E3598E61CE99).
The old dos signature will be removed by a write command.

Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-124735454, default 2048): 65536
Last sector, +/-sectors or +/-size{K,M,G,T,P} (65536-124735454, default 124735454): 116346879

Created a new partition 1 of type 'Linux filesystem' and of size 55.5 GiB.

You can then format this partition with mkfs.ext4 /dev/sdc1.

Bootloader

Note 01.04.2021: I wrote a followup here.

You can, in theory, build the entire bootloader from open-source components. However, it seems that the memory timing setup is then somehow messed up. You will get the Linux kernel to panic due to memory errors while trying to access random chunks of RAM. I decided not to dig any deeper into it and use the mini loader provided by Rockchip, which comes with its own mess that I describe later. Anyways, you can start with vanilla U-boot:

]==> sudo apt-get install gcc-aarch64-linux-gnu bison flex u-boot-tools
]==> sudo apt-get install python-pyelftools device-tree-compiler libncurses-dev rsync
]==> git clone https://gitlab.denx.de/u-boot/u-boot.git
]==> cd u-boot
]==> git checkout v2019.10
]==> make  nanopi-m4-rk3399_defconfig
]==> make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu-

Once you have the U-boot image, you need to create the Rockchip bootloader payload. I run the commands below in a container because I am not that fond of running binaries of suspicious origin unchecked. Yeah, they produce other binaries that I then run uncontrolled on the target board. I see the irony, but I'd still rather keep my workstation safe.

]==> git clone https://github.com/rockchip-linux/rkbin.git
]==> cd rkbin
]==> ./tools/trust_merger RKTRUST/RK3399TRUST.ini
]==> ./tools/loaderimage --pack --uboot /path/to/u-boot/u-boot-dtb.bin uboot.img

Finally, you need to build the mini loader with the right DRAM timing settings and flash all that stuff to the disk:

]==> mkimage -n rk3399 -T rksd -d bin/rk33/rk3399_ddr_933MHz_v1.24.bin idbloader.img
]==> cat bin/rk33/rk3399_miniloader_v1.19.bin >> idbloader.img
]==> sudo dd if=idbloader.img of=/dev/sdc seek=64
]==> sudo dd if=trust.img of=/dev/sdc seek=24576
]==> sudo dd if=uboot.img of=/dev/sdc seek=16384
]==> sync

That should give you a working bootloader capable of starting Linux from an ext4 filesystem.

Base System

You need to create the Debian filesystem for aarch64 the usual way. I add mdadm here because I will want to run my disks in RAID4 mode.

]==> apt-get install qemu-user-static debootstrap
]==> mount /dev/sdc1 /mnt
]==> sudo qemu-debootstrap --include=u-boot-tools,mc,initramfs-tools,network-manager,openssh-server,mdadm --arch=arm64 testing /mnt/ http://mirror.init7.net/debian/

Modify the necessary configuration files, in particular the fstab, and set up the required user accounts:

]==> chroot /mnt /usr/bin/qemu-aarch64-static /bin/bash
]==> cat /etc/fstab
UUID=15c39b7d-fad1-4a76-93eb-050b94791312 /               ext4    errors=remount-ro 0       1
]==> cat /etc/hostname
your-hostname
]==> cat /etc/apt/sources.list
deb http://mirror.init7.net/debian/ testing main contrib non-free
deb-src http://mirror.init7.net/debian/ testing main contrib non-free
deb http://security.debian.org testing-security main contrib non-free
deb-src http://security.debian.org testing-security main contrib non-free
]==> useradd -m your-user
]==> passwd your-user
]==> passwd
]==> exit

You then need to build the kernel. I mentioned before that the Rockchip's bootloader and the ATF come with their mess. The main problem is that they reserve a chunk of memory at EL3. The kernel running at EL1 can then trip over this memory, get denied access, and Oops. So we need to declare it as inaccessible in the device tree. It is what the memory.patch does.

]==> wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.3.tar.xz
]==> tar xf linux-5.6.3.tar.xz
]==> cd linux-5.6.3
]==> patch -Np1 -i ../memory.patch
]==> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
]==> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
]==> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- KBUILD_IMAGE=arch/arm64/boot/Image -j12 bindeb-pkg
]==> sudo cp ../linux-headers-5.6.3_5.6.3-1_arm64.deb ../linux-libc-dev_5.6.3-1_arm64.deb ../linux-libc-dev_5.6.3-1_arm64.deb /mnt
]==> chroot /mnt /usr/bin/qemu-aarch64-static /bin/bash
]==> dpkg -i linux-*deb
]==> rm linux-*
]==> exit

As you probably have noticed, I manually include the Image instead of zImage into the package. It is because the aarch64 kernel does not currently provide a decompressor. Therefore, either the bootloader needs to ungzip it, or you need to have an already decompressed kernel image.

Bootloader setup and automation

The last thing before being able to boot into the new system is telling the bootloader how to do it. U-boot will look for the boot.scr file in the root of your filesystem. For the first boot, it's best to set things up statically. Here's what I do:

]==> cat /boot.cmd
setenv bootargs 'root=/dev/mmcblk1p1 rootfstype=ext4 rootwait console=ttyS2,1500000 console=tty1 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u memtest=4 earlycon=uart8250,mmio32,0xff1a0000'
load mmc 1:1 ${kernel_addr_r} /boot/vmlinuz-5.6.3
load mmc 1:1 ${fdt_addr_r} /usr/lib/linux-image-5.6.3/rockchip/rk3399-nanopi-m4.dtb
load mmc 1:1 ${ramdisk_addr_r} /boot/uinitrd.img-5.6.3
booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
]==> mkimage -A arm64 -O linux -T ramdisk -C gzip -d /boot/initrd.img-5.6.3 /boot/uinitrd.img-5.6.3
]==> mkimage -A arm -O linux -T script -C none -n "Initial u-boot script" -d /boot.cmd /boot.scr

There are three tricks to the above. Firstly, we use the booti command instead of the usual bootz. It is because of the decompressor issue mentioned above. Secondly, we need to convert the initial ramdisk to the format digestible by U-boot. Thirdly, we need to convert the text boot script to the U-boot's binary format.

That's it. You can now boot your system.

It's a pain to run all this by hand every time you want to update your kernel, so it's a good idea to automate the process. To that end, I create a template file in the root of a filesystem:

]==> cat /boot.cmd.in
setenv bootargs 'root=/dev/mmcblk1p1 rootfstype=ext4 rootwait console=ttyS2,1500000 console=tty1 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u'
load mmc 1:1 ${kernel_addr_r} /boot/vmlinuz-__VERSION__
load mmc 1:1 ${fdt_addr_r} /usr/lib/linux-image-__VERSION__/rockchip/rk3399-nanopi-m4.dtb
load mmc 1:1 ${ramdisk_addr_r} /boot/uinitrd.img-__VERSION__
booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}

And then run a hook that converts the ramdisk, creates boot.cmd, and converts it to the U-boot's binary format.

]==> cat /etc/kernel/postinst.d/uboot
#!/bin/sh -e
version="$1"
/usr/bin/mkimage -A arm64 -O linux -T ramdisk -C gzip -d /boot/initrd.img-${version} /boot/uinitrd.img-${version}
/bin/cat /boot.cmd.in  | /usr/bin/sed -e "s/__VERSION__/${version}/g" > /boot.cmd
/usr/bin/mkimage -A arm -O linux -T script -C none -n "Initial u-boot script" -d /boot.cmd /boot.scr

Post boot setup and media

After you've booted up the system, you can install some desktop environment and add your user account to all the useful groups. Reconfiguring locales and time zone data gets rid of annoying internationalization warnings and lets you handle the time correctly.

]==> apt-get update
]==> apt-get dist-upgrade
]==> apt-get install xfce4 alsa-utils mesa-utils pulseaudio mdadm locales tzdata mesa-utils-extra
]==> dpkg-reconfigure locales
]==> dpkg-reconfigure tzdata
]==> usermod -a -G pulse,pulse-access,netdev,plugdev,video,audio,sudo,dialout,users,render your-user

If you have some fancy surround sound setup as I do, you can set PulseAudio up to use the Alsa sink by adding load-module module-alsa-sink to /etc/pulse/default.pa. You can also configure the number of audio channels in /etc/pulse/daemon.conf. The setting is called default-sample-channels. As for ALSA itself, this is what I put in /etc/asound.conf:

]==> cat /etc/asound.conf
pcm.dmixer  {
        type dmix
        ipc_key 1024
        slave {
                pcm "hw"
                channels 6
        }
}

pcm.!default "plug:dmixer

The last step is to make the hardware media decoder work. Rockchip has its proprietary acceleration hardware and the software driving it. Here's how to get it:

]==> sudo apt-get install build-essential dh-exec git cmake
]==> git clone https://github.com/rockchip-linux/mpp.git mpp-1.4.0
]==> git checkout 50a96555

It happens to have a rules file to create a Debian package, but the file is messed up, so you will have to apply the following patch:

diff --git a/debian/rules b/debian/rules
index 876d6e6e..26686f39 100755
--- a/debian/rules
+++ b/debian/rules
@@ -24,7 +24,5 @@ include /usr/share/dpkg/default.mk
 # This is example for Cmake (See http://bugs.debian.org/641051 )
 override_dh_auto_configure:
        dh_auto_configure -- \
-       -DCMAKE_TOOLCHAIN_FILE=/etc/dpkg-cross/cmake/CMakeCross.txt \
        -DCMAKE_BUILD_TYPE=Release \
-       -DHAVE_DRM=ON \
-       -DARM_MIX_32_64=ON
+       -DHAVE_DRM=ON

Then build and install it the usual way:

]==> tar czf mpp_1.4.0.orig.tar.gz mpp-1.4.0/
]==> cd mpp-1.4.0
]==> DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage -rfakeroot --no-sign
]==> dpkg -i ../librockchip-mpp-dev_1.4.0-1_arm64.deb ../librockchip-mpp1_1.4.0-1_arm64.deb ../librockchip-vpu0_1.4.0-1_arm64.deb 

That's not the end. You still need to make ffmpeg use the hardware acceleration.

]==> apt-get source ffmpeg
]==> apt-get build-dep ffmpeg

Apply the following patch to enable Rockchip's MPP:

diff -Naur ffmpeg-4.2.2.orig/debian/rules ffmpeg-4.2.2/debian/rules
--- ffmpeg-4.2.2.orig/debian/rules      2020-01-25 17:22:32.000000000 +0100
+++ ffmpeg-4.2.2/debian/rules   2020-04-11 23:15:44.465637513 +0200
@@ -104,7 +104,10 @@
        --enable-openal \
        --enable-opencl \
        --enable-opengl \
-       --enable-sdl2
+       --enable-sdl2 \
+       --enable-rkmpp \
+       --enable-version3
+
 
 # The standard configuration only uses the shared CONFIG.
 CONFIG_standard = --enable-shared

Then build and install it the usual way:

]==> cd ffmpeg-4.2.2
]==> dpkg-buildpackage -rfakeroot --no-sign
]==> sudo dpkg -i ../libpostproc-dev_*_arm64.deb ../libavformat-dev_*_arm64.deb ../libavcodec-dev_*_arm64.deb ../libavformat58_*_arm64.deb ../libavutil-dev_*_arm64.deb ../libavutil56_*_arm64.deb ../libswresample-dev_*_arm64.deb ../libswresample3_*_arm64.deb ../libavfilter-dev_*_arm64.deb  ../libavfilter7_*_arm64.deb ../libswscale-dev_*_arm64.deb ../libswscale5_*_arm64.deb ../ffmpeg_*_arm64.deb ../libavresample4_*_arm64.deb ../libavresample-dev_*_arm64.deb ../libavcodec58_*_arm64.deb ../libpostproc55_*_arm64.deb ../libavdevice58_*_arm64.deb  ../libavdevice-dev_*_arm64.deb
]==> sudo apt-mark hold libpostproc-dev libavformat-dev libavcodec-dev libavformat58 libavutil-dev libavutil56 libswresample-dev libswresample3 libavfilter-dev libavfilter7 libswscale-dev libswscale5 ffmpeg libavresample4 libavresample-dev libavcodec58 libpostproc55 libavdevice58 libavdevice-dev

The last command tells apt not to update these packages as the default system version does not support hardware acceleration on this platform.

Conclusion

The board works quite well, and I am reasonably happy with it. I am currently waiting for a 3D printer to come to put it in a case together with the disks. I also intend to set up NFS4 with Kerberos, so there will be follow up articles.

Intro

It's been quite a long while since the 64-bit CPUs took over the world. It seems that most of the ecosystem moved on, and people stopped paying close attention to the 32-bit compatibility issues. I know I did. It's all fine - compatibility can be quite a pain for very negligible gain. However, some notable software projects are still stuck in the 32-bit-only rut. Furthermore, it seems to be the case even when their primary target platforms have been 64-bit capable for quite a long time now.

One of the projects in the first category is Bitcoin. The Bitcoin developers made their stable release without noticing that the crypto unit tests fail on 32-bit platforms. On the other side of the spectrum, there is the RaspberryPi software stack. It could not run well in 64-bit mode up until a couple of days ago. The good people at Balena Linux distro did the necessary porting work and submitted it to the upstream RPi kernel. This work is so fresh and has been done mainly by a third-party, even though the boards from RaspberryPi 2B 1.2 onwards have an ARMv8 CPU. That's early 2016. I guess that this situation is the result of not pushing the changes to the upstream kernel that seems to be very common in the embedded world and the lack of documentation.

The reason I am writing about this is that I have just hit both of these issues when trying to run a full Bitcoin node on my fresh and new RaspberryPi 4B. I ended up having to install stock Debian on it. It was not exactly hard, but there seemed to have been no instruction on the Internet, so I write down what I did here.

Prerequisites

Except for a RaspberryPi board, you will need a microSD card and a Linux system that can write to this card. Following what Raspbian does, I partitioned my card in the following way:

]==> fdisk /dev/sdb

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/sdb: 7.41 GiB, 7948206080 bytes, 15523840 sectors
Disk model: USB SD Reader   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x8c7e33d1

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sdb1         8192   532480   524289  256M  c W95 FAT32 (LBA)
/dev/sdb2       540672 15523839 14983168  7.1G 83 Linux

The first partition is the boot partition holding the VideoCore GPU firmware necessary to boot a kernel as well as the Linux kernel image. I will discuss how to get both later. The second partition is for the root filesystem. I mounted them in my host system as described below and the rest of this document follows this convention.

]==> mount /dev/sdb2 /mnt
]==> mkdir /mnt/boot
]==> mount /dev/sdb1 /mnt/boot/

Furthermore, you will need to have the following packages installed in the host system:

]==> apt-get install debootstrap qemu-user-static
]==> apt-get install gcc-aarch64-linux-gnu bison flex python-pyelftools device-tree-compiler

The packages in the first group are necessary to bootstrap the root filesystem, while the ones in the second group are needed to cross-compile the kernel image.

System installation

You'll need the arm64 flavor of Debian. I use the testing distribution for pretty much every system I have, but you probably can successfully run any other. I also like using Midnight Commander, so I include that package in the target installation. The include specification is a comma-separated list, and you can declare any package you want there. My network operator is Init7, so I use their mirror. You should select one that is close to you. Finally, the target for the installation is /mnt.

]==> qemu-debootstrap --include=mc --arch=arm64 testing /mnt/ http://mirror.init7.net/debian/

After the bootstrapping process completes, you need to build the kernel image that can run on the board. The official website describes the process quite well, but I like building proper Debian packages instead. The commit with hash db690083a4 worked fine for me.

]==> cd /tmp
]==> git clone https://github.com/raspberrypi/linux.git
]==> cd linux
]==> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig
]==> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- KBUILD_IMAGE=arch/arm64/boot/Image -j12 bindeb-pkg
]==> make ARCH=arm64 CROSS_COMPILE=/usr/bin/aarch64-linux-gnu- dtbs
]==> cp /tmp/linux/arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb /mnt/boot/
]==> mkdir /mnt/boot/overlays
]==> cp /tmp/linux/arch/arm64/boot/dts/overlays/vc4-fkms-v3d.dtbo /mnt/boot/overlays/
]==> cp /tmp/linux*deb /mnt

The first make command configures the kernel for this particular board. The second one builds the Debian packages with the kernel, the kernel headers, and the kernel userspace headers. The third one compiles the Device Tree files. The last four commands install the necessary Device Tree files in the boot partition and copy the kernel packages to the root filesystem.

You can now chroot into the root filesystem, install the kernel, create a user account, and change passwords. We also need to prevent the linux-libc-dev package from being updated using apt-mark.

]==> chroot /mnt /usr/bin/qemu-aarch64-static /bin/bash
]==> dpkg -i linux-*deb
]==> rm linux*deb
]==> apt-mark hold linux-libc-dev
]==> useradd -m youruser
]==> passwd youruser
]==> passwd root

You then need to edit the files that play a role in the boot process. The relevant listings are presented below. Make sure that you got the partition setup right. You can get the PARTUUIDs by listing: ls -l /dev/disk/by-partuuid.

]==> cat /etc/hostname 
cryptopi
]==> cat /etc/fstab   
PARTUUID=8c7e33d1-01  /boot           vfat    defaults          0       2
PARTUUID=8c7e33d1-02  /               ext4    defaults,noatime  0       1
]==> cat /etc/apt/sources.list
deb http://mirror.init7.net/debian/ testing main contrib non-free
deb-src http://mirror.init7.net/debian/ testing main contrib non-free

deb http://security.debian.org testing-security main contrib non-free
deb-src http://security.debian.org testing-security main contrib non-free

Finally, all that's left is the installation of the VideoCore GPU firmware files and the configuration of the bootloader. Again, make sure that the partition IDs are right and that you name the kernel image name correctly.

]==> cd /mnt/boot
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/start4.elf
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/start4cd.elf
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/start4db.elf
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/start4x.elf
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/fixup4.dat
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/fixup4cd.dat
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/fixup4db.dat
]==> wget https://github.com/raspberrypi/firmware/raw/master/boot/fixup4x.dat
]==> cat config.txt 
enable_uart=1
dtparam=audio=on
kernel=vmlinuz-4.19.73-v8+
disable_overscan=1

[pi4]
dtoverlay=vc4-fkms-v3d
max_framebuffers=2
arm_64bit=1
enable_gic=1

]==> cat cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=8c7e33d1-02 rootfstype=ext4 elevator=deadline fsck.repair=yes quiet rootwait

That's it. You can now unmount the filesystems from the host, insert the microSD card into your RaspberryPi, and the Debian system should boot.

Post-installation setup

Generally, it's nice to have the network interface start up at the boot time so that you can download stuff from the Internet without a hassle. NetworkManager does that job well. The SSH daemon is also useful for other reasons.

]==> dhclient eth0
]==> apt-get install network-manager
]==> systemctl enable NetworkManager
]==> apt-get install openssh-server
]==> systemctl enable ssh

You may also want to suppress the annoying Perl warnings about missing locale files and set up your favorite time zone.

]==> apt-get install locales tzdata
]==> dpkg-reconfigure locales
]==> dpkg-reconfigure tzdata

Conclusion

My Bitcoin node runs fine with all its >250GB of blockchain data, but I have not checked if anything else works at all. In particular, I have not tested the display drivers nor any camera setup. However, random people on the Internet claim that the GPU drivers are now in the kernel, so things should be fine.

Have fun!