Intro

It starts to feel like this blog has become mostly about installing Debian on more or less esoteric pieces of hardware. One of the reasons for this is that I have lately been getting more and more into building embedded systems of various kinds and probably the largest annoyance in all this is the issue of the operating system that controls the hardware. If the device is powerful enough to run Linux, more often than not, it comes with some more or less useless vendor-specific and horrendously outdated IoT distribution. The only reasonable argument that I have heard for this being the state of affairs is that these embedded devices become obsolete pretty quickly, and the vendors are not keen on providing proper support. That's, in fact, an argument for making things more open, if you ask me, and for enabling the community to provide adequate support. I would have gladly put some time into it if getting the documentation was not close to impossible.

Even disregarding the fact that running this outdated software usually is a huge security risk, it ends up being an enormous drag when you want to build something that the vendor has not foreseen. The good people at Armbian do a pretty great job of bringing mainstream Linux to these small devices. However, I'd still instead run a vanilla distribution whenever possible. In most cases, making it work is just a matter of building the bootloader and tweaking the kernel.

So what's the deal this time around? Well, I need a file server and a media player for my home network. I found a suitable device with a bunch of SATA controllers attached over PCIe, a relatively powerful GPU, and a hardware video decoder. But then again, making vanilla Linux distribution run on it is somewhat of a challenge, so here's a howto.

NanoPi M4 with a SATA shield

Partitioning

You need to get a MicroSD card and partition it. You only need one partition because the preloader will look for the SPL, U-boot, and ARM Trusted Firmware directly in the block device. U-boot can boot from ext4 just fine, so there is no need for any FAT. That said, you may consider adding some swap if your use case calls for it. I created a GPT disklabel with the first partition starting at the 65536th block to have enough room for all the bootloader payload.

]==> fdisk /dev/sdc

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): g
Created a new GPT disklabel (GUID: 3124F0FC-9528-A44C-BD06-E3598E61CE99).
The old dos signature will be removed by a write command.

Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-124735454, default 2048): 65536
Last sector, +/-sectors or +/-size{K,M,G,T,P} (65536-124735454, default 124735454): 116346879

Created a new partition 1 of type 'Linux filesystem' and of size 55.5 GiB.

You can then format this partition with mkfs.ext4 /dev/sdc1.

Bootloader

Note 01.04.2021: I wrote a followup here.

You can, in theory, build the entire bootloader from open-source components. However, it seems that the memory timing setup is then somehow messed up. You will get the Linux kernel to panic due to memory errors while trying to access random chunks of RAM. I decided not to dig any deeper into it and use the mini loader provided by Rockchip, which comes with its own mess that I describe later. Anyways, you can start with vanilla U-boot:

]==> sudo apt-get install gcc-aarch64-linux-gnu bison flex u-boot-tools
]==> sudo apt-get install python-pyelftools device-tree-compiler libncurses-dev rsync
]==> git clone https://gitlab.denx.de/u-boot/u-boot.git
]==> cd u-boot
]==> git checkout v2019.10
]==> make  nanopi-m4-rk3399_defconfig
]==> make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu-

Once you have the U-boot image, you need to create the Rockchip bootloader payload. I run the commands below in a container because I am not that fond of running binaries of suspicious origin unchecked. Yeah, they produce other binaries that I then run uncontrolled on the target board. I see the irony, but I'd still rather keep my workstation safe.

]==> git clone https://github.com/rockchip-linux/rkbin.git
]==> cd rkbin
]==> ./tools/trust_merger RKTRUST/RK3399TRUST.ini
]==> ./tools/loaderimage --pack --uboot /path/to/u-boot/u-boot-dtb.bin uboot.img

Finally, you need to build the mini loader with the right DRAM timing settings and flash all that stuff to the disk:

]==> mkimage -n rk3399 -T rksd -d bin/rk33/rk3399_ddr_933MHz_v1.24.bin idbloader.img
]==> cat bin/rk33/rk3399_miniloader_v1.19.bin >> idbloader.img
]==> sudo dd if=idbloader.img of=/dev/sdc seek=64
]==> sudo dd if=trust.img of=/dev/sdc seek=24576
]==> sudo dd if=uboot.img of=/dev/sdc seek=16384
]==> sync

That should give you a working bootloader capable of starting Linux from an ext4 filesystem.

Base System

You need to create the Debian filesystem for aarch64 the usual way. I add mdadm here because I will want to run my disks in RAID4 mode.

]==> apt-get install qemu-user-static debootstrap
]==> mount /dev/sdc1 /mnt
]==> sudo qemu-debootstrap --include=u-boot-tools,mc,initramfs-tools,network-manager,openssh-server,mdadm --arch=arm64 testing /mnt/ http://mirror.init7.net/debian/

Modify the necessary configuration files, in particular the fstab, and set up the required user accounts:

]==> chroot /mnt /usr/bin/qemu-aarch64-static /bin/bash
]==> cat /etc/fstab
UUID=15c39b7d-fad1-4a76-93eb-050b94791312 /               ext4    errors=remount-ro 0       1
]==> cat /etc/hostname
your-hostname
]==> cat /etc/apt/sources.list
deb http://mirror.init7.net/debian/ testing main contrib non-free
deb-src http://mirror.init7.net/debian/ testing main contrib non-free
deb http://security.debian.org testing-security main contrib non-free
deb-src http://security.debian.org testing-security main contrib non-free
]==> useradd -m your-user
]==> passwd your-user
]==> passwd
]==> exit

You then need to build the kernel. I mentioned before that the Rockchip's bootloader and the ATF come with their mess. The main problem is that they reserve a chunk of memory at EL3. The kernel running at EL1 can then trip over this memory, get denied access, and Oops. So we need to declare it as inaccessible in the device tree. It is what the memory.patch does.

]==> wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.3.tar.xz
]==> tar xf linux-5.6.3.tar.xz
]==> cd linux-5.6.3
]==> patch -Np1 -i ../memory.patch
]==> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
]==> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
]==> make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- KBUILD_IMAGE=arch/arm64/boot/Image -j12 bindeb-pkg
]==> sudo cp ../linux-headers-5.6.3_5.6.3-1_arm64.deb ../linux-libc-dev_5.6.3-1_arm64.deb ../linux-libc-dev_5.6.3-1_arm64.deb /mnt
]==> chroot /mnt /usr/bin/qemu-aarch64-static /bin/bash
]==> dpkg -i linux-*deb
]==> rm linux-*
]==> exit

As you probably have noticed, I manually include the Image instead of zImage into the package. It is because the aarch64 kernel does not currently provide a decompressor. Therefore, either the bootloader needs to ungzip it, or you need to have an already decompressed kernel image.

Bootloader setup and automation

The last thing before being able to boot into the new system is telling the bootloader how to do it. U-boot will look for the boot.scr file in the root of your filesystem. For the first boot, it's best to set things up statically. Here's what I do:

]==> cat /boot.cmd
setenv bootargs 'root=/dev/mmcblk1p1 rootfstype=ext4 rootwait console=ttyS2,1500000 console=tty1 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u memtest=4 earlycon=uart8250,mmio32,0xff1a0000'
load mmc 1:1 ${kernel_addr_r} /boot/vmlinuz-5.6.3
load mmc 1:1 ${fdt_addr_r} /usr/lib/linux-image-5.6.3/rockchip/rk3399-nanopi-m4.dtb
load mmc 1:1 ${ramdisk_addr_r} /boot/uinitrd.img-5.6.3
booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
]==> mkimage -A arm64 -O linux -T ramdisk -C gzip -d /boot/initrd.img-5.6.3 /boot/uinitrd.img-5.6.3
]==> mkimage -A arm -O linux -T script -C none -n "Initial u-boot script" -d /boot.cmd /boot.scr

There are three tricks to the above. Firstly, we use the booti command instead of the usual bootz. It is because of the decompressor issue mentioned above. Secondly, we need to convert the initial ramdisk to the format digestible by U-boot. Thirdly, we need to convert the text boot script to the U-boot's binary format.

That's it. You can now boot your system.

It's a pain to run all this by hand every time you want to update your kernel, so it's a good idea to automate the process. To that end, I create a template file in the root of a filesystem:

]==> cat /boot.cmd.in
setenv bootargs 'root=/dev/mmcblk1p1 rootfstype=ext4 rootwait console=ttyS2,1500000 console=tty1 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u'
load mmc 1:1 ${kernel_addr_r} /boot/vmlinuz-__VERSION__
load mmc 1:1 ${fdt_addr_r} /usr/lib/linux-image-__VERSION__/rockchip/rk3399-nanopi-m4.dtb
load mmc 1:1 ${ramdisk_addr_r} /boot/uinitrd.img-__VERSION__
booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}

And then run a hook that converts the ramdisk, creates boot.cmd, and converts it to the U-boot's binary format.

]==> cat /etc/kernel/postinst.d/uboot
#!/bin/sh -e
version="$1"
/usr/bin/mkimage -A arm64 -O linux -T ramdisk -C gzip -d /boot/initrd.img-${version} /boot/uinitrd.img-${version}
/bin/cat /boot.cmd.in  | /usr/bin/sed -e "s/__VERSION__/${version}/g" > /boot.cmd
/usr/bin/mkimage -A arm -O linux -T script -C none -n "Initial u-boot script" -d /boot.cmd /boot.scr

Post boot setup and media

After you've booted up the system, you can install some desktop environment and add your user account to all the useful groups. Reconfiguring locales and time zone data gets rid of annoying internationalization warnings and lets you handle the time correctly.

]==> apt-get update
]==> apt-get dist-upgrade
]==> apt-get install xfce4 alsa-utils mesa-utils pulseaudio mdadm locales tzdata mesa-utils-extra
]==> dpkg-reconfigure locales
]==> dpkg-reconfigure tzdata
]==> usermod -a -G pulse,pulse-access,netdev,plugdev,video,audio,sudo,dialout,users,render your-user

If you have some fancy surround sound setup as I do, you can set PulseAudio up to use the Alsa sink by adding load-module module-alsa-sink to /etc/pulse/default.pa. You can also configure the number of audio channels in /etc/pulse/daemon.conf. The setting is called default-sample-channels. As for ALSA itself, this is what I put in /etc/asound.conf:

]==> cat /etc/asound.conf
pcm.dmixer  {
        type dmix
        ipc_key 1024
        slave {
                pcm "hw"
                channels 6
        }
}

pcm.!default "plug:dmixer

The last step is to make the hardware media decoder work. Rockchip has its proprietary acceleration hardware and the software driving it. Here's how to get it:

]==> sudo apt-get install build-essential dh-exec git cmake
]==> git clone https://github.com/rockchip-linux/mpp.git mpp-1.4.0
]==> git checkout 50a96555

It happens to have a rules file to create a Debian package, but the file is messed up, so you will have to apply the following patch:

diff --git a/debian/rules b/debian/rules
index 876d6e6e..26686f39 100755
--- a/debian/rules
+++ b/debian/rules
@@ -24,7 +24,5 @@ include /usr/share/dpkg/default.mk
 # This is example for Cmake (See http://bugs.debian.org/641051 )
 override_dh_auto_configure:
        dh_auto_configure -- \
-       -DCMAKE_TOOLCHAIN_FILE=/etc/dpkg-cross/cmake/CMakeCross.txt \
        -DCMAKE_BUILD_TYPE=Release \
-       -DHAVE_DRM=ON \
-       -DARM_MIX_32_64=ON
+       -DHAVE_DRM=ON

Then build and install it the usual way:

]==> tar czf mpp_1.4.0.orig.tar.gz mpp-1.4.0/
]==> cd mpp-1.4.0
]==> DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage -rfakeroot --no-sign
]==> dpkg -i ../librockchip-mpp-dev_1.4.0-1_arm64.deb ../librockchip-mpp1_1.4.0-1_arm64.deb ../librockchip-vpu0_1.4.0-1_arm64.deb 

That's not the end. You still need to make ffmpeg use the hardware acceleration.

]==> apt-get source ffmpeg
]==> apt-get build-dep ffmpeg

Apply the following patch to enable Rockchip's MPP:

diff -Naur ffmpeg-4.2.2.orig/debian/rules ffmpeg-4.2.2/debian/rules
--- ffmpeg-4.2.2.orig/debian/rules      2020-01-25 17:22:32.000000000 +0100
+++ ffmpeg-4.2.2/debian/rules   2020-04-11 23:15:44.465637513 +0200
@@ -104,7 +104,10 @@
        --enable-openal \
        --enable-opencl \
        --enable-opengl \
-       --enable-sdl2
+       --enable-sdl2 \
+       --enable-rkmpp \
+       --enable-version3
+
 
 # The standard configuration only uses the shared CONFIG.
 CONFIG_standard = --enable-shared

Then build and install it the usual way:

]==> cd ffmpeg-4.2.2
]==> dpkg-buildpackage -rfakeroot --no-sign
]==> sudo dpkg -i ../libpostproc-dev_*_arm64.deb ../libavformat-dev_*_arm64.deb ../libavcodec-dev_*_arm64.deb ../libavformat58_*_arm64.deb ../libavutil-dev_*_arm64.deb ../libavutil56_*_arm64.deb ../libswresample-dev_*_arm64.deb ../libswresample3_*_arm64.deb ../libavfilter-dev_*_arm64.deb  ../libavfilter7_*_arm64.deb ../libswscale-dev_*_arm64.deb ../libswscale5_*_arm64.deb ../ffmpeg_*_arm64.deb ../libavresample4_*_arm64.deb ../libavresample-dev_*_arm64.deb ../libavcodec58_*_arm64.deb ../libpostproc55_*_arm64.deb ../libavdevice58_*_arm64.deb  ../libavdevice-dev_*_arm64.deb
]==> sudo apt-mark hold libpostproc-dev libavformat-dev libavcodec-dev libavformat58 libavutil-dev libavutil56 libswresample-dev libswresample3 libavfilter-dev libavfilter7 libswscale-dev libswscale5 ffmpeg libavresample4 libavresample-dev libavcodec58 libpostproc55 libavdevice58 libavdevice-dev

The last command tells apt not to update these packages as the default system version does not support hardware acceleration on this platform.

Conclusion

The board works quite well, and I am reasonably happy with it. I am currently waiting for a 3D printer to come to put it in a case together with the disks. I also intend to set up NFS4 with Kerberos, so there will be follow up articles.