At $DAY_JOB I do a lot of work on embedded linux systems. In
particular, a large part of my job is defining and building the base
linux platform that we build on top of. We generally use Ubuntu
(because that's what $SOC_VENDOR supports), but it's not a good fit
for embedded development. I've always thought that Guix would solve a
lot of problems that we encounter; let's give it a try!
First, to clarify what I'm after and why I think Guix is a good fit:
Systems are built and deployed instead of managed. Installing
packages to the rootfs shouldn't require running scripts
(e.g. post-install scripts) or otherwise chroot'ing into the
rootfs.
Building the system should be hermetic and deterministic. The build should depend on a small fixed set of tools on the host machine and should be reproducible on any host distro.
The system should be customizable. Installing packages, enabling services, modifying the initramfs, etc. are all important to customizing the embedded distro for the hardware and product that is being built.
The base system should be small. Embedded systems generally have limited flash space, so fitting the rootfs in as little space as possible is quite important.
Ubuntu fails all of these points, but is often the vendor's first
choice due to "popularity". Due to this reason and the fact that most
non-embedded linux developers are familiar with it, this is what we
generally use at $DAY_JOB. Working with Ubuntu on an embedded system
often feels like trying to fit a square peg in a round hole; we always
make it work, but it requires more force than it should.
Yocto/openembedded/other bitbake-based distros are generally pretty bad at (2). While the core openembedded system might be hermetic and deterministic (note that I've never tested this), vendor extensions and recipes almost never are. Vendors usually require the system is built from a given distro---usually a specific Ubuntu release---with a set of packages pre-installed. (One particularly egregious vendor required building different parts of the system on two different releases of Ubuntu!)
Buildroot ticks all these boxes, but usually doesn't have first-class
support from vendors. Also, while it tries to be hermetic, this isn't
enforce via containerization in any way, so it can break subtly. (One
time I discovered that not having gawk on my $PATH caused SSH to
silently be disabled during the build, resulting in a broken rootfs.)
Guix solves 1-3 pretty well and is (IMO) much easier and more pleasant to work with than any other solution I've tried. What about rootfs size? How small can we make a base guix system? Let's find out!
For the purposes of comparison, let's start with an operating-system
with default settings:
(operating-system
(host-name "default-guix")
(bootloader (bootloader-configuration
(bootloader grub-efi-bootloader)
(targets '("/boot/efi"))))
(file-systems (append
(list (file-system
(device "/dev/vda1")
(mount-point "/")
(type "ext4")))
%base-file-systems)))(I'm using grub-efi-bootloader just as a basis for comparison; grub
is not typically used in embedded systems, but I doubt it will affect
the rootfs size too much)
Building this with guix system image --image-type=efi-raw default.scm results in a disk image of ~1.6GB; not great, but we can
trim this down quite a bit without much effort.
Full config is here.
First, defining requirements that the "minified system" should meet (essentially all embedded linux systems I've worked on requires these):
The system should boot correctly.
I can log on over a serial console and have access to basic utilities (coreutils, busybox, etc.).
syslog (in some form) and udev should be running.
We can then strip things we don't need from %base-packages and
%base-services to trim down the rootfs bu quite a bit:
(operating-system
(host-name "milli-guix")
(bootloader (bootloader-configuration
(bootloader grub-efi-bootloader)
(targets '("/boot/efi"))))
(file-systems (append
(list (file-system
(device "/dev/vda1")
(mount-point "/")
(type "ext4")))
%base-file-systems))
(setuid-programs '())
(services
(list (service login-service-type)
(service virtual-terminal-service-type)
(service syslog-service-type)
(service mingetty-service-type (mingetty-configuration
(tty "tty1")))
(service static-networking-service-type
(list %loopback-static-networking))
(service udev-service-type)
(service sysctl-service-type)
(service special-files-service-type
`(("/bin/sh" ,(file-append bash "/bin/sh"))
("/usr/bin/env" ,(file-append coreutils "/bin/env"))))))
(packages (list coreutils)))The loopback-static-networking, sysctl, and special-files
services fall under the "system should boot correctly" category,
though I'm sure you could leave them out in some custom systems. I've
also removed all setuid programs, since we're running a single-user
system anywyas, setuid is fairly pointless.
Now we're down to just 725MB (and guix system vm milli.scm confirms
we can still boot and login as root). That's making progress, but
still a long way off from where it should be for an embedded system.
Full config is here.
Guix ships with a bunch of locales enabled by default, which can take up a lot of space. Let's try eliminating those:
(operating-system
(host-name "micro-guix")
(bootloader (bootloader-configuration
(bootloader grub-efi-bootloader)
(targets '("/boot/efi"))))
(file-systems (append
(list (file-system
(device "/dev/vda1")
(mount-point "/")
(type "ext4")))
%base-file-systems))
(setuid-programs '())
(services
(list (service login-service-type)
(service virtual-terminal-service-type)
(service syslog-service-type)
(service mingetty-service-type (mingetty-configuration
(tty "tty1")))
(service static-networking-service-type
(list %loopback-static-networking))
(service udev-service-type)
(service sysctl-service-type)
(service special-files-service-type
`(("/bin/sh" ,(file-append bash "/bin/sh"))
("/usr/bin/env" ,(file-append coreutils "/bin/env"))))))
(packages (list coreutils))
(locale "en_US.utf8")
(locale-definitions (list (locale-definition
(name "en_US.utf8")
(source "en_US")
(charset "UTF-8"))))
(locale-libcs (list glibc)))This shaves off another few MB, bringing the disk image down to 693MB total.
Full config is here.
This is potentially one of the biggest areas of improvement, but is difficult to tackle in this worklog format. The kernel is going to be highly customized for a given system, reducing its footprint far more than I will here.
Most of the on-disk footprint for the kernel comes from the kernel modules, so the easiest way to shrink disk usage is to prune unused modules. On guix, there are thousands installed by default, but only 10s that are actually used to boot in a VM. The process for pruning these is roughly:
From the booted image, ensure you have the actual config in use
from /proc/config.gz and the list of kernel modules that are
loaded (from lsmod).
In the config, disable all modules, running something like:
sed -i -E 's/(CONFIG_[^=]+)=m/# \1 is not set/' .config
Re-activate the kernel modules that are actually required using
make menuconfig.
Ensure the system boots and re-enable additional options as necessary.
The defconfig I'm using is here. We can then use this config in the operating system definition:
(define tinylinux
(customize-linux
#:name "tinylinux"
#:defconfig (local-file "tinylinux_defconfig")))
(operating-system
(host-name "nano-guix")
(kernel tinylinux)
...)The resulting image is down to 558MB---another 135MB of savings.
Full config is here.
Another common technique for shrinking the rootfs is to use
busybox instead of the normal coreutils,
util-linux, bash, etc. A couple of these replacements are
immediately obvious: the /bin/sh and /usr/bin/env symlinks can
come from busybox instead of bash and coreutils (respectively).
Another simple (but maybe less obvious) change is to use busybox for
the root user's login shell. Guix makes this easy through the
inherit keyword:
(operating-system
...
(users
(list (user-account
(inherit (@@ (gnu system) %root-account))
(shell (file-append busybox "/bin/sh")))))
...)Note that if we add a root user to the user list, guix will
automatically detect this and not add root as a default account.
Busybox also has a syslogd implementation we can use. Unfortunately,
it doesn't accept the same arguments or config file as the default
syslogd, so we can't reuse guix's syslog-service-type. Instead,
let's define our own based on the existing syslog service definition:
(define %default-busybox-syslog.conf
(plain-file "syslog.conf" "\
# Log all error messages, authentication messages of
# level notice or higher and anything of level err or
# higher to the console.
# Don't log private authentication messages!
*.alert;auth.notice;authpriv.none /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none /var/log/messages
# The authpriv file has restricted access.
# Also include unprivileged auth logs of info or higher level
# to conveniently gather the authentication data at the same place.
authpriv.*;auth.info /var/log/secure
"))
(define-record-type* <busybox-syslog-configuration>
busybox-syslog-configuration make-busybox-syslog-configuration
busybox-syslog-configuration?
(syslogd busybox-syslog-configuration-syslogd
(default (file-append busybox "/sbin/syslogd")))
(config-file busybox-syslog-configuration-config-file
(default %default-busybox-syslog.conf)))
(define (busybox-syslog-shepherd-service config)
(define config-file
(busybox-syslog-configuration-config-file config))
(shepherd-service
(documentation "Run the busybox syslog daemon (syslogd).")
(provision '(syslogd))
(requirement '(user-processes))
(actions
(list (shepherd-configuration-action config-file)))
(start #~(make-forkexec-constructor
(list #$(busybox-syslog-configuration-syslogd config)
"-n" "-f" #$config-file)
#:file-creation-mask #o137))
(stop #~(make-kill-destructor))))
(define busybox-syslog-service-type
(service-type
(name 'syslog)
(default-value (busybox-syslog-configuration))
(extensions (list (service-extension
shepherd-root-service-type
(compose list busybox-syslog-shepherd-service))))
(description "Run busybox's @command{syslogd} as the system logger.")))Similarly, busybox has sysctl as well (again accepting slightly
different arguments from guix's sysctl-service-type). This time,
let's inherit from the default sysctl service, just changing the
arguments:
(define busybox-sysctl-shepherd-service
(match-lambda
(($ (@@ (gnu services sysctl) <sysctl-configuration>) sysctl settings)
(let ((sysctl.conf
((@@ (gnu services sysctl) sysctl-configuration-settings->sysctl.conf) settings)))
(shepherd-service
(documentation "Configure kernel parameters at boot.")
(provision '(sysctl))
(start #~(lambda _
(zero? (system* #$sysctl "-p" #$sysctl.conf))))
(one-shot? #t))))))
(define busybox-sysctl-service-type
(service-type
(inherit sysctl-service-type)
(extensions
(list (service-extension
shepherd-root-service-type
(compose list busybox-sysctl-shepherd-service))))))With these simple changes, we're down to 493MB. Writing a replacement
for udev-service-type using busybox's mdev is left as an exercise
for the reader.
Full config is here.
While 493MB is small enough to be a valid Ubuntu replacement for embedded systems, it's quite a ways off from a replacement for buildroot-based systems. While there's still a bit more space to be trimmed via configuration, I suspect package transformations are the next step to getting this much smaller. In particular, I'd like to try these sometime in the future:
System.map from the kernel package, since it isn't very
useful on an embedded system.guile packages (there are three in the store
on the rootfs and I hope we could get rid of two).-Os similar to the existing --tune=<CPU>
transform.$ guix describe --format=channels
(list (channel
(name 'guix)
(url "https://git.savannah.gnu.org/git/guix.git")
(branch "master")
(commit
"0f3a25a25e212bfa8ab9db37d267fb260a087e5d")
(introduction
(make-channel-introduction
"9edb3f66fd807b096b48283debdcddccfea34bad"
(openpgp-fingerprint
"BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))
(channel
(name 'nonguix)
(url "https://gitlab.com/nonguix/nonguix")
(branch "master")
(commit
"92f4921c6603e81693a01b0cd7c50977e0b92788")
(introduction
(make-channel-introduction
"897c1a470da759236cc11798f4e0a5f7d4d59fbc"
(openpgp-fingerprint
"2A39 3FFF 68F4 EF7A 3D29 12AF 6F51 20A0 22FB B2D5")))))