Introduction

Since the Yocto warrior release, the RAUC (Robust Auto-Update Controller) mechanism support has been added to Yogurt. It controls the procedure of updating a device with new firmware. This includes updating Linux kernel, Device Tree, and root filesystem. For eMMC devices only, it can also update the bootloader. For detailed information about RAUC, visitrauc.readthedocs.io.

This manual describes how RAUC is used and implemented on various PHYTEC platforms. Note, that different modules use different bootloaders and flash storage devices, which affects the way things are handled by RAUC. Make sure to read the correct sections fitting your platform.

System Configuration

RAUC can be used with both eMMC and NAND flash storage. It is not, by default, enabled but our example can be configured and activated with the instructions shown below. RAUC can be used in different update scenarios. As an example, we configured the BSP to use an A/B setup to have a completely redundant system (including the bootloader on eMMC devices).

The partition layout is defined in the /etc/rauc/system.conf file. As an example, this is how it looks like for i.MX 8M Mini with eMMC flash storage:

[system]
compatible=phyboard-polis-imx8mm-3
bootloader=uboot
mountprefix=/mnt/rauc

[handlers]
pre-install=/usr/bin/rauc_downgrade_barrier.sh

[keyring]
path=ca.cert.pem

# System A
[slot.rootfs.0]
device=/dev/mmcblk2p2
type=ext4
bootname=system0

[slot.boot.0]
device=/dev/mmcblk2p1
type=vfat
parent=rootfs.0
                     
# System B           
[slot.rootfs.1]      
device=/dev/mmcblk2p4
type=ext4            
bootname=system1     
                     
[slot.boot.1]        
device=/dev/mmcblk2p3
type=vfat            
parent=rootfs.1

Note, that the devices specified in the slots are different depending on the selected machine.

Design Considerations

In order to prevent the system from locking up, it may be a good idea to utilize a hardware watchdog. In case the Linux Kernel does not boot or another catastrophic event occurs that prevents the system from operating normally, the hardware watchdog then resets the system. By default, the hardware watchdog is disabled. To enable it, refer to the corresponding BSP manual that fits your SoM.

Initial Setup

To use RAUC, the flash device needs to be written and the bootloader must be set up to start the A/B system.

Flash storage

eMMC

To flash the eMMC with the correct partitions one line needs to be added in the local.conf file of the BSP:

# Select a preconfigured A/B system setup for eMMC images.
#WKS_FILES_mx6 = "imx6-rauc-sdimage.wks"
#WKS_FILES_mx6ul = "imx6-rauc-sdimage.wks"
WKS_FILES_mx8m = "imx8m-rauc-sdimage.wks"

Enable the correct partitioning scheme by removing the comment as shown above. Build the image as usual:

host$ bitbake phytec-headless-image

Flash the .sdcard image onto a running target's eMMC from the host:

host$ dd if=phytec-headless-image-phyboard-polis-imx8mm-3.sdcard | ssh root@192.168.3.11 "dd of=/dev/mmcblk2 bs=100M"

Now the target is able to boot the flashed A/B system.

NAND

With raw NAND flash the kernel, device tree, and root filesystem are written individually. Initialize the NAND flash with the correct volumes from a Linux on the target:

target$ rauc-flash-nand -k /path/to/zImage -d /path/to/oftree -r /path/to/root.ubifs

The initialization script will automatically utilize all available space of the NAND flash. The NAND device is also determined automatically by finding the device root in /proc/mtd.

The A/B system on NAND flash is now ready to be booted.

Bootloader

Booting the A/B System by Default

Booting of the A/B system is done mostly automatically by the bootloader since Yocto release hardknott. For devices with eMMC flash storage, the corresponding setting is written into the bootloader environment during the building of the BSP. In particular, if WKS_FILES is set to a RAUC image, as described previously, the bootloader should automatically start the A/B system and set variables for the updated client RAUC accordingly.

This automatic setting can be manually changed by setting one variable in the bootloader. The procedure is described in more detail in the following chapters for U-Boot and barebox.

U-Boot

After a successful boot into a Linux environment, this command is used to view the available parameters:

target$ fw_printenv

You may see this parameter along with others in the output:

doraucboot=1

To manually disable or enable booting the A/B system with RAUC, set this variable to "0" or "1":

target$ fw_setenv doraucboot 1

This parameter can also be edited in U-Boot. Restart your board and hit any key to stop the automatic boot. The environment variables can now be viewed:

bootloader$ printenv

and set:

bootloader$ setenv doraucboot 1
bootloader$ saveenv

Barebox

In barebox, the system to be booted can be selected directly by its name. To boot the A/B system including RAUC, bootchooser is used. To boot e.g. an SD card use mmc instead, or nand for NAND devices.

bootloader$ nv boot.default=bootchooser

Creating RAUC Bundles

To update your system with RAUC, a RAUC bundle (.raucb) needs to be created. It contains all required images and scripts for the update and a RAUC manifest.raucm that describes the content of the bundle for the RAUC update on the target. The BSP includes a Yocto target that lets you build a RAUC bundle from your Yocto build.

To create the bundle with Yocto, run:

host$ bitbake phytec-headless-bundle

This results in the creation of a .raucb bundle file in deploy/images/<MACHINE>/ which can be used for updating the system as described later. There is no need to create a manifest.raucm manually as it is created automatically during the build of the bundle. As a reference, the created manifest would look something like this:

[update]
compatible=phyboard-polis-imx8mm-3
version=r0
description=PHYTEC rauc bundle based on BSP-Yocto-FSL-i.MX8MM-PD20.1.0
build=20200624074335

[image.rootfs]
sha256=cc3f65cd1c1993951d7a39bdb7b7d723617ac46460f8b640cd8d1622ad6e4c17
size=99942000
filename=phytec-headless-image-phyboard-polis-imx8mm-3.tar.gz

[image.boot]
sha256=bafe46679af8c6292dba22b9d402e3119ef78c6f8b458bcb6993326060de3aa4
size=12410534
filename=boot.tar.gz.img

For more information about the manifest format, see https://rauc.readthedocs.io/en/latest/reference.html#manifest.

Updating with RAUC

To update the target system with RAUC, the RAUC bundle file previously created first needs to be copied to the board or to a memory device that can be mounted in Linux. One way is to copy the bundle file with scp, but this requires enough space left on the board's filesystem. To do this, boot the target board to Linux and connect it via Ethernet to your host PC.

On the host, run:

host$ scp phytec-headless-bundle-phyboard-polis-imx8mm-3.raucb root@192.168.3.11:/root/

On the target, the bundle can be verified:

target$ rauc info /root/phytec-headless-bundle-phyboard-polis-imx8mm-3.raucb

and the output should look similar to this:

rauc-Message: 12:52:49.821: Reading bundle: /phytec-headless-bundle-phyboard-polis-imx8mm-3.raucb
rauc-Message: 12:52:49.830: Verifying bundle... 
Compatible:     'phyboard-polis-imx8mm-3'
Version:        'r0'
Description:    'PHYTEC rauc bundle based on BSP-Yocto-FSL-i.MX8MM-PD20.1.0'
Build:          '20200624073212'
Hooks:          ''
2 Images:
(1)     phytec-headless-image-phyboard-polis-imx8mm-3.tar.gz
        Slotclass: rootfs
        Checksum:  342f67f7678d7af3f77710e1b68979f638c7f4d20393f6ffd0c36beff2789070
        Size:      180407809
        Hooks:     
(2)     boot.tar.gz.img
        Slotclass: boot
        Checksum:  8c84465b4715cc142eca2785fea09804bd970755142c9ff57e08c791e2b71f28
        Size:      12411786
        Hooks:     
0 Files

Certificate Chain:
 0 Subject: /O=PHYTEC Messtechnik GmbH/CN=PHYTEC Messtechnik GmbH Development-1
   Issuer: /O=PHYTEC Messtechnik GmbH/CN=PHYTEC Messtechnik GmbH PHYTEC BSP CA Development
   SPKI sha256: E2:47:5F:32:05:37:04:D4:8C:48:8D:A6:74:A8:21:2E:97:41:EE:88:74:B5:F4:65:75:97:76:1D:FF:1D:7B:EE
   Not Before: Jan  1 00:00:00 1970 GMT
   Not After:  Dec 31 23:59:59 9999 GMT
 1 Subject: /O=PHYTEC Messtechnik GmbH/CN=PHYTEC Messtechnik GmbH PHYTEC BSP CA Development
   Issuer: /O=PHYTEC Messtechnik GmbH/CN=PHYTEC Messtechnik GmbH PHYTEC BSP CA Development
   SPKI sha256: AB:5C:DB:C6:0A:ED:A4:48:B9:40:AC:B1:48:06:AA:BA:92:09:83:8C:DC:6F:E1:5F:B6:FB:0C:39:3C:3B:E6:A2
   Not Before: Jan  1 00:00:00 1970 GMT
   Not After:  Dec 31 23:59:59 9999 GMT

To check the current state of the system, run:

target$ rauc status

and get output similar to this:

Compatible:  phyboard-polis-imx8mm-3
Variant:
Booted from: rootfs.0 (system0)
Activated:   rootfs.0 (system0)
slot states:
  rootfs.0: class=rootfs, device=/dev/mmcblk2p2, type=ext4, bootname=system0
      state=booted, description=, parent=(none), mountpoint=/
      boot status=good
  boot.0: class=boot, device=/dev/mmcblk2p1, type=vfat, bootname=(null)
      state=active, description=, parent=rootfs.0, mountpoint=(none)

  rootfs.1: class=rootfs, device=/dev/mmcblk2p4, type=ext4, bootname=system1
      state=inactive, description=, parent=(none), mountpoint=(none)
      boot status=good
  boot.1: class=boot, device=/dev/mmcblk2p3, type=vfat, bootname=(null)
      state=inactive, description=, parent=rootfs.1, mountpoint=(none)

To update the currently inactive system with the downloaded bundle, run:

target$ rauc install /root/phytec-headless-bundle-phyboard-polis-imx8mm-3.raucb

and reboot afterward:

target$ reboot

With the success of the update, RAUC automatically switches the active system to the newly updated system. Now during reboot, RAUC counts the boot attempts of the kernel and if it fails more often than specified in the state framework of the system, RAUC switches back to the old system and marks the new system as bad. If the boot attempt to the kernel is successful, the new system is marked as good and the old system can now be updated with the same instructions. After two successful rauc install and reboot, both systems are updated.

Changing the Active Boot Slot

It is possible to switch the active system manually:

target$ rauc status mark-active other

After a reboot, the target now starts from the other system.

Switching RAUC Keyrings

PHYTEC's distribution comes with keys and certificates used for development and demonstration purposes only. To change to a different PKI when devices are already rolled out, RAUC's keyring must be changed. This chapter describes the full procedure from a development state to a production state. Keep in mind, that it is always a better idea to roll out your devices with a production keyring in the first place, instead of relying on a development one for too long. The following diagram shows the general process of switching keyrings for RAUC:

Create new certificates and keys for your own PKI using the openssl-ca.sh script available in the meta-yogurt/meta-ampliphy Yocto layer. See our security manual for a detailed description of the usage of this script.

Navigate to the directory of the meta-yogurt/meta-ampliphy layer and execute the script with the option -r:

host$ ./scripts/openssl-ca.sh -r

Enter an appropriate name and organization. For this document, we refer to this newly created PKI as "production", as opposed to the existing "development" keys.

Move the generated keys and certificates, contained in the openssl-ca directory, outside of the Yocto layer. E.g. to your main Yocto build directory root, alongside with build/ and sources/.

Now, a RAUC bundle must be created that contains the new "production" CA keyring in its root filesystem but is still signed by the "development" CA. With this, the system is converted from a "development" system to a "production" system. To achieve this, exchange the file ca.cert.pem installed by the RAUC recipe in the Yocto sources. Create a file rauc_%.bbappend in your own Yocto layer:

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

RAUC_KEYRING_FILE = "${OEROOT}/../../openssl-ca/rauc-customer/ca.cert.pem"

Build the same RAUC bundle as before, now with the exchanged keyring:

host$ source source/poky/oe-init-build-env
host$ bitbake phytec-headless-bundle  # Build the desired RAUC bundle

Install the resulting RAUC bundle as usual. The target now has the image with the "production" keyring installed in its other slot ("System B" in the figure above). Reboot to start that system.

All future RAUC bundles for the "production" system must now also be signed by the "production" CA. For this, change the key and certificate to your newly generated "production" ones in the bundle recipe:

require phytec-base-bundle.inc

RAUC_SLOT_rootfs ?= "phytec-headless-image"

RAUC_KEY_FILE = "${OEROOT}/../../openssl-ca/rauc-customer/private/production-1.key.pem"
RAUC_CERT_FILE = "${OEROOT}/../../openssl-ca/rauc-customer/production-1.cert.pem"

Rebuild the RAUC bundle:

host$ bitbake customer-headless-bundle

These and any future bundles are now ready to be installed on your "production" target system and have been fully migrated away from the "development" system. This also means that now only bundles signed by the "production" CA can be installed on the target (and e.g. "development" bundles cannot).

Use Case Examples

Use Case 1: Automatic Update from USB Flash Drive with RAUC

One of the most prominent use cases for RAUC might be an automatic update system from a USB flash drive. This use case is implemented in the BSP as a reference example. We combine only standard Linux mechanisms with RAUC to build the system. The kernel notifies udev when a device gets plugged into the USB port. We use a custom udev rule to trigger a systemd service when this event happens.

KERNEL!="sd[a-z][0-9]", GOTO="media_by_label_auto_mount_end"
 
# Trigger systemd service
ACTION=="add", TAG+="systemd", ENV{SYSTEMD_WANTS}="update-usb@%k.service"
 
# Exit  
LABEL="media_by_label_auto_mount_end"

The service automatically mounts the USB flash drive and notifies the application.

[Unit]
Description=usb media RAUC service
After=multi-user.target
Requires=rauc.service
 
[Service]
Type=oneshot
Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/dbus/system_bus_socket
ExecStartPre=/bin/mkdir -p /media/%I
ExecStartPre=/bin/mount -t auto /dev/%I /media/%I
ExecStart=/usr/bin/update_usb.sh %I
ExecStop=/bin/umount -l /media/%i
ExecStopPost=-/bin/rmdir /media/%I

In our reference implementation, we simply use a bash script for the application logic.

#!/bin/sh
 
MOUNT=/media/$1
 
NUMRAUCM=$(find ${MOUNT}/*.raucb -maxdepth 0 | wc -l)
 
[ "$NUMRAUCM" -eq 0 ] && echo "${MOUNT}*.raucb not found" && exit
[ "$NUMRAUCM" -ne 1 ] && echo "more than one ${MOUNT}/*.raucb" && exit
 
rauc install $MOUNT/*.raucb
if [ "$?" -ne 0 ]; then
	echo "Failed to install RAUC bundle."
else
	echo "Update successful."
fi
exit $?

The update logic can be integrated into an application by using systemd's D-Bus API. RAUC does not need to be called by its command-line interface but can be integrated with D-Bus.

Use Case 2: Security Measurement: Downgrade Barrier

As a second reference example, we will implement a security mechanism: a downgrade barrier. When you detect a security vulnerability on your system, you will fix it and update your system. The systems with the new software will now be secure again. If an attacker gets ahold of the old software update bundle, which still has a valid signature, the attacker might have the possibility to install the old software and still take advantage of the previously fixed security vulnerability. To prevent this from happening, you could revoke the update certificate for every single update and create a new one. This might be difficult to handle, depending on the environment. A simpler solution would be to allow updates only in one direction using a version check.

#!/bin/sh                                       
                                                  
VERSION_FILE=/etc/rauc/downgrade_barrier_version  
MANIFEST_FILE=${RAUC_UPDATE_SOURCE}/manifest.raucm
                                   
[ ! -f ${VERSION_FILE} ] && exit 1 
[ ! -f ${MANIFEST_FILE} ] && exit 2            
                                                                      
VERSION=`cat ${VERSION_FILE} | cut -d 'r' -f 2`                       
BUNDLE_VERSION=`grep "version" -rI ${MANIFEST_FILE} | cut -d 'r' -f 3`
                                     
# check from empty or unset variables
[ -z "${VERSION}" ] && exit 3       
[ -z "${BUNDLE_VERSION}" ] && exit 4                
                                                    
# developer mode, allow all updates if version is r0
#[ ${VERSION} -eq 0 ] && exit 0
                                             
# downgrade barrier                                          
if [ ${VERSION} -gt ${BUNDLE_VERSION} ]; then                
        echo "Downgrade barrier blocked rauc update! CODE5\n"
else          
        exit 0
fi            
exit 5

The script is installed on the target but it is not activated. You need to remove the developer mode line in the script to activate it.

Reference

Boot Logic Implementation

U-Boot Environment Variables

For U-Boot, the bootlogic that selects the correct partitions to boot from is implemented in its environment. As a reference, these are the most important U-Boot variables that are used for the A/B system with RAUC:

These environment variables are defined in include/configs/phycore_<SOC>.h in the u-boot source code.

Barebox Bootchooser Framework

For the barebox, the bootlogic that selects the correct partitions to boot from is implemented using the bootchooser and state framework. See the barebox documentation for detailed information about these: Barebox Bootchooser Framework, Barebox State Framework.

First, the state framework configuration needs to be added to the barebox device tree. Check out the Customizing the BSP chapter in the Yocto reference manual. The state framework configuration is already included with our BSP for the supported SoC and can be directly included in the main barebox device tree. E.g. for i.MX6 based module:

#include "imx6qdl-phytec-state.dtsi"

Afterward, rebuild the image and flash the new bootloader.

The following device tree snippet shows an example of the state framework configuration used with the BSP. As can be seen, the EEPROM is used as a backend for the state information.

/ {
    aliases {
        state = &state;
    };
  
    state: imx6qdl_phytec_boot_state {
        magic = <0x883b86a6>;
        compatible = "barebox,state";
        backend-type = "raw";
        backend = <&backend_update_eeprom>;
        backend-stridesize = <54>;
  
        #address-cells = <1>;
        #size-cells = <1>;
        bootstate {
            #address-cells = <1>;
            #size-cells = <1>;
            last_chosen {
                reg = <0x0 0x4>;
                type = "uint32";
            };
            system0 {
                #address-cells = <1>;
                #size-cells = <1>;
                remaining_attempts {
                    reg = <0x4 0x4>;
                    type = "uint32";
                    default = <3>;
                };
                priority {
                    reg = <0x8 0x4>;
                    type = "uint32";
                    default = <21>;
                };
                ok {
                    reg = <0xc 0x4>;
                    type = "uint32";
                    default = <0>;
                };
            };
            system1 {
                #address-cells = <1>;
                #size-cells = <1>;
                remaining_attempts {
                    reg = <0x10 0x4>;
                    type = "uint32";
                    default = <3>;
                };
                priority {
                    reg = <0x14 0x4>;
                    type = "uint32";
                    default = <20>;
                };
                ok {
                    reg = <0x18 0x4>;
                    type = "uint32";
                    default = <0>;
                };
            };
        };
    };
};
  
&eeprom {
    status = "okay";
    partitions {
        compatible = "fixed-partitions";
        #size-cells = <1>;
        #address-cells = <1>;
        backend_update_eeprom: state@0 {
            reg = <0x0 0x100>;
            label = "update-eeprom";
        };
    };
};

To be able to boot from two systems alternately, the bootchooser needs to be aware of the state framework configuration. For each system, a boot script is required. For a system with NAND flash, the boot script of the first system may look like the following:

#!/bin/sh

[ -e /env/config-expansions ] && /env/config-expansions

[ ! -e /dev/nand0.root.ubi ] && ubiattach /dev/nand0.root

global.bootm.image="/dev/nand0.root.ubi.kernel0"
global.bootm.oftree="/dev/nand0.root.ubi.oftree0"
global.linux.bootargs.dyn.root="root=ubi0:root0 ubi.mtd=root rootfstype=ubifs"

The second boot script has the same structure but uses the partitions containing the second system. Machines with eMMC flash use similar boot scripts, albeit the mounting and boot arguments look different.

Run the following commands to create the required bootchooser non-volatile environment variables:

bootloader$ nv bootchooser.state_prefix=state.bootstate
bootloader$ nv bootchooser.system0.boot=system0
bootloader$ nv bootchooser.system1.boot=system1
bootloader$ nv bootchooser.targets="system0 system1"

eMMC Boot Partitions

With eMMC flash storage it is possible to use the dedicated boot partitions for redundantly storing the bootloader.

By default, bundles built with our BSP (e.g. phytec-headless-bundle) contain the bootloader for updating eMMC boot partitions accordingly.

Note, that the U-Boot environment still resides in the user area before the first partition. The user area also still contains the bootloader which the image first shipped during its initialization process.

To manually write the bootloader to the eMMC boot partitions, first disable the write protection of these:

target$ echo 0 > /sys/block/mmcblk2boot0/force_ro
target$ echo 0 > /sys/block/mmcblk2boot1/force_ro

Write the bootloader to the eMMC boot partitions:

target$ dd if=imx-boot of=/dev/mmcblk2boot0 bs=1k seek=33
target$ dd if=imx-boot of=/dev/mmcblk2boot1 bs=1k seek=33

This example is valid for the i.MX8M Mini SoC. Note, that other SoCs may have different bootloader files and require different offsets where the bootloader is expected, specified by the seek parameter. See the following table for the different offsets being required by each SoC:

Note that the offset is different, depending on whether the bootloader resides in the user area or the boot partitions of the eMMC.

After a bootloader has been written to the eMMC boot partitions, booting from these can be enabled by using the following command:

target$ mmc bootpart enable 1 0 /dev/mmcblk2

This also means that only the bootloaders written in the eMMC boot partitions are used. The bootloader in the user area is not used anymore. These steps are also executed by RAUC internally when updating the target system with a bundle.

To disable booting from the eMMC boot partitions simply enter the following command:

target$ mmc bootpart enable 0 0 /dev/mmcblk2

After this command, the eMMC user area is used to provide the bootloader.