Zephyr-Linux Co-Integration – Meeting Early Bootup Requirements

Modern application processors increasingly ship with more than one kind of CPU in the same package. This allows a much more direct integration of a feature-rich multi-purpose operating system together with an operating system for real-time or safety applications, compared to what would be possible with separate, dedicated microprocessors or controllers.

A common combination consists of an ARM Cortex-A application processor together with a separate Cortex-M real-time processor. The first of the two typically runs Linux and takes care of networking, heavy computation or a user interface, while the latter runs an RTOS like Zephyr and is responsible for applications with real-time or safety requirements or managing low-level peripherals.

Such a system requires careful, well-defined coordination between the involved processors, which is also referred to as lifecycle management: The startup ordering of the processors must be clearly defined, as well as the communication channels and used protocols between them, and sometimes also the coordination of shutdown sequences.

In the context of Linux, this is achieved with the remoteproc framework. Once booted, a Linux userspace component can use this framework to control the real-time processor (also referred to as the remote processor), load an application to it, and help establish necessary communication channels. The entire process is well documented, and plenty of documentation (remoteproc, rpmsg, virtio 1.3), articles, and conference talks are already available (ST Microelectronics: Linux RPMsg framework overview, rt-rk: RPMsg-Based Inter-Core Communication in Heterogeneous Embedded Systems, Toradex: How To Use RemoteProc, Iuliana Prodan: Enhancing OpenAMP: Making Linux and Zephyr Work Better Together). However, this solution is not suitable for systems which require a very fast bootup of the real-time processor. Even if the Linux system is optimized for fast bootup, the resulting bootup delay for the real-time processor will still be measured in seconds. This is too long if the real-time system is required to be up and running within milliseconds.

One solution for this problem is booting the realtime core before Linux, then let Linux attach to it. While this can easily be achieved by small changes in the bootloader configuration, the real challenge is adapting the lifecycle management. With the same Linux-side configuration as before, Linux will assume it has full control over the lifecycle and resets and reconfigures the already running realtime processor.

So the question is, how can we establish a bootup process where Linux does not follow its default load model but instead attaches itself to the already running processor? And how are the communication channels established, if the lifecycle is no longer managed only by Linux? In contrast to the well documented load model, there is little to no documentation available on how to implement the attach model.

This article is split into two parts: this first installment establishes the foundational concepts and technical background, while the second part, following shortly, will walk through a complete hands-on implementation.

Available Building Blocks

The building blocks for establishing communication between two independent processors are already available. The following sections assign them to their respective layer in the ISO/OSI layering model and briefly describe their role.

Shared Memory and Inter-Processor Interrupts

Communication between two independent processors requires a means of data transfer and a way of signalling each other. Correctly sequencing both allows implementing robust inter-processor communication.

Although other methods exist, shared memory is the most common mechanism for transferring data between independent cores on a single chip. With shared memory, the sending processor writes data into a memory region accessible by both processors and triggers an inter-processor interrupt to notify the receiving processor. The receiving processor then reads the data and acknowledges completion back to the sender.

Establishing and using shared memory for inter-processor communication comes with three general challenges: both involved sides must agree on the concrete memory location and size of the shared memory carveout, they may need to maintain cache coherence, and accessing the shared memory region might impact performance. In addition, inter-processor interrupts are mostly implemented with vendor-specific solutions. In most cases, a dedicated on-chip peripheral provides the necessary functionality but typically requires driver support from both involved operating systems. For this part of the article series, it is sufficient to know that these challenges exist. The second part will go into more detail about how they are solved in a concrete example.

Together, shared memory and inter-processor interrupts form the physical layer of the ISO/OSI layering model.

The Resource Table

A shared memory carveout alone is not sufficient. Both sides also need to agree on a shared data structure definition, and it must be clarified which side is responsible for its initialization. This is the task of the resource table: it is a supporting data structure for establishing communication channels on top of shared memory.

The format and purpose of the resource table are defined by Linux’s remoteproc framework. It is used by the remote processor to tell Linux what it needs to properly function. This includes the number and type of communication channels for higher-level protocols as well as their configuration and status information.

The remoteproc framework establishes four default resource categories inside the resource table. These categories include carveout, devmem, trace, and vdev. Among them, only vdev plays a role throughout this article series. Furthermore, the label carveout creates a semantic overlap: on one hand, it describes the statically defined shared memory sections, and on the other hand, a resource type built into remoteproc and its resource table. For proprietary requirements, developers may implement custom protocols through a designated range of vendor-specific resource types.

Crucially, the interaction between the remote and master processors is implemented as a strict system contract rather than an active conversation during runtime. The remote processor statically defines its demands within the resource table at compile time, and the master processor reads these demands to provision the environment. If the master side fails to provide the resources requested via the table, communication cannot succeed. This mechanism is independent of the underlying real-time operating system and functions identically whether using Zephyr, another RTOS, or a bare-metal implementation.

As with the shared memory carveout, the memory location for the resource table must be agreed on by both processors. When using the load model, the resource table is placed in a special section within the ELF image that contains the application for the remote processor. In that case, Linux’s remoteproc framework will extract the resource table from this ELF section and place it at the correct memory location before the application on the remote processor is started. In contrast, the attach model requires that the resource table is placed in memory before Linux boots. It must, however, still be initialized by Linux. How this is achieved in detail will be explained in the second part of this article.

The resource table cannot be attributed to a singular ISO/OSI layer: it contains configuration information relevant throughout the entire communication stack.

VirtIO and VRings

As mentioned above, the resource table tells the controlling side how many communication channels of what type it shall create, such that both sides can use them. Although custom transports could be used, the remote side typically requests communication using Virtual I/O (short: virtio), using two vrings.

The virtio specification is mostly known in the context of virtual machines, but its concepts also apply to situations where two processors communicate with each other over shared memory. Virtio knows two types of communication participants: devices and drivers. Here, Linux provides the driver side, and the remote processor acts as a device.

Virtio offers a number of predefined device types. Each one has, among other properties, a well-known device id, a specific number of vrings for predefined purposes, and a specified runtime behavior.

With virtio, device and driver exchange bulk data with the help of so-called vrings (the virtio specification calls them virtqueues, Linux and Zephyr refer to them as vring). Instead of the actual payload, a vring transports buffer descriptors, and handles pieces of memory allocated from a shared buffer pool. A noteworthy detail is the unidirectionality of a vring: communication from driver to device happens through a different vring than communication in the other direction. 

Virtio and vrings implement the transport layer of the ISO/OSI layering model.

RPMsg

On top of the communication facilities implemented by virtio lives rpmsg, also known as “Remote Processor Messaging”. Rpmsg is a messaging bus used to open communication endpoints and announce services by name. Rpmsg is directly supported by virtio as a dedicated, well-known device type. Its specification is part of the Linux kernel documentation.

As soon as a remote virtio device creates an rpmsg endpoint, the Linux kernel tries to find an rpmsg driver that matches the announced channel name. Upon success, Linux loads that driver and begins interacting with the virtual device. If, for example, Zephyr on the remote side creates an endpoint with the name rpmsg-tty, and if Linux has a matching rpmsg device driver, then Linux will instantiate a /dev/ttyRPMSG0 device. This new tty device implements the other side of the connection. If Linux has no matching driver, udev will see the announcement, but no driver will be loaded.

Announcing a new service is also possible from the Linux side. Analog to the example above, a Linux driver could as well declare a new endpoint, and Zephyr could react to this event. In this article, the focus is on announcements made by Zephyr.

Load and Attach Model Compared

The above building blocks are everything that is needed to implement both use cases: the load model and also the attach model. Since the ultimate goal is booting the remote core as early as possible, it is worth taking one step back and visualizing the bootup flow of both models.

Figure 1 depicts the well documented load model. After the device is powered on, the bootloader starts executing on the Cortex-A53 processor. The bootloader loads the Linux kernel into memory and starts the kernel. After the Linux kernel brings up a functioning userspace, a process is able to locate, load, and deploy the ELF image for the remote Cortex-M7 core. Only now, the M7 core is released, boots up Zephyr, and a handshake between both processors can take place.

In contrast, Figure 2 shows the attach model. The bootloader first takes care of loading and starting the application for the remote processor, and only afterwards the Linux kernel is started. The M7 core begins executing even before the Linux kernel is loaded into memory. It can run critical business logic while waiting for Linux to become ready in the background, which is exactly what is needed for an early boot-up scenario.