Multicore

Since release 3.0, Miosix supports multicore (SMP) microcontrollers, such as the RP2040. SMP support is required in three key areas: boot, locking, interrupts and scheduler. The kernel is configured with SMP enabled if the board configuration defines the macro WITH_SMP, otherwise only the first core is used.

SMP interfaces

SMP-specific functions that need to be implemented by each architecture port are defined in smp.h. Other interfaces related to SMP support, but not specific to it, are os_timer.h, interrupts.h, cpu_const.h and atomic_ops.h.

Boot

At boot, if SMP is enabled, Miosix calls a platform-specific function defined in smp.h called IRQinitSMP(). This function is responsible early at boot for doing the following:

• Allocating a system stack (in ARM parlance, the stack for interrupt handlers) for all non-zero cores.
• Starting up all other cores (except core 0 which is already started up) and configuring their system stacks and process stacks (in ARM parlance, the stack for running non-interrupt code).
• Configuring the interrupt vectors for all cores to be the same as core 0 (the interrupt vectors are shared for all cores, but the enabled/disabled state must be per-core).
• Initializing and starting the OS timer for all cores included core 0.
• Jumping to the specified main function in all cores.

At the end of this function, all other cores must be up enough to be able to immediately process a IRQinvokeSchedulerOnCore or an IRQcallOnCore, which will be discussed later.

Locking

When in kernel mode, the hardware implicitly creates one special "thread" that needs to be synchronized against: interrupt handlers. Interrupt handlers access OS state (specifically, peripheral driver state, or scheduler state for the timer interrupt) but --- barring magic shenanigans that are complicated to implement and add significant overhead --- cannot yield to another thread. Hence one cannot use conventional synchronization primitives.

In the single core case, the easiest way to ensure mutual exclusion with an interrupt handler is to disable interrupts altogether. Since preemption is driven by interrupts (even wakeups, at least in ARM architectures, due to the use of PendSV for implementing the Yield primitive) this also has the effect of disabling preemption. Disabling preemption means we also are in mutual exclusion with other pieces of code that disable/enable interrupts. There is no need to disable interrupts in interrupt handlers because they already are assumed to run in mutual exclusion between each other. In reality this is not strictly true on platforms that have nested interrupts but Miosix disables nested interrupts.

In an SMP architectures, being in an interrupt handler and/or disabling interrupts does not ensure mutual exclusion with other interrupt handlers or with other pieces of code which run with interrupts disabled. This is because interrupts are per-core, and the interrupt disabling flag is also per-core. Even with interrupts disabled, or in an interrupt handler, the other cores can run code in parallel, with interrupts disabled or not, or even in an interrupt handler.

The solution to this is to use spinlocks. A fundamental property of spinlocks, as opposed to other synchronization methods, is that they do not require anything from the operating system, they can always be used even with preemption disabled or in an interrupt handler. Unfortunately they also burn CPU cycles, but at least on ARM the WFE instruction can be used to put a core to sleep while waiting for a spinlock to be released, and the SEV instruction can wake all cores sleeping on a WFE when a spinlock is released. These instructions work even with interrupts disabled.