PREEMPT_RT Linux: Configuration and Cyclictest Latency Tests

Practical guide to PREEMPT_RT on embedded Linux covering kernel configuration, IRQ threading, latency measurement with cyclictest, and tuning techniques with explicit test conditions and interpretation limits.

PREEMPT_RT Linux: Configuration and Cyclictest Latency Tests

PREEMPT_RT transforms the Linux kernel into a viable platform for soft and firm real-time applications. Use an architecture and kernel branch that support CONFIG_PREEMPT_RT, then verify the vendor configuration. This guide covers what PREEMPT_RT actually does, how to configure and tune it, and - critically - how to document observed scheduling latency and the conditions a test covers.

What PREEMPT_RT changes

Standard Linux is optimized for throughput. The kernel disables preemption during critical sections, holds spinlocks with interrupts disabled, and batches work for efficiency. These paths affect scheduling response and need measurement on the selected configuration.

PREEMPT_RT makes the following changes:

  • Spinlocks become sleeping locks: Most kernel spinlocks are replaced with rt-mutexes that can sleep, allowing higher-priority threads to preempt lower-priority ones even inside kernel code paths.
  • Interrupt handlers run as threads: Hardware interrupt handlers are converted to kernel threads with configurable priorities. This means a high-priority real-time task can preempt interrupt handling.
  • Priority inheritance: rt-mutexes implement priority inheritance to prevent priority inversion - a low-priority task holding a lock needed by a high-priority task temporarily inherits the high priority.
  • High-resolution timers: Timer resolution is improved to the hardware timer's native resolution (typically 1 µs or better).

Measure the scheduling response on your selected hardware and kernel. PREEMPT_RT alone establishes no numerical latency bound.

Kernel configuration

The following configuration names are a starting checklist for Linux 6.12. Verify architecture support, dependencies and the vendor kernel configuration before enabling them:

CONFIG_PREEMPT_RT=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_NO_HZ_FULL=y
CONFIG_CPU_FREQ=n        # or use performance governor
CONFIG_CPU_IDLE=n         # or configure shallow idle states

Frequency and idle policies can affect latency, temperature and power. Compare supported policies on the target under sustained load. CPU idle-state names and available controls vary by architecture. Keep the original configuration and record each tested change.

IRQ thread priorities

PREEMPT_RT threads most interrupt handlers, with exceptions for low-level and explicitly non-threaded paths. You can see them with ps:

ps -eo pid,cls,rtprio,comm | grep -i irq

Set priorities based on your application's requirements. This illustrative priority ordering needs validation against the actual dependency graph:

A real-time thread must not starve an IRQ thread or worker on which it depends. Record the complete wakeup path before assigning priorities. Include watchdogs, communication, storage and recovery work in that review. A universal numeric priority table cannot establish a safe ordering for an unknown device.

Use chrt to set thread priorities:

chrt -p THREAD_PID  # inspect a known thread before planning changes

Or configure them in udev rules or systemd unit files for persistence.

Measuring latency with cyclictest

cyclictest is the standard tool for measuring scheduling latency on PREEMPT_RT systems. It creates high-priority threads that sleep for a defined interval and measures how long they actually slept (the difference is the scheduling latency).

The command below uses options documented for rt-tests 2.6. Run it on disposable test equipment with appropriate scheduling permissions. Record the actual output of cyclictest --version and keep the kernel configuration with the results.

cyclictest --version
uname -a
cyclictest --mlockall --threads=1 --priority=80 --interval=1000 --duration=10m --histogram=100 --quiet --json=run.json > run.txt

This page supplies a synthetic histogram and annotated summary and a normalised JSON fixture. The JSON uses ProteanOS fixture schema 1, with explicit units and provenance. It is an instructional representation. Native cyclictest JSON must be interpreted against its generating rt-tests version.

The synthetic histogram contains 50 samples at 1 microsecond, 40 at 2 and 10 at 10. It has 100 samples, mean 2.3 microseconds and maximum 10. Under the nearest-rank convention, p95 and p99 are both 10. Summary-only output cannot supply these percentiles. If samples overflow the recorded histogram, percentiles in that unobserved tail remain unavailable or bounded below by the overflow threshold.

Thread identifiers identify measurement threads. CPU affinity requires separate recorded evidence. Never add a total histogram column to its constituent thread columns. When combining means, weight each thread by its sample count. Record missing hardware, kernel, workload and duration as unknown. All values in these downloads are synthetic and represent no device test.

Stress during measurement

Keep an idle baseline and compare it with representative application load. Run stress workloads simultaneously:

# CPU stress
stress-ng --cpu $(nproc) --timeout 86400 &

# I/O stress
stress-ng --hdd 4 --timeout 86400 &

# Network stress (if applicable)
iperf3 -s &  # on target
iperf3 -c target-ip -t 86400 &  # from host

# Memory pressure
stress-ng --vm 2 --vm-bytes 80% --timeout 86400 &

Run cyclictest with all stressors active. The maximum is the largest observed scheduling delay in that run. Unobserved loads, firmware paths and hardware states can produce larger delays.

Common latency sources and fixes

CPU frequency and idle policy

Correlate spikes with frequency transitions and idle residency before changing policy. Compare available governors on a disposable target and record the thermal and power effects. A performance governor is one candidate configuration, not a universal correction.

Firmware and secure-world activity

On x86, firmware handling of System Management Interrupts can delay kernel scheduling. Other architectures have different firmware paths, including secure-world calls. Use platform tracing and vendor documentation to investigate. Do not disable firmware functions without understanding their thermal, reliability and security roles.

USB and GPU drivers

Correlate traces with USB traffic and display activity. Before moving IRQ priorities or affinity, map which threads the application depends on. Lowering a necessary interrupt thread can delay the work you intended to protect. Repeat the application and recovery tests after a change.

Memory allocation

Pre-allocate and touch the required memory before entering the measured loop. Check that memory locking succeeds under the configured limits, then verify the application avoids allocations and unwanted faults during the deadline-sensitive path. The following fragment shows the failure check; it is not a complete real-time application.

#include <sys/mman.h>
#include <stdio.h>

int main() {
    if (mlockall(MCL_CURRENT | MCL_FUTURE) != 0) {
        perror("mlockall");
        return 1;
    }
    // Pre-allocate buffers
    // Enter real-time loop
}

CPU isolation

CPU isolation can reduce interference from unrelated work. It does not remove every interrupt or firmware path. The following boot arguments are illustrative; verify supported CPU indices, kernel configuration and housekeeping capacity on the target:

# Kernel command line
isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3

This removes CPUs 2 and 3 from the general scheduler. Pin your real-time application to these cores using taskset or pthread_setaffinity_np. The isolated cores handle only your real-time threads and their explicitly assigned IRQs.

Documenting latency coverage

Measuring worst-case latency is necessary but not sufficient for safety-critical systems. Record the test coverage as follows:

  1. Define the requirement: "The control loop must complete within 100 µs of the trigger event, 100% of the time"
  2. Measure under worst-case conditions: Choose duration and stress workloads from the product risk and usage profile, then retain the logs
  3. Add margin: A chosen engineering margin is an assumption to validate. Adding a margin to an observed maximum does not establish an upper bound.
  4. Document the test conditions: Hardware, kernel version, RT config, stress workloads, test duration, ambient temperature
  5. Repeat after any kernel or hardware change

Certification and assurance depend on the complete product, applicable standard and assessment evidence. A cyclictest run measures scheduling behaviour. Application execution and end-to-end I/O deadlines require separate measurements.

Strengthen your OS fundamentals

Real-time tuning relies heavily on understanding how the kernel schedules tasks, handles interrupts, and manages memory. If any of these concepts feel unfamiliar, the Fundamentals of Operating Systems course builds that foundation from the ground up.

When PREEMPT_RT is not enough

If timing and isolation requirements exceed the evidence for the tested Linux configuration, evaluate:

  • A dedicated RTOS co-processor (see RTOS vs embedded Linux)
  • Xenomai or RTAI for hard real-time with Linux as a secondary OS
  • FPGA-based control paths with timing verified against the implementation

Record the application deadline, allowed failure behaviour and hardware validation still needed before selecting an architecture.

Integration with ProteanOS

Check the kernel configuration supplied for your target. Use the installation guide for project context and retain your own kernel and application test records. Teams accelerating kernel and embedded C development with AI assistance can find a practical survey of the best coding tools for this kind of systems-level work.

Summary

PREEMPT_RT in 2026 is a mature, mainline technology. Configure it, isolate cores, lock memory, measure under stress, and document your results. Select the architecture using end-to-end application deadlines, fault handling and maintained board support.