Discussion: Real Time Cores (Preempt-RT, RTAI, Xenomai and others)

[hdiethelm]

After the long discussions in #3873 and #3875, I investigated the real time options a bit more in depth and created a summary.

LinuxCNC has quite a number of available real time cores. Some of them are broken, others work fine. What I found so far:

All userspace variants are configured by --with-realtime=uspace and built if the needed library is installed. During runtime, a variant is detected using the kernel name, version and other hints and selected automatically or also not if your kernel name doesn't match expectations.

Format: {Text shown at start of LinuxCNC}:{My short description}

• User Space: --with-realtime=uspace
  • POSIX non-realtime: Any old kernel or you forgot make setuid
    • Works, but big latency
    • Basically what starts when you don't have any other realtime or it is not properly detected
  • POSIX realtime: PREEMPT_RT
    • Works
    • Probably the most used option
    • Debian stock RT kernel works fine
  • XENOMAI (posix-skin) realtime: Xenomai2: Might be / Xenomai3: Works / Xenomai4: Not compatible
    • Works: I managed to get it running with Xenomai3
    • Probably seldom used due to no up to date compiled kernel package is available
  • LXRT realtime: RTAI Userspace
    • Seams to be broken after 2.8.4. I had no success, 2.8.4 is buster where I don't have a VM. In Bookwork/Trixie I was not able to compile 2.8.4. 2.9.* and master builds but has an error starting.
• Kernel Space: --with-realtime=/usr/realtime-$VERSION
  • Nothing: RTAI Kernelspace
    • Works in 2.9
    • Broken in Master: Ruckig broke RTAI #3875
    • @NTULINUX maintains a kernel which is available in the LinuxCNC Debian repo: https://github.com/NTULINUX/RTAI

RTAI is not maintained any more, last commit ~4 years ago.

To help others wanting to try Xenomai3, I created a repo with all you need to get it running:
https://github.com/hdiethelm/xenomai3-linuxcnc

Xenomai4 has a new API: EVL. I did not found a posix skin. I managed to get the kernel and tools built for anybody interested in creating a Xenomai4 module for LinuxCNC:
https://github.com/hdiethelm/xenomai4-linuxcnc

If there is interest, I could try to create a Xenomai4 userspace module. Looking at the code and what I have read so far, this should be not to complicated.

BTW: Is it fine starting this discussion here or should I move it to the forum?

[hdiethelm]

Benchmarks so far on my CNC PC:
Hardware:

• H110M-A Mainboard
• i5-6500
• AMD Radeon RX 460 2GB (IGPU or Nvidia was crap, so I bought it second hand, way better)

---

Results 1:

Kernel command line: isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll nohz=on nohz_full=2-3 intel_idle.max_cstate=1 cpufreq.off=1

Results with latency-histogram, load is 2x glx gears and watching video on youtube with sound:

• PREEMPT_RT: Idle: 1.1us / Load: 136us
• Xenomai3: Idle: 1.0us / Load: 132us
• Xenomai4: Idle: 1.1us / Load: 143us
• LXRT: Idle: 37us / Load: 99us
• RTAI: Idle: 8.3us / Load: 67us

Caveats:

• I have this 140us latency spikes, about 2 in 3 min testing when I watch youtube with sound and they are in all tests
• LXRT and might be RTAI use CPU 0/1 even if 2/3 are isolated. There is an issue somewhere in the CPU assignement

---

Results 2:

Kernel command line: isolcpus=managed_irq,domain,2-3 irqaffinity=0-1 rcu_nocbs=2-3 rcu_nocb_poll nohz=on nohz_full=2-3 intel_idle.max_cstate=1 cpufreq.off=1 nosoftlockup nowatchdog mce=off

mce (machine check) gave interrupts on the isolated CPU's. However, the latency's did not change a lot with this change, with or without load.

The 140us spikes seam to come from the GPU and watching youtube is not really needed for CNC work... ;-) So I used stress-ng.

Results with latency-histogram, load is 2x glx gears and stress-ng --cpu 2 --iomix 2 --vm 2 --vm-bytes 128M --fork 2:

• PREEMPT_RT: Idle: 1.7us / Load: 2.9us
• Xenomai3: Idle: 1.1us / Load: 12us
• Xenomai4: Idle: 1.2us / Load: 1.9us
• LXRT: Idle: 28us / Load: 24us
• RTAI: Idle: 6.9us / Load: 13us

So at least for my PC, PREEMPT_RT and Xenomai4 are the best, followed by Xenomai3.

RTAI/LXRT perform way worse. The new isolcpus syntax is not recognized. As soon as I use isolcpus=2,3 the system gets unstable and crashes often on load. So I forced the thread to CPU 3, so there is at leas no interrupt noise.

Loading hm2_eth, needed for my machine, i use a ethernet mesa card:

• PREEMPT_RT: Works out of the box, latency ~25us
• Xenomai3/Xenomai4: Works out of the box
  • However, for RTnet, additional work and an e1000e ethernet card is required
  • Without RTnet, the normal kernel processes the ethernet workload but the latency is still fine (~50us)
• RTAI: Not supported
• LXRT: Immediate freeze of the whole PC

[rene-dev]

In my option rt_preempt is the only future. Being in mainline means it will always keep working, and be maintained. And it works with Ethernet/Ethercat hardware.

[NTULINUX]

If there is interest, I could try to create a Xenomai4 userspace module. Looking at the code and what I have read so far, this should be not to complicated.

Sweet! Good luck!

[hdiethelm]

I had to try it anyway... ;-) It doesn't look that bad. configure, automake and that WithRoot() was the biggest issue so far. Now I have proper EVL threads:

linuxcnc:

Note: Using XENOMAI4 EVL realtime
linuxcnc-thread:88560
linuxcnc-thread:88561

evl ps -l

CPU   PID   SCHED   PRIO  ISW     CTXSW     SYS       RWA       STAT     TIMEOUT      %CPU      CPUTIME       WCHAN                 NAME
 15   88560  fifo    98   1       1         1         0          X          -           0.0      0:000.003    (null)                linuxcnc-thread:88560
 15   88561  fifo    97   1       1         1         0          X          -           0.0      0:000.001    (null)                linuxcnc-thread:88561

The harder part is finding anything which uses functions not suited for real time and port that to EVL. And I wont think about HW drivers so far... ;-)

[andypugh]

In my option rt_preempt is the only future. Being in mainline means it will always keep working, and be maintained. And it works with Ethernet/Ethercat hardware.

RTAI and Ethernet is probably not impossible, it just wasn't implemented.
Xenomai? Who knows.

LXRT RTAI should work with Ethernet too, and does have slightly better latency on my test system than preempt-rt.

(But my test system is a freak, less than 1k latency on preempt-rt and on RTAI so who knows what the typical LXRT/RTAI latency is?)

[hdiethelm]

@andypugh You have LXRT RTAI working with LinuxCNC 2.9 / master? For me it fails. Any tricks?

In Xenomai3, network runs out of the box, however, probably not as good as it could. My machine has a Ethernet Mesa card and just works with 2.9.8 / Xenomai3 kernel. However, I did not yet do extensive testing.

Xenomai3 and Xenomai4 are different and bot a bit special concerning syscalls / device drivers / network.
Both: As soon as you call a non-Xenomai part, it switches from the Xenomai to the normal kernel, does the work and switches back. Due to you can have a PREEMT_RT kernel with Xenomai, this should still be fast but not optimal. There is a way of tracing this, I have to look into it.

Xenomai3: It has a set of ported network drivers: https://v3.xenomai.org/rtnet/config/index.html
Xenomai4: A driver has to be "enabled" to work, what ever this is: https://v4.xenomai.org/core/net/index.html#evl-net-port

This presentation explains it: https://www.youtube.com/watch?v=WXKsLTfrOiE 23:40 for Xenomai3 / 27:33 for Xenomai4

[hdiethelm]

Xenomai3 network drivers: https://gitlab.com/Xenomai/xenomai3/xenomai/-/tree/master/kernel/drivers/net/drivers?ref_type=heads
Xenomai4: https://v4.xenomai.org/core/drivers/net/index.html :-D Looking in Git and grep for NET_OOB probably only Intel e1000e and 82575/82576: https://gitlab.com/Xenomai/xenomai4/linux-evl/-/blob/v6.12.y-cip-evl-rebase/drivers/net/ethernet/intel/Kconfig?ref_type=heads

It seams that Xenomai4 is still under heavy construction.

[andypugh]

@andypugh You have LXRT RTAI working with LinuxCNC 2.9 / master?

No, I haven't tried LXRT RTAI since 2022.

However LXRT RTAI has run at least once on 2.9:
https://sourceforge.net/p/emc/mailman/message/37718519/
(note the link to the .png)

[hdiethelm]

@andypugh You have LXRT RTAI working with LinuxCNC 2.9 / master?

No, I haven't tried LXRT RTAI since 2022.

However LXRT RTAI has run at least once on 2.9: https://sourceforge.net/p/emc/mailman/message/37718519/ (note the link to the .png)

I found it: #3879 Seams no one used this for a pretty long time...

[NTULINUX]

You should be able to use Xenomai 4 in a kernel thread and not worry about ethernet drivers, similar as to how it's used with RTAI and a PCI(e) parallel port card.

I found a 5.4 tree that is still active!

https://github.com/oracle/linux-uek/tree/v5.4.17-2136.354.4

edit: bit off more than I could chew trying to build that..

What if we went back to 4.19 but use the cip branch that is currently maintained?

https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git/log/?h=linux-4.19.y-cip

[hdiethelm]

Is there any reason to run in kernel space? Xenomai 3 works well also in user space and I will start test with Xenomai 4 in a few days. So now I think I should have all configs running including RTAI, so I can compare all of them on my CNC PC in the exact same configuration except different kernels.

All information I found was pretty outdated and most just recommend to test your app. I did not find any reference yet that Xenomai3/4 still support kernel mode.

Does it really make sense to setup a new system based on that old kernels? I did my test on 6.12.67 both on
Xenomai3: https://gitlab.com/Xenomai/linux-dovetail/-/tree/v6.12.67-dovetail1?ref_type=tags
Xenomai4: https://gitlab.com/Xenomai/xenomai4/linux-evl/-/tree/v6.12.67-evl2-rebase?ref_type=tags

[NTULINUX]

The whole point of the co-kernel approach, whether it's IPIPE or dovetail is that you isolate tasks at kernel level. People talk about isolcpus being all great and wonderful but on most systems I've seen it doesn't help latency because the spikes come from hardware interrupts, most frequently APIC, even when paired with nohz_full and rcu_nocbs, and in some cases I've actually seen latency higher than if the kernel parameters are left alone. isolcpus=cores here nohz_full=cores here rcu_nocbs=cores here don't help with hardware interrupts if they're happening on the isolated core and there's really no way around that once you start digging into the backtraces of using rtla timerlat top and you can actually see where the spikes come from on PREEMPT_RT, instead of just looking at numbers.

IPIPE (RTAI's backbone) only goes up to 5.4 but there is a currently maintained kernel repository that's based on 4.19 which is why I'm asking, should we start looking into using the 4.19 cip branch since it's actively maintained, or is an older, completely EOL 5.4 kernel better in your opinion?

If implemented correctly, doing RT in kernel space using a dual kernel will always be superior, whether it be Xenomai 4 or RTAI. I personally would never use uspace with either of those though, because if I want uspace, I'd just use plain ole PREEMPT_RT.

I have RTAI for linux-cip-4.19.325-cip130 now. (just locally atm)

[hdiethelm]

Finally, I got the first full testing round done. It took some patience to get all variants running. I tested with this branch where I back ported everything to 2.9.8: https://github.com/hdiethelm/linuxcnc-fork/tree/2.9.8-hdiethelm
The pure xenomai4 uspace code base master is here, but RTAI is not compiling: https://github.com/hdiethelm/linuxcnc-fork/tree/xenomai4

I updated this comment: #3878 (comment)
The results are not yet conclusive. I have to figure out what generates this ~140us spike.

Hacks needed:

• Fix rtai headers

--- rtai_lxrt.h.orig	2026-03-23 21:21:53.462988753 +0100
+++ /usr/include/rtai/asm-x86/rtai_lxrt.h	2026-03-23 21:22:24.946592950 +0100
@@ -167,7 +167,7 @@
         return ret;
 }
 
-#define rtai_iopl()  do { extern int iopl(int); iopl(3); } while (0)
+#define rtai_iopl()  do { extern int iopl(int) __THROW; iopl(3); } while (0)
 
 #endif /* __KERNEL__ */

• Fix module list so kernel finds hard drive to boot:
  • Add sd_mod to /etc/initramfs-tools/modules
  • sudo update-initramfs -u -k 5.4.279-rtai-amd64

[BsAtHome]

There may be additional issues. I discovered possible races between RT/non-RT while looking into the latency-plot/latency-histogram code. The data pulled out of RT (components timedelta and histobins) is not latched and values read may not be of one and the same cycle.
I've been working on a python replacement, but am still battling matplotlib/pyplot. However, my new components where data is latched may possibly be backported.

Additionally, using glxgears on modern machines may no longer have a real effect because drawing is offloaded almost entirely to the GPU on modern machines.