From ad9e0782f07707ea9cd7030779c725f4899e8ecb Mon Sep 17 00:00:00 2001 From: Liam Girdwood Date: Tue, 10 Mar 2026 14:02:29 +0000 Subject: [PATCH 1/7] readme: Add README.md with scheduler architecture. Add some information about the different schedulers. Signed-off-by: Liam Girdwood --- src/schedule/README.md | 169 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 src/schedule/README.md diff --git a/src/schedule/README.md b/src/schedule/README.md new file mode 100644 index 000000000000..764b6c08ac5c --- /dev/null +++ b/src/schedule/README.md @@ -0,0 +1,169 @@ +# SOF Scheduling Architecture + +This directory (`src/schedule`) contains the Sound Open Firmware (SOF) scheduling infrastructure, deeply integrated with the underlying Zephyr RTOS. SOF utilizes a multi-tiered scheduling approach to cater to different real-time constraints, ranging from hard real-time, low-latency requirements to more relaxed, compute-intensive data processing tasks. + +## Overview of Schedulers + +SOF categorizes tasks and assigns them to specialized schedulers: + +1. **LL (Low Latency) Scheduler**: For tasks that require strict, predictable, and lowest possible latency bounded to hardware events (Timers, DMA interrupts). +2. **DP (Data Processing) Scheduler**: For compute-intensive components that process large chunks of data and operate on deadlines rather than strict cycles. +3. **TWB (Thread With Budget) Scheduler**: For tasks that are allotted a specific execution "budget" (in CPU cycles) per tick to prevent CPU starvation. + +Below is a high-level component interaction architecture of the SOF scheduling domains on top of Zephyr. + +```mermaid +graph TD + subgraph Zephyr RTOS + Timer[Hardware Timer] + DMA[DMA Controller] + Threads[Zephyr Threads] + end + + subgraph Generic Scheduler API + API[schedule.c API] + end + + subgraph LL Scheduler Domain + LL[zephyr_ll.c] + LLDomain[zephyr_domain.c] + DMADomain[zephyr_dma_domain.c] + end + + subgraph DP Scheduler Domain + DP[zephyr_dp_schedule.c] + DPThread[zephyr_dp_schedule_thread.c] + end + + subgraph TWB Scheduler Domain + TWB[zephyr_twb_schedule.c] + end + + API --> LL + API --> DP + API --> TWB + + Timer -.->|Interrupt| LLDomain + DMA -.->|Interrupt| DMADomain + + LLDomain --> |Wakeup| LL + DMADomain --> |Wakeup| LL + LL -->|Runs tasks| Threads + + LL -->|NOTIFIER_ID_LL_POST_RUN| DP + DP -->|Recalculate Deadlines| DPThread + DPThread -->|Update Thread Deadlines| Threads + + LL -->|LL Tick Source| TWB + TWB -->|Update Time Slices| Threads +``` + +--- + +## 1. LL (Low Latency) Scheduler + +The LL scheduler (`zephyr_ll.c`) is designed for extreme low-latency processing. It bypasses complex generic Zephyr scheduling for its internal tasks to minimize overhead, executing a list of registered SOF tasks in a strict priority order. + +### Architecture + +- **Domain Threads**: The LL scheduler runs within a dedicated high-priority Zephyr thread (`ll_thread0`, etc.) pinned to each core (`zephyr_domain.c`). +- **Triggers**: It is woken up by a hardware timer (e.g., a 1ms tick) or directly by hardware DMA interrupts (`zephyr_dma_domain.c`). +- **Execution**: Once woken up, it locks the domain, iterates through all `QUEUED` tasks ordered by priority, moves them to a temporary list, and calls their `.run()` functions. +- **Post-Run**: After all tasks execute, it triggers a `NOTIFIER_ID_LL_POST_RUN` event. This event cascades to wake up other dependent schedulers like DP and TWB. + +### Task State Diagram + +```mermaid +stateDiagram-v2 + [*] --> INIT: task_init + INIT --> QUEUED: schedule_task + QUEUED --> RUNNING: zephyr_ll_run (Timer/DMA Tick) + RUNNING --> QUEUED: return RESCHEDULE + RUNNING --> COMPLETED: return COMPLETED + RUNNING --> CANCEL: task_cancel + + QUEUED --> CANCEL: task_cancel + CANCEL --> FREE: task_free + COMPLETED --> FREE: task_free + + FREE --> [*] +``` + +*(Note: State transitions handle Zephyr SMP locking to ensure a task is safely dequeued before state shifts)* + +--- + +## 2. DP (Data Processing) Scheduler + +The DP scheduler (`zephyr_dp_schedule.c`) manages asynchronous, compute-heavy tasks that process data when enough input is available and sufficient output space is free. It effectively relies on Zephyr's EDF (Earliest Deadline First) or standard preemptive scheduling capabilities. + +### Architecture + +- **Separate Threads**: Unlike LL which multiplexes tasks inside a single thread, **each DP task is assigned its own Zephyr thread**. +- **Wakeup Mechanism**: DP scheduling is evaluated at the end of each LL tick (`scheduler_dp_recalculate()`). +- **Readiness**: It checks if a component has sufficient data across its sinks and sources. If so, it transitions to `RUNNING` and signals the individual DP thread via a Zephyr Event object. +- **Deadlines**: Once ready, the DP thread computes its deadline absolute timestamp (`module_get_deadline()`) and calls `k_thread_absolute_deadline_set()`, submitting to the Zephyr kernel's EDF scheduler. + +### Task State Diagram + +```mermaid +stateDiagram-v2 + [*] --> INIT: task_init + INIT --> QUEUED: schedule_task + + note right of QUEUED + Wait for LL POST RUN event + to evaluate Readiness. + end note + + QUEUED --> RUNNING: resources ready (set priority/deadline) + RUNNING --> QUEUED: return RESCHEDULE (processed chunk) + RUNNING --> COMPLETED: return COMPLETED + RUNNING --> CANCEL: task_cancel + + QUEUED --> CANCEL: task_cancel + COMPLETED --> FREE: task_free + CANCEL --> FREE: task_free + + FREE --> [*] +``` + +--- + +## 3. TWB (Thread With Budget) Scheduler + +The TWB scheduler (`zephyr_twb_schedule.c`) provides execution budget limits for specific tasks to prevent them from starving the CPU. This is useful for intensive workloads that shouldn't disrupt the overall systemic low-latency chain. + +### Architecture + +- **Separate Threads**: Similar to DP, each TWB task executes in its own Zephyr thread. +- **Time Slicing**: When scheduled, the thread's granted cycle budget is translated to OS ticks (`k_thread_time_slice_set()`). +- **Budget Exhaustion**: If the thread consumes its budget before completing its work for the tick, a callback (`scheduler_twb_task_cb()`) is invoked by the Zephyr kernel. This callback immediately drops the thread's priority to a background level (`CONFIG_TWB_THREAD_LOW_PRIORITY`), preventing starvation of other threads. +- **Replenishment**: On the next LL tick (`scheduler_twb_ll_tick()`), the consumed cycles are reset, and the thread's original priority and time slice are restored, granting it a fresh budget. + +### Task State Diagram + +```mermaid +stateDiagram-v2 + [*] --> INIT: task_init + INIT --> RUNNING: schedule_task (Thread Created) + + state RUNNING { + [*] --> HighPriority: Replete Budget + HighPriority --> LowPriority: Budget Exhausted (Callback) + LowPriority --> HighPriority: Next LL Tick + } + + RUNNING --> QUEUED: return RESCHEDULE + QUEUED --> RUNNING: Next LL Tick (Restore Priority) + + RUNNING --> CANCEL: task_cancel + QUEUED --> CANCEL: task_cancel + + RUNNING --> COMPLETED: return COMPLETED + + CANCEL --> FREE: task_free + COMPLETED --> FREE: task_free + + FREE --> [*] +``` From 829b436ec6d5e8fa1fcf87ec029dc0c97f0e8aee Mon Sep 17 00:00:00 2001 From: Liam Girdwood Date: Tue, 10 Mar 2026 18:45:03 +0000 Subject: [PATCH 2/7] Update src/schedule/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Liam Girdwood --- src/schedule/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/schedule/README.md b/src/schedule/README.md index 764b6c08ac5c..c19faea288ca 100644 --- a/src/schedule/README.md +++ b/src/schedule/README.md @@ -137,9 +137,9 @@ The TWB scheduler (`zephyr_twb_schedule.c`) provides execution budget limits for ### Architecture - **Separate Threads**: Similar to DP, each TWB task executes in its own Zephyr thread. -- **Time Slicing**: When scheduled, the thread's granted cycle budget is translated to OS ticks (`k_thread_time_slice_set()`). -- **Budget Exhaustion**: If the thread consumes its budget before completing its work for the tick, a callback (`scheduler_twb_task_cb()`) is invoked by the Zephyr kernel. This callback immediately drops the thread's priority to a background level (`CONFIG_TWB_THREAD_LOW_PRIORITY`), preventing starvation of other threads. -- **Replenishment**: On the next LL tick (`scheduler_twb_ll_tick()`), the consumed cycles are reset, and the thread's original priority and time slice are restored, granting it a fresh budget. +- **Time Slicing**: When scheduled, the thread's execution budget is configured in OS ticks via `k_thread_time_slice_set()`. This tick-based budget is internally converted to hardware cycles for accounting against the CPU cycles actually consumed. +- **Budget Exhaustion**: If the thread consumes its budget (as measured in hardware cycles derived from the tick budget) before completing its work for the tick, a callback (`scheduler_twb_task_cb()`) is invoked by the Zephyr kernel. This callback immediately drops the thread's priority to a background level (`CONFIG_TWB_THREAD_LOW_PRIORITY`), preventing starvation of other threads. +- **Replenishment**: On the next LL tick (`scheduler_twb_ll_tick()`), the consumed hardware cycles are reset, and the thread's original priority and time slice are restored, granting it a fresh tick-based budget. ### Task State Diagram From 9ccb8aba8fdc6741654ea6730220a3091ff30685 Mon Sep 17 00:00:00 2001 From: Liam Girdwood Date: Tue, 10 Mar 2026 18:45:28 +0000 Subject: [PATCH 3/7] Update src/schedule/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Liam Girdwood --- src/schedule/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/schedule/README.md b/src/schedule/README.md index c19faea288ca..f7e677447ced 100644 --- a/src/schedule/README.md +++ b/src/schedule/README.md @@ -149,7 +149,7 @@ stateDiagram-v2 INIT --> RUNNING: schedule_task (Thread Created) state RUNNING { - [*] --> HighPriority: Replete Budget + [*] --> HighPriority: Budget replenished HighPriority --> LowPriority: Budget Exhausted (Callback) LowPriority --> HighPriority: Next LL Tick } From 2718d07d21d7e08949e46c393d6b37dcaf26f9d1 Mon Sep 17 00:00:00 2001 From: Liam Girdwood Date: Tue, 10 Mar 2026 18:45:37 +0000 Subject: [PATCH 4/7] Update src/schedule/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Liam Girdwood --- src/schedule/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/schedule/README.md b/src/schedule/README.md index f7e677447ced..8ed2276e785e 100644 --- a/src/schedule/README.md +++ b/src/schedule/README.md @@ -6,7 +6,7 @@ This directory (`src/schedule`) contains the Sound Open Firmware (SOF) schedulin SOF categorizes tasks and assigns them to specialized schedulers: -1. **LL (Low Latency) Scheduler**: For tasks that require strict, predictable, and lowest possible latency bounded to hardware events (Timers, DMA interrupts). +1. **LL (Low Latency) Scheduler**: For tasks that require strict, predictable, and lowest possible latency bound to hardware events (Timers, DMA interrupts). 2. **DP (Data Processing) Scheduler**: For compute-intensive components that process large chunks of data and operate on deadlines rather than strict cycles. 3. **TWB (Thread With Budget) Scheduler**: For tasks that are allotted a specific execution "budget" (in CPU cycles) per tick to prevent CPU starvation. From c5ac0290510e2414b96c086de0386255e548cccf Mon Sep 17 00:00:00 2001 From: Liam Girdwood Date: Tue, 10 Mar 2026 18:45:55 +0000 Subject: [PATCH 5/7] Update src/schedule/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Liam Girdwood --- src/schedule/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/schedule/README.md b/src/schedule/README.md index 8ed2276e785e..7923ad4d6d70 100644 --- a/src/schedule/README.md +++ b/src/schedule/README.md @@ -68,7 +68,7 @@ The LL scheduler (`zephyr_ll.c`) is designed for extreme low-latency processing. - **Domain Threads**: The LL scheduler runs within a dedicated high-priority Zephyr thread (`ll_thread0`, etc.) pinned to each core (`zephyr_domain.c`). - **Triggers**: It is woken up by a hardware timer (e.g., a 1ms tick) or directly by hardware DMA interrupts (`zephyr_dma_domain.c`). -- **Execution**: Once woken up, it locks the domain, iterates through all `QUEUED` tasks ordered by priority, moves them to a temporary list, and calls their `.run()` functions. +- **Execution**: Once woken up, it locks the domain, iterates through all scheduled tasks in priority order, moves them to a temporary list, and calls their `.run()` functions. - **Post-Run**: After all tasks execute, it triggers a `NOTIFIER_ID_LL_POST_RUN` event. This event cascades to wake up other dependent schedulers like DP and TWB. ### Task State Diagram From d4a2ebe03cc65ec00ea955da8600b2ce40489dbb Mon Sep 17 00:00:00 2001 From: Liam Girdwood Date: Tue, 10 Mar 2026 18:46:31 +0000 Subject: [PATCH 6/7] Update src/schedule/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Liam Girdwood --- src/schedule/README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/schedule/README.md b/src/schedule/README.md index 7923ad4d6d70..ddd2e8c641a3 100644 --- a/src/schedule/README.md +++ b/src/schedule/README.md @@ -78,13 +78,12 @@ stateDiagram-v2 [*] --> INIT: task_init INIT --> QUEUED: schedule_task QUEUED --> RUNNING: zephyr_ll_run (Timer/DMA Tick) - RUNNING --> QUEUED: return RESCHEDULE - RUNNING --> COMPLETED: return COMPLETED + RUNNING --> RUNNING: return RESCHEDULE + RUNNING --> FREE: return COMPLETED RUNNING --> CANCEL: task_cancel QUEUED --> CANCEL: task_cancel CANCEL --> FREE: task_free - COMPLETED --> FREE: task_free FREE --> [*] ``` From 8a8da0065eeb91d7b3602af5bcc19ccfed376599 Mon Sep 17 00:00:00 2001 From: Liam Girdwood Date: Tue, 10 Mar 2026 18:46:50 +0000 Subject: [PATCH 7/7] Update src/schedule/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Liam Girdwood --- src/schedule/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/schedule/README.md b/src/schedule/README.md index ddd2e8c641a3..fd6b8f27c46c 100644 --- a/src/schedule/README.md +++ b/src/schedule/README.md @@ -8,7 +8,7 @@ SOF categorizes tasks and assigns them to specialized schedulers: 1. **LL (Low Latency) Scheduler**: For tasks that require strict, predictable, and lowest possible latency bound to hardware events (Timers, DMA interrupts). 2. **DP (Data Processing) Scheduler**: For compute-intensive components that process large chunks of data and operate on deadlines rather than strict cycles. -3. **TWB (Thread With Budget) Scheduler**: For tasks that are allotted a specific execution "budget" (in CPU cycles) per tick to prevent CPU starvation. +3. **TWB (Thread With Budget) Scheduler**: For tasks that are allotted a specific execution "budget" per scheduler tick, expressed in Zephyr time-slice ticks (e.g. derived from `ZEPHYR_TWB_BUDGET_MAX` in OS ticks), which the runtime then uses to limit and account for CPU cycles to prevent starvation. Below is a high-level component interaction architecture of the SOF scheduling domains on top of Zephyr.