Skip to content

USHIFT-6887: Fix microshift-low-latency for RHEL10#6590

Merged
openshift-merge-bot[bot] merged 3 commits intoopenshift:mainfrom
pmtk:low-latency-el10
Apr 30, 2026
Merged

USHIFT-6887: Fix microshift-low-latency for RHEL10#6590
openshift-merge-bot[bot] merged 3 commits intoopenshift:mainfrom
pmtk:low-latency-el10

Conversation

@pmtk
Copy link
Copy Markdown
Member

@pmtk pmtk commented Apr 27, 2026

RHEL10 uses different path for the profiles, so while we're still using RHEL9 for the builders, we need a workaround to make the package usage on both systems

Summary by CodeRabbit

  • Refactor

    • Low-latency profile packaging updated to support RHEL 10+ and older layout variants, with compatibility symlink handling and installing profiles into the appropriate tuned profile directory.
    • TuneD profile lookup expanded to also consider profiles/ subdirectories for improved profile resolution.
  • New Features

    • Added preconfigured EL9/EL10 container images that enable tuned and apply MicroShift low-latency tuning.
    • Added periodic test scenarios that wait for boot activation and run ordered tuned test suites.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 27, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 27, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 27, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

RPM spec makes TuneD profile installation RHEL-version-aware; tuned profile paths and symlink behavior adjusted. New EL9/EL10 bootc containerfiles add microshift-tuned and write MicroShift/tuned configs. New periodic scenario scripts create a VM, wait for ≥2 boots, then run three ordered Robot test suites with exit-on-failure.

Changes

Cohort / File(s) Summary
RPM packaging (low-latency / tuned)
packaging/rpm/microshift.spec
Add %tuned_profile_dir macro keyed on %{?rhel}; install microshift-baseline into %{tuned_profile_dir}; on RHEL<10 create /usr/lib/tuned/profiles and symlink back; update %files low-latency.
Tuned helper
packaging/tuned/microshift-tuned.py
Extend get_profile_path() to search /etc/tuned/profiles/ and /usr/lib/tuned/profiles/ in addition to legacy tuned locations.
EL10 bootc tuned image & scenario
test/image-blueprints-bootc/el10/layer3-periodic/group1/rhel102-bootc-source-tuned.containerfile, test/scenarios-bootc/el10/periodics/el102-src@low-latency.sh
New containerfile enabling microshift-tuned, writing /etc/microshift/* and /etc/tuned/microshift-baseline-variables.conf; new periodic script to create/remove VM, poll for ≥2 boots (60s timeout), then run three Robot suites sequentially with --exitonfailure.
EL9 bootc tuned image & scenario
test/image-blueprints-bootc/el9/layer3-periodic/group1/rhel98-bootc-source-tuned.containerfile, test/scenarios-bootc/el9/periodics/el98-src@low-latency.sh
New containerfile enabling microshift-tuned, writing /etc/microshift/* and tuned variables; new periodic script to create/remove VM, poll for ≥2 boots (60s timeout), then run three Robot suites sequentially with --exitonfailure.

Sequence Diagram(s)

sequenceDiagram
    participant CI as CI/Test Runner
    participant ImageBuilder as Image Builder
    participant VMHost as VM Host
    participant VM as Test VM
    participant Tuned as tuned service
    participant MicroShift as MicroShift

    CI->>ImageBuilder: build tuned bootc image (EL9/EL10) with microshift-tuned
    ImageBuilder-->>CI: tuned image artifact
    CI->>VMHost: create VM from tuned image (host1)
    VMHost->>VM: boot
    VM->>Tuned: apply microshift-baseline profile
    Tuned-->>VM: reboot (profile applied)
    VM->>CI: report boot entries via journalctl
    CI->>CI: wait until >=2 boots (timeout 60s)
    CI->>VM: run tuned Robot Framework suites (--exitonfailure)
    VM-->>CI: test results
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main change: fixing the microshift low-latency configuration for RHEL10 compatibility, which is the core purpose of the PR.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The pull request does not introduce any Ginkgo test definitions. Modifications consist of RPM spec, containerfile, bash scripts, and Python code. Existing Robot test files use static, descriptive test names without dynamic content.
Test Structure And Quality ✅ Passed PR contains no Ginkgo test files; check not applicable to provided changes.
Microshift Test Compatibility ✅ Passed This PR does not add any Ginkgo e2e tests. The changes include RPM spec file modifications, container image build recipes, and bash scenario scripts, not Go tests with Ginkgo patterns.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR adds bash scenario scripts, Containerfiles, RPM spec modifications, and Python helper scripts—no Ginkgo e2e tests added. SNO compatibility check not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed No Kubernetes deployment manifests, operators, or controllers with pod scheduling constraints introduced.
Ote Binary Stdout Contract ✅ Passed PR modifies only RPM specs, Containerfiles, Bash scripts, and Python code—no Go source or OTE binaries affected.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR does not add new Ginkgo e2e tests; changes are RPM specs, Python scripts, container recipes, and bash orchestration scripts for existing robot test suites.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 27, 2026
@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 27, 2026

/test ?

@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 27, 2026

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-el9

1 similar comment
@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 28, 2026

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-el9

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/scenarios-bootc/el10/periodics/el102-src`@low-latency.sh:
- Around line 28-39: The loop that polls boots lacks numeric validation and uses
too-short timeout; make boot_num a local variable (local boot_num), validate its
value with a regex like [[ "${boot_num}" =~ ^[0-9]+$ ]] before numeric
comparison, only treat it as >=2 when it passes validation, and extend the
timeout check from 60 to 180 seconds (use start_time and date +%s as in the
existing logic with the updated timeout value). Use run_command_on_vm, boot_num
and start_time to locate the code to change.

In `@test/scenarios-bootc/el9/periodics/el98-src`@low-latency.sh:
- Around line 28-39: The boot-count polling loop using boot_num from
run_command_on_vm can fail if boot_num is empty or non-numeric; change the loop
in which start_time and boot_num are used to first validate boot_num with a
regex (e.g. ^[0-9]+$) before doing the numeric comparison, treating non-matches
as not-yet-ready and continuing the loop; replace the hardcoded 60s timeout with
a configurable TUNED_REBOOT_WAIT_TIMEOUT (default to 180 if unset) and use that
variable in the elapsed-time check; update references to boot_num, start_time
and the loop that contains run_command_on_vm accordingly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: df014366-557b-48e9-b084-bd46f95e85a5

📥 Commits

Reviewing files that changed from the base of the PR and between 0bfdade and df16594.

📒 Files selected for processing (4)
  • test/image-blueprints-bootc/el10/layer3-periodic/group1/rhel102-bootc-source-tuned.containerfile
  • test/image-blueprints-bootc/el9/layer3-periodic/group1/rhel98-bootc-source-tuned.containerfile
  • test/scenarios-bootc/el10/periodics/el102-src@low-latency.sh
  • test/scenarios-bootc/el9/periodics/el98-src@low-latency.sh

Comment on lines +28 to +39
local -r start_time=$(date +%s)
while true; do
boot_num=$(run_command_on_vm host1 "sudo journalctl --list-boots --quiet | wc -l" || true)
boot_num="${boot_num%$'\r'*}"
if [[ "${boot_num}" -ge 2 ]]; then
break
fi
if [ $(( $(date +%s) - start_time )) -gt 60 ]; then
echo "Timed out waiting for VM having 2 boots"
exit 1
fi
sleep 5
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# Read the specific file and check lines around 28-39
cat -n "test/scenarios-bootc/el10/periodics/el102-src@low-latency.sh" | sed -n '20,45p'

Repository: openshift/microshift

Length of output: 1062


🏁 Script executed:

# Search for similar boot-polling patterns to understand the "hardening" mentioned
rg "boot_num|list-boots" --type sh -B 3 -A 3

Repository: openshift/microshift

Length of output: 10872


🏁 Script executed:

# Search for TUNED_REBOOT_WAIT_TIMEOUT to see if it's used elsewhere
rg "TUNED_REBOOT_WAIT_TIMEOUT" --type sh

Repository: openshift/microshift

Length of output: 46


🏁 Script executed:

# Search for boot_num with regex validation (the hardening pattern)
rg 'boot_num.*=~|boot_num.*\[0-9\]' --type sh -B 2 -A 2

Repository: openshift/microshift

Length of output: 46


🏁 Script executed:

# Search for longer timeout values in similar context
rg 'boot_wait_timeout|TUNED_REBOOT_WAIT_TIMEOUT|boot.*timeout.*18[0-9]|boot.*timeout.*2[0-9]{2}' --type sh -B 3 -A 3

Repository: openshift/microshift

Length of output: 46


🏁 Script executed:

# Look for helper functions or common boot polling patterns
rg 'wait.*boot|boot.*poll' --type sh -i

Repository: openshift/microshift

Length of output: 3821


🏁 Script executed:

# Check if there are any helper functions related to boot polling
rg 'run_command_on_vm' --type sh -A 5 | head -50

Repository: openshift/microshift

Length of output: 4035


🏁 Script executed:

# Look at the test/bin/scenario.sh to see if there's a better pattern
cat test/bin/scenario.sh | head -100

Repository: openshift/microshift

Length of output: 3645


🏁 Script executed:

# Check if there are any recent commits or patterns with numeric validation
rg '\[0-9\]\+|\^\[0-9\]' --type sh -B 2 -A 2 | head -40

Repository: openshift/microshift

Length of output: 3239


Harden boot-polling to prevent CI flakes.

The current code lacks numeric validation for boot_num (can fail with non-numeric output during reboots) and uses a 60-second timeout that's too tight for periodic environments. Add regex validation (^[0-9]+$), increase timeout to 180 seconds, and declare boot_num as local.

Proposed fix
 scenario_run_tests() {
     exit_if_image_not_found "${start_image}"

     # Should not be run immediately after creating VM because of
     # microshift-tuned rebooting the node to activate the profile.
     local -r start_time=$(date +%s)
+    local -r boot_wait_timeout="${TUNED_REBOOT_WAIT_TIMEOUT:-180}"
     while true; do
-        boot_num=$(run_command_on_vm host1 "sudo journalctl --list-boots --quiet | wc -l" || true)
+        local boot_num
+        boot_num=$(run_command_on_vm host1 "sudo journalctl --list-boots --quiet | wc -l" 2>/dev/null || true)
         boot_num="${boot_num%$'\r'*}"
-        if [[ "${boot_num}" -ge 2 ]]; then
+        if [[ "${boot_num}" =~ ^[0-9]+$ ]] && [[ "${boot_num}" -ge 2 ]]; then
             break
         fi
-        if [ $(( $(date +%s) - start_time )) -gt 60 ]; then
+        if [ $(( $(date +%s) - start_time )) -gt "${boot_wait_timeout}" ]; then
             echo "Timed out waiting for VM having 2 boots"
             exit 1
         fi
         sleep 5
     done
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
local -r start_time=$(date +%s)
while true; do
boot_num=$(run_command_on_vm host1 "sudo journalctl --list-boots --quiet | wc -l" || true)
boot_num="${boot_num%$'\r'*}"
if [[ "${boot_num}" -ge 2 ]]; then
break
fi
if [ $(( $(date +%s) - start_time )) -gt 60 ]; then
echo "Timed out waiting for VM having 2 boots"
exit 1
fi
sleep 5
local -r start_time=$(date +%s)
local -r boot_wait_timeout="${TUNED_REBOOT_WAIT_TIMEOUT:-180}"
while true; do
local boot_num
boot_num=$(run_command_on_vm host1 "sudo journalctl --list-boots --quiet | wc -l" 2>/dev/null || true)
boot_num="${boot_num%$'\r'*}"
if [[ "${boot_num}" =~ ^[0-9]+$ ]] && [[ "${boot_num}" -ge 2 ]]; then
break
fi
if [ $(( $(date +%s) - start_time )) -gt "${boot_wait_timeout}" ]; then
echo "Timed out waiting for VM having 2 boots"
exit 1
fi
sleep 5
done
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/scenarios-bootc/el10/periodics/el102-src`@low-latency.sh around lines 28
- 39, The loop that polls boots lacks numeric validation and uses too-short
timeout; make boot_num a local variable (local boot_num), validate its value
with a regex like [[ "${boot_num}" =~ ^[0-9]+$ ]] before numeric comparison,
only treat it as >=2 when it passes validation, and extend the timeout check
from 60 to 180 seconds (use start_time and date +%s as in the existing logic
with the updated timeout value). Use run_command_on_vm, boot_num and start_time
to locate the code to change.

Comment on lines +28 to +39
local -r start_time=$(date +%s)
while true; do
boot_num=$(run_command_on_vm host1 "sudo journalctl --list-boots --quiet | wc -l" || true)
boot_num="${boot_num%$'\r'*}"
if [[ "${boot_num}" -ge 2 ]]; then
break
fi
if [ $(( $(date +%s) - start_time )) -gt 60 ]; then
echo "Timed out waiting for VM having 2 boots"
exit 1
fi
sleep 5
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Verify the current code pattern in both low-latency scenario files.
for f in \
  test/scenarios-bootc/el9/periodics/el98-src@low-latency.sh \
  test/scenarios-bootc/el10/periodics/el102-src@low-latency.sh
do
  echo "== ${f} =="
  rg -n 'boot_num=.*journalctl|"\$\{boot_num\}" -ge 2| -gt 60' "${f}"
done

# Reproduce bash integer-compare behavior with empty value.
bash -lc 'boot_num=""; if [[ "${boot_num}" -ge 2 ]]; then echo "ok"; fi' || true

Repository: openshift/microshift

Length of output: 735


Harden boot-count polling to prevent flaky failures.

The [[ "${boot_num}" -ge 2 ]] comparison fails when boot_num is empty or non-numeric during reboots. The hardcoded 60s timeout is also brittle for CI environments. Add regex validation before numeric comparison and use a configurable timeout via TUNED_REBOOT_WAIT_TIMEOUT environment variable (default 180s).

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/scenarios-bootc/el9/periodics/el98-src`@low-latency.sh around lines 28 -
39, The boot-count polling loop using boot_num from run_command_on_vm can fail
if boot_num is empty or non-numeric; change the loop in which start_time and
boot_num are used to first validate boot_num with a regex (e.g. ^[0-9]+$) before
doing the numeric comparison, treating non-matches as not-yet-ready and
continuing the loop; replace the hardcoded 60s timeout with a configurable
TUNED_REBOOT_WAIT_TIMEOUT (default to 180 if unset) and use that variable in the
elapsed-time check; update references to boot_num, start_time and the loop that
contains run_command_on_vm accordingly.

@pmtk pmtk force-pushed the low-latency-el10 branch from df16594 to 5795332 Compare April 28, 2026 09:48
@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 28, 2026

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-el9

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packaging/rpm/microshift.spec`:
- Around line 19-24: get_profile_path in packaging/tuned/microshift-tuned.py
currently only checks /etc/tuned/{profile} and /usr/lib/tuned/{profile}; update
this function to also check /usr/lib/tuned/profiles/{profile} (the relocated
EL10 profile path) so that when the spec installs profiles to
/usr/lib/tuned/profiles/microshift-baseline the function will find them; modify
the search_paths list/logic inside get_profile_path() to include the new path
before failing.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: e557f24e-6d40-4741-b231-ac3310b6ca1e

📥 Commits

Reviewing files that changed from the base of the PR and between df16594 and 5795332.

📒 Files selected for processing (5)
  • packaging/rpm/microshift.spec
  • test/image-blueprints-bootc/el10/layer3-periodic/group1/rhel102-bootc-source-tuned.containerfile
  • test/image-blueprints-bootc/el9/layer3-periodic/group1/rhel98-bootc-source-tuned.containerfile
  • test/scenarios-bootc/el10/periodics/el102-src@low-latency.sh
  • test/scenarios-bootc/el9/periodics/el98-src@low-latency.sh
✅ Files skipped from review due to trivial changes (1)
  • test/image-blueprints-bootc/el10/layer3-periodic/group1/rhel102-bootc-source-tuned.containerfile
🚧 Files skipped from review as they are similar to previous changes (3)

Comment thread packaging/rpm/microshift.spec
@pmtk pmtk force-pushed the low-latency-el10 branch from 5795332 to 852a45f Compare April 28, 2026 10:04
@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 28, 2026

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-el9

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
packaging/tuned/microshift-tuned.py (1)

121-122: ⚠️ Potential issue | 🔴 Critical

Missing comma makes EL10 profile path unreachable

At Line 121 and Line 122, the two adjacent f-strings are concatenated into one invalid path, so /usr/lib/tuned/profiles/{profile} is never checked.

Proposed fix
     paths = [
         f"/etc/tuned/{profile}",
         f"/etc/tuned/profiles/{profile}",
-        f"/usr/lib/tuned/{profile}"
-        f"/usr/lib/tuned/profiles/{profile}"
+        f"/usr/lib/tuned/{profile}",
+        f"/usr/lib/tuned/profiles/{profile}",
     ]
#!/bin/bash
set -euo pipefail

echo "Inspect get_profile_path list entries:"
rg -n -A10 'def get_profile_path' packaging/tuned/microshift-tuned.py

echo
echo "Verify both /usr/lib/tuned entries are separate comma-terminated list items:"
rg -nP 'f"/usr/lib/tuned/\{profile\}",\s*$' packaging/tuned/microshift-tuned.py
rg -nP 'f"/usr/lib/tuned/profiles/\{profile\}",\s*$' packaging/tuned/microshift-tuned.py
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packaging/tuned/microshift-tuned.py` around lines 121 - 122, In
get_profile_path, two adjacent f-strings for "/usr/lib/tuned/{profile}" and
"/usr/lib/tuned/profiles/{profile}" were accidentally concatenated because a
comma was missing; fix by adding a comma between those two list entries so both
paths are separate list items (locate the f-strings in get_profile_path and
insert the missing comma to ensure "/usr/lib/tuned/profiles/{profile}" is
checked).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@packaging/tuned/microshift-tuned.py`:
- Around line 121-122: In get_profile_path, two adjacent f-strings for
"/usr/lib/tuned/{profile}" and "/usr/lib/tuned/profiles/{profile}" were
accidentally concatenated because a comma was missing; fix by adding a comma
between those two list entries so both paths are separate list items (locate the
f-strings in get_profile_path and insert the missing comma to ensure
"/usr/lib/tuned/profiles/{profile}" is checked).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: fddcba34-60b4-4372-ab58-f49c4a98fb86

📥 Commits

Reviewing files that changed from the base of the PR and between 5795332 and 852a45f.

📒 Files selected for processing (6)
  • packaging/rpm/microshift.spec
  • packaging/tuned/microshift-tuned.py
  • test/image-blueprints-bootc/el10/layer3-periodic/group1/rhel102-bootc-source-tuned.containerfile
  • test/image-blueprints-bootc/el9/layer3-periodic/group1/rhel98-bootc-source-tuned.containerfile
  • test/scenarios-bootc/el10/periodics/el102-src@low-latency.sh
  • test/scenarios-bootc/el9/periodics/el98-src@low-latency.sh
🚧 Files skipped from review as they are similar to previous changes (4)
  • test/image-blueprints-bootc/el9/layer3-periodic/group1/rhel98-bootc-source-tuned.containerfile
  • test/scenarios-bootc/el9/periodics/el98-src@low-latency.sh
  • test/scenarios-bootc/el10/periodics/el102-src@low-latency.sh
  • test/image-blueprints-bootc/el10/layer3-periodic/group1/rhel102-bootc-source-tuned.containerfile

@pmtk pmtk changed the title microshift-low-latency for RHEL10 USHIFT-6887: Fix microshift-low-latency for RHEL10 Apr 28, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Apr 28, 2026
@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented Apr 28, 2026

@pmtk: This pull request references USHIFT-6887 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set.

Details

In response to this:

RHEL10 uses different path for the profiles, so while we're still using RHEL9 for the builders, we need a workaround to make the package usage on both systems

Summary by CodeRabbit

  • Refactor

  • Low-latency profile packaging updated to support RHEL 10+ and older layout variants, including compatibility symlink handling and installing the profile into the appropriate tuned profile directory.

  • TuneD profile lookup expanded to also consider additional system profile directories for improved profile resolution.

  • New Features

  • Added preconfigured container images and EL9/EL10 low-latency periodic scenarios that enable tuned, apply MicroShift tuning, wait for profile activation, and run ordered tuned test suites.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

pmtk added 2 commits April 28, 2026 14:20
RHEL10 uses different path for the profiles, so while we're still using
RHEL9 for the builders, we need a workaround to make the package usage
on both systems
@pmtk pmtk force-pushed the low-latency-el10 branch from 852a45f to 3547a30 Compare April 28, 2026 12:20
@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 28, 2026

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-el9

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
test/scenarios-bootc/el9/periodics/el98-src@low-latency.sh (1)

28-39: ⚠️ Potential issue | 🟠 Major

Harden boot polling against non-numeric output and make timeout configurable.
Line 32 can fail when boot_num is empty/non-numeric during reboot windows, and Line 35’s fixed 60 seconds is brittle in CI.

Proposed patch
-    local -r start_time=$(date +%s)
+    local -r start_time=$(date +%s)
+    local -r reboot_wait_timeout="${TUNED_REBOOT_WAIT_TIMEOUT:-180}"
+    local boot_num
     while true; do
         boot_num=$(run_command_on_vm host1 "sudo journalctl --list-boots --quiet | wc -l" || true)
         boot_num="${boot_num%$'\r'*}"
-        if [[ "${boot_num}" -ge 2 ]]; then
+        if [[ "${boot_num}" =~ ^[0-9]+$ ]] && [[ "${boot_num}" -ge 2 ]]; then
             break
         fi
-        if [ $(( $(date +%s) - start_time )) -gt 60 ]; then
-            echo "Timed out waiting for VM having 2 boots"
+        if [ $(( $(date +%s) - start_time )) -gt "${reboot_wait_timeout}" ]; then
+            echo "Timed out waiting for VM having 2 boots (timeout: ${reboot_wait_timeout}s)"
             exit 1
         fi
         sleep 5
     done
#!/bin/bash
set -euo pipefail

# Verify current pattern in both EL9/EL10 low-latency scenarios.
for f in \
  test/scenarios-bootc/el9/periodics/el98-src@low-latency.sh \
  test/scenarios-bootc/el10/periodics/el102-src@low-latency.sh
do
  echo "== ${f} =="
  rg -n -C2 'boot_num=.*journalctl|"\$\{boot_num\}" -ge 2| -gt 60|TUNED_REBOOT_WAIT_TIMEOUT' "${f}" || true
done

# Reproduce bash behavior with empty value in numeric compare.
bash -lc 'boot_num=""; if [[ "${boot_num}" -ge 2 ]]; then echo "ok"; fi' || true
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/scenarios-bootc/el9/periodics/el98-src`@low-latency.sh around lines 28 -
39, Replace the brittle numeric compare and hard-coded timeout: validate that
boot_num (set by run_command_on_vm host1) is numeric before comparing (e.g., use
a regex like ^[0-9]+$ or bash arithmetic with a safe check) and only treat it as
>=2 when numeric; also make the timeout configurable by introducing a
TUNED_REBOOT_WAIT_TIMEOUT (default 60) used in the elapsed-time check instead of
the hard-coded 60, and keep the existing carriage-return strip for boot_num to
avoid spurious non-numeric characters.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@test/scenarios-bootc/el9/periodics/el98-src`@low-latency.sh:
- Around line 28-39: Replace the brittle numeric compare and hard-coded timeout:
validate that boot_num (set by run_command_on_vm host1) is numeric before
comparing (e.g., use a regex like ^[0-9]+$ or bash arithmetic with a safe check)
and only treat it as >=2 when numeric; also make the timeout configurable by
introducing a TUNED_REBOOT_WAIT_TIMEOUT (default 60) used in the elapsed-time
check instead of the hard-coded 60, and keep the existing carriage-return strip
for boot_num to avoid spurious non-numeric characters.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 6036c903-31aa-43b9-ab87-a1c6f8d955d8

📥 Commits

Reviewing files that changed from the base of the PR and between 852a45f and 3547a30.

📒 Files selected for processing (6)
  • packaging/rpm/microshift.spec
  • packaging/tuned/microshift-tuned.py
  • test/image-blueprints-bootc/el10/layer3-periodic/group1/rhel102-bootc-source-tuned.containerfile
  • test/image-blueprints-bootc/el9/layer3-periodic/group1/rhel98-bootc-source-tuned.containerfile
  • test/scenarios-bootc/el10/periodics/el102-src@low-latency.sh
  • test/scenarios-bootc/el9/periodics/el98-src@low-latency.sh
✅ Files skipped from review due to trivial changes (1)
  • packaging/tuned/microshift-tuned.py
🚧 Files skipped from review as they are similar to previous changes (2)
  • test/image-blueprints-bootc/el10/layer3-periodic/group1/rhel102-bootc-source-tuned.containerfile
  • test/scenarios-bootc/el10/periodics/el102-src@low-latency.sh

@pmtk
Copy link
Copy Markdown
Member Author

pmtk commented Apr 29, 2026

/test e2e-aws-tests-bootc-periodic-el10 e2e-aws-tests-bootc-periodic-el9

@pmtk pmtk force-pushed the low-latency-el10 branch from 2229575 to 5e30311 Compare April 29, 2026 14:19
@pmtk pmtk marked this pull request as ready for review April 29, 2026 14:20
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 29, 2026
@openshift-ci openshift-ci Bot requested review from eslutsky and kasturinarra April 29, 2026 14:24
@agullon
Copy link
Copy Markdown
Contributor

agullon commented Apr 30, 2026

/retest

@agullon
Copy link
Copy Markdown
Contributor

agullon commented Apr 30, 2026

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Apr 30, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 30, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: agullon, pmtk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@agullon
Copy link
Copy Markdown
Contributor

agullon commented Apr 30, 2026

/verified by CI

@agullon
Copy link
Copy Markdown
Contributor

agullon commented Apr 30, 2026

/cherrypick release-4.22

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Apr 30, 2026
@openshift-ci-robot
Copy link
Copy Markdown

@agullon: This PR has been marked as verified by CI.

Details

In response to this:

/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-cherrypick-robot
Copy link
Copy Markdown

@agullon: once the present PR merges, I will cherry-pick it on top of release-4.22 in a new PR and assign it to you.

Details

In response to this:

/cherrypick release-4.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-merge-bot openshift-merge-bot Bot merged commit 3964b46 into openshift:main Apr 30, 2026
14 checks passed
@openshift-cherrypick-robot
Copy link
Copy Markdown

@agullon: new pull request created: #6614

Details

In response to this:

/cherrypick release-4.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants