Skip to content

[WIP] MetricsV2 + OTel+Prometheus sample and Grafana dashboard#3154

Draft
csviri wants to merge 49 commits intooperator-framework:nextfrom
csviri:otel-metrics-grafana
Draft

[WIP] MetricsV2 + OTel+Prometheus sample and Grafana dashboard#3154
csviri wants to merge 49 commits intooperator-framework:nextfrom
csviri:otel-metrics-grafana

Conversation

@csviri
Copy link
Collaborator

@csviri csviri commented Feb 4, 2026 •
edited
Loading

Goal of this PR is to provide a OTel + Prometheus + Grafana setup. So we:

  1. verify integration with OTel
  2. Provide a default Grafana dashboard for metrics.
  3. Changes webpage sample to use Otel and deploy the app and dasboard to a cluster with OTel + Prometheus + Grafana. This should be easy to done by a users, so dey can check and validate the dashboard.
  4. new metrics implementations

Notes on new metrics implementation:

  • only static gauges (lives until the operator lives) - that makes much easir to manage them
  • no manual counter removal (should not be done)
  • sanitized names and labels according to best practices
  • added some metrics counter for last attempt retries.

csviri and others added 17 commits February 4, 2026 15:30
@csviri
improve: add license headers to source files (operator-framework#2980)
9942882 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
chore: version to 5.3.0-SNAPSHOT
aef4730 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@shawkins @csviri
Annotation removal using locking (operator-framework#3015)
3aa3cbb 
Signed-off-by: Steve Hawkins <shawkins@redhat.com>
@csviri
improve: complete comparable resource version configs (operator-frame…
a4949d0 
…work#3027)

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
improve: run pr-s checks for v5.3 (operator-framework#3042)
22442ab 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
fix: rebase on main after release
658423b 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
fix(javadoc): invalid method ref blocks snapshot release (operator-fr…
d09a575 
…amework#3076)

simplified the javadoc

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@metacosm @csviri
feat: record desired state in Context (operator-framework#3082)
5eb8d10 
Signed-off-by: Chris Laprun <metacosm@gmail.com>
@csviri
improve: rename junit5 module to junit (operator-framework#3081)
379e727 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
fix: delete empty files result of rebase on main (operator-framework#…
efc2e8f 
…3093)

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri @shawkins
feat: ReconcileUtils for strongly consistent updates (operator-framew…
700d642 
…ork#3106)

- Adds utility that provides methods to update resources using comparable resource versions
- Integrates this utility to the core of the framework (thus simplifying `ReconciliationDispatcher`)
  -  note that this introduces a change in the behavior for the `UpdateControl.patchStatus` (and others), since it won't trigger the reconiliation for the event in that update.
-  Renames former `ReconcilerUtils` to `ReconcilerUtilsInternal`, this is breaking but that utils was never advertised for non-internal usage 
- Includes also fixes for ControllerEventSource event filtering
- Improves TemporaryResourceCache event filtering algorithm

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
Signed-off-by: Steve Hawkins <shawkins@redhat.com>
Co-authored-by: Steve Hawkins <shawkins@redhat.com>
@csviri
Event filtering now records resource action and previous resource (op…
fdc210d 
…erator-framework#3127)

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
improve: facelift samples to use ReconcileUtils (operator-framework#3135
03073ab 
)

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
improve: move compare resource version methods to internal utils (ope…
e17a739 
…rator-framework#3137)

This should not be user facing. At least not in any obvious scenerio.

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri @metacosm
feat: move ReconcileUtils methods to ResourceOperations accessible fr…
f3caf00 
…om Context (operator-framework#3142)

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
Signed-off-by: Chris Laprun <metacosm@gmail.com>
Co-authored-by: Chris Laprun <metacosm@gmail.com>
@csviri
improve: KubernetesDependentResource uses resource operations directly (
258d975 
operator-framework#3146)

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@metacosm @csviri
feat: provide de-duplicated secondary resources stream on Context (op…
4f1d011 
…erator-framework#3141)

Signed-off-by: Chris Laprun <metacosm@gmail.com>
Co-authored-by: Attila Mészáros <a_meszaros@apple.com>
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 4, 2026
This was linked to issues Feb 4, 2026
Provide a default Grafana dashboard for the default metrics #2975
Open
Investigate integration possibilities with Opentelemetry #2135
Open
@csviri csviri changed the title OTel+Prometheus sample and Graphana dashboard [WIP] OTel+Prometheus sample and Graphana dashboard Feb 4, 2026
metacosm and others added 3 commits February 5, 2026 16:20
@metacosm
refactor: avoid creating intermediate collections when unneeded (oper…
8fd7ded 
…ator-framework#3156)

* refactor: avoid creating intermediate collections when unneeded

Also use constant filters and collectors

Signed-off-by: Chris Laprun <metacosm@gmail.com>

* fix: incorrect test

Signed-off-by: Chris Laprun <metacosm@gmail.com>

---------

Signed-off-by: Chris Laprun <metacosm@gmail.com>
@csviri
improve: event filtering algorithm for multiple parallel updates (ope…
a182791 
…rator-framework#3155)

Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@metacosm
improve: prepare for removal of exitOnStopLeading from public API (op…
280b162 
…erator-framework#3161)

* improve: prepare for removal of exitOnStopLeading from public API

Signed-off-by: Chris Laprun <metacosm@gmail.com>

* fix: missed deprecated annotation

[skip ci]

Signed-off-by: Chris Laprun <metacosm@gmail.com>

---------

Signed-off-by: Chris Laprun <metacosm@gmail.com>
@csviri csviri changed the title [WIP] OTel+Prometheus sample and Graphana dashboard [WIP] OTel+Prometheus sample and Grfana dashboard Feb 8, 2026
@csviri csviri changed the title [WIP] OTel+Prometheus sample and Grfana dashboard [WIP] OTel+Prometheus sample and Grafana dashboard Feb 8, 2026
csviri added 4 commits February 8, 2026 16:25
@csviri
feat: observability with otel and default grafana dashboard
586728d 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
c6a688c 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
afab176 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
b0af1e2 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
csviri added 7 commits February 9, 2026 09:27
@csviri
wip
899e345 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
ff05901 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
f7e2565 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
98f200f 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
77307e2 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
1daab47 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
cefad78 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
Copy link
Collaborator Author

csviri commented Feb 9, 2026

JVM metrics:

image image

@csviri
Copy link
Collaborator Author

csviri commented Feb 9, 2026

JODSK metrics:

image

Added TODOs to improve those, like we should not have controller name as suffix, rather as a tag:

operator_sdk_reconciliations_executions_webpagestandalonedependentsreconciler

@csviri
wip
b40766e 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri csviri mentioned this pull request Feb 9, 2026
Open
@csviri csviri linked an issue Feb 10, 2026 that may be closed by this pull request
Micrometer metrics enhancement requests #2974
Open
@csviri
improve: micrometer metrics improvements
6efffd6 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
4f38ca9 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri csviri mentioned this pull request Feb 10, 2026
Closed
@csviri csviri requested review from metacosm and xstefank February 10, 2026 11:50
csviri added 10 commits February 10, 2026 14:17
@csviri
wip
87fd299 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
98b260a 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
a0b9803 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
5a89db6 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
5c662dd 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
c0bed14 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
19b68cb 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
a419dfb 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
wip
243e9a6 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri
e2e test skeleton
7acaaf4 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
@csviri csviri changed the title [WIP] OTel+Prometheus sample and Grafana dashboard [WIP] MetricsV2 + OTel+Prometheus sample and Grafana dashboard Feb 11, 2026
@csviri
wip
5f446ac 
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xstefank xstefank Awaiting requested review from xstefank

@metacosm metacosm Awaiting requested review from metacosm

Copilot code review Copilot Awaiting requested review from Copilot Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Milestone

No milestone

3 participants

@csviri @shawkins @metacosm