From a96d0ed311491fac5c15832770b427b879c8d370 Mon Sep 17 00:00:00 2001 From: Blockchains Validator Service <87995279+Bela1401@users.noreply.github.com> Date: Mon, 27 Apr 2026 11:25:37 +0300 Subject: [PATCH 1/4] docs: add single-host monitoring guide Add Prometheus and Grafana setup instructions for a single Arc EL + CL node on one host. --- docs/monitoring.md | 243 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 243 insertions(+) create mode 100644 docs/monitoring.md diff --git a/docs/monitoring.md b/docs/monitoring.md new file mode 100644 index 0000000..759eaff --- /dev/null +++ b/docs/monitoring.md @@ -0,0 +1,243 @@ +# Monitoring an Arc Node + +This guide describes a simple monitoring setup for operators running one Arc +Execution Layer (EL) and one Arc Consensus Layer (CL) on the same Linux host. + +It uses: + +- **Prometheus** for metrics collection; +- **Grafana** for dashboards; +- **SSH port forwarding** for safe access to local-only monitoring endpoints. + +This guide assumes you already have a working Arc node by following +[installation](./installation.md) and +[running an Arc node](./running-an-arc-node.md). + +## Prerequisites + +Before setting up monitoring, confirm that: + +- your Arc node is already running; +- the execution layer is exposing metrics on `127.0.0.1:9001`; +- the consensus layer is exposing metrics on `127.0.0.1:29000`; +- Docker Engine and Docker Compose are installed on the host. + +The examples in this guide assume the following metrics flags are present: + +### Execution layer + +```sh +--metrics 127.0.0.1:9001 +Consensus layer +--metrics 127.0.0.1:29000 +Verify metrics endpoints +Before deploying Prometheus and Grafana, verify that both metrics endpoints are +reachable on the host. + +The execution layer exposes metrics at /: + +curl -s http://127.0.0.1:9001 | head +The consensus layer exposes metrics at /metrics: + +curl -s http://127.0.0.1:29000/metrics | head +If either command fails, first fix the Arc node startup or metrics flags before +continuing. + +Create a monitoring directory +This guide stores monitoring files under $ARC_HOME/monitoring. + +ARC_HOME="${ARC_HOME:-$HOME/.arc}" +ARC_MONITORING="${ARC_MONITORING:-$ARC_HOME/monitoring}" + +mkdir -p "$ARC_MONITORING"/grafana-provisioning/datasources +mkdir -p "$ARC_MONITORING"/grafana-provisioning/dashboards +mkdir -p "$ARC_MONITORING"/prometheus-data +mkdir -p "$ARC_MONITORING"/dashboards +Reuse the existing Arc dashboards +The repository already contains Grafana dashboard JSON files under +deployments/monitoring/config-grafana/provisioning/dashboards-data. + +From the arc-node repository root, copy them into the monitoring directory: + +cp -r deployments/monitoring/config-grafana/provisioning/dashboards-data/* \ + "$ARC_MONITORING"/dashboards/ +Create the Prometheus configuration +Write the following file to $ARC_MONITORING/prometheus.yml: + +global: + scrape_interval: 1s + +scrape_configs: + - job_name: "arc_execution" + metrics_path: "/" + scrape_interval: 1s + static_configs: + - targets: ["127.0.0.1:9001"] + labels: + client_name: "reth" + client_type: "execution" + + - job_name: "arc_consensus" + metrics_path: "/metrics" + scrape_interval: 1s + static_configs: + - targets: ["127.0.0.1:29000"] + labels: + client_name: "malachite" + client_type: "consensus" +Create the Grafana datasource configuration +Write the following file to +$ARC_MONITORING/grafana-provisioning/datasources/prometheus.yml: + +apiVersion: 1 + +datasources: + - name: prometheus + uid: prometheus + type: prometheus + url: http://127.0.0.1:9090 + isDefault: true + editable: true +Create the Grafana dashboards provisioning file +Write the following file to +$ARC_MONITORING/grafana-provisioning/dashboards/default.yml: + +apiVersion: 1 + +providers: + - name: "arc" + orgId: 1 + folder: "" + type: file + disableDeletion: false + editable: true + updateIntervalSeconds: 10 + options: + path: /var/lib/grafana/dashboards +Create the Docker Compose file +Write the following file to $ARC_MONITORING/compose.yaml: + +services: + prometheus: + image: prom/prometheus + user: "0" + network_mode: host + command: + - --config.file=/etc/prometheus/prometheus.yml + - --storage.tsdb.path=/prometheus + - --web.listen-address=127.0.0.1:9090 + volumes: + - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro + - ./prometheus-data:/prometheus + restart: unless-stopped + + grafana: + image: grafana/grafana-oss + network_mode: host + environment: + GF_SERVER_HTTP_ADDR: 127.0.0.1 + GF_SERVER_HTTP_PORT: 3000 + GF_SECURITY_ADMIN_USER: admin + GF_SECURITY_ADMIN_PASSWORD: admin + volumes: + - ./grafana-provisioning:/etc/grafana/provisioning:ro + - ./dashboards:/var/lib/grafana/dashboards:ro + restart: unless-stopped +Why network_mode: host? The Arc metrics endpoints in this guide are bound +to 127.0.0.1, so host networking is the simplest way for Prometheus to scrape +them without changing the Arc node configuration or exposing metrics publicly. + +Start the monitoring stack +cd "$ARC_MONITORING" +docker compose up -d +Check that both containers are running: + +docker compose ps +Verify Prometheus and Grafana +Check that Grafana is healthy: + +curl -s http://127.0.0.1:3000/api/health +Check that Prometheus is ready: + +curl -s http://127.0.0.1:9090/-/ready +You can also inspect Prometheus targets in the browser later at: + +http://localhost:9090/targets +Access the dashboards +This setup binds Grafana and Prometheus to 127.0.0.1 only. +That is intentional: the services are not exposed to the public internet. + +If you are on the same machine, open: + +http://127.0.0.1:3000 for Grafana; +http://127.0.0.1:9090 for Prometheus. +If you are connecting to a remote server, use SSH port forwarding from your +local machine: + +ssh -N \ + -L 3000:127.0.0.1:3000 \ + -L 9090:127.0.0.1:9090 \ + user@YOUR_SERVER_IP +Then open on your local machine: + +http://localhost:3000 +http://localhost:9090 +Grafana default credentials from the Compose file above are: + +username: admin +password: admin +Change them before exposing Grafana through any reverse proxy or shared access setup. + +What to expect +Once Prometheus starts scraping both endpoints and Grafana loads the provisioned +dashboards, you should see metrics for: + +execution layer activity; +consensus layer activity; +block height; +validator-related consensus telemetry; +connected peers and chain progress. +If some panels show no data immediately after startup, wait a minute and refresh. +A newly started Prometheus instance needs a short time to collect enough samples. + +Troubleshooting +Browser shows ERR_CONNECTION_REFUSED +If you are using a remote server, make sure you are opening: + +http://localhost:3000 +http://localhost:9090 +on your local machine, not http://YOUR_SERVER_IP:3000. + +Also confirm that the SSH tunnel is still running. + +Prometheus is restarting with a permissions error +If Prometheus logs include an error about queries.active or write permission +under /prometheus, fix the ownership of the local data directory: + +sudo chown -R 65534:65534 "$ARC_MONITORING"/prometheus-data +docker compose restart prometheus +Grafana is healthy but dashboards show no data +First, confirm Prometheus targets are up: + +curl -s http://127.0.0.1:9090/api/v1/targets +Then re-check the Arc metrics endpoints: + +curl -s http://127.0.0.1:9001 | head +curl -s http://127.0.0.1:29000/metrics | head +If Prometheus can reach the targets but some dashboard panels are still empty, +wait for more samples to accumulate and refresh the dashboard. + +Check Arc node health directly +You can still verify the Arc node independently of Grafana: + +cast block-number --rpc-url http://127.0.0.1:8545 +sudo journalctl -u arc-execution -f +sudo journalctl -u arc-consensus -f +Next steps +This guide keeps monitoring intentionally minimal and local-only. +For more advanced deployments, operators may want to add: + +persistent Grafana storage; +HTTPS and authentication through a reverse proxy; +Alertmanager-based alerting; +system metrics exporters such as node_exporter. From df7556e9c28a83bc2105b46535803c448ad18348 Mon Sep 17 00:00:00 2001 From: Blockchains Validator Service <87995279+Bela1401@users.noreply.github.com> Date: Mon, 27 Apr 2026 11:37:05 +0300 Subject: [PATCH 2/4] docs: link monitoring guide from node operations docs Add a reference to docs/monitoring.md from running-an-arc-node.md. --- docs/running-an-arc-node.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/running-an-arc-node.md b/docs/running-an-arc-node.md index 0a20af6..cf06299 100644 --- a/docs/running-an-arc-node.md +++ b/docs/running-an-arc-node.md @@ -404,6 +404,8 @@ sudo systemctl start arc-execution arc-consensus ### Monitoring +For a Prometheus + Grafana setup on a single host, see [Monitoring an Arc Node](./monitoring.md). + ```sh # Check service status sudo systemctl status arc-execution From 30b7ef063ef04b72094fe325bbfa487788367dbe Mon Sep 17 00:00:00 2001 From: Blockchains Validator Service <87995279+Bela1401@users.noreply.github.com> Date: Mon, 27 Apr 2026 15:10:28 +0300 Subject: [PATCH 3/4] docs: fix markdown code fences in monitoring guide --- docs/monitoring.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/monitoring.md b/docs/monitoring.md index 759eaff..66e2c73 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -28,12 +28,20 @@ The examples in this guide assume the following metrics flags are present: ```sh --metrics 127.0.0.1:9001 -Consensus layer +``` + +### Consensus layer + +```sh --metrics 127.0.0.1:29000 -Verify metrics endpoints +``` + +## Verify metrics endpoints + Before deploying Prometheus and Grafana, verify that both metrics endpoints are reachable on the host. + The execution layer exposes metrics at /: curl -s http://127.0.0.1:9001 | head From 70d24596406472f868135ed75e87af521a2e044f Mon Sep 17 00:00:00 2001 From: Stelios Daveas Date: Mon, 27 Apr 2026 17:58:33 +0000 Subject: [PATCH 4/4] docs: fix markdown fences and heading structure in monitoring guide Add missing code fences (```sh/```yaml) around all shell commands and config blocks. Promote plain-text section titles to proper ## and ### headings. Format inline paths, URLs, and lists as markdown. No content changes. --- docs/monitoring.md | 172 ++++++++++++++++++++++++++++++++------------- 1 file changed, 122 insertions(+), 50 deletions(-) diff --git a/docs/monitoring.md b/docs/monitoring.md index 66e2c73..c0f2c09 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -41,19 +41,26 @@ The examples in this guide assume the following metrics flags are present: Before deploying Prometheus and Grafana, verify that both metrics endpoints are reachable on the host. +The execution layer exposes metrics at `/`: -The execution layer exposes metrics at /: - +```sh curl -s http://127.0.0.1:9001 | head -The consensus layer exposes metrics at /metrics: +``` +The consensus layer exposes metrics at `/metrics`: + +```sh curl -s http://127.0.0.1:29000/metrics | head +``` + If either command fails, first fix the Arc node startup or metrics flags before continuing. -Create a monitoring directory -This guide stores monitoring files under $ARC_HOME/monitoring. +## Create a monitoring directory +This guide stores monitoring files under `$ARC_HOME/monitoring`. + +```sh ARC_HOME="${ARC_HOME:-$HOME/.arc}" ARC_MONITORING="${ARC_MONITORING:-$ARC_HOME/monitoring}" @@ -61,17 +68,25 @@ mkdir -p "$ARC_MONITORING"/grafana-provisioning/datasources mkdir -p "$ARC_MONITORING"/grafana-provisioning/dashboards mkdir -p "$ARC_MONITORING"/prometheus-data mkdir -p "$ARC_MONITORING"/dashboards -Reuse the existing Arc dashboards +``` + +## Reuse the existing Arc dashboards + The repository already contains Grafana dashboard JSON files under -deployments/monitoring/config-grafana/provisioning/dashboards-data. +`deployments/monitoring/config-grafana/provisioning/dashboards-data`. From the arc-node repository root, copy them into the monitoring directory: +```sh cp -r deployments/monitoring/config-grafana/provisioning/dashboards-data/* \ "$ARC_MONITORING"/dashboards/ -Create the Prometheus configuration -Write the following file to $ARC_MONITORING/prometheus.yml: +``` +## Create the Prometheus configuration + +Write the following file to `$ARC_MONITORING/prometheus.yml`: + +```yaml global: scrape_interval: 1s @@ -93,10 +108,14 @@ scrape_configs: labels: client_name: "malachite" client_type: "consensus" -Create the Grafana datasource configuration +``` + +## Create the Grafana datasource configuration + Write the following file to -$ARC_MONITORING/grafana-provisioning/datasources/prometheus.yml: +`$ARC_MONITORING/grafana-provisioning/datasources/prometheus.yml`: +```yaml apiVersion: 1 datasources: @@ -106,10 +125,14 @@ datasources: url: http://127.0.0.1:9090 isDefault: true editable: true -Create the Grafana dashboards provisioning file +``` + +## Create the Grafana dashboards provisioning file + Write the following file to -$ARC_MONITORING/grafana-provisioning/dashboards/default.yml: +`$ARC_MONITORING/grafana-provisioning/dashboards/default.yml`: +```yaml apiVersion: 1 providers: @@ -122,9 +145,13 @@ providers: updateIntervalSeconds: 10 options: path: /var/lib/grafana/dashboards -Create the Docker Compose file -Write the following file to $ARC_MONITORING/compose.yaml: +``` +## Create the Docker Compose file + +Write the following file to `$ARC_MONITORING/compose.yaml`: + +```yaml services: prometheus: image: prom/prometheus @@ -151,101 +178,146 @@ services: - ./grafana-provisioning:/etc/grafana/provisioning:ro - ./dashboards:/var/lib/grafana/dashboards:ro restart: unless-stopped -Why network_mode: host? The Arc metrics endpoints in this guide are bound -to 127.0.0.1, so host networking is the simplest way for Prometheus to scrape -them without changing the Arc node configuration or exposing metrics publicly. +``` + +> **Why `network_mode: host`?** The Arc metrics endpoints in this guide are bound +> to `127.0.0.1`, so host networking is the simplest way for Prometheus to scrape +> them without changing the Arc node configuration or exposing metrics publicly. -Start the monitoring stack +## Start the monitoring stack + +```sh cd "$ARC_MONITORING" docker compose up -d +``` + Check that both containers are running: +```sh docker compose ps -Verify Prometheus and Grafana +``` + +## Verify Prometheus and Grafana + Check that Grafana is healthy: +```sh curl -s http://127.0.0.1:3000/api/health +``` + Check that Prometheus is ready: +```sh curl -s http://127.0.0.1:9090/-/ready +``` + You can also inspect Prometheus targets in the browser later at: -http://localhost:9090/targets -Access the dashboards -This setup binds Grafana and Prometheus to 127.0.0.1 only. +`http://localhost:9090/targets` + +## Access the dashboards + +This setup binds Grafana and Prometheus to `127.0.0.1` only. That is intentional: the services are not exposed to the public internet. If you are on the same machine, open: -http://127.0.0.1:3000 for Grafana; -http://127.0.0.1:9090 for Prometheus. +- `http://127.0.0.1:3000` for Grafana; +- `http://127.0.0.1:9090` for Prometheus. + If you are connecting to a remote server, use SSH port forwarding from your local machine: +```sh ssh -N \ -L 3000:127.0.0.1:3000 \ -L 9090:127.0.0.1:9090 \ user@YOUR_SERVER_IP +``` + Then open on your local machine: -http://localhost:3000 -http://localhost:9090 +- `http://localhost:3000` +- `http://localhost:9090` + Grafana default credentials from the Compose file above are: -username: admin -password: admin +- username: `admin` +- password: `admin` + Change them before exposing Grafana through any reverse proxy or shared access setup. -What to expect +## What to expect + Once Prometheus starts scraping both endpoints and Grafana loads the provisioned dashboards, you should see metrics for: -execution layer activity; -consensus layer activity; -block height; -validator-related consensus telemetry; -connected peers and chain progress. +- execution layer activity; +- consensus layer activity; +- block height; +- validator-related consensus telemetry; +- connected peers and chain progress. + If some panels show no data immediately after startup, wait a minute and refresh. A newly started Prometheus instance needs a short time to collect enough samples. -Troubleshooting -Browser shows ERR_CONNECTION_REFUSED +## Troubleshooting + +### Browser shows ERR_CONNECTION_REFUSED + If you are using a remote server, make sure you are opening: -http://localhost:3000 -http://localhost:9090 -on your local machine, not http://YOUR_SERVER_IP:3000. +- `http://localhost:3000` +- `http://localhost:9090` + +on your local machine, not `http://YOUR_SERVER_IP:3000`. Also confirm that the SSH tunnel is still running. -Prometheus is restarting with a permissions error -If Prometheus logs include an error about queries.active or write permission -under /prometheus, fix the ownership of the local data directory: +### Prometheus is restarting with a permissions error + +If Prometheus logs include an error about `queries.active` or write permission +under `/prometheus`, fix the ownership of the local data directory: +```sh sudo chown -R 65534:65534 "$ARC_MONITORING"/prometheus-data docker compose restart prometheus -Grafana is healthy but dashboards show no data +``` + +### Grafana is healthy but dashboards show no data + First, confirm Prometheus targets are up: +```sh curl -s http://127.0.0.1:9090/api/v1/targets +``` + Then re-check the Arc metrics endpoints: +```sh curl -s http://127.0.0.1:9001 | head curl -s http://127.0.0.1:29000/metrics | head +``` + If Prometheus can reach the targets but some dashboard panels are still empty, wait for more samples to accumulate and refresh the dashboard. -Check Arc node health directly +### Check Arc node health directly + You can still verify the Arc node independently of Grafana: +```sh cast block-number --rpc-url http://127.0.0.1:8545 sudo journalctl -u arc-execution -f sudo journalctl -u arc-consensus -f -Next steps +``` + +## Next steps + This guide keeps monitoring intentionally minimal and local-only. For more advanced deployments, operators may want to add: -persistent Grafana storage; -HTTPS and authentication through a reverse proxy; -Alertmanager-based alerting; -system metrics exporters such as node_exporter. +- persistent Grafana storage; +- HTTPS and authentication through a reverse proxy; +- Alertmanager-based alerting; +- system metrics exporters such as `node_exporter`.