On a network with two validators, monitoring is not observability — it is the difference between noticing finality stopped and finding out from a user. This post is the monitoring baseline EmpoorioChain asks of every validator, and the alerts that matter.
Per node (Prometheus, port 9615, monitoring network only)
Substrate's standard exporter provides peer count, best block, finalized block, block import and authoring timings, transaction pool size, database and memory metrics. The operator guide's minimum set:
- process running
- peer count
- best block and finalized block — and the lag between them
- missed authoring slots
- disk usage, memory pressure
- RPC latency
- reward and slashing events
Finality (network-wide)
empoorio_finality_seconds{quantile="latest"}
empoorio_finality_seconds{quantile="p50"}
empoorio_finality_seconds{quantile="p95"}
empoorio_finality_seconds{quantile="p99"}
empoorio_finalized_head
empoorio_finalized_head_age_seconds
empoorio_finality_exporter_up
The exporter runs alongside the node and Prometheus scrapes it. finalized_head_age_seconds is the single most important number on the network today: on the testnet it should sit around 17 s. If it climbs past a minute, a validator is down or partitioned, and with two validators there is no third to vote around it.
Transaction pool
emp_txpool_* gauges — pending count, DAG width and depth, duplicates, propagation delay p95 — tell an operator whether a stall is in consensus or in the network layer.
Dashboard
deploy/prometheus-grafana/dashboards/validator-decentralization.json: validators active (from chain state, not from a file), peer count and block heights per node, best–finalized lag, uptime per validator over 7 days, client version per node, RPC error rate. It is the visual form of the decentralization gate, and its uptime panel is what decides whether an operator counts as meeting the ≥ 99 % condition.
What pages
| Condition | Why |
|---|---|
empoorio_finalized_head_age_seconds > 60 | Finality stalled; on two validators this means one is gone |
empoorio_finality_exporter_up = 0 | You are blind, not fine |
| Peer count = 0 on a validator | Genesis mismatch or network isolation — the node may be authoring alone |
| Missed slots rising | The authoring key or the host is struggling |
| Best–finalized lag growing for 10 min | GRANDPA behind; check both validators |
| Disk > 85 % | A full disk kills a node quietly |
| Client version differs between validators | Consensus risk; align before the next upgrade |
What is not monitored yet
Public RPC uptime over 30 days (a mainnet-gate item); stake and operator concentration (waits on the indexer); block-weight utilisation. Each is listed in the decentralization report as pending rather than left off.
The audit script
python3 scripts/testnet_validator_audit.py --rpc https://rpc.testnet.empooriochain.org --rpc https://rpc2.testnet.empooriochain.org
Reads Session::Validators from both nodes, compares with the inventory, exits non-zero on disagreement or silence. Run weekly and before every significant runtime upgrade; its JSON output is the evidence behind the published validator count.
Based on VALIDATOR_GUIDE.md, CONSENSUS_STATUS.md, MEMPOOL_DAG_ORDERING.md and DECENTRALIZATION_REPORT.md.