Validator Operations: Hardware, Keys, Monitoring and Incident Response

A validator is production infrastructure holding keys that can author blocks and vote on finality. EmpoorioChain's operator documentation treats it that way — as a security-sensitive service, not a casual process — and this post collects the baseline it asks operators to meet.

Hardware

Sized for a mainnet-class node; the live network today is the public testnet.

The reference machine for runtime weight measurement is separate and smaller — a 4-vCPU, 7 GB EPYC-Milan instance chosen as the signed floor. Weights have not yet been measured on it, because renting the exact machine is still pending; the docs record that as an open item rather than a done one.

Build and binary

Run a pinned binary from a tagged release or a reviewed commit, built after ./scripts/restaurar_sdk_vendorizado.sh restores the vendored SDK patches. Never operate a validator from a locally modified, non-reproducible build.

Keys

Three kinds of keys live on or near a validator:

• the network key (secret_ed25519) — the node's p2p identity, generated once with key generate-node-key, file mode 600;
• the session keys — Aura authoring and GRANDPA voting, generated by author_rotateKeys and registered on chain from the stash;
• the stash/controller accounts — which hold the bonded DMS and should never be on the validator host.

Rotate session keys if the server is reinstalled, if access credentials were shared, if a backup may have leaked, or if monitoring shows unknown sessions. Seeds go nowhere near shell history, public docs or chat. Backups of configuration are encrypted; backups of unencrypted keys do not exist.

Monitoring

Minimum set, exported to Prometheus on port 9615 (monitoring network only):

• process running; peer count; best block; finalized block and lag between them;
• missed authoring slots;
• disk usage and memory pressure;
• RPC latency;
• reward and slashing events.

Finality gets its own alerting: empoorio_finality_seconds at p50/p95/p99 and empoorio_finalized_head_age_seconds. On a two-validator network a finality stall is a page.

Routine checks:

systemctl status empooriochain
journalctl -u empooriochain -n 200 --no-pager
curl -s -H 'content-type: application/json' \
  -d '{"id":1,"jsonrpc":"2.0","method":"system_health","params":[]}' http://127.0.0.1:9944
python3 scripts/testnet_validator_audit.py --rpc http://127.0.0.1:9944

Maintenance

OS security patches current. Binary upgrades in planned windows — runtime upgrades arrive on chain with a timelock (12 hours minimum for normal, 48 for security and economic classes on testnet), so an operator always has notice. Snapshot chain data before risky changes. RPC exposed only when needed, behind a reverse proxy and firewall. Telemetry: there is no telemetry.empooriochain.org (the hostname does not resolve, and the guide dropped it); do not point a private testnet validator at public telemetry by default.

Incident response

If the validator host is compromised:

1. Stop the service.
2. Remove the validator from the active set when possible.
3. Rotate session keys.
4. Move funds controlled by potentially exposed keys.
5. Preserve logs for analysis.
6. Redeploy from clean infrastructure.

Accepting this playbook is one of the six conditions for being counted as an independent operator. The ecosystem has already lived through a host compromise — a cryptominer was found and removed from another project's server in July 2026, and a rootkit on a second one in September — which is why these steps are written as a sequence and not as advice.

Based on VALIDATOR_GUIDE.md (verified 2026-09-05), VALIDATOR_IN_30_MINUTES.md and CONSENSUS_STATUS.md.