Monad Snapshots: Operator Runbook for Generation, Hosting, and Verification

Restoring a Monad node from official rolling snapshots is a surgical process—precision matters at every step. Always stop the Monad node with systemctl stop monad before any restore to prevent data corruption. Skipping this step risks irrecoverable state inconsistencies.

Before touching any archives, verify available disk space with df -h. Snapshot archives routinely exceed 100GB, and extraction can require even more. Running out of space mid-restore is a common, avoidable failure.

Download both the snapshot and its checksum from https://snapshots.monad.natsai.xyz using wget --continue to ensure resumable transfers, especially over unreliable links. Interrupted downloads are a leading cause of corrupted restores.

Strictly verify archive integrity with sha256sum -c before extraction; abort on any mismatch. Even a single byte off means starting over—never attempt to extract a failed checksum.

Never restore onto a non-empty or partially restored data directory. Backup or clear /var/lib/monad before extraction; never mix operational data with configs or keys. Mixing snapshot data with live keys or configs is a recipe for silent failures.

Extract with lz4 -dc ... | tar -x -C /var/lib/monad to minimize IO and temp disk usage. This avoids double-handling large files and keeps the restore process within commodity hardware limits.

After extraction, set correct ownership with chown -R monad:monad /var/lib/monad to avoid permission errors on startup. Skipping this step leads to subtle runtime issues and failed syncs.

Start the node with systemctl start monad and monitor logs for permission errors, missing files, or version mismatches. Expected log output includes a clean startup sequence, rapid block import, and no panics or repeated warnings.

Sanity-check the restored block height via direct RPC: call eth_blockNumber on port 8545. The result should match the snapshot’s documented height; discrepancies indicate incomplete or failed restores.

Always document the snapshot version compatibility matrix for different Monad releases. Match snapshot metadata (VERSION/README) to the node binary—running mismatched versions is a top source of restore failures. Refer to official docs for the latest compatibility notes.

If the node fails to start, execute an explicit rollback procedure: restore the previous backup and re-verify all steps. Never attempt to patch over a failed restore; full rollback is safer and faster.

Log and timestamp every restore attempt for auditability. This is critical for diagnosing issues and proving operational diligence during post-mortems.

Typical snapshot size is 100–150GB, with restore times ranging from 15 to 45 minutes on commodity hardware. Practice the full restore process periodically to surface latent issues before production incidents—don’t wait for an outage to discover a broken workflow.

Operational safety depends on never restoring snapshots onto non-empty or partially restored data directories. Mixing remnants from previous attempts or other chains in /var/lib/monad can introduce subtle, hard-to-diagnose bugs that only surface under load or after upgrades.

When extracting, always use the recommended lz4 -dc ... | tar -x -C /var/lib/monad pipeline. This approach minimizes both IO and temporary disk usage, which is especially important on servers with limited resources or when running multiple nodes in parallel.

After restore, immediately set ownership with chown -R monad:monad /var/lib/monad. Permission mismatches are a frequent cause of startup failures, and can manifest as missing file errors or unexplained sync stalls in logs.

Every restore attempt should be logged and timestamped, including disk checks, checksum results, and version metadata. This audit trail is essential for troubleshooting and for compliance in regulated environments.

If a restore fails, rollback to the last known-good backup without delay. Attempting to salvage a half-applied snapshot or mismatched version wastes time and increases operational risk.