How the ContactOS production redundancy is built, how it works, the hardening behind it, and exactly what to do when something goes wrong. Validated by game-day 2026-06-12
26781c18 · identical pinned image · NODE_ROLE decides who runs the timers · certs auto-sync active → standbycontactos_prodcontactos.uqnqpph.mongodb.net · 3-node replica set · auto-failover · PITR + snapshotsExactly one node is active at a time. DNS points at a Reserved IP, and the Reserved IP points at the active node. Both nodes always answer HTTP, but only the active one runs the background timers that send messages — so customers never get double-sent. Failover = remap the Reserved IP to the standby. No DNS change, no waiting on TTL.
The api reads NODE_ROLE=active|standby. On standby it skips the two timer loops (CON-94 inactivity sender + stale-conversation cleanup) — the only paths that could double-fire, since inbound & broadcasts already reach one node only. Default is active, so a single node is unaffected. /status exposes the live role for monitoring.
The standby can't complete an HTTP-01 challenge without holding the IP, so acme-sync.sh runs on both nodes daily and the active node pushes its live Let's Encrypt acme.json to the standby. It self-gates on NODE_ROLE, so a promote flips the sync direction automatically.
Both nodes share one Atlas M10 replica set (DB contactos_prod). A primary failure triggers an automatic election; the driver reconnects transparently — RPO ≈ 0, no app action. Continuous PITR + on-demand snapshots back it up.
Deploys pin IMAGE_TAG=<sha> in the droplet .env → deterministic one-command rollback.
App user contactos_app (readWrite on the app DBs only); the over-privileged atlasAdmin user deleted; creds rotated.
Timer loops run on the active node only — no duplicate sends across two nodes.
Inbound WhatsApp deduped by message-id in Redis → Meta retries can't double-process.
mem_limit on every container; UI healthcheck; DO uptime checks & down-alerts.
IP access list restricted to prod + standby + Hetzner; removed 0.0.0.0/0.
Upgraded Flex → M10: 3-node replica set, auto-failover, PITR, termination protection.
MaxStartups 100:30:200 on both nodes so the internet brute-force flood can't drop ops connections.
DO droplet backups + Atlas PITR + on-demand snapshots before any destructive change.
| Scenario | What we did | Result |
|---|---|---|
| Planned flip | Promote standby → remap Reserved IP → flip back | PASS exactly-one-active throughout |
| Unplanned hard-kill | Powered off the active droplet, failed over to standby | PASS RTO ≈ 52s outage→recovery |
| Database failover | Atlas Test Failover while writing through the app | PASS RPO ≈ 0, zero failed writes |
| Agent WebSocket | Live session through a flip | PASS auto-reconnect, no reload / re-login |
role:active. Two active = double-send risk (demote one). Two standby = silent outage (promote one). Check any time with: curl -s https://api.contactos.ai/statushttps://api.contactos.ai times out or 5xx; DO shows the active droplet down.Fail over to the standby — one command (best-effort demote means a dead node doesn't block it):
# from a host with SSH to both droplets + a DO token DO_TOKEN=<digitalocean token> scripts/contactos-promote.sh b # promote standby (green) # verify exactly one active, then bring the old node back later: curl -s https://api.contactos.ai/status # expect {"role":"active",...}
Manual equivalent if the script/token isn't handy: SSH the standby → set NODE_ROLE=active in backend/.env → docker compose -f docker-compose.prod.yml up -d --force-recreate api → remap the Reserved IP 129.212.145.125 to droplet 576964180 (DO console / doctl / MCP).
DO_TOKEN=<token> scripts/contactos-promote.sh b # go green # ...do the work on the now-idle blue node, smoke-test... DO_TOKEN=<token> scripts/contactos-promote.sh a # back to blue
Tip: for a planned flip, the script demotes-first to guarantee no double-send (a brief serving gap while the new active's api recreates). To avoid the gap, promote the new node to healthy before remapping the IP.
Nothing to do — the M10 replica set elects a new primary and the driver reconnects automatically (RPO ≈ 0). Confirm the app is healthy: curl -s https://api.contactos.ai/health. Investigate the cause in the Atlas UI afterwards.
Roll back to a known-good immutable tag — no rebuild:
# re-run the Deploy workflow with image_tag set to a prior commit SHA, # or the pre-launch snapshot: prelaunch-2026-06-11 # or on the droplet directly: ssh root@<active> "cd /opt/apps/datastudios-contactos && \ sed -i 's/^IMAGE_TAG=.*/IMAGE_TAG=<good-sha>/' .env && \ docker compose -f docker-compose.prod.yml up -d"
Re-sync the live cert from the active node (the live volume is traefik_traefik-letsencrypt, not the stale bare traefik-letsencrypt):
ssh root@<active-node> /root/acme-sync.sh # pushes acme.json + restarts standby traefik
Use the Hetzner bastion (static IP, never flooded):
ssh -J root@hetzner-devbox root@<droplet-ip> '<command>'
contactos_prod.Restore from Atlas — PITR (any second in the retention window) or an on-demand snapshot, via the Atlas UI/API. For a selective recovery (one collection) without reverting everything: restore the snapshot to a temporary cluster, export the wanted collections, re-import into contactos_prod.