ops
New
2026-04-13Backup & Disaster Recovery
RTO/RPO targets, PostgreSQL WAL archiving with wal-g, Velero hourly K8s backups, PITR runbooks, and quarterly DR test history.
backup
disaster-recovery
rto
rpo
velero
wal-g
pitr
Recovery Objectives
Measured from DR tests performed in Q1 2026.
| Tier | Scope | RTO | RPO |
|---|---|---|---|
| Tier 1 — Full outage | All services down; database inaccessible | **4 hours** | **1 hour** |
| Tier 2 — Single-service failure | One microservice crashed; DB healthy | **15 minutes** | 0 (stateless restart) |
| Tier 3 — Data corruption | Data loss in PostgreSQL or Redis | **2 hours** | **1 hour** |
| Tier 4 — Region loss | Entire cloud region unavailable | **8 hours** | **1 hour** |
PostgreSQL WAL Archiving
WAL segments are continuously streamed to object storage using wal-g or pgBackRest, achieving RPO ≈ 1 minute in steady state. A full base backup runs daily at 02:00 UTC via a Kubernetes CronJob, retained for 30 days.
yaml
1postgresql:
2 walArchiving:
3 enabled: true
4 walg:
5 enabled: true
6 s3Bucket: "your-org-aegissovereign-wal"
7 s3Region: "us-east-1"
8 archiveTimeout: 60 # secondsPoint-in-Time Recovery (PITR)
Restore to any point within the 30-day retention window.
bash
1# 1. Stop writes
2kubectl scale deployment aegissovereign-backend --replicas=0
3
4# 2. Restore base backup
5wal-g backup-fetch /var/lib/postgresql/data LATEST
6
7# 3. Set recovery target
8cat >> /var/lib/postgresql/data/postgresql.conf <<EOF
9restore_command = 'wal-g wal-fetch %f %p'
10recovery_target_time = '2026-04-10 14:30:00+00'
11recovery_target_action = 'promote'
12EOF
13touch /var/lib/postgresql/data/recovery.signal
14
15# 4. Restart
16kubectl scale deployment aegissovereign-postgres --replicas=1
17kubectl scale deployment aegissovereign-backend --replicas=3Kubernetes Backup (Velero)
Velero backs up K8s resources, ConfigMaps, Secrets, and PVC snapshots on an hourly schedule with 7-day retention, and daily with 30-day retention.
bash
1velero schedule create aegissovereign-hourly \
2 --schedule="0 * * * *" \
3 --include-namespaces aegissovereign \
4 --ttl 168h
5
6velero schedule create aegissovereign-daily \
7 --schedule="0 3 * * *" \
8 --include-namespaces aegissovereign \
9 --ttl 720hDR Test History
DR tests are scheduled quarterly. Results are reviewed by Engineering and shared with Enterprise customers on request.
| Date | Type | RTO Achieved | RPO Achieved | Notes |
|---|---|---|---|---|
| 2026-01-15 | PITR drill (Tier 3) | 1h 45m | 3 minutes | Restored to 48h prior timestamp |
| 2026-02-12 | Full cluster restore (Tier 1) | 3h 20m | 58 minutes | Within RTO/RPO targets |
| 2026-03-05 | Redis loss simulation (Tier 2) | 8 minutes | 0 | Cache warmed from DB queries |
Backup Encryption
All backups are encrypted at rest.
| Layer | Encryption |
|---|---|
| WAL archives (S3/GCS) | AES-256 (SSE-S3 or CMEK) |
| Base backups | wal-g built-in AES-256-CTR |
| Velero PVC snapshots | Cloud-native volume encryption (EBS/Persistent Disk) |
| Redis RDB/AOF | PVC-level encryption via storage class |