ops
    New
    2026-04-13

    Backup & Disaster Recovery

    RTO/RPO targets, PostgreSQL WAL archiving with wal-g, Velero hourly K8s backups, PITR runbooks, and quarterly DR test history.

    backup
    disaster-recovery
    rto
    rpo
    velero
    wal-g
    pitr

    Recovery Objectives

    Measured from DR tests performed in Q1 2026.

    TierScopeRTORPO
    Tier 1 — Full outageAll services down; database inaccessible**4 hours****1 hour**
    Tier 2 — Single-service failureOne microservice crashed; DB healthy**15 minutes**0 (stateless restart)
    Tier 3 — Data corruptionData loss in PostgreSQL or Redis**2 hours****1 hour**
    Tier 4 — Region lossEntire cloud region unavailable**8 hours****1 hour**

    PostgreSQL WAL Archiving

    WAL segments are continuously streamed to object storage using wal-g or pgBackRest, achieving RPO ≈ 1 minute in steady state. A full base backup runs daily at 02:00 UTC via a Kubernetes CronJob, retained for 30 days.

    yaml
    1postgresql:
    2  walArchiving:
    3    enabled: true
    4    walg:
    5      enabled: true
    6      s3Bucket: "your-org-aegissovereign-wal"
    7      s3Region: "us-east-1"
    8    archiveTimeout: 60   # seconds

    Point-in-Time Recovery (PITR)

    Restore to any point within the 30-day retention window.

    bash
    1# 1. Stop writes
    2kubectl scale deployment aegissovereign-backend --replicas=0
    3
    4# 2. Restore base backup
    5wal-g backup-fetch /var/lib/postgresql/data LATEST
    6
    7# 3. Set recovery target
    8cat >> /var/lib/postgresql/data/postgresql.conf <<EOF
    9restore_command = 'wal-g wal-fetch %f %p'
    10recovery_target_time = '2026-04-10 14:30:00+00'
    11recovery_target_action = 'promote'
    12EOF
    13touch /var/lib/postgresql/data/recovery.signal
    14
    15# 4. Restart
    16kubectl scale deployment aegissovereign-postgres --replicas=1
    17kubectl scale deployment aegissovereign-backend --replicas=3

    Kubernetes Backup (Velero)

    Velero backs up K8s resources, ConfigMaps, Secrets, and PVC snapshots on an hourly schedule with 7-day retention, and daily with 30-day retention.

    bash
    1velero schedule create aegissovereign-hourly \
    2  --schedule="0 * * * *" \
    3  --include-namespaces aegissovereign \
    4  --ttl 168h
    5
    6velero schedule create aegissovereign-daily \
    7  --schedule="0 3 * * *" \
    8  --include-namespaces aegissovereign \
    9  --ttl 720h

    DR Test History

    DR tests are scheduled quarterly. Results are reviewed by Engineering and shared with Enterprise customers on request.

    DateTypeRTO AchievedRPO AchievedNotes
    2026-01-15PITR drill (Tier 3)1h 45m3 minutesRestored to 48h prior timestamp
    2026-02-12Full cluster restore (Tier 1)3h 20m58 minutesWithin RTO/RPO targets
    2026-03-05Redis loss simulation (Tier 2)8 minutes0Cache warmed from DB queries

    Backup Encryption

    All backups are encrypted at rest.

    LayerEncryption
    WAL archives (S3/GCS)AES-256 (SSE-S3 or CMEK)
    Base backupswal-g built-in AES-256-CTR
    Velero PVC snapshotsCloud-native volume encryption (EBS/Persistent Disk)
    Redis RDB/AOFPVC-level encryption via storage class
    Edit this page on GitHub