Health Endpoints

Three health endpoints are available without authentication for Kubernetes probes and uptime monitoring.

json

Readiness response

1{
2  "status": "ready",
3  "checks": { "database": "ok", "redis": "ok" },
4  "version": "1.0.0"
5}

Endpoint	Auth	Description
`GET /health`	None	Liveness — returns 200 if the process is running
`GET /api/v1/health/live`	None	Alias for liveness (explicit K8s livenessProbe path)
`GET /api/v1/health/ready`	None	Readiness — checks DB + Redis; returns 503 if either is down

Monitoring dashboard — The in-app monitoring dashboard — live metrics, drift monitor, agent traces, and per-agent kill switches.

Monitoring — Detail panel (opens when you click a row).

Prometheus Metrics

GET /metrics returns Prometheus text format. No authentication required — protect at network level via NetworkPolicy or ingress allow-list.

Metric	Type	Description
`http_requests_total`	Counter	Total HTTP requests by method, path, status
`http_request_duration_seconds`	Histogram	Request latency (p50/p95/p99)
`http_requests_in_progress`	Gauge	Current in-flight requests
`aegissovereign_cpu_usage_percent`	Gauge	Host CPU utilisation (all cores, averaged)
`aegissovereign_gpu_usage_percent`	Gauge	Primary GPU utilisation (0 when no GPU)
`aegissovereign_gpu_memory_used_bytes`	Gauge	GPU VRAM in use
`aegissovereign_celery_active_tasks`	Gauge	Celery tasks currently executing

Grafana Dashboard

Import the pre-built Grafana dashboard from grafana/opensovereign-dashboard.json in the repository (the file keeps the legacy runtime name). Key panels: request rate and error rate by endpoint, p95/p99 latency trends, LLM token cost per workspace over time, active agent runs, and Celery task queue depth — metrics are exported under the opensovereign_* prefix.

bash

Port-forward Grafana

kubectl port-forward svc/grafana 3000:3000 -n monitoring
# Then import grafana/opensovereign-dashboard.json via the Grafana UI

Workspace Metrics API

GET /api/v1/monitoring/metrics?workspace_id=ws-prod returns workspace-scoped metrics derived from real database aggregates. cpu_pct and gpu_pct are sourced from the Prometheus collector; they return null when psutil/pynvml are not installed.

json

Response

1{
2  "active_models": 12,
3  "predictions_per_hr": 47230,
4  "latency_p99_ms": 87.4,
5  "cpu_pct": 34.2,
6  "gpu_pct": 71.8
7}

Structured Logging

All log output is structured JSON (set LOG_FORMAT=dev for human-readable output during development). Every log line in request context carries request_id from the X-Request-Id header, enabling correlation of all log lines for a single request in Splunk, Datadog, or CloudWatch.

json

Log entry

1{
2  "timestamp": "2026-04-05T10:23:41.123Z",
3  "level": "INFO",
4  "logger": "app.routers.registry",
5  "message": "Model promoted to production",
6  "request_id": "req-550e8400",
7  "workspace_id": "ws-prod",
8  "model_id": "model-uuid",
9  "initiated_by": "user@company.com"
10}

SIEM Integration

Every AuditEntry is forwarded asynchronously to configured SIEM backends. All adapters are fire-and-forget — a SIEM failure never blocks the API response or rolls back the audit write. Test connectivity via POST /api/v1/audit/siem/test.

Provider	Required env vars
Splunk HEC	`SIEM_SPLUNK_HEC_URL`, `SIEM_SPLUNK_HEC_TOKEN`, `SIEM_SPLUNK_INDEX`
Datadog Logs	`SIEM_DATADOG_API_KEY`, `SIEM_DATADOG_SITE` (default: `datadoghq.com`)
AWS CloudWatch	`SIEM_CLOUDWATCH_LOG_GROUP`, `SIEM_CLOUDWATCH_LOG_STREAM` (+ boto3 credentials)
GCP Cloud Logging	`SIEM_GCP_LOG_NAME`, `GCP_PROJECT_ID` (+ Application Default Credentials)
GCP Security Command Center	`SIEM_GCP_SCC_ORG_ID`, `GCP_PROJECT_ID`
GCP Cloud Pub/Sub	`SIEM_GCP_PUBSUB_TOPIC`, `GCP_PROJECT_ID`