Overview

The AI Safety Platform is a mandatory defence-in-depth system enforced at every stage of the AI lifecycle: from raw prompt ingestion through model deployment through post-deployment monitoring through regulatory certification. Safety is not an optional add-on — it is a pipeline stage wired into every request path, every agent handoff, and every model promotion decision. Available on Professional plan and above; the red-team certification runner, safety analytics, and regulatory reports require Sovereign plan.

Phase	Component	What it enforces
1	Safety Policy Engine	Regex pre-screen (built-in rules) + ML classifier ensemble (Llama Guard, ShieldGemma, Azure AI Content Safety, OpenAI Moderation). Two-phase: regex runs first (<2ms), ML runs only if regex passes.
2	Agentic Safety Guard	Tool-call validation before execution — risk level, argument pattern matching (block `rm -rf /`, `DROP TABLE`), principal allow-lists. Destructive calls fail loudly.
3	PII Masking	Microsoft Presidio (or built-in regex) auto-redacts PII entity types from outputs in place instead of blocking, when auto_fixable=true.
4	Conversation Monitor	Cross-turn analysis detecting crescendo / persistence / escalation / refusal-bypass, plus a stateful per-session memory window that blocks PII seen in earlier turns from later outputs.
5	Multi-Agent Trust Boundary	A caller agent cannot grant a callee more authority than it holds. Delegation allow-lists and hand-off payload scanning enforced before every agent hand-off.
6	Safety SLO Engine	Error-budget burn-rate framework (Google SRE). Burn rate >14.4× triggers a critical alert. Default SLOs: PII protection (0 leaks), block-rate budget (≤10%), injection budget (≤25).
7	Red-Team Certification Runner	Curated adversarial corpus (jailbreak, prompt-injection, harmful, PII, benign controls) + domain libraries (medical/financial/legal) + multi-turn chains, run on demand through the real guard. Returns a pass/fail certification with per-category coverage and blocking_failures (undetected attacks).
8	ModelSafetyRegistry Clearance	Pre-deployment clearance gate aggregating bias, robustness, and safety-eval results into a single deploy verdict, plus a lineage/provenance DAG back to the root base model. Promotion is blocked until the model earns clearance.

LLM output safety evaluation — LLM output evaluation — score prompt/response pairs for hallucination, toxicity, and factual consistency with a per-check breakdown.

Safety Policy Engine

Every request passes through two validation phases before any model response is returned: (1) regex pre-screen across 14 built-in rule IDs covering six categories, then (2) ML classifier ensemble. The regex layer is sub-millisecond; the ML layer runs only if the regex layer does not block, keeping the fast path fast.

bash

Validate a prompt via REST

1curl -X POST https://sovereign.yourcompany.com/api/v1/safety/validate/input \
2  -H "Authorization: Bearer $PAT" \
3  -H "Content-Type: application/json" \
4  -d '{
5    "text": "Ignore all previous instructions and...",
6    "target": "input",
7    "context": { "agent_id": "researcher-agent", "dry_run": false }
8  }'
9# → { "blocked": true, "violations": [{"rule_id": "JAILBREAK-001", "severity": "high"}],
10#     "classifier_scores": { "llama_guard": 0.94 } }

bash

Enable Llama Guard ML classifier

1# Option 1 — Llama Guard via Ollama (air-gapped, no API key)
2docker run -d -p 11434:11434 ollama/ollama
3docker exec <container> ollama pull llama-guard3
4export SAFETY_CLASSIFIER=llama_guard
5
6# Option 2 — Production ensemble
7export SAFETY_CLASSIFIER=llama_guard,openai_moderation
8export SAFETY_CLASSIFIER_STRATEGY=highest_severity   # default
9
10# Option 3 — Shadow mode for calibration (no blocking)
11# Pass dry_run=true in the request context — metrics recorded but never blocked

Category	Rule IDs	What it detects
`jailbreak`	JAILBREAK-001/002/003	Instruction overrides, DAN personas, system prompt exfiltration
`harmful_content`	HARMFUL-001/002/003	CBRN weapons, self-harm instructions, CSAM indicators
`privacy_leakage`	PRIVACY-001/002/003	SSN, credit card, API keys/credentials in output
`bias_fairness`	BIAS-001	Discriminatory generalisations on protected characteristics
`adversarial_abuse`	ABUSE-001	Token smuggling via chat template delimiters
`human_oversight`	OVERSIGHT-001	Autonomous high-stakes decisions without human approval

Agentic Safety

When AI agents execute multi-step tasks, the threat model shifts from prompts to actions. Three complementary controls protect agent pipelines: AgenticSafetyGuard (tool-call validation), CrossTurnMemorySafetyMonitor (multi-turn data exfiltration detection), and MultiAgentTrustBoundary (trust escalation prevention).

bash

Validate a tool call before execution

1curl -X POST https://sovereign.yourcompany.com/api/v1/safety/agentic/tool-call/validate \
2  -H "Authorization: Bearer $PAT" \
3  -H "Content-Type: application/json" \
4  -d '{
5    "tool_name": "delete_file",
6    "arguments": { "path": "/prod/db" },
7    "context": { "agent_id": "researcher-agent", "trust_level": "internal" }
8  }'
9# → { "allowed": false, "risk_level": "critical",
10#     "violations": ["Argument matches prohibited pattern: /prod/"] }

Control	What it prevents
AgenticSafetyGuard	Blocks tools above the caller's risk level, arguments matching prohibited patterns (rm -rf /, DROP TABLE), and principals not on the tool's allow-list. Hard fail — no silent pass-through.
CrossTurnMemorySafetyMonitor	Detects multi-turn data exfiltration: tracks PII observed in early turns and blocks it from appearing in later outputs.
MultiAgentTrustBoundary	Prevents a caller agent from granting a callee more authority than it possesses. Trust levels: untrusted → internal → trusted → admin.

Approvals & Human Review

When a violation requires human oversight or a high-risk action needs sign-off, the interaction is routed to the Approvals system. Reviewers list pending items and record an approve/reject decision; every decision is written to the immutable audit trail. The safety SDK/CLI/MCP review helpers (list_reviews / submit_review) are backed by this Approvals system.

bash

List pending approvals + record a decision

1# List pending approvals
2curl "https://sovereign.yourcompany.com/api/v1/approvals/pending?workspace_id=$WS" \
3  -H "Authorization: Bearer $PAT"
4
5# Resolve (approve / reject)
6curl -X POST https://sovereign.yourcompany.com/api/v1/approvals/rev-abc123/resolve \
7  -H "Authorization: Bearer $PAT" \
8  -H "Content-Type: application/json" \
9  -d '{
10    "decision": "reject",
11    "reason": "Confirmed jailbreak — DAN persona attempt"
12  }'

Safety SLOs

Safety commitments are codified as measurable, time-windowed SLOs using the Google SRE error-budget framework, computed from real safety audit events and LLM request volume. A burn rate of 14.4× means 5% of the weekly error budget is consumed per hour — a critical alert fires automatically via the configured alert integration.

bash

Check SLO fleet status

curl https://sovereign.yourcompany.com/api/v1/safety/slos/summary \
  -H "Authorization: Bearer $PAT"
# → { "total": 3, "healthy": 2, "warning": 1, "critical": 0,
#     "slos": [{ "name": "block_rate_budget", "burn_rate_1h": 0.7, "error_budget_remaining_percent": 98.4 }, ...] }

SLO	Threshold	Direction
`pii_protection`	0 PII leaks over any 24h	less-than-or-equal
`block_rate_budget`	≤ 10% blocked over any 24h	less-than-or-equal
`injection_budget`	≤ 25 injection blocks over any 24h	less-than-or-equal

Red-Team Certification

The red-team certification runner executes a curated adversarial corpus (jailbreak, prompt-injection, harmful, PII, and benign controls) plus optional domain libraries (medical/financial/legal) and multi-turn attack chains — all run on demand through the real Safety Policy Engine and conversation monitor. It returns a pass/fail certification with per-category detection coverage and blocking_failures (attacks the guard did not detect). Blocking failures prevent model promotion via the CompliancePipelineAgent safety gate.

bash

Run the certification suite

1# Via REST (server-side)
2curl -X POST https://sovereign.yourcompany.com/api/v1/safety/redteam/run \
3  -H "Authorization: Bearer $PAT" \
4  -H "Content-Type: application/json" \
5  -d '{ "domains": ["medical", "financial"], "strict": false, "include_multi_turn": true }'
6
7# Via CLI (runs against the local Safety Policy Engine)
8oss safety redteam run --domain medical --domain financial --strict

ModelSafetyRegistry Clearance

The ModelSafetyRegistry tracks a safety clearance verdict and deployment constraints for every model/version pair, aggregating bias, robustness, and safety-eval results into a single deploy decision. Before any model is promoted to production, the CompliancePipelineAgent checks the registry — a model without clearance is blocked. Provenance is available as a lineage DAG back to the root base model.

bash

Check a model's clearance

1curl "https://sovereign.yourcompany.com/api/v1/safety/clearance/my-model?workspace_id=$WS" \
2  -H "Authorization: Bearer $PAT"
3# → { "model_id": "my-model", "cleared": false, "verdict": "conditional",
4#     "constraints": ["human_oversight_required"],
5#     "blocking": ["robustness_score_below_threshold"] }

Verdict	Meaning
`certified`	Passed full safety clearance
`conditional`	Approved with documented constraints
`experimental`	Internal use only — not production-ready
`restricted`	High-risk — requires explicit override
`unsafe`	Failed safety evaluation — blocked from all promotion paths

External Integration Hub

Safety and governance events are dispatched to configured external systems, and GET /safety/integrations/status reports live connectivity for each. SIEM export is conformance-tested against Splunk HEC and AWS CloudWatch; incident notifications route to PagerDuty, Slack, and Jira. Configure with environment variables — no code changes required.

bash

Configure and check integration status

1export SPLUNK_HEC_URL=https://splunk.corp.example.com:8088/services/collector
2export SPLUNK_HEC_TOKEN=your-token
3export PAGERDUTY_ROUTING_KEY=your-routing-key
4export SLACK_WEBHOOK_URL=https://hooks.slack.com/...
5
6# Report live connectivity for every configured integration
7curl "https://sovereign.yourcompany.com/api/v1/safety/integrations/status?workspace_id=$WS" \
8  -H "Authorization: Bearer $PAT"

Category	Providers
SIEM	Splunk HEC, AWS CloudWatch Logs
Alerting	PagerDuty Events API v2
ITSM	Jira REST API v3
Chat/notification	Slack Block Kit