LLM Gateway — API Reference
Full REST API reference for the LLM Gateway — routing policies, safety eval endpoints, budget configuration, provider management, and the route simulator.
Overview
The LLM Gateway provides a unified OpenAI-compatible API. Existing code using openai.OpenAI(base_url=...) works without modification. Every request flows through: safety pre-screening → data classification check → provider routing → response streaming → async token counting + cost tracking → async audit log write.
Gateway Endpoints
All endpoints follow the OpenAI API spec.
| Endpoint | Description |
|---|---|
| `POST /api/v1/gateway/v1/chat/completions` | Chat completion (streaming SSE) |
| `POST /api/v1/gateway/v1/completions` | Legacy text completion |
| `POST /api/v1/gateway/v1/embeddings` | Text embeddings |
| `GET /api/v1/gateway/v1/models` | List available models |
Supported Providers
Register multiple providers per workspace. The router selects based on priority, tags, data classification, and budget.
| Type | Description |
|---|---|
| `openai` | OpenAI API (GPT-4o, GPT-4, GPT-3.5) |
| `anthropic` | Anthropic API (Claude models) |
| `azure_openai` | Azure OpenAI Service |
| `bedrock` | AWS Bedrock (Claude, Titan, Llama via AWS — IRSA, no API keys) |
| `vertex_ai` | Google Vertex AI / Gemini (Workload Identity, no service account keys) |
| `ollama` | Self-hosted Ollama (local/air-gapped deployments) |
| `custom` | Any OpenAI-compatible endpoint |
Routing Logic
Provider selection order: (1) Priority — lower number = higher priority. (2) Tags — via X-Provider-Tags header. (3) Data classification — X-Data-Classification: confidential routes to sovereign/on-premise providers only when LLM_ALLOW_EXTERNAL_PROVIDERS=false. (4) Budget — providers over monthly cap are skipped. (5) Fallback — 5xx triggers next-provider fallback.
Safety Evaluation (Input Screening)
Prompts are screened before being forwarded. The safety evaluator classifies for harmful content, PII exposure, jailbreak attempts, and data classification violations. Results: safe / unsafe / requires_review. Unsafe prompts are blocked at the gateway and logged.
curl -X POST /api/v1/safety/evaluations \
-d '{ "artifact_type": "prompt", "content": "...", "workspace_id": "ws-prod" }'LLM Output Evaluation
Output evaluation (Enterprise+) uses an LLM-as-judge pattern to score responses for hallucination, toxicity, and factual consistency. Pass thresholds: hallucination score < 0.30, toxicity < 0.20, factual consistency ≥ 0.60.
1curl -X POST /api/v1/llm/output-evals?workspace_id=ws-prod \
2 -d '{
3 "prompt": "What is the customer balance?",
4 "response": "Your balance is $4,231.00.",
5 "context": "account_id=ACC-001, balance=$4,231.00",
6 "checks": ["hallucination", "toxicity", "factual_consistency"]
7 }'Cost Tracking & Budget Alerts
Every gateway request writes a TokenUsageEvent record with workspace, provider, model, input/output tokens, cost in USD, and latency. Budget alerts fire via webhook when a workspace reaches LLM_BUDGET_ALERT_THRESHOLD (default 80%) of its monthly budget.
GET /api/v1/llm/usage?workspace_id=ws-prod&from=2026-04-01