Bifrost Gateway

Bifrost is a compiled Go binary that exposes a unified LLM gateway — OpenAI-format requests on /v1 plus provider-native passthroughs (e.g. /anthropic). Vigil routes all LLM traffic through Bifrost so caching, cost tracking, and budget enforcement live in one place.

How Vigil uses Bifrost

As of GH #84 PR-B, there is a single routing path:

• Anthropic traffic (including extended-thinking calls) hits Bifrost’s Anthropic-compatible passthrough at {BIFROST_URL}/anthropic using the regular Anthropic SDK with a swapped base_url. This preserves extended thinking, cache_control blocks, and cache-token usage counters.
• OpenAI / Ollama / other providers hit Bifrost’s /v1 OpenAI-format surface.

The single source of truth for Anthropic client construction is core/llm/providers/clients.py — every Anthropic(...) / AsyncAnthropic(...) call site in the repo goes through create_anthropic_client / create_async_anthropic_client, which point at Bifrost. The lone exception is backend/api/llm_providers.py, which validates user-supplied API keys against the upstream provider and therefore deliberately bypasses the gateway.

The routing decision lives in core/llm/router/router.py. There is no direct-SDK bypass — if Bifrost is unhealthy, LLM traffic fails loudly rather than silently taking a second path.

Pre-flight capability probe (merge blocker)

Before shipping any change that modifies how Vigil talks to Bifrost, run:

BIFROST_URL=http://localhost:8080 \
ANTHROPIC_API_KEY=sk-ant-... \
python scripts/bifrost_capability_probe.py

This verifies Bifrost passes through:

• basic round-trip
• the thinking parameter (extended thinking)
• cache_control: {"type": "ephemeral"} on system blocks → cache creation
• second-call cache hits surfaced as cache_read_input_tokens in the usage object

If any probe fails, do not ship. File an upstream issue on https://github.com/maximhq/bifrost and hold the change until it’s fixed. Per project policy (see memory: “Single LLM routing path”), we never reintroduce a direct-SDK bypass to work around a gateway limitation.

Starting Bifrost

Bifrost is always-on — docker compose up brings it up alongside the rest of the stack:

docker compose up postgres redis bifrost backend llm-worker

Health check: curl http://localhost:8080/health.

Configuration

docker/bifrost/config.json declares the providers, the models Bifrost will expose, and the cache backend. API keys are not written into the config file — they are injected as environment variables at container start time (env.ANTHROPIC_API_KEY, env.OPENAI_API_KEY, env.OLLAMA_URL). Vigil’s backend reads per-provider keys from its own secrets_manager; what’s in Bifrost’s env are the fallback/default keys used when a provider row in llm_provider_configs doesn’t override them.

Model allow-list: runtime sync, not the config file

The models array under each providers.<name>.keys[0] in config.json is a cold-boot bootstrap list only. The backend runs sync_all_provider_models() (core/llm/bifrost/admin.py) which:

1. Queries each upstream provider’s live catalog (Anthropic /v1/models, OpenAI /v1/models, Ollama /api/tags) via core/llm/providers/discovery.py.
2. Writes the per-row list into the backend’s dropdown cache (_MODEL_LIST_CACHE in core/llm/providers/registry.py).
3. Unions the model IDs across all active providers of the same type and PUTs that union to Bifrost’s admin API (PUT /api/providers/{name} with keys[0].models updated).

Single source of truth. sync_all_provider_models() is the only writer to both caches — the UI dropdown and Bifrost’s allow-list come from the same call over the same upstream fetch, so they cannot drift. fetch_provider_models() on the read path is a pure cache lookup that falls back to a lazy sync on a cold-start miss.

When it runs:

• At backend startup — launched as a background task (services/api/main.py startup handler).
• Periodically — MODEL_CATALOG_REFRESH_INTERVAL_S (default 300s / 5min). Set to 0 to disable the loop and only sync once at startup.
• On demand — POST /api/llm-providers/refresh-models (all) or POST /api/llm-providers/{id}/refresh-models (one).
• Whenever a provider is added, updated, or deleted via the Providers UI — CRUD handlers invalidate the cache and schedule a background resync.

The bootstrap list in config.json only matters for the cold-start window (seconds) before the first sync iteration completes, or for environments where the backend can’t reach upstream at startup.

Local Ollama: wildcard routing + private-network access

Self-hosted Ollama serves whatever the user has pulled or built, so a finite discovered allow-list is wrong — a custom model (e.g. qwythos-custom:latest) would 400 with “no keys found that support model”. Two settings make local Ollama route reliably:

• Wildcard allow-list. The ollama key uses "models": ["*"] in config.json, and sync_provider_models() forces ["*"] for ollama rather than the discovered union — so routing never depends on /api/tags enumerating a custom model.
• network_config.allow_private_network: true. Bifrost blocks RFC1918 targets (10.x, 172.16.x, 192.168.x) by default. In the container path OLLAMA_URL resolves to host.docker.internal → the Docker Desktop gateway (192.168.65.254, RFC1918), so the flag is required or every call fails with “connection to private IP … is not allowed”. It belongs in the provider’s network_config, not at the key level — Bifrost’s schema silently drops the key-level form. Loopback (127.0.0.1, ::1) is always allowed regardless.

Bifrost seeds /app/data/config.db (SQLite) from config.json only on first boot; the store lives in the container’s writable layer, so a docker compose restart keeps the stale copy. After editing config.json, re-seed with docker compose up -d --force-recreate bifrost.

Legacy / non-listed models (extras)

Some upstream providers drop older model IDs from their /v1/models listing even when those IDs are still callable (Anthropic did this with the Claude 3.x family). To surface those in the UI and let Bifrost route traffic for them, the backend unions a configurable “extras” set into both the dropdown and Bifrost’s allow-list. Extras are flagged deprecated=True in the API response so the UI can badge them.

The defaults live in core/llm/providers/registry.py (_DEFAULT_EXTRA_MODELS) and currently cover Claude 3.5 Haiku, 3.5 Sonnet v2, and 3 Haiku. Override per deployment via env:

• ANTHROPIC_EXTRA_MODELS="id1,id2,..." — replaces the default list.
• ANTHROPIC_EXTRA_MODELS="" — disables extras for Anthropic entirely.
• OPENAI_EXTRA_MODELS="..." — same mechanism for OpenAI if you need it.

Logging — authoritative cost source (#185)

Bifrost’s logging plugin is the source of truth for per-call cost. The config at docker/bifrost/config.json enables it explicitly with a local SQLite store (./logs.db inside the Bifrost container). Vigil’s /api/analytics/cost time-series proxies Bifrost’s /api/logs/histogram/cost endpoint; on a provider repricing, an admin hits POST /api/analytics/recalculate-cost (admin-only) which calls Bifrost’s POST /api/logs/recalculate-cost to re-cost historical rows without a redeploy.

Per-call correlation: Vigil attaches a custom header x-bf-lh-vigil-interaction-id: <uuid> to every upstream call (the x-bf-lh-* prefix is Bifrost’s logging-headers convention). The UUID lands in Bifrost’s LogEntry.metadata field for manual audit. We do not wire automated per-call cost reconcile from Bifrost back into LLMInteractionLog.cost_usd because Bifrost’s GET /api/logs query parameters don’t currently support filtering by custom metadata — reconciling would require pulling a time-window of logs and matching locally on (model, timestamp, token_counts), which is brittle. Local compute_call_cost() remains the synchronous source for LLMInteractionLog.cost_usd; Bifrost is the authority for aggregate analytics. When Bifrost adds a metadata filter, swap to Bifrost as the authority for the cost_usd column too.

Production: switch to Postgres. SQLite is fine for dev (the default config above); for production set:

{
  "logs_store": {
    "enabled": true,
    "type": "postgres",
    "config": {
      "host": "postgres",
      "port": "5432",
      "user": "bifrost",
      "password": "<secret>",
      "db_name": "bifrost_logs"
    }
  }
}

Reuse the existing postgres service in docker-compose.yml with a separate database (UTF8 required by Bifrost). Inject the password via your secrets manager — same pattern as the provider API keys above.

Caching — two layers

Vigil benefits from two independent caching layers. They’re complementary, not redundant:

Anthropic prefix caching is turned on by default (ANTHROPIC_PROMPT_CACHE_ENABLED=true, kill-switch in Settings → AI Config → AI Operations). The cache_control markers are added in core/llm/harness/claude.py:_apply_prompt_cache_controls and preserved by Bifrost’s /anthropic passthrough — verified by scripts/bifrost_capability_probe.py.

Bifrost’s semantic cache is opt-in and configured through the Bifrost UI at http://localhost:8080 (not via docker/bifrost/config.json — v1.4.23 rejects a top-level cache block). Enabling it requires:

1. A vector store (Redis recommended — reuse the existing Redis, a separate DB number)
2. An embedding provider (OpenAI text-embedding-3-small or a local Ollama model)
3. Enabling the semantic_cache plugin in the UI

Run python scripts/bifrost_cache_status.py to check whether it’s live. Vigil’s unified routing does not depend on this layer — it’s optional cost gravy on top of Anthropic’s native prefix caching.

Production deployment: keep API keys out of .env

Local docker compose up injects ANTHROPIC_API_KEY / OPENAI_API_KEY into Bifrost via the environment: block, which resolves from .env in the compose project directory. That’s fine for development; it’s not fine for production — the .env file lives on the host, is read by every process in the compose network, and doesn’t rotate.

For production deployments, inject Bifrost’s provider keys from a real secrets manager at container start. A few patterns:

HashiCorp Vault (agent-injector or init container)

services:
  bifrost:
    image: maximhq/bifrost:latest
    # Vault agent writes rendered env file to a tmpfs shared volume;
    # the container sources it before the main process starts.
    command: ["sh", "-c", ". /vault/secrets/bifrost.env && exec /app/main"]
    volumes:
      - vault-secrets:/vault/secrets:ro

AWS Secrets Manager (ECS task definition)

Use ECS secrets (not environment) so the agent resolves the ARN at task launch and injects the value as an env var that never hits the task-definition JSON:

"containerDefinitions": [
  {
    "name": "bifrost",
    "secrets": [
      {"name": "ANTHROPIC_API_KEY", "valueFrom": "arn:aws:secretsmanager:...:vigil/anthropic-api-key"},
      {"name": "OPENAI_API_KEY",    "valueFrom": "arn:aws:secretsmanager:...:vigil/openai-api-key"}
    ]
  }
]

Kubernetes (Secret → envFrom)

envFrom:
  - secretRef:
      name: vigil-llm-provider-keys

With the Secret populated via Sealed Secrets, External Secrets Operator, or whatever mechanism you already use for other app secrets — Bifrost doesn’t care, it just wants the env var present at startup.

What stays in .env

Non-secret Bifrost config (URL, log level, etc.) is fine to keep in plaintext. The distinction is: if leaking the value to a host-mounted .env file is acceptable, leave it in env; otherwise, route it through your secrets manager of choice.

Tool-use support matrix