Architecture — Vigil SOC

Overview

Vigil uses backend tool integration via Claude Agent SDK. No desktop application or separate MCP servers required for core functionality.

Architecture Diagram

┌─────────────┐
│   Browser   │
│   (Web UI)  │
└──────┬──────┘
       │ HTTPS
       ↓
┌─────────────────────┐
│  FastAPI Backend    │
│  (Python)           │
│  ├── API Endpoints  │
│  ├── Tool Routing   │
│  └── Claude Service │
└──────┬──────────────┘
       │ Agent SDK
       ↓
┌─────────────────────┐
│  Anthropic Servers  │
│  (Claude 4.5)       │
│  Agent SDK          │
└──────┬──────────────┘
       │ Autonomous tool execution
       ↓
┌─────────────────────┐
│   Backend Tools     │
│  ├── Security       │
│  │   Detections    │
│  ├── Case Mgmt      │
│  ├── Approvals      │
│  └── ATT&CK         │
└──────┬──────────────┘
       │
       ↓
┌─────────────────────┐
│   Data Layer        │
│  ├── PostgreSQL     │
│  ├── Detection      │
│  │   Rules (6.7K)  │
│  └── Services       │
└─────────────────────┘

Stack Components

Frontend

React/TypeScript
Vite dev server
Port: 6988

Backend

FastAPI (Python 3.10+)
Claude API integration
Backend tools (23 tools)
Port: 6987

Database

PostgreSQL 16
Findings, cases, embeddings
Port: 5432

Detection Rules

7,200+ rules (Sigma, Splunk, Elastic, KQL)
Loaded from: ~/security-detections/
Indexed in memory on first use

Tool Categories

1. Security Detection Tools (5)

Access to detection rule database:

analyze_coverage - Coverage by MITRE technique
search_detections - Keyword search across rules
identify_gaps - Gap analysis
get_coverage_stats - Statistics
get_detection_count - Counts by source

2. Finding & Case Tools (10)

SOC investigation workflow:

list_findings - Query findings
search_findings - Keyword search across findings
get_findings_stats - Finding statistics
get_finding - Get specific finding
nearest_neighbors - Similar findings
list_cases - Query cases
get_case - Get specific case
create_case - Create new case
add_finding_to_case - Associate findings
update_case - Update case status and metadata
add_resolution_step - Document resolution steps

3. MITRE ATT&CK Tools (2)

Technique mapping and visualization:

get_attack_layer - Generate Navigator layer
get_technique_rollup - Technique statistics

4. Approval Tools (5)

Autonomous response workflow:

list_pending_approvals - Pending actions
get_approval_action - Action details
approve_action - Approve action
reject_action - Reject action
get_approval_stats - Statistics

Total: 23 backend tools

Request Flow

Example: Detection Coverage Query

1. User: "What's our coverage for T1059.001?"
   ↓
2. Web UI → POST /api/claude/chat
   ↓
3. Backend → Claude API (with tool definitions)
   ↓
4. Claude decides to use: analyze_coverage
   ↓
5. Backend executes: security_tools.analyze_coverage(["T1059.001"])
   ↓
6. Loads detection rules (if not cached)
   ↓
7. Searches Sigma, Splunk, Elastic, KQL rules
   ↓
8. Returns: { "T1059.001": { count: 234, by_source: {...} } }
   ↓
9. Claude synthesizes response
   ↓
10. Web UI displays result

Configuration

LLM provider credentials (Anthropic / OpenAI / Ollama) are configured in the web UI under Settings → AI / LLM Providers and stored in the encrypted secret store at ~/.vigil/secrets.enc. The backend pushes them to Bifrost over its admin API in the same request. See CONFIGURATION.md and STATE.md for the full split between bootstrap config and secrets.

Environment Variables (.env)

Bootstrap-only — nothing sensitive belongs here.

# Detection Rules
SIGMA_PATHS="${HOME}/security-detections/sigma/rules"
SPLUNK_PATHS="${HOME}/security-detections/security_content/detections"
ELASTIC_PATHS="${HOME}/security-detections/detection-rules/rules"
KQL_PATHS="${HOME}/security-detections/Hunting-Queries-Detection-Rules"

# Database
DATABASE_URL="postgresql://deeptempo:deeptempo_secure_password_change_me@localhost:5432/deeptempo_soc"

# LLM gateway (Bifrost) — provider keys are NOT set here
BIFROST_URL="http://bifrost:8080"

One loop, and where it is

There is exactly one tool-calling loop in Vigil, and it is TypeScript: services/agent/core/stream.ts. It owns the budget, the approval gate, the prompt-injection scan and the turn cap, so anything that reasons with tools gets all four by construction rather than by remembering to.

Python holds no loop. ClaudeService.chat() is a one-shot completion for the ~20 callers that want a single answer and no tools:

from core.llm.harness.claude import ClaudeService

answer = ClaudeService().chat(message="...", system_prompt="...")

Work that needs tools is enqueued as a run and the worker drives it:

from core.agents.queue import build_start_job, enqueue_run, new_run_id

run_id = new_run_id()
await enqueue_run(build_start_job(run_id, "investigate", request, enqueued_by="api"))

Five loops preceded this — claude_service, agent_runner, workflows_service, orchestrator and openai_agent_service — with three tool dispatchers and four transports between them. scripts/check_one_loop.py runs in CI and fails when a second one appears, because each addition looked reasonable on its own and review did not catch the fifth.

Deployment

Development

# Terminal 1: Database
cd docker && docker-compose up -d postgres

# Terminal 2: Backend
source venv/bin/activate
uvicorn services.api.main:app --host 127.0.0.1 --port 6987 --reload

# Terminal 3: Frontend
cd clients/web && npm run dev

Production (Docker)

cd docker
docker-compose up -d

Includes:

PostgreSQL
Backend API
Nginx (optional)

Performance

Tool Execution Latency

Tool	Avg Latency
Detection search	100-500ms
Coverage analysis	200-1000ms
Database queries	10-50ms
ATT&CK layer	200-500ms

Resource Usage

Component	Memory	CPU
Backend	500MB	10-20%
Detection rules (loaded)	200MB	0%
PostgreSQL	100MB	5%

Scalability

Concurrent users: 50-100 (single instance)
Detection rule index: Shared across all users
Claude API: Rate limited by Anthropic
Database: PostgreSQL with connection pooling enabled

Monitoring

Health Endpoints

# Backend health
curl http://localhost:6987/api/health

# Claude API status
curl http://localhost:6987/api/claude/status

# Database connectivity
curl http://localhost:6987/api/storage/status

Logs

# Backend logs (stdout when running uvicorn directly, or Docker logs)
docker logs deeptempo-postgres  # PostgreSQL logs
docker logs -f <backend-container>  # Backend API logs

# Tool execution (debug) — enable via LOG_LEVEL=DEBUG in .env

Security

API Key Management

Stored in secure keychain (macOS/Linux)
Fallback to environment variables
Never logged or exposed

Tool Permissions

No dangerous operations exposed
Approval required for response actions
Rate limiting on API endpoints
Authentication required (configurable)

Comparison: Previous vs Current Architecture

Before (MCP-Based)

Web UI → Backend → MCP Client → MCP Servers (stdio)
                                     ↓
                              Tool Implementations

Required desktop application for local use
Complex MCP server configuration
Not web UI compatible
Higher latency (protocol overhead)

Now (Agent SDK)

Web UI → Backend → Claude Agent SDK → Backend Tools → Services

Autonomous tool execution via Agent SDK
Simple configuration
Production-ready
Lower latency

Troubleshooting

“No tools loaded”

# Tools are the registry's, not an LLM client's. Startup populates it.
from core.integrations.mcp.registry import get_mcp_registry
print(get_mcp_registry().get_tool_names())

An empty list means no MCP server connected this boot: the disk cache is a warm-start artifact and a server that failed to connect is deliberately not registered, so a model cannot claim a capability it has no session for.

“Detection rules not found”

# Check repositories
ls -la ~/security-detections/
# Should show: sigma/, security_content/, detection-rules/, Hunting-Queries-Detection-Rules/

# Clone if missing
./scripts/setup_detection_repos.sh

“Database connection failed”

# Check PostgreSQL
docker ps | grep postgres

# Check connection string
echo $POSTGRESQL_CONNECTION_STRING

Development

Adding New Tools

Define tool schema in core/llm/tool_schemas.py:

{
    "name": "my_new_tool",
    "description": "What the tool does",
    "input_schema": {
        "type": "object",
        "properties": {
            "param1": {"type": "string", "description": "..."}
        },
        "required": ["param1"]
    }
}

Implement tool in appropriate service
Route tool in core/llm/harness/claude.py → _process_backend_tool_use():

elif tool_name == 'my_new_tool':
    from services.my_service import MyService
    service = MyService()
    result = service.my_method(**arguments)

Test:

python tests/test_backend_tools.py

Running Tests

# Backend tool tests
python tests/test_backend_tools.py

# Integration tests
python tests/test_integration_backend_tools.py

# Unit tests
pytest tests/unit/

Documentation

Backend Tools Guide - Detailed tool documentation
Detection Engineering - Detection rule usage
Integrations - Backend tool integration overview
API Reference - FastAPI documentation

Support

For issues or questions:

GitHub Issues: https://github.com/Vigil-SOC/vigil/issues
Label: backend-tools
Include: Backend logs, tool name, error message