Complete guide to testing the Vigil platform.

Table of Contents

Overview

The Vigil testing strategy includes:

  • Unit Tests: Test individual functions and classes in isolation
  • Integration Tests: Test API endpoints with database and mocked external services
  • End-to-End Tests: Test complete user workflows (future)
  • Performance Tests: Benchmark critical operations

Test Frameworks

Backend: pytest with coverage, async support, and mocking Frontend: Vitest with React Testing Library

Test Infrastructure

Backend Setup

# Install dependencies
pip install -r requirements.txt

# Test dependencies include:
# - pytest (test framework)
# - pytest-asyncio (async test support)
# - pytest-cov (coverage reporting)
# - pytest-mock (mocking)
# - faker (test data generation)
# - freezegun (time mocking)
# - responses (HTTP mocking)

Frontend Setup

cd clients/web
npm install

# Test dependencies include:
# - vitest (test framework)
# - @testing-library/react (component testing)
# - @testing-library/user-event (user interaction)
# - jsdom (DOM environment)
# - msw (API mocking)

Running Tests

Backend Tests

# Run all tests
pytest

# Run unit tests only
pytest tests/unit/

# Run integration tests only
pytest tests/integration/

# Run with coverage
pytest --cov=services --cov=core --cov-report=html

# Run specific test file
pytest tests/unit/test_auth.py

# Run specific test
pytest tests/unit/test_auth.py::TestPasswordHashing::test_hash_password

# Run tests matching pattern
pytest -k "auth"

# Run with verbose output
pytest -v

# Run in parallel (faster)
pytest -n auto

Frontend Tests

cd clients/web

# Run all tests
npm test

# Run tests once (CI mode)
npm run test:unit

# Watch mode (auto-rerun on changes)
npm run test:watch

# Run with coverage
npm run test:coverage

# Run specific test file
npm test src/utils/api.test.ts

Quick Test Script

# Run the full suite (backend pytest + frontend vitest)
./tests/run-tests.sh

Writing Tests

Unit Test Example

File: tests/unit/test_service.py

import pytest
from services.my_service import MyService

class TestMyService:
    """Test MyService functionality."""
    
    def test_basic_function(self):
        """Test basic function returns expected result."""
        result = MyService.calculate(10, 5)
        assert result == 15
    
    @pytest.mark.asyncio
    async def test_async_function(self):
        """Test async function."""
        result = await MyService.fetch_data("123")
        assert result is not None
    
    def test_with_mock(self, mocker):
        """Test with mocked dependency."""
        mock_db = mocker.patch('services.my_service.database')
        mock_db.query.return_value = [{"id": "1"}]
        
        result = MyService.get_records()
        assert len(result) == 1

Integration Test Example

File: tests/integration/test_api.py

import pytest
from fastapi.testclient import TestClient

@pytest.mark.integration
class TestCaseAPI:
    """Integration tests for case API."""
    
    def test_create_case(self, test_client, auth_headers):
        """Test case creation endpoint."""
        response = test_client.post(
            "/api/cases",
            headers=auth_headers,
            json={
                "title": "Test Case",
                "priority": "high",
                "severity": "high"
            }
        )
        
        assert response.status_code == 201
        data = response.json()
        assert data["title"] == "Test Case"
        assert "id" in data
    
    def test_list_cases(self, test_client, auth_headers):
        """Test listing cases."""
        response = test_client.get(
            "/api/cases",
            headers=auth_headers
        )
        
        assert response.status_code == 200
        assert isinstance(response.json(), list)

Frontend Test Example

File: clients/web/src/components/CaseList.test.tsx

import { describe, it, expect, vi } from 'vitest'
import { render, screen, fireEvent } from '@testing-library/react'
import CaseList from './CaseList'

describe('CaseList', () => {
  it('renders case list', () => {
    const cases = [
      { id: '1', title: 'Case 1', priority: 'high' },
      { id: '2', title: 'Case 2', priority: 'medium' }
    ]
    
    render(<CaseList cases={cases} />)
    
    expect(screen.getByText('Case 1')).toBeInTheDocument()
    expect(screen.getByText('Case 2')).toBeInTheDocument()
  })
  
  it('calls onClick when case is clicked', () => {
    const handleClick = vi.fn()
    const cases = [{ id: '1', title: 'Case 1', priority: 'high' }]
    
    render(<CaseList cases={cases} onCaseClick={handleClick} />)
    
    fireEvent.click(screen.getByText('Case 1'))
    expect(handleClick).toHaveBeenCalledWith('1')
  })
})

Test Fixtures

Using Shared Fixtures

Located in tests/conftest.py:

def test_with_fixtures(test_db_session, sample_user, auth_headers):
    """Test using multiple fixtures."""
    # test_db_session: Database session
    # sample_user: Pre-created user
    # auth_headers: Authorization headers
    pass

JSON Fixtures

Located in tests/fixtures/:

import json

def test_with_json_fixture():
    """Load test data from JSON file."""
    with open('tests/fixtures/sample_findings.json') as f:
        findings = json.load(f)
    
    assert len(findings) > 0

Coverage Requirements

Target Coverage

Component Unit Coverage Integration Coverage
Backend API 85% 75%
Services 80% 70%
Daemon 75% 60%
Database 70% 80%
Frontend 70% N/A

Viewing Coverage Reports

# Backend
pytest --cov --cov-report=html
open htmlcov/index.html

# Frontend
npm run test:coverage
open coverage/index.html

Coverage Commands

# Check if coverage meets thresholds
pytest --cov --cov-fail-under=80

# Generate XML coverage for CI
pytest --cov --cov-report=xml

# Show missing lines
pytest --cov --cov-report=term-missing

Test Markers

Use markers to categorize tests:

@pytest.mark.unit
def test_unit():
    pass

@pytest.mark.integration
def test_integration():
    pass

@pytest.mark.slow
def test_slow_operation():
    pass

@pytest.mark.performance
def test_performance_benchmark():
    pass

Run tests by marker:

# Only unit tests
pytest -m unit

# Skip slow tests
pytest -m "not slow"

# Only integration tests
pytest -m integration

Substituting Services

Services are constructed once by whoever owns the process and injected from there. Nothing reaches for a module-global accessor, so a test never has to monkeypatch one — it supplies its own instance. Three rules:

Where How the service arrives
services/api/main.py lifespan Constructs each service onto app.state (_build_services)
FastAPI handlers A Depends provider from core/deps.py reads request.app.state.<key>
Everything else (services, daemon, LLM harnesses) A constructor keyword argument, defaulting to a locally-built instance

tests/unit/_ratchets/test_no_ambient_state.py enforces this: it fails on a new module-level service instantiation and on the lazy global _x accessor shape that #459 retired.

Overriding a service in an API test

Use app.dependency_overrides, keyed on the provider function itself. The handler then talks to your stub and touches no database, LLM, or MCP process. Full worked example: tests/unit/api/test_service_dependency_overrides.py.

from core.deps import provide_approvals
from core.response.approvals_router import router as approvals_router


class StubApprovals:
    def list_pending_approvals(self):
        return []


def test_pending_is_empty():
    app = FastAPI()
    app.include_router(approvals_router, prefix="/api")
    app.dependency_overrides[provide_approvals] = lambda: StubApprovals()

    resp = TestClient(app).get("/api/approvals/pending")
    assert resp.json() == {"actions": []}

A bare FastAPI() like the one above runs no lifespan, so app.state holds nothing — the override is the wiring. If you instead build a TestClient over the real app as a context manager (with TestClient(app) as c:), the lifespan does run and populates app.state; overrides still win.

Overriding elsewhere

Outside a request there is no Depends, so pass the instance in:

svc = WorkflowsService(workflow_runs=WorkflowRunService(), approvals=ApprovalService())
agent = OpenAIAgentService(approvals=stub_approvals)
claude = ClaudeService(mcp_client=fake_client, mcp_registry=MCPRegistry())

The one exception: MCPClient

MCPClient owns persistent stdio child processes that only its creator closes, so exactly one may exist per process. The owner installs it (set_process_mcp_client) and process_mcp_client() returns it.

“One per process, installed by the owner” only holds if the processes are enumerated. There are three, and two of them own a client:

Process Owns an MCPClient?
API (services/api/main.py lifespan) yes
Daemon (services/daemon/main.py _init_components) yes
ARQ llm-worker (python -m services.worker) no — it runs with use_mcp_tools=False and reaches data through backend tools

Before any owner starts — which is the case in a plain unit test — it returns None, so importing the harness never spawns child processes. To supply a fake, pass mcp_client= to the constructor or override provide_mcp_client; only reach for set_process_mcp_client when the code under test resolves the process slot directly.

Mocking External Services

Mocking Claude API

def test_claude_enrichment(mocker):
    """Test with mocked Claude API."""
    mock_response = {
        "content": [{"type": "text", "text": "Test response"}]
    }
    mocker.patch('core.llm.harness.claude.anthropic.Anthropic')
    
    result = ClaudeService.enrich_finding({"id": "test"})
    assert result is not None

Mocking Splunk

@responses.activate
def test_splunk_search():
    """Test with mocked Splunk API."""
    responses.add(
        responses.GET,
        'https://splunk.example.com/services/search',
        json={"results": []},
        status=200
    )
    
    results = SplunkService.search("query")
    assert results == []

Best Practices

1. Test Naming

# Good
def test_create_case_with_valid_data():
    pass

def test_create_case_with_missing_title_returns_400():
    pass

# Bad
def test1():
    pass

def test_case():
    pass

2. Arrange-Act-Assert Pattern

def test_calculate_priority():
    # Arrange
    finding = {"severity": "high", "confidence": 0.9}
    
    # Act
    priority = calculate_priority(finding)
    
    # Assert
    assert priority == "high"

3. One Assertion Per Test (when possible)

# Good
def test_user_has_correct_username():
    user = create_user("testuser")
    assert user.username == "testuser"

def test_user_has_correct_email():
    user = create_user("testuser", "test@example.com")
    assert user.email == "test@example.com"

# Acceptable when testing related properties
def test_user_properties():
    user = create_user("testuser", "test@example.com")
    assert user.username == "testuser"
    assert user.email == "test@example.com"
    assert user.is_active is True

4. Avoid Test Interdependence

# Bad - tests depend on order
class TestCases:
    case_id = None
    
    def test_create_case(self):
        self.case_id = create_case()
    
    def test_update_case(self):
        update_case(self.case_id)  # Fails if create_case didn't run

# Good - independent tests
class TestCases:
    def test_create_case(self, test_db):
        case_id = create_case()
        assert case_id is not None
    
    def test_update_case(self, test_db, sample_case):
        result = update_case(sample_case.id)
        assert result is True

5. Use Factories for Test Data

from faker import Faker

fake = Faker()

def create_test_user(**kwargs):
    """Factory for creating test users."""
    defaults = {
        "username": fake.user_name(),
        "email": fake.email(),
        "password": fake.password()
    }
    defaults.update(kwargs)
    return User(**defaults)

# Usage
def test_user_creation():
    user = create_test_user(username="specific_name")
    assert user.username == "specific_name"

Debugging Tests

Running in Debug Mode

# Run with print statements visible
pytest -s

# Stop on first failure
pytest -x

# Drop into debugger on failure
pytest --pdb

# Run last failed tests
pytest --lf

Using pdb

def test_complex_logic():
    import pdb; pdb.set_trace()
    
    result = complex_function()
    assert result == expected

CI/CD Integration

Tests run automatically in GitHub Actions:

  • On Pull Request: Unit + Integration tests
  • On Push to main: Full test suite + coverage
  • Nightly: Full suite + performance tests

See CI_CD_GUIDE.md for details.

Troubleshooting

Common Issues

Issue: ImportError: No module named 'backend' Solution: Ensure you’re running from project root and PYTHONPATH is set

Issue: Database tests failing Solution: Ensure PostgreSQL is running: docker-compose up -d postgres

Issue: Async tests timing out Solution: Increase timeout in pytest.ini or use @pytest.mark.timeout(30)

Issue: Fixtures not found Solution: Check conftest.py is in the right location and fixtures are properly defined

Additional Resources