CI/CD Guide

Complete guide to the CI/CD pipeline for Vigil.

Overview
GitHub Actions Workflows
Pipeline Stages
Secrets Configuration
Deployment Process
Monitoring & Alerts

Overview

Vigil uses GitHub Actions for CI/CD with three main workflows:

ci-cd.yml: Main pipeline for PRs and pushes
release.yml: Production deployment on tags
nightly.yml: Scheduled comprehensive testing

Pipeline Architecture

┌──────────────┐
│ Pull Request │
│  or Push     │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│   Linting    │
│ Backend/FE   │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│  Unit Tests  │
│ Backend/FE   │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│ Integration  │
│    Tests     │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│   Security   │
│   Scanning   │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│ Build Docker │
│    Images    │
└──────┬───────┘
       │
       ▼ (main branch only)
┌──────────────┐
│Deploy Staging│
└──────────────┘

GitHub Actions Workflows

Main CI/CD Workflow

File: .github/workflows/ci-cd.yml

Triggers:

Pull requests to main or develop
Pushes to main or develop

Jobs:

lint-backend - Python linting (flake8, black, isort, mypy)
lint-frontend - TypeScript/React linting (ESLint)
lint-docker - Dockerfile linting (hadolint)
test-unit-backend - Backend unit tests with coverage
test-unit-frontend - Frontend unit tests
test-integration - Integration tests with PostgreSQL
security-scan-python - Bandit SAST scanning
security-scan-npm - NPM audit
build-images - Build and push Docker images
build-frontend - Build frontend static assets
scan-images - Trivy vulnerability scanning
deploy-staging - Deploy to staging (main branch only) — currently commented out in ci-cd.yml; no staging deploy runs.

Release Workflow

File: .github/workflows/release.yml

Triggers:

Tags matching v*.*.* (e.g., v1.0.0)

Jobs:

version - Resolve the vX.Y.Z version from the pushed tag
build-backend - Build & push ghcr.io/vigil-soc/vigil-backend (also reused by the llm-worker)
build-daemon - Build & push ghcr.io/vigil-soc/vigil-daemon
smoke-test - Pull the published images and verify they start
update-release - Append the image digests to the GitHub Release notes

No deployment runs here. release.yml publishes images only; it does not SSH anywhere or deploy. The GitHub Release object itself is created by release-please.yml, not this workflow. See Stage 7 for the (currently disabled) manual deploy path.

Nightly Tests

File: .github/workflows/nightly.yml

Triggers:

Scheduled: 2 AM UTC daily
Manual: workflow_dispatch

Jobs:

full-test-suite - Complete test coverage
performance-tests - Performance regression testing
security-audit - pip-audit vulnerability scan
migration-tests - Database migration validation

security-audit audits requirements.lock, not requirements.txt, so it sees the versions that actually install. Findings are reported, not fatal — but a pip-audit that fails to run is, because it writes no report at all when it errors and the job used to pass on that silence.

Pipeline Stages

Stage 1: Linting & Code Quality

Backend Linting:

- flake8: Style checking          (gates)
- black: Code formatting          (gates)
- isort: Import sorting           (gates)
- import-linter: Layering         (gates)
- mypy: Type checking             (advisory)

Versions are pinned in requirements-dev.txt, which is also what .pre-commit-config.yaml mirrors, so a local pre-commit run and CI agree.

Frontend Linting:

- ESLint: JavaScript/TypeScript linting
- Prettier: Code formatting (optional)

Exit Criteria:

All linters pass
No critical issues found

Stage 2: Unit Tests

Backend:

pytest tests/unit/ tests/security/ -v --cov=services --cov=core

Frontend:

npm run test:unit

Exit Criteria:

All tests pass
Coverage thresholds met (80% backend, 70% frontend)

Stage 3: Integration Tests

Setup:

PostgreSQL container via GitHub Actions services
Test database: deeptempo_test

Execution:

pytest tests/integration/ -v --cov

Exit Criteria:

All integration tests pass
No database errors
External API mocks working

Stage 4: Security Scanning

Python (Bandit):

bandit -r services/ core/ -f json -o bandit-report.json

NPM:

npm audit --audit-level=moderate

Exit Criteria: neither tool gates — both are continue-on-error, so this stage cannot fail the pipeline today. It produces a report, uploaded as an artifact, for triage by hand.

Stage 5: Build Images

Backend Image:

FROM python:3.10-slim
COPY . /app
RUN pip install -r requirements.txt
CMD ["uvicorn", "services.api.main:app", "--host", "0.0.0.0"]

Daemon Image:

FROM python:3.10-slim
COPY . /app
CMD ["python", "-m", "daemon.main"]

Optimization:

Docker BuildKit for caching
Multi-stage builds
Layer caching via GitHub Actions

Exit Criteria:

Images build successfully
Tagged with branch and SHA
Pushed to GitHub Container Registry

Stage 6: Container Scanning

Trivy Scanner:

trivy image --severity CRITICAL,HIGH ghcr.io/user/image:tag

Exit Criteria:

No critical vulnerabilities
Scan results uploaded to GitHub Security

Stage 7: Deployment

Not currently enabled. CI builds, scans, and publishes images, but no automated deployment runs — neither staging nor production. The release.yml workflow only builds and publishes images. The commands below describe the manual deploy path via scripts/deploy_to_vm.sh (and the intended automated wiring for when deployment is turned on, post-1.0). The script’s default IMAGE_NAME is stale — override it, e.g. IMAGE_NAME=vigil-soc/vigil, so it pulls the published images.

Staging (intended: automatic on main):

./scripts/deploy_to_vm.sh staging

Production (manual, against release-tagged images):

./scripts/deploy_to_vm.sh production

Exit Criteria:

Services healthy
Smoke tests pass
Rollback available

Secrets Configuration

Required Secrets

Configure in Settings → Secrets → Actions:

SSH & VM Access

Only needed for the manual scripts/deploy_to_vm.sh path (see Stage 7). No wired workflow consumes these today — leave them unset until automated deployment is turned on.

SSH_PRIVATE_KEY         # SSH key for VM access
STAGING_VM_HOST         # Staging VM hostname/IP
STAGING_VM_USER         # SSH username (staging)
PROD_VM_HOST            # Production VM hostname/IP
PROD_VM_USER            # SSH username (production)

Container Registry

GITHUB_TOKEN            # Auto-provided by GitHub Actions

API Keys

ANTHROPIC_API_KEY       # Claude API key for testing

Notifications

SLACK_WEBHOOK_URL       # Slack webhook for deployment notifications

Monitoring

SENTRY_DSN              # Sentry error tracking DSN

Setting Secrets

# Via GitHub CLI
gh secret set SSH_PRIVATE_KEY < ~/.ssh/id_rsa
gh secret set SLACK_WEBHOOK_URL -b "https://hooks.slack.com/..."

# Via GitHub Web UI
1. Go to repository Settings
2. Click Secrets → Actions
3. Click "New repository secret"
4. Add name and value

Deployment Process

Staging Deployment

Automatic on main branch merge:

All tests pass
Images built and scanned
SSH to staging VM
Pull latest images
Run migrations
Rolling restart services
Health checks
Slack notification

Manual Rollback:

ssh user@staging-host
cd /opt/vigil
docker-compose down
git checkout previous-commit
docker-compose up -d

Production Deployment

Not automated. Pushing a version tag runs release.yml, which builds and publishes images only — it does not deploy. Deploying those images to a VM is a separate, manual step (Stage 7 / scripts/deploy_to_vm.sh) that is not currently wired into CI.

What a version tag (e.g. v1.2.3) actually triggers (release.yml):

Build & push vigil-backend and vigil-daemon images to GHCR
Trivy-scan the published images
Smoke-test that the images start
Annotate the GitHub Release with the image digests

Creating a Release — releases are driven by release-please, not by hand-pushing tags:

Merge the open release PR that release-please maintains on main.
On merge, release-please pushes the vX.Y.Z tag and creates the GitHub Release; that tag push triggers release.yml (above).

See RELEASING.md for the full process.

Rollback Procedure

The automatic-rollback step below is illustrative — it belongs to the not-yet-wired VM deploy path, not an active workflow. Manual rollback is the real procedure today.

Automatic Rollback (on health check failure):

- name: Rollback on failure
  if: failure()
  run: |
    ssh $VM_USER@$VM_HOST '
      cd /opt/vigil &&
      docker-compose down &&
      docker-compose up -d --force-recreate
    '

Manual Rollback:

# SSH to production
ssh prod-user@prod-host

# View available images
docker images | grep vigil

# Rollback to previous version
cd /opt/vigil
export IMAGE_TAG=1.2.2  # Previous version (image tags are pushed without the v prefix; git tags still use v)
docker-compose up -d --force-recreate

Performance Optimization

Caching Strategy

Docker Layer Caching:

- name: Build and push
  uses: docker/build-push-action@v5
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max

Dependency Caching:

- name: Cache pip dependencies
  uses: actions/cache@v3
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}

NPM Caching:

- name: Setup Node.js
  uses: actions/setup-node@v5
  with:
    cache: 'npm'
    cache-dependency-path: clients/web/package-lock.json

Parallel Execution

Jobs run in parallel when possible:

All linting jobs run concurrently
Unit tests (backend/frontend) run concurrently
Security scans run in parallel

Estimated Pipeline Duration:

Linting: 2-3 minutes
Tests: 5-7 minutes
Build: 8-10 minutes
Deploy: 3-5 minutes
Total: ~15 minutes (with caching)

Monitoring & Alerts

Deployment Notifications

Slack Integration:

- name: Notify Slack
  uses: slackapi/slack-github-action@v1
  with:
    payload: |
      {
        "text": "Deployment Successful",
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "*Version:* ${{ github.ref_name }}"
            }
          }
        ]
      }

Notification Triggers:

Staging deployment success/failure
Production deployment success/failure
Nightly test failures
Security scan failures

GitHub Status Checks

Required Checks (configure in branch protection):

lint-backend
lint-frontend
test-unit-backend
test-integration
security-scan-python

Branch Protection Rules:

Settings → Branches → Branch protection rules
✓ Require status checks to pass before merging
✓ Require branches to be up to date before merging
✓ Require linear history

Sentry Integration

Error Tracking:

Automatic error capture in production
Performance monitoring
Release tracking

Configuration:

# backend/monitoring.py
sentry_sdk.init(
    dsn=os.getenv("SENTRY_DSN"),
    environment=os.getenv("ENVIRONMENT"),
    release=os.getenv("RELEASE_VERSION")
)

Troubleshooting

Pipeline Failures

Build Failures:

# Check logs
gh run view --log

# Rerun failed jobs
gh run rerun --failed

Test Failures:

# Run tests locally
pytest tests/unit/ -v --lf  # Run last failed

# Check test logs in GitHub Actions
Actions → Failed workflow → test-unit-backend → Logs

Deployment Failures:

# SSH to VM and check logs
ssh user@host
cd /opt/vigil
docker-compose logs --tail=100 backend

# Check service status
docker-compose ps

Common Issues

Issue: Cache not working Solution: Clear cache and rerun workflow

Issue: Docker build timeout Solution: Optimize Dockerfile, reduce image size

Issue: Test database connection failed Solution: Ensure PostgreSQL service is healthy in workflow

Issue: Permission denied on deployment Solution: Check SSH key is correct and VM user has docker permissions

Best Practices

1. Semantic Versioning

Use semantic versioning for releases:

v1.0.0: Major release
v1.1.0: Minor feature addition
v1.1.1: Bug fix/patch

2. Commit Messages

Follow conventional commits:

feat: Add new case search functionality
fix: Resolve authentication token expiry issue
docs: Update API documentation
test: Add unit tests for Claude service
ci: Optimize Docker build caching

3. Pull Request Workflow

Create feature branch from develop
Make changes and commit
Push and create PR to develop
Wait for CI checks to pass
Request code review
Merge to develop
Periodic merges from develop to main
Tag releases from main

4. Emergency Hotfixes

# Create hotfix branch from main
git checkout main
git checkout -b hotfix/critical-bug

# Make fix and test locally
pytest tests/

# Commit and push
git commit -m "fix: Resolve critical bug"
git push origin hotfix/critical-bug

# Create PR directly to main
# After merge, tag new patch version
git tag v1.2.4
git push origin v1.2.4

Table of Contents

Overview

Pipeline Architecture

GitHub Actions Workflows

Main CI/CD Workflow

Release Workflow

Nightly Tests

Pipeline Stages

Stage 1: Linting & Code Quality

Stage 2: Unit Tests

Stage 3: Integration Tests

Stage 4: Security Scanning

Stage 5: Build Images

Stage 6: Container Scanning

Stage 7: Deployment

Secrets Configuration

Required Secrets

SSH & VM Access

Container Registry

API Keys

Notifications

Monitoring

Setting Secrets

Deployment Process

Staging Deployment

Production Deployment

Rollback Procedure

Performance Optimization

Caching Strategy

Parallel Execution

Monitoring & Alerts

Deployment Notifications

GitHub Status Checks

Sentry Integration

Troubleshooting

Pipeline Failures

Common Issues

Best Practices

1. Semantic Versioning

2. Commit Messages

3. Pull Request Workflow

4. Emergency Hotfixes

Additional Resources