Deployment Guide

Complete guide to deploying Vigil to VMs and managing production environments.

Overview
VM Requirements
Initial Setup
Deployment Methods
Configuration
Monitoring
Maintenance
Troubleshooting

Overview

Vigil can be deployed in several configurations:

Single VM: All services on one machine (development/testing)
Multi-VM: Distributed deployment (staging/production)
Docker Compose: Container-based deployment
Kubernetes: Orchestrated deployment (future)

This guide focuses on VM deployment with Docker Compose.

VM Requirements

Minimum Requirements (Single VM)

OS: Ubuntu 22.04 LTS
CPU: 4 cores
RAM: 8 GB
Disk: 50 GB SSD
Network: Public IP with firewall

Recommended Requirements (Production)

OS: Ubuntu 22.04 LTS
CPU: 8 cores
RAM: 16 GB
Disk: 100 GB SSD
Network: Load balancer + multiple VMs

Multi-VM Architecture

Recommended Production Setup:

┌─────────────────┐
│ Load Balancer   │
│  (Nginx/HAProxy)│
└────────┬────────┘
         │
    ┌────┴────┐
    │         │
┌───▼────┐ ┌─▼──────┐
│ VM 1   │ │ VM 2   │
│Backend │ │Backend │
└────────┘ └────────┘
    │         │
    └────┬────┘
         │
    ┌────▼────┐
    │ VM 3    │
    │Postgres │
    └────┬────┘
         │
    ┌────▼────┐
    │ VM 4    │
    │ Daemon  │
    └─────────┘

VM Roles:

VM 1-2: Backend API (load balanced)
VM 3: PostgreSQL database (with replication)
VM 4: SOC Daemon (autonomous operations)

Initial Setup

1. Prepare VM

# Update system
sudo apt update && sudo apt upgrade -y

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# Verify installation
docker --version
docker-compose --version

# Logout and login again for group changes

2. Create Deployment User

# Create deployer user
sudo adduser deployer
sudo usermod -aG docker deployer
sudo usermod -aG sudo deployer

# Setup SSH key for deployer
sudo su - deployer
mkdir ~/.ssh
chmod 700 ~/.ssh
vi ~/.ssh/authorized_keys  # Paste GitHub Actions public key
chmod 600 ~/.ssh/authorized_keys

3. Prepare Deployment Directory

# As deployer user
sudo mkdir -p /opt/vigil
sudo chown deployer:deployer /opt/vigil
cd /opt/vigil

# Clone repository
git clone https://github.com/your-org/vigil.git .

# Create necessary directories
mkdir -p logs evidence backups

4. Configure Firewall

# Allow SSH
sudo ufw allow 22/tcp

# Allow API
sudo ufw allow 6987/tcp

# Allow Frontend
sudo ufw allow 6988/tcp

# Allow Prometheus metrics
sudo ufw allow 9090/tcp

# Allow Webhook ingestion
sudo ufw allow 8081/tcp

# Enable firewall
sudo ufw enable
sudo ufw status

Deployment Methods

Method 1: Automated Deployment (CI/CD)

Prerequisites:

GitHub Actions configured
SSH keys added to secrets
VM accessible from GitHub

Staging Deployment:

# Push to main branch
git push origin main

# GitHub Actions automatically:
# 1. Runs tests
# 2. Builds images
# 3. Deploys to staging

Production Deployment:

# Tag a release
git tag -a v1.2.3 -m "Release version 1.2.3"
git push origin v1.2.3

# GitHub Actions automatically:
# 1. Creates release
# 2. Builds production images
# 3. Deploys to production
# 4. Runs health checks

Method 2: Manual Deployment

# On deployment machine
cd /opt/vigil

# Pull latest code
git pull origin main

# Set environment variables
export REGISTRY=ghcr.io
export IMAGE_NAME=your-org/vigil
export IMAGE_TAG=latest

# Run deployment script
chmod +x scripts/deploy_to_vm.sh
./scripts/deploy_to_vm.sh production

Method 3: Docker Compose Direct

# On deployment machine
cd /opt/vigil

# Set environment variables in .env file
cp env.example .env
vi .env  # Edit configuration

# Pull images
docker-compose pull

# Start services
docker-compose up -d

# Check status
docker-compose ps

Configuration

Environment Variables

File: /opt/vigil/.env

.env is for bootstrap-only settings. LLM provider keys, SIEM credentials, and other secrets are configured through the web UI (Settings → AI / LLM Providers, Settings → Integrations) and stored encrypted at ~/.vigil/secrets.enc. For server-side / orchestrated deployments where the UI isn’t available at first boot, inject secrets via the Helm chart values (see HELM.md) or your secret manager of choice.

# Database
DATABASE_URL=postgresql://deeptempo:secure_password@postgres:5432/deeptempo_soc
POSTGRES_PASSWORD=secure_password_change_me

# Backend
SECRET_KEY=your-secret-key-here
ENVIRONMENT=production

# LLM gateway (Bifrost) — provider keys are NOT set here
BIFROST_URL=http://bifrost:8080

# Monitoring
SENTRY_DSN=https://your-sentry-dsn
RELEASE_VERSION=v1.2.3

Docker Compose Configuration

File: /opt/vigil/docker-compose.yml

version: '3.8'

services:
  postgres:
    image: postgres:16-alpine
    container_name: deeptempo-postgres
    environment:
      POSTGRES_DB: deeptempo_soc
      POSTGRES_USER: deeptempo
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped

  backend:
    image: ${REGISTRY}/${IMAGE_NAME}-backend:${IMAGE_TAG}
    container_name: deeptempo-backend
    environment:
      - DATABASE_URL
      - SECRET_KEY
      - SENTRY_DSN
      - BIFROST_URL
    volumes:
      # Persist the encrypted secret store across container restarts so
      # provider keys configured via the UI survive image upgrades.
      - vigil_secrets:/root/.vigil
    ports:
      - "6987:6987"
    depends_on:
      - postgres
    restart: unless-stopped

  soc-daemon:
    image: ${REGISTRY}/${IMAGE_NAME}-daemon:${IMAGE_TAG}
    container_name: deeptempo-daemon
    environment:
      - DATABASE_URL
      - BIFROST_URL
    volumes:
      - vigil_secrets:/root/.vigil
    ports:
      - "8081:8081"  # Webhook
      - "9090:9090"  # Metrics
    depends_on:
      - postgres
    restart: unless-stopped

volumes:
  postgres_data:
  vigil_secrets:

Nginx Reverse Proxy (Optional)

File: /etc/nginx/sites-available/vigil

server {
    listen 80;
    server_name app.deeptempo.ai;
    
    # Redirect to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name app.deeptempo.ai;
    
    # SSL certificates
    ssl_certificate /etc/letsencrypt/live/app.deeptempo.ai/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.deeptempo.ai/privkey.pem;
    
    # API
    location /api {
        proxy_pass http://localhost:6987;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    
    # Frontend
    location / {
        root /opt/vigil/clients/web/build;
        try_files $uri $uri/ /index.html;
    }
}

Monitoring

Health Checks

# API health
curl http://localhost:6987/health

# Daemon metrics
curl http://localhost:9090/metrics

# Docker container status
docker-compose ps

# View logs
docker-compose logs -f backend
docker-compose logs -f soc-daemon

Log Management

View Logs:

# All services
docker-compose logs --tail=100

# Specific service
docker-compose logs -f backend

# Save logs to file
docker-compose logs > logs/deployment-$(date +%Y%m%d).log

Log Rotation:

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Metrics Collection

Prometheus Metrics:

# Access metrics endpoint
curl http://localhost:9090/metrics

# Example metrics:
# - http_requests_total
# - http_request_duration_seconds
# - active_cases_total
# - findings_processed_total

Grafana Dashboard (Optional):

# Install Grafana
docker run -d -p 3000:3000 grafana/grafana

# Add Prometheus datasource
# Import Vigil dashboard

Maintenance

Database Backups

Automated Daily Backups:

# Create backup script
vi /opt/vigil/scripts/backup.sh

#!/bin/bash
BACKUP_DIR="/opt/vigil/backups"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/deeptempo_$DATE.sql"

# Create backup
docker-compose exec -T postgres pg_dump -U deeptempo deeptempo_soc > $BACKUP_FILE

# Compress
gzip $BACKUP_FILE

# Remove backups older than 30 days
find $BACKUP_DIR -name "*.sql.gz" -mtime +30 -delete

echo "Backup completed: ${BACKUP_FILE}.gz"

Schedule with Cron:

# Edit crontab
crontab -e

# Add daily backup at 2 AM
0 2 * * * /opt/vigil/scripts/backup.sh >> /opt/vigil/logs/backup.log 2>&1

Database Restore

# Stop services
docker-compose stop backend soc-daemon

# Restore from backup
gunzip -c backups/deeptempo_20260127.sql.gz | docker-compose exec -T postgres psql -U deeptempo deeptempo_soc

# Start services
docker-compose start backend soc-daemon

Update Deployment

# Pull latest images
docker-compose pull

# Recreate containers
docker-compose up -d --force-recreate

# Verify
docker-compose ps

Certificate Renewal (Let’s Encrypt)

# Renew certificates
sudo certbot renew --nginx

# Verify renewal
sudo certbot certificates

# Add to cron for auto-renewal
0 0 1 * * certbot renew --nginx >> /var/log/letsencrypt/renew.log 2>&1

Troubleshooting

Service Not Starting

# Check logs
docker-compose logs backend

# Check configuration
docker-compose config

# Restart service
docker-compose restart backend

# Rebuild and restart
docker-compose up -d --build --force-recreate backend

Database Connection Issues

# Check PostgreSQL status
docker-compose ps postgres

# Test connection
docker-compose exec postgres psql -U deeptempo -d deeptempo_soc -c "SELECT 1;"

# Check network
docker network ls
docker network inspect vigil_default

High Memory Usage

# Check resource usage
docker stats

# Restart memory-heavy service
docker-compose restart soc-daemon

# Increase Docker memory limit
# Edit /etc/docker/daemon.json
{
  "default-ulimits": {
    "memlock": {
      "soft": -1,
      "hard": -1
    }
  }
}

Disk Space Issues

# Check disk usage
df -h

# Clean Docker resources
docker system prune -a --volumes

# Remove old images
docker images | grep vigil | grep -v latest | awk '{print $3}' | xargs docker rmi

# Cleanup old logs
find /opt/vigil/logs -name "*.log" -mtime +7 -delete

Security Hardening

1. Secure SSH

# Disable root login
sudo vi /etc/ssh/sshd_config
# Set: PermitRootLogin no
# Set: PasswordAuthentication no

# Restart SSH
sudo systemctl restart sshd

2. Enable Fail2Ban

# Install fail2ban
sudo apt install fail2ban

# Configure
sudo vi /etc/fail2ban/jail.local
# Add SSH protection
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

3. Database Security

# Use strong passwords
# Enable SSL for PostgreSQL connections
# Restrict PostgreSQL to localhost only

4. API Security

# Use HTTPS only
# Enable rate limiting
# Configure CORS properly
# Use secure session cookies

Rollback Procedures

Quick Rollback

# Stop current deployment
docker-compose down

# Pull previous version
export IMAGE_TAG=1.2.2  # image tags are pushed without the v prefix (git tags still use v)
docker-compose pull
docker-compose up -d

# Verify
curl http://localhost:6987/health

Complete Rollback with Database

# 1. Stop services
docker-compose down

# 2. Restore database backup
gunzip -c backups/pre-v1.2.3.sql.gz | docker-compose exec -T postgres psql -U deeptempo deeptempo_soc

# 3. Revert code
git checkout v1.2.2

# 4. Deploy previous version
export IMAGE_TAG=1.2.2  # image tags are pushed without the v prefix (git tags still use v)
docker-compose up -d

# 5. Verify
docker-compose ps
curl http://localhost:6987/health

Best Practices

Always test in staging first
Backup before major updates
Monitor logs during deployment
Keep rollback images available
Document all configuration changes
Use infrastructure as code (IaC)
Automate routine maintenance
Set up alerts for critical issues

Table of Contents