Kitos 222979574a feat(phase-38): automatic intelligence — OSINT enrichment + stale coverage detection
Tarea 4.1 — OSINT Enrichment:
- Add OsintItem model with source_type, severity, CVSS metadata, review flag
- Add Alembic migration b022 with osint_items table and optimized indexes
- Add osint_enrichment_service with NVD API integration, deduplication, rate limiting
- Add OSINT router: GET /osint/items, /osint/summary, /osint/technique/{id}
- Add POST /osint/items/{id}/review to mark items as reviewed
- Add POST /osint/enrich/{technique_id} for manual single-technique enrichment
- Techniques with new CVEs are automatically flagged review_required=True
- Register weekly enrichment job in APScheduler
- Add NVD_API_KEY config setting for optional increased rate limits

Tarea 4.2 — Stale Coverage Detection:
- Add stale_detection_service that flags techniques with no validated test
  in the last N days, or never-validated but with a coverage status
- Configurable threshold via STALE_THRESHOLD_DAYS setting (default 365)
- Register daily stale detection job in APScheduler
- Only flags techniques not already marked review_required
2026-02-17 17:47:47 +01:00

Aegis — MITRE ATT&CK Coverage Platform

Aegis is a comprehensive platform for tracking and managing security coverage against the MITRE ATT&CK framework. It enables security teams to document, validate, and visualize their defensive capabilities against known adversary techniques through a structured Red Team / Blue Team validation workflow.

Features

Core (V1)

  • MITRE ATT&CK Integration — Automatic synchronization via TAXII 2.0 (with GitHub fallback), scheduled every 24h
  • Red/Blue Validation Workflow — Structured dual-validation lifecycle: draft → red_executing → blue_evaluating → in_review → validated/rejected
  • Dual Validation — Independent Red Lead / Blue Lead approval before finalization
  • Coverage Tracking — Per-technique status (validated, partial, not covered, in progress)
  • Evidence Storage — Secure evidence with SHA256 integrity, separated by team (red/blue)
  • Role-Based Access Control — Granular permissions for 6 roles (admin, red_tech, blue_tech, red_lead, blue_lead, viewer)

Enhanced (V2)

  • Test Template Catalog — Import from Atomic Red Team, CALDERA, LOLBAS, GTFOBins; create custom templates; bulk activate/deactivate
  • In-App Notifications — Real-time notification bell with polling and automatic state-change alerts
  • Reports & Export — Coverage summary, test results, and remediation reports in JSON and CSV
  • Remediation Tracking — Step-by-step remediation assignments with status tracking
  • Metrics Dashboard — Pipeline funnel, team activity, validation rates

Advanced (V3)

  • Multi-Source Data Import — Sigma, CALDERA, LOLBAS, GTFOBins, D3FEND, MITRE CTI threat actors, compliance mappings (NIST 800-53, CIS Controls v8)
  • Detection Rule Tracking — Import and evaluate Sigma/Elastic detection rules per test
  • ATT&CK Heatmap — Interactive Navigator-style heatmap with layers, filters, and export
  • Threat Actor Intelligence — Track intrusion sets and their technique coverage
  • Campaign Management — Group tests into campaigns with dependencies, scheduling, and recurring execution
  • Compliance Mapping — Map NIST 800-53 and CIS Controls v8 to ATT&CK techniques with gap analysis
  • Granular Scoring — 0100 scoring for techniques, tactics, actors, and organization with configurable weights
  • Operational Metrics — MTTD, MTTR, detection efficacy, alert fidelity, coverage velocity
  • Executive Dashboard — High-level KPIs for leadership (leads + admin)
  • Temporal Comparison — Coverage snapshots with historical comparison and trend analysis
  • Auto Re-testing — Automatic retest creation after remediation completion (configurable limit)
  • Performance Optimizations — Score caching, lazy loading, React.memo, database index optimization

Red Team / Blue Team Validation Flow

┌──────┐    ┌──────────────┐    ┌─────────────────┐    ┌───────────┐
│ DRAFT│───▶│RED_EXECUTING │───▶│ BLUE_EVALUATING  │───▶│ IN_REVIEW │
└──────┘    └──────────────┘    └─────────────────┘    └─────┬─────┘
                                                             │
                                         ┌───────────────────┤
                                         ▼                   ▼
                                   ┌──────────┐       ┌──────────┐
                                   │ REJECTED │       │VALIDATED │
                                   └────┬─────┘       └──────────┘
                                        │                    │
                                        └──▶ Back to DRAFT   ├──▶ Remediation
                                                             └──▶ Auto Re-test

States

State Description Who acts
draft Created, pending execution Red Tech
red_executing Red Team documents attack & uploads evidence Red Tech
blue_evaluating Blue Team documents detection & uploads evidence Blue Tech
in_review Both managers review evidence Red Lead, Blue Lead
validated Approved by both managers — (terminal)
rejected Rejected — returns to draft for redo Red/Blue Lead can reopen

Dual Validation

Both Red Lead and Blue Lead must independently vote:

  • Both approve → test moves to validated
  • Either rejects → test moves to rejected
  • One votes, other pending → stays in in_review

User Roles

Role Description Capabilities
admin Full system access Everything
red_tech Red team technician Create tests, document attacks, upload red evidence
blue_tech Blue team technician Document detection, upload blue evidence
red_lead Red team lead Validate/reject the red side of tests
blue_lead Blue team lead Validate/reject the blue side of tests
viewer Read-only access View all data

Tech Stack

  • Backend: FastAPI (Python 3.11)
  • Database: PostgreSQL 15 with UUID primary keys and JSONB columns
  • Object Storage: MinIO (S3-compatible)
  • ORM: SQLAlchemy with Alembic migrations (18 migration files)
  • Frontend: React 19 + TypeScript + Vite 7 + Tailwind CSS v4 + TanStack Query + TanStack Virtual
  • Scheduler: APScheduler (MITRE sync, Intel scan, Notification cleanup, Snapshots, Recurring campaigns)
  • Charts: Recharts

Quick Start

Prerequisites

  • Docker and Docker Compose
  • Git
  • Linux / macOS (or WSL on Windows)

Production Deployment

The recommended way to deploy Aegis in production:

git clone <repository-url>
cd Aegis
chmod +x scripts/install.sh
./scripts/install.sh

The interactive install wizard will guide you through:

  1. Domain configuration — your domain or IP, protocol (HTTP/HTTPS), and port
  2. Admin account — custom username and password (or auto-generated secure password)
  3. Database — name, user, and password (or auto-generated)
  4. Session duration — JWT token expiry (default: 15 minutes)
  5. MITRE ATT&CK sync — optionally import ~700 techniques on first run

The script automatically generates cryptographically secure random secrets for SECRET_KEY, database password, and MinIO credentials. A summary with all credentials is displayed at the end of the installation.

Access the application at the URL shown in the installation summary.

Development Setup

For local development with hot-reload:

git clone <repository-url>
cd Aegis
docker-compose up -d
./scripts/init.sh

Access at http://localhost:5173 (frontend dev server) and http://localhost:8000/docs (API docs).

Authentication

JWT-based authentication with HttpOnly cookies. Admin credentials are configured during installation:

  • If you set a custom password in the wizard, use that.
  • If you left it blank, a secure random password was auto-generated and displayed in the installation summary and backend logs.

To retrieve auto-generated credentials after installation:

docker logs aegis-backend 2>&1 | grep -A 5 "Initial Admin User Created"

Note: Passwords must meet complexity requirements: minimum 12 characters with at least one uppercase letter, one lowercase letter, one digit, and one special character.

Importing Data Sources

On startup, the backend automatically seeds the initial data sources (Atomic Red Team, SigmaHQ, CALDERA, LOLBAS, GTFOBins, D3FEND). You can then sync each source from the UI:

  1. Navigate to Data Sources in the sidebar
  2. Click Sync on each data source to import its content
  3. Trigger a MITRE ATT&CK Sync from the System page

Alternatively, use the API:

# Sync MITRE ATT&CK techniques
curl -X POST http://your-server/api/v1/system/sync-mitre -H "Authorization: Bearer $TOKEN"

# Sync all data sources at once
curl -X POST http://your-server/api/v1/data-sources/sync-all -H "Authorization: Bearer $TOKEN"

Production Considerations

  • HTTPS/TLS: For internet-facing deployments, place a reverse proxy with TLS in front (e.g., Traefik, Caddy, or Nginx with Let's Encrypt). Uncomment the HSTS header in frontend/nginx.conf once HTTPS is configured.
  • Backups: Set up regular PostgreSQL backups: docker exec aegis-postgres pg_dump -U postgres attackdb > backup.sql
  • Updates: To update, pull the latest code and run: docker compose -f docker-compose.prod.yml up -d --build
  • Firewall: Only expose port 80/443. All other services (DB, MinIO, backend) are internal only.
  • Reconfigure: Run ./scripts/install.sh again to reconfigure the environment (domain, credentials, etc.).

Configuring Scoring Weights

Default scoring weights can be adjusted via environment variables:

SCORING_WEIGHT_TESTS=40
SCORING_WEIGHT_DETECTION_RULES=20
SCORING_WEIGHT_D3FEND=15
SCORING_WEIGHT_FRESHNESS=15
SCORING_WEIGHT_PLATFORM_DIVERSITY=10

Or at runtime via the admin API — see docs/SCORING.md.

Configuring Campaigns

  1. Navigate to Campaigns in the sidebar
  2. Create a new campaign (custom or from a threat actor)
  3. Add tests with dependencies and phases
  4. Optionally enable recurring scheduling (weekly, monthly, quarterly)

Services

Service Port Description
Frontend 5173 React dev server (Vite)
Backend 8000 FastAPI REST API
PostgreSQL 5433 Database
MinIO API 9000 S3-compatible object storage
MinIO Console 9001 MinIO web interface

Navigation

📊 Dashboard
📊 Executive Dashboard          (leads + admin)
🔲 ATT&CK Matrix                (advanced heatmap)
🧪 Tests
  ├─ All Tests
  ├─ My Pending Tasks
  └─ Test Catalog
📋 Campaigns
👤 Threat Actors
📜 Compliance
📈 Comparison                    (leads + admin)
📄 Reports
⚙️ System                       (admin only)
  ├─ Data Sources               (sync Atomic, Sigma, CALDERA, LOLBAS, GTFOBins, D3FEND)
  ├─ MITRE Sync                 (ATT&CK sync, intel scan, template management)
  ├─ Users
  └─ Audit Log

API Documentation

Interactive API documentation is available in development only (disabled in production for security):

In production (AEGIS_ENV=production), these endpoints are disabled. Use the development environment or refer to docs/API.md.

API Endpoints

Group Prefix Key Operations
Auth /api/v1/auth Login, get current user
Techniques /api/v1/techniques CRUD, list with filters, mark reviewed
Tests /api/v1/tests Full Red/Blue workflow, remediation, retest chain
Test Templates /api/v1/test-templates CRUD, stats, toggle active, bulk activate/deactivate
Evidence /api/v1/tests/{id}/evidence Upload evidence, get presigned URLs
Campaigns /api/v1/campaigns CRUD, scheduling, history
Threat Actors /api/v1/threat-actors CRUD, technique mappings
Detection Rules /api/v1/detection-rules List, filter by source/technique
D3FEND /api/v1/d3fend Defensive techniques and mappings
Compliance /api/v1/compliance Frameworks, controls, gaps
Scores /api/v1/scores Technique/tactic/actor/org scores, config
Operational Metrics /api/v1/metrics/operational MTTD, MTTR, trends, team breakdown
Heatmap /api/v1/heatmap ATT&CK Navigator-style data
Snapshots /api/v1/snapshots Create, compare, list snapshots
Reports /api/v1/reports Coverage, results, remediation exports
Notifications /api/v1/notifications List, read, mark all read
Metrics /api/v1/metrics Summary, by-tactic, pipeline, team activity
System /api/v1/system MITRE sync, import, scheduler status
Users /api/v1/users User CRUD (admin)
Audit Logs /api/v1/audit-logs Audit trail (admin)
Data Sources /api/v1/data-sources Data source management (admin)

See docs/API.md for the full endpoint reference.

Configuration

All variables are configured automatically by scripts/install.sh. For manual setup, copy .env.example to .env and fill in the values.

Required (production)

Variable Description
SECRET_KEY JWT signing key — required in production (app refuses to start without it). Generate with openssl rand -hex 32
DB_PASSWORD PostgreSQL password
MINIO_SECRET_KEY MinIO secret key

Security & Auth

Variable Default Description
AEGIS_ENV Set to production to enforce security settings
ADMIN_USERNAME admin Initial admin account username
ADMIN_PASSWORD (auto-generated) Initial admin password. If empty, a secure random password is generated and shown in logs
ACCESS_TOKEN_EXPIRE_MINUTES 15 JWT token lifetime in minutes
CORS_ORIGINS http://localhost:5173 Comma-separated list of allowed frontend origins

Infrastructure

Variable Default Description
DB_USER postgres PostgreSQL username
DB_NAME attackdb PostgreSQL database name
MINIO_ENDPOINT minio:9000 MinIO server address
MINIO_ACCESS_KEY minioadmin MinIO access key
MINIO_BUCKET evidence MinIO bucket for evidence files
MINIO_SECURE false Set to true if MinIO is behind TLS
FRONTEND_PORT 80 Port exposed by the frontend container

Scoring Weights

Variable Default Description
SCORING_WEIGHT_TESTS 40 Weight for test validation component
SCORING_WEIGHT_DETECTION_RULES 20 Weight for detection rules component
SCORING_WEIGHT_D3FEND 15 Weight for D3FEND coverage component
SCORING_WEIGHT_FRESHNESS 15 Weight for freshness component
SCORING_WEIGHT_PLATFORM_DIVERSITY 10 Weight for platform diversity component
MAX_RETEST_COUNT 3 Max automatic retests per original test

Security

Aegis includes several security hardening measures:

  • Authentication: JWT tokens stored in HttpOnly/Secure/SameSite cookies (immune to XSS theft). Token revocation via in-memory blacklist on logout.
  • Rate limiting: Login endpoint limited to 5 attempts per minute per IP (via slowapi).
  • Password policy: Minimum 12 characters with uppercase, lowercase, digit, and special character.
  • CORS: Configurable origins via CORS_ORIGINS environment variable. Restrictive method and header lists.
  • Nginx security headers: CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy.
  • Non-root container: Backend runs as appuser (UID 1001), not root.
  • File uploads: 50 MB size limit, extension whitelist, filename sanitization.
  • ZIP imports: Zip Slip (path traversal) and Zip Bomb (size/entry limit) protection.
  • API surface: Swagger UI, ReDoc, and OpenAPI schema disabled in production.
  • Health endpoint: Restricted to internal networks via Nginx ACL.
  • Input sanitization: LIKE wildcard escaping in all search queries; Pydantic schemas on all endpoints.
  • XML parsing: Uses defusedxml to prevent Billion Laughs / XXE attacks.
  • Error handling: Internal exception details are logged server-side only, never exposed to clients.

Project Structure

Aegis/
├── docker-compose.yml
├── docker-compose.prod.yml
├── docs/
│   ├── API.md                     # Full API endpoint reference
│   ├── ARCHITECTURE.md            # System architecture and DB schema
│   ├── DATA_SOURCES.md            # External data source documentation
│   └── SCORING.md                 # Scoring system and metrics
├── backend/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── alembic.ini
│   ├── alembic/versions/          # b001b018 migration files
│   ├── pytest.ini
│   └── app/
│       ├── main.py                # FastAPI app with all routers + lifespan
│       ├── config.py              # Settings from environment
│       ├── database.py            # SQLAlchemy engine + session (lazy init)
│       ├── storage.py             # MinIO/S3 helpers
│       ├── auth.py                # Password hashing + JWT tokens
│       ├── models/                # 18 model files (SQLAlchemy ORM)
│       ├── schemas/               # Pydantic request/response schemas
│       ├── routers/               # 21 API routers
│       ├── services/              # 20 business logic services
│       ├── dependencies/          # Auth dependencies (get_current_user, require_role)
│       └── jobs/
│           └── mitre_sync_job.py  # APScheduler: 5 background jobs
├── frontend/src/
│   ├── App.tsx                    # Routes with lazy loading + role protection
│   ├── api/                       # 22 API client modules (Axios + TanStack Query)
│   ├── components/
│   │   ├── Layout.tsx             # Sidebar + header + NotificationBell
│   │   ├── Sidebar.tsx            # Role-aware collapsible navigation
│   │   ├── heatmap/               # ATT&CK heatmap (6 components)
│   │   ├── compliance/            # Compliance UI (gauge, controls table)
│   │   └── test-detail/           # Test detail sub-components
│   ├── hooks/
│   │   └── useDebounce.ts         # Debounce hook for search inputs
│   ├── context/
│   │   └── AuthContext.tsx         # Auth state management
│   └── pages/                     # 21 page components
└── backend/tests/
    ├── conftest.py                # SQLite test DB with JSONB/UUID compatibility
    ├── fixtures/                  # YAML/TOML/JSON test fixtures
    ├── test_data_sources.py       # Data source parsing tests
    ├── test_scoring_and_compliance.py # Scoring + metrics + compliance tests
    ├── test_campaigns_and_snapshots.py # Campaign, snapshot, and retest tests
    ├── test_workflow.py           # Red/Blue workflow tests
    ├── test_templates_crud.py     # Template CRUD tests
    ├── test_metrics_v2.py         # V2 metrics tests
    └── test_integration_v2.py     # Full integration E2E tests

Development

Running Migrations

docker exec aegis-backend alembic upgrade head
docker exec aegis-backend alembic revision --autogenerate -m "description"
docker exec aegis-backend alembic downgrade -1

Running Tests

# Run all V3 tests inside the container (recommended)
docker exec aegis-backend python -m pytest tests/ -v --tb=short

# Run specific test suites
docker exec aegis-backend python -m pytest tests/test_data_sources.py -v
docker exec aegis-backend python -m pytest tests/test_scoring_and_compliance.py -v
docker exec aegis-backend python -m pytest tests/test_campaigns_and_snapshots.py -v

# Skip integration tests (require network)
docker exec aegis-backend python -m pytest tests/ -v -m "not integration"

Generating Reports

# Coverage summary (JSON)
GET /api/v1/reports/coverage-summary

# Coverage CSV export
GET /api/v1/reports/coverage-csv

# Compliance gap analysis
GET /api/v1/compliance/{framework_id}/gaps

Further Documentation

  • Architecture — Database schema, service layer, state machine diagrams
  • Data Sources — All external data sources with import instructions
  • Scoring — Scoring system explained with examples and configuration
  • API Reference — Full endpoint documentation

License

This project is proprietary software. All rights reserved.

Disclaimer

This project has been developed with the assistance of Cursor and Claude Opus 4.6 (Anthropic).

Description
El proyecto es una plataforma interna de gestión de cobertura MITRE ATT&CK con FastAPI, PostgreSQL, MinIO y React.
Readme 6.2 MiB
Languages
Python 65.1%
TypeScript 33.1%
Shell 1%
HTML 0.5%
CSS 0.2%