Aegis/README.md

# Aegis — MITRE ATT&CK Coverage Platform

Continuous integration (lint + tests against PostgreSQL and Redis) is defined in [`.github/workflows/ci.yml`](.github/workflows/ci.yml).

Aegis is a comprehensive platform for tracking and managing security coverage against the MITRE ATT&CK framework. It enables security teams to document, validate, and visualize their defensive capabilities against known adversary techniques through a structured Red Team / Blue Team validation workflow.

## Features

### Core (V1)
- **MITRE ATT&CK Integration** — Automatic synchronization via TAXII 2.0 (with GitHub fallback), scheduled every 24h
- **Red/Blue Validation Workflow** — Structured dual-validation lifecycle: draft → red_executing → blue_evaluating → in_review → validated/rejected
- **Dual Validation** — Independent Red Lead / Blue Lead approval before finalization
- **Coverage Tracking** — Per-technique status (validated, partial, not covered, in progress)
- **Evidence Storage** — Secure evidence with SHA256 integrity, separated by team (red/blue)
- **Role-Based Access Control** — Granular permissions for 6 roles (admin, red_tech, blue_tech, red_lead, blue_lead, viewer)

### Enhanced (V2)
- **Test Template Catalog** — Import from Atomic Red Team, CALDERA, LOLBAS, GTFOBins; create custom templates; bulk activate/deactivate
- **In-App Notifications** — Real-time notification bell with polling and automatic state-change alerts
- **Reports & Export** — Coverage summary, test results, and remediation reports in JSON and CSV
- **Remediation Tracking** — Step-by-step remediation assignments with status tracking
- **Metrics Dashboard** — Pipeline funnel, team activity, validation rates

### Advanced (V3)
- **Multi-Source Data Import** — Sigma, CALDERA, LOLBAS, GTFOBins, D3FEND, MITRE CTI threat actors, compliance mappings (NIST 800-53, CIS Controls v8)
- **Detection Rule Tracking** — Import and evaluate Sigma/Elastic detection rules per test
- **ATT&CK Heatmap** — Interactive Navigator-style heatmap with layers, filters, and export
- **Threat Actor Intelligence** — Track intrusion sets and their technique coverage
- **Campaign Management** — Group tests into campaigns with dependencies, scheduling, and recurring execution
- **Compliance Mapping** — Map NIST 800-53 and CIS Controls v8 to ATT&CK techniques with gap analysis
- **Granular Scoring** — 0–100 scoring for techniques, tactics, actors, and organization with configurable weights
- **Operational Metrics** — MTTD, MTTR, detection efficacy, alert fidelity, coverage velocity
- **Executive Dashboard** — High-level KPIs for leadership (leads + admin)
- **Temporal Comparison** — Coverage snapshots with historical comparison and trend analysis
- **Auto Re-testing** — Automatic retest creation after remediation completion (configurable limit)
- **Performance Optimizations** — Score caching, lazy loading, React.memo, database index optimization

## Red Team / Blue Team Validation Flow

```
┌──────┐    ┌──────────────┐    ┌─────────────────┐    ┌───────────┐
│ DRAFT│───▶│RED_EXECUTING │───▶│ BLUE_EVALUATING  │───▶│ IN_REVIEW │
└──────┘    └──────────────┘    └─────────────────┘    └─────┬─────┘
                                                             │
                                         ┌───────────────────┤
                                         ▼                   ▼
                                   ┌──────────┐       ┌──────────┐
                                   │ REJECTED │       │VALIDATED │
                                   └────┬─────┘       └──────────┘
                                        │                    │
                                        └──▶ Back to DRAFT   ├──▶ Remediation
                                                             └──▶ Auto Re-test
```

### States

| State | Description | Who acts |
|-------|-------------|----------|
| `draft` | Created, pending execution | Red Tech |
| `red_executing` | Red Team documents attack & uploads evidence | Red Tech |
| `blue_evaluating` | Blue Team documents detection & uploads evidence | Blue Tech |
| `in_review` | Both managers review evidence | Red Lead, Blue Lead |
| `validated` | Approved by both managers | — (terminal) |
| `rejected` | Rejected — returns to draft for redo | Red/Blue Lead can reopen |

### Dual Validation

Both Red Lead and Blue Lead must independently vote:
- **Both approve** → test moves to `validated`
- **Either rejects** → test moves to `rejected`
- **One votes, other pending** → stays in `in_review`

## User Roles

| Role | Description | Capabilities |
|------|-------------|-------------|
| `admin` | Full system access | Everything |
| `red_tech` | Red team technician | Create tests, document attacks, upload red evidence |
| `blue_tech` | Blue team technician | Document detection, upload blue evidence |
| `red_lead` | Red team lead | Validate/reject the red side of tests |
| `blue_lead` | Blue team lead | Validate/reject the blue side of tests |
| `viewer` | Read-only access | View all data |

## Tech Stack

- **Backend**: FastAPI (Python 3.11) — Clean Modular Monolith with domain entities, services, and repository pattern
- **Database**: PostgreSQL 16 with UUID primary keys and JSONB columns
- **Object Storage**: MinIO (S3-compatible)
- **ORM**: SQLAlchemy 2.x with Alembic migrations
- **Frontend**: React 19 + TypeScript + Vite 7 + Tailwind CSS v4 + TanStack Query + TanStack Virtual
- **Cache / Token Store**: Redis (token blacklist, score caching)
- **Scheduler**: APScheduler (MITRE sync, Intel scan, Notification cleanup, Snapshots, Recurring campaigns)
- **Testing**: Pytest (367+ tests), Ruff (linting), GitHub Actions CI
- **Charts**: Recharts

## Quick Start

### Prerequisites

- Docker and Docker Compose
- Git
- Linux / macOS (or WSL on Windows)

### Production Deployment

The recommended way to deploy Aegis in production:

```bash
git clone <repository-url>
cd Aegis
chmod +x scripts/install.sh
./scripts/install.sh
```

The interactive install wizard will guide you through:
1. **Domain configuration** — your domain or IP, protocol (HTTP/HTTPS), and port
2. **Admin account** — custom username and password (or auto-generated secure password)
3. **Database** — name, user, and password (or auto-generated)
4. **Session duration** — JWT token expiry (default: 15 minutes)
5. **MITRE ATT&CK sync** — optionally import ~700 techniques on first run

The script automatically generates cryptographically secure random secrets for `SECRET_KEY`, database password, and MinIO credentials. A summary with all credentials is displayed at the end of the installation.

Access the application at the URL shown in the installation summary.

### Development Setup

For local development with hot-reload:

```bash
git clone <repository-url>
cd Aegis
docker-compose up -d
./scripts/init.sh
```

Access at **http://localhost:5173** (frontend dev server) and **http://localhost:8000/docs** (API docs).

### Authentication

JWT-based authentication with HttpOnly cookies. Admin credentials are configured during installation:

- If you set a custom password in the wizard, use that.
- If you left it blank, a secure random password was auto-generated and displayed in the installation summary and backend logs.

To retrieve auto-generated credentials after installation:

```bash
docker logs aegis-backend 2>&1 | grep -A 5 "Initial Admin User Created"
```

> **Note:** Passwords must meet complexity requirements: minimum 12 characters with at least one uppercase letter, one lowercase letter, one digit, and one special character.

### Importing Data Sources

On startup, the backend automatically seeds the initial data sources (Atomic Red Team, SigmaHQ, CALDERA, LOLBAS, GTFOBins, D3FEND). You can then sync each source from the UI:

1. Navigate to **Data Sources** in the sidebar
2. Click **Sync** on each data source to import its content
3. Trigger a **MITRE ATT&CK Sync** from the **System** page

Alternatively, use the API:

```bash
# Sync MITRE ATT&CK techniques
curl -X POST http://your-server/api/v1/system/sync-mitre -H "Authorization: Bearer $TOKEN"

# Sync all data sources at once
curl -X POST http://your-server/api/v1/data-sources/sync-all -H "Authorization: Bearer $TOKEN"
```

### Production Considerations

- **HTTPS/TLS:** For internet-facing deployments, place a reverse proxy with TLS in front (e.g., Traefik, Caddy, or Nginx with Let's Encrypt). Uncomment the HSTS header in `frontend/nginx.conf` once HTTPS is configured.
- **Backups:** Set up regular PostgreSQL backups: `docker exec aegis-postgres pg_dump -U postgres attackdb > backup.sql`
- **Updates:** To update, pull the latest code and run: `docker compose -f docker-compose.prod.yml up -d --build`
- **Firewall:** Only expose port 80/443. All other services (DB, MinIO, backend) are internal only.
- **Reconfigure:** Run `./scripts/install.sh` again to reconfigure the environment (domain, credentials, etc.).

### Configuring Scoring Weights

Default scoring weights can be adjusted via environment variables:

```env
SCORING_WEIGHT_TESTS=40
SCORING_WEIGHT_DETECTION_RULES=20
SCORING_WEIGHT_D3FEND=15
SCORING_WEIGHT_FRESHNESS=15
SCORING_WEIGHT_PLATFORM_DIVERSITY=10
```

Or at runtime via the admin API — see [docs/SCORING.md](docs/SCORING.md).

### Configuring Campaigns

1. Navigate to **Campaigns** in the sidebar
2. Create a new campaign (custom or from a threat actor)
3. Add tests with dependencies and phases
4. Optionally enable **recurring scheduling** (weekly, monthly, quarterly)

## Services

| Service | Port | Description |
|---------|------|-------------|
| Frontend | 5173 | React dev server (Vite) |
| Backend | 8000 | FastAPI REST API |
| PostgreSQL | 5433 | Database |
| MinIO API | 9000 | S3-compatible object storage |
| MinIO Console | 9001 | MinIO web interface |

## Navigation

```
📊 Dashboard
📊 Executive Dashboard          (leads + admin)
🔲 ATT&CK Matrix                (advanced heatmap)
🧪 Tests
  ├─ All Tests
  ├─ My Pending Tasks
  └─ Test Catalog
📋 Campaigns
👤 Threat Actors
📜 Compliance
📈 Comparison                    (leads + admin)
📄 Reports
⚙️ System                       (admin only)
  ├─ Data Sources               (sync Atomic, Sigma, CALDERA, LOLBAS, GTFOBins, D3FEND)
  ├─ MITRE Sync                 (ATT&CK sync, intel scan, template management)
  ├─ Users
  └─ Audit Log
```

## API Documentation

Interactive API documentation is available **in development only** (disabled in production for security):

- **Swagger UI**: http://localhost:8000/docs
- **ReDoc**: http://localhost:8000/redoc

> In production (`AEGIS_ENV=production`), these endpoints are disabled. Use the development environment or refer to [docs/API.md](docs/API.md).

### API Endpoints

| Group | Prefix | Key Operations |
|-------|--------|---------------|
| Auth | `/api/v1/auth` | Login, get current user |
| Techniques | `/api/v1/techniques` | CRUD, list with filters, mark reviewed |
| Tests | `/api/v1/tests` | Full Red/Blue workflow, remediation, retest chain |
| Test Templates | `/api/v1/test-templates` | CRUD, stats, toggle active, bulk activate/deactivate |
| Evidence | `/api/v1/tests/{id}/evidence` | Upload evidence, get presigned URLs |
| Campaigns | `/api/v1/campaigns` | CRUD, scheduling, history |
| Threat Actors | `/api/v1/threat-actors` | CRUD, technique mappings |
| Detection Rules | `/api/v1/detection-rules` | List, filter by source/technique |
| D3FEND | `/api/v1/d3fend` | Defensive techniques and mappings |
| Compliance | `/api/v1/compliance` | Frameworks, controls, gaps |
| Scores | `/api/v1/scores` | Technique/tactic/actor/org scores, config |
| Operational Metrics | `/api/v1/metrics/operational` | MTTD, MTTR, trends, team breakdown |
| Heatmap | `/api/v1/heatmap` | ATT&CK Navigator-style data |
| Snapshots | `/api/v1/snapshots` | Create, compare, list snapshots |
| Reports | `/api/v1/reports` | Coverage, results, remediation exports |
| Notifications | `/api/v1/notifications` | List, read, mark all read |
| Metrics | `/api/v1/metrics` | Summary, by-tactic, pipeline, team activity |
| System | `/api/v1/system` | MITRE sync, import, scheduler status |
| Users | `/api/v1/users` | User CRUD (admin) |
| Audit Logs | `/api/v1/audit-logs` | Audit trail (admin) |
| Data Sources | `/api/v1/data-sources` | Data source management (admin) |

See [docs/API.md](docs/API.md) for the full endpoint reference.

## Configuration

All variables are configured automatically by `scripts/install.sh`. For manual setup, copy `.env.example` to `.env` and fill in the values.

### Required (production)

| Variable | Description |
|----------|-------------|
| `SECRET_KEY` | JWT signing key — **required** in production (app refuses to start without it). Generate with `openssl rand -hex 32` |
| `DB_PASSWORD` | PostgreSQL password |
| `MINIO_SECRET_KEY` | MinIO secret key |

### Security & Auth

| Variable | Default | Description |
|----------|---------|-------------|
| `AEGIS_ENV` | — | Set to `production` to enforce security settings |
| `ADMIN_USERNAME` | `admin` | Initial admin account username |
| `ADMIN_PASSWORD` | *(auto-generated)* | Initial admin password. If empty, a secure random password is generated and shown in logs |
| `ACCESS_TOKEN_EXPIRE_MINUTES` | `15` | JWT token lifetime in minutes |
| `CORS_ORIGINS` | `http://localhost:5173` | Comma-separated list of allowed frontend origins |

### Infrastructure

| Variable | Default | Description |
|----------|---------|-------------|
| `DB_USER` | `postgres` | PostgreSQL username |
| `DB_NAME` | `attackdb` | PostgreSQL database name |
| `MINIO_ENDPOINT` | `minio:9000` | MinIO server address |
| `MINIO_ACCESS_KEY` | `minioadmin` | MinIO access key |
| `MINIO_BUCKET` | `evidence` | MinIO bucket for evidence files |
| `MINIO_SECURE` | `false` | Set to `true` if MinIO is behind TLS |
| `FRONTEND_PORT` | `80` | Port exposed by the frontend container |

### Scoring Weights

| Variable | Default | Description |
|----------|---------|-------------|
| `SCORING_WEIGHT_TESTS` | `40` | Weight for test validation component |
| `SCORING_WEIGHT_DETECTION_RULES` | `20` | Weight for detection rules component |
| `SCORING_WEIGHT_D3FEND` | `15` | Weight for D3FEND coverage component |
| `SCORING_WEIGHT_FRESHNESS` | `15` | Weight for freshness component |
| `SCORING_WEIGHT_PLATFORM_DIVERSITY` | `10` | Weight for platform diversity component |
| `MAX_RETEST_COUNT` | `3` | Max automatic retests per original test |

## Security

Aegis includes several security hardening measures:

- **Authentication:** JWT tokens stored in HttpOnly/Secure/SameSite cookies (immune to XSS theft). Token revocation via Redis-backed blacklist on logout.
- **Rate limiting:** Login endpoint limited to 5 attempts per minute per IP (via slowapi).
- **Password policy:** Minimum 12 characters with uppercase, lowercase, digit, and special character.
- **CORS:** Configurable origins via `CORS_ORIGINS` environment variable. Restrictive method and header lists.
- **Nginx security headers:** CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy.
- **Non-root container:** Backend runs as `appuser` (UID 1001), not root.
- **File uploads:** 50 MB size limit, extension whitelist, filename sanitization.
- **ZIP imports:** Zip Slip (path traversal) and Zip Bomb (size/entry limit) protection.
- **API surface:** Swagger UI, ReDoc, and OpenAPI schema disabled in production.
- **Health endpoint:** Restricted to internal networks via Nginx ACL.
- **Input sanitization:** LIKE wildcard escaping in all search queries; Pydantic schemas on all endpoints.
- **XML parsing:** Uses `defusedxml` to prevent Billion Laughs / XXE attacks.
- **Error handling:** Internal exception details are logged server-side only, never exposed to clients.

## Project Structure

```
Aegis/
├── docker-compose.yml
├── docker-compose.prod.yml
├── .github/workflows/ci.yml      # GitHub Actions: ruff + pytest on PostgreSQL + Redis
├── docs/
│   ├── API.md                     # Full API endpoint reference
│   ├── ARCHITECTURE.md            # System architecture, DB schema, service map
│   ├── ADR.md                     # Architecture Decision Records
│   ├── DATA_SOURCES.md            # External data source documentation
│   ├── SCORING.md                 # Scoring system and metrics
│   ├── TECHNOLOGY_JUSTIFICATION.md
│   ├── C4_CONTEXT_DIAGRAM.md      # System context (C4 Level 1)
│   └── C4_CONTAINER_DIAGRAM.md    # Container architecture (C4 Level 2)
├── backend/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── alembic.ini
│   ├── alembic/versions/          # Database migration files
│   ├── pytest.ini
│   ├── tests/                     # 367+ pytest tests (domain, service, API)
│   └── app/
│       ├── main.py                # FastAPI app with all routers + lifespan
│       ├── config.py              # Pydantic Settings from environment
│       ├── database.py            # SQLAlchemy engine + session (lazy init)
│       ├── storage.py             # MinIO/S3 helpers
│       ├── auth.py                # Password hashing + JWT tokens
│       ├── domain/                # Pure business logic (zero framework imports)
│       │   ├── entities/          # Rich domain entities (Technique, Campaign, etc.)
│       │   ├── ports/             # Protocol interfaces (repos, ImportService)
│       │   ├── value_objects/     # Immutable types (MitreId, ScoringWeights)
│       │   ├── errors.py          # Domain exception hierarchy
│       │   └── unit_of_work.py    # Transaction management
│       ├── infrastructure/        # SQLAlchemy repos, Redis, mappers
│       ├── models/                # SQLAlchemy ORM models
│       ├── schemas/               # Pydantic request/response schemas
│       ├── routers/               # 27 thin HTTP adapter routers
│       ├── services/              # 46 framework-agnostic business services
│       ├── middleware/            # Error handler (domain exceptions → HTTP)
│       ├── dependencies/          # FastAPI dependency injection (auth, repos)
│       └── jobs/                  # APScheduler background jobs
└── frontend/src/
    ├── App.tsx                    # Routes with lazy loading + role protection
    ├── api/                       # API client modules (Axios + TanStack Query)
    ├── components/                # Reusable UI components
    ├── hooks/                     # Custom hooks (useDebounce, etc.)
    ├── context/                   # Auth state management
    └── pages/                     # Page components
```

## Development

### Running Migrations

```bash
docker exec aegis-backend alembic upgrade head
docker exec aegis-backend alembic revision --autogenerate -m "description"
docker exec aegis-backend alembic downgrade -1
```

### Running Tests

```bash
# Run all V3 tests inside the container (recommended)
docker exec aegis-backend python -m pytest tests/ -v --tb=short

# Run specific test suites
docker exec aegis-backend python -m pytest tests/test_data_sources.py -v
docker exec aegis-backend python -m pytest tests/test_scoring_and_compliance.py -v
docker exec aegis-backend python -m pytest tests/test_campaigns_and_snapshots.py -v

# Skip integration tests (require network)
docker exec aegis-backend python -m pytest tests/ -v -m "not integration"
```

### Generating Reports

```bash
# Coverage summary (JSON)
GET /api/v1/reports/coverage-summary

# Coverage CSV export
GET /api/v1/reports/coverage-csv

# Compliance gap analysis
GET /api/v1/compliance/{framework_id}/gaps
```

## Further Documentation

- **[Architecture](docs/ARCHITECTURE.md)** — Database schema, backend layers, domain entities, service map
- **[API Reference](docs/API.md)** — Full endpoint documentation
- **[Scoring](docs/SCORING.md)** — Scoring system explained with examples and configuration
- **[Data Sources](docs/DATA_SOURCES.md)** — All external data sources with import instructions
- **[ADRs](docs/ADR.md)** — Architecture Decision Records
- **[Technology Justification](docs/TECHNOLOGY_JUSTIFICATION.md)** — Technology choices and rationale
- **[C4 Diagrams](docs/C4_CONTEXT_DIAGRAM.md)** — System context and container architecture

## License

This project is proprietary software. All rights reserved.

## Disclaimer

This project has been developed with the assistance of [Cursor](https://cursor.com) and Claude Opus 4.6 (Anthropic).