Aegis/README.md

# Aegis — MITRE ATT&CK Coverage Platform

Aegis is a comprehensive platform for tracking and managing security coverage against the MITRE ATT&CK framework. It enables security teams to document, validate, and visualize their defensive capabilities against known adversary techniques through a structured Red Team / Blue Team validation workflow.

## Features

### Core (V1)
- **MITRE ATT&CK Integration** — Automatic synchronization via TAXII 2.0 (with GitHub fallback), scheduled every 24h
- **Red/Blue Validation Workflow** — Structured dual-validation lifecycle: draft → red_executing → blue_evaluating → in_review → validated/rejected
- **Dual Validation** — Independent Red Lead / Blue Lead approval before finalization
- **Coverage Tracking** — Per-technique status (validated, partial, not covered, in progress)
- **Evidence Storage** — Secure evidence with SHA256 integrity, separated by team (red/blue)
- **Role-Based Access Control** — Granular permissions for 6 roles (admin, red_tech, blue_tech, red_lead, blue_lead, viewer)

### Enhanced (V2)
- **Test Template Catalog** — Import from Atomic Red Team, create custom templates, instantiate tests
- **In-App Notifications** — Real-time notification bell with polling and automatic state-change alerts
- **Reports & Export** — Coverage summary, test results, and remediation reports in JSON and CSV
- **Remediation Tracking** — Step-by-step remediation assignments with status tracking
- **Metrics Dashboard** — Pipeline funnel, team activity, validation rates

### Advanced (V3)
- **Multi-Source Data Import** — Sigma, Elastic, CALDERA, LOLBAS, D3FEND, MITRE CTI threat actors, compliance mappings
- **Detection Rule Tracking** — Import and evaluate Sigma/Elastic detection rules per test
- **ATT&CK Heatmap** — Interactive Navigator-style heatmap with layers, filters, and export
- **Threat Actor Intelligence** — Track intrusion sets and their technique coverage
- **Campaign Management** — Group tests into campaigns with dependencies, scheduling, and recurring execution
- **Compliance Mapping** — Map NIST 800-53 controls to ATT&CK techniques with gap analysis
- **Granular Scoring** — 0–100 scoring for techniques, tactics, actors, and organization with configurable weights
- **Operational Metrics** — MTTD, MTTR, detection efficacy, alert fidelity, coverage velocity
- **Executive Dashboard** — High-level KPIs for leadership (leads + admin)
- **Temporal Comparison** — Coverage snapshots with historical comparison and trend analysis
- **Auto Re-testing** — Automatic retest creation after remediation completion (configurable limit)
- **Performance Optimizations** — Score caching, lazy loading, React.memo, database index optimization

## Red Team / Blue Team Validation Flow

```
┌──────┐    ┌──────────────┐    ┌─────────────────┐    ┌───────────┐
│ DRAFT│───▶│RED_EXECUTING │───▶│ BLUE_EVALUATING  │───▶│ IN_REVIEW │
└──────┘    └──────────────┘    └─────────────────┘    └─────┬─────┘
                                                             │
                                         ┌───────────────────┤
                                         ▼                   ▼
                                   ┌──────────┐       ┌──────────┐
                                   │ REJECTED │       │VALIDATED │
                                   └────┬─────┘       └──────────┘
                                        │                    │
                                        └──▶ Back to DRAFT   ├──▶ Remediation
                                                             └──▶ Auto Re-test
```

### States

| State | Description | Who acts |
|-------|-------------|----------|
| `draft` | Created, pending execution | Red Tech |
| `red_executing` | Red Team documents attack & uploads evidence | Red Tech |
| `blue_evaluating` | Blue Team documents detection & uploads evidence | Blue Tech |
| `in_review` | Both managers review evidence | Red Lead, Blue Lead |
| `validated` | Approved by both managers | — (terminal) |
| `rejected` | Rejected — returns to draft for redo | Red/Blue Lead can reopen |

### Dual Validation

Both Red Lead and Blue Lead must independently vote:
- **Both approve** → test moves to `validated`
- **Either rejects** → test moves to `rejected`
- **One votes, other pending** → stays in `in_review`

## User Roles

| Role | Description | Capabilities |
|------|-------------|-------------|
| `admin` | Full system access | Everything |
| `red_tech` | Red team technician | Create tests, document attacks, upload red evidence |
| `blue_tech` | Blue team technician | Document detection, upload blue evidence |
| `red_lead` | Red team lead | Validate/reject the red side of tests |
| `blue_lead` | Blue team lead | Validate/reject the blue side of tests |
| `viewer` | Read-only access | View all data |

## Tech Stack

- **Backend**: FastAPI (Python 3.11)
- **Database**: PostgreSQL 15 with UUID primary keys and JSONB columns
- **Object Storage**: MinIO (S3-compatible)
- **ORM**: SQLAlchemy with Alembic migrations (18 migration files)
- **Frontend**: React 19 + TypeScript + Vite 7 + Tailwind CSS v4 + TanStack Query + TanStack Virtual
- **Scheduler**: APScheduler (MITRE sync, Intel scan, Notification cleanup, Snapshots, Recurring campaigns)
- **Charts**: Recharts

## Quick Start

### Prerequisites

- Docker and Docker Compose
- Git

### Installation

1. Clone the repository:
```bash
git clone <repository-url>
cd Aegis
```

2. Start all services:
```bash
docker-compose up -d
```

3. Run database migrations:
```bash
docker exec aegis-backend alembic upgrade head
```

4. Seed the admin user:
```bash
docker exec aegis-backend python -m app.seed
```

5. Access the application:
```bash
# API health check
curl http://localhost:8000/health
# Expected: {"status":"ok"}

# Open http://localhost:5173 — Aegis login page
```

### Authentication

JWT-based authentication. Default admin credentials after seeding:

```
Username: admin
Password: admin123
```

> **Important:** Change the default `admin123` password and `SECRET_KEY` in production.

### Importing Data Sources

After initial setup, populate the platform with data:

```bash
# 1. Sync MITRE ATT&CK techniques
curl -X POST http://localhost:8000/api/v1/system/sync-mitre -H "Authorization: Bearer $TOKEN"

# 2. Import test templates from Atomic Red Team
curl -X POST http://localhost:8000/api/v1/system/import-atomic-red-team -H "Authorization: Bearer $TOKEN"

# 3. Import additional sources via the Data Sources admin page
#    Navigate to System → Data Sources in the UI
```

See [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) for detailed instructions on all data sources.

### Configuring Scoring Weights

Default scoring weights can be adjusted via environment variables:

```env
SCORING_WEIGHT_TESTS=40
SCORING_WEIGHT_DETECTION_RULES=20
SCORING_WEIGHT_D3FEND=15
SCORING_WEIGHT_FRESHNESS=15
SCORING_WEIGHT_PLATFORM_DIVERSITY=10
```

Or at runtime via the admin API — see [docs/SCORING.md](docs/SCORING.md).

### Configuring Campaigns

1. Navigate to **Campaigns** in the sidebar
2. Create a new campaign (custom or from a threat actor)
3. Add tests with dependencies and phases
4. Optionally enable **recurring scheduling** (weekly, monthly, quarterly)

## Services

| Service | Port | Description |
|---------|------|-------------|
| Frontend | 5173 | React dev server (Vite) |
| Backend | 8000 | FastAPI REST API |
| PostgreSQL | 5433 | Database |
| MinIO API | 9000 | S3-compatible object storage |
| MinIO Console | 9001 | MinIO web interface |

## Navigation

```
📊 Dashboard
📊 Executive Dashboard          (leads + admin)
🔲 ATT&CK Matrix                (advanced heatmap)
🧪 Tests
  ├─ All Tests
  ├─ My Pending Tasks
  └─ Test Catalog
📋 Campaigns
👤 Threat Actors
📜 Compliance
📈 Comparison                    (leads + admin)
📄 Reports
⚙️ System                       (admin only)
  ├─ Data Sources
  ├─ MITRE Sync
  ├─ Users
  └─ Audit Log
```

## API Documentation

Interactive API documentation available at:

- **Swagger UI**: http://localhost:8000/docs
- **ReDoc**: http://localhost:8000/redoc

### API Endpoints

| Group | Prefix | Key Operations |
|-------|--------|---------------|
| Auth | `/api/v1/auth` | Login, get current user |
| Techniques | `/api/v1/techniques` | CRUD, list with filters, mark reviewed |
| Tests | `/api/v1/tests` | Full Red/Blue workflow, remediation, retest chain |
| Test Templates | `/api/v1/test-templates` | CRUD, import, stats, toggle active |
| Evidence | `/api/v1/tests/{id}/evidence` | Upload evidence, get presigned URLs |
| Campaigns | `/api/v1/campaigns` | CRUD, scheduling, history |
| Threat Actors | `/api/v1/threat-actors` | CRUD, technique mappings |
| Detection Rules | `/api/v1/detection-rules` | List, filter by source/technique |
| D3FEND | `/api/v1/d3fend` | Defensive techniques and mappings |
| Compliance | `/api/v1/compliance` | Frameworks, controls, gaps |
| Scores | `/api/v1/scores` | Technique/tactic/actor/org scores, config |
| Operational Metrics | `/api/v1/metrics/operational` | MTTD, MTTR, trends, team breakdown |
| Heatmap | `/api/v1/heatmap` | ATT&CK Navigator-style data |
| Snapshots | `/api/v1/snapshots` | Create, compare, list snapshots |
| Reports | `/api/v1/reports` | Coverage, results, remediation exports |
| Notifications | `/api/v1/notifications` | List, read, mark all read |
| Metrics | `/api/v1/metrics` | Summary, by-tactic, pipeline, team activity |
| System | `/api/v1/system` | MITRE sync, import, scheduler status |
| Users | `/api/v1/users` | User CRUD (admin) |
| Audit Logs | `/api/v1/audit-logs` | Audit trail (admin) |
| Data Sources | `/api/v1/data-sources` | Data source management (admin) |

See [docs/API.md](docs/API.md) for the full endpoint reference.

## Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_URL` | `postgresql://postgres:postgres@postgres:5432/attackdb` | PostgreSQL connection |
| `SECRET_KEY` | `change-me-in-production` | JWT signing key |
| `ALGORITHM` | `HS256` | JWT signing algorithm |
| `ACCESS_TOKEN_EXPIRE_MINUTES` | `60` | Token lifetime |
| `MINIO_ENDPOINT` | `minio:9000` | MinIO server |
| `MINIO_ACCESS_KEY` | `minioadmin` | MinIO access key |
| `MINIO_SECRET_KEY` | `minioadmin` | MinIO secret key |
| `MINIO_BUCKET` | `evidence` | Evidence bucket |
| `MAX_RETEST_COUNT` | `3` | Max automatic retests per original test |
| `SCORING_WEIGHT_TESTS` | `40` | Weight for test validation component |
| `SCORING_WEIGHT_DETECTION_RULES` | `20` | Weight for detection rules component |
| `SCORING_WEIGHT_D3FEND` | `15` | Weight for D3FEND coverage component |
| `SCORING_WEIGHT_FRESHNESS` | `15` | Weight for freshness component |
| `SCORING_WEIGHT_PLATFORM_DIVERSITY` | `10` | Weight for platform diversity component |

## Project Structure

```
Aegis/
├── docker-compose.yml
├── docker-compose.prod.yml
├── docs/
│   ├── API.md                     # Full API endpoint reference
│   ├── ARCHITECTURE.md            # System architecture and DB schema
│   ├── DATA_SOURCES.md            # External data source documentation
│   └── SCORING.md                 # Scoring system and metrics
├── backend/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── alembic.ini
│   ├── alembic/versions/          # b001–b018 migration files
│   ├── pytest.ini
│   └── app/
│       ├── main.py                # FastAPI app with all routers + lifespan
│       ├── config.py              # Settings from environment
│       ├── database.py            # SQLAlchemy engine + session (lazy init)
│       ├── storage.py             # MinIO/S3 helpers
│       ├── auth.py                # Password hashing + JWT tokens
│       ├── models/                # 18 model files (SQLAlchemy ORM)
│       ├── schemas/               # Pydantic request/response schemas
│       ├── routers/               # 21 API routers
│       ├── services/              # 20 business logic services
│       ├── dependencies/          # Auth dependencies (get_current_user, require_role)
│       └── jobs/
│           └── mitre_sync_job.py  # APScheduler: 5 background jobs
├── frontend/src/
│   ├── App.tsx                    # Routes with lazy loading + role protection
│   ├── api/                       # 22 API client modules (Axios + TanStack Query)
│   ├── components/
│   │   ├── Layout.tsx             # Sidebar + header + NotificationBell
│   │   ├── Sidebar.tsx            # Role-aware collapsible navigation
│   │   ├── heatmap/               # ATT&CK heatmap (6 components)
│   │   ├── compliance/            # Compliance UI (gauge, controls table)
│   │   └── test-detail/           # Test detail sub-components
│   ├── hooks/
│   │   └── useDebounce.ts         # Debounce hook for search inputs
│   ├── context/
│   │   └── AuthContext.tsx         # Auth state management
│   └── pages/                     # 21 page components
└── backend/tests/
    ├── conftest.py                # SQLite test DB with JSONB/UUID compatibility
    ├── fixtures/                  # YAML/TOML/JSON test fixtures
    ├── test_data_sources.py       # Data source parsing tests
    ├── test_scoring_and_compliance.py # Scoring + metrics + compliance tests
    ├── test_campaigns_and_snapshots.py # Campaign, snapshot, and retest tests
    ├── test_workflow.py           # Red/Blue workflow tests
    ├── test_templates_crud.py     # Template CRUD tests
    ├── test_metrics_v2.py         # V2 metrics tests
    └── test_integration_v2.py     # Full integration E2E tests
```

## Development

### Running Migrations

```bash
docker exec aegis-backend alembic upgrade head
docker exec aegis-backend alembic revision --autogenerate -m "description"
docker exec aegis-backend alembic downgrade -1
```

### Running Tests

```bash
# Run all V3 tests inside the container (recommended)
docker exec aegis-backend python -m pytest tests/ -v --tb=short

# Run specific test suites
docker exec aegis-backend python -m pytest tests/test_data_sources.py -v
docker exec aegis-backend python -m pytest tests/test_scoring_and_compliance.py -v
docker exec aegis-backend python -m pytest tests/test_campaigns_and_snapshots.py -v

# Skip integration tests (require network)
docker exec aegis-backend python -m pytest tests/ -v -m "not integration"
```

### Generating Reports

```bash
# Coverage summary (JSON)
GET /api/v1/reports/coverage-summary

# Coverage CSV export
GET /api/v1/reports/coverage-csv

# Compliance gap analysis
GET /api/v1/compliance/{framework_id}/gaps
```

## Further Documentation

- **[Architecture](docs/ARCHITECTURE.md)** — Database schema, service layer, state machine diagrams
- **[Data Sources](docs/DATA_SOURCES.md)** — All external data sources with import instructions
- **[Scoring](docs/SCORING.md)** — Scoring system explained with examples and configuration
- **[API Reference](docs/API.md)** — Full endpoint documentation

## License

This project is proprietary software. All rights reserved.