# Aegis — Architecture Decision Records (ADR)

> **Date:** February 11, 2026  
> **Status:** All decisions are **Accepted** and currently in effect.

---

## Index

| ADR | Title | Status |
|-----|-------|--------|
| [ADR-001](#adr-001-fastapi-as-backend-framework) | FastAPI as Backend Framework | Accepted |
| [ADR-002](#adr-002-postgresql-with-jsonb-as-primary-database) | PostgreSQL with JSONB as Primary Database | Accepted |
| [ADR-003](#adr-003-minio-for-evidence-storage) | MinIO for Evidence Storage | Accepted |
| [ADR-004](#adr-004-docker-compose-for-deployment) | Docker Compose for Deployment | Accepted |
| [ADR-005](#adr-005-modular-monolith-over-microservices) | Modular Monolith over Microservices | Accepted |
| [ADR-006](#adr-006-apscheduler-in-process-over-external-job-system) | APScheduler In-Process over External Job System | Accepted |

---

## ADR-001: FastAPI as Backend Framework

**Date:** Project inception  
**Status:** Accepted

### Context

Aegis is an internal security platform for managing MITRE ATT&CK coverage through Red/Blue team validation workflows. The backend must:

- Expose a REST API consumed by a React SPA (21 pages, 80+ endpoints).
- Handle CRUD operations for 18+ domain entities with complex filtering and joins.
- Support file uploads (evidence) and streaming downloads (CSV/JSON exports).
- Integrate with external APIs (MITRE TAXII 2.0, GitHub REST, D3FEND REST).
- Enforce RBAC authorization across 6 roles.
- Be developed and maintained by a small team requiring fast iteration.
- Run in a containerized environment with Python as the team's primary language.

### Decision

We chose **FastAPI** as the backend framework, served by **Uvicorn** (ASGI).

Key factors:
- **Automatic OpenAPI/Swagger** generation from type hints reduces documentation burden for 80+ endpoints.
- **Pydantic integration** provides request/response validation with zero boilerplate, critical for a schema-heavy domain (test workflows, scoring payloads, compliance data).
- **`Depends()` system** provides clean dependency injection for auth, DB sessions, and role checks without a third-party DI container.
- **Async-capable** but allows synchronous route handlers, which matters because SQLAlchemy (sync) is the ORM and all external data imports are CPU/IO-bound synchronous operations.
- **Performance** is sufficient for an internal tool (< 100 concurrent users) without needing Go/Rust-level throughput.
- **Python ecosystem** gives direct access to `taxii2-client`, `pySigma`, `boto3`, `PyYAML`, and `toml` — all required for the 8 external data source integrations.

### Consequences

**Positive:**
- Swagger UI available in development (`/docs`) for rapid API exploration and testing.
- Pydantic schemas act as living documentation for the API contract.
- `Depends()` chain for `get_db` → `get_current_user` → `require_role()` is concise and composable.
- `python-jose` + `passlib` integrate naturally for JWT/bcrypt auth.
- SlowAPI integrates directly with FastAPI for rate limiting.

**Negative:**
- The `Depends()` system encourages passing `db: Session` directly into route handlers, which has led to routers containing raw SQLAlchemy queries instead of delegating to a service/repository layer (see ADR analysis — 11 of 21 routers query the DB directly).
- Synchronous route handlers block the event loop when performing long operations (MITRE sync ZIP downloads can take 30+ seconds), mitigated by Nginx proxy timeout of 300s.
- No built-in background task system beyond `BackgroundTasks` (which is request-scoped), requiring APScheduler for scheduled jobs (see ADR-006).

**Risks:**
- FastAPI's ease of putting logic in route handlers has contributed to "fat controllers" — this is a developer discipline issue, not a framework limitation.

### Alternatives Considered

| Alternative | Reason Rejected |
|------------|-----------------|
| **Django + DRF** | Heavier ORM opinions, admin panel unnecessary, slower startup. Django's ORM lacks SQLAlchemy's flexibility with JSONB and complex joins. |
| **Flask + Flask-RESTful** | No built-in validation, no auto-generated OpenAPI, manual Swagger setup. Would require marshmallow or similar for schema validation. |
| **Go (Gin/Echo)** | Team's primary expertise is Python. The 8 data source integrations rely heavily on Python libraries (pySigma, taxii2-client, PyYAML). |
| **NestJS (Node.js)** | Would split the team across two runtimes. Python libraries for STIX/TAXII and Sigma rule parsing have no mature Node.js equivalents. |

---

## ADR-002: PostgreSQL with JSONB as Primary Database

**Date:** Project inception  
**Status:** Accepted

### Context

Aegis manages a complex relational domain: techniques have tests, tests belong to campaigns, threat actors map to techniques, compliance controls map to techniques, detection rules map to techniques and tests. This is a deeply relational model with 18+ tables and many-to-many relationships.

However, several entities also carry semi-structured data that varies by source:
- **Audit logs** — `details` field contains arbitrary action metadata (different structure per action type).
- **Threat actors** — `aliases`, `target_sectors`, `target_regions`, `references` are variable-length arrays/objects from STIX 2.0 bundles.
- **Detection rules** — `platforms` (array), `log_sources` (object with varying keys like `product`, `service`, `category`).
- **Data sources** — `last_sync_stats` (object with import-specific counters), `config` (source-specific configuration).
- **Techniques** — `platforms` (array of OS names from ATT&CK).
- **Campaigns** — `tags` (user-defined array).

This data is imported from external sources with varying schemas (STIX JSON, Sigma YAML, Elastic TOML) and must be stored without rigid column definitions.

### Decision

We chose **PostgreSQL 15** as the primary database, using its native **JSONB** column type for semi-structured fields alongside traditional relational columns for the core domain.

The schema is managed by **Alembic** (18 migration versions) with **SQLAlchemy** ORM using `sqlalchemy.dialects.postgresql.JSONB`.

### Consequences

**Positive:**
- Relational integrity enforced with foreign keys for the core domain (test → technique, campaign → test, evidence → test, etc.).
- JSONB columns store variable-structure data without schema migrations when external sources change their format.
- JSONB supports GIN indexing for efficient containment queries (`@>` operator) on arrays like `platforms` and `target_sectors`.
- Single database to operate — no need for a separate document store.
- PostgreSQL's mature ecosystem: `pg_dump` for backups, `pg_isready` for health checks, extensive monitoring tooling.
- SQLAlchemy's `JSONB` type allows Python dict/list access with full query support.

**Negative:**
- JSONB fields bypass ORM-level validation — the schema for `details`, `config`, `references` etc. is only enforced by application code (Pydantic schemas on input), not by the database.
- Complex queries mixing relational joins with JSONB containment can be harder to optimize and debug.
- No GIN indexes are currently defined in migrations for JSONB columns, meaning array containment queries may perform full scans on large datasets.
- JSONB fields in audit logs make structured querying across action types difficult (e.g., "find all audit entries where details.old_state = 'draft'").

**Risks:**
- As JSONB usage grows, the boundary between "should be a column" and "should be JSONB" can blur. Currently well-contained to arrays and metadata fields.

### Alternatives Considered

| Alternative | Reason Rejected |
|------------|-----------------|
| **PostgreSQL without JSONB** | Would require separate junction tables for every array field (technique_platforms, actor_aliases, actor_sectors, etc.), adding 10+ tables for data that is always read as a whole array. |
| **MongoDB** | The core domain is deeply relational (techniques ↔ tests ↔ campaigns ↔ threat actors). Modeling this in MongoDB would require denormalization, embedded documents, or manual reference integrity — trading JSONB flexibility for relational integrity loss. |
| **PostgreSQL + MongoDB (dual)** | Operational complexity of two database systems is unjustified for the current JSONB usage (~12 columns across 6 tables). |
| **MySQL 8 with JSON** | PostgreSQL's JSONB is binary-indexed and faster for containment queries. MySQL's JSON type is text-based with function-based indexing. PostgreSQL also has superior support for UUID primary keys (native type vs BINARY(16)). |

---

## ADR-003: MinIO for Evidence Storage

**Date:** Project inception  
**Status:** Accepted

### Context

The Red/Blue team validation workflow requires both teams to upload evidence files (screenshots, log files, PCAPs, documents) to support their test findings. Requirements:

- Files range from small screenshots (KB) to large PCAPs (hundreds of MB).
- Files must be associated with specific tests and teams (red/blue).
- Files must be downloadable by authorized users via the browser.
- Storage must be independent from the application database (no BLOBs in PostgreSQL).
- The platform is deployed on-premise via Docker Compose — cloud-native S3 is not available.
- The upload/download API must be simple and well-supported in Python.

### Decision

We chose **MinIO** as an S3-compatible object storage system, accessed via **boto3** (AWS S3 SDK for Python).

Implementation details:
- A single `evidence` bucket is auto-created on backend startup (`ensure_bucket_exists()`).
- Files are uploaded with `put_object()` using a generated UUID-based key.
- Downloads use presigned URLs (`generate_presigned_url()`) with 1-hour expiration.
- The MinIO client is a module-level singleton in `storage.py`.
- Evidence metadata (filename, MIME type, size, team, test association) is stored in PostgreSQL; only the binary content lives in MinIO.

### Consequences

**Positive:**
- S3-compatible API means zero code changes if migrating to AWS S3, GCS, or any S3-compatible service.
- boto3 is the most mature and well-documented S3 client library in Python.
- Presigned URLs offload download bandwidth from the backend — the browser fetches directly from MinIO.
- Binary data stays out of PostgreSQL, keeping the database lean and backups fast.
- MinIO runs as a single Docker container with a persistent volume — simple to deploy and back up.
- MinIO Console (port 9001) provides a web UI for administrators to inspect stored files.

**Negative:**
- Presigned URLs currently point to `minio:9000` (Docker internal hostname), which is not accessible from the browser in production without additional Nginx configuration or a public MinIO endpoint.
- No file virus scanning or content validation before storage.
- No lifecycle policies configured (no automatic deletion of old evidence).
- The module-level singleton client means the MinIO connection configuration cannot be changed at runtime (acceptable for the current deployment model).

**Risks:**
- If MinIO container is lost and the volume is not backed up, all evidence files are permanently lost. Evidence metadata in PostgreSQL would reference non-existent files.

### Alternatives Considered

| Alternative | Reason Rejected |
|------------|-----------------|
| **PostgreSQL BYTEA/BLOB** | Storing binary files in the database bloats backups, degrades query performance, and makes streaming large files complex. PostgreSQL is not designed as a file store. |
| **Local filesystem** | Not portable across container restarts without host volume mounts. No presigned URL support, requiring the backend to proxy all downloads. No built-in replication or management UI. |
| **AWS S3** | Requires cloud account and internet connectivity. The platform is designed for on-premise deployment where external cloud services may not be permitted. |
| **SeaweedFS** | Less mature ecosystem, smaller community. The S3-compatible layer is less complete than MinIO's. boto3 compatibility is not guaranteed. |

---

## ADR-004: Docker Compose for Deployment

**Date:** Project inception  
**Status:** Accepted

### Context

Aegis is a multi-component platform deployed on-premise within organizations' security environments:

- 4 services: Frontend (Nginx), Backend (Uvicorn), PostgreSQL, MinIO.
- Target environments range from a single server to small clusters.
- Security teams typically have Docker available but may not have Kubernetes.
- The platform must be installable by a security engineer (not necessarily a DevOps specialist).
- Both development and production environments should use the same orchestration approach for consistency.

### Decision

We chose **Docker Compose** as the deployment and orchestration tool, with two compose files:

- `docker-compose.yml` — Development: source volumes mounted, dev servers, exposed ports.
- `docker-compose.prod.yml` — Production: multi-stage builds, Nginx serving static assets, only frontend port exposed, `SECRET_KEY` required.

Supporting infrastructure:
- `scripts/install.sh` — Interactive production installer that generates secrets, prompts for configuration, writes `.env`, and runs `docker compose up -d --build`.
- `scripts/init.sh` — Development setup that waits for services, runs migrations, and seeds data.
- All services connected via a `aegis-network` bridge network.
- Named volumes for PostgreSQL and MinIO data persistence.
- Health checks on PostgreSQL (`pg_isready`) and backend (`/health`).
- Service dependency ordering: backend waits for `postgres: service_healthy` and `minio: service_started`.

### Consequences

**Positive:**
- Single-command deployment: `docker compose -f docker-compose.prod.yml up -d --build`.
- The `install.sh` wizard makes production setup accessible to non-DevOps personnel.
- Consistent environments between development and production (same containers, same network topology).
- Named volumes survive container rebuilds — data persists across upgrades.
- No external dependencies beyond Docker and Docker Compose.
- Multi-stage Dockerfile for frontend produces a minimal Nginx image (~25MB) from a full Node.js build stage.
- Non-root user (`appuser`, UID 1001) in backend Dockerfile follows container security best practices.

**Negative:**
- No built-in horizontal scaling — running multiple backend instances requires manual Nginx upstream configuration and a shared token blacklist (currently in-memory).
- No rolling deployments — `docker compose up -d --build` causes brief downtime during image rebuilds.
- No built-in secrets management — secrets are in `.env` files on the host filesystem.
- No container orchestration beyond restart policies (`restart: always`).
- No centralized logging — each container logs to its own stdout/stderr.

**Risks:**
- Single point of failure: if the host machine goes down, all services go down.
- No automated backup strategy — `pg_dump` is documented but not automated.

### Alternatives Considered

| Alternative | Reason Rejected |
|------------|-----------------|
| **Kubernetes (k8s)** | Significantly higher operational complexity. Requires a cluster, kubectl expertise, Helm charts or manifests, ingress controllers, PVCs. Overkill for a single-server deployment targeting security teams. |
| **Docker Swarm** | Adds orchestration complexity with minimal benefit over Compose for < 5 services. The project does not need multi-node scheduling or service mesh. Swarm's future is uncertain compared to Compose V2. |
| **Bare metal / systemd** | Loses containerization benefits (isolation, reproducibility, dependency management). Would require manual installation of Python, Node.js, PostgreSQL, MinIO on each target system. |
| **Ansible + Docker** | Adds a configuration management layer that is unnecessary for a 4-service application. Could be valuable in the future for multi-server deployments but is premature now. |

---

## ADR-005: Modular Monolith over Microservices

**Date:** Project inception  
**Status:** Accepted

### Context

Aegis has distinct functional domains that could theoretically be separate services:
- **Test Workflow** — Red/Blue validation state machine, evidence management.
- **Coverage Analytics** — Scoring engine, heatmaps, metrics, reports.
- **Data Import** — 8 external source integrations (MITRE, Sigma, Elastic, CALDERA, etc.).
- **Campaign Management** — Campaign lifecycle, scheduling, threat actor generation.
- **Compliance** — Framework mappings, gap analysis, control tracking.
- **User/Auth** — Authentication, RBAC, audit logging.

However:
- These domains share the same database and have tight data dependencies (e.g., scoring reads tests, techniques, detection rules, and D3FEND mappings in a single calculation).
- The development team is small.
- The deployment target is single-server Docker Compose.
- Latency between services would complicate the scoring engine (which aggregates across 5+ tables).

### Decision

We chose a **modular monolith** architecture: a single deployable backend process organized into internal modules (routers, services, models) rather than separate microservices.

Module boundaries:
- **Routers** (21 files) — HTTP endpoint definitions grouped by domain.
- **Services** (20 files) — Business logic grouped by capability (workflow, scoring, notifications, imports).
- **Models** (18 files) — ORM entities grouped by domain concept.
- **Schemas** (10 files) — Pydantic DTOs grouped by domain concept.

All modules share a single database, a single process, and a single deployment artifact.

### Consequences

**Positive:**
- No network overhead between domains — scoring can join 5+ tables in a single SQL query.
- Single deployment artifact simplifies CI/CD, monitoring, and debugging.
- Shared database means ACID transactions across domains (e.g., creating a test + logging the audit entry + sending a notification in one commit).
- No service discovery, API gateways, circuit breakers, or distributed tracing needed.
- Faster development iteration — change any module, rebuild one container.

**Negative:**
- All domains scale together — cannot scale the data import workers independently from the API.
- A bug in one module (e.g., a memory leak in scoring) can crash the entire application.
- Module boundaries are not enforced at the language level — routers currently import services and models freely across domains (e.g., `heatmap.py` imports 6 models from different domains).
- The monolith has grown to 21 routers and 20 services without explicit boundary enforcement, leading to "fat controllers" and cross-cutting concerns.

**Risks:**
- Without explicit module boundaries (enforced by code structure or linting rules), the modular monolith can degrade into a traditional monolith where everything depends on everything.
- The Clean Architecture refactor proposed in `ARCHITECTURAL_ANALYSIS.md` would restore module boundaries via the domain/application/infrastructure/presentation layers.

### Alternatives Considered

| Alternative | Reason Rejected |
|------------|-----------------|
| **Microservices** | The 8 data source integrations would each become a service, requiring inter-service communication for writing to the shared technique/rule tables. Scoring would need to call 3-4 services to gather data, adding latency and failure modes. Operational overhead (8+ containers, service mesh, distributed tracing) unjustified for a small team and single-server deployment. |
| **Microservices with shared DB** | Anti-pattern. Multiple services sharing a database lose the main benefit of microservices (independent deployment and schema evolution) while keeping the operational complexity. |
| **Modular monolith with enforced boundaries** | This is the recommended evolution (see ADR analysis). The current implementation has module structure but no boundary enforcement. Adding domain-layer interfaces (Protocol/ABC), a repository pattern, and import linting rules would achieve this without a microservices migration. |

---

## ADR-006: APScheduler In-Process over External Job System

**Date:** Project inception  
**Status:** Accepted

### Context

Aegis requires periodic background tasks:

| Task | Frequency | Duration | Description |
|------|-----------|----------|-------------|
| MITRE ATT&CK sync | Every 24 hours | 30-120 seconds | Download STIX/TAXII feed, upsert ~700 techniques |
| Intel scan | Every 7 days | 10-60 seconds | Scan threat intelligence sources |
| Notification cleanup | Every 24 hours | < 5 seconds | Delete read notifications older than 90 days |
| Coverage snapshot | Weekly (Sunday 00:00) | 5-30 seconds | Capture point-in-time coverage state across all techniques |
| Recurring campaigns | Every 24 hours | < 10 seconds | Check and spawn due recurring test campaigns |

Requirements:
- Jobs must access the same database as the API.
- Jobs must not block API request handling.
- No additional infrastructure should be required beyond what Docker Compose already provides.
- Job failure should not crash the API server.
- Jobs do not need distributed execution (single-server deployment).

### Decision

We chose **APScheduler** (`BackgroundScheduler`) running as an in-process thread within the FastAPI application.

Implementation details:
- The scheduler is started during FastAPI's `lifespan` startup event and shut down on application exit.
- Each job function creates its own `SessionLocal()` instance, independent from request-scoped sessions.
- All jobs use try/except/finally to ensure sessions are closed even on failure.
- Jobs are registered with `replace_existing=True` to handle server restarts cleanly.
- The scheduler is a module-level singleton in `jobs/mitre_sync_job.py`.

### Consequences

**Positive:**
- Zero additional infrastructure — no message broker, no worker containers, no job database.
- Jobs share the same Python process, so they can import services directly (`sync_mitre`, `scan_intel`, `create_snapshot`, etc.) without serialization or RPC.
- Simple debugging — job logs appear in the same stdout as API logs.
- Session isolation per job prevents interference with request-scoped transactions.
- `replace_existing=True` prevents duplicate job registrations on hot reload.

**Negative:**
- **No persistence:** If the server crashes mid-job, the job state is lost. There is no retry mechanism — the job simply runs again at the next scheduled interval.
- **No distributed execution:** Cannot run jobs on a separate worker node. If the API is under heavy load, jobs compete for the same CPU and memory.
- **No dead letter queue:** Failed jobs are logged but not queued for retry. A failed MITRE sync silently waits 24 hours before trying again.
- **No job history:** There is no record of when jobs last ran, how long they took, or whether they succeeded — only log lines.
- **Single-instance constraint:** If multiple backend instances are running (horizontal scaling), each instance runs its own scheduler, causing duplicate job execution (double MITRE sync, double snapshots, etc.).
- **No manual trigger via scheduler:** Admin-triggered syncs go through the API endpoints (`/api/v1/system/*`), bypassing the scheduler entirely. There are effectively two paths to the same operations.

**Risks:**
- The single-instance constraint is the most significant risk. If Aegis scales horizontally, APScheduler must be replaced or augmented with a distributed lock (e.g., PostgreSQL advisory locks or Redis-based locking).

### Alternatives Considered

| Alternative | Reason Rejected |
|------------|-----------------|
| **Celery + Redis/RabbitMQ** | Requires an additional broker container (Redis or RabbitMQ), a separate worker process, and Celery configuration. Significant operational overhead for 5 periodic tasks that each run for < 2 minutes. Would be justified if job volume grows or horizontal scaling is needed. |
| **Dramatiq + Redis** | Similar to Celery but lighter. Still requires a Redis container and a separate worker process. Same operational overhead concern. |
| **Cron jobs (host-level)** | Would require the host to have cron configured and scripts that call API endpoints or run Python commands inside the container. Breaks the "single Docker Compose" deployment model. Not portable. |
| **PostgreSQL `pg_cron`** | Runs inside the database, limited to SQL operations. Cannot execute Python logic (downloading ZIPs, parsing YAML, upserting with business rules). Would require stored procedures or external triggers. |
| **Kubernetes CronJobs** | Requires Kubernetes. Not applicable to the Docker Compose deployment model (see ADR-004). |
| **APScheduler with JobStore (PostgreSQL)** | APScheduler supports persistent job stores that would solve the single-instance problem via database locking. This is a viable evolution path — same library, minimal code change, adds distributed-safe execution. **Recommended as the first upgrade when horizontal scaling is needed.** |

---

## ADR Evolution Path

The following table summarizes when each decision should be revisited:

| ADR | Revisit When | Likely Evolution |
|-----|-------------|-----------------|
| ADR-001 (FastAPI) | Stable — no change needed | Add structured logging, OpenTelemetry tracing |
| ADR-002 (PostgreSQL + JSONB) | JSONB query performance degrades | Add GIN indexes on JSONB columns, evaluate moving high-query fields to dedicated columns |
| ADR-003 (MinIO) | Cloud deployment required | Swap boto3 endpoint to AWS S3 / GCS (zero code change) |
| ADR-004 (Docker Compose) | Multi-server deployment needed | Migrate to Kubernetes with Helm charts, or add Ansible playbooks |
| ADR-005 (Modular Monolith) | Team grows > 5 developers, or domains need independent scaling | Enforce boundaries first (Clean Architecture refactor), then extract high-traffic domains as services if needed |
| ADR-006 (APScheduler) | Horizontal scaling required, or jobs need retry/history | Add APScheduler PostgreSQL JobStore first; migrate to Celery if job complexity grows significantly |