Replace sa.Enum column types with raw SQL DO $$ IF NOT EXISTS blocks
for enum creation, then sa.Text columns + ALTER TYPE USING casts.
This completely bypasses SQLAlchemy's _on_table_create hook that
triggers CREATE TYPE without checkfirst, causing DuplicateObject
on PostgreSQL when the entrypoint retries after a failed migration.
Use create_type=False on sa.Enum column references inside
op.create_table so PostgreSQL does not attempt to CREATE TYPE
again after we already created them with checkfirst=True.
Full Jira/Tempo pipeline: link Aegis entities to Jira issues, auto-sync
status hourly, log time internally with integrity hashing, and optionally
push worklogs to Tempo.
- 1.1 JiraLink model + Worklog model: Alembic migration b020 with indexes,
enums (jiralinkentitytype, jirasyncdirection), and integrity_hash column
- 1.2 Jira service: atlassian-python-api wrapper with lazy singleton client,
search/create/sync operations, feature-flagged via JIRA_ENABLED
- 1.3 Jira router: CRUD endpoints for /jira/links, /jira/search,
/jira/create-issue with audit logging and entity-to-issue auto-creation
- 1.4 Tempo service: worklog push via tempo-api-python-client, auto-log from
test completions when TEMPO_ENABLED, graceful fallback on failure
- 1.5 Worklog service + router: immutable internal time records with SHA-256
integrity hash, CRUD at /worklogs, /worklogs/{id}/verify endpoint
- 1.6 Frontend: JiraLinkPanel component (search, link, sync, unlink) and
WorklogTimeline component (timeline view, manual log form) integrated into
TestDetailPage sidebar, CampaignDetailPage grid, TechniqueDetailPage
- 1.7 Jira sync job: APScheduler hourly job syncs all links from Jira,
registered in background scheduler alongside existing jobs
Foundational changes required before any new feature work can begin.
- 0.1 Redis infrastructure: add redis:7-alpine to docker-compose dev and prod,
REDIS_URL config, singleton client in app/infrastructure/redis_client.py
- 0.2 Token blacklist on Redis SEC-001: replace in-memory dict with Redis SETEX
keyed by jti, auto-expiring TTL derived from token exp
- 0.3 Database indexes SR-006: Alembic migration b019 with 5 composite indexes
for scoring, MTTD/MTTR, remediation, and notification queries
- 0.4 Domain exceptions TD-003: app/domain/exceptions.py with typed errors,
error_handler middleware mapping them to HTTP, services decoupled from FastAPI
- 0.5 Fix silenced exceptions TD-007: replace 4 bare except-pass blocks in
test_workflow_service with logger.warning with exc_info
- 0.6 CI pipeline TD-009: GitHub Actions workflow with Postgres and Redis
service containers, ruff lint, pytest; ruff.toml for baseline config
Critical (1-3):
- Replace hardcoded admin credentials with secure auto-generation (seed.py)
- Enforce SECRET_KEY configuration, fail in production if missing (config.py)
- Add Zip Slip and Zip Bomb protection to all ZIP import services
High/Medium (4-9):
- Add 50MB file size limit and extension whitelist to evidence uploads
- Configure CORS origins via environment variable instead of hardcoded
- Migrate JWT storage from localStorage to HttpOnly cookies (frontend+backend)
- Add rate limiting (5/min) on login endpoint via slowapi
- Replace generic dict payloads with Pydantic schemas (mass assignment)
Medium (10-17):
- Check is_active on login to prevent disabled users from authenticating
- Sanitize exception messages in API responses (system, data_sources)
- Escape LIKE wildcards in all ilike search filters across 8 routers
- Run Docker container as non-root user (appuser)
- Make MINIO_SECURE configurable via environment variable
- Add password complexity policy (12+ chars, upper/lower/digit/special)
- Implement JWT token revocation via in-memory blacklist + reduce TTL to 15min
- Replace xml.etree with defusedxml to prevent Billion Laughs attacks
Low (18-20):
- Add security headers to Nginx (CSP, X-Frame-Options, HSTS-ready, etc.)
- Disable Swagger UI/ReDoc/OpenAPI in production
- Restrict /health endpoint to internal networks via Nginx ACL
Also: rewrite install.sh as interactive wizard for guided deployment,
fix test-from-template validation error (technique_id UUID vs MITRE ID)
- Fix hardcoded localhost:8000 URLs in frontend to use relative /api/v1
path (works with Nginx proxy in prod and VITE_API_URL in dev)
- Create production entrypoint (entrypoint.prod.sh) that runs migrations,
seeds, and starts uvicorn with 4 workers (no --reload)
- Create comprehensive install.sh script for production deployment that
generates secure .env, builds containers, waits for health, and
optionally triggers initial MITRE sync
- Update docker-compose.prod.yml to use production entrypoint
- Update Dockerfile to make both entrypoints executable
- Remove init.ps1 (production will always be Linux)
- Update README with production deployment instructions
- Rewrite D3FEND import to use tactic-level APIs for reliable technique
fetching with proper ontology IRIs, descriptions, and tactic assignments
- Fix D3FEND technique URLs to use canonical IRI (no more 404s)
- All 255 D3FEND techniques now have descriptions from the official API
- Change Deactivate button color to red in template management table
- Add custom Aegis logo and favicon replacing default Vite assets
- Remove unused old API parsing code and clean up fallback list
- Make D3FEND defense cards clickable with expandable details and external link
- Fix D3FEND URLs to use PascalCase technique names matching the ontology
- Remove duplicate Import Atomic Red Team from System page (use Data Sources)
- Add bulk Activate All / Deactivate All buttons with confirmation modal
- Fix template admin list to show both active and inactive templates
- Add PATCH /test-templates/bulk-activate backend endpoint
- Auto-seed data sources on container startup via entrypoint.sh
- Fix SigmaHQ, CALDERA, GTFOBins import issues
- Register D3FEND sync handler in data sources router
- Add CIS Controls v8 compliance framework import
- Expand Test Catalog source filters (CALDERA, LOLBAS, GTFOBins)
- Campaign Generate from Threat Actor now opens actor selector modal
- Add coverage snapshot creation button to Comparison page
- Update README with accurate data source and feature documentation
T-119: TestCatalogPage with search, filters (source/platform/severity), template cards grid, and pagination
T-120: TestFromTemplateForm modal with pre-filled fields from template, required field validation, and redirect on creation
T-121: Integrate Available Test Templates section in TechniqueDetailPage with Run This Test buttons; fix missing testStateBadgeColors for new states
Also: add backend entrypoint.sh for automatic Alembic migrations + seed on container startup, add curl to Dockerfile for healthcheck
T-109: Rewrite tests router with full Red/Blue workflow endpoints - list with filters, create from template, Red/Blue team updates with state guards, start-execution, submit-red, submit-blue, validate-red, validate-blue, reopen, and timeline. All using workflow service from Phase 11.
T-110: Rewrite evidence router with Red/Blue separation - upload with team field, list with team filter, delete with state-based permissions. Red Team edits in draft/red_executing, Blue Team in blue_evaluating, admin bypasses all.
T-111: Create test_templates router with full CRUD - paginated list with source/platform/severity/search filters, by-technique lookup, admin-only create/update, and soft delete. Registered in main.py.
T-112: Add POST /system/import-atomic-tests endpoint to system router - admin-only trigger for Atomic Red Team import with error handling and statistics response.
Includes validation tests for all four tasks (35 checks total).
T-106: Create test_workflow_service.py with state-machine transitions for the complete test lifecycle (draft -> red_executing -> blue_evaluating -> in_review -> validated/rejected), dual validation by Red/Blue leads, and reopen capability with field cleanup.
T-107: Update status_service.py to use detection_result from Blue Team instead of legacy result field, and differentiate between partial progress (some validated) vs all-in-progress states.
T-108: Create atomic_import_service.py that downloads the Atomic Red Team repo as a ZIP (avoiding API rate limits), parses all atomics YAML files, and creates idempotent TestTemplate records mapped to MITRE techniques.
Includes validation tests for all three tasks (19 checks total).
- Add intel_service.py: RSS feed scanner for threat intelligence
Searches CISA, NIST NVD, SANS ISC, BleepingComputer, The Hacker News,
Krebs on Security for mentions of MITRE technique IDs and names
- New intel items stored in intel_items table with URL deduplication
- Techniques with new intel flagged with review_required=True
- Add POST /system/run-intel-scan endpoint (admin only)
- Register weekly intel scan job in APScheduler (every 7 days)
- Audit log records each scan execution with summary stats
- Update README with new endpoint and project structure
- Add GET /metrics/summary endpoint with global coverage counts and percentage
- Add GET /metrics/by-tactic endpoint with per-tactic coverage breakdown
- Handle multi-tactic techniques (comma-separated) counting in each tactic
- Add CoverageSummary and TacticCoverage Pydantic schemas
- Update README with metrics endpoints and project structure
- Add MITRE sync service via TAXII 2.0 with GitHub fallback
- Upsert attack-pattern objects into techniques table (691 techniques)
- Detect name/description changes and flag review_required on re-sync
- Add APScheduler background job running every 24h
- Add POST /system/sync-mitre endpoint (admin only)
- Add GET /system/scheduler-status endpoint (admin only)
- Configure logging for scheduler and sync visibility
- Update README with new endpoints and project structure
- Add Pydantic schemas for Technique, Test and Evidence
- Add CRUD endpoints for Techniques (list with filters, detail, create, update, review)
- Add CRUD endpoints for Tests (create, detail, update, validate, reject)
- Add evidence upload with SHA-256 integrity and presigned download URLs
- Add MinIO/S3 storage client with bucket auto-creation on startup
- Add status_service to recalculate technique coverage from test results
- Add require_any_role RBAC dependency for multi-role authorization
- Update README with API endpoints reference and project structure
Implements all database models for the Aegis platform with full
Alembic migration support.
Models created:
- User: Authentication with role-based access control
- Technique: MITRE ATT&CK techniques with coverage status tracking
- Test: Security tests with validation workflow (draft/review/validated)
- Evidence: File metadata for test evidence (stored in MinIO)
- IntelItem: Threat intelligence items linked to techniques
- AuditLog: System-wide audit trail with JSONB details
Enumerations:
- TechniqueStatus: not_evaluated, in_progress, validated, partial, etc.
- TestState: draft, in_review, validated, rejected
- TestResult: detected, not_detected, partially_detected
Services:
- audit_service.py: log_action() helper for audit logging
All models include proper foreign key relationships and PostgreSQL
enum types are managed correctly in migrations (create/drop).
This commit establishes the foundational infrastructure for the Aegis
MITRE ATT&CK Coverage Platform.
T-001: Initialize project and Docker Compose
- Set up Docker Compose with PostgreSQL 15, MinIO, and FastAPI backend
- Create basic FastAPI application with health endpoint
- Configure persistent volumes for data storage
T-002: Configuration and database connection
- Add centralized configuration using pydantic-settings
- Implement SQLAlchemy database connection with session management
- Configure MinIO and JWT settings
T-003: Initialize Alembic for migrations
- Set up Alembic with PostgreSQL connection from settings
- Create initial empty migration
- Configure autogenerate support for future models
Also includes:
- Professional README with setup instructions
- Comprehensive .gitignore for Python/Node/Docker
- Project task plan (AegisTestPlan.md)
