SQLAlchemy fires before_create for ALL known enum types when any table
is created via op.create_table, causing DuplicateObject even with
create_type=False. Rewrite both CREATE TABLE statements as raw SQL via
conn.execute(sa.text(...)) and use CREATE TABLE IF NOT EXISTS / CREATE
INDEX IF NOT EXISTS for full idempotency.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace _enum_exists() helper (which had connection context issues in
Alembic) with PostgreSQL DO $$ BEGIN ... EXCEPTION WHEN duplicate_object
THEN NULL; END $$ blocks, which are truly idempotent regardless of
transaction state.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Phase 6.1: WebhookConfig model, CRUD router (/api/v1/webhooks, admin-only),
dispatch_webhook() with HMAC signing; integrated into test validation,
campaign completion, and MITRE sync job
- Phase 7.1: SMTP email service with send_test_validated_email,
send_campaign_completed_email, send_new_mitre_techniques_email;
notify_role_with_email() added to notification_service
- Phase 7.2: notification_preferences and jira_account_id on User model;
PATCH /users/me/preferences endpoint; Alembic migrations b031phase6 and b032phase7
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add UniqueConstraint(test_id, detection_rule_id) named uq_tdr_test_rule to TestDetectionResult model
- Alembic b025: safely deduplicate existing rows before creating constraint
- Declare __table_args__ on Test with 5 indexes: technique_id, state, created_at, (technique_id,state), (state,created_at)
- Declare __table_args__ on AuditLog with 3 indexes: (entity_type,entity_id), timestamp, (entity_type,entity_id,action)
- Alembic b024: create only the 2 new indexes (ix_tests_created_at, ix_tests_state_created_at); existing indexes from b005/b018/b019 are preserved
- Model index names aligned with existing migration names to prevent duplicates
- Add must_change_password field to User model with migration b023
- Add POST /auth/change-password endpoint with password policy validation
- Add require_password_changed dependency to block requests until password is changed
- Add ChangePasswordModal with live password policy checklist (forced on first login)
- Show password policy in CreateUserModal and EditUserModal
- Fix backend permissions: tests, campaigns, templates, reports, evidence, worklogs
- red_tech/blue_tech: execute only, cannot create tests/campaigns/templates
- red_lead/blue_lead: create/edit tests/campaigns/templates, generate reports, no system access
- viewer: read-only everywhere, can generate reports
- Fix frontend role checks across TestDetailPage, TestDetailHeader, TeamTabs, TestsPage, CampaignsPage, CampaignDetailPage, Sidebar
Tarea 4.1 — OSINT Enrichment:
- Add OsintItem model with source_type, severity, CVSS metadata, review flag
- Add Alembic migration b022 with osint_items table and optimized indexes
- Add osint_enrichment_service with NVD API integration, deduplication, rate limiting
- Add OSINT router: GET /osint/items, /osint/summary, /osint/technique/{id}
- Add POST /osint/items/{id}/review to mark items as reviewed
- Add POST /osint/enrich/{technique_id} for manual single-technique enrichment
- Techniques with new CVEs are automatically flagged review_required=True
- Register weekly enrichment job in APScheduler
- Add NVD_API_KEY config setting for optional increased rate limits
Tarea 4.2 — Stale Coverage Detection:
- Add stale_detection_service that flags techniques with no validated test
in the last N days, or never-validated but with a coverage status
- Configurable threshold via STALE_THRESHOLD_DAYS setting (default 365)
- Register daily stale detection job in APScheduler
- Only flags techniques not already marked review_required
Pause/Resume timer:
- Add paused_at, red_paused_seconds, blue_paused_seconds fields to Test model
- Add pause_timer/resume_timer workflow functions with accumulated pause tracking
- Auto-resume on phase submit; subtract paused time from worklog duration
- Add POST /tests/{id}/pause-timer and resume-timer endpoints
- Update LiveTimer component with pause/resume button and paused visual state
- Wire pause/resume mutations through TestDetailPage and TestDetailHeader
Professional Reporting Engine - Fase 2:
- Add ReportEngine service with Jinja2 HTML rendering, WeasyPrint PDF, and docxtpl DOCX
- Add corporate CSS stylesheet with cover page, data tables, stats grid, findings
- Create purple_campaign, coverage_report, and executive_summary HTML templates
- Add report_generation_service collecting domain data for each report type
- Add professional_reports router: GET /reports/generate/purple-campaign/{id}, coverage-summary, executive-summary
- Add analytics router with flat JSON endpoints for PowerBI: /coverage, /tests, /trends, /operators
- Add advanced_metrics router: /coverage-by-tactic, /never-tested, /avg-validation-time, /detection-rate-trend
- Add weasyprint and docxtpl to requirements.txt
- Add REPORT_TEMPLATES_DIR, REPORT_OUTPUT_DIR, COMPANY_NAME, COMPANY_LOGO_PATH to config
- Add red_started_at/blue_started_at timing fields to Test model with Alembic migration
- Modify workflow transitions to auto-create integrity-hashed worklogs: Start Execution records red_started_at, Submit to Blue Team stops Red timer and creates worklog then starts Blue timer, Submit for Review stops Blue timer and creates worklog
- Auto-sync worklogs to Tempo when test has a Jira link
- Add LiveTimer component showing real-time elapsed counter during active phases
- Clear timing fields on test reopen
- Fix campaign test management: replace broken navigate-to-tests flow with AddTestToCampaignModal that lets users search and add existing tests directly from the campaign detail page
Replace all sa.Enum / op.create_table / ALTER TABLE approach with a
single op.execute() containing raw DDL. This sidesteps every
SQLAlchemy hook (enum auto-create, default cast conflicts) by letting
PostgreSQL handle CREATE TYPE IF NOT EXISTS, CREATE TABLE IF NOT
EXISTS, and CREATE INDEX IF NOT EXISTS directly.
Replace sa.Enum column types with raw SQL DO $$ IF NOT EXISTS blocks
for enum creation, then sa.Text columns + ALTER TYPE USING casts.
This completely bypasses SQLAlchemy's _on_table_create hook that
triggers CREATE TYPE without checkfirst, causing DuplicateObject
on PostgreSQL when the entrypoint retries after a failed migration.
Use create_type=False on sa.Enum column references inside
op.create_table so PostgreSQL does not attempt to CREATE TYPE
again after we already created them with checkfirst=True.
Full Jira/Tempo pipeline: link Aegis entities to Jira issues, auto-sync
status hourly, log time internally with integrity hashing, and optionally
push worklogs to Tempo.
- 1.1 JiraLink model + Worklog model: Alembic migration b020 with indexes,
enums (jiralinkentitytype, jirasyncdirection), and integrity_hash column
- 1.2 Jira service: atlassian-python-api wrapper with lazy singleton client,
search/create/sync operations, feature-flagged via JIRA_ENABLED
- 1.3 Jira router: CRUD endpoints for /jira/links, /jira/search,
/jira/create-issue with audit logging and entity-to-issue auto-creation
- 1.4 Tempo service: worklog push via tempo-api-python-client, auto-log from
test completions when TEMPO_ENABLED, graceful fallback on failure
- 1.5 Worklog service + router: immutable internal time records with SHA-256
integrity hash, CRUD at /worklogs, /worklogs/{id}/verify endpoint
- 1.6 Frontend: JiraLinkPanel component (search, link, sync, unlink) and
WorklogTimeline component (timeline view, manual log form) integrated into
TestDetailPage sidebar, CampaignDetailPage grid, TechniqueDetailPage
- 1.7 Jira sync job: APScheduler hourly job syncs all links from Jira,
registered in background scheduler alongside existing jobs
Foundational changes required before any new feature work can begin.
- 0.1 Redis infrastructure: add redis:7-alpine to docker-compose dev and prod,
REDIS_URL config, singleton client in app/infrastructure/redis_client.py
- 0.2 Token blacklist on Redis SEC-001: replace in-memory dict with Redis SETEX
keyed by jti, auto-expiring TTL derived from token exp
- 0.3 Database indexes SR-006: Alembic migration b019 with 5 composite indexes
for scoring, MTTD/MTTR, remediation, and notification queries
- 0.4 Domain exceptions TD-003: app/domain/exceptions.py with typed errors,
error_handler middleware mapping them to HTTP, services decoupled from FastAPI
- 0.5 Fix silenced exceptions TD-007: replace 4 bare except-pass blocks in
test_workflow_service with logger.warning with exc_info
- 0.6 CI pipeline TD-009: GitHub Actions workflow with Postgres and Redis
service containers, ruff lint, pytest; ruff.toml for baseline config
T-106: Create test_workflow_service.py with state-machine transitions for the complete test lifecycle (draft -> red_executing -> blue_evaluating -> in_review -> validated/rejected), dual validation by Red/Blue leads, and reopen capability with field cleanup.
T-107: Update status_service.py to use detection_result from Blue Team instead of legacy result field, and differentiate between partial progress (some validated) vs all-in-progress states.
T-108: Create atomic_import_service.py that downloads the Atomic Red Team repo as a ZIP (avoiding API rate limits), parses all atomics YAML files, and creates idempotent TestTemplate records mapped to MITRE techniques.
Includes validation tests for all three tasks (19 checks total).
Implements all database models for the Aegis platform with full
Alembic migration support.
Models created:
- User: Authentication with role-based access control
- Technique: MITRE ATT&CK techniques with coverage status tracking
- Test: Security tests with validation workflow (draft/review/validated)
- Evidence: File metadata for test evidence (stored in MinIO)
- IntelItem: Threat intelligence items linked to techniques
- AuditLog: System-wide audit trail with JSONB details
Enumerations:
- TechniqueStatus: not_evaluated, in_progress, validated, partial, etc.
- TestState: draft, in_review, validated, rejected
- TestResult: detected, not_detected, partially_detected
Services:
- audit_service.py: log_action() helper for audit logging
All models include proper foreign key relationships and PostgreSQL
enum types are managed correctly in migrations (create/drop).
This commit establishes the foundational infrastructure for the Aegis
MITRE ATT&CK Coverage Platform.
T-001: Initialize project and Docker Compose
- Set up Docker Compose with PostgreSQL 15, MinIO, and FastAPI backend
- Create basic FastAPI application with health endpoint
- Configure persistent volumes for data storage
T-002: Configuration and database connection
- Add centralized configuration using pydantic-settings
- Implement SQLAlchemy database connection with session management
- Configure MinIO and JWT settings
T-003: Initialize Alembic for migrations
- Set up Alembic with PostgreSQL connection from settings
- Create initial empty migration
- Configure autogenerate support for future models
Also includes:
- Professional README with setup instructions
- Comprehensive .gitignore for Python/Node/Docker
- Project task plan (AegisTestPlan.md)
