Commit Graph

59 Commits

Author SHA1 Message Date
kitos 9ff0f04ba3 refactor(types): add comprehensive type annotations across backend Python codebase
Enable ANN rules in ruff.toml (flake8-annotations) and resolve all 221 violations:

ANN201/ANN202 — return types on 168 public/private functions:
- All 28 FastAPI routers: endpoints annotated with dict/list/specific schema/
  StreamingResponse/FileResponse/JSONResponse as appropriate
- main.py: lifespan→AsyncGenerator[None,None], exception handlers→JSONResponse
- database.py: get_db→Generator[Session,None,None], proxy methods→correct types
- middleware/request_context.py: dispatch→Response with Callable call_next type

ANN001/ANN002/ANN003 — 32 missing argument types:
- seed_demo.py: all db parameters typed as Session
- domain/unit_of_work.py: __aexit__ exc_type/exc_val/exc_tb typed with TracebackType
- services: audit_service user_id→UUID|None, heatmap_service query/model/builder,
  notification_service test→Test, tempo_service test→Test/user→User,
  test_workflow_service test_id→UUID, campaign_crud **fields→object,
  test_crud **fields→object (4 sites)

ANN401 — 16 Any usages resolved:
- Domain entities (campaign/technique/threat_actor/test_entity): replaced Any with
  actual ORM types via TYPE_CHECKING guards to avoid circular imports
- detection_rule_service: test_id/detection_rule_id/evaluator_id→UUID
- score_cache: kept Any with # noqa: ANN401 (genuinely generic cache)
- jira_service/tempo_service: kept Any with # noqa: ANN401 (lazy optional deps)
- d3fend_import_service: _to_str(v: Any) kept with # noqa: ANN401

ANN204/ANN205/ANN206 — special/static/class methods:
- database.py proxy __call__/__getattr__: *args: object/**kwargs: object
- schemas/test.py model_validate: obj→object, **kwargs→object
- sa_technique_repository._int_type→type

All 439 unit tests pass. ruff check app/ → All checks passed!
2026-06-11 11:06:54 +02:00
kitos 8f98bdd273 refactor(pep8): enforce full PEP8 compliance across backend Python codebase
- ruff.toml: select E/W/F/I/N rules, line-length=120, drop legacy ignores
- Auto-fix: sort 82 import blocks (isort), remove 29 unused imports,
  strip 6 trailing-whitespace blank lines in docstrings
- main.py: move setup_logging and settings imports to top (E402)
- errors.py: noqa N818 on DDD exception names (96 call sites, safe)
- intel_service.py: noqa N817 for universal ET alias
- atomic/elastic/sigma import services: move _MAX_UNCOMPRESSED_SIZE and
  _MAX_ENTRIES to module level (N806)
- compliance_import_service.py: move SAMPLE_CONTROLS / CIS_CONTROLS to
  module level; wrap long description strings (N806 + E501)
- snapshot_service.py: move STATUS_ORDER dict to module level (N806)
- sigma_import_service.py: remove dead dedup_key expression (F841)
- threat_actor_import_service.py: remove dead stix_to_actor expression (F841)
- data_source.py, seed_demo.py, campaign_scheduler_service.py,
  lolbas_import_service.py: wrap lines exceeding 120 chars (E501)
- d3fend_import_service.py: per-file E501 ignore (data file with long strings)

All 439 unit tests pass. ruff check app/ → All checks passed!
2026-06-11 11:06:54 +02:00
kitos 05b221a22d feat(scoring): composite recency decay and severity weights persisted in DB [FASE-5.1] 2026-05-18 15:07:12 +02:00
kitos 3e854b7b79 feat(security): extend rate limits on sync, tests, evidence and reports [FASE-3.4] 2026-05-18 14:16:53 +02:00
kitos 6b076f52b2 feat(auth): audit login success and failure attempts [FASE-3.2] 2026-05-18 14:16:53 +02:00
kitos b6f23f385d fix(analytics): restrict operators endpoint to admin [FASE-2.5]
Align with BI security spec and add flat JSON API tests for coverage, tests, and operators.
2026-05-18 14:00:47 +02:00
kitos 6ab950ec42 feat(reports): add quarterly and technique download routes [FASE-2.4]
Expose GET endpoints for quarterly-summary and technique reports with PDF, DOCX, and HTML formats.
2026-05-18 14:00:46 +02:00
kitos b8c9c4ac6a test(jira): add hourly sync job tests [FASE-1.7]
Aegis CI / lint-and-test (push) Has been cancelled
Verify skip when disabled, per-link sync invocation, and continued batch on single-link failures.
2026-05-18 13:33:40 +02:00
kitos c5eb6f6dc1 feat(auth): move JWT blacklist to Redis with TTL [FASE-0.2]
Revoke tokens by jti in a dedicated Redis DB, honor TTL from JWT exp on logout, reject revoked tokens in get_current_user, and add FakeRedis-backed API tests.
2026-05-18 13:19:15 +02:00
kitos a9255e15ce refactor: remove db.commit() from audit_service.log_action, all callers use UoW 2026-02-20 15:33:23 +01:00
kitos 14d995b40c refactor: remove db.commit() from business services, callers use UnitOfWork (Tier 3) 2026-02-20 14:42:20 +01:00
kitos 339d669498 feat: move all remaining inline logic from routers to services (Tier 2) 2026-02-20 14:34:24 +01:00
kitos 9e22fde746 feat: extract advanced_metrics, analytics, test_templates, and auth to services (Tier 1 complete) 2026-02-20 14:28:52 +01:00
kitos c0c6cda11d feat: add Campaign/Compliance domain entities and extract users/audit/data_sources to services (LP-2 through LP-6) 2026-02-20 13:28:14 +01:00
kitos f4c74230ec refactor(campaigns): extract CRUD/business logic to campaign_crud_service, use domain exceptions 2026-02-19 19:04:32 +01:00
kitos 50b70704ae refactor(evidence): extract permission validation and queries to evidence_service, use domain exceptions 2026-02-19 19:02:36 +01:00
kitos 20738d11b3 refactor(tests): extract CRUD/query logic to test_crud_service, router delegates to service with domain exceptions 2026-02-19 18:35:09 +01:00
kitos 4e3787d091 refactor(scoring): persist weights in DB table, replace mutable Settings with scoring_config_service 2026-02-19 17:46:02 +01:00
kitos 93fde55389 refactor(threat-actors): extract query/business logic to threat_actor_service, fix N+1 with grouped subqueries
Aegis CI / lint-and-test (push) Has been cancelled
2026-02-19 17:40:00 +01:00
kitos 560fc0c9f0 refactor(detection-rules): extract query/business logic to detection_rule_service, router is thin HTTP adapter 2026-02-19 17:39:31 +01:00
kitos d305db8794 refactor(compliance): extract business logic to compliance_service, use domain exceptions instead of HTTPException 2026-02-19 17:06:32 +01:00
kitos 25fddad17c refactor(metrics): extract query logic to metrics_query_service, thin down router to HTTP adapter 2026-02-19 17:06:07 +01:00
kitos 8d5c5fa80e refactor(reports): extract query and aggregation logic to coverage_report_service, fix N+1 test-count pattern 2026-02-19 15:56:42 +01:00
kitos 2b6d9090c9 refactor(techniques): wire TechniqueRepository into techniques router replacing direct db.query() with repo pattern, domain exceptions, and UnitOfWork 2026-02-19 15:13:52 +01:00
kitos f41b8fd8c2 fix(security): add username validation, constant-time login, default credential rejection, and tooling 2026-02-18 19:11:14 +01:00
kitos e651ef8a8c refactor(heatmap): extract business logic to dedicated service
Aegis CI / lint-and-test (push) Has been cancelled
Move layer dispatch, entity-not-found checks, and validation from router to heatmap_service. Router now only validates requests, calls service, and formats responses (no HTTPException, no business logic). Service raises EntityNotFoundError/BusinessRuleViolation instead of returning None. Add build_navigator_export() for centralized dispatch. 29 new tests (253 total, 0 failures).
2026-02-18 16:09:51 +01:00
kitos 6147abc87a refactor(heatmap): extract business logic to dedicated service
Aegis CI / lint-and-test (push) Has been cancelled
- Create heatmap_service.py with all layer-building logic (coverage, threat-actor, detection-rules, campaign)

- Service is framework-agnostic: no FastAPI imports, no HTTPException, no db.commit()

- Fix N+1 in coverage and threat-actor layers: bulk-fetch test_counts and rule_counts with GROUP BY

- Router reduced from 528 to 140 lines: validates request, calls service, returns response
2026-02-18 13:14:41 +01:00
kitos bfce1a8a0e refactor(core): introduce Unit of Work and remove commits from services
Aegis CI / lint-and-test (push) Has been cancelled
- Add UnitOfWork context manager in domain/unit_of_work.py with commit/rollback/flush API and auto-rollback on exception

- Remove all db.commit() from test_workflow_service (8 calls), notification_service (4 calls), status_service (1 call)

- Services now only stage changes via db.add/db.flush; caller owns the transaction boundary

- Update routers/tests.py: wrap 9 workflow endpoints in UnitOfWork context managers

- Update routers/notifications.py: wrap mark_as_read and mark_all_as_read in UnitOfWork
2026-02-18 12:51:55 +01:00
kitos a4a2adccee feat(phase-39): role-based access control overhaul + forced password change
Aegis CI / lint-and-test (push) Has been cancelled
- Add must_change_password field to User model with migration b023

- Add POST /auth/change-password endpoint with password policy validation

- Add require_password_changed dependency to block requests until password is changed

- Add ChangePasswordModal with live password policy checklist (forced on first login)

- Show password policy in CreateUserModal and EditUserModal

- Fix backend permissions: tests, campaigns, templates, reports, evidence, worklogs

- red_tech/blue_tech: execute only, cannot create tests/campaigns/templates

- red_lead/blue_lead: create/edit tests/campaigns/templates, generate reports, no system access

- viewer: read-only everywhere, can generate reports

- Fix frontend role checks across TestDetailPage, TestDetailHeader, TeamTabs, TestsPage, CampaignsPage, CampaignDetailPage, Sidebar
2026-02-18 10:37:02 +01:00
kitos 8f764d8e39 fix: auto-detect kill chain phase when adding tests to custom campaigns 2026-02-17 17:53:15 +01:00
kitos 222979574a feat(phase-38): automatic intelligence — OSINT enrichment + stale coverage detection
Tarea 4.1 — OSINT Enrichment:
- Add OsintItem model with source_type, severity, CVSS metadata, review flag
- Add Alembic migration b022 with osint_items table and optimized indexes
- Add osint_enrichment_service with NVD API integration, deduplication, rate limiting
- Add OSINT router: GET /osint/items, /osint/summary, /osint/technique/{id}
- Add POST /osint/items/{id}/review to mark items as reviewed
- Add POST /osint/enrich/{technique_id} for manual single-technique enrichment
- Techniques with new CVEs are automatically flagged review_required=True
- Register weekly enrichment job in APScheduler
- Add NVD_API_KEY config setting for optional increased rate limits

Tarea 4.2 — Stale Coverage Detection:
- Add stale_detection_service that flags techniques with no validated test
  in the last N days, or never-validated but with a coverage status
- Configurable threshold via STALE_THRESHOLD_DAYS setting (default 365)
- Register daily stale detection job in APScheduler
- Only flags techniques not already marked review_required
2026-02-17 17:47:47 +01:00
kitos 31e116b4ba feat(phase-37): timer pause/resume + professional reporting engine
Aegis CI / lint-and-test (push) Has been cancelled
Pause/Resume timer:
- Add paused_at, red_paused_seconds, blue_paused_seconds fields to Test model
- Add pause_timer/resume_timer workflow functions with accumulated pause tracking
- Auto-resume on phase submit; subtract paused time from worklog duration
- Add POST /tests/{id}/pause-timer and resume-timer endpoints
- Update LiveTimer component with pause/resume button and paused visual state
- Wire pause/resume mutations through TestDetailPage and TestDetailHeader

Professional Reporting Engine - Fase 2:
- Add ReportEngine service with Jinja2 HTML rendering, WeasyPrint PDF, and docxtpl DOCX
- Add corporate CSS stylesheet with cover page, data tables, stats grid, findings
- Create purple_campaign, coverage_report, and executive_summary HTML templates
- Add report_generation_service collecting domain data for each report type
- Add professional_reports router: GET /reports/generate/purple-campaign/{id}, coverage-summary, executive-summary
- Add analytics router with flat JSON endpoints for PowerBI: /coverage, /tests, /trends, /operators
- Add advanced_metrics router: /coverage-by-tactic, /never-tested, /avg-validation-time, /detection-rate-trend
- Add weasyprint and docxtpl to requirements.txt
- Add REPORT_TEMPLATES_DIR, REPORT_OUTPUT_DIR, COMPANY_NAME, COMPANY_LOGO_PATH to config
2026-02-17 17:20:45 +01:00
kitos 9b98f60a9a feat(phase-35): Jira + Tempo integration with internal worklogs
Aegis CI / lint-and-test (push) Has been cancelled
Full Jira/Tempo pipeline: link Aegis entities to Jira issues, auto-sync
status hourly, log time internally with integrity hashing, and optionally
push worklogs to Tempo.

- 1.1 JiraLink model + Worklog model: Alembic migration b020 with indexes,
  enums (jiralinkentitytype, jirasyncdirection), and integrity_hash column
- 1.2 Jira service: atlassian-python-api wrapper with lazy singleton client,
  search/create/sync operations, feature-flagged via JIRA_ENABLED
- 1.3 Jira router: CRUD endpoints for /jira/links, /jira/search,
  /jira/create-issue with audit logging and entity-to-issue auto-creation
- 1.4 Tempo service: worklog push via tempo-api-python-client, auto-log from
  test completions when TEMPO_ENABLED, graceful fallback on failure
- 1.5 Worklog service + router: immutable internal time records with SHA-256
  integrity hash, CRUD at /worklogs, /worklogs/{id}/verify endpoint
- 1.6 Frontend: JiraLinkPanel component (search, link, sync, unlink) and
  WorklogTimeline component (timeline view, manual log form) integrated into
  TestDetailPage sidebar, CampaignDetailPage grid, TechniqueDetailPage
- 1.7 Jira sync job: APScheduler hourly job syncs all links from Jira,
  registered in background scheduler alongside existing jobs
2026-02-17 15:57:39 +01:00
kitos 6d18a5417d feat(phase-34): resolve blocking tech debt — Redis, domain exceptions, indexes, CI
Aegis CI / lint-and-test (push) Has been cancelled
Foundational changes required before any new feature work can begin.

- 0.1 Redis infrastructure: add redis:7-alpine to docker-compose dev and prod,
  REDIS_URL config, singleton client in app/infrastructure/redis_client.py
- 0.2 Token blacklist on Redis SEC-001: replace in-memory dict with Redis SETEX
  keyed by jti, auto-expiring TTL derived from token exp
- 0.3 Database indexes SR-006: Alembic migration b019 with 5 composite indexes
  for scoring, MTTD/MTTR, remediation, and notification queries
- 0.4 Domain exceptions TD-003: app/domain/exceptions.py with typed errors,
  error_handler middleware mapping them to HTTP, services decoupled from FastAPI
- 0.5 Fix silenced exceptions TD-007: replace 4 bare except-pass blocks in
  test_workflow_service with logger.warning with exc_info
- 0.6 CI pipeline TD-009: GitHub Actions workflow with Postgres and Redis
  service containers, ruff lint, pytest; ruff.toml for baseline config
2026-02-17 15:43:05 +01:00
kitos 64d64080e0 fix: resolve 20 security vulnerabilities from comprehensive audit
Critical (1-3):
- Replace hardcoded admin credentials with secure auto-generation (seed.py)
- Enforce SECRET_KEY configuration, fail in production if missing (config.py)
- Add Zip Slip and Zip Bomb protection to all ZIP import services

High/Medium (4-9):
- Add 50MB file size limit and extension whitelist to evidence uploads
- Configure CORS origins via environment variable instead of hardcoded
- Migrate JWT storage from localStorage to HttpOnly cookies (frontend+backend)
- Add rate limiting (5/min) on login endpoint via slowapi
- Replace generic dict payloads with Pydantic schemas (mass assignment)

Medium (10-17):
- Check is_active on login to prevent disabled users from authenticating
- Sanitize exception messages in API responses (system, data_sources)
- Escape LIKE wildcards in all ilike search filters across 8 routers
- Run Docker container as non-root user (appuser)
- Make MINIO_SECURE configurable via environment variable
- Add password complexity policy (12+ chars, upper/lower/digit/special)
- Implement JWT token revocation via in-memory blacklist + reduce TTL to 15min
- Replace xml.etree with defusedxml to prevent Billion Laughs attacks

Low (18-20):
- Add security headers to Nginx (CSP, X-Frame-Options, HSTS-ready, etc.)
- Disable Swagger UI/ReDoc/OpenAPI in production
- Restrict /health endpoint to internal networks via Nginx ACL

Also: rewrite install.sh as interactive wizard for guided deployment,
fix test-from-template validation error (technique_id UUID vs MITRE ID)
2026-02-11 08:56:26 +01:00
kitos c2e9c687f4 fix: D3FEND expandable cards, System page cleanup, and multi-source improvements
- Make D3FEND defense cards clickable with expandable details and external link
- Fix D3FEND URLs to use PascalCase technique names matching the ontology
- Remove duplicate Import Atomic Red Team from System page (use Data Sources)
- Add bulk Activate All / Deactivate All buttons with confirmation modal
- Fix template admin list to show both active and inactive templates
- Add PATCH /test-templates/bulk-activate backend endpoint
- Auto-seed data sources on container startup via entrypoint.sh
- Fix SigmaHQ, CALDERA, GTFOBins import issues
- Register D3FEND sync handler in data sources router
- Add CIS Controls v8 compliance framework import
- Expand Test Catalog source filters (CALDERA, LOLBAS, GTFOBins)
- Campaign Generate from Threat Actor now opens actor selector modal
- Add coverage snapshot creation button to Comparison page
- Update README with accurate data source and feature documentation
2026-02-10 13:22:23 +01:00
kitos 14f8485f06 feat(phase-33): final polish V3 - navigation, performance, and documentation (T-238 to T-240) 2026-02-10 09:21:35 +01:00
kitos 02034d60f0 feat(phase-31): add campaign scheduling and recurring automation (T-233 to T-234) 2026-02-10 08:38:00 +01:00
kitos 4d124b42dd feat(phase-30): add coverage snapshots, temporal comparison and auto re-testing (T-230 to T-232) 2026-02-10 08:34:29 +01:00
kitos 2ac8e7f4a5 feat(phase-29): add compliance framework mapping, reports and UI (T-227 to T-229) 2026-02-09 18:41:24 +01:00
kitos 12f33307fd feat(phase-28): add scoring system, operational metrics and executive dashboard (T-224 to T-226) 2026-02-09 17:24:44 +01:00
kitos a911ddeb52 feat(phase-27): add advanced ATT&CK Navigator-style heatmap with layers, filters and export (T-221 to T-223) 2026-02-09 17:16:59 +01:00
kitos 57b47c296d feat(phase-26): add Campaign models, endpoints, service with kill chain timeline UI (T-217 to T-220) 2026-02-09 16:52:52 +01:00
kitos f4de12d8ab feat(phase-25): add detection rule associations, checklist UI and evaluation workflow (T-215, T-216) 2026-02-09 16:44:35 +01:00
kitos cd124b655b feat(phase-24): integrate MITRE D3FEND defensive techniques with ATT&CK mapping (T-213, T-214) 2026-02-09 16:38:59 +01:00
kitos 2fc0e2cafd feat(phase-23): add Threat Actor profiles with MITRE CTI import, API, heatmap and gap analysis (T-208 to T-212) 2026-02-09 16:27:38 +01:00
kitos f4c8cbf768 feat(phase-22): add import services for Sigma, LOLBAS, GTFOBins, CALDERA, Elastic and data sources panel (T-203 to T-207) 2026-02-09 16:19:44 +01:00
kitos 29eab4ef77 feat(phase-20): navigation, error handling, integration tests, and V2 docs (T-132 to T-135) 2026-02-09 14:19:42 +01:00
kitos 9ea6ce1326 feat(phase-19): add remediation fields and reports system (T-130, T-131) 2026-02-09 13:58:35 +01:00
kitos fb7f340038 feat(phase-18): add in-app notification system (T-128, T-129) 2026-02-09 13:52:04 +01:00