feat: Phase 6 - Automated intel scanning (T-021, T-022)

- Add intel_service.py: RSS feed scanner for threat intelligence
  Searches CISA, NIST NVD, SANS ISC, BleepingComputer, The Hacker News,
  Krebs on Security for mentions of MITRE technique IDs and names
- New intel items stored in intel_items table with URL deduplication
- Techniques with new intel flagged with review_required=True
- Add POST /system/run-intel-scan endpoint (admin only)
- Register weekly intel scan job in APScheduler (every 7 days)
- Audit log records each scan execution with summary stats
- Update README with new endpoint and project structure
This commit is contained in:
2026-02-06 15:48:57 +01:00
parent abdb23be33
commit 52d230628d
4 changed files with 329 additions and 13 deletions

View File

@@ -1,9 +1,13 @@
"""Scheduled job for periodic MITRE ATT&CK synchronisation.
"""Scheduled background jobs.
Uses APScheduler's ``BackgroundScheduler`` to run :func:`sync_mitre` every
24 hours. The job manages its own database session (created on entry,
closed in ``finally``) so it is fully independent from FastAPI's
request-scoped sessions.
Registers periodic tasks on an APScheduler ``BackgroundScheduler``:
* **MITRE sync** — every 24 hours (see :func:`sync_mitre`)
* **Intel scan** — every 7 days (see :func:`scan_intel`)
Each job manages its own database session (created on entry, closed in
``finally``) so it is fully independent from FastAPI's request-scoped
sessions.
"""
import logging
@@ -12,6 +16,7 @@ from apscheduler.schedulers.background import BackgroundScheduler
from app.database import SessionLocal
from app.services.mitre_sync_service import sync_mitre
from app.services.intel_service import scan_intel
logger = logging.getLogger(__name__)
@@ -22,6 +27,11 @@ logger = logging.getLogger(__name__)
scheduler = BackgroundScheduler()
# ---------------------------------------------------------------------------
# Job functions
# ---------------------------------------------------------------------------
def _run_mitre_sync() -> None:
"""Execute a MITRE sync inside its own DB session."""
logger.info("Scheduled MITRE sync job starting...")
@@ -35,11 +45,33 @@ def _run_mitre_sync() -> None:
db.close()
def start_scheduler() -> None:
"""Register the MITRE sync job and start the background scheduler.
def _run_intel_scan() -> None:
"""Execute an intel scan inside its own DB session."""
logger.info("Scheduled intel scan job starting...")
db = SessionLocal()
try:
summary = scan_intel(db)
logger.info("Scheduled intel scan job finished — %s", summary)
except Exception:
logger.exception("Scheduled intel scan job failed")
finally:
db.close()
The job runs every **24 hours**. It does **not** fire immediately on
startup — the first execution happens 24 h after the application boots.
# ---------------------------------------------------------------------------
# Scheduler bootstrap
# ---------------------------------------------------------------------------
def start_scheduler() -> None:
"""Register all periodic jobs and start the background scheduler.
Jobs registered:
* ``mitre_sync`` — every **24 hours**
* ``intel_scan`` — every **7 days**
Neither job fires immediately on startup.
"""
scheduler.add_job(
_run_mitre_sync,
@@ -49,5 +81,13 @@ def start_scheduler() -> None:
name="MITRE ATT&CK sync (every 24h)",
replace_existing=True,
)
scheduler.add_job(
_run_intel_scan,
trigger="interval",
weeks=1,
id="intel_scan",
name="Intel scan (every 7d)",
replace_existing=True,
)
scheduler.start()
logger.info("MITRE sync scheduler started (interval=24h)")
logger.info("Background scheduler started — mitre_sync (24h), intel_scan (7d)")