# Aegis — Data Sources

Aegis imports security data from multiple external sources to populate its test catalog, detection rules, defensive techniques, threat actors, and compliance mappings. This document describes each source, its format, and how to manage imports.

---

## Overview

| Source | Type | Format | Destination |
|--------|------|--------|-------------|
| MITRE ATT&CK | Techniques | STIX 2.0 (TAXII / GitHub) | `techniques` |
| Atomic Red Team | Test Templates | YAML | `test_templates` |
| SigmaHQ | Detection Rules | YAML | `detection_rules` |
| Elastic Detection Rules | Detection Rules | TOML | `detection_rules` |
| CALDERA | Test Templates | YAML (multi-doc) | `test_templates` |
| LOLBAS | Test Templates | YAML | `test_templates` |
| MITRE D3FEND | Defensive Techniques | JSON-LD | `defensive_techniques` + mappings |
| MITRE CTI | Threat Actors | STIX 2.0 (JSON) | `threat_actors` + technique mappings |
| NIST 800-53 → ATT&CK | Compliance Mappings | STIX 2.0 (JSON) | `compliance_*` tables |

---

## MITRE ATT&CK (Techniques)

**Repository:** https://github.com/mitre/cti (Enterprise ATT&CK)
**Protocol:** TAXII 2.0 with GitHub JSON fallback
**Format:** STIX 2.0 bundles
**Service:** `mitre_sync_service.py`
**Schedule:** Automatic every 24 hours via APScheduler

**Extracted fields:**
- `mitre_id` — External ID (e.g., T1059, T1059.001)
- `name` — Technique name
- `description` — Full description
- `tactic` — ATT&CK tactic (execution, persistence, etc.)
- `platforms` — Target platforms (windows, linux, macos, etc.)
- `is_subtechnique` — Whether it's a sub-technique
- `url` — Link to MITRE page

**Manual trigger:**
```bash
# Via API
curl -X POST http://localhost:8000/api/v1/system/sync-mitre \
  -H "Authorization: Bearer $TOKEN"

# Via container
docker exec aegis-backend python -c "from app.services.mitre_sync_service import sync_mitre_attack; sync_mitre_attack()"
```

**Volume:** ~700 techniques (Enterprise ATT&CK v16)

**Troubleshooting:**
- If TAXII fails (timeout/rate limit), the service automatically falls back to the GitHub JSON bundle
- Check `data_sources` table for `last_sync_at` and `last_sync_stats`

---

## Atomic Red Team (Test Templates)

**Repository:** https://github.com/redcanaryco/atomic-red-team
**Format:** YAML (one file per technique under `atomics/T*/T*.yaml`)
**Service:** `atomic_import_service.py`

**Extracted fields:**
- `name` — Test name
- `description` — What the test does
- `mitre_technique_id` — ATT&CK technique ID
- `attack_commands` — Commands to execute
- `expected_detection` — What should be detected
- `platform` — Target OS
- `severity` — Derived from technique context
- `cleanup_commands` — Cleanup procedures

**Import:**
```bash
curl -X POST http://localhost:8000/api/v1/system/import-atomic-red-team \
  -H "Authorization: Bearer $TOKEN"
```

**Volume:** ~3,500 test templates
**Frequency:** Monthly or after ATT&CK version updates

---

## SigmaHQ (Detection Rules)

**Repository:** https://github.com/SigmaHQ/sigma
**Format:** YAML (Sigma rule format)
**Service:** `sigma_import_service.py`

**Extracted fields:**
- `name` — Rule title
- `description` — Rule description
- `query` — Sigma detection logic
- `mitre_technique_id` — Extracted from `tags: attack.tXXXX`
- `severity` — From `level` field (low, medium, high, critical)
- `platforms` — From `logsource.product`
- `source` = `"sigma"`

**Example Sigma rule tags:**
```yaml
tags:
  - attack.execution
  - attack.t1059.001
  - attack.defense_evasion
  - attack.t1562.001
```

**Volume:** ~3,000 detection rules
**Frequency:** Monthly recommended

**Troubleshooting:**
- Rules without MITRE technique tags in `attack.tXXXX` format are skipped
- Duplicate detection is by `name` + `source` + `mitre_technique_id`

---

## Elastic Detection Rules

**Repository:** https://github.com/elastic/detection-rules
**Format:** TOML (one file per rule under `rules/`)
**Service:** `elastic_import_service.py`

**Extracted fields:**
- `name` — Rule name from `[rule]`
- `description` — Rule description
- `query` — KQL/EQL query
- `mitre_technique_id` — From `[[rule.threat]]` entries
- `severity` — From `rule.severity`
- `rule_type` — eql, query, threshold, etc.
- `source` = `"elastic"`

**TOML structure:**
```toml
[rule]
name = "Scheduled Task Created via Schtasks"
severity = "medium"
type = "eql"

[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1053"
name = "Scheduled Task/Job"
[[rule.threat.technique.subtechnique]]
id = "T1053.005"
```

**Volume:** ~1,200 detection rules
**Frequency:** Quarterly recommended

---

## CALDERA (Test Templates)

**Repository:** https://github.com/mitre/caldera
**Format:** YAML (multi-document, abilities under `data/abilities/`)
**Service:** `caldera_import_service.py`

**Extracted fields:**
- `name` — Ability name
- `description` — What the ability does
- `mitre_technique_id` — From `technique.attack_id`
- `tactic` — ATT&CK tactic
- `platforms` — Target platforms
- `attack_commands` — Commands per platform/executor

**Volume:** ~500 abilities
**Frequency:** Quarterly recommended

---

## LOLBAS (Test Templates)

**Repository:** https://github.com/LOLBAS-Project/LOLBAS
**Format:** YAML (one file per binary under `yml/OSBinaries/`, `yml/OtherMSBinaries/`, etc.)
**Service:** `lolbas_import_service.py`

**Extracted fields:**
- `name` — Binary name (e.g., Mshta.exe)
- `mitre_technique_id` — From `Commands[].MitreID`
- `attack_commands` — From `Commands[].Command`
- `description` — From `Commands[].Description`
- `usecase` — From `Commands[].Usecase`

**Volume:** ~200 living-off-the-land binaries with ~500 commands
**Frequency:** Quarterly recommended

---

## MITRE D3FEND (Defensive Techniques)

**Repository:** https://d3fend.mitre.org/
**Format:** JSON-LD (REST API at `https://d3fend.mitre.org/api/`)
**Service:** `d3fend_import_service.py`

**Extracted fields:**
- `d3fend_id` — D3FEND identifier (e.g., D3-AL, D3-NI)
- `name` — Defensive technique name
- `description` — Definition or comment
- `tactic` — Defensive tactic (Detect, Isolate, Deceive, Evict, Harden)
- ATT&CK ↔ D3FEND mappings stored in `defensive_technique_mappings`

**Volume:** ~200 defensive techniques
**Frequency:** Annually (D3FEND updates are infrequent)

---

## MITRE CTI — Threat Actors

**Repository:** https://github.com/mitre/cti (enterprise-attack)
**Format:** STIX 2.0 JSON bundles (`intrusion-set`, `relationship`, `attack-pattern`)
**Service:** `threat_actor_import_service.py`

**Extracted fields:**
- `name` — Actor name (e.g., APT28)
- `mitre_id` — MITRE group ID (e.g., G0007)
- `aliases` — Known aliases (JSONB array)
- `description` — Full description
- `country` — Attribution (when available)
- `motivation` — Espionage, financial, etc.
- `target_sectors` / `target_regions` — JSONB arrays
- Technique mappings via `relationship` objects

**Volume:** ~140 threat actors with ~2,000 technique mappings
**Frequency:** Quarterly recommended

---

## NIST 800-53 → ATT&CK (Compliance)

**Repository:** https://github.com/center-for-threat-informed-defense/attack-control-framework-mappings
**Format:** STIX 2.0 JSON bundles
**Service:** `compliance_import_service.py`

**Extracted fields:**
- `ComplianceFramework` — Framework name and version
- `ComplianceControl` — Control ID (e.g., AC-2), title, category
- `ComplianceControlMapping` — Control ↔ ATT&CK technique associations

**Volume:** ~1,000 controls with ~5,000 mappings
**Frequency:** Annually (mappings are versioned with the framework)

---

## Managing Data Sources

### Admin UI

Navigate to **System → Data Sources** in the Aegis frontend to:
- View all configured data sources and their sync status
- Trigger manual imports
- Enable/disable individual sources
- View import statistics (imported, updated, errors)

### API Endpoints

```bash
# List data sources
GET /api/v1/data-sources

# Trigger import for a specific source
POST /api/v1/data-sources/{id}/sync

# Enable/disable a source
PATCH /api/v1/data-sources/{id}
```

### Recommended Update Schedule

| Source | Frequency | Reason |
|--------|-----------|--------|
| MITRE ATT&CK | Automatic (24h) | Core framework, frequent updates |
| Atomic Red Team | Monthly | Active community contributions |
| SigmaHQ | Monthly | Active community contributions |
| Elastic Rules | Quarterly | Major version-aligned releases |
| CALDERA | Quarterly | Less frequent updates |
| LOLBAS | Quarterly | Less frequent updates |
| D3FEND | Annually | Infrequent updates |
| CTI Actors | Quarterly | New groups and campaigns |
| NIST 800-53 | Annually | Framework revision cycles |
