diff --git a/QA-Testing-Guide.-.md b/QA-Testing-Guide.-.md index e69de29..2d547f7 100644 --- a/QA-Testing-Guide.-.md +++ b/QA-Testing-Guide.-.md @@ -0,0 +1,285 @@ +# QA Testing Guide + +This page documents the automated QA runner and manual testing checklist for Aegis. +The QA suite validates access control, test lifecycle, API key scoping, and security +controls. + +--- + +## Automated QA Runner + +The automated QA runner is located at `scripts/qa_runner.py`. + +### What it Tests + +The runner executes **77 automated tests** organized as follows: + +1. **Setup** — Creates test users for all 6 roles (admin, red_lead, blue_lead, + red_tech, blue_tech, viewer). Creates a test technique. + +2. **Authentication tests** + - Login for each role + - Verify `/auth/me` returns correct role + - Verify `must_change_password` blocks all endpoints + - Verify logout revokes token (subsequent request returns 401) + +3. **Full test lifecycle** (happy path) + - Create test as red_lead → draft + - Start execution as red_tech → red_executing + - Verify blue_tech cannot submit-red (403) + - Submit red as red_tech → blue_evaluating + - Verify red_tech cannot update blue fields (403) + - Update blue fields as blue_tech + - Submit blue as blue_tech → in_review + - Validate red as red_lead → red approved + - Validate blue as blue_lead → both approved → validated + - Verify technique coverage updated + +4. **Access control denials (403 checks)** + For each protected endpoint, verify forbidden roles receive 403: + - viewer cannot POST /tests + - viewer cannot POST /tests/{id}/start-execution + - blue_tech cannot PATCH /tests/{id}/red + - red_tech cannot PATCH /tests/{id}/blue + - blue_lead cannot POST /tests/{id}/complete (campaign) + - red_tech cannot GET /audit-logs + - non-admin cannot POST /webhooks + - non-admin cannot DELETE /snapshots/{id} + - non-admin cannot POST /system/sync-mitre + +5. **API Key scope enforcement** + - Create read-scope key as red_lead + - Verify GET /tests returns 200 with read key + - Verify POST /tests returns 403 with read key + - Verify PATCH /tests/{id} returns 403 with read key + - Create write-scope key + - Verify POST /tests returns 201 with write key + +6. **SSRF webhook protection** + - Attempt to create webhook pointing to 192.168.1.1 → expect 400 (SSRF blocked) + - Attempt to create webhook pointing to 127.0.0.1 → expect 400 + - Attempt to create webhook pointing to 10.0.0.1 → expect 400 + - Attempt with valid public URL → expect 201 + +7. **Campaign lifecycle** + - Create campaign as red_lead + - Add tests to campaign + - Activate campaign + - Verify viewer can read campaign progress + - Verify blue_lead cannot complete campaign (403) + - Complete campaign as red_lead + +8. **Cleanup** + - Delete all created tests + - Delete all created campaigns + - Delete all created users + - Verify cleanup success + +### Running the QA Suite + +```bash +# Against local dev +python scripts/qa_runner.py --base-url http://localhost:8000 + +# Against production (use with caution) +python scripts/qa_runner.py --base-url https://aegis.yourcompany.com + +# Verbose mode (shows each HTTP request) +python scripts/qa_runner.py --base-url http://localhost:8000 --verbose + +# Run specific phase only +python scripts/qa_runner.py --base-url http://localhost:8000 --phase lifecycle +``` + +### Exit Codes + +| Code | Meaning | +|------|---------| +| 0 | All tests passed | +| 1 | One or more tests failed | +| 2 | Setup failed (cannot run further tests) | + +### Sample Output + +``` +Creating Aegis QA Test Run... +[SETUP] Creating test users... + [OK] Created admin user: qa_admin_20240315 + [OK] Created red_lead user: qa_rl_20240315 + [OK] Created blue_lead user: qa_bl_20240315 + [OK] Created red_tech user: qa_rt_20240315 + [OK] Created blue_tech user: qa_bt_20240315 + [OK] Created viewer user: qa_v_20240315 + +[PHASE 1] Authentication... + [OK] Login as admin + [OK] Login as red_lead + ... + +[PHASE 2] Test Lifecycle... + [OK] Create test (red_lead) + [OK] Start execution (red_tech) + [FAIL] 403 for viewer trying start-execution — expected 403, got 403 ✓ + ... + +Results: 77/77 passed +``` + +--- + +## Manual QA Checklist + +### admin role + +- [ ] Login succeeds +- [ ] Change password on first login +- [ ] Can list all users (`GET /users`) +- [ ] Can create a new user (POST /users) +- [ ] Can delete a user (DELETE /users/{id}) +- [ ] Can access audit logs (`GET /audit-logs`) +- [ ] Can trigger MITRE sync (POST /system/sync-mitre) +- [ ] Can create a webhook (POST /webhooks) +- [ ] Can view scheduler status (`GET /system/scheduler-status`) +- [ ] Can delete a snapshot (DELETE /snapshots/{id}) +- [ ] Can set data classification on a test (PATCH /tests/{id}/classification) +- [ ] Can complete a campaign (POST /campaigns/{id}/complete) +- [ ] Can validate both red and blue sides of a test + +### red_lead role + +- [ ] Login succeeds +- [ ] Can create a test (POST /tests) +- [ ] Can start test execution +- [ ] Can validate red side of test in `in_review` +- [ ] Can create a campaign and add tests +- [ ] Can complete a campaign +- [ ] Can create a playbook +- [ ] Can create a lesson learned +- [ ] Cannot access audit logs → 403 +- [ ] Cannot access webhooks → 403 +- [ ] Cannot validate blue side → 403 +- [ ] Cannot delete snapshots → 403 +- [ ] Can generate a PDF report + +### blue_lead role + +- [ ] Login succeeds +- [ ] Can validate blue side of test in `in_review` +- [ ] Can create detection assets and link to techniques +- [ ] Can create detection validations +- [ ] Can generate revalidation queue +- [ ] Cannot start test execution → 403 +- [ ] Cannot complete a campaign → 403 +- [ ] Cannot validate red side → 403 +- [ ] Cannot access webhooks → 403 +- [ ] Can generate a PDF report + +### red_tech role + +- [ ] Login succeeds +- [ ] Can start test execution (test must be in draft) +- [ ] Can update red fields (PATCH /tests/{id}/red) in red_executing state +- [ ] Can upload red evidence +- [ ] Can submit red (POST /tests/{id}/submit-red) +- [ ] Cannot create a test → 403 +- [ ] Cannot update blue fields → 403 +- [ ] Cannot validate either side → 403 +- [ ] Cannot generate reports → 403 +- [ ] Can pause/resume timer + +### blue_tech role + +- [ ] Login succeeds +- [ ] Can update blue fields (PATCH /tests/{id}/blue) in blue_evaluating state +- [ ] Can upload blue evidence (only in blue_evaluating state) +- [ ] Can submit blue (POST /tests/{id}/submit-blue) +- [ ] Can evaluate detection rules (POST /detection-rules/evaluate) +- [ ] Cannot start execution → 403 +- [ ] Cannot submit red → 403 +- [ ] Cannot validate either side → 403 +- [ ] Cannot create tests → 403 +- [ ] Can pause/resume timer + +### viewer role + +- [ ] Login succeeds +- [ ] Can read tests, campaigns, techniques, coverage, heatmap, dashboard +- [ ] Can generate PDF/DOCX/HTML reports +- [ ] Cannot create a test → 403 +- [ ] Cannot start execution → 403 +- [ ] Cannot update any test fields → 403 +- [ ] Cannot upload evidence → 403 +- [ ] Cannot validate → 403 +- [ ] Cannot create campaigns → 403 +- [ ] Cannot create playbooks or lessons → 403 +- [ ] Cannot pause/resume timer → 403 + +--- + +## Security QA Checklist + +### SSRF Protection + +- [ ] POST /webhooks with URL `http://192.168.1.1/evil` → 400 Bad Request +- [ ] POST /webhooks with URL `http://10.0.0.1/evil` → 400 Bad Request +- [ ] POST /webhooks with URL `http://127.0.0.1/evil` → 400 Bad Request +- [ ] POST /webhooks with URL `http://169.254.169.254/latest/meta-data` → 400 +- [ ] POST /webhooks with URL `https://hooks.slack.com/services/valid` → 201 OK + +### API Key Scopes + +- [ ] Read-scope key: GET requests succeed (200) +- [ ] Read-scope key: POST requests fail (403) +- [ ] Read-scope key: PATCH requests fail (403) +- [ ] Read-scope key: DELETE requests fail (403) +- [ ] Write-scope key: POST requests succeed (201) +- [ ] Admin-scope key: admin-only endpoints succeed + +### Token Blacklisting + +- [ ] Logout request succeeds +- [ ] Using old token after logout returns 401 +- [ ] New login after logout works correctly + +### Rate Limiting + +- [ ] 6 rapid failed login attempts → 429 Too Many Requests on 6th + > Note: This test cannot be run from loopback (127.0.0.1) if the rate limiter + > is configured to skip localhost. Run from an external IP in production testing. + +--- + +## Known Test Limitations + +| Limitation | Reason | Workaround | +|------------|--------|------------| +| Rate limit not testable from loopback | Redis rate limiter may skip 127.0.0.1 | Test from external network | +| SSRF blocks may not apply to all DNS names | DNS resolves at request time, not save time | Use IP addresses in SSRF tests | +| PDF generation requires wkhtmltopdf/weasyprint | Must be installed in container | Check container logs if PDF tests fail | +| SAML SSO testing requires an IdP | Real IdP needed for full SSO test | Use mock IdP (e.g. samltest.id) | +| Evidence download requires MinIO | MinIO must be running and bucket configured | Use dev docker-compose | +| Cleanup may fail if tests are in validated state | Validated tests may be immutable | Admin override required for cleanup | + +--- + +## Phase-by-Phase QA Scripts + +In addition to `qa_runner.py`, the `scripts/` directory contains phase-specific +scripts for testing individual feature areas: + +| Script | Phase covered | +|--------|--------------| +| `qa_phases_6_7.py` | Attack paths, knowledge management | +| `qa_phase8.py` | Ownership, risk, analytics | +| `qa_phase9.py` | Alerts, detection lifecycle | +| `qa_phase10.py` | Reports, snapshots, heatmap | +| `qa_phase11.py` | Threat actors, compliance | +| `qa_phase12.py` | SSO, API keys, webhooks | +| `qa_phase13.py` | Operational alerts (Phase 13 gaps) | +| `qa_phase14.py` | Final integration tests | +| `qa_full.py` | Full end-to-end suite | + +Run any of these directly: +```bash +python scripts/qa_phase13.py --base-url http://localhost:8000 +```