Add wiki page: QA-Testing-Guide

2026-05-22 12:33:10 +00:00
parent a2db142359
commit bf9f1ffd21
1 changed files with 285 additions and 0 deletions
--- a/QA-Testing-Guide.-.md
+++ b/QA-Testing-Guide.-.md
@@ -0,0 +1,285 @@
+# QA Testing Guide
+
+This page documents the automated QA runner and manual testing checklist for Aegis.
+The QA suite validates access control, test lifecycle, API key scoping, and security
+controls.
+
+---
+
+## Automated QA Runner
+
+The automated QA runner is located at `scripts/qa_runner.py`.
+
+### What it Tests
+
+The runner executes **77 automated tests** organized as follows:
+
+1. **Setup** — Creates test users for all 6 roles (admin, red_lead, blue_lead,
+   red_tech, blue_tech, viewer). Creates a test technique.
+
+2. **Authentication tests**
+   - Login for each role
+   - Verify `/auth/me` returns correct role
+   - Verify `must_change_password` blocks all endpoints
+   - Verify logout revokes token (subsequent request returns 401)
+
+3. **Full test lifecycle** (happy path)
+   - Create test as red_lead → draft
+   - Start execution as red_tech → red_executing
+   - Verify blue_tech cannot submit-red (403)
+   - Submit red as red_tech → blue_evaluating
+   - Verify red_tech cannot update blue fields (403)
+   - Update blue fields as blue_tech
+   - Submit blue as blue_tech → in_review
+   - Validate red as red_lead → red approved
+   - Validate blue as blue_lead → both approved → validated
+   - Verify technique coverage updated
+
+4. **Access control denials (403 checks)**
+   For each protected endpoint, verify forbidden roles receive 403:
+   - viewer cannot POST /tests
+   - viewer cannot POST /tests/{id}/start-execution
+   - blue_tech cannot PATCH /tests/{id}/red
+   - red_tech cannot PATCH /tests/{id}/blue
+   - blue_lead cannot POST /tests/{id}/complete (campaign)
+   - red_tech cannot GET /audit-logs
+   - non-admin cannot POST /webhooks
+   - non-admin cannot DELETE /snapshots/{id}
+   - non-admin cannot POST /system/sync-mitre
+
+5. **API Key scope enforcement**
+   - Create read-scope key as red_lead
+   - Verify GET /tests returns 200 with read key
+   - Verify POST /tests returns 403 with read key
+   - Verify PATCH /tests/{id} returns 403 with read key
+   - Create write-scope key
+   - Verify POST /tests returns 201 with write key
+
+6. **SSRF webhook protection**
+   - Attempt to create webhook pointing to 192.168.1.1 → expect 400 (SSRF blocked)
+   - Attempt to create webhook pointing to 127.0.0.1 → expect 400
+   - Attempt to create webhook pointing to 10.0.0.1 → expect 400
+   - Attempt with valid public URL → expect 201
+
+7. **Campaign lifecycle**
+   - Create campaign as red_lead
+   - Add tests to campaign
+   - Activate campaign
+   - Verify viewer can read campaign progress
+   - Verify blue_lead cannot complete campaign (403)
+   - Complete campaign as red_lead
+
+8. **Cleanup**
+   - Delete all created tests
+   - Delete all created campaigns
+   - Delete all created users
+   - Verify cleanup success
+
+### Running the QA Suite
+
+```bash
+# Against local dev
+python scripts/qa_runner.py --base-url http://localhost:8000
+
+# Against production (use with caution)
+python scripts/qa_runner.py --base-url https://aegis.yourcompany.com
+
+# Verbose mode (shows each HTTP request)
+python scripts/qa_runner.py --base-url http://localhost:8000 --verbose
+
+# Run specific phase only
+python scripts/qa_runner.py --base-url http://localhost:8000 --phase lifecycle
+```
+
+### Exit Codes
+
+| Code | Meaning |
+|------|---------|
+| 0 | All tests passed |
+| 1 | One or more tests failed |
+| 2 | Setup failed (cannot run further tests) |
+
+### Sample Output
+
+```
+Creating Aegis QA Test Run...
+[SETUP] Creating test users...
+  [OK] Created admin user: qa_admin_20240315
+  [OK] Created red_lead user: qa_rl_20240315
+  [OK] Created blue_lead user: qa_bl_20240315
+  [OK] Created red_tech user: qa_rt_20240315
+  [OK] Created blue_tech user: qa_bt_20240315
+  [OK] Created viewer user: qa_v_20240315
+
+[PHASE 1] Authentication...
+  [OK] Login as admin
+  [OK] Login as red_lead
+  ...
+
+[PHASE 2] Test Lifecycle...
+  [OK] Create test (red_lead)
+  [OK] Start execution (red_tech)
+  [FAIL] 403 for viewer trying start-execution — expected 403, got 403 ✓
+  ...
+
+Results: 77/77 passed
+```
+
+---
+
+## Manual QA Checklist
+
+### admin role
+
+- [ ] Login succeeds
+- [ ] Change password on first login
+- [ ] Can list all users (`GET /users`)
+- [ ] Can create a new user (POST /users)
+- [ ] Can delete a user (DELETE /users/{id})
+- [ ] Can access audit logs (`GET /audit-logs`)
+- [ ] Can trigger MITRE sync (POST /system/sync-mitre)
+- [ ] Can create a webhook (POST /webhooks)
+- [ ] Can view scheduler status (`GET /system/scheduler-status`)
+- [ ] Can delete a snapshot (DELETE /snapshots/{id})
+- [ ] Can set data classification on a test (PATCH /tests/{id}/classification)
+- [ ] Can complete a campaign (POST /campaigns/{id}/complete)
+- [ ] Can validate both red and blue sides of a test
+
+### red_lead role
+
+- [ ] Login succeeds
+- [ ] Can create a test (POST /tests)
+- [ ] Can start test execution
+- [ ] Can validate red side of test in `in_review`
+- [ ] Can create a campaign and add tests
+- [ ] Can complete a campaign
+- [ ] Can create a playbook
+- [ ] Can create a lesson learned
+- [ ] Cannot access audit logs → 403
+- [ ] Cannot access webhooks → 403
+- [ ] Cannot validate blue side → 403
+- [ ] Cannot delete snapshots → 403
+- [ ] Can generate a PDF report
+
+### blue_lead role
+
+- [ ] Login succeeds
+- [ ] Can validate blue side of test in `in_review`
+- [ ] Can create detection assets and link to techniques
+- [ ] Can create detection validations
+- [ ] Can generate revalidation queue
+- [ ] Cannot start test execution → 403
+- [ ] Cannot complete a campaign → 403
+- [ ] Cannot validate red side → 403
+- [ ] Cannot access webhooks → 403
+- [ ] Can generate a PDF report
+
+### red_tech role
+
+- [ ] Login succeeds
+- [ ] Can start test execution (test must be in draft)
+- [ ] Can update red fields (PATCH /tests/{id}/red) in red_executing state
+- [ ] Can upload red evidence
+- [ ] Can submit red (POST /tests/{id}/submit-red)
+- [ ] Cannot create a test → 403
+- [ ] Cannot update blue fields → 403
+- [ ] Cannot validate either side → 403
+- [ ] Cannot generate reports → 403
+- [ ] Can pause/resume timer
+
+### blue_tech role
+
+- [ ] Login succeeds
+- [ ] Can update blue fields (PATCH /tests/{id}/blue) in blue_evaluating state
+- [ ] Can upload blue evidence (only in blue_evaluating state)
+- [ ] Can submit blue (POST /tests/{id}/submit-blue)
+- [ ] Can evaluate detection rules (POST /detection-rules/evaluate)
+- [ ] Cannot start execution → 403
+- [ ] Cannot submit red → 403
+- [ ] Cannot validate either side → 403
+- [ ] Cannot create tests → 403
+- [ ] Can pause/resume timer
+
+### viewer role
+
+- [ ] Login succeeds
+- [ ] Can read tests, campaigns, techniques, coverage, heatmap, dashboard
+- [ ] Can generate PDF/DOCX/HTML reports
+- [ ] Cannot create a test → 403
+- [ ] Cannot start execution → 403
+- [ ] Cannot update any test fields → 403
+- [ ] Cannot upload evidence → 403
+- [ ] Cannot validate → 403
+- [ ] Cannot create campaigns → 403
+- [ ] Cannot create playbooks or lessons → 403
+- [ ] Cannot pause/resume timer → 403
+
+---
+
+## Security QA Checklist
+
+### SSRF Protection
+
+- [ ] POST /webhooks with URL `http://192.168.1.1/evil` → 400 Bad Request
+- [ ] POST /webhooks with URL `http://10.0.0.1/evil` → 400 Bad Request
+- [ ] POST /webhooks with URL `http://127.0.0.1/evil` → 400 Bad Request
+- [ ] POST /webhooks with URL `http://169.254.169.254/latest/meta-data` → 400
+- [ ] POST /webhooks with URL `https://hooks.slack.com/services/valid` → 201 OK
+
+### API Key Scopes
+
+- [ ] Read-scope key: GET requests succeed (200)
+- [ ] Read-scope key: POST requests fail (403)
+- [ ] Read-scope key: PATCH requests fail (403)
+- [ ] Read-scope key: DELETE requests fail (403)
+- [ ] Write-scope key: POST requests succeed (201)
+- [ ] Admin-scope key: admin-only endpoints succeed
+
+### Token Blacklisting
+
+- [ ] Logout request succeeds
+- [ ] Using old token after logout returns 401
+- [ ] New login after logout works correctly
+
+### Rate Limiting
+
+- [ ] 6 rapid failed login attempts → 429 Too Many Requests on 6th
+  > Note: This test cannot be run from loopback (127.0.0.1) if the rate limiter
+  > is configured to skip localhost. Run from an external IP in production testing.
+
+---
+
+## Known Test Limitations
+
+| Limitation | Reason | Workaround |
+|------------|--------|------------|
+| Rate limit not testable from loopback | Redis rate limiter may skip 127.0.0.1 | Test from external network |
+| SSRF blocks may not apply to all DNS names | DNS resolves at request time, not save time | Use IP addresses in SSRF tests |
+| PDF generation requires wkhtmltopdf/weasyprint | Must be installed in container | Check container logs if PDF tests fail |
+| SAML SSO testing requires an IdP | Real IdP needed for full SSO test | Use mock IdP (e.g. samltest.id) |
+| Evidence download requires MinIO | MinIO must be running and bucket configured | Use dev docker-compose |
+| Cleanup may fail if tests are in validated state | Validated tests may be immutable | Admin override required for cleanup |
+
+---
+
+## Phase-by-Phase QA Scripts
+
+In addition to `qa_runner.py`, the `scripts/` directory contains phase-specific
+scripts for testing individual feature areas:
+
+| Script | Phase covered |
+|--------|--------------|
+| `qa_phases_6_7.py` | Attack paths, knowledge management |
+| `qa_phase8.py` | Ownership, risk, analytics |
+| `qa_phase9.py` | Alerts, detection lifecycle |
+| `qa_phase10.py` | Reports, snapshots, heatmap |
+| `qa_phase11.py` | Threat actors, compliance |
+| `qa_phase12.py` | SSO, API keys, webhooks |
+| `qa_phase13.py` | Operational alerts (Phase 13 gaps) |
+| `qa_phase14.py` | Final integration tests |
+| `qa_full.py` | Full end-to-end suite |
+
+Run any of these directly:
+```bash
+python scripts/qa_phase13.py --base-url http://localhost:8000
+```