kitos/Aegis
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Page: QA-Testing-Guide
Pages
API-Reference Architecture Authentication-and-Security Campaigns Deployment-Guide Detection-Lifecycle Executive-Dashboard-and-Reports Home Knowledge-Management MITRE-ATT-CK-Coverage Operational-Alerts QA-Testing-Guide Roles-and-Permissions Test-Lifecycle
Clone
2
QA-Testing-Guide
kitos edited this page 2026-05-22 12:33:10 +00:00
Table of Contents

QA Testing Guide

This page documents the automated QA runner and manual testing checklist for Aegis. The QA suite validates access control, test lifecycle, API key scoping, and security controls.

Automated QA Runner

The automated QA runner is located at scripts/qa_runner.py.

What it Tests

The runner executes 77 automated tests organized as follows:

  1. Setup — Creates test users for all 6 roles (admin, red_lead, blue_lead, red_tech, blue_tech, viewer). Creates a test technique.

  2. Authentication tests

    • Login for each role
    • Verify /auth/me returns correct role
    • Verify must_change_password blocks all endpoints
    • Verify logout revokes token (subsequent request returns 401)

  3. Full test lifecycle (happy path)

    • Create test as red_lead → draft
    • Start execution as red_tech → red_executing
    • Verify blue_tech cannot submit-red (403)
    • Submit red as red_tech → blue_evaluating
    • Verify red_tech cannot update blue fields (403)
    • Update blue fields as blue_tech
    • Submit blue as blue_tech → in_review
    • Validate red as red_lead → red approved
    • Validate blue as blue_lead → both approved → validated
    • Verify technique coverage updated

  4. Access control denials (403 checks) For each protected endpoint, verify forbidden roles receive 403:

    • viewer cannot POST /tests
    • viewer cannot POST /tests/{id}/start-execution
    • blue_tech cannot PATCH /tests/{id}/red
    • red_tech cannot PATCH /tests/{id}/blue
    • blue_lead cannot POST /tests/{id}/complete (campaign)
    • red_tech cannot GET /audit-logs
    • non-admin cannot POST /webhooks
    • non-admin cannot DELETE /snapshots/{id}
    • non-admin cannot POST /system/sync-mitre

  5. API Key scope enforcement

    • Create read-scope key as red_lead
    • Verify GET /tests returns 200 with read key
    • Verify POST /tests returns 403 with read key
    • Verify PATCH /tests/{id} returns 403 with read key
    • Create write-scope key
    • Verify POST /tests returns 201 with write key

  6. SSRF webhook protection

    • Attempt to create webhook pointing to 192.168.1.1 → expect 400 (SSRF blocked)
    • Attempt to create webhook pointing to 127.0.0.1 → expect 400
    • Attempt to create webhook pointing to 10.0.0.1 → expect 400
    • Attempt with valid public URL → expect 201

  7. Campaign lifecycle

    • Create campaign as red_lead
    • Add tests to campaign
    • Activate campaign
    • Verify viewer can read campaign progress
    • Verify blue_lead cannot complete campaign (403)
    • Complete campaign as red_lead

  8. Cleanup

    • Delete all created tests
    • Delete all created campaigns
    • Delete all created users
    • Verify cleanup success

Running the QA Suite

# Against local dev
python scripts/qa_runner.py --base-url http://localhost:8000

# Against production (use with caution)
python scripts/qa_runner.py --base-url https://aegis.yourcompany.com

# Verbose mode (shows each HTTP request)
python scripts/qa_runner.py --base-url http://localhost:8000 --verbose

# Run specific phase only
python scripts/qa_runner.py --base-url http://localhost:8000 --phase lifecycle

Exit Codes

Code Meaning
0 All tests passed
1 One or more tests failed
2 Setup failed (cannot run further tests)

Sample Output

Creating Aegis QA Test Run...
[SETUP] Creating test users...
  [OK] Created admin user: qa_admin_20240315
  [OK] Created red_lead user: qa_rl_20240315
  [OK] Created blue_lead user: qa_bl_20240315
  [OK] Created red_tech user: qa_rt_20240315
  [OK] Created blue_tech user: qa_bt_20240315
  [OK] Created viewer user: qa_v_20240315

[PHASE 1] Authentication...
  [OK] Login as admin
  [OK] Login as red_lead
  ...

[PHASE 2] Test Lifecycle...
  [OK] Create test (red_lead)
  [OK] Start execution (red_tech)
  [FAIL] 403 for viewer trying start-execution — expected 403, got 403 ✓
  ...

Results: 77/77 passed

Manual QA Checklist

admin role

  • Login succeeds
  • Change password on first login
  • Can list all users (GET /users)
  • Can create a new user (POST /users)
  • Can delete a user (DELETE /users/{id})
  • Can access audit logs (GET /audit-logs)
  • Can trigger MITRE sync (POST /system/sync-mitre)
  • Can create a webhook (POST /webhooks)
  • Can view scheduler status (GET /system/scheduler-status)
  • Can delete a snapshot (DELETE /snapshots/{id})
  • Can set data classification on a test (PATCH /tests/{id}/classification)
  • Can complete a campaign (POST /campaigns/{id}/complete)
  • Can validate both red and blue sides of a test

red_lead role

  • Login succeeds
  • Can create a test (POST /tests)
  • Can start test execution
  • Can validate red side of test in in_review
  • Can create a campaign and add tests
  • Can complete a campaign
  • Can create a playbook
  • Can create a lesson learned
  • Cannot access audit logs → 403
  • Cannot access webhooks → 403
  • Cannot validate blue side → 403
  • Cannot delete snapshots → 403
  • Can generate a PDF report

blue_lead role

  • Login succeeds
  • Can validate blue side of test in in_review
  • Can create detection assets and link to techniques
  • Can create detection validations
  • Can generate revalidation queue
  • Cannot start test execution → 403
  • Cannot complete a campaign → 403
  • Cannot validate red side → 403
  • Cannot access webhooks → 403
  • Can generate a PDF report

red_tech role

  • Login succeeds
  • Can start test execution (test must be in draft)
  • Can update red fields (PATCH /tests/{id}/red) in red_executing state
  • Can upload red evidence
  • Can submit red (POST /tests/{id}/submit-red)
  • Cannot create a test → 403
  • Cannot update blue fields → 403
  • Cannot validate either side → 403
  • Cannot generate reports → 403
  • Can pause/resume timer

blue_tech role

  • Login succeeds
  • Can update blue fields (PATCH /tests/{id}/blue) in blue_evaluating state
  • Can upload blue evidence (only in blue_evaluating state)
  • Can submit blue (POST /tests/{id}/submit-blue)
  • Can evaluate detection rules (POST /detection-rules/evaluate)
  • Cannot start execution → 403
  • Cannot submit red → 403
  • Cannot validate either side → 403
  • Cannot create tests → 403
  • Can pause/resume timer

viewer role

  • Login succeeds
  • Can read tests, campaigns, techniques, coverage, heatmap, dashboard
  • Can generate PDF/DOCX/HTML reports
  • Cannot create a test → 403
  • Cannot start execution → 403
  • Cannot update any test fields → 403
  • Cannot upload evidence → 403
  • Cannot validate → 403
  • Cannot create campaigns → 403
  • Cannot create playbooks or lessons → 403
  • Cannot pause/resume timer → 403

Security QA Checklist

SSRF Protection

  • POST /webhooks with URL http://192.168.1.1/evil → 400 Bad Request
  • POST /webhooks with URL http://10.0.0.1/evil → 400 Bad Request
  • POST /webhooks with URL http://127.0.0.1/evil → 400 Bad Request
  • POST /webhooks with URL http://169.254.169.254/latest/meta-data → 400
  • POST /webhooks with URL https://hooks.slack.com/services/valid → 201 OK

API Key Scopes

  • Read-scope key: GET requests succeed (200)
  • Read-scope key: POST requests fail (403)
  • Read-scope key: PATCH requests fail (403)
  • Read-scope key: DELETE requests fail (403)
  • Write-scope key: POST requests succeed (201)
  • Admin-scope key: admin-only endpoints succeed

Token Blacklisting

  • Logout request succeeds
  • Using old token after logout returns 401
  • New login after logout works correctly

Rate Limiting

  • 6 rapid failed login attempts → 429 Too Many Requests on 6th

    Note: This test cannot be run from loopback (127.0.0.1) if the rate limiter is configured to skip localhost. Run from an external IP in production testing.

Known Test Limitations

Limitation Reason Workaround
Rate limit not testable from loopback Redis rate limiter may skip 127.0.0.1 Test from external network
SSRF blocks may not apply to all DNS names DNS resolves at request time, not save time Use IP addresses in SSRF tests
PDF generation requires wkhtmltopdf/weasyprint Must be installed in container Check container logs if PDF tests fail
SAML SSO testing requires an IdP Real IdP needed for full SSO test Use mock IdP (e.g. samltest.id)
Evidence download requires MinIO MinIO must be running and bucket configured Use dev docker-compose
Cleanup may fail if tests are in validated state Validated tests may be immutable Admin override required for cleanup

Phase-by-Phase QA Scripts

In addition to qa_runner.py, the scripts/ directory contains phase-specific scripts for testing individual feature areas:

Script Phase covered
qa_phases_6_7.py Attack paths, knowledge management
qa_phase8.py Ownership, risk, analytics
qa_phase9.py Alerts, detection lifecycle
qa_phase10.py Reports, snapshots, heatmap
qa_phase11.py Threat actors, compliance
qa_phase12.py SSO, API keys, webhooks
qa_phase13.py Operational alerts (Phase 13 gaps)
qa_phase14.py Final integration tests
qa_full.py Full end-to-end suite

Run any of these directly:

python scripts/qa_phase13.py --base-url http://localhost:8000