Running Scout Scans¶

Scans run Inspect Scout scanners over transcripts from previous evaluations.

Scan Config¶

scanners:
  - package: git+https://github.com/METR/inspect-agents
    name: metr_scanners
    items:
      - name: reward_hacking_scanner
      - name: sandbagging_scanner
      - name: broken_env_scanner

models:
  - package: openai
    name: openai
    items:
      - name: gpt-5

transcripts:
  sources:
    - eval_set_id: inspect-eval-set-abc123
  filter:
    where:
      - eval_status: success
    limit: 10
    shuffle: true

Submitting a Scan¶

hawk scan run config.yaml

Pass secrets the same way as evaluations:

hawk scan run config.yaml --secret MY_API_KEY --secrets-file .env

Transcript Filtering¶

Filter which transcripts to scan:

transcripts:
  filter:
    where:
      - eval_status: success           # Only successful runs
      - score: {gt: 0.5}               # Score above threshold
      - model: {like: "gpt-4%"}        # Pattern matching
      - metadata.agent.name: react     # Nested field access
    limit: 100
    shuffle: true

Available Operators¶

Equality: field: value or field: [val1, val2] (IN list)
Comparison: {gt: 0.5}, {ge: 0.5}, {lt: 1.0}, {le: 1.0}, {between: [0.5, 1.0]}
Pattern matching: {like: "pattern"}, {ilike: "PATTERN"} (case-insensitive)
Logical: {not: condition}, {or: [cond1, cond2]}
Null checks: field: null

Per-scanner filters can override the global filter via scanners[].items[].filter.

Resuming Scans¶

Scan configs are saved to S3. Resume an interrupted scan:

hawk scan resume              # resume the last scan
hawk scan resume SCAN_RUN_ID  # resume a specific scan

Secrets must be re-provided via --secret or --secrets-file when resuming.

Running Scans Locally¶

hawk local scan examples/simple.scan.yaml
hawk local scan config.yaml --secrets-file .env --secret MY_API_KEY