Research Reports

Analyzing Shannon — AI Pentester by Keygraph

12 min read

Analyzing Shannon — AI Pentester by Keygraph

Source: KeygraphHQ/shannon @ d26f3b6

Overview

Shannon is an autonomous AI penetration testing agent for web applications and APIs. It combines white-box source-code analysis with live exploitation, executing real proof-of-concept attacks and reporting only vulnerabilities it can demonstrably prove. Built by Keygraph, it’s the open-source engine (AGPL-3.0) behind the commercial Keygraph platform.

The project is organized as a pnpm monorepo with two packages: a CLI (@keygraph/shannon, npm-published) containing only Docker orchestration logic, and a worker (private) containing the Temporal workflow engine, pi-based agent execution, and all business logic. The CLI bundles its compose infrastructure and dispatches ephemeral Docker workers, each running a full pentest lifecycle orchestrated via Temporal.

Key Findings

How It Works

A user runs npx @keygraph/shannon start -u <url> -r <repo> (or ./shannon start in local mode). The CLI spins up Temporal Server via Docker Compose, then spawns one ephemeral docker run --rm worker container per scan. The worker registers a Temporal workflow that drives 12 agents across 5 phases, each agent running as a pi coding-agent session with Playwright for browser automation.

The pipeline has no inter-agent synchronization barriers — each vuln→exploit pipeline starts as soon as its vuln agent finishes, not after all vulns complete. Results are git-checkpointed per agent, enabling interrupt-resume.

Architecture

┌──────────────────────────────────────────────────────────┐
│                    CLI (@keygraph/shannon)                 │
│  ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌─────────────┐  │
│  │ start   │ │ stop     │ │ logs     │ │ workspaces  │  │
│  │ (spawns │ │ (compose │ │ (tail    │ │ (list local │  │
│  │  worker)│ │  down)   │ │  volume) │ │  workspaces)│  │
│  └─────────┘ └──────────┘ └──────────┘ └─────────────┘  │
│  ┌──────────────────────────────────────────────────┐    │
│  │ mode.ts: auto-detect local vs npx (SHANNON_LOCAL)│    │
│  │ docker.ts: compose lifecycle + docker run        │    │
│  │ env.ts: .env / TOML → provider-scoped env flags  │    │
│  └──────────────────────────────────────────────────┘    │
└────────────────────────┬─────────────────────────────────┘
                         │ docker run --rm (ephemeral)
┌────────────────────────▼─────────────────────────────────┐
│               Worker Container (Chainguard Wolfi)          │
│                                                           │
│  ┌──────────────────────────────────────────────────┐    │
│  │         Temporal Workflow (pentestPipeline)        │    │
│  │                                                    │    │
│  │  preflight → auth-validation → init deliverables   │    │
│  │              ↓                                     │    │
│  │  Phase 1: pre-recon (sequential)                   │    │
│  │  Phase 2: recon (sequential)                       │    │
│  │  Phase 3-4: 5× vuln→exploit pipelines (parallel)   │    │
│  │    ┌──────────┐  ┌──────────┐  ┌──────┐  ┌──────┐  │    │
│  │    │injection │  │  xss     │  │ auth │  │ssrf  │  │    │
│  │    │ vuln→exp │  │ vuln→exp │  │ ...  │  │authz │  │    │
│  │    └──────────┘  └──────────┘  └──────┘  └──────┘  │    │
│  │  Phase 5: report (sequential)                       │    │
│  │              ↓                                     │    │
│  │        Markdown report + SARIF                     │    │
│  └──────────────────────────────────────────────────┘    │
│                                                           │
│  ┌──────────────────────────────────────────────────┐    │
│  │        Agent Execution (pi coding-agent)           │    │
│  │  ┌──────────┐  ┌──────────┐  ┌────────────┐    │    │
│  │  │ pi loop  │  │Playwright│  │ git        │    │    │
│  │  │ (LLM→    │  │ (browser │  │ checkpoint │    │    │
│  │  │  tool→..)│  │  agents) │  │ + rollback │    │    │
│  │  └──────────┘  └──────────┘  └────────────┘    │    │
│  └──────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────┘

The Spine

The pipeline has five phases with clear dependencies:

  1. Preflight — quick sanity checks (config validity, target reachability, Docker readiness). Not an agent phase; just validation.

  2. Pre-reconnaissance — the first agent reads the target repo at {{REPO_PATH}}, identifies frameworks, entry points, data flows, and likely attack surfaces.

  3. Reconnaissance — the second agent correlates the pre-recon output with live application behavior using Playwright browser automation, producing a structured attack surface map.

  4. Vulnerability Analysis + Exploitation — five pipelined pairs run concurrently: injection, xss, auth, ssrf, authz. Each vuln agent writes findings to typed JSON exploitation queues (injection_exploitation_queue.json, etc.). A queue check determines whether the paired exploit agent should run. If a vuln agent finds nothing exploitable, the exploit agent is skipped.

  5. Reporting — the report agent reads all per-class deliverables, concatenates them, and runs through a final LLM pass that adds an executive summary and cleans up formatting. Model metadata is injected into the final document.

Concurrency Model

The 5 vuln→exploit pipelines run with a concurrency limit of 5 (all at once), implemented via a thunk-aware runWithConcurrencyLimit that uses Promise.race on an in-flight set. Results are processed in completion order, not input order — the aggregatePipelineResults function keys on the vulnType field in each result, not array position.

Pipeline failure semantics

If all 5 pipelines fail, the workflow hard-fails with PipelineFailedError and no report is produced. If only some fail, the status becomes 'partial' and a report is produced with the failed classes listed as “analysis did not complete.” A cancelled pipeline is caught and rethrown to produce a clean cancelled state with spend metrics preserved.

Agent Execution Lifecycle

Each agent follows a strict lifecycle in AgentExecutionService.execute():

  1. Load config (pre-parsed, raw YAML, or file path, cascading)
  2. Load prompt template with template variables ({{WEB_URL}}, {{REPO_PATH}}, {{LOGIN_INSTRUCTIONS}}, etc.)
  3. Create a git checkpoint
  4. Start audit logging
  5. Run the pi coding-agent session with a submit tool (vuln agents: capture findings to queue JSON)
  6. On failure: rollback git workspace, record audit
  7. Write structured output (vuln agents: serialized queue JSON)
  8. Validate output
  9. Write deliverable (renderer hook — e.g., markdown report)
  10. Commit on success
  11. Record metrics (duration, tokens, cost, model)

Steps 7-11 are atomic under a git repo lock so concurrent sibling agents don’t race on the deliverable directory.

Key Patterns

The CLI/Worker Separation

The CLI package is deliberately thin — it publishes the npm package @keygraph/shannon, bundles with tsdown to single-file ESM, and contains zero business logic. Everything in it is Docker orchestration: compose lifecycle (pull vs build), ephemeral worker spawning, state directory management, env var collection, TOML config writing, and terminal UI. The worker container holds the full Temporal stack, pi runtime, Playwright browser, and all prompts.

This is a strong architectural boundary: the CLI can be replaced or rewritten entirely without touching the pentesting logic, and vice versa.

Git-Based Checkpointing

Each agent creates a git checkpoint in a deliverables git repo before execution begins. On success, the agent commits its output. On failure, the workspace is rolled back. This enables:

  • Resumable scans: if the worker container dies (OOM, timeout, network), a new worker can restore the git state and pick up where the last committed agent left off
  • Audit trail: every deliverable change is a git commit with agent name and attempt number
  • No database needed for state — just git + a small session.json with metadata

The resume logic in workflows.ts loads a resume state, cross-checks which agents have completed deliverables, restores the git checkpoint hash, and short-circuits if all expected agents are already done.

Temporal as Orchestration Backbone

Temporal provides:

  • Durable execution: workflow state survives worker crashes
  • Queryable progress: the UI at localhost:8233 exposes getProgress, showing current phase, completed agents, elapsed time
  • Configurable retry: production retry starts at 5 minutes with 2x backoff up to 30 minutes, 50 max attempts — designed to ride through rate-limit windows
  • Cancellation handling: cancellation propagates through the pipeline, shields the finalization I/O from the cancellation signal, and produces a structured cancelled state with real metrics
  • Non-retryable errors: AuthenticationError, ConfigurationError, InvalidTargetError, AuthLoginFailedError, PermanentError are never retried

The workflow has a pipelineTestingMode switch that shortens all timeouts and retries for fast iteration during development.

The pi Harness

Shannon uses pi, Mario Zechner’s TypeScript coding-agent toolkit, as its agent execution engine. Each agent runs as a full pi session with:

  • Built-in tools (read, bash, edit, write, grep, find, ls)
  • Custom submit tool for vuln agents (captures structured exploitation queue)
  • Playwright CLI skill for browser automation agents
  • Task delegation tool for code-heavy agents (recon, pre-recon)
  • Bounded bash timeouts via a timeout extension
  • Permission system (optional, from pi’s agent_dir)
  • Provider-level retry (pi handles API-level retries; Temporal handles agent-level retries — the system keeps only one retry layer at each boundary)

Dual retry layers, one policy each

A notable design choice: agent-level retry is disabled in pi’s PI_RETRY_SETTINGS so Temporal owns restart decisions. Provider-level retry (API calls) remains on in pi. This avoids competing retry policies and ensures Temporal has the full picture for backoff.

Non-Obvious Details

5 Playwright Sessions, Dedicated to Specific Agents

PLAYWRIGHT_SESSION_MAPPING assigns each prompt template to one of 5 browser sessions (agent1–agent5). The mapping ensures no two agents running concurrently share a browser context — critical because parallel exploit agents would otherwise interfere with each other’s cookies, localStorage, and navigation state. The pre-recon agent uses agent1, recon uses agent2, and the 5 vuln→exploit pairs each get a dedicated session that they reuse across their two phases (vuln then exploit).

Exploitation Queue as a Typed JSON Contract

Vuln agents don’t return free-form text to their exploit counterparts. Instead, they populate typed JSON queue files via a custom submit tool that captures structured findings. The exploit agent then reads its queue and systematically attempts to weaponize each entry. This is a design-level decision to make the vuln→exploit handshake deterministic and machine-readable — the queue schema lives in queue-schemas.ts and both the vuln agent’s submit tool and the exploitation checker (checkExploitationQueue) reference it.

/etc/hosts Forwarding

The CLI’s docker.ts reads the host’s /etc/hosts and emits one --add-host flag per valid user-added entry into the ephemeral worker container. Loopback IPs are rewritten to host-gateway; IPv6 addresses are bracketed. This lets the worker resolve custom hostnames present on the host, which is essential for testing apps behind internal DNS or corporate proxies. Disabled per-scan via SHANNON_FORWARD_HOSTS=false.

Dual Deployment Mode

The project supports two fully parallel deployment paths:

Aspectnpx modeLocal mode
Installnpx @keygraph/shannongit clone && ./shannon build
Worker imageDocker Hub pullLocal build
State dir~/.shannon/./workspaces/
Config~/.shannon/config.toml (via setup TUI).env
PromptsBundled in Docker imageLive-editable from apps/worker/prompts/

The mode is auto-detected in mode.ts — if SHANNON_LOCAL=1 is set (by the ./shannon entrypoint), it’s local; otherwise npx. This means the same binary running from a clone behaves differently than from npm, which could surprise users who install via npm in the same directory as a clone.

Prompt-templating System

Prompts live as .txt files under apps/worker/prompts/ with an @include(shared/...) directive system for shared fragments. Template variables use {{MUSTACHE}} syntax. A dedicated PromptManagerService resolves includes and interpolates variables. The shared directory contains fragments like _target.txt, _shared-session.txt, _rules-of-engagement.txt, _code-path-rules.txt, and _exploit-scope.txt — these are reused across multiple agent types.

Pipeline Testing Mode

The --pipeline-testing flag configures: minimal prompts (shortened templates), 10-second retry intervals (max 30s, 5 attempts), 30-minute start-to-close timeouts, and minimal prompt content. This is used for development iteration where full 1-hour waits are prohibitive.

Assessment

Strengths

  • Proof-by-exploitation is the right model. Shannon doesn’t flag CVEs — it runs actual exploit payloads and reports only what works. This eliminates the false-positive problem that plagues static analyzers.
  • Git-based checkpointing for resume is elegant. No database dependency, inspectable by anyone with git log, and the rollback-on-failure pattern prevents partial state from leaking between agent runs.
  • The CLI/Worker separation is a clean boundary. The npm-published CLI is tiny and dependency-free of the LLM stack. Users get a fast install with no heavy SDK downloads.
  • Temporal is a good fit. The pentest pipeline is a textbook Temporal workflow: long-running, failure-prone, with state you want to query mid-flight and resume after crashes.
  • Parallel vuln→exploit pairs without a sync barrier. Each pipeline starts as soon as its vuln agent finishes — this is faster than a naive barrier and simpler than a DAG scheduler.
  • 5 isolated Playwright sessions. Prevents cross-agent contamination during parallel exploitation — a detail that would only become obvious after debugging a race condition at 3am.

Concerns

  • Prompt files are .txt, not .md. The project uses .txt extensions for prompts (e.g., exploit-auth.txt). These are plain text files loaded by the prompt manager, not markdown. This works but means no syntax highlighting or Obsidian rendering if you’re in the vault.
  • The npx/local dual-mode auto-detection could surprise. The mode detection relies on SHANNON_LOCAL=1 being set by the ./shannon entrypoint. If a user runs node apps/cli/dist/index.mjs from a clone manually, they get npx mode with wrong paths. The entrypoint catches this correctly via its shebang script, but the bifurcation adds a subtle footgun.
  • Local mode has no setup wizard. In local mode, users must hand-write .env files. The setup TUI (npx @keygraph/shannon setup) only works in npx mode. First-time local users need to know the provider env var names by reading docs.
  • Temporal dev server has no persistence. The compose file uses temporal server start-dev with --db-filename /home/temporal/temporal.db (SQLite), which is fine for local use but has no replication or backup. Long-running or production-facing deployments would need a real Temporal cluster.
  • No external code contributions accepted. At the time of this analysis, the README states Keygraph is not accepting external code contributions. This means no community bug fixes or improvements — all development is internal.

Operational Decision Rules

Use Shannon when: you have a web application or API with source code access, need to run proof-based pentests on demand (per-build, per-PR), and can dedicate ~1–1.5 hours plus LLM API costs. The white-box approach is strongest against injectable classes (SQLi, XSS, SSRF, broken auth/authz).

Don’t use Shannon when: you need black-box testing without source code, require passive scanning only (no state mutation), are testing adversarial/untrusted codebases (prompt injection risk), or want broader coverage including dependency scanning, secrets detection, or IaC misconfigurations — those are addressed by the commercial Keygraph platform.

  • analyzing-pentestagent — Python-based pentesting agent with multi-agent crew mode and MCP integration
  • analyzing-openclaw-pi — Another project built on pi, showing the framework’s flexibility beyond coding agents
  • analyzing-nuclei — Template-based vulnerability scanner, different approach but same problem space
  • analyzing-zaproxy — Traditional open-source pentest proxy, predecessor in automation

Graph View