Research Reports

Analyzing SpiderFoot

8 min read

Analyzing SpiderFoot

Source: smicallef/spiderfoot @ 0f815a2

How It Works

SpiderFoot is an event-driven OSINT automation engine. You give it a target — an IP address, domain, email, username, phone number, Bitcoin address, or human name — and it fans out across 200+ data source modules, each querying external APIs, web services, or local datasets. Results from one module become inputs to others, creating a cascading graph of discovered intelligence. A scan against a single domain can cascade through DNS lookups, WHOIS queries, certificate transparency logs, breach database checks, web scraping, and social media lookups, with each discovery triggering further modules.

The system operates on a publish-subscribe model. Each module declares which event types it consumes (e.g., IP_ADDRESS, DOMAIN_NAME) and which it produces (e.g., BLACKLISTED_IPADDR, MALICIOUS_IPADDR). A central dispatcher routes events from producing modules to all consuming modules through per-module queues. This creates an implicit data-flow graph determined entirely by module declarations — no hard-coded wiring exists between modules.

After all modules finish, SpiderFoot runs a post-scan correlation engine using YAML-defined rules (e.g., “critical vulnerability found on host,” “email appears in multiple breaches”). Correlations query the SQLite result store and generate higher-level findings. The entire pipeline — scan execution and correlation — runs in a separate process, with the web UI or CLI polling SQLite for status updates.

Architecture

sf.py                         # CLI entry point, flag parsing, web server boot
 │
 ├── sfwebui.py               # CherryPy web UI (Mako templates, JSON API)
 ├── sfcli.py                 # Interactive CLI (talks to web API)
 ├── sfscan.py                # SpiderFootScanner: scan lifecycle orchestration
 │
 ├── sflib.py                 # SpiderFoot: shared utility class (fetchUrl, caching, etc.)
 │
 └── spiderfoot/              # Core library package
      ├── plugin.py           # SpiderFootPlugin base class
      ├── event.py            # SpiderFootEvent (typed data + provenance chain)
      ├── target.py           # SpiderFootTarget (target + aliases)
      ├── db.py               # SpiderFootDb (SQLite schema, CRUD, WAL mode)
      ├── helpers.py          # SpiderFootHelpers (module loading, graph export, etc.)
      ├── correlation.py      # SpiderFootCorrelator (YAML rule engine)
      ├── threadpool.py       # SpiderFootThreadPool (reusable worker pool)
      └── logger.py           # Multiprocessing-aware logging setup

modules/                      # 230+ OSINT modules (sfp_*.py)
correlations/                 # ~40 YAML correlation rules

Storage layer: SQLite with WAL journaling. All scan results, event types, scan configs, and correlation results live in a single spiderfoot.db file. The SpiderFootDb class wraps all access with a threading.RLock for thread safety. A tbl_scan_results table stores every event with its type, confidence/visibility/risk scores, source event hash, and module of origin — forming a queryable provenance graph.

Module catalog: Modules are Python files in modules/ that export a class inheriting SpiderFootPlugin. Each module declares a meta dict (name, summary, flags, use cases, categories), an opts dict (configuration including API keys), watchedEvents() (what it consumes), and producedEvents() (what it outputs). The loader (SpiderFootHelpers.loadModulesAsDict) imports them at startup and extracts this metadata into a registry.

The Spine

1. Startup — sf.py parses CLI args, initializes logging (multiprocessing Queue-based), loads all modules from modules/, loads correlation rules from correlations/, initializes SQLite, and branches:

  • -l ip:port → start CherryPy web server (start_web_server)
  • -s target → start CLI scan (start_scan)

2. Scan launch — Both paths create a multiprocessing.Process targeting startSpiderFootScanner, passing the target, module list, config, and logging queue. The scan runs in an isolated process.

3. SpiderFootScanner.init — Inside the child process:

  • Creates a SpiderFootTarget from the raw target string and detected type
  • Sets up SOCKS proxy, DNS resolver, user-agent, TLD list
  • Instantiates each module, calls setup(), assigns target/dbh/scanId/queues
  • Sorts modules by _priority (lower runs first)
  • Creates a shared SpiderFootThreadPool for concurrent module execution

4. Event dispatch — The scanner creates a ROOT event and a target-typed event (e.g., INTERNET_NAME), then pushes them onto a shared eventQueue. The waitForThreads() loop pulls events from this queue and distributes them to each module’s incomingEventQueue if the module watches that event type. Each module runs its own threadWorker that pulls from its incoming queue and calls handleEvent().

# Core dispatch loop (simplified)
while True:
    sfEvent = self.eventQueue.get_nowait()
    for mod in self.__moduleInstances.values():
        if sfEvent.eventType in mod.watchedEvents():
            mod.incomingEventQueue.put(deepcopy(sfEvent))

5. Event propagation — When a module produces a new event (e.g., sfp_abuseipdb finds a blacklisted IP), it calls self.notifyListeners(sfEvent) which puts the event back onto the shared eventQueue, closing the loop. The notifyListeners method includes cycle detection: it walks the source event chain and suppresses re-notifications of duplicate data to prevent infinite loops.

6. Completion — waitForThreads detects completion when all module queues are empty and no threads are running. It does three final passes (sending FINISHED to each module) before setting status to FINISHED and running correlations.

Key Patterns

Plugin contract. Every module follows the same lifecycle: setup() → watchedEvents() / producedEvents() declarations → handleEvent(sfEvent) called per event → finish() at scan end. Storage modules (sfp__stor_*) are treated specially: they execute synchronously (not via the thread pool) and have _priority = 0 to run first.

Event provenance chain. Each SpiderFootEvent carries a sourceEvent reference forming a linked list back to the ROOT. This chain is used for cycle detection, result display (source-to-child relationships), and correlation. Event hashes (SHA256 of a composite ID) provide identity without storing the full chain.

Target aliasing. The SpiderFootTarget class maintains a list of aliases (e.g., if you scan an IP, the reverse DNS hostname becomes an alias). Modules use target.matches(value) to check if a discovered piece of data relates to the original target, with logic for parent/child domain relationships and subnet membership.

Configuration layering. Global options (user-agent, proxy, DNS server, thread count) are merged with per-module options (API keys, module-specific thresholds). The DB stores serialized configs per scan, enabling scan re-runs with the original settings.

Shared thread pool. Modules don’t manage their own concurrency. Each module submits work to a shared SpiderFootThreadPool, which uses per-module task naming to enforce maxThreads limits per module while sharing a fixed worker pool globally.

Non-Obvious Details

Double-ended SQLite handles

The SpiderFootPlugin.threadWorker creates a new SpiderFootDb instance per thread because SQLite connections cannot be shared across threads. This means each module thread opens its own DB connection — with 200+ modules, that’s hundreds of concurrent SQLite writers. WAL mode mitigates this, but it’s a scalability ceiling.

Cycle detection via source event walking. notifyListeners walks the sourceEvent chain looking for ancestors with matching event type and data. If found, it sets storeOnly = True, which means the event gets stored but not propagated to non-storage modules. This prevents the most common infinite loop pattern (module A finds X, module B processes X and finds X again from a different source).

The __outputFilter__ gate. Each module has an optional output filter set via strict mode (-x flag). This is checked in notifyListeners — if the event type isn’t in the filter (plus ROOT and the target type), it’s silently dropped. This is how SpiderFoot limits module cascading when users only want specific output types.

Correlation rules are YAML mini-DSLs. Rules like vulnerability_critical.yaml specify collection criteria (method: exact, field: type, value: VULNERABILITY_CVE_CRITICAL), aggregation fields, and analysis methods (thresholds, uniqueness checks, percentage-based detection). The correlator runs post-scan and produces findings with risk ratings, linked to the underlying events.

CLI talks to the web server. sfcli.py is not a direct scan launcher — it’s an HTTP client that connects to a running SpiderFoot web instance. This is a deliberate separation: the CLI is a remote control, not an alternative execution path. Only sf.py -s runs scans directly without the web server.

sflib.py is the God class

At 1,664 lines, SpiderFoot in sflib.py is a utility monolith containing fetchUrl, caching, DNS resolution, configuration serialization, module dependency resolution (modulesProducing, modulesConsuming, eventsToModules), and more. It’s the Swiss Army knife passed to every module as self.sf.

Assessment

Strengths:

  • The event-driven plugin architecture is clean and extensible. Adding a new OSINT source means writing one Python file with a well-defined contract. The 230+ modules prove this scales.
  • Cycle detection in notifyListeners is a practical solution to the graph traversal problem inherent in cascading OSINT lookups.
  • Target aliasing and the matches() method with domain/subnet awareness is a thoughtful design for OSINT scope management.

Concerns:

  • SQLite at the center. Every event, status check, and log entry hits the same database. The RLock serialization and per-thread connection pattern work for single scans but will bottleneck under concurrent multi-scan workloads.
  • The SpiderFoot utility class in sflib.py (1,664 lines) and SpiderFootDb (1,802 lines) are large, tightly coupled classes. Module authors depend on self.sf.fetchUrl() and self.sf.cacheGet() — the boundary between framework and module is porous.
  • No type-safe event contract. Event types are free-form strings matched at runtime. A typo in watchedEvents() or producedEvents() silently breaks the data flow.
  • Global mutable state (sfConfig['__modules__']) is shared across the process and mutated during scan setup. Combined with multiprocessing, this works because fork copies-on-write, but it’s fragile.

Recommendations:

  • Extract fetchUrl, caching, and config management from sflib.py into focused modules to reduce the surface area passed to each plugin.
  • Consider an event type registry (enum or constants module) to catch typos at load time rather than runtime.
  • For concurrent scan support, evaluate moving to a proper message queue (Redis, ZeroMQ) instead of in-process queue.Queue + SQLite polling.
  • analyzing-nuclei — template-driven vulnerability scanner with similar plugin/extensibility patterns
  • analyzing-bloodhound — attack path mapping tool; complementary post-exploitation recon
  • analyzing-misp — threat intelligence platform; SpiderFoot feeds into MISP via its modules
  • analyzing-opencti — open CTI platform; another event-driven intelligence aggregation system

Graph View

Backlinks