Research Reports

Analyzing Maigret

8 min read

Overview

Maigret (v0.6.0) is an open-source OSINT username enumeration tool that collects dossiers on individuals by checking for account existence across thousands of websites. Named after the fictional detective, it requires no API keys and operates purely through HTTP requests. The tool scans a curated database of 3,154 sites (as of the bundled data.json), extracts biographical and identity data from discovered profiles, performs recursive searches on newly discovered identifiers, and generates reports in nine output formats. It is used professionally by platforms like Social Links, Crimewall, and UserSearch.ai.

The codebase lives in approximately 6,400 lines of Python across 18 source modules under maigret/, with a 36,226-line JSON site database (maigret/resources/data.json), and roughly 3,800 lines of test code across 17 test modules. The project requires Python 3.10+ and is built on asyncio/aiohttp for high-concurrency HTTP checking.

Key Findings

Architecture and Async Design

The core execution flow follows a layered async pipeline:

  1. CLI entry point (maigret/maigret.py, run() → main()): An asyncio.run() wrapper parses arguments via setup_arguments_parser(), loads settings, resolves the site database path, and orchestrates the search loop. The main loop at line 796 processes usernames one at a time, delegating each to the maigret() async function.

  2. Core search function (maigret/checking.py, line 1024, async def maigret()): This ~180-line function sets up protocol-specific checkers (clearweb via SimpleAiohttpChecker, Tor via ProxiedAiohttpChecker, I2P, and DNS via AiodnsDomainResolver), then feeds tasks to the AsyncioQueueGeneratorExecutor.

  3. Executor framework (maigret/executors.py): The active executor is AsyncioQueueGeneratorExecutor (line 184), which runs an async worker pool with configurable concurrency (default 100 parallel connections). Workers pull (function, args, kwargs) tuples from an asyncio.Queue, execute them with timeout guards, and yield results through a second queue. Several deprecated executor classes remain in the file (AsyncioSimpleExecutor, AsyncioProgressbarExecutor, etc.) with deprecation comments.

  4. Per-site checking (maigret/checking.py, check_site_for_username() at line 945): Each site check selects an appropriate HTTP checker, sends the request, processes the response through process_site_result(), which applies the site’s detection logic (check_type: message, status_code, or response_url), and extracts identity data if parsing is enabled.

The checker hierarchy is notable:

CheckerFile:LinePurpose
SimpleAiohttpCheckerchecking.py:103Default aiohttp-based HTTP client
ProxiedAiohttpCheckerchecking.py:224Proxy-routed requests (Tor, I2P)
AiodnsDomainResolverchecking.py:231DNS A-record checks for domain enumeration
CurlCffiCheckerchecking.py:270TLS fingerprint impersonation via curl_cffi
CloudflareWebgateCheckerchecking.py:338FlareSolverr/CloudflareBypassForScraping integration
CheckerMockchecking.py:553No-op checker for disabled protocols

Checker selection in make_site_result() (line 774) follows a precedence: Cloudflare webgate if protection triggers match, then curl_cffi for TLS fingerprint sites, falling back to protocol-specific aiohttp checkers.

Site Database Structure

The site database (maigret/resources/data.json, ~1.2 MB) is the heart of the tool. It is a JSON object with three top-level keys:

  • sites: Object keyed by site name (e.g., “Facebook”, “YouTube”), each containing detection configuration
  • engines: Reusable detection templates shared across sites with similar platforms
  • tags: Array of category strings used for filtering (e.g., “social”, “photo”, “dating”, country codes)

Each site entry defines:

{
  "url": "https://example.com/{username}",
  "urlMain": "https://example.com/",
  "checkType": "message",
  "presenseStrs": ["first_name"],
  "absenceStrs": ["Page not found"],
  "alexaRank": 2,
  "regexCheck": "^[a-zA-Z0-9_\\.]{3,49}$",
  "usernameClaimed": "zuck",
  "usernameUnclaimed": "noonewouldeverusethis7",
  "tags": ["social"],
  "headers": {"User-Agent": "..."},
  "errors": {"Login text": "Login required"},
  "activation": {"method": "twitter", "marks": [...], "url": "..."},
  "protection": ["tls_fingerprint"]
}

The MaigretSite class (maigret/sites.py:23) loads this data, with MaigretEngine objects for shared engine templates. Sites are ranked by alexaRank and filtered through MaigretDatabase.ranked_sites_dict() (line 340), which supports tag filtering, exclusion lists, disabled-site handling, and a mirror-append logic that includes alternative viewers for high-ranking disabled sites.

The database auto-updates from GitHub every 24 hours via maigret/db_updater.py (342 lines), which checks a lightweight db_meta.json for version compatibility, SHA-256 integrity verification, and atomic file replacement. The bundled fallback is at maigret/resources/data.json.

Detection Logic and Check Types

Username presence is determined through three checkType strategies in process_site_result() (line 605):

  • message (default): Searches HTML for presenseStrs and absenceStrs substrings. If absence strings are not found and presence strings are found (or no presence strings are defined), the account is CLAIMED.
  • status_code: HTTP 2xx → CLAIMED, otherwise AVAILABLE.
  • response_url: Disallows redirects; 2xx status with presence detection → CLAIMED.

Error detection (maigret/errors.py) maintains a dictionary of known error page signatures (Cloudflare challenges, DDoS-Guard, censorship blocks) and groups error statistics with user-facing remediation suggestions.

When parsing is enabled (default), Maigret uses socid-extractor (version >=0.0.27) to extract structured identity data from profile pages: full names, locations, gender, bios, profile images, linked accounts, and various platform-specific IDs. The SUPPORTED_IDS tuple in checking.py:37 defines nine identifier types:

username, yandex_public_id, gaia_id, vk_id, ok_id,
wikimapia_uid, steam_id, uidme_uguid, yelp_userid

Recursive search (enabled by default) feeds discovered usernames and IDs back into the search loop, building a graph of connected identities across platforms.

Activation System

Some sites require dynamic token acquisition before querying. The ParsingActivator class (maigret/activation.py, 137 lines) implements activation methods for Twitter (guest token refresh), Vimeo (JWT acquisition), OnlyFans (request signing with SHA1 checksums and dynamic rules), and Weibo (multi-stage visitor token flow). These are triggered when response content matches configured activation marks.

CLI Interface and Report Formats

The CLI (maigret/maigret.py, setup_arguments_parser() at line 122) exposes extensive options organized into argument groups:

  • Site filtering: --all-sites, --top-sites N, --tags, --exclude-tags, --site, --use-disabled-sites
  • Operating modes: --parse URL, --submit URL, --self-check, --stats, --web PORT
  • Report formats: --html, --pdf, --xmind, --txt, --csv, --json TYPE, --graph, --md, --ai
  • Output options: --verbose, --info, --debug, --no-color, --print-not-found, --print-errors

Report generation (maigret/report.py, 718 lines) produces HTML/PDF via Jinja2 templates and xhtml2pdf, XMind mindmaps, interactive D3 graph visualizations via pyvis/networkx, CSV, TXT, JSON (simple and NDJSON), and Markdown. The Markdown report includes structured sections for summary, extracted information, country/interest tags, and per-account details.

Web Interface

A Flask-based web UI (maigret/web/app.py, 352 lines) provides a browser-based alternative to the CLI. It uses background threads to run searches, generates all report formats, and serves interactive graph visualizations. Security hardening in commit f413603 fixed path traversal in the /reports/ endpoint using send_from_directory.

AI Analysis Mode

The --ai flag (maigret/ai.py, 162 lines) sends the generated Markdown report to an OpenAI-compatible API with streaming token output. It supports configurable model selection and base URL, making it compatible with any OpenAI API endpoint.

Code Quality Observations

Positive:

  • Consistent use of type hints throughout (typing imports, Optional, Dict, List, Tuple)
  • Well-structured test suite with 17 test modules covering core functionality, including a dedicated test_cloudflare_webgate.py (256 lines)
  • pytest-asyncio with asyncio_mode=auto configured in pytest.ini
  • Dev dependencies include flake8, mypy, black, pytest-cov, and coverage
  • CI workflows for Python package testing, Docker builds, PyInstaller packaging, and CodeQL analysis
  • Proper SSL context handling (line 191-193) that avoids TLS fingerprinting while handling invalid certs

Concerns:

  • Several # TODO comments scattered throughout (executor tests, username merge logic, retry tracking)
  • Multiple deprecated executor classes still present in executors.py with deprecation comments but no removal timeline
  • QueryResultWrapper is typed as Dict[str, Any] (types.py:11) — a loose type that weakens type safety across the codebase
  • The Settings class (settings.py) uses __dict__.update() to load configuration rather than explicit field assignment, which bypasses type checking
  • socid_extractor is imported at the top level of maigret.py with a try/except that raises a custom import error — functional but could be cleaner with lazy imports
  • The activation module (activation.py) uses synchronous requests calls within an async context, which blocks the event loop

Dependencies

The project has substantial dependencies (30+ production packages in pyproject.toml), including:

  • HTTP: aiohttp ^3.12.14, aiohttp-socks, requests, curl_cffi >=0.14, cloudscraper
  • Parsing: socid-extractor, lxml, html5lib, beautifulsoup4 (transitive)
  • Reporting: Jinja2, xhtml2pdf, XMind, PyPDF2, networkx, pyvis, reportlab
  • Web: flask ^3.1.1 (with async extras), asgiref
  • DNS: aiodns
  • AI: aiohttp (for streaming API calls)

Assessment

Strengths

  • Scale and coverage: 3,154 sites with auto-updating database is a significant competitive advantage. The engine/template system with MaigretEngine avoids configuration duplication across similar platforms.

  • Sophisticated anti-detection: Multiple checker strategies (aiohttp, curl_cffi for TLS impersonation, FlareSolverr for Cloudflare bypass) show thoughtful handling of modern web protections. The activation system handles token refresh for protected APIs.

  • Rich output ecosystem: Nine report formats plus an AI analysis mode cover virtually any downstream use case. The interactive graph visualization (networkx + pyvis) is particularly useful for visualizing identity correlations.

  • Operational flexibility: Tor/I2P support, proxy configuration, cookie jars, recursive search, username permutation, and tag-based site filtering make it adaptable to diverse OSINT scenarios.

  • Embeddable design: Can be used as a Python library (documented library usage), not just a CLI tool. The MaigretDatabase and MaigretSite classes are well-structured for programmatic access.

Concerns

  • Blocking sync calls in async context: The ParsingActivator methods (lines 13, 28, 44, 87 of activation.py) use synchronous requests calls that block the event loop. Under high concurrency with many activated sites, this could cause significant slowdowns.

  • Type safety: The pervasive use of Dict[str, Any] for QueryResultWrapper and QueryOptions makes static analysis difficult. A dataclass-based approach would improve maintainability.

  • Technical debt: Deprecated executor classes, scattered TODOs, and the # Deprecated: will be removed soon comments suggest incomplete refactoring. The codebase has grown organically.

  • Dependency surface: 30+ production dependencies is substantial for a tool of this nature. The inclusion of mock as a production dependency (rather than dev-only) is a minor packaging issue.

Recommendations

  • Migrate ParsingActivator methods to async (aiohttp) to avoid event loop blocking under load.
  • Replace QueryResultWrapper = Dict[str, Any] with a proper dataclass or TypedDict for better static analysis.
  • Remove deprecated executor classes to reduce maintenance burden and code confusion.
  • Move mock from production to dev-only dependencies in pyproject.toml.
  • Consider adding py.typed marker and stricter mypy configuration for the well-typed portions of the codebase.

Graph View