AI Search & Crawler Engineering for Crypto

See what AI search engines actually read on your protocol.

In live audits across 11 DeFi and Layer 1 protocols ($1.38B TVL), 7 of 11 block AI crawlers at Cloudflare WAF or robots.txt, 4 of 11 expose 0 to 8 raw HTML words without JavaScript, and owned domains scored 0 of 44 on non-brand Perplexity category searches.

11 Protocols $1.38B live TVL audited
0 / 44 Hits Non-brand Perplexity category citations
7 / 11 Blocked HTTP 403 or robots.txt exclusions
Live Instant Probe

Test any domain against 6 AI crawlers and raw HTML extraction

Sample live output: axis.to (2026-11-19)
Live presets:
Verified Corpus (11 Protocols · $1.38B TVL)

Benchmark Explorer & Live AI Retrieval Telemetry

Inspect real pillar scores, Chromium JS-on vs. JS-off word extraction, edge WAF blocks, and live Perplexity + Gemini answer-engine results.

86 / 100 GRADE A
Monad monad.xyz
$891.3M TVL 2026-11-15
RAW HTML: 709w JS-ON: 650w

Measured Infrastructure Findings
    Live Retrieval Matrix
    Tested Query Owned Rank Top Displacing Result
    Priority Engineering Remediation

    Methodology & Observed Failure Modes

    Six weighted technical checks grounded in live telemetry

    Every score comes from verifiable HTTP traces, Chromium DOM diffs, open registry APIs, and web-grounded AI queries. Nothing is simulated.

    01 · Crawler Access 7/11 Blocked

    RFC 9309 + Targeted vs. Site-Wide WAF

    Separates intentional robots.txt opt-outs from silent firewall blocks. On infinifi.xyz and loopscale.com, Cloudflare drops all 14 bots; on axis.to, robots.txt line 143 allows PerplexityBot while Cloudflare drops it with a 25-byte HTTP 403.

    02 · JS-Off Rendering 4/11 CSR Blackouts

    Headless Chromium JS-On vs. JS-Off Diff

    Captures what non-JS crawlers extract vs. human browsers. current.finance drops from 494 rendered words to 0 raw HTML words, prjx.com from 964 to 7, and mezo.org sub-routes /docs and /earn/lock drop to 0.

    03 · Schema & Graph Canonical Leaks

    JSON-LD Entities & Subdomain Consistency

    Validates Organization, WebSite, and FAQPage in raw HTML. Catches missing root schemas on nado.xyz, mezo.org, and loopscale.com, plus multi-host splits across entropy.io, pro.entropy.io, and entropy.trade.

    04 · Machine Corpus 7/11 Missing llms.txt

    Sitemap Hygiene & /llms.txt Endpoint DNS

    Verifies markdown validity and tests referenced URLs and RPC hosts live. Catches nado.xyz leaking /emailtest in sitemap.xml, prjx.com serving HTML on /llms.txt, and monad.xyz/llms.txt pointing to unresolved rpc.testnet.monad.xyz.

    05 · Entity Footprint Off-Site Trust

    DefiLlama, CoinGecko, GitHub & Wikidata

    Audits third-party truth sources that LLMs cross-reference for safety and disambiguation. Prevents unverified aggregators like tradingstrategy.ai or iq.wiki from defining your protocol risk and token identity.

    06 · Live Citations 0/44 Category Hits

    Multi-Engine Web-Grounded Query Benchmarking

    Measures brand, comparison, and category share across Perplexity Search and Google Gemini grounding. On Monad (86/100 A), Gemini still hedged between 400ms and 300ms block times because official AI docs drifted (v0.16.1 vs. v0.16.5).

    Engagement Options

    Transparent, fixed-scope engineering engagements

    Run the instant crawler probe for free, commission a 48-hour remediation sprint, or track weekly WAF and citation drift continuously.

    Self-Serve Diagnostics

    Instant Probe

    $0

    Live HTTP, WAF scope, and /llms.txt DNS check right on this page.

    • RFC 9309 robots.txt longest-match check
    • 6-crawler WAF probe (site-wide vs. targeted)
    • Raw HTML word count and JSON-LD types
    • Live /llms.txt and /sitemap.xml DNS validation
    Run Free Probe
    Ongoing Protection

    Drift & Citation Monitor

    $1,200 / mo

    Automated weekly regression alerts and monthly competitor citation tracking.

    • Weekly live WAF, robots.txt & /llms.txt RPC drift checks
    • Instant alert when edge rules silently block search bots
    • Monthly Perplexity + Gemini comparison & category tracking
    • Architecture & copy review for comparison and security docs
    Request Monitor
    Sample Sprint Artifacts

    Concrete repository patches delivered with every Sprint

    public/llms.txt
    
              
    Technical FAQ

    Questions engineering & growth teams ask

    Why does our React / Vite site look complete in Chrome while AI search misses it?

    Desktop Chrome executes JavaScript bundles and mounts DOM copy after initial load. Search crawlers parse initial raw HTML first—or get stopped at Cloudflare with HTTP 403 before HTML arrives. Across our 11-protocol corpus, 4 protocols expose 0 to 8 words in raw HTML and 7 block crawlers at the edge or via robots.txt.

    If robots.txt allows PerplexityBot or OAI-SearchBot, can our firewall still block them?

    Yes. In our live probe suite, 3 of 11 protocols explicitly permitted live search bots in robots.txt while Cloudflare Bot Fight Mode or custom WAF rules still returned HTTP 403. On axis.to, regular Chrome requests return 200 while PerplexityBot receives a 25-byte HTTP 403 block.

    Why run live Perplexity and Google Gemini queries in addition to static HTML checks?

    Technical hygiene alone does not capture non-brand citations. Even monad.xyz (86/100 Grade A) scored 0 of 4 on category queries while Gemini answers contradicted 400ms vs. 300ms specs because /llms.txt pointed to an unresolvable testnet RPC and docs listed mismatched versions.

    How does a 48-hour AI Visibility Sprint work?

    Fixed $750 scope paid via standard invoice or USDC, with zero production downtime. You submit your root URL and subdomains below; within 48 hours we deliver the full 6-pillar report, 12-query live AI search breakdown, verified patch files ready for engineering review, and a 14-day Before/After check once merged.

    Book an Engagement

    Request an AI Visibility Sprint or Monitor

    Submit your root domain, target scope, and team contact. We prepare a structured intake ticket ready to dispatch via email or X DM and copy it straight to your clipboard.

    TURNAROUND 48h Full Audit + PR Patch Bundle
    DIRECT EMAIL hello@proofofcite.com
    X / PUBLIC FEED x.com/ProofOfCite