Crawl a site for RAG

Your agent needs to ground a model in an entire body of content — a documentation set, a knowledge base, a category of articles — not one page. That means crawling: discovering every URL under a path, rendering each one, and returning clean markdown ready to chunk and embed. Doing it as a hand-rolled loop of fetches means writing link discovery, dedup, rate-limiting and retry yourself. This page shows the agent-native path on a single Auxiliar key.

What the agent is trying to do

The agent has a concrete job: enumerate a path-scoped set of pages and return each as clean markdown — a corpus it can chunk, embed, and retrieve over. Coverage is the whole game: a crawl that silently misses a third of the pages produces a RAG system that confidently omits a third of the answers.

That requires more than fetching one URL:

• Link discovery — follow in-scope links, stay within the path, and not wander off-site.
• Deduplication and scope — one copy of each page, bounded to the section that matters.
• Per-page rendering — every page still needs JS rendering and boilerplate stripping, at scale.
• Coverage under limits — reach the long tail without hammering the origin or getting rate-limited.

Why the agent cannot solve this alone

Crawling is infrastructure, not reasoning — you cannot re-prompt your way to full coverage of a site. On its own, an agent cannot:

• Create or manage provider accounts — it can’t open a Firecrawl, Spider, or Tavily account and accept terms for a human.
• Pass signup, payment, and bot flows — provider onboarding sits behind email verification, card entry, and its own bot gates that need a real human browser.
• Safely hold many upstream keys — spreading provider secrets across an agent’s context or environment is a credential-sprawl and exfiltration risk.
• Know which crawler is winning right now — which crawler gets the best coverage at the lowest cost per useful page today is live measured data the model can’t sample.
• Run a crawl fleet — a distributed queue with dedup, politeness, rendering and retry is a system you operate, not a conclusion you think your way to.

So the blocker isn’t reasoning. Crawling needs accounts, credentials, crawl infrastructure, and current benchmark data — none of which live inside the model.

How Auxiliar solves it

Auxiliar is a web-access API gateway that collapses all of the above into one credential:

• One AUXILIAR_API_KEY — the only secret the agent ever holds.
• Upstream keys server-side — Firecrawl, Spider, Tavily and the rest are authenticated by Auxiliar; the agent never sees a provider key.
• Native provider routes — call https://api.auxiliar.ai/{provider}/{native-path} with the provider’s own request shape; you just change the base URL.
• One balance — credit-metered against a single account, no per-provider subscriptions.
• Eval-backed selection — crawler ranking comes from the Auxiliar curator fleet measuring coverage, latency and cost per useful page on one shared corpus, published at /evals.json and the /best/ pages.
• One-line crawler swaps — because every crawler is on the same key, the agent can default to the highest-coverage crawler and switch to a cheaper or faster one where coverage allows, with no new signup.

First call

Point at the top-ranked crawler — Firecrawl — with your one key. The body is Firecrawl’s own native shape; Auxiliar injects the Firecrawl key server-side.

curl -X POST "https://api.auxiliar.ai/firecrawl/v1/crawl" \
  -H "Authorization: Bearer $AUXILIAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "limit": 50, "scrapeOptions": {"formats": ["markdown"]}}'

A successful call kicks off a crawl and returns the discovered pages as clean markdown — every in-scope URL rendered and ready to chunk. Firecrawl led our crawl corpus on coverage (0.92) at $0.024 / useful page and ~16.3 s; Spider trades some coverage (0.77) for far lower cost ($0.000062 / useful page) and much lower latency. See the full crawler ranking.

Fallback ladder

No single crawler is best for every corpus. Start with the highest coverage and escalate — or downgrade — by what the corpus needs — every provider below is on the same key, so switching is a one-line path change, not a new account. Values are measured by the Auxiliar curator fleet on one corpus.

For managed crawl queues and site-specific jobs, Crawlbase offers pay-for-success by domain complexity and Apify offers pre-built Actors for a specific site — both on the same key when a bespoke crawl beats a generic one.

The rule of thumb: Firecrawl for maximum coverage, Spider or Tavily when volume, cost or speed dominate and the site is clean.

Eval-backed, not marketing claims

Every number above comes from the Auxiliar curator fleet running each provider on the same corpus — a merit-only ranking, not a listicle. Verify or route from the source:

• Best web crawler API — the full ranking with methodology (coverage, latency, cost).
• RAG over live web pages — the retrieve-then-read pipeline this corpus feeds.
• Firecrawl · Spider · Tavily — per-provider scorecards.
• /evals.json — the machine-readable catalog + eval endpoint your agent (or the auxiliar-mcp server) can read directly.

Optimizing for cost

Coverage first, then squeeze cost — in that order:

1. Default to coverage for the seed crawl. Missing pages become missing answers; a cheap crawl that skips a third of the corpus is the expensive one. Firecrawl by default for the first full pass.
2. Scope tightly. Bound the crawl to the path you actually need and set a sane limit. Crawling everything is the fastest way to overspend.
3. Downgrade re-crawls to a cheaper crawler. Once you know a site is clean and fully mapped, refresh it with Spider or Tavily at a fraction of the cost per page.
4. Track cost per useful page. Measure cost ÷ useful pages per crawler per site, not raw request price — pages you can’t use aren’t coverage.
5. Reserve unblockers for defended sites. Bright Data and heavy unblockers cost more; escalate to them only when a site’s anti-bot actually blocks the crawl.

Because all crawlers sit behind one key and one balance, this is routing logic in your agent — no renegotiated contracts, no new signups to change the mix.