Your agent asks for a page and gets back a Cloudflare interstitial, a DataDome challenge, an Akamai sensor script, an HTTP 403, or an HTML shell with none of the content it needed because the site renders in JavaScript. No amount of re-prompting fixes this — the block is infrastructure, not intelligence. This page shows the agent-native path through it, on a single Auxiliar key.
What the agent is trying to do
The agent has a concrete job: fetch the readable content of a URL — an article, a product page, a listing, a docs page — and hand back clean text or markdown it can reason over. It doesn’t want to become a scraping company. It wants the page.
What stands between it and the page is a stack of anti-bot systems and rendering requirements that change per site:
- Bot fingerprinting — Cloudflare, DataDome, Akamai, PerimeterX inspect TLS, headers, and behavior and return a challenge or a
403. - JavaScript rendering — the content only exists after the page’s JS runs, so a plain HTTP GET returns an empty shell.
- IP reputation — datacenter IPs get blocked; the request needs residential or mobile egress.
- Per-site drift — what works on one target fails on the next, and what worked yesterday breaks today.
Why the agent cannot solve this alone
This is the important part: these failures are not things an LLM can reason its way out of. Trying harder produces the same 403. Specifically, an agent on its own cannot:
- Create or manage provider accounts — it can’t decide to open a Firecrawl, Scrapfly, or Bright Data account and agree to terms on a human’s behalf.
- Pass signup, payment, and bot flows — provider onboarding sits behind email verification, card entry, and its own Cloudflare/CAPTCHA gates that require a real human browser.
- Safely hold many upstream keys — even if it had accounts, stuffing a dozen provider secrets into an agent’s context or environment is a credential-sprawl and exfiltration risk.
- Know current provider success and cost — which unblocker is actually clearing this target today, and at what cost per successful page, is live measured data the agent has no way to sample on its own.
- Run proxy and browser infrastructure — residential proxy pools, headless browser farms, and fingerprint rotation are systems you operate, not conclusions you reach by thinking harder.
So the blocker isn’t the agent’s reasoning. It’s that solving blocked scraping requires accounts, credentials, infrastructure, and current benchmark data — none of which live inside the model.
How Auxiliar solves it
Auxiliar is a web-access API gateway that collapses all of the above into one credential:
- One
AUXILIAR_API_KEY— the only secret the agent ever holds. - Upstream keys server-side — Firecrawl, Scrapfly, Zyte, Bright Data and the rest are authenticated by Auxiliar; the agent never sees or manages a provider key.
- Native provider routes — call
https://api.auxiliar.ai/{provider}/{native-path}with the provider’s own request shape; existing SDKs and code keep working, you just change the base URL and path. - One balance — credit-metered against a single account, no per-provider subscriptions or invoices.
- Eval-backed selection — provider ranking comes from the Auxiliar curator fleet measuring bypass rate, latency and cost per success on one shared corpus, published at
/evals.jsonand the/best/pages. - Fallback across providers — because every unblocker is on the same key, the agent can try the fastest/cheapest sufficient provider and escalate to a heavier one on failure, without a single new signup.
First call
Point at the top-ranked unblocker — Firecrawl — with your one key. The request body is Firecrawl’s own native shape; Auxiliar injects the Firecrawl key server-side.
curl -X POST "https://api.auxiliar.ai/firecrawl/v1/scrape" \
-H "Authorization: Bearer $AUXILIAR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://a-site-that-blocked-you.com", "formats": ["markdown"]}'
A successful call returns HTTP 200 with the page as clean markdown — anti-bot cleared, JS rendered, no proxy or browser for you to run. Firecrawl cleared 100% of the anti-bot vendors in our corpus at ~962 ms and $0.0021 / success — see the full anti-bot scraping ranking.
Fallback ladder
No single provider wins every target. Start at the top and escalate by failure mode — every provider below is on the same key, so escalation is a one-line path change, not a new account. Values are measured by the Auxiliar curator fleet on one corpus.
| Order | Provider | Reach for it when | Anti-bot bypass | Cost / success | Latency p50 |
|---|---|---|---|---|---|
| 1 | Firecrawl | Default. Clean markdown, fast, cheapest of the 100%-bypass tier. | 100% | $0.0021 | 962 ms |
| 2 | Scrapfly | Firecrawl returns a challenge; you want stealth + AI extraction on the retry. | 100% | $0.0025 | 4.5 s |
| 2 | Zyte | You want automatic ban-management and the lowest cost per success at full bypass. | 100% | $0.00069 | 6.5 s |
| 3 | Bright Data | Hardest targets; you need the strongest unblocker and compliance posture. | 75% | $0.0019 | 3.9 s |
| 3 | Oxylabs | Enterprise scale, dedicated parsers, or the NL-agent mode. | 75% | $0.0013 | 10.5 s |
| 3 | Crawlbase | You want a managed crawl queue and pay-for-success by domain complexity. | 75% | $0.006 | 3.5 s |
| 3 | ZenRows | You need one key spanning a scraper API and a full scraping browser. | 62% | $0.0047 | 4.8 s |
| 3 | ScraperAPI | You want the simplest possible unblocking call. | 62% | $0.01 | 2.4 s |
| 3 | Spider | High-volume crawling where transparent per-page pricing matters most. | 50% | $0.00013 | 12.1 s |
The rule of thumb: tier-1 (Firecrawl → Scrapfly/Zyte) clears everything anti-bot in our corpus; drop to Bright Data / Oxylabs for the genuinely hardest targets, and use the rest to trade latency, price, or a specific feature.
Eval-backed, not marketing claims
Every number above comes from the Auxiliar curator fleet running each provider on the same corpus — a merit-only ranking, not a listicle. Verify or route from the source:
- Best anti-bot scraping API — the full ranking with methodology.
- Firecrawl · Scrapfly · Zyte · Bright Data — per-provider scorecards.
/evals.json— the machine-readable catalog + eval endpoint your agent (or theauxiliar-mcpserver) can read directly.
Optimizing for cost
Success first, then squeeze cost — in that order:
- Start success-first. Route the first attempt to a provider that clears the target (Firecrawl by default). A cheap failure is more expensive than a slightly pricier success.
- Route easy pages to the cheapest sufficient provider. Not every URL is defended. For pages that don’t need heavy unblocking, send them to the cheapest provider that still returns clean content.
- Reserve browser/enterprise unblockers for failures. Bright Data, Oxylabs and full scraping-browser modes cost and take more time — escalate to them only when the cheaper tier actually fails, not by default.
- Log cost per successful page. Track
cost ÷ successesper provider per domain, not raw request cost. That’s the number that reflects real spend. - Downgrade once the success threshold is met. When a cheaper provider’s measured success rate on a domain crosses your bar, make it the default for that domain and keep the heavier one as fallback.
Because all providers sit behind one key and one balance, this is routing logic in your agent — no renegotiated contracts, no new signups to change the mix.