The Auxiliar curator fleet ran the major web-access APIs head-to-head on the same corpus, scoring each on quality, latency, cost per success and error rate — per capability. This is the merit-only result: no house provider, no thumb on the scale, the same tests for everyone. Here’s what 22 providers across 13 capabilities actually look like when you measure them.
Headline findings
- Firecrawl won 5 of 13 capabilities — more than any other provider — leading scrape, crawl, screenshot, PDF parsing and change detection.
- No provider wins everything. The best SERP provider, the best AI-search index, the best extractor and the best unblocker are four different companies. Standardizing on one vendor means using its weakest verb somewhere.
- Anti-bot is solved — for some. Firecrawl, Scrapfly cleared 100% of the anti-bot vendors in the corpus; others failed outright on the same targets.
- Price spans orders of magnitude. On SERP, Serper came in cheapest at $0.0003 / call — a fraction of the premium incumbents for comparable quality.
Winners by capability
Every score is a composite (0–10) from the same corpus. Each links through to the full scorecard.
| Capability | Winner | Score | Runner-up | Measured on |
|---|---|---|---|---|
| Web search (RAG grounding) | Jina | 7.69/10 | You.com (7.41) | Recall@10 |
| Scrape to markdown | Firecrawl | 9.81/10 | Scrapfly (9.48) | Anti-bot bypass |
| SERP / Google results | Serper | 9.56/10 | SearchAPI.io (9.02) | Quality |
| Cited answers & research | Exa | 7.9/10 | You.com (7.79) | Correctness |
| Site crawling | Firecrawl | 8.61/10 | Spider (8.54) | Coverage |
| AI / schema extraction | Scrapfly | 8.96/10 | Firecrawl (8.81) | Field accuracy |
| Rule-based (CSS) extraction | ScrapingBee | 9.23/10 | Oxylabs (8.52) | Field accuracy |
| Screenshots | Firecrawl | 9.5/10 | Zyte (9.04) | Valid image |
| Structured domain scraping | Oxylabs | 8.74/10 | Apify (8.23) | Accuracy |
| PDF / document parsing | Firecrawl | 8.82/10 | Jina (8.43) | Text accuracy |
| Declarative browser actions | ScrapingBee | 8.72/10 | Zyte (8.5) | Task success |
| Natural-language browser agents | Oxylabs | 7.99/10 | Firecrawl (5.66) | Task success |
| Change detection | Firecrawl | 8.33/10 | — | Class. accuracy |
The standouts
- Best scrape quality — Firecrawl. Top markdown-cleanliness score (9.6 / 10) at 962 ms, $0.0021 / success. See the best web scraping API ranking.
- Best anti-bot bypass — Firecrawl. 100% of protected targets cleared. Full list: best anti-bot scraping API.
- Cheapest Google/SERP — Serper. $0.0003 / call per call at 1.0 s. See cheapest search API and best SERP API.
- Best search for RAG — Jina. Highest recall composite for agent grounding: best search API for AI agents.
- Best cited answers — Exa. Leads on correctness and citation faithfulness: best AI answer & research API.
Why we published this
The web-access market is dozens of providers all claiming to be fastest, cheapest and most reliable. Almost none publish head-to-head numbers — and the few “benchmarks” that exist are run by vendors grading their own homework. Auxiliar has no reason to favor one provider over another: it resells all of them on one key. So the ranking above is the honest one, and you can act on it directly — every provider here is reachable through a single Auxiliar API key, so you can route each job to the winner (or fall back between them) without a new signup.
Methodology
Each provider was exercised on identical inputs per capability (“verb”) — the same URLs to scrape, the same queries to search, the same documents to parse. We scored verb-specific quality (e.g. anti-bot bypass rate, markdown cleanliness, recall, field accuracy), latency p50, cost per successful result, and error rate, then combined them into a composite. Providers that couldn’t be scored on a synchronous request/response corpus (e.g. stateful browser sessions) are noted on their scorecards rather than ranked. Numbers are refreshed as the fleet re-runs; see any provider’s page for its latest detail.