If you’ve ever gotten a 403, an endless CAPTCHA, or a “Checking your browser…” page while scraping, you’ve met an anti-bot system — Cloudflare, DataDome, Akamai, PerimeterX. The traditional fix is a proxy stack you build and babysit: residential IPs, rotation, fingerprints, retries. This guide skips all of that. You call one Auxiliar key, a managed scraper handles the unblocking, and if one provider gets blocked you fall back to another — in a single line.
The one-flag approach
The best scraping APIs turn on stealth with a single parameter and return the page as clean markdown. Through the gateway that’s one request:
import os, requests
AUX = "https://api.auxiliar.ai"
H = {"Authorization": f"Bearer {os.environ['AUXILIAR_API_KEY']}"}
def scrape(url):
r = requests.post(f"{AUX}/firecrawl/v1/scrape", headers=H,
json={"url": url, "formats": ["markdown"]}, timeout=90)
r.raise_for_status()
return r.json()["data"]["markdown"]
No IPs to rotate, no browser to run. Firecrawl cleared 100% of the anti-bot vendors in our corpus at ~962 ms — see the full anti-bot scraping ranking.
Fallback: when one provider is blocked, try the next
No single scraper wins every site. The advantage of one key over every provider is that a fallback is trivial — loop through providers until one returns the page:
PROVIDERS = [
("firecrawl", "/firecrawl/v1/scrape", lambda u: {"url": u, "formats": ["markdown"]}),
("scrapfly", "/scrapfly/scrape", lambda u: {"url": u, "asp": True, "format": "markdown"}),
("brightdata","/brightdata/unlocker", lambda u: {"url": u}),
]
def scrape_resilient(url):
for name, path, body in PROVIDERS:
try:
r = requests.post(f"{AUX}{path}", headers=H, json=body(url), timeout=90)
if r.ok and r.text.strip():
return name, r.text
except requests.RequestException:
continue
raise RuntimeError(f"all providers failed for {url}")
Request/response shapes are each provider’s native API — Auxiliar passes them through unchanged — so check docs.auxiliar.ai for the exact body per provider. The point stands: three of the strongest unblockers on the market, tried in sequence, on one key.
This kind of cross-provider fallback is the single biggest reliability win of a gateway. For an LLM, a call rarely “fails”; for a scraper, one target blocking you is routine — so being able to reroute to the next-best provider mid-run is worth more here than almost anywhere else.
Choosing the right unblocker
Different providers lead on different targets and price points. Scrapfly and Firecrawl both cleared every anti-bot vendor in testing; Bright Data and Oxylabs are the enterprise heavyweights. Compare them in best anti-bot scraping API and head-to-head in Bright Data vs Oxylabs — then, since they’re all on one key, let your fallback chain pick the winner per site.