Read web pages as clean markdown

Your agent has a URL and needs the page’s actual content — an article, a docs page, a product listing — as clean markdown it can reason over. A plain HTTP GET gives it a tag-soup of navigation, ads, and scripts, or an empty shell because the content renders in JavaScript. Getting from URL to trustworthy text is a real step, and re-prompting doesn’t fix it. This page shows the agent-native path on a single Auxiliar key.

What the agent is trying to do

The agent has a concrete job: turn a URL into the readable content of that page as clean markdown — headings, paragraphs, tables and links preserved, boilerplate stripped. Garbled input poisons every downstream step, so the quality of this conversion sets the ceiling for the whole task.

That requires more than an HTTP client:

• JavaScript rendering — much of the modern web assembles content client-side; a raw GET returns an empty frame.
• Boilerplate removal — nav, cookie banners, ads and scripts have to come out, or they dominate the tokens and confuse the model.
• Faithful structure — headings, lists and tables must survive the conversion, not collapse into a wall of text.
• Blocked pages — some targets sit behind anti-bot systems, so the reader also needs to clear a challenge to see the content at all.

Why the agent cannot solve this alone

Clean rendering isn’t a reasoning problem — you cannot re-prompt your way past an empty JS shell. On its own, an agent cannot:

• Create or manage provider accounts — it can’t open a Firecrawl, Jina, or Scrapfly account and accept terms for a human.
• Pass signup, payment, and bot flows — provider onboarding sits behind email verification, card entry, and its own bot gates that need a real human browser.
• Safely hold many upstream keys — spreading provider secrets across an agent’s context or environment is a credential-sprawl and exfiltration risk.
• Know which reader is winning right now — which reader produces the cleanest markdown at the lowest cost per successful page today is live measured data the model can’t sample.
• Run headless-browser infrastructure — rendering farms, boilerplate extraction, and anti-bot handling are systems you operate, not conclusions you think your way to.

So the blocker isn’t reasoning. Clean reading needs accounts, credentials, rendering infrastructure, and current benchmark data — none of which live inside the model.

How Auxiliar solves it

Auxiliar is a web-access API gateway that collapses all of the above into one credential:

• One AUXILIAR_API_KEY — the only secret the agent ever holds.
• Upstream keys server-side — Firecrawl, Jina, Scrapfly and Spider are authenticated by Auxiliar; the agent never sees a provider key.
• Native provider routes — call https://api.auxiliar.ai/{provider}/{native-path} with the provider’s own request shape; you just change the base URL.
• One balance — credit-metered against a single account, no per-provider subscriptions.
• Eval-backed selection — reader ranking comes from the Auxiliar curator fleet measuring markdown quality, anti-bot bypass, latency and cost per successful page on one shared corpus, published at /evals.json and the /best/ pages.
• One-line reader swaps — because every reader is on the same key, the agent can default to the highest-quality reader and fall back to a stealth one on blocked pages, with no new signup.

First call

Point at the top-ranked reader — Firecrawl — with your one key. The body is Firecrawl’s own native shape; Auxiliar injects the Firecrawl key server-side.

curl -X POST "https://api.auxiliar.ai/firecrawl/v1/scrape" \
  -H "Authorization: Bearer $AUXILIAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/some-article", "formats": ["markdown"]}'

A successful call returns HTTP 200 with the page as clean markdown — JS rendered, boilerplate stripped, structure intact. Firecrawl led our scraping corpus on markdown quality while clearing 100% of the anti-bot vendors at ~962 ms and $0.0021 / successful page. For a dedicated URL→markdown Reader, Jina is purpose-built and can search-plus-read in a single call. See the full scraping ranking.

Fallback ladder

No single reader is best on every page. Start with the highest-quality reader and escalate by failure mode — every provider below is on the same key, so escalation is a one-line path change, not a new account. Values are measured by the Auxiliar curator fleet on one corpus.

The rule of thumb: Firecrawl for the cleanest markdown by default, Jina when you want a Reader-shaped call, and Scrapfly / Zyte when the page is defended and the fetch would otherwise fail.

Eval-backed, not marketing claims

Every number above comes from the Auxiliar curator fleet running each provider on the same corpus — a merit-only ranking, not a listicle. Verify or route from the source:

• Best web scraping API — the full ranking with methodology (markdown quality, bypass, cost).
• Firecrawl vs Jina — the two leading URL-to-markdown options head to head.
• Firecrawl · Jina · Scrapfly · Spider — per-provider scorecards.
• /evals.json — the machine-readable catalog + eval endpoint your agent (or the auxiliar-mcp server) can read directly.

Optimizing for cost

Quality first, then squeeze cost — in that order:

1. Default to the cleanest reader. A garbled fetch that forces a re-read or a wrong answer costs more than a slightly pricier clean one. Firecrawl by default.
2. Route easy pages to the cheapest sufficient reader. Undefended, mostly-static pages don’t need heavy rendering — send them to a low-cost reader that still returns clean markdown.
3. Reserve stealth readers for blocked pages. Scrapfly and other anti-bot readers cost and take more; escalate to them only when the cheaper tier actually returns an empty or challenged page.
4. Track cost per successful page. Measure cost ÷ successes per reader per domain, not raw request price — that’s what reflects real spend.
5. Trim what you feed downstream. Ask for markdown only (not screenshots or raw HTML you won’t use) and truncate to the section you need before sending to the model.

Because all readers sit behind one key and one balance, this is routing logic in your agent — no renegotiated contracts, no new signups to change the mix.