Extract structured data from pages

Your agent doesn’t need the whole page — it needs three fields from it: the price, the availability, the publish date. Getting clean markdown and then re-parsing it with brittle regex or a second LLM pass is slow, lossy, and expensive. Schema-driven extraction returns a typed JSON record directly. This page shows the agent-native path to structured extraction on a single Auxiliar key.

What the agent is trying to do

The agent has a concrete job: turn a page into a typed record that matches a schema — the fields it asked for, correctly filled, in JSON it can act on without post-processing. Field accuracy is the whole point: a record with the wrong price is worse than no record.

That requires more than a scrape:

• Render then locate — run the page’s JS, then find the right values among the noise.
• Schema conformance — return exactly the fields requested, typed, so downstream code doesn’t guess.
• Accuracy on messy pages — real pages vary in layout; the extractor has to be robust to that variance.
• Bypass when needed — many high-value pages (product, listing) are also defended, so extraction must clear anti-bot first.

Why the agent cannot solve this alone

Accurate extraction at scale is infrastructure plus measurement, not reasoning. On its own, an agent cannot:

• Create or manage provider accounts — it can’t open a Scrapfly, Firecrawl, or Zyte account and accept terms for a human.
• Pass signup, payment, and bot flows — provider onboarding sits behind email verification, card entry, and its own bot gates that need a real human browser.
• Safely hold many upstream keys — spreading provider secrets across an agent’s context or environment is a credential-sprawl and exfiltration risk.
• Know which extractor is winning right now — which provider has the highest field accuracy at the lowest cost per useful record today is live measured data the model can’t sample.
• Run rendering and extraction infrastructure — headless rendering, anti-bot handling, and tuned extraction pipelines are systems you operate, not conclusions you reach by thinking harder.

So the blocker isn’t reasoning. Structured extraction needs accounts, credentials, rendering infrastructure, and current benchmark data — none of which live inside the model.

How Auxiliar solves it

Auxiliar is a web-access API gateway that collapses all of the above into one credential:

• One AUXILIAR_API_KEY — the only secret the agent ever holds.
• Upstream keys server-side — Scrapfly, Firecrawl, ScrapingBee and Zyte are authenticated by Auxiliar; the agent never sees a provider key.
• Native provider routes — call https://api.auxiliar.ai/{provider}/{native-path} with the provider’s own request shape; you just change the base URL.
• One balance — credit-metered against a single account, no per-provider subscriptions.
• Eval-backed selection — extractor ranking comes from the Auxiliar curator fleet measuring field accuracy, latency and cost per useful record on one shared corpus, published at /evals.json and the /best/ pages.
• One-line extractor swaps — because every extractor is on the same key, the agent can pick the most accurate one and fall back to a typed-schema extractor for common page types, with no new signup.

First call

For prompt/schema-driven JSON in a single scrape call, use Firecrawl’s native json format with your one key. Auxiliar injects the Firecrawl key server-side.

curl -X POST "https://api.auxiliar.ai/firecrawl/v1/scrape" \
  -H "Authorization: Bearer $AUXILIAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.example.com/product/123",
    "formats": ["json"],
    "jsonOptions": {
      "prompt": "Extract the product name, price, and availability.",
      "schema": {
        "type": "object",
        "properties": {
          "name": {"type": "string"},
          "price": {"type": "number"},
          "availability": {"type": "string"}
        }
      }
    }
  }'

A successful call returns a typed JSON record matching the schema — no re-parsing. Firecrawl scored field accuracy 0.96 at $0.0052 / useful record; the accuracy leader is Scrapfly at 1.00 and $0.002 / useful record. See the full extraction ranking.

Fallback ladder

No single extractor is best on every page type. Start with the most accurate and escalate — or specialize — by what the page needs — every provider below is on the same key, so switching is a one-line path change, not a new account. Values are measured by the Auxiliar curator fleet on one corpus.

For a site with a bespoke shape, Apify offers pre-built Actors that emit site-specific structured items, and when schema extraction is overkill, Jina returns clean markdown to pull fields from — both on the same key.

The rule of thumb: Scrapfly or Firecrawl for accurate schema extraction by default; drop to Zyte for common typed page types where its fixed schema fits and cost matters.

Eval-backed, not marketing claims

Every number above comes from the Auxiliar curator fleet running each provider on the same corpus — a merit-only ranking, not a listicle. Verify or route from the source:

• Best AI data extraction API — the full ranking with methodology (field accuracy, latency, cost).
• Scrapfly · Firecrawl · ScrapingBee · Zyte — per-provider scorecards.
• /evals.json — the machine-readable catalog + eval endpoint your agent (or the auxiliar-mcp server) can read directly.

Optimizing for cost

Accuracy first, then squeeze cost — in that order:

1. Default to the accurate extractor. A wrong field is more expensive than a slightly pricier correct one — bad data propagates. Scrapfly or Firecrawl by default.
2. Use typed presets for common page types. For standard product/article/job pages, Zyte’s fixed-schema extraction is cheap and fits; reserve prompt/schema extraction for bespoke fields.
3. Extract once, don’t re-parse. Schema-driven JSON avoids a second LLM pass over markdown — fewer tokens, fewer failures, lower cost.
4. Track cost per useful record. Measure cost ÷ correct records per extractor per page type, not raw request price — a record you can’t trust isn’t useful.
5. Reserve stealth for defended pages. Heavy anti-bot extraction costs more; escalate to it only when a plain extract is actually blocked.

Because all extractors sit behind one key and one balance, this is routing logic in your agent — no renegotiated contracts, no new signups to change the mix.