Key takeaways
- Decide access routes per field and per source, not per project: use an official API where it exposes the field at a workable quota, and scrape only the gaps.
- Anti-bot costs are expected to keep rising, so the price of a scraped field is a moving number you should track, not assume.
- Regulatory access routes such as DSA Article 40 exist, but they are aimed at researchers and regulators, so don’t plan a commercial pipeline around them.
- Record which route produced each record and when; that makes the pipeline auditable and lets you switch routes without rewriting downstream code.
A hybrid data sourcing strategy means choosing the access route field by field: take data from an official API where one covers it, and scrape only what the API leaves out. Most teams still pick one route per project and live with the consequences, which usually means either an API quota that blocks them or a scraper fleet maintained for data a vendor would have handed over for free.
This post sets out how to make that choice per field, what it costs you to get it wrong, and how to build the routing so you can change your mind later.
Table of contents
Why is the "scrape or API" question changing now?
Two pressures point in opposite directions. Anti-bot systems keep getting more capable: one 2026 overview describes vendors such as Cloudflare Turnstile, PerimeterX and DataDome checking TLS fingerprints, JavaScript environment signals and behavioural patterns, and expects anti-bot costs to rise (Apify, future of web scraping 2026). That makes every scraped field a little more expensive over time.
At the same time, more structured routes are opening up. The same overview lists official APIs such as Google Places and limited Twitter and LinkedIn APIs, and recommends using them when the fields you need are exposed and the quotas fit, with scraping reserved for gaps. It also points to the EU Digital Services Act, whose Article 40 ("Data access and scrutiny") requires very large platforms to provide data access to researchers and regulators; the provision is in Regulation (EU) 2022/2065 on EUR-Lex.
That last point needs care. Article 40 is not a commercial access right. Treat it as evidence that regulated routes are multiplying, not as a source for your pipeline unless you are actually a qualifying researcher.
How do you build a hybrid data sourcing strategy field by field?
Treat the decision as a small table you fill in per source and per field, rather than a philosophy. Four questions do most of the work:
| Question | If yes | If no |
|---|---|---|
| Does the official API expose this field? | Candidate for API | Scrape, or drop the field |
| Do quotas cover your volume and refresh rate? | API is viable | Split: API for core fields, scrape for the rest |
| Do the API terms allow your use (storage, redistribution)? | Proceed | Check the terms before you build, not after |
| Is the field stable on the page, and is the site tolerant of access? | Scraping is cheap to run | Expect rising cost; weigh against a licensed or API route |
A worked example: say you track listings across ten sources. Source A has an API with price, availability and location, but no seller reviews. Sources B to J have no API at all. The sensible split is API for A’s price, availability and location, a scraper for A’s reviews, and scrapers for everything else. One source, two routes, and you only pay anti-bot costs where you must.
If your question is the broader one of search API, crawl API or your own fleet for AI agents, we covered that in our decision guide to web data access for AI agents. This post is the narrower follow-on: once you have several routes available, how do you combine them.
What does a routing layer look like in practice?
Keep route choice in configuration, not in scraper code. A minimal version:
sources:
source_a:
fields:
price: { route: api, endpoint: /v1/places, refresh: 6h }
availability: { route: api, endpoint: /v1/places, refresh: 6h }
reviews: { route: scrape, extractor: reviews_v3, refresh: 24h }
source_b:
fields:
price: { route: scrape, extractor: listing_v2, refresh: 12h }Every record that leaves this layer carries the same envelope regardless of route:
{
"source": "source_a",
"field": "price",
"route": "api",
"collected_at": "2026-10-28T09:14:00Z",
"policy_version": "source_a-terms-2026-09",
"value": 129.0
}The route and policy_version fields are what make the pipeline auditable. When an API changes its quota or a site changes its terms, you change one line of configuration and the downstream schema stays the same. That is the practical payoff of the hybrid approach: switching routes becomes a config change rather than a rewrite.
Where does LLM extraction fit in the scrape half?
When a scraped field needs more than a stable selector, LLM extraction is an option, but it is a cost decision. One 2026 guide estimates AI extraction at roughly 10 to 50 times the cost of CSS parsing for the same page, and says CSS is often more reliable on stable, well-structured pages. It suggests AI is often justified at around 1,000 products across 10 sites, while hybrid approaches work better at around 100,000 products (Apify, web scraping with AI and LLMs in 2026). Those are one vendor’s estimates, so measure your own.
The pattern it recommends mirrors the sourcing logic above: selectors first for stable, high-volume fields, LLM fallback when selectors fail or return null on a critical field, and caching so extraction re-runs only when the content hash changes. The last point is where incremental crawling and change detection pays for itself.
What should you document so the hybrid setup survives an audit?
Mixing routes means mixing legal bases and terms. API access runs under the provider’s terms; scraping runs under whatever the site’s policies and applicable law allow. A practical record per source covers the route used, the terms or policy version, the date you checked them, and any personal data involved, since GDPR obligations around lawful basis and minimisation apply to scraped personal data regardless of the route (Apify).
If downstream users, such as AI teams, will be asked where their training data came from, this record is the thing they’ll need. For AI training specifically, the EU’s template for summarising training content asks for the crawlers used, collection period and the top 10% of scraped domains by content size, plus the measures taken to honour text-and-data-mining opt-outs (Slaughter and May). A route-tagged record gets you most of the way to answering that.
Conclusion
Stop asking "scrape or API" once per project. Ask it per field, put the answer in configuration, tag every record with its route and the terms that applied, and revisit the table when an API appears or a site hardens. The saving comes from not paying scraping costs for data someone will serve you properly, and from being able to change routes without touching your warehouse schema.
The limits: the cost figures above come from vendor blogs and are estimates, and regulatory access routes like DSA Article 40 will not apply to most commercial users.
If your pipeline needs each record documented with its source, collection date and legal basis, that is how Dataizi datasets are delivered, and we run the scraping and API collection behind them as a managed service. See our datasets or talk to us.
Sources
- Apify: Future of web scraping 2026
- Apify: Web scraping with AI and LLMs in 2026
- EUR-Lex: Regulation (EU) 2022/2065 (Digital Services Act)
- Slaughter and May: EU AI Office template for summarising GPAI training content
Facts reflect the sources as read on 2026-10-28.
When should I use an official API instead of scraping?
Use the API when it exposes the fields you need and its quotas and terms fit your volume. Scrape only for the fields or sources the API does not cover.
Is a hybrid approach more expensive to maintain than scraping alone?
It adds a routing layer and a normalisation step, but it removes the most fragile and costly scraping work for fields an API serves reliably. The extra code is small compared with maintaining selectors and anti-bot countermeasures for everything.
Does the EU Digital Services Act give companies access to platform data?
Article 40 requires very large platforms to give data access to researchers and regulators. It is not a general commercial data-access right, so check whether your use case actually qualifies.
What should I log for each record in a hybrid pipeline?
Log the source, the access route (API or scrape), the collection timestamp and the terms or policy version that applied. That lets you audit a record and re-route a field later without losing lineage.