Dataizi — web crawling & data extraction
Get started for free

Incremental Crawling Done Right: Change Detection, Deduplication and Entity Resolution for Scraped Data at Scale

Most scraping jobs are wasteful in the same way. Every night they fetch a few million pages, and on a typical day only a small fraction of those pages differ from yesterday’s copy. You pay for the proxies, the browsers and the bandwidth on all of them, and you take on the block risk that comes with all that traffic.

Incremental crawling is the obvious fix, and it’s also easy to get subtly wrong. This post walks through the three pieces that make it work: deciding what changed, keeping duplicates out, and resolving records that describe the same real-world thing. It also covers where each piece tends to fail.

What incremental crawling buys you, and what it costs

Incremental scraping means collecting only data that is new or changed since the last run, instead of re-collecting everything. Browse AI’s glossary lists the benefits you’d expect: less bandwidth and processing, faster refreshes, lower proxy and cloud spend, and fewer requests to a target site, which makes you less likely to be flagged by anti-bot systems.

The same page is honest about the costs, and they’re worth reading as a checklist:

That last point is the one that bites in production. Incremental crawling moves your risk from "too expensive" to "quietly wrong", so the rest of the design should assume the change signal will sometimes lie.

Change detection: don’t trust the site’s timestamps

If a site gives you a reliable Last-Modified header, an ETag or a sitemap lastmod, use them first. They’re free. But treat them as a hint, not a fact, for the reason above: some sites update them for trivial edits, and others don’t have them at all.

The dependable fallback is to hash the part of the record you care about. Hash the extracted fields, not the raw HTML. Raw HTML changes every time a session token, a rotating ad slot or a "viewed 14 times" counter changes, and you’ll re-process pages that haven’t meaningfully changed.

import hashlib, json TRACKED = ("title", "price", "currency", "availability", "seller") def content_hash(record: dict) -> str: # Hash only the fields that matter, in a stable order. payload = {k: record.get(k) for k in TRACKED} blob = json.dumps(payload, sort_keys=True, ensure_ascii=False) return hashlib.sha256(blob.encode("utf-8")).hexdigest() def classify(record: dict, state: dict) -> str: key = record["canonical_id"] h = content_hash(record) if key not in state: return "new" return "unchanged" if state[key]["hash"] == h else "changed"

Browse AI points out a gap in the simple version of this: scraping only listing pages efficiently catches additions and removals, but not changes inside existing items. Their suggested fix is to re-scrape only records whose listing-level fields differ from what you’ve stored. In practice that means a two-tier crawl. A cheap listing crawl runs often, and a deeper detail crawl runs only for records the listing flagged, plus a slow background sample to catch drift the listing never shows.

For deletions, don’t delete on the first miss. Mark a record missing and only retire it after it’s been absent for N consecutive crawls. The right N depends on how flaky the source is, so tune it per site.

Deduplication: exact matching isn’t enough

Duplicates arrive from several directions. ScrapeHero’s guide to cleaning scraped data lists the usual ones: URLs that change between crawls, products listed in multiple categories, pagination repeats and session parameters. It says duplicate rates in retail monitoring "frequently exceed 25%" when marketplaces syndicate identical listings. That figure comes from a vendor with no primary source given, so read it as "can be large" rather than as a benchmark.

The article also argues that exact-match deduplication is no longer sufficient, and suggests hash fingerprints, similarity scoring, SKU matching, fuzzy title comparison, image similarity and canonical URL mapping. I’d order those by cost:

  1. Canonicalize URLs first. Strip session and tracking parameters, normalize case and trailing slashes, and follow rel=canonical where it’s trustworthy. This alone removes a lot of noise and costs almost nothing.
  2. Match on strong identifiers. SKU, GTIN, a listing ID from the source. If you have one, use it.
  3. Fall back to fuzzy matching on normalized title, brand and key attributes, but only for records that survive steps 1 and 2.

The same article makes a point that’s easy to skip: standardize before you load. Currencies, date formats, units, encodings and null representations all need one convention, or you’ll compare "USD 129" with "129 dollars" and conclude they’re different prices. Do this at ingestion, before the data reaches the warehouse, using whatever you already run (dbt and Spark are the examples they name).

Entity resolution: when two records are the same thing

Deduplication removes copies of the same record. Entity resolution answers a harder question: are these two different records about the same product, company or listing?

Take two marketplaces selling what looks like the same item. One calls it "Acme TrailRunner 2 Men’s 10", the other "TrailRunner II – Acme – US10". No shared SKU, different titles, different images. A reasonable approach:

I’d push hard on that last item. Teams that overwrite source records with the merged result can’t undo a bad merge. Teams that store links can.

Putting it together, and testing it

A workable pipeline looks like this: a cheap listing crawl feeds a change classifier; only new and changed records go to the detail crawl; extracted fields get standardized; canonicalization and dedup run next; then entity resolution assigns or updates the entity ID; and only validated records are published. ScrapeHero recommends validating schema, ranges and freshness and quarantining bad records instead of publishing them. Incremental ingestion into the warehouse also reduces compute cost and helps keep pipelines stable.

Because the failure modes here are silent, test for them directly:

What to take away

Incremental crawling saves real money, but only if the change signal, the identity scheme and the matching logic are treated as first-class parts of the pipeline. Hash extracted fields instead of HTML, canonicalize before you fuzzy-match, retire deletions slowly, and store links between source records and entities so you can fix mistakes later.

One limit on all of this: the numbers in the public sources are mostly qualitative or vendor-supplied. Measure duplicate rate and change rate on your own targets before you size a project around anyone’s claimed percentages.

Sources