A Third of New Web Pages Are Written With AI: Filtering Synthetic Content Out of Scraped Datasets

About a third of the new, dated pages on the web now show signs of AI authorship. If your pipeline scrapes text and treats it as human-written, that assumption is already shaky. You can’t remove synthetic content perfectly, but you can measure it, label it per record and let the people using the data decide what to do with it.

How much of the web is AI-written, and how sure can we be?

Pew Research Center’s data essay of 20 August 2026, How Much of the Internet Is Written With AI?, estimates that 35% of pages in the July 2026 Common Crawl snapshot with post-ChatGPT publication dates show signs of AI authorship. Pew sampled 10,000 English-language pages from each of 49 crawls between January 2021 and July 2026, 490,000 pages in total. Pew says the number is broadly consistent with a separate 2026 study by Dolezal et al., which found that 35% of newly published websites contained AI-generated or AI-assisted text.

Two caveats from the same essay matter for anyone building on this number.

First, only about 10-15% of pages include a publication date, so the estimate covers dated content. Pew also notes that paywalled and login-only sites are probably underrepresented. Second, the detector counts a page as having "meaningful" signs of AI authorship or editing at a score of 0.2 or higher. That is a threshold someone chose, and a different choice gives a different headline number.

So read "35%" as: a large and growing share, measured one particular way. Not as a precise fact about your own data.

What does a detector score actually tell you?

Pew scored each page’s body text from 0 to 1 with an open model from Pangram (editlens_Llama-3.2-3B). It then checked the open model against Pangram’s commercial model 3.3 on 62,370 pages from seven crawls. They agreed 96% of the time, with a Cohen’s kappa of 0.61.

Both numbers are worth sitting with. 96% agreement sounds great, but kappa corrects for chance agreement, and 0.61 is usually described as substantial rather than near-perfect. In other words, two competent detectors still disagree on a meaningful number of pages.

The essay also reports that the open model likely produced false positives on the 2021 and 2022 crawls, with shares around 1%, at a time when there was no ChatGPT to blame. That 1% is a useful floor: any detector you run will flag some human text.

The practical reading: a score is a probability-like signal, and your threshold is a policy decision. Write it down.

Where should the filter sit in the pipeline?

Put detection after extraction and before anything that treats text as a unit of evidence. Boilerplate, navigation and cookie banners will confuse a detector, so score the cleaned main-content text, not raw HTML.

A minimal shape, assuming you wrap whichever detector you use behind a score_ai(text) function that returns a number from 0 to 1:

from dataclasses import dataclass
DETECTOR = "pangram-open"        # record exactly what you ran
DETECTOR_VERSION = "2026-10"
MIN_CHARS = 400                  # short text gives unstable scores
@dataclass
class Record:
url: str
text: str
ai_score: float | None = None
ai_detector: str | None = None
ai_scored_at: str | None = None
def annotate(rec: Record, now: str) -> Record:
if len(rec.text) < MIN_CHARS:
return rec               # leave unscored, don't guess
rec.ai_score = score_ai(rec.text)
rec.ai_detector = f"{DETECTOR}@{DETECTOR_VERSION}"
rec.ai_scored_at = now
return rec

Three design choices are in there on purpose. Short texts stay unscored rather than getting a made-up number. The detector name and version travel with the score, so you can re-score later. And nothing is deleted. Filtering becomes a query (WHERE ai_score < 0.2 OR ai_score IS NULL) instead of a one-way door.

How do you choose a threshold without fooling yourself?

Don’t borrow 0.2 because Pew used it. Pew used it to describe a trend across the web; you are trying to protect a specific use case.

A workable approach:

1. Pull a sample of a few hundred records per source type, stratified by site, not pooled.
2. Have a person label them, or at least a subset, as human, edited or generated. It’s tedious, and it’s the only ground truth you will have.
3. Run two detectors if you can, and look at where they disagree. Disagreement clusters tell you which source types the filter can’t be trusted on.
4. Choose thresholds per use case: strict for training text, loose for structured extraction.
5. Re-run the sample every quarter. Detectors and generators both move.

This is the same discipline we described in Measuring AI Search Answers Without Fooling Yourself: sample deliberately, fix your settings, and keep an evidence trail for every number you report.

Why does the source mix change the answer?

A single global filter rate hides the interesting part. A product catalogue is mostly structured fields and short descriptions, and a news archive is long prose. A forum has human quirks that detectors usually score as human. Review sites and affiliate content are where you would expect the highest synthetic share, though that is a hypothesis you should test on your own sources rather than a finding from the research above.

This is why per-source reporting beats a dataset-wide percentage. If one domain contributes 40% of your records and most of your flagged ones, that’s a sourcing decision, not a filtering problem.

It also interacts with deduplication. Templated, lightly varied generated pages tend to look like near-duplicates, so the clustering step in incremental crawling will catch some of them before a detector ever sees them. Run both and compare what each one catches.

What should you tell the people who use the data?

Ship the score, the detector, the threshold used for any filtered release and the date. Say plainly what the filter can’t do. A buyer of a training corpus or a sentiment dataset needs to know that "filtered" means "scored at or below X by detector Y on date Z", not "human-written".

If some of your inputs come from official feeds rather than scraping, the problem shrinks for those fields, which is one more argument for the routing logic in Scrape, API or Both?.

Where this leaves you

Synthetic text isn’t going away, and no detector will give you a clean binary. What you can control is whether the uncertainty is visible: scores stored per record, thresholds chosen per use case, samples checked by a person, and sources reported separately.

If your use case needs to know where each record came from and when it was collected, that is how Dataizi datasets are delivered, with the source list, collection dates and legal basis documented. See Dataizi datasets or talk to us.

Sources

• Pew Research Center, "How Much of the Internet Is Written With AI?", 20 August 2026: https://www.pewresearch.org/?p=356393
• Allaham and Diakopoulos, "Synthetic Sources?: Auditing Generative Search Engine Citations for Evidence of AI-Generated Sources", arXiv 2605.23684 (I could only read the title and authorship from the PDF, so no findings are cited from it; relevant background on AI-generated sources in generative search): https://arxiv.org/pdf/2605.23684