One Runner, Many Patterns: An AI-Driven Scraping Framework

When the data you sell is data you collect, the collection tier is not infrastructure. It is the supply chain. A large research estate typically gets there the same way: several independent collection platforms, each built by a different team in a different era, each a reasonable local decision at the time, and collectively a liability nobody chose.

This is about consolidating that onto one AI-driven collection framework, and about the analysis that came before the build. The interesting result was never the AI. It was finding that a fleet of many hundreds of scrapers is a small set of patterns wearing different configuration, and that once you know the patterns, most of the rest is engineering you can buy or generate.

Many platforms, many specialties, one job

Each business area had solved collection for itself, so the estate grew into parallel platforms with no shared services between them. Different languages. Different orchestration philosophies, from managed state machines to a message bus to a workflow scheduler to hand-rolled script runners. Pipeline shapes that did not agree on how many stages a pipeline has. Output formats and destinations that agreed on nothing at all.

The consequence is easy to state and expensive to live with: nothing transferred. Not code, not monitoring, not people. A fix on one platform needed one language and one message bus; the same class of fix elsewhere needed a different language and a different scheduler. Every platform paid its own maintenance bill for the same problems, and every hire, on-call rotation and tooling decision multiplied by the number of stacks.

Manual effort was the operating model

Underneath the fragmentation sat a deeper issue. A deterministic scraper assumes the source never changes, and modern publishing portals change constantly.

The break-fix loop never closed. Layout changes and format churn broke parsers continuously. Engineers hand-patched selectors, redeployed, and waited for the next change. A meaningful share of sources sat behind bot protection or needed a full browser to render at all.

Housekeeping shipped as code. Certificate renewals, password rotations and schedule changes all went out as deployments through each platform's own release process. Routine credential work consumed engineering capacity and carried release risk, for changes that should have been configuration.

Failures were found downstream. With almost no source-level observability, a silent break was usually noticed in a downstream table, a model output, or by a client. By then the bad data was already in the product. When your customer is your monitoring system, you have a commercial problem rather than an operational one.

The stakes were not back-office

These feeds price trades, track physical infrastructure, and land directly in client-facing products. Refresh cadences ran from sub-minute polling at one end to annual regulatory filings at the other, with everything in between. A late file is a stale price, and a stale price does not queue politely. It shows up on client screens.

The control gaps compounded it. Several distinct authentication patterns, from anonymous endpoints through API keys, OAuth, mutual TLS client certificates and mailbox logins, each managed ad hoc inside scraper code, separately on every platform. Run-to-output lineage existed only in pockets. And the inventories themselves disagreed: different systems of record reported different totals, and one platform's own breakdown did not reconcile with its own stated total.

That last point is the one I would put in front of an executive. When the inventory cannot be reconciled, auditability is not a policy question. The data you would need in order to audit does not exist.

The insight: a scraper is configuration, not a program

Before designing anything we inventoried a representative fleet completely, field by field. The question was not "how do we rewrite these" but "what are these, actually."

Every scraper turned out to be a combination of four things: a transport (how you reach it), a container (what comes back), a discovery rule (how you know which URL or file to fetch) and an auth mode (how you get in). Enumerate the distinct combinations that actually occur and you do not get hundreds. You get a short, closed list, and the long tail you expect to find simply is not there.

So a source stops being code and becomes a catalog entry:

name:             regional-balancing-error
pattern:          ftp-single-file
transport:
  host:           ftp.example-operator.org
  path:           /reports/balancing/ACE_current.csv
auth:
  mode:           anonymous
discovery:
  kind:           static_path         # publisher overwrites one file in place
  snapshot_on_pull: true              # so keep every pull, history survives
schedule:         rate(5 minutes)
container:        csv
schema_ref:       balancing-error@v3  # the contract, versioned separately
politeness:
  max_rps:        0.5
  respect_robots: true
owner:            grid-analytics

That entry attaches an approval gate, vault-managed credentials, run-level lineage and freshness monitoring automatically, because those are properties of the platform rather than of the scraper. The snapshot_on_pull flag is worth pausing on: the publisher overwrites a single static file on a schedule, so without it you have current state and no history. With it you have a time series, and not one line of bespoke code was written to get it.

The schema is the contract, and it deserves its own lifecycle

The commercial scraping platforms worked this out before most in-house teams did. Look at how the mature ones are built, whether that is Bright Data's studio, an actor-style marketplace or a managed extraction API, and the schema is a first-class object with its own editor, not a byproduct of whatever the parser happened to emit. We copied that deliberately.

A registered source points at a versioned schema rather than carrying its own field list. The schema editor lets an owner set types, nullability, units, formats and value ranges per field, and infers a first draft from a sample so the starting point is close. Three properties matter more than the editing experience.

Schema evolution is governed. Additive changes are allowed by default. Removing a field or changing a type mints a new version, and consumers pin to a version rather than tracking head. A publisher renaming a column becomes a versioning decision made by a named owner, not a silent break in a downstream table.

Extraction is mapped, not hardcoded. Each field records how it was obtained: which selector or strategy produced it, and with what confidence. That per-field provenance is what makes the difference between an alert saying a run failed and an alert saying one field stopped resolving while the rest stayed fine.

Drift is classified, not just detected. Every run infers a schema from what actually arrived and compares it against the contract. A new column, a missing column, a widened type and a changed format are four different events with four different responses. Only some of them should stop a pipeline. Treating them all as parse failures is why so much collection tooling cries wolf until people stop reading the alerts.

schema: balancing-error@v3
evolution: additive_only          # breaking changes mint v4
fields:
  - name: interval_start
    type: timestamp
    tz: source_local
    required: true
    extract: { strategy: css, selector: "td[data-col='ts']",
               fallbacks: [ xpath, text_anchor ] }
  - name: ace_mw
    type: decimal(12,3)
    unit: MW
    range: [-10000, 10000]        # a value outside this is quarantined
    required: true
    extract: { strategy: column_index, index: 2 }
  - name: revision
    type: int
    required: false               # added in v3, additive, no new version needed
on_drift:
  new_column:     warn_and_capture
  missing_column: fail_if_required
  type_widened:   warn
  type_narrowed:  fail

Where AI belongs, and where it must not go

This is the part most teams get backwards, so I want to be precise. AI sits in two places, and deliberately nowhere else.

At build time, generating the collector

Give the platform a sample page or response and it generates the extraction code, infers the schema and column types, analyses the document structure and proposes selectors. For browser-rendered sources there is a recording mode: drive the page once by hand and the session becomes a replayable script, which is the same authoring pattern the commercial studios use.

Two build-time techniques repaid the effort far beyond the obvious code generation.

API discovery from the network waterfall. Before generating a single selector, capture the page's network traffic and look at what the page itself calls. A large share of modern portals render from a JSON endpoint that is easier to call directly, more stable than the DOM, and often paginated and filterable in ways the visible page is not. The best scraper is frequently no scraper at all, and finding that out is an automated analysis rather than an act of inspiration. When it works, the source gets promoted from an HTML pattern to an API pattern and stops being fragile.

Semantic anchors rather than positional selectors. Generated selectors default to anchoring on stable text, labels and accessibility attributes rather than on position in the tree. A selector written as the third cell of the fourth row breaks the moment a column is inserted. A selector anchored on a header label survives it. This single generation constraint removed a large class of future breakage, and it costs nothing at build time.

At run time, on the exception path only

Run-time AI engages on failure, never by default. The important design detail is that self-healing is not a single attempt at repair. It is an escalation ladder, and each rung is more capable and more expensive than the last.

The scheduled hot path stays deterministic. This is the design decision I would defend hardest. If every scheduled pull invokes a model you have bought unpredictable cost and unpredictable output on a fleet running thousands of jobs at cadences down to a minute. Concentrating AI at build time and on the exception path means spend scales with breakage rather than with volume, and the daily behaviour of the fleet is the behaviour you tested.

The unblocking tier is a commodity, so buy it

A minority of sources sit behind bot protection, and that minority consumes a wildly disproportionate share of engineering attention if you try to fight it in-house. Anti-bot detection is an arms race between well-funded specialists, and it is not a race a data business creates value by entering.

The commercial platforms have converged on a small set of primitives here, and they are worth naming because the shape is now standard across vendors:

• A remote browser endpoint. A managed headless browser you attach to over the standard debugging protocol, with fingerprint management, challenge solving and retries handled on their side. Your code drives a normal browser session; someone else runs the browser farm and keeps the fingerprints current.
• A request-level unblocking API. You send a URL and get HTML back. Underneath it handles TLS and HTTP/2 fingerprints, header ordering, challenge pages and retry logic. Priced per successful response, which aligns the vendor with the outcome rather than the attempt.
• Tiered proxy pools. Datacenter, ISP, residential and mobile, with geo-targeting and session persistence. Session stickiness matters more than people expect: any source that builds state across a multi-step flow needs the same exit address for the whole sequence.
• Dataset products. For some targets the vendor already collects what you need and sells the feed. Buying the record is cheaper and more stable than collecting it, and the only reason teams miss this is that nobody checks.

The framework treats all of these as pluggable transports behind one interface, so a source can move between direct fetch, unblocking API and remote browser by changing configuration rather than code. That also keeps the commercial position honest, because switching providers is a config change rather than a migration.

What no vendor sells is the part that makes an estate like this hard: file transfer sessions, mailbox-sourced feeds, certificate-bound portals, embargo-aware backfills, and landing into your own governed storage with lineage. Buy the commodity, build the differentiator, and be clear-eyed about which is which.

Politeness is a platform control, not a per-scraper habit

Every source carries a rate policy, and the runner enforces it centrally with a per-domain token bucket rather than trusting each collector to behave. Crawl-delay directives are honoured. Repeated rejection responses trigger adaptive backoff and reduce the domain's budget for a cooling-off window rather than retrying into a wall.

This is partly courtesy and mostly self-interest. Aggressive polling is how a public endpoint that has served you for years decides to start blocking you, and recovering that relationship costs far more than the data you gained by hammering it.

The architecture

Four details in there carry more weight than a line each.

The API mirrors the UI exactly. A person registering one source and a migration script registering hundreds use the same contract. This sounds like a minor symmetry and it is the reason a large migration is feasible at all: you are not maintaining a bulk path that drifts from the interactive one.

Retries end somewhere visible. Bounded retries with backoff, then a dead letter queue. That queue is what converts a silent failure into an owned, visible event, which was the single largest operational gap we set out to close.

The raw zone is immutable and versioned. Whatever the publisher sent is preserved exactly as sent, partitioned per run. Every reprocessing question later, and there are always reprocessing questions, is answerable without going back to the source.

The output path is deterministic. Every run lands at a predictable partitioned path carrying its run and schema identifiers, and the record of config version, inputs and outputs is written as a by-product. Lineage is not a feature someone remembered to add. It is the shape of the write.

Validation with an agentic browser

Here is the failure mode that keeps me up at night on a self-healing system. A parser breaks, the platform repairs it automatically, and the repaired selector now points at the adjacent column. Collection succeeds. Monitoring is green. Wrong numbers flow into the product with full lineage and a clean audit trail, which makes them more credible, not less.

Schema and range checks catch some of that, and they do not catch the case where the wrong value is a plausible value. What does catch it is looking at the page the way a person would.

Agentic browser models are the useful new capability here. Something like Amazon Nova Act takes a natural-language instruction and performs it reliably in a real browser, broken into small verifiable steps. We use that in three places, none of which is the scheduled hot path:

• Post-collection visual validation. After a repair, and on a sampled basis otherwise, the agent opens the live page and is asked to find the values we believe we extracted. If the agent cannot see them where we say they are, the run is quarantined rather than published. This is the direct counter to silent repair drift, and it is the reason I would fund the capability at all.
• Multi-step and hostile flows. Sources requiring a login, a date picker, a form submission and a download are not expressible as a declarative config. An agentic browser handles them as a sequence of instructions, which is far more durable than a recorded click script that breaks when a button moves.
• Change triage. When a source breaks in a way the ladder cannot repair, the agent drives the page and reports what changed in plain language. The owner gets an explanation instead of a stack trace, which is the difference between a ticket that gets picked up and one that ages.

The cost discipline is the same as everywhere else: this runs on repair, on a sample, and on the sources that need it, not on every pull.

Governance as the default path

The operating model change matters more than any single component. Onboarding moves from an engineering ticket to a catalog entry with an approval gate. Credential rotation moves from a code change and deployment to a policy-managed operation. Failure detection moves from downstream discovery to source-level freshness, schema and value checks that alert a named owner. Ownership moves from tribal knowledge to a recorded field.

The point is not that these controls exist. It is that the governed path is the only path. Approval, managed credentials, run lineage and monitoring apply because a source was registered, not because someone remembered to configure them. The audit trail is created as a by-product of doing the work, which is the only kind that stays current.

Growing the same spine agentic

The framework has four stages: scrape, extract, curate, vectorize. The extensions that make it agentic are deliberately built as additional step types in the same pipeline executor, recorded in the same run lineage, rather than as a second platform alongside the first.

• Scrape. Agentic browser sessions for validation and for the hostile tier, as above.
• Extract. Semantic partitioning for the PDF and HTML already landing in the raw zone, preserving titles, narrative and tables as structured output rather than flattening a document to text.
• Curate. Confidence scoring on extracted output. High-confidence records flow straight through, low-confidence records route to human review before publication.
• Vectorize. Embeddings built per run and served from a vector store alongside the warehouse, so collected content is retrievable for language model workloads rather than only queryable as tables.

That last stage is what connects this to the rest of the AI platform, because it turns a collection tier into a retrieval corpus.

Registering it as a custom workflow extension

Rather than wiring a bespoke integration, the collection framework registers on the central AI platform as a workflow extension, the same way the model estate does. It appears to agents through the same gateway and inherits identity, entitlements, tracing, evaluation and human review rather than reimplementing any of them.

list_sources(filter)
  -> [ { name, pattern, owner, cadence, last_run, health } ]

describe_source(name)
  -> { pattern, schema_ref, auth_mode, cadence, rate_policy,
       destination, owner, required_entitlement }

trigger_collection(name, window)              # async, off-cadence pull
  -> { run_id, accepted_at }

get_run_status(run_id)
  -> { state, records, bytes, drift_events, diagnosis }

get_run_artifacts(run_id, projection)         # reference, not payload
  -> { raw_uri, curated_uri, schema_version, row_count, confidence }

search_collected(query, filters)              # over the vectorized layer
  -> [ { chunk, source, run_id, published_at, score } ]

propose_source(url_or_sample)                 # drafts, never activates
  -> { suggested_pattern, draft_config, inferred_schema,
       discovered_api, policy_check,
       status: "pending_human_approval" }

The last tool is the interesting one and needs the most discipline around it.

An agent given a URL can do the work that used to take an engineer an afternoon: fetch a sample, check whether a cleaner API sits behind the page, infer which pattern fits, generate the extraction config and draft schema, and submit it. What it cannot do is activate it. The draft lands in the same approval queue a human-authored source lands in, and a person assigns the owner and signs it off.

That separation is not bureaucratic caution. Rate limits, terms of service and robots directives are commitments your business is making, and they belong in platform policy enforced at the extension boundary, not in an agent's judgment about whether a site seems fine to scrape. An agent that can talk itself past a rate policy is a liability with a nice interface.

Evaluating the extension

Because it is registered on the platform, the extension inherits the evaluation system, and the golden set for collection is specific: saved page fixtures with known correct extraction, replayed in continuous integration so a change to the generator is regression-tested without touching the live source. Field-level accuracy is the headline score. Two checks matter more than the headline.

First, that generated extraction produces the expected fields rather than plausible-looking wrong ones. Second, that self-healing has not silently changed a schema. A parser that repairs itself into the wrong column publishes; a broken one alerts. The repaired one is the more dangerous outcome, and it is the reason the agentic validation step earns its cost.

What it changes

Onboarding goes from weeks of engineering backlog to days through a catalog. Detection moves from the client to the source owner. Credential rotation stops requiring a release. Several sets of tooling, on-call rotations and tribal knowledge collapse toward one. And engineers stop spending their week on selector patches.

One thing I would insist on in any programme like this: decommission is the deliverable. Each phase has to end with legacy scope switched off, not with a demo of the new thing. The savings and the risk reduction land only when the old stack stops running, and a consolidation programme that never retires anything has simply added one more platform to the pile.

The closing thought is the same one I reached on the modelling side. The AI is the smallest part of this. What made the estate tractable was the boring inventory work that revealed a short pattern library underneath a large fleet. AI then made the long tail cheap to build and made breakages partly self-correcting, and an agentic browser made silent corruption detectable, all of which are real gains worth having. But if you start with the AI you end up with a very capable generator producing collectors into the same ungoverned mess, and the thing you actually needed, which is knowing what you collect, who owns it, and whether it is right, is still missing.