One Version of the Truth: Building the Governed Data Foundation Behind Analytics and AI

A data platform is judged by one question: when two people ask for the same number, do they get the same answer? In a business that sells research and data, that question carries revenue, because the data is the product. Now a third party is asking, an AI agent, and it will answer confidently whether or not the data underneath agrees. This is the design of the governed foundation that our analysts, dashboards, customers and agents all read from: Snowflake as the lakehouse on a medallion architecture, dbt for transformation, master data and an ontology for meaning, DataHub for metadata and lineage, and a retrieval pipeline that turns the same governed content into something language models can use safely.

Three answers to one question

The starting point will be familiar to anyone who has run data at scale. The same metric came out differently depending on which team you asked, because each team had its own extract, its own join logic and its own idea of what a region or a customer was. The same company appeared under many spellings across research systems, sales systems and collected filings. Nobody could say with confidence which tables held personal data, which content was licensed from third parties and could not be redistributed, or what would break if a column changed.

None of that is new. What raised the stakes was AI. An analyst who finds two revenue tables asks which one is right. An agent picks one. A retrieval system built on a raw document dump returns last year's report and content the user was never entitled to see. The lesson from the AI-DLC rollout applies here too: AI amplifies the system it lands in. The data foundation had to be right before the agents on top of it could be trusted.

Principles, and the shape of the platform

Six principles settled most of the arguments that came later. Land data raw and keep it, because you cannot reprocess what you threw away. Transform inside the warehouse, as code, reviewed like code. Put a contract at every boundary, between producers and Bronze and between Gold and its consumers. Treat metadata as a product with owners, not as documentation someone writes later. Give every business entity exactly one identity. And make every workload pay for its own compute, so cost follows usage instead of hiding in a shared bill.

Snowflake is the lakehouse for practical reasons. Separating storage from compute let every workload run on its own warehouse. Zero-copy cloning made development and CI environments nearly free. And secure sharing let customers query data products in place instead of receiving files.

Getting data in without losing it

Five ingestion paths cover nearly every source, and each one is a template rather than a project.

• Operational databases use log-based change data capture, so inserts, updates and deletes arrive as events and history survives.
• Files and SaaS extracts land in object storage and load through external stages, with Snowpipe picking them up on arrival.
• Event streams flow from Kafka through Snowpipe Streaming, row by row with seconds of latency and no staging files.
• Documents, such as reports, PDFs, slide decks and spreadsheets, are parsed with Unstructured.io. The original files stay in storage, and the parsed elements land in Bronze with their page and section structure intact, feeding both Silver tables and the retrieval pipeline.
• Collected data from the scraping framework arrives with more context than any other source, because every run is already a governed object with a source definition, pattern, schema version and run ID. Raw payloads land in Bronze keyed by run ID, curated records enter Silver against their registered schema, and the run ID becomes the first link in the lineage chain. Any number in a Gold table can be traced back to the exact pull that produced it.

Every path lands the same way. Bronze stores what arrived, as it arrived, with semi-structured payloads kept in VARIANT columns rather than flattened early. Each row carries load metadata: source system, batch or run ID, load time and a content hash. Loads are idempotent, so a rerun replaces rather than duplicates. Schema drift is tolerated in Bronze and caught in Silver, where a contract either absorbs the change or fails loudly. A source that changes shape should never be able to corrupt anything downstream without someone knowing.

The medallion, with a contract per layer

Bronze, Silver and Gold are only useful if each layer means something specific. We wrote down what each one guarantees, and enforced it.

Physically, each layer is its own database with a schema per domain inside it, which keeps access rules simple. Workloads run on separate warehouses for ingestion, transformation, BI, ad hoc analysis and AI, each with auto-suspend and a resource monitor, so a heavy notebook cannot slow down a customer dashboard. Development and CI environments are zero-copy clones of production, created in seconds and paid for only in changed storage. Time Travel turns recovery from a bad load into a query, not a restore project.

Silver is where most of the real work happens, and conformance is the core of it. A Silver table does not just clean a source. It maps every company, asset and product to its master data ID, and every code to the shared ontology, so two sources describing the same thing become joinable without anyone writing a one-off mapping. Gold builds data products on conformed Silver, never on Bronze and never on another team's private tables.

dbt as the only way data moves

Every transformation between layers is a dbt model. No hand-run scripts, no stored procedures nobody reviews, no BI-tool calculations that quietly redefine a metric. dbt's conventions map cleanly onto the layers: staging models read Bronze and produce typed, renamed inputs, intermediate models do the conforming and joining in Silver, and marts are Gold.

A handful of practices did most of the work. Large tables build incrementally with merge, keyed on stable IDs, so a daily run touches only what changed. Snapshots capture history where a source overwrites records in place. Tests run at every layer: the built-in uniqueness, not-null, accepted-values and relationship tests, the dbt-expectations package for ranges and distributions, and singular tests for rules only one model needs. Gold models carry enforced contracts and versions, so a breaking change ships as a new version alongside the old one with a deprecation date. Model access is explicit, so private models stay inside their domain and only public Gold models can be referenced across projects. Exposures register the dashboards, shares and agents that read each model, which is what makes impact analysis complete.

models:
  - name: company_profile
    config:
      contract: { enforced: true }   # types and constraints checked at build
      access: public                 # other projects may reference it
      group: company_data
      meta:
        tier: 1                      # sets the rules in CI and on-call
        owner: company-data-team
        freshness_sla_hours: 24
    latest_version: 2
    versions:
      - v: 2
      - v: 1
        deprecation_date: 2026-12-31 # consumers get a window to move
    columns:
      - name: company_id
        data_type: varchar
        description: Golden record ID from master data
        constraints: [{ type: not_null }, { type: primary_key }]
        data_tests: [unique]
      - name: primary_contact_email
        data_type: varchar
        meta: { sensitivity: pii_direct }   # synced to a Snowflake tag

In CI, a pull request builds only the models it changed and their dependants into a zero-copy clone, deferring to production state for everything else, then runs tests, contract checks, SQL linting and tier rules. DataHub posts the downstream impact as a comment on the pull request, so a reviewer sees which dashboards, shares and data products a change touches before approving it. In production, each run builds into an audit schema, runs the full test suite there, and swaps into the live schema only if everything passes. A failed run pages the owner instead of publishing a broken number and finding out from a customer.

Master data and ontology: one ID, one meaning

Conformance needs two things that are easy to confuse. Master data decides which specific things exist: that three records spelled differently in the sales system, the research database and a scraped filing are one company, with one ID. The ontology decides what kinds of things exist and how they relate: a company owns assets, an asset is located in a region and produces a commodity, and one company can be the parent of another. Master data supplies the instances. The ontology supplies the types and the relationships. You need both, and they have to agree.

Master data: match, merge, steward

We chose a consolidation hub, built in Snowflake with dbt and Snowpark Python, over a registry that only links IDs or a coexistence model that writes golden records back into every source. Consolidation gave one place to build golden records without forcing source systems to change on our timeline, and it kept master data inside the same lineage, tests and access controls as everything else.

The pipeline runs in Silver. It standardizes names, addresses, identifiers and legal suffixes, then blocks records into candidate groups so matching compares likely pairs rather than everything against everything. Matching runs in three passes: exact matches on strong identifiers such as registration numbers, fuzzy matching on names and locations, and a model that scores what is left. Survivorship rules then build the golden record attribute by attribute, by source priority, recency and completeness. Each golden record gets a stable ID that is never reissued, and a cross-reference table maps every source ID to it. Hierarchies such as parent and subsidiary are relationships between golden records, not fields on them.

Thresholds send the uncertain middle to people. High-confidence matches merge automatically, low scores stay separate, and everything in between goes to a stewardship queue with the evidence for and against. Steward decisions are recorded and fed back as training data for the matching model, and because the cross-reference keeps every source record intact, any merge can be undone.

Ontology: the shared vocabulary

The ontology lives in the repository as a versioned definition of entity types, attributes and relationships, reviewed like code. From there it is published three ways. Into DataHub as the business glossary, so a column tagged as a commodity means the same thing in every dataset. Into Silver as conformed reference tables and code mappings, so each source's vocabulary resolves to shared terms. And into the knowledge graph, where golden record IDs become nodes and ontology relationships become edges. That is what lets an agent answer which subsidiaries of a company operate assets in a region without a seven-way join. Ontology changes follow the same versioning rules as Gold contracts, because renaming a relationship is a breaking change for everything that reasons over it.

Metadata and lineage: DataHub as the control plane

A catalog that people update by hand is out of date within a quarter. DataHub works here because nearly everything in it is collected automatically, and because it drives behavior rather than just answering searches.

What flows in: Snowflake schemas, usage and query history, from which DataHub derives column-level lineage; dbt manifests, run results and test results, which become documentation, lineage and assertions; orchestration runs from Airflow as OpenLineage events; dashboards and their fields from the BI tool; Kafka topics and schemas; and the scraper's run lineage. Ownership, domains, data products, tiers and glossary terms are declared in the repository and arrive through the same ingestion, so the source of truth for metadata is code, not a form.

Lineage is the part that pays for itself. It runs end to end at column level, from a scraper run or source table, through Bronze, Silver and Gold, to the dashboards, customer shares and RAG indexes that read it. That chain answers four questions that used to take days. What breaks if I change this column? Where did this number come from? Which downstream assets inherit this sensitivity tag? And which tables, shares and indexes must change when a record is deleted or a license expires?

Two more uses turned DataHub from a catalog into a control plane. The first is actions: tag and glossary changes trigger automation, so a column classified as personal data in DataHub is tagged in Snowflake and picked up by the masking policy without a ticket. The second is agents. The AI platform reaches DataHub through its MCP server, so before an agent writes SQL it looks up which Gold table is certified for the question, what the columns mean and who owns it. An agent that can read the catalog stops guessing between two similar tables.

Tiers: how much an asset matters

Layers describe how refined data is. They say nothing about how much it matters. A Gold table behind a customer data product and a Gold table behind one analyst's exploration need very different care, so every asset also carries a tier, declared in dbt metadata and visible in DataHub.

Tiers are enforced, not suggested. CI rejects a Tier 1 model without an owner, an enforced contract or tests on its keys. A lineage check blocks any Tier 1 asset from depending on Tier 3 upstream, because a critical product built on an unowned sandbox table is an incident waiting for a date. Agents and the RAG pipeline read Tier 1 and certified Tier 2 sources only. And tiers set the incident rules, so a Tier 1 freshness breach pages someone while a Tier 3 failure waits for the morning.

Sensitive data: classify, tag, enforce

Sensitivity is the third independent label. One tag taxonomy covers every asset: public, internal, confidential, personal data split into direct and indirect identifiers, and licensed third-party content with redistribution limits. That last category matters as much as personal data in a business that sells data, because redistributing licensed content is a contract breach even when no individual is involved.

Classification starts automatically. Snowflake's built-in classification and DataHub's classifiers scan new Bronze and Silver columns and propose tags, and an owner confirms them. Confirmed tags then propagate downstream through lineage, so a Gold column derived from a personal-data column inherits its tag unless the owner records why the transformation removed the sensitivity, for example by aggregation.

create tag governance.tags.sensitivity
  allowed_values 'public', 'internal', 'confidential',
                 'pii_direct', 'pii_indirect', 'licensed';

create masking policy governance.policies.mask_sensitive
  as (val string) returns string ->
  case
    when system$get_tag_on_current_column('governance.tags.sensitivity')
         in ('public', 'internal')        then val
    when is_role_in_session('SENSITIVE_READER') then val
    else 'MASKED'
  end;

-- every column that carries the tag is masked from the moment it is tagged
alter tag governance.tags.sensitivity
  set masking policy governance.policies.mask_sensitive;

Because the policy hangs off the tag, a newly tagged column is protected the moment the tag lands, with no policy change and no deploy. Row access policies cover the other dimension, limiting licensed datasets and customer-specific data to entitled roles. Roles follow an access role and functional role pattern managed in Terraform, so who can see what is reviewable in a pull request, and Snowflake's access history gives the audit trail of who read which columns. Deletion requests run through lineage too: the platform finds every table, snapshot, share and index holding the subject's records, and a controlled purge job is the one permitted exception to append-only Bronze. Time Travel retention on those tables is set so deleted data does not outlive the policy.

Vectorization: RAG on governed data

The retrieval system in the AI platform reads from this foundation, and that is the point. Most enterprise RAG failures are data failures: stale documents, duplicate versions, content the user is not entitled to, and chunks with no idea what they are about. Building the index from governed layers fixed most of them before any model was involved.

Indexing practices

Index governed content only. Sources are Silver and Gold: parsed documents, curated content from the scraping framework, and Gold records rendered as text, each with a known owner, tier and sensitivity. Nothing is embedded straight from a file share.

Parse for structure first. Layout-aware parsing keeps headings, tables and page boundaries, because most retrieval failures start as parsing failures. Tables stay whole or become structured rows instead of being cut through the middle.

Chunk by structure, not by character count. Chunks follow sections and paragraphs, sized to a few hundred tokens with a small overlap. A parent and child design retrieves on small chunks for precision and returns the surrounding section for context.

Add context before embedding. Each chunk is prefixed with its document title and section path, so a paragraph that says output rose eight percent still knows what it is about. It then carries metadata: master data IDs for every entity it mentions, ontology terms, sensitivity, license, entitlements, tier, document version and the lineage run ID.

chunk_id:        doc-4471#s3.2#c2
parent_id:       doc-4471#s3.2            # the section returned for context
context_header:  "Q3 Outlook > Supply > Regional capacity"
entity_ids:      [CO-000123, AS-004512]   # golden record IDs, joinable to Gold
ontology_terms:  [Company, Asset, Supply outlook]
sensitivity:     internal
license:         redistribution_restricted
entitlements:    [research_subscribers]   # filtered inside the search
tier:            1
doc_version:     2026-09-30
embedding_model: text-embed@v3            # a model change builds a new index
content_hash:    9f2c41ab                 # unchanged chunks are not re-embedded
lineage_run_id:  r-8841

Version the embedding model. Every vector records the model that produced it. A model change builds a new index alongside the old one, evaluates it, and switches an alias: write-audit-publish applied to vectors.

Update incrementally and delete faithfully. The content hash means only changed chunks are re-embedded. Deletions and license expiries propagate through lineage into the index, so a removed document stops being retrievable, not just stops being updated.

Retrieval practices

Retrieval is hybrid: dense vectors for meaning and lexical search for exact names, codes and identifiers, combined and then reranked by a cross-encoder. Entitlement filters run inside the search, matching the user's roles against each chunk's entitlements, so restricted content is never retrieved in the first place. Master data IDs let the retriever expand a question through the knowledge graph, so a question about a company also finds content about its subsidiaries and their assets. Answers cite their chunks, and lineage links each citation back to the original document and version. Retrieval is evaluated continuously against a reference set of questions with known answers, tracking recall, precision after reranking and groundedness, so a parsing change or a new embedding model is measured before it ships.

Asking questions of the data

People ask questions in different ways, so the platform offers several paths. All of them read governed Gold under the same access controls.

• SQL and notebooks for analysts and data scientists, on their own warehouses with query tags and timeouts.
• A semantic layer for metrics. Every business metric is defined once in dbt's semantic layer, with its measures, dimensions and filters. Dashboards, spreadsheets and agents query metrics through it, so revenue means one thing everywhere.
• Natural language to SQL through an agent on the AI platform. It grounds itself in the semantic layer first and DataHub second, prefers certified metrics over raw tables, runs read-only against Tier 1 and certified Tier 2 Gold, shows the SQL it ran, and has cost and time limits. It queries as the user, not as a service account, so row access and masking still apply. A reference set of questions runs on every change to prompts, models or the semantic layer.
• Graph queries on the knowledge graph for relationship-heavy questions.
• Document questions through the RAG pipeline above.
• Customers and applications through secure sharing and private listings, so customers query live Gold data products in their own accounts without copies or file deliveries, and through cached data APIs where an application needs low latency.

Performance tuning stays deliberately boring: clustering keys on the largest Gold tables, search optimization for selective lookups, dynamic tables for aggregates that must stay fresh without a schedule, and the result cache for repeated dashboard queries.

Quality, cost and ownership

Quality is monitored where consumers feel it: freshness SLAs on every Tier 1 and Tier 2 product, volume and distribution checks on Silver, and data contracts agreed with the teams that produce source data, so a breaking upstream change is caught in their pipeline instead of ours. Incidents route to the owner recorded in DataHub.

Cost is visible per team and per data product. dbt sets a query tag on every model, warehouses are split by workload with auto-suspend and resource monitors, and a monthly view of cost per data product turns the conversation from the size of the bill to the value of each product.

Ownership follows a simple split. The platform team owns the paved road: ingestion templates, dbt project scaffolding, CI, DataHub, the security model and the RAG pipeline. Domain teams own their Silver conformance and their Gold data products, each with a named owner, a tier and a contract. Domains own meaning. The platform owns the mechanics.

What changed

The argument about which number is right mostly stopped, because there is one certified Gold product per question and the semantic layer defines each metric once. Onboarding a source became a template and a contract instead of a project. A change that once would have broken a customer feed now shows its impact on the pull request before it merges. Personal and licensed data is masked by tag, audited by default and deletable end to end. And the agents and retrieval system became more reliable without a single prompt change, because they finally read data that knew what it was, who owned it and who was allowed to see it.

What I would tell another team

Keep layers, tiers and tags separate. Refinement, criticality and sensitivity are different questions, and a platform that collapses them into one label cannot answer any of them well.

Make master data and the ontology part of Silver, not a side project. Conformance is where trust is created, and everything downstream, from dashboards to agents, inherits it.

Collect metadata automatically, then make it do things. A catalog nobody updates is a museum. A catalog that drives masking, blocks risky dependencies and grounds agents is infrastructure.

Build RAG on the governed layers, never beside them. Most retrieval problems are data problems with a model on top.

And publish with write-audit-publish everywhere, for tables and vector indexes alike, so consumers only ever see data that passed its checks.