Most enterprise AI programs stall in the same place. A team builds a convincing demo, the room gets excited, and then the second use case costs just as much as the first, and the third costs more. Nothing compounds. Over two quarters I led the build of an AI platform designed to break that pattern, and this is an account of what we actually built: the agentic framework, the retrieval system, the evaluation system, the workflow onboarding layer, the data foundation, and the MLOps underneath it.
None of it started with a model. It started with a decision about what kind of problem this was.
Treat AI as a platform problem
Left alone, every product team builds its own stack. Its own ingestion, its own prompts, its own connection to a model provider, its own idea of what a guardrail is. You end up with six half-platforms, none of them good, and a security review for each. We chose instead to centralize the parts nobody differentiates on and push everything else out to the teams.
Five principles carried the work. Every capability is an independent building block, usable on its own or composed into something larger. Centralize the heavy lifting, meaning security, compliance, infrastructure, data, identity and model access, so product teams spend their time on what makes their product different. Componentize for reuse, with golden paths and shared SDKs that remove the boilerplate and the decisions nobody should have to make twice. Keep platform and business logic strictly separated, so the stack can be upgraded without disrupting anyone's delivery. And solve for one team in a way that generalizes to all of them, so each initiative funds the next.
The measure of success was straightforward. The platform stops being a cost center when the second and third use cases cost a fraction of the first. Adoption should lower the cost of the next thing, not raise it.
The model is the smallest part of an AI product. What decides whether it ships is everything around it: retrieval, evaluation, identity, guardrails, and the plumbing that lets the next team go faster than the last one did.
A layered architecture that stays swappable
The platform is four layers. Applications and front-end experiences sit on top. Below them an API and workflow layer handles orchestration and exposes a stable contract. Below that the AI services layer holds the agent runtime, retrieval, and the model gateway. At the bottom sit the data and infrastructure foundations.
The most important property is that the model is not load-bearing. All model connectivity flows through one gateway that abstracts provider differences, with a router that selects a model per use case based on capability, latency and cost. Product code never names a provider. When a better or cheaper model appears, and one appears every few weeks, adopting it is a routing change rather than a rewrite.
We made the same choice about orchestration. Rather than betting the platform on a single agent framework, the runtime supports several, because that part of the ecosystem is still moving and being locked to one opinionated framework is a liability. Agents are model-agnostic and data-independent by design. You can swap a model, add a data source, or compose a new workflow without touching shared infrastructure.
One gateway instead of an integration mesh
The fastest way to make an AI platform unmaintainable is to let every application connect directly to every service. With a handful of apps and a handful of tools you already have dozens of bespoke integrations, each with its own auth, its own logging, and its own idea of a rate limit. That is the classic N times M problem, and it gets worse every quarter.
We put a single gateway in the middle, using the Model Context Protocol as the standard interface, in a hub and spoke arrangement. Every application and every agent connects through the hub. Every data service and tool registers behind it. Adding a consumer or a service does not touch anything else.
What makes the hub worth the indirection is that the cross-cutting concerns get enforced exactly once: identity and entitlements, observability, evaluation and guardrails, the tool and skill registry, rate limiting, data protection and compliance. None of that is re-implemented per spoke, and none of it can be quietly skipped by a team in a hurry.
The same gateway then became a distribution channel. Because it speaks a standard protocol, the same governed tools and data can be exposed outward into other AI ecosystems and enterprise assistants, so customers reach the content through whatever assistant they already use. That turned an internal architecture decision into a commercial one.
Advanced retrieval, where precision is the product
Retrieval is where most of the answer quality lives, and it is the part teams most often treat as a solved problem. It is not solved. We built one ingestion and retrieval pipeline and reused it across products rather than letting each team build a worse version of it.
The pipeline runs in stages. Multi-format ingestion takes in documents, web content and spreadsheets and pushes them through a document processing service that parses, chunks and cleans, because most retrieval failures are really parsing failures. Chunks are embedded and written to a hybrid index that combines vector similarity with lexical search, with an index tree that narrows the search space before scoring. Real questions mix fuzzy meaning with exact terms such as product codes, names and identifiers, and a pure vector search will blur exactly the tokens that matter most.
On the retrieval side we use hypothetical document embedding to lift recall on sparse or badly phrased queries, a cross-encoder reranker to restore precision on the candidate set, and a query cache so repeat questions do not pay the full cost. Output is dual: embeddings to the vector store and structured records to the governed lake, plus structured JSON for runtime consumption, so the same pipeline serves conversational answers and downstream analytics.
Every response is grounded in source material with citations that trace back to the document. In an enterprise setting an uncited answer is not a feature, it is a liability.
Evaluation as a first-class system
The single biggest difference between a demo and a product is whether you can measure quality and defend it. We built an evaluation system and treated it as platform infrastructure, not as a testing chore.
At the center is a curated golden test set, versioned like code, that captures the questions the business actually cares about along with correct answers and the sources they should cite. Every change to retrieval, prompts, chunking or model selection runs against it automatically. We score precision, recall, F1 and latency per run, and compare variants side by side, typically a current baseline, a tuned candidate, and something more experimental. Nothing is promoted on a hunch or a demo that went well.
Around that sit tracing and observability on every call, automated retrieval quality scoring, human review for the hard and ambiguous cases, and cost tracking per agent and per workflow so quality is always read next to what it costs to produce. Guardrails run inline rather than as a post-hoc filter.
The working rule we settled on: no agent ships against a use case until there is an eval set for it, because the eval set is the real specification. Once that loop exists, improvement becomes routine. Test, measure, tune, deploy, repeat, with a regression signal that tells you on every change whether you made the product better or just different. On our golden set that loop moved retrieval quality into the mid nineties for precision and recall, and more importantly it stopped quality from silently regressing.
Onboarding workflows without the platform team in the loop
A platform that requires its own team for every new use case is a bottleneck with good branding. The point of the investment is that other people can use it without asking permission or booking your calendar.
So we built a studio, a single control plane where teams create agents from templates, register tools and connectors, assemble multi-step workflows in a visual builder, and test everything in an integrated chat before it goes anywhere. It supports multiple personas, so the same workflow behaves correctly for different roles and entitlement levels. Publishing runs through the normal software delivery lifecycle with the guardrails already attached, rather than through a side door.
Underneath, the tool and connector registry is what makes this work. Once a capability is registered once, every agent can discover and use it, with identity and entitlements enforced at the gateway. Onboarding a workflow becomes configuration rather than a project, and the backlog of internal processes that could be accelerated turns from an aspiration into a queue teams can work through themselves.
The data foundation underneath
Retrieval quality is a data platform problem wearing an AI costume. If the underlying data is fragmented, duplicated or ungoverned, no amount of prompt engineering rescues it.
The foundation is a governed lakehouse on a medallion structure, raw to refined to business-ready, with versioned transformations and the analytics and BI layer reading from the same governed tables the AI does. Metadata is not an afterthought. Datasets surface automatically in a catalog with full lineage back to source, schema and column-level detail, quality and freshness statistics, and ownership and tags flowing through as governance metadata. When someone asks where a number came from, the answer is a link rather than an investigation.
On top of that we added zero-copy data sharing. Curated datasets are published as private listings and appear live in a consumer's own warehouse account, with no duplication and no extract-transform-load pipeline in between. Access stays governed and revocable, and the customer works with current data instead of a stale copy.
AI-assisted data acquisition
A large share of the data arriving in any enterprise platform comes from sources that were never designed to be consumed programmatically. That work is traditionally hand-written scrapers, which break constantly and are nobody's favorite thing to maintain.
We replaced that with a governed acquisition framework where the language model does the tedious part. It generates extraction code, infers schemas, analyzes page structure, diagnoses failures and repairs pipelines when a source changes, which is the difference between a pipeline that self-heals and one that pages a person. A scheduler handles cadence, retries, backoff and dead letter queues with per-source rate rules. Execution runs on managed workers supporting plain HTTP, headless browser automation and standard crawling frameworks, depending on what the source requires.
Everything lands through a type-safe columnar transformation layer with schema validation and column mapping, into partitioned, immutable, versioned object storage, and from there into the warehouse and the vectorization pipeline. It is governed by construction: run identifiers, configuration versions, and inputs and outputs are tracked on every execution, credentials come from a vault rather than a config file, and alerts route to the owner of the source. Auditability and lineage are properties of the design rather than something bolted on for a compliance review.
MLOps and the operational bar
Models and agents need the same delivery discipline as any other production system, and a bit more, because they degrade quietly. Managed pipelines cover the full lifecycle from experiment to production: data validation, training, evaluation against thresholds, registration, controlled deployment to serving, and monitoring that watches for drift and performance decay and triggers retraining. Fine-tuning and distillation sit in the same pipeline, so a smaller and cheaper model can take over serving when it holds quality.
The operational requirements were set to match the fact that this is now customer-facing. Blue and green deployment for zero-downtime releases with instant rollback. Auto-scaling and right-sizing for load spikes. Circuit breakers and rate limiting so a failure in one dependency does not cascade. Health checks and proactive alerting rather than waiting for a report. Graceful degradation paths so a partial outage costs a feature rather than the whole experience. The availability target for the identity and access path, which everything else depends on, was 99.95 percent.
Identity deserves its own mention. The platform runs zero-trust, with fine-grained entitlements covering both workforce and customer identity, and entitlement handoff extended to agents and tools. An agent acting on behalf of a person must never see more than that person is entitled to see. Getting that right is what makes agentic access to enterprise data defensible.
The developer multiplier
The last piece was making the engineers faster, since a platform that only speeds up AI features leaves most of the organization where it was. We rolled out AI coding assistants with real adoption tracking, moved observability onto a modern stack with unified logging, modernized the API gateway, and stood up a developer portal for documentation, onboarding and self-service tooling.
The piece I would build again first is the operations agent. It watches production, and when a priority incident fires it investigates on its own: it ingests the alert, queries the monitoring and metrics systems, maps service dependencies, correlates telemetry with recent code and deployment activity to isolate a root cause, applies a remediation such as scaling infrastructure or throttling a workload, notifies the right channel, and then writes a prevention specification describing what should change so it does not recur. It runs against the standard operations toolchain through the same gateway everything else uses.
What actually moved
By the end of the second quarter the platform stopped being a build and started being infrastructure other people depended on. The first production agentic workflow went live in a customer-facing product. One retrieval pipeline served multiple products instead of each team building its own. Connectors and backend services were reused across both internal and external workflows. The gateway was live inside third-party AI ecosystems, which opened a commercial channel that did not exist before. Several proofs of concept in different business areas ran on shared components rather than bespoke stacks, which is the clearest sign the platform is doing its job.
The honest summary is that almost none of the hard work was model work. It was retrieval quality, evaluation, identity and entitlements, governed data, self-service onboarding, and operational discipline. That is the unglamorous 90 percent, and it is the entire difference between a demo that impresses a room and a platform a business can run on.