In a research business, the product is a number with a defensible story behind it. Clients pay for a view of where supply, demand and prices are going, and they expect that view to move when the market moves. So when the process that produces the number takes weeks of manual work, the problem is not an engineering inconvenience. It is the product arriving late.
This is about taking a modelling process that ran on spreadsheets, scripts and expert memory and turning it into something with pipelines, versions and an audit trail. The estate covered five different families of model, from deterministic valuations to solvers to simulation, and only one of them was machine learning. That mix is what decides how far an MLOps programme actually reaches.
The business problem, stated plainly
Expert analysts were spending the majority of their time on three things that are not analysis: finding the data, wrangling it into the shape a model expects, and checking every step by hand. That cost was paid once to produce a forecast, and then paid again in full for every update, because none of the work was captured anywhere a machine could re-run it.
Three consequences followed, and they compound.
The update cycle was long. Long enough that between refreshes the published view drifted away from the market it was describing. In a slow market that is tolerable. In a volatile one it is the whole ballgame.
Scenarios were projects. Answering a question like "what happens to this outlook if prices run twenty percent higher for three years" required an analyst to redo the work. So those questions got asked rarely, answered slowly, and only for the largest clients. The ability to test futures on demand, which is the actual value of owning a model, was locked behind human effort.
Nothing was reproducible. Six months after publishing a number, reconstructing exactly how it was produced meant finding the right spreadsheet, the right analyst, and hoping both still remembered. For a business whose credibility rests on defending its methodology, that is a serious exposure.
What "model" actually meant
The first useful thing we did was stop saying "the models" as though they were one kind of thing. Across the business the estate covered at least five families, and only one of them was machine learning.
- Deterministic models. Asset valuations, cost curves. Structured arithmetic over structured inputs, with a great deal of domain logic and no statistics at all.
- Linear optimization. Demand allocation and dispatch problems, solved with a commercial or open-source solver against constraints.
- Machine learning. Price forecasting where the relationships are learned rather than specified.
- Statistical and predictive. Short-horizon forecasting from time series, closer to econometrics than to deep learning.
- Simulation. Process simulation and Monte Carlo runs over distributions of inputs.
Tooling had settled wherever each team could get the job done, and it tracked two variables: how complex the business rules were, and how much data the model chewed through. Simple rules at low volume lived in spreadsheets. A bit more of either and you got spreadsheet macros, then Python scripts, then stored procedures, then a dedicated platform. Every team made a locally reasonable choice, and the aggregate was a zoo.
That zoo is the thing people usually try to fix first, by picking one modelling tool and mandating it. We ran that evaluation, scoring the options on data curation, integration, visualization, model execution, storage, security and permissions, and performance at scale. The evaluation was worth doing, and it produced a clear answer about where new work should go. But it did not solve the problem, because the estate that already existed was not going to be rewritten, and because the actual pain was never really about which language the model was written in.
The pain was that none of these models, whatever their family, had an operating discipline around them. They all needed the same things: versioned inputs, a reproducible run, somewhere to register the artifact, a way to be triggered on a schedule or on demand, monitoring on what came out, and a path to publication. That list is MLOps. The machine learning part is a special case of it, not the point of it.
So the framing we used, and the one I would use again, is that this is model operations. If you scope it as an ML programme you will build something the ML team likes and the other four families ignore, and you will have solved a fraction of the problem at full cost.
The target lifecycle
The shape we designed toward has two loops joined at a registry. The left loop is continuous integration and delivery for models: data engineering, model development, and an automated pipeline that trains, evaluates and packages. The right loop is automated operations: orchestration, inference, and continuous monitoring that feeds back into the next cycle. The model registry sits at the join, and the promotion of an artifact through it is the only legitimate way for work to cross from one loop to the other.
That last point matters more than the diagram makes it look. If a model can reach production any other way, by a notebook someone ran or a file someone copied, then the registry is documentation rather than control, and every guarantee downstream of it is a hope.
The architecture
Five layers, each with a job, and cross-cutting concerns that belong to the platform rather than to any model. The reference implementation ran on one cloud, but nothing in the shape is specific to it: every box has an equivalent everywhere, and I have named the capability first so the substitution is obvious.
The parts that carry the weight
A run is reproducible or it is an anecdote
The single most valuable artifact we introduced was not a model. It was the run manifest: the record that says exactly what produced a number. A result is reproducible when three things are pinned, and a business that publishes forecasts needs all three pinned for years, not weeks.
Run manifest, written before execution and stored with the results{
"run_id": "run_2026_03_14_am_supply_0417",
"model": { "name": "supply_outlook", "version": "7.2.1",
"registry_stage": "production", "approved_by": "..." },
"data": { "snapshot": "2026-03-14T06:00Z",
"tables": [ "prices@v412", "asset_master@v88",
"capacity_additions@v53" ] },
"parameters": { "set": "high_price_case", "version": 4,
"overrides": { "price_shift_pct": 20, "horizon_years": 15 } },
"code": { "pipeline_commit": "b9f4c21", "image": "runner:3.4.0" },
"compute": { "family": "solver", "instances": 12, "wall_clock_s": 5140 },
"outputs": { "uri": "s3://.../run_2026_03_14_am_supply_0417/",
"row_count": 2841600, "checksum": "sha256:..." }
}
Model version, data version, parameter version. Pin those three and the run is a function you can call again in two years and get the same answer. Leave any one of them floating and you have a number nobody can defend. Note that the parameter set is versioned as a first-class object rather than passed in ad hoc, because in a scenario business the parameters are the question being asked.
Contracts at the boundary, not validation at the end
The manual process put validation at the end, where an analyst checked whether the output looked sensible. That is the most expensive place to catch an error, because you have already paid for the run. We moved it to the front: every model declares the schema it expects, the ranges each field is allowed to take, and the invariants that must hold. A run that violates the contract fails in seconds instead of producing plausible nonsense in hours.
For the machine learning models this is standard data validation. For the deterministic and optimization models it turned out to matter more, because those models will happily compute a precise answer from an input that makes no physical sense, and nothing about the output will look wrong.
Transformations as pipelines, not as notebooks
The wrangling work that consumed analyst time was real work, and most of it was the same work every cycle. Moving it into versioned pipelines did two things: it stopped being re-done by hand, and it became the same code path for training and for inference, which is the cheap way to avoid the skew that appears when those two paths are written separately by different people at different times.
Long runs need an asynchronous contract
A solver over a large constraint set or a Monte Carlo simulation with enough draws to be useful does not complete inside an HTTP timeout. Anything that pretends otherwise breaks the first time a run gets big. So execution is a job API rather than a request and response: start a run, receive an identifier, poll status or receive an event on completion, then fetch results by reference. Results go to object storage and callers get a pointer and summary statistics, not a multi-million-row payload.
Wrapping that in a state machine rather than a script is what makes retries, timeouts, partial failure and approval steps first-class rather than something each model re-implements badly.
Monitoring has to fit the model family
Drift detection on input distributions is the right instrument for a learned model and close to meaningless for a deterministic one. For the non-ML families the useful checks are invariants and ranges on outputs, and comparison of this run against the last one to surface material movements for a human to look at. Across all families, the check that actually earns trust is back-testing: how did what we published last year compare with what happened. That is a slower signal and a harder one to build, and it is the one clients care about.
Coordination between dependent models
Several models consumed the outputs of other models, which in the manual world meant a downstream view could be quietly built on a stale upstream one. Once runs carry manifests, the dependency is explicit and checkable: a downstream run records which upstream run it consumed, and the orchestration layer refuses to publish a chain where an input has been superseded without a deliberate override.
Making the estate callable by agents
The last piece is what turns this from an efficiency project into a different product. Once models are versioned, callable and grounded, they can be invoked by an agent rather than only by a scheduler or a human with a form.
We did not build a bespoke integration for that. The central AI platform already had the pieces: an agent runtime, a gateway that fronts every tool and data service, a tool registry, and a studio for onboarding new workflows. The MLOps platform gets registered as a custom workflow extension, which means it shows up to agents the same way every other capability does, and inherits the same identity, entitlement, observability and evaluation machinery rather than reinventing them.
The tool contract
The extension exposes a small, typed surface. Small is deliberate: a wide tool surface makes agent behaviour harder to predict and harder to evaluate.
Tools registered by the extensionlist_models()
-> [ { name, family, description, owner, latest_version } ]
describe_model(name)
-> { input_schema, parameter_schema, allowed_ranges,
typical_runtime_s, cost_band, required_entitlement }
start_run(name, parameter_set, overrides) # async, returns immediately
-> { run_id, accepted_at, estimated_completion_s }
get_run_status(run_id)
-> { state: queued|running|failed|complete, progress, message }
get_run_results(run_id, projection) # reference, not payload
-> { uri, row_count, summary_stats, manifest }
compare_runs(baseline_run_id, scenario_run_id, dimensions)
-> { deltas: [ { dimension, metric, baseline, scenario,
abs_change, pct_change, material: bool } ] }
Two design decisions in there are worth pulling out.
describe_model returns the allowed ranges. The agent does not guess what a sensible price shift is. It reads the declared bounds from the parameter store and is rejected at the boundary if it proposes something outside them. The contract is the guardrail, which is more reliable than instructing a model to behave and hoping.
compare_runs is code, not generation. This is the rule I would enforce hardest on any team building this: the language model narrates, it does not calculate. Every delta, percentage and ranking in the output is computed by the extension and handed to the model as structured input. The model's job is to decide what matters and say it clearly. When arithmetic leaks into the generation step you get outputs that read beautifully and are wrong in ways that are very hard to catch.
The scenario comparison workflow
The workflow that made the value obvious to the business was scenario comparison, which used to be a multi-day analyst task. A user picks a model, a baseline case and a scenario case, and asks for the difference. The extension runs six steps, and only the last one is generative.
Entitlements have to travel with the call
An agent acting for a user must not be able to do more than that user can. The gateway resolves the caller's identity and passes it down, so a run the user is not entitled to start is refused at the extension, and results the user cannot see are never returned. This sounds obvious and is the thing most commonly skipped, because during a prototype the agent runs with a service account that can see everything and nobody notices until the pilot widens.
Evaluating a workflow, not just a prompt
Because the extension is registered on the platform, it inherits the evaluation system. The golden set for this one is a collection of scenario comparisons with known answers, and the scoring checks something specific: does every quantitative claim in the generated narrative match a delta the extension actually computed. That is a groundedness test with a precise definition, which is only possible because the arithmetic lives in code where it can be compared against. It also catches the failure mode that matters most here, which is a fluent summary that overstates a movement.
Keep a human in the loop where the output is external
Internal exploration can run unattended. Anything heading for a client goes through analyst review, with the manifest attached so the reviewer can see exactly which model version and data snapshot produced it. The point of the automation was never to remove the expert. It was to stop spending the expert on data wrangling so they can spend their time on the judgment that only they can provide.
What changes, and what does not
The visible change is speed: a scenario goes from a piece of work someone schedules to a question someone asks. The less visible change matters more. A published number acquires a provenance record, which means it can be defended, corrected precisely, and rebuilt years later. Dependent models stop silently consuming stale inputs. A new model arrives through a golden path rather than as a bespoke build, so the cost of the next one falls.
What does not change is that the model still has to be right. None of this improves the methodology, and a platform that industrializes a flawed model just produces wrong answers faster and with better provenance. The engineering buys speed, reproducibility and reach. The judgment still belongs to the people who understand the domain, which is the argument for taking the wrangling away from them in the first place.
The sequencing lesson is the one I would repeat. The agent layer is the part everyone wants to build first, and it is the part that only works last. An agent in front of a manual, unversioned, unreproducible process is a faster way to be confidently wrong. Make the models callable, versioned and grounded, and the agent becomes a thin and genuinely useful layer on top. Do it in the other order and you will spend a year demonstrating prototypes that cannot be put in front of a client.