On paper, consolidating four legacy business-unit data and search platforms into one centralized lakehouse is an architecture diagram. In practice, it's an exercise in changing an airplane's engines mid-flight. Every one of those four platforms had real customers, real revenue, and a real team that built it and didn't want it touched.
We did it. The result was a single AWS lakehouse that became the shared foundation for AI-powered experiences across the company, plus $23M in annual savings from re-engineering the core processing along the way. But the parts that mattered most weren't the parts I'd have predicted at the start.
Why four platforms existed in the first place
Fragmentation is rarely incompetence. It's usually history. Each business unit had solved its own problem at its own time with the best tools available then. Four reasonable local decisions added up to one expensive global problem: duplicated data, inconsistent definitions of the same metric, four security models, and an AI strategy that couldn't get clean fuel because the fuel lived in four incompatible tanks.
You can't build cross-functional AI on top of data that can't agree with itself. Consolidation isn't a tidiness project. It's the precondition for everything downstream.
The sequencing problem
The instinct is to design the perfect target state and migrate everyone to it at once. That's how you get an 18-month project that gets cancelled at month 14. Instead, we sequenced around risk and value.
Start with the seams, not the cores
The fastest wins came from unifying the connective tissue, meaning identity, access, and metadata, before touching anyone's core processing. A single governed catalog and a consistent access model let the four platforms keep running while we built the shared layer underneath them.
Make the new platform the easy choice
Migrations don't fail because people resist; they fail because the new thing is harder than the old thing. We treated the lakehouse as a product with internal customers. If landing data in the new platform was faster, cheaper, and better-governed than maintaining a legacy pipeline, teams migrated because they wanted to, not because a mandate told them to.
Re-engineer while you migrate
Lift-and-shift preserves your problems at a new address. Because we were moving the data anyway, we re-engineered the core processing on Spark, Airflow, and EMR at the same time. That is where the $23M in annual savings came from. The migration paid for itself instead of being a cost center.
Governance is what makes it work, not a tax
Centralizing data without centralizing governance just creates a bigger, more dangerous mess. We established enterprise data governance, metadata classification, and security guardrails as a first-class part of the platform. That is what made GDPR and CCPA compliance across millions of accounts tractable, and what made responsible AI training and inference possible afterward.
The reframe that helped: governance isn't the thing slowing AI down. Ungoverned data is. A clean, classified, access-controlled lakehouse is what lets you say "yes" to an AI use case in days instead of quarters.
The org chart was harder than the architecture
The technical design was the part I worried about least. The real work was aligning four teams, and the business-unit leaders above them, around a shared foundation that meant giving up some local control for global capability. That's a leadership problem, not an engineering one: you earn it with transparency, by making the trade visible, and by ensuring the teams that give something up are also the first to get something back.
What I'd tell anyone starting one of these
Sequence around risk and value, not around the org chart. Unify the seams before the cores. Treat your platform as a product and your colleagues as customers. Re-engineer while you migrate so the project funds itself. And build governance in from the first day, because the entire point of consolidation is to make the next thing, usually AI, dramatically easier.