The Federated Data Nexus
Data fragmentation is the default state of any system landscape past a certain size — ten systems, five formats, no unified view anywhere. This blueprint documents a federated approach to resolving it, without centralising records or paying the compliance cost of a persistent data warehouse.
The Unified View — Why It's Harder Than It Looks
Most serious platforms eventually need one thing: a single, up-to-date view of an entity — a customer, an account, an order — built from every system that touches it. Not a report generated overnight. A live view, queryable on demand, that reflects what every source system currently knows.
The business case is straightforward. Without that view, whoever needs the full picture has to reconstruct it by hand from several disconnected systems — a support agent checking a customer's account across a CRM, a billing system and a usage dashboard. The extra clicks and context-switching are real. So is the risk of missing something that was sitting in a system nobody thought to check. The time cost, multiplied across every person who does this every day, adds up fast.
The problem is that this view doesn't exist by default — it has to be built. And the default way people build it, pulling everything into one central data warehouse, creates a second set of problems that's often worse than the one it was meant to solve.
Any organisation running for more than a few years accumulates systems the same way: one tool per function, bought or built independently, at different times, by different teams — none of them designed to share a format with the others. Whatever "single view" already exists is usually a snapshot generated on a schedule, not a live, complete record. This shape — many systems, no shared format, no single owner — shows up in almost every industry once it grows past a certain size.
Centralising into one warehouse tends to fail for three predictable reasons. First, it's a migration project — pulling data out of every source, reshaping it into one common format, and loading it in — expensive to build and fragile every time a source system changes. Second, it creates a new compliance problem: every record copied into the warehouse becomes a new thing you're legally responsible for under GDPR or whichever regime applies to your industry, on top of the obligations the source systems already carry. Third, it goes stale — a warehouse that refreshes every four hours is fine for a report, not for a decision that needs to be right now.
The Federated Nexus Pattern avoids all three problems by flipping the approach: instead of moving the data to one place, it sends the query out to where the data already lives.
Centralise or Federate — Choosing the Right Tool
Centralising into a warehouse is the right answer for analytical workloads — historical data, ML training, scheduled reporting. Federation is the right answer for operational workloads — real-time queries on live data from systems you may not own or cannot copy. Most mature data architectures use both. The mistake is reaching for one when the other fits better.
| Dimension | Data Warehouse | Federated Nexus |
|---|---|---|
| Data freshness | Batch refresh — hours to days behind live | Real-time — queried at request time |
| Query speed | Very fast — pre-computed, indexed | Depends on source system response time |
| Historical depth | Excellent — years of data in one place | Limited to what source systems currently expose |
| Compliance cost | High — every copy is a new processing obligation | Low — no persistent copy, minimal new obligation |
| Best for | Analytics, ML training, scheduled reporting, dashboards | Operational queries, real-time views, cross-org regulated data |
| Breaks when | Source schemas change frequently — ETL pipelines break | Source systems are slow, unreachable, or expose no read access |
Legacy Formats vs Modern APIs — The Standards Gap That Breaks Integration
The data landscape inside any organisation that's grown past a handful of systems isn't random chaos — it's organised chaos. Every system was built to some standard, they just weren't the same standard, and they weren't built at the same time. A single entity's record can span an old messaging protocol built decades ago for one system, a modern REST API for another, and a proprietary vendor format for a third — each one internally consistent, none of them compatible with the others.
| Dimension | Legacy Protocol | Modern Standard |
|---|---|---|
| Protocol | Fixed-format message segments | RESTful API · JSON or XML resources |
| Data model | Source-specific, undocumented shortcuts | Well-defined, named resources with clear semantics |
| Query method | One-way push messages · no way to ask a question | HTTP GET with structured search parameters |
| Shared vocabulary | Local, inconsistent codes per source | Mandatory shared taxonomy across the industry |
| Regulatory status | Still running most legacy systems, decades on | Increasingly mandated across regulated industries |
The practical consequence: ten source systems, each needing its own custom connector to query and reconcile, creates an integration burden that grows much faster than the number of systems you add. Change one source system's schema, and every connector built against it breaks.
The performance problem compounds it. Legacy systems weren't built to answer real-time API queries from an aggregation layer sitting on top of them — a 2 to 15 second response per source under load is common. Query ten of them one after another and the wall-clock time is too slow to be usable.
The Federated Nexus Pattern — Federate, Don't Centralise
The Federated Nexus Pattern leaves data where it already lives and builds the aggregation layer above it. Instead of solving the connector problem by centralising everything, it solves it by adding one normalisation layer that each source connects to once — turning "every source needs a connector to every other source" into "every source needs one connector to the Nexus." Three parts make this work.
-
1AI Semantic Mapper. Every source exposes data in its own format. The Semantic Mapper translates all of it into one shared, canonical format — without anyone hand-writing a translation rule for every single source. A trained model figures out what maps to what: a field in a legacy format becomes a properly named resource in the shared schema, local codes get reconciled against a common vocabulary, and the same real-world record is matched across systems even without a shared ID. When a source system changes its format, the mapper absorbs the change — nothing downstream has to know or care.
The Semantic Mapper produces confident-looking output even when a mapping is incorrect. In a regulated domain — healthcare, financial services, insurance — a wrong field mapping doesn't surface as a visible error. It surfaces as correct-looking data that leads to a wrong decision. Before production, canonical mappings need explicit sign-off from a domain expert who understands both the source format and the target schema. Low-confidence mappings should be flagged and routed for manual review rather than surfaced as authoritative output.
-
2Async Orchestrator. When a query comes in, the Orchestrator sends a request to every relevant source at the same time, instead of waiting for one to finish before asking the next. Each source gets its own timeout, so a slow or unresponsive one doesn't hold up the rest. A source that answers in 300ms and a legacy system that takes 9 seconds don't block each other — the query returns as soon as it has what it needs.
-
3Mediated Schema — held only for the session. The assembled response is kept in a temporary, session-scoped cache inside a secure network boundary — never written to a permanent data store. It gives the requesting application the single unified view it needs, without creating a new place where that data now lives and has to be protected. Once the session ends, it's gone. The original record stays governed by whichever system owns it. No new retention policy needed, no new processing obligation beyond the query log itself.
This pattern follows the Data Mesh philosophy introduced by Zhamak Dehghani (2019): each domain stays the authoritative owner of its own data and exposes it through a well-defined interface, rather than handing ownership to one central platform team. The Federated Nexus applies the same idea regardless of domain — each source system stays the owner of its data, and the Nexus layer just provides a single place to query across all of them.
The Honest Trade-offs
Latency depends on source system health
Even with parallel queries, total response time is bounded by the slowest source that answers within its timeout. A source under load or behind a slow network degrades the entire view. A centralised warehouse doesn't have this dependency at query time — the data is already there.
Mapping accuracy requires senior domain expertise
The AI Semantic Mapper needs expert-validated canonical mappings before it can be trusted in production. This isn't a one-time setup cost — it's an ongoing governance commitment. Every time a source system changes its format, the mapping layer needs review. Teams without the domain expertise to validate mappings should not run this pattern in regulated environments.
Source unavailability means gaps, not stale data
If a source system is unreachable at query time, that portion of the unified view is missing — not stale, missing. A warehouse returns yesterday's data when a source goes down; a federation returns a gap. Depending on the use case, a gap can be worse than stale. The pattern needs an explicit strategy for partial responses.
Access control complexity is higher than with a warehouse
Row-level and field-level access control has to be enforced at the Nexus layer across every source system it queries. Getting this wrong means the federation layer becomes an access control bypass — a user who can query the Nexus but not the underlying system directly should never see more than they're authorised to see at the source.
Architecture — From Legacy Silos to a Unified View
Unified timeline · Real-time
One canonical API surface
Each column does one job. The source column exposes data through whatever interfaces already exist — no migration, no schema changes, no vendor engagement required. The Nexus turns that data into one common shape, assembles it in parallel across sources with wildly different response times, and holds it only for the length of the session. The unified interface gets one clean API to query — with no idea what protocols or quirks sit behind it.
The property this architecture protects above all: the original record never leaves its source system. The Nexus only ever holds a temporary, session-scoped copy. Compliance responsibility stays with whoever owns the source system. No new processing obligation is created beyond the query log at the Nexus layer.
One implementation decision the blueprint doesn't resolve automatically: what happens when two source systems return conflicting values for the same field on the same entity — different addresses, different account statuses, different balances. The Nexus has three options: designate one source as authoritative per field; apply a recency rule and surface the most recently updated value; or surface both values and flag the conflict for the consuming application to handle. This decision needs to be made explicitly during canonical schema design — not discovered at query time in a production environment.
One Pattern, Several Industries
Fragmented source systems aren't a healthcare-only problem — they're the default state of any organisation that's grown past a handful of tools. The same shape shows up wherever one entity's history is scattered across systems that were never built to talk to each other. A few examples:
Patient records scattered across EHR, lab, imaging and pharmacy systems
A clinician needs one view of a patient's history. Instead they open the EHR for notes, a separate portal for lab results, an imaging system for scans, and a pharmacy system for medication — each speaking a different format.
Customer accounts scattered across core banking, cards and fintech partner systems
A single customer's financial picture spans a core banking platform, a card processor, and however many fintech partners have been integrated — each with its own data model and its own refresh schedule.
Product and inventory data scattered across POS, warehouse and marketplace systems
Knowing what's actually in stock means reconciling a point-of-sale system, a warehouse management system, and however many marketplace channels the business sells through — none of which update each other in real time.
Claims data scattered across policy admin, claims processing and third-party assessor systems
Processing a claim end to end means pulling from a policy administration system, a claims workflow tool, and reports from outside assessors or repair networks — usually stitched together by hand.
Customer data scattered across CRM, billing, support and product usage systems
A single customer relationship spans a CRM, a billing platform, a support desk and the product's own usage data — the same shape as everywhere else on this list, just at a smaller scale.
The three moving parts (AI Semantic Mapper, Async Orchestrator, session-scoped Mediated Schema) stay the same regardless of industry. What changes is the canonical schema you translate everything into, and the specific regulatory regime you're operating under.
Is the Federated Nexus the right architecture?
This pattern fits when
Data lives across three or more source systems with distinct schemas and no existing common query interface
The target product needs a real-time or near-real-time unified view — not a batch analytics report or a scheduled export
Centralising the data would create unacceptable compliance overhead under GDPR or an equivalent regime for the industry
A canonical interoperability standard is mandated or preferred for the target market or regulatory environment
The product is a decision-support tool, customer-facing portal, or coordination layer — not a primary system of record
This pattern does not fit when
The use case is large-scale retrospective analytics over historical data — a centralised warehouse is a better fit for batch processing at that scale
Source systems can't expose read access in any form — no legacy message feed, no modern API, no database access under a data-sharing agreement
The environment needs full offline capability — this pattern needs network access to source systems at query time
The team lacks senior domain expertise to design and validate the mapping layer — a wrong mapping produces confidently wrong data, and in a regulated domain that's a compliance or safety risk, not just a bug
Building AI features that need data from multiple source systems? Explore AI & Automation →