The last twenty per cent: why integration eats the back half of a rollout

There is a familiar shape to platform rollouts. The build goes well, the demo is convincing, the team is confident — and then the last stretch, the part that connects the new system to everything around it, consumes far more of the schedule than anyone allowed for.

This is not a planning failure so much as an estimation blind spot. The work that remains at eighty per cent complete is not the same kind of work as the eighty per cent already done, and estimating it by extrapolation is what goes wrong.

Two different kinds of work

The first eighty per cent is mostly construction: build a screen, model an entity, implement a rule. The scope is knowable in advance, the work is largely under your control, and progress is roughly linear in effort.

The remainder is mostly negotiation with reality: making your system agree with systems you did not build, do not control, and cannot always inspect. Its defining property is that you cannot see the work until you attempt it.

You cannot know that the partner's API returns null where the documentation says it returns an empty array until you call it. You cannot know that their rate limit is enforced per-account rather than per-key until you hit it in load testing. You cannot know that two systems disagree about what a cancelled order means until the reconciliation report shows a discrepancy nobody can explain.

Extrapolating from construction velocity to discovery work is the error. The first kind of work tells you very little about the second.

Where the time actually goes

Identity and matching. Systems rarely share a key. Reconciling records means matching on attributes, and matching on attributes means edge cases: near-duplicates, historical name changes, records created twice during a previous migration. Each ambiguous case needs a rule, and rules need someone with authority to decide.

Semantic mismatch. Both systems have a field called status. They do not mean the same thing, the mapping is not one-to-one, and the gaps are where behaviour has to be invented. This is the most under-estimated category we see, because on a field-mapping spreadsheet it looks like a solved problem.

Failure paths. The happy path is a small fraction of integration code. Timeouts, partial writes, duplicate deliveries, out-of-order events, replays after an outage — each needs a decision and a test. Systems that skip this appear finished and then produce data drift nobody detects for a month.

Other people's release cycles. Your counterpart is also changing. If their sandbox lags production, you are testing against something that no longer exists. If they ship a breaking change mid-project, you absorb it. Neither is in your plan and neither is under your control.

Environment fidelity. Integration testing needs environments that resemble production in the ways that matter — data volume, data shape, credentials, network paths. Most staging environments resemble production in none of these, so problems surface at cutover.

Idempotency is the property that pays for itself

If there is one technical decision that reduces this tail more than any other, it is designing every integration point to be safely repeatable.

An idempotent operation can be applied more than once without changing the result beyond the first application. It sounds like a purist concern. It is in fact the difference between an incident that takes ten minutes and one that takes two days.

When a batch fails halfway with idempotent writes, you re-run it. Without them, you first determine exactly what was written, then construct a bespoke recovery for the remainder, under time pressure, usually with incomplete logs. The recovery is often more dangerous than the original fault.

The same property makes replays safe after an outage, makes duplicate webhook deliveries harmless, and makes retries a first resort rather than a risk. Retrofitting it late is expensive; building it in costs almost nothing.

The practical requirement is that every message carries a stable identifier from its originating system and every consumer records what it has already processed. That is the whole of it.

Sequencing beats estimating

Since discovery work resists estimation, the useful lever is ordering rather than accuracy.

Do the hardest integration first. Not the most important — the least understood. It is uncomfortable early and it converts your largest unknown into a known while the schedule can still absorb the answer. Deferring it means finding out at the point of minimum flexibility.

Get a real payload in week one. Not the documentation, not a sample from the sales engineer: an actual response from the actual system with actual data. Documentation drift is universal and cheap to detect early.

Test at real volume before you believe the design. Behaviour at ten records tells you nothing about behaviour at a hundred thousand. Pagination, timeouts and rate limits are all volume-dependent, and all of them change designs.

Build the reconciliation report before you need it. A daily comparison of record counts and key totals across the boundary catches drift while it is small. Teams that build it after the first serious discrepancy always wish they had built it first.

Treat third-party dependencies as scheduling risk. If you need something from a counterpart, ask early and in writing, and plan on the assumption that it arrives late.

Do less of it

The cheapest integration is the one you do not build. Two questions worth asking of every proposed interface:

Does this data need to move, or does something need to read it? A synchronous read against a system of record is often simpler and safer than a copy that must be kept current. Copies create reconciliation obligations forever.

Does this need to be real time? Real-time synchronisation is materially harder than a scheduled batch — ordering, idempotency, backpressure and failure handling all become live concerns. A great many "real-time" requirements turn out on inspection to mean "within the hour", which is a batch job.

Every interface you remove removes a permanent maintenance obligation, not just a build task.

Contract tests catch the drift

The integration you finish is not the integration you keep. Both sides continue to change, and the usual way you discover a breaking change is that something stops working in production.

Contract testing addresses this directly. Rather than testing your system against a live counterpart — slow, flaky, and dependent on their environment being up — you record the shape of the exchange you depend on and test both sides against that recorded contract.

Concretely, the consumer declares what it needs: these fields, these types, this behaviour when a record is missing. That declaration becomes a test the provider can run. If the provider changes something the consumer relies on, their build fails, before the change ships.

Two properties make this worth the setup cost:

It runs without the other system. Your pipeline does not depend on a sandbox being available, which removes the most common source of flaky integration tests and the temptation to disable them.

It documents the real dependency. A contract states exactly which parts of an interface you depend on, which is almost always a small subset of what the interface offers. That subset is the thing a provider needs to know about, and it is otherwise invisible to them.

The obvious limitation is that it requires cooperation. With an internal team or a group company, running the contract tests is a reasonable ask. With a large external provider it usually is not, and the fallback is a lighter version: keep the recorded contract, run it against their sandbox on a schedule, and alert when the shape changes. That does not prevent a breaking change, but it tells you about it before your users do — which, for a dependency you do not control, is most of the available value.

Either way, the underlying discipline is the same: write down what you actually depend on, and make a change to it something that announces itself.

How we plan for it

We scope integration and APIs as a distinct workstream with its own budget rather than folding it into implementation, precisely so it cannot quietly consume contingency intended for something else. In enterprise platform engagements the first delivery phase deliberately includes the nastiest integration rather than the easiest, on the reasoning above.

The estimate is still uncertain. The difference is that the uncertainty is visible in the plan, and it resolves early enough to do something about.