External Courier Gateway

The Assumption

Every system accumulates assumptions that were never decisions. Grubhub Managed Delivery had a good one: a delivery is served by a Grubhub courier. Not as a rule written in a config file — as a shape the code had grown into. Courier identity was an employee record. Location came from our app on a phone we knew the number of. Trip state advanced because a courier tapped a button. Dispatch was a choice we made, so somebody was always going to be assigned, and every downstream service was written expecting that.

Serving a delivery with a Relay courier breaks all four at once, and it breaks them in the awkward direction. It's not that we needed less from those services — the diner still needs a name, a photo, a phone number, a moving dot on a map, and a status that changes at the right moments. We needed the same outputs from a courier who is not in our system, not carrying our app, and not ours to dispatch.

The interesting part of this project isn't the Relay integration. It's that a system built around an assumption that strong usually has to be renovated to lose it. This one didn't, and the reason is architectural.

What Loose Coupling Actually Bought

The delivery lifecycle at Grubhub was already distributed, asynchronous, and event-driven — services publishing what happened rather than calling each other to say what to do. The backbone event stream, cmbndDlvryV2, carried the combined delivery view: state transitions, assignment changes, timing updates, published by Delivery API and consumed by whoever cared.

That style is usually defended in terms of failure isolation and independent deploys. The payoff here was different and much larger: a new consumer could be added to the delivery lifecycle without a single existing service being told about it. Nothing upstream needed a branch for "and if this is an external courier." The External Courier Gateway subscribed to the same events every other consumer already got, and decided for itself which deliveries were its business.

The inverse is the version of this project that didn't happen. In a synchronous, tightly coupled system, "sometimes a third party has the courier" is a conditional threaded through every service that touches assignment — dispatch, tracking, status, support tooling — each one needing to know a distinction it has no business knowing. The event stream is what let the distinction live in exactly one service.

The Gateway Where the App Used To Be

The design follows from asking what a Grubhub courier actually is, from the perspective of the backend. Not a person — a source of events. The Delivery Mobile Gateway was the edge our courier apps talked to: it authenticated the courier, took their location pings, accepted their taps, and translated all of it into calls and events against the internal services.

So ECG was built as a peer of the DMG, not as a feature inside anything. Same position in the topology, same downstream contracts, different upstream: instead of an authenticated courier holding a phone, the upstream is a provider's webhooks. Everything past that seam is indistinguishable, on purpose. Courier, Colo, Trip, Delivery Service, and Order Status were never modified to know that Relay exists, because from where they sit, it doesn't.

That gave ECG a small, blunt job description:

• Decide, per delivery, whether an external provider should serve it.
• Stop the delivery from being dispatched to one of our own couriers in the meantime.
• Ask the provider to take it, and wait — for an unbounded amount of time.
• Turn whoever they assign into a courier our system can represent.
• Produce, from webhooks, the same event sequence a courier's thumb and GPS would have produced.

Minting a Courier Who Doesn't Work Here

The identity problem is the one that looks trivial and isn't. Relay tells you who they assigned: a name, a photo URL, a phone number, and an opaque identifier of their own. Every service we own keys on a Grubhub courier id — a long, stable, internal number that Trip references, Colo indexes location by, Order Status renders from, and support tooling searches. You cannot hand those services a string from someone else's namespace.

So ECG mints one. The Courier service gained a notion of an externally-sourced courier, identified by the pair (provider, external_id), upserted idempotently: the first webhook for a given pair creates the record and allocates a real courier id, and every webhook after that resolves to the same one. That single decision is what makes the rest possible — downstream, an external courier is a courier, with an id shaped like every other id.

Stability is the requirement doing the work. A courier who runs three deliveries for us over a week has to be the same courier id all three times, or tracking history fragments, support can't correlate anything, and a reassignment mid-delivery is indistinguishable from a brand new person. Getting that right means treating the provider's id as the natural key and our id as derived and permanent — never the other way around, and never allocated per delivery.

The rest of the profile is data hygiene with teeth. The photo is fetched and re-hosted rather than hot-linked, so the diner's tracking page doesn't depend on a third party's CDN being up, and doesn't leak a request to it. The name is normalized to the same presentation our own couriers get. And the phone number is never shown to anyone.

Two Numbers That Are Both Fake

Grubhub couriers and diners have always talked through masked numbers — a Twilio proxy session per delivery, one leg bound to the diner, one to the courier, neither side ever holding the other's real number, and the whole session torn down at the end. It is a privacy control, and it is not optional just because the courier is somebody else's.

If anything it gets more important. Handing a diner's phone number to a third-party provider so their courier can call is exactly the thing masking exists to prevent, and it's a disclosure that happens once and cannot be walked back. ECG creates the proxy session before the delivery is ever offered to Relay: the number that goes out in the request is the masked one. When the courier assignment webhook arrives, their number gets bound to the other leg of the same session.

Which means a mid-delivery reassignment is a rebind, not a new number. The diner's contact doesn't change under them when Relay swaps couriers — same masked endpoint, different leg behind it. That's visible in the demo with the reassignment fault switched on.

The Hold

There's a window between deciding an external provider should serve a delivery and finding out whether they will. It might be a few seconds. It might be much longer. During that window the delivery is sitting in exactly the state autodispatch is looking for: unassigned, ready, within range of couriers who would happily take it.

So ECG places a hold on the delivery before it asks anyone anything. The hold makes the delivery invisible to dispatch, carries a reason so it's diagnosable rather than mysterious, and — critically — carries a TTL. A hold that can leak is a delivery that silently never gets a courier, which is a far worse failure than any of the ones the hold is protecting against. Expiry returns it to the normal path automatically.

The ordering matters and is easy to get backwards: hold first, then ask. Asking first opens a race where the provider accepts and our own dispatch assigns, and now two couriers are en route to the same restaurant with no mechanism to discover each other. The hold is released in the same operation that records the external assignment, so the delivery is never simultaneously un-held and un-assigned.

Emulating a Courier

With identity and the hold in place, the remaining work is translation, and this is where the gateway earns the name. Each provider webhook maps to the internal calls and events that courier action would have generated:

courier.assigned          → Courier.upsertExternal → Trip.create + assign
                            Delivery.assignCourier + releaseHold
courier.location          → Colo.ping(courier_id, lat, lng, at)
courier.arrived_at_pickup → Trip.event(ARRIVED_PICKUP)
order.picked_up           → Trip.event(PICKED_UP) + Delivery.pickedUp
order.delivered           → Trip.complete + Delivery.complete

Order Status — the service behind the diner's tracking page — was never touched. It consumes queue updates from Colo, Trip, and Delivery Service, and it produces the correct diner experience for a Relay delivery for the simple reason that it is receiving the same updates it always did, from the same three services. The provider distinction dies at ECG's boundary.

Region Data, consulted over a queue, is what decides any of this applies: which provider (if any) serves a region, and what that provider can actually do. Those capability flags are the difference between an integration and a gateway, and they're the reason the next section exists.

Webhooks Are Not an Event Bus

Internally, we had ordering guarantees, a schema registry, replay, and consumer lag we could measure. A third party's webhooks have none of that, and the failure modes are not exotic — they're Tuesday:

• Duplicates. Any at-least-once sender retries on a timeout it experienced and you didn't. Every handler is keyed on the provider's event id and is a no-op the second time.
• Reordering. Two webhooks dispatched a second apart can land in either order. Lifecycle events carry a sequence, and anything ahead of the current position goes into a reorder buffer and is released when its predecessor lands — rather than being applied and dragging the trip backwards through states it already left.
• Gaps. The location stream is best-effort. Colo holds a last-known fix with a timestamp, and staleness is surfaced rather than smoothed: an interpolated courier position is a lie that gets rendered on a map and then written into an ETA.
• Contradictions. A provider can tell you a delivery was picked up by a courier it already told you was unassigned. The state machine rejects the transition and says so loudly, because quietly accepting it produces data nobody can debug a week later.

None of this is clever. It's the standard anti-corruption-layer discipline, applied at the one place in the topology where an outside system's guarantees stop and ours begin. The value of concentrating it there is that no other service needs any of it.

Testing a Dispatcher You Don't Control

The integration cannot be exercised in staging by asking Relay to dispatch real couriers to real restaurants. Their sandbox could produce assignments, but it could not make a courier move — no way to drive a location stream along a plausible route, which is most of what there is to test.

We already had the machine for this. The actor-based simulation gave our pre-production environment a heartbeat: statistically realistic diners, merchants, and couriers driving real road geometry through a simulated day. Its couriers already did everything a Relay courier would need to do — accept, drive, arrive, pick up, deliver — they just did it by calling the Delivery Mobile Gateway, the same as a real app.

So we gave the simulation a second output mode. The same courier actors, the same routes, the same timing distributions; instead of posting to the DMG, they signed and posted webhooks at ECG in Relay's format, from a fake dispatcher that accepted delivery requests and assigned simulated couriers to them. Location updates fell out for free — the actors were already driving the road network at 1 Hz.

That gave us the thing the vendor sandbox couldn't: a full external-courier delivery, end to end, on demand, deterministic enough to assert against — and cheap to make hostile. Duplicate a webhook. Deliver two out of order. Drop the location stream for six minutes. Reassign the courier after pickup. Every one of those is a switch in the fake dispatcher, and every one of them found something.

Then It Flew

Some time later, Wonder needed drone deliveries in the same lifecycle. The honest expectation for a system built as a Relay integration would be a second integration built beside it, with a third one implied.

What it actually took was a new provider adapter and a capability profile, because the abstractions had been drawn around the problem rather than around Relay:

• Identity was already provider-scoped. A tail number is an external_id like any other. An aircraft becomes a courier with a stable id, a name, and a photo, through the same idempotent upsert — no new concept, no new column.
• The hold was never about Relay. "Don't dispatch this to one of ours while somebody else decides" is provider-agnostic on its face, and it survived unchanged.
• Capabilities absorbed the differences. Drones have no phone, so the masking step is skipped by configuration rather than stubbed out. They have no taps, so geofence crossings synthesize the pickup and dropoff events. Telemetry arrives far faster than a phone's GPS, so it's downsampled on the way into Colo. Each of those is a branch that already existed as a flag, because Relay had already differed from our own couriers along the same axes.

The event emulation, the ordering defenses, the Colo and Trip and Order Status wiring, the simulation harness — all reused. The generalization wasn't foresight about drones specifically. It came from a narrower discipline: when Relay differed from a Grubhub courier, we modeled the difference as a capability the provider does or doesn't have, instead of writing if (relay). Do that consistently and the second provider is configuration. Skip it and the second provider is a rewrite.

The demo below is a re-creation, not the original code. The service topology, the message kinds, and the ordering hazards are the real ones; the timings are chosen to be legible rather than measured.