A dirty flag beat a consensus protocol
Two systems, both able to write the same appointment, thousands of clinics apart. The overwrite race that quietly ate users' changes looked like a distributed-locking problem. It was a boolean and a rule that reads it — and I'd make that trade again.
Our platform and the clinics' external clinical systems both hold appointments, and both can change them. A front-desk staffer reschedules inside the external system; a patient books or cancels through our platform. For the product to make sense, the two worlds have to agree — in both directions, continuously, across thousands of clinics.
"Keep two systems in sync" sounds like a checkbox and is actually a swamp. The moment both sides can write, you're in distributed-systems territory whether you wanted to be or not: propagation delay, acknowledgement, retries — and the one that ruins your week, what happens when both sides touch the same record at nearly the same time.
Two writers and a race
sounds like it demands heavy machinery.
The original sync was one-directional and batch-shaped — a reasonable v1. External systems periodically pulled appointments from us over HTTP and reconciled. But it was built to communicate created appointments; updates and deletes were second-class, and a periodic poll gives you neither promptness nor a clean story for changes originating on our side. As our platform became a real place to reschedule and cancel, two failure modes surfaced — worth separating, because they have different fixes.
One — changes got stuck or lost on the way out. With only a periodic pull and no acknowledgement, there was no answer to "did this change actually land over there?" A change could sit pending forever, indistinguishable from lost.
Two — the nasty one: overwrite races. A patient changes an appointment on our side. Before that reaches the external system, its periodic pull hands us its version — the stale one. Apply it naively and we overwrite our own un-synced change with old data. The user's action silently evaporates.
That second failure erodes trust one confusing incident at a time. "I definitely rescheduled that — why did it revert?" No error, no crash. Just a lost write, because two sources of truth disagreed and the wrong one won.
the fix criterion isn't "make it faster" or "lock something" — it's "don't let inbound stomp an un-acked local change"
There's nothing to profile here. The investigation was modeling the flow of truth precisely enough to see where it broke. An outbound change and an inbound pull are two events racing; the bug is one specific interleaving. Draw it, and the fix criterion stops being about speed or locks.
That reframing is the whole ballgame. The problem isn't concurrency in the classic shared-memory sense — it's that the inbound path lacks a piece of knowledge: "this record is dirty on our side," which would let it make the right call.
Most people — me too, on a bad day — hear "two writers, same record, race" and immediately think locking or consensus. But locks across a boundary you don't control are a nightmare, and consensus is absurd overkill. The real output of the investigation was recognizing this needed a state marker, not a coordination protocol.
A change originated on our side and hasn't been acknowledged yet.
No pending local change; both sides believe the same thing.
"local change wins while it's pending; the other side retries until it catches up" — no locks, no consensus, no distributed transaction
The whole consistency model fits in a sentence on purpose. I distrust elegance — but this earns it, because the elegance is the reduction: from a coordination protocol to a boolean and a rule that reads it.
The instinct to reach for the sophisticated thing is exactly what I had to reject.
More frequent batch pull
RejectedPolling harder gives no near-real-time outbound, no acknowledgement loop, and nothing against the overwrite race. It just does the inadequate thing more often.
Fire-and-forget outbound push
RejectedNo acknowledgement means no way to know it landed, no basis for retry, and still no protection against inbound clobbering. Speed without an ack loop just makes the loss faster.
Distributed locks / coordination protocol
Rejected firmlyYou cannot reliably hold a lock across a boundary to an external, on-premise system that can be slow, unreachable, or down for maintenance. A held lock against an offline participant is a deadlock waiting to happen — a sledgehammer where the problem needs a light switch.
Exactly-once transactional two-way sync
DisproportionateThe heavyweight "correct" answer. Guarantees stronger than the problem needs, high cost, and it assumes a level of control over the external systems we simply don't have.
Dirty-flag guard + ack loop + pull fallback
ChosenTrack whether a record has an un-acked local change; have the inbound path respect that flag; use a real-time channel for prompt propagation and keep the pull as a reliable fallback; close the loop with an explicit acknowledgement. Light, robust, and honest that the external world is unreliable.
Three cooperating mechanisms, each doing exactly one job.
The dirty-flag guard — the heart of it
Every appointment carries a marker: "has an un-synced local change." Set it true when a change originates on our side. The inbound path is taught one rule — skip any record whose marker is set. When the external system finally acknowledges our change, flip it back to false and inbound processing resumes.
Real-time propagation, with a pull fallback
Our-side changes push to the external systems in near-real-time. But real-time delivery is best-effort — an on-premise system can be offline — so the periodic pull remains as the reconciliation path. The fast path is never the only path. (The real-time transport itself was a platform choice; mine is the sync logic that rides on it.)
The acknowledgement loop — and updates/deletes done right
I rebuilt the pull so a single query returns created, modified, and deleted appointments, each tagged by the datastore with its change kind — one query, correctly typed results, instead of multiple application-side passes. A status write-back records completed or failed, so "in flight," "done," and "lost" are finally distinguishable. (The status/queue concept was co-designed with the scheduler team; I built our side.)
Ask what one side doesn't know
before asking how to lock it.
Outbound is fast with a safety net. Inbound is safe because it reads one flag.
The dirty flag sits between the two directions as the arbiter. Everything else is transport and bookkeeping around that single piece of knowledge.
reschedules, cancels
inbound skips if dirty
ack flips dirty=false
Outbound changes push promptly; the pull is the fallback that makes the fast path safe to rely on.
Created / modified / deleted returned already tagged by the datastore — classification pushed down, not done in app code.
The skip-if-dirty rule is honored across every inbound path, including the per-vendor handlers — a guard in nine of ten paths is a race with better odds.
Local-change-wins while pending
Two propagation paths
Unbounded retry, for now
The honest framing is "local-change-wins-while-pending," not "we handle every conflict perfectly." Naming the edge is part of the trade.
Distributed locks in the design. The overwrite race — the trust-eroding one where a user's change silently reverts — is structurally prevented by the dirty-flag guard, not coordinated away.
correctnessFailure modes eliminated. Outbound changes reach the external systems near-real-time with a reliable pull fallback; updates and deletes are first-class, carried by a single well-formed query.
reliabilityClinics kept consistent with their external systems, in both directions, continuously — without a single distributed lock. (Per-day change volume and a before/after on "stuck pending" are numbers I'd pull from monitoring, not assert from memory.)
reachThe core idea — a dirty flag and a retry instead of a lock and a consensus — I'd keep without hesitation. Three things I'd finish.
Bound the retry; define the unreachable case
The one honest soft spot is what happens when a system never acknowledges. I'd add an explicit policy — alert after N failed attempts, a distinct "chronically unreachable" state, operator visibility — so the tail case is designed rather than incidental.
Make the invariant structural, not disciplinary
Today "respect the dirty flag" is a rule enforced across many handlers — a new one could forget it. I'd route all inbound application through a single choke point that checks the flag once, so the guard can't be accidentally omitted. Turn a discipline into a structural guarantee.
Build the sync-health observability
Per-client, per-direction: is sync flowing? how stale is it? how many records are dirty and un-acked, and for how long? That dashboard turns "a client reports their changes aren't syncing" into "we saw it first, and here's exactly which records."
The instinct I'd want a reader to leave with isn't the flag itself — it's the reflex to ask how little coordination the problem actually requires before reaching for the sophisticated thing. The best distributed-systems solution is usually the least distributed one you can get away with.