Replicated key-value stores hold copies of the same data on many machines and are upgraded without downtime by replacing nodes one at a time. During the rollout the cluster is mixed: machines running old and new software both serve the same keys. Published incident reports show that this window quietly corrupts data through stale reads, lost writes, and clients that cannot see their own writes. Linearizability, the standard check, is too slow to run live, and no public corpus of upgrade histories exists to develop a cheaper check against. MIXGUARD evaluates four correctness rules as operations arrive, over a sliding window of the last W operations, at O(1) amortised time and O(W) memory. The check costs 2.1 microseconds in isolation and 0.13% of a real etcd operation’s end-to-end latency. Two limits are stated up front and measured. A bounded window misses violations whose evidence has scrolled out of it, and the predicates are client centric, so even an unbounded window sees only the 58% of corrupted operations that ever reach a client. To label every alarm we built a 400 line simulator of a mid-upgrade cluster with per operation ground truth, cross checked against an independent replay checker, and injected six classes of version incompatibility bug taken from documented breaking changes. A gradient boosted classifier names the responsible class at macro F1 0.92 under a leakage free grouped split, and 0.91 with the predicate firing features removed. An abort rule prevents 95.6% of corrupted operations, against 13.8% for a fixed timeout policy, with zero false aborts across 60 clean upgrades. On a live three node etcd cluster the same monitor stays silent through a clean v3.4 to v3.5 rolling upgrade and fires correctly when a stalled replica breaks read your writes for pinned clients, confirming that the predicates work on real responses. We calibrate the simulator’s timing on a real ZooKeeper trace and release the benchmark generator MIXWIN.
Post comments