ADR 0007: Single-writer state with versioned pull as the dashboard concurrency model#

  • Status: accepted

  • Deciders: Simon

  • Date: 2026-07-20

Context#

The dashboard runs on two kinds of threads: the shared ingestion thread (DashboardServices._update_loop, draining Kafka and driving orchestrator updates at ~50 ms), and each session’s Tornado/Bokeh IOLoop (UI callbacks and the 100 ms session poll, under that session’s document lock; the fixed-cadence poll has since been replaced by wake ticks plus a slow housekeeping tick — see the end of ADR 0005). Layer activation runs on the latter — the poll pass calls PlotDataService.set_active, so activation flushes and Kafka-delta builds enter the same gate from opposite threads. Session-bound objects (Pipe, DynamicMap, Bokeh documents) may only be mutated on their own session’s IOLoop — a hard constraint established in ADR 0005.

One path crosses that boundary by design: the stale-session reaper runs on the ingestion thread (SessionRegistry.cleanup_stale_sessions), so SessionUpdater cleanup handlers execute off their own session’s IOLoop and must be safe there.

Historically, state changes propagated between these threads by two competing mechanisms:

  1. Synchronous push: callbacks fanning out from the mutating thread, carrying data or lifecycle events — DataService extracting and delivering per subscriber inside ingestion (_notify), the JobOrchestrator subscription registry notifying LayerSubscription / PlotOrchestrator on job changes, and PlotOrchestrator lifecycle notifications rebuilding every session’s grid widgets from the mutating session’s thread.

  2. Versioned pull: a single writer updates authoritative state and bumps a version counter; each consumer polls the version on its own clock and pulls a snapshot when it changed — the pattern prescribed in .claude/rules/dashboard-widgets.md, followed by the workflow-status widgets, and (since ADR 0005) by the frame-gated session flush.

The 2026-07 audits (#1039, #1044; standalone issues #1040, #955; assessment #1042) showed the two mechanisms have sharply different defect profiles. Every confirmed cross-thread race and session leak was located in push machinery, and the audited compute waste concentrated in its delivery path: registration racing ingestion, notify loops iterating live sets mutated by the UI thread, cross-session document mutation outside the document lock, the viewer gate and stash bolted onto the push path along with their documented races, and the rebinding chain that exists only to push “the job key changed” to consumers. The code built on versioned pull had no concurrency findings (it has ordinary bugs like any code — e.g. buffer edge cases — but no races). The same failure was fixed once before (#1007), in push code.

The structural reason: under push, thread-safety is a per-call-site obligation. Each new entry point (add-layer-while-running, heartbeat deactivation, the stale-session reaper) must remember every guard, and the audit trail shows each one independently missed some. Under versioned pull, safety is a property of the pattern: the writer is single, the version bump is atomic, and consumers read snapshots on their own thread.

Decision#

Single-writer state with versioned pull is the dashboard’s concurrency and delivery model. Push is the exception and requires explicit justification.

Concretely:

  • Every piece of shared state has exactly one writing thread. Other threads request changes by enqueueing for the owner or mutating via a narrowly-scoped, documented lock — never by direct multi-step mutation.

  • State changes are observed by version, not by callback. A writer bumps a monotonic version (per key, grid, or subsystem as granularity requires); consumers poll the version on their own clock (session poll, frame flush) and pull an immutable snapshot when it advanced. Callbacks, where they survive, carry no data — at most “something changed” wake-ups that degrade to the next poll tick if lost. The sanctioned instance is the ingestion thread scheduling a data-free session tick via doc.add_next_tick_callback once flush_frames has committed a grid: only the scheduling crosses threads, the woken tick body runs in the owning session’s document context, and a lost or duplicated wake is harmless because the tick is idempotent and generation-gated. ADR 0005’s rejection of add_next_tick_callback scopes to data-carrying push and to pn.state.execute, not to wake-ups — see its 2026-07-06 amendment.

  • Data flows by pull at display cadence. Ingestion records what changed (dirty keys); extraction, copy, and compute happen when a consumer with a viewer pulls — not eagerly per delta on the ingestion thread. “No viewers, no work” is the default, not a gate.

  • Session-bound mutation happens only on the owning session’s IOLoop (reaffirming ADR 0005), including teardown paths such as the stale-session reaper.

  • Subscription identity is stable. Keys a consumer subscribes under must not embed ephemeral tokens (e.g. per-commit job_number); generation discrimination is a filter at the ingest boundary, not part of consumer identity. (Assessment: #1042, Option B.)

Alternatives considered#

  • Keep push, add locks (the direction of the standalone lock/gate fixes, e.g. serializing DataService, guarding PlotOrchestrator topology). Rejected as the end state: each lock protects one call-site pattern, the lock-ordering surface grows (ingestion_guard → service lock → document lock), and nothing prevents the next feature from bypassing the guards — the audited failure mode. Locks survive, but narrowly, protecting buffer writes rather than delivery.

  • Marshal all mutation onto session IOLoops (add_next_tick_callback everywhere). Rejected: fan-out to N sessions multiplies the work per change, the mutating thread ends up owning each session’s update order, and shared (non-session) state still needs a single writer regardless. Scheduling itself is sound — hence the data-free wake-up above — but it is a trigger mechanism, not a state-ownership model.

  • Full actor model / message-passing rewrite. Rejected as disproportionate: versioned pull achieves the same isolation for this system’s shapes (one ingestion writer, per-session readers) with the patterns already proven in the codebase.

Consequences#

  • The three push planes converge on the model (tracked in #1046): data delivery inverts to pull-on-frame (#1045, subsuming the viewer gate #1043 and shrinking the #1009 lock to buffer writes); the job-identity rebinding chain is deleted via stable keys (#1042 Option B); grid widgets move to version polling and reaper teardown moves to the owning IOLoop (#1040, #955).

  • Audited races #1039-2/3/9 and the session-leak item #1039-11 are removed structurally rather than guarded; field symptoms #789 and #714 are expected to resolve (verify after landing). (#1039-5 is a deterministic index-mapping bug, not a race — it is fixed directly, outside this program.)

  • New features inherit safety by construction: a new consumer polls a version and pulls a snapshot; a new producer bumps a version. Review can enforce a simple rule — “who is the single writer, and where is the version?” — instead of re-deriving interleavings per PR.

  • Worst-case freshness is bounded by the consumer’s poll cadence (100 ms session tick; see ADR 0005 for why this does not add visible latency over push). Where a wake-up backs the poll, the bound tightens to data cadence plus milliseconds and the poll becomes a safety net: the 100 ms fast tick — and with it the per-tick hold+freeze recompute and O(grids×cells×layers) layer scan (#1055 Target 3, #1044 scalability note) — can then be retired rather than optimized.

  • Cost: dirty-flag/version bookkeeping replaces eager delivery, and during migration both mechanisms coexist — the migration is sequenced so each step deletes more push machinery than it adds pull machinery.