# ADR 0005: Frame-gated per-session plot flush - Status: accepted - Deciders: Simon - Date: 2026-06-25 ## Context Reduced data reaches the browser through two dashboard threads. A shared ingestion thread (`DashboardServices._update_loop`) drains a Kafka batch, forwards it through `DataService`, and runs `Plotter.compute` synchronously for every visible layer -- the compute records `created_at` and marks the layer's presenter dirty. A *separate* per-session callback (`pn.state.add_periodic_callback` → `SessionUpdater.periodic_update` → `PlotGridTabs._poll_for_plot_updates`) then pushes the dirty presenter's data to that session's HoloViews `Pipe` via `pipe.send`. The push could only happen on the session callback because session-bound objects (`hv.streams.Pipe`, `DynamicMap`, `pn.io.hold`) **must** be mutated in their own Bokeh document context, on that session's IOLoop. Panel cannot even resolve the right session context from a background thread: `pn.state` reflects the *current* session, which is absent off the session IOLoop (Panel #5488). Mutating session-bound objects from the ingestion thread therefore corrupts the document and misroutes updates to the wrong tab -- the empirical failure that drove the per-session polling architecture (original root-cause analysis: `git show 2b349005a:docs/developer/plans/multi-session-architecture.md`). This is a hard constraint, not a preference. The session callback ran at a fixed 1000 ms period, unrelated to data arrival. Consequences (issue #1011): - A freshly computed frame sat undisplayed for up to one poll period. The freshness pill (data age, `now − data_end`) and the per-layer pipeline lag (`created_at − data_end`) read the *same* `TimeBounds`, so the observed gap was exactly `now_poll − created_at` -- the undisplayed wait. A ~1.1 s pipeline lag showed as a ~2.0 s pill. - The fixed period doubled as a frame synchronizer: `periodic_update` wraps the whole pass in `pn.io.hold()` + `doc.models.freeze()`, collapsing every layer's `pipe.send` into one WebSocket flush. Naively shortening the period would scatter a burst's layers (whose computes finish at staggered times) across ticks, producing distracting cross-plot stagger. The design problem: cut the latency without losing per-frame synchronization and without pushing from the ingestion thread. ## Decision Decouple the flush *trigger* from the clock while keeping the flush *itself* on the session thread, coordinated by a frame counter. A `FrameClock` (one per `DashboardServices`, shared with `PlotOrchestrator`) carries a `generation` counter keyed by grid (tab). As a burst drains, the ingestion thread buckets each visible layer's due recompute by grid (`PlotOrchestrator._enqueue_compute`) rather than computing inline. Once the burst is drained (after each `Orchestrator.update()` in the loop), `flush_frames` runs the buckets grid by grid and `commit(grid_id)`s each grid the moment its own layers finish -- so a session showing one tab sees its frame without waiting on any other tab's compute. The per-session poll period drops to 100 ms, but `_poll_for_plot_updates` gates the data flush (`update_pipe` → `pipe.send`, plus the per-layer time/lag row) on the *active grid's* generation having advanced since this session last flushed, or the active tab having changed. The cheap per-tick work -- lifecycle/version scan, layer activation, freshness-pill aging -- still runs every tick. All session-bound mutation stays inside the periodic callback, so the threading constraint holds. The freshness pill refreshes in step with the data flush (reading the true lag of the frame just shown) and otherwise on a slow stall cadence (`_FRESHNESS_STALL_INTERVAL_S = 2 s`) so a stalled stream still visibly ages. ## Alternatives considered - **Push from the ingestion thread** (event-driven `pipe.send`, or `doc.add_next_tick_callback` onto each captured session document). Rejected: it reintroduces the cross-thread session mutation the polling architecture exists to avoid. `pn.state.execute` does not help -- it targets the *current* session context, useless from the ingestion thread. - **Naively lower the fixed poll period.** Rejected: each pass stays batched, but computes finishing in different sub-windows flush on different ticks, so multi-plot updates stagger. - **User-selectable poll period.** Rejected: it exposes the latency/stagger trade-off to the user instead of resolving it; the low-latency setting still staggers. ## Key design choices - **Generation gate, not push.** Synchronization needs a flush *per data burst*, not a slow timer. The burst boundary is data-defined (`commit(grid_id)` on drain), so gating a fast pull on it gives both low latency and one-flush-per-grid-per-burst batching. A burst only *approximates* a backend frame -- see *Scope and limits of synchronization*. - **Visibility falls out of the existing compute gate.** A build is due only for layers holding a viewer interest token (the active tab), so bucketing the due builds inherently tracks only visible recomputes. Hidden tabs stash without computing and never bucket, so their generation never advances. - **Generation keyed per grid, not global.** With multiple sessions each showing a different tab, a single global counter would wake every session on any tab's data: harmless for the data push (dirty-gated, so it shows nothing wrong) but it would re-tick each session's freshness pill on other tabs' frames, undoing the per-frame pill cadence. Keying the generation by grid means a session is woken only by bursts in the tab it is displaying. - **Commit per grid, not once per burst.** Bucketing the burst's computes by grid and committing each grid as its layers finish means a session waits only on its own tab's compute, never the union of every visible tab's compute across all sessions. Deferring dispatch out of the inline `DataService._notify` to the post-drain `flush_frames` also stamps `created_at` closer to display time. - **`pipe.send` stays dirty-gated** by `has_pending_update`, so it fires at most at the data rate regardless of poll frequency. The faster poll therefore adds only cheap no-op scans, not extra Bokeh model work. - **Flush after `activate_layer`**, so a tab-switch 0→1 synchronous build is pushed on the same tick. - **Stall cadence above backend cadence.** With the stall interval equal to the ~1 s publish cadence, the stall tick beat against the flush and double-updated the age; keeping it at 2 s lets each flush reset the timer first, so a healthy stream updates the pill once per frame. Slower-than-2 s streams age the pill, which is informative rather than distracting. ## Scope and limits of synchronization The guarantee is *intra-grid within a burst*: one `commit(grid_id)` collapses that grid's layers computed from one drained batch into a single `hold`+`freeze` flush on each session showing it. Because `get_messages` drains the whole consumer queue, a burst coalesces however many Kafka batches accumulated since the last `update()`. What that burst aligns with on the backend rests on the following assumptions, which are reasoned, not measured: - **No backend frame exists.** Compute is partitioned into independent workers (detector-view, monitor-histograms, data-reduction, device-metadata), each publishing ~1 Hz at its own phase, some slower (e.g. motor positions). They are not mutually synchronized and carry no cross-worker frame marker -- the backend never declares "these results are one moment." Per-frame sync is therefore an approximation made at the dashboard, not a property of the data. - **Sync only matters within a tab**, since the gate is per grid and a session views one tab. Tabs are *typically* fed by a single worker, whose outputs are computed per tick and published together. The working assumption is that such a worker's co-published outputs arrive close enough to drain into one burst, so the cells sharing a moment flush together. We have *no hard proof* that a worker's output set always lands in one consumer batch / one `update()`. - **Cross-worker data is not synchronized** (different phases, sub-Hz sources). Accepted: those sources share no backend frame and mostly live on separate tabs that the per-grid gate isolates. A custom tab mixing workers gets no cross-worker alignment -- but neither did the old fixed period, which only aligned sources by the luck of its 1 s bucket phase, never true coherence, and at up to 1 s latency. These expectations are unverified. We also lack a current measurement of how real ~1 Hz traffic (with ~200 ms compute per batch) partitions into bursts -- one burst per second plus empty ticks, or several. The design degrades gracefully if an assumption fails: a torn frame staggers by one ~100 ms tick and self-corrects. Partial sync is the accepted fallback; real-world use will confirm or refute the above, and this section should be revisited if intra-tab stagger proves visible. ## Consequences - The *polling* component of display latency drops from up to one 1000 ms period to roughly one 100 ms tick. For a single visible layer that is the entire wait. Committing per grid also removes *cross-tab* interference: a session no longer waits on other sessions' tabs' computes before its own frame commits. The residual wait is the *within-tab* serial compute -- a grid's layers still run sequentially on the ingestion thread before that grid commits -- plus the 100 ms tick. Parallelizing that per-grid compute is a separate concern, out of scope here. - The per-layer pill lag (`created_at − min_end`) and the headline age (`now_flush − min_end`) differ by exactly the display latency, so they coincide only for the last-computed layer of a grid (and any single-plot tab); the first-computed layer's age sits a full grid-compute above its lag. The end-to-end age is identical for every layer in a grid, since one flush shows them all. - A grid's layers from one burst flush together (one `hold`+`freeze` flush per grid generation). This is intra-grid only; cross-grid, cross-burst, and cross-worker alignment are not guaranteed -- see *Scope and limits of synchronization*. - `PlotOrchestrator` gains a `frame_generation(grid_id)` accessor and a `flush_frames` that runs the per-grid compute buckets and commits each grid; `DashboardServices` owns the `FrameClock` and calls `flush_frames` after each drain; the ingestion idle sleep dropped to 50 ms so burst detection is not the new floor. - The pill's stall cadence is coupled to the backend publish cadence. This is accepted: at slower cadences ticking the age every 2 s is reasonable. - A logical frame can split across bursts when its arrival straddles an `update()` cycle boundary -- made marginally likelier by the ~200 ms serial compute per batch (observed in practice) holding the loop. The result is a one-tick (~100 ms) stagger: acceptable and self-correcting. How often this happens in real traffic is unmeasured.