Declarative session reconciler — migration plan#

Companion to the declarative session reconciler proposal. Four phases, each shippable and useful on its own; phase 2 doubles as the go/no-go spike for the rest.

        flowchart LR
    P0["Phase 0<br/>characterization<br/>tests"]
    P1["Phase 1<br/>occupancy from<br/>topology"]
    P2["Phase 2<br/>desired() + differ<br/>(the spike)"]
    P3["Phase 3<br/>fold in flush +<br/>freshness"]
    P4["Phase 4<br/>stable widget shell<br/>(speculative)"]

    P0 --> P2
    P1 --> P2
    P2 -- "criteria met" --> P3 --> P4
    P2 -- "criteria missed" --> Stop(["stop, keep 0+1,<br/>write down why"])

    classDef ship fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20;
    classDef spec fill:#fff3e0,stroke:#ef6c00,color:#e65100;
    class P0,P1 ship;
    class P2,P3 spec;
    class P4 spec;

Phase 0 — characterization tests#

Status: implemented in #1229.

Lock in what the current pass does before changing how it does it. The tests from #1220 (hidden cell not built, reveal builds once, watcher preserves pre-warm, deferred rebuild) and #1226 (pipe subscribers severed on dispose, subscriber count flat across a rebuild) are the seed; add:

Tab reconcile: focus preserved across rebuilds, local-creation focus, disabled-grid tab omission, mid-removal identity lookups.
Orphan sweep: cell removed from topology → widget removed and disposed; disabled grid’s cells survive for re-enable.
Flush gating: one flush per frame generation, tab-switch flush without a generation change does not consume the next frame’s flush.
Freshness: pill updates on flush/rebuild, stall aging at the 2 s cadence, no double-update beat.
Predicate/pass agreement: for each gate, “pass would do work ⇒ predicate fires” (this test is only writable per-gate today; under phase 2 it becomes a single generic property).
Resource symmetry: dispose releases everything build acquired — total pipe-subscriber count across a session’s layers is flat across rebuilds and removals, and hv.DynamicMap callback executions per pass stay flat (#1224’s acceptance checks; cheap to instrument in the pass).

Also enumerate the framework-trap invariants that tests cannot easily cover as a checklist in the module docstring (they are currently scattered across IMPORTANT: comments): modal container has one stable parent at top level; wake registration deferred until browser load; batching around multi-widget updates; disposal breaks toolbar reference cycles and severs pipe subscriptions before the replacement renders (holoviews severs all weak plot-refresh subscribers on the touched streams, so ordering carries the correctness — #1224); markup-pane reveal workaround.

Value: independent of this proposal — it is the safety net any future change to this module needs.

Phase 1 — occupancy from topology (#1219, structural half)#

Status: merged as #1221.

PlotGrid stops deriving free positions from inserted widgets. Instead it receives a callable returning the grid’s topology geometries (wired to the orchestrator by PlotGridTabs), and the second click of the selection gesture re-validates the region before opening the wizard. _occupied_cells shrinks to “which widget sits at which geometry” (needed for removal/replacement) and no longer feeds selection decisions.

Small, local, already agreed as the right fix in #1219. Kills the stale-display path (empty cells offered over positions topology holds). One residue is unavoidable by construction: two sessions can still complete the wizard simultaneously on the same genuinely-free region, so the loser hits the overlap ValueError — #1220’s error handler stays necessary as the last line of defence, and no occupancy model removes it.

Phase 2 — `desired()` + differ for structure and materialization#

Status: implemented in #1230 (desired_cells in dashboard/cell_plan.py; plans compared against the build inputs each widget records). Outcome against the criteria below: 2–5 met cleanly; criterion 1 met with one honest nuance — _last_layer_version survives, but purely as the wake stamp with no rebuild semantics, and its snapshot-early/record-late rule now applies uniformly to all stamps instead of being a per-field choreography. The build inputs turned out to need one element beyond the sketch below: has_plot per layer, because a STOPPED layer’s retained data is computed by the reveal’s first-viewer activation without a version bump — the one upgrade a counter cannot express.

The spike. Scope: concerns 1–4 of the current pass (tabs, cell composition, layer lifecycle, tokens) move to the new shape; flush (5) and freshness (6) stay as they are, driven after the apply step exactly as today.

Sketch of the types (final naming via the glossary before implementation):

@dataclass(frozen=True)
class LayerBuildInput:
    layer_id: LayerId
    plotter: object | None      # identity comparison only
    has_plot: bool              # plotter holds a computed frame

@dataclass(frozen=True)
class CellPlan:
    grid_id: GridId
    geometry: CellGeometry
    materialize: bool
    build_inputs: tuple[...]    # composition + per-layer LayerBuildInput

def desired(
    topology: Mapping[GridId, PlotGridConfig],   # orchestrator snapshot
    layer_states: Mapping[LayerId, LayerSnapshot],
    view: SessionView,                            # active grid, modal open
    watched: Callable[[LayerId], bool],           # other sessions' tokens
) -> dict[CellId, CellPlan]: ...

The differ compares CellPlans against the applied record (which stores each widget’s build_inputs) and emits build/insert/dispose actions; token acquire/release is derived from the same plans (materialized-and-visible ⇒ hold tokens). The wake predicate becomes: input version stamp ≠ stamp at last apply, plus the freshness time term.

Deleted on success: _cell_signatures, SessionLayer.last_seen_version (as a rebuild trigger), _last_layer_version and its before/after choreography, the rebuild half of _has_pending_work, and the deliberately-stale-records device from #1220. _cell_grid folds into the applied record. _tab_composition and the tab-focus logic can stay initially — tabs are a small, self-contained concern.

Order-of-operations invariants to carry over explicitly (they do not disappear, they become documented steps of the apply stage):

Token 0→1 computes synchronously; sample time bounds and flush after activation so a reveal renders on its own pass.
Rebuild+insert runs inside the same batched update as removal, in one pass.
Records update only after a successful build (exception ⇒ retry next tick) — under the differ this is automatic: a failed build leaves the applied record unchanged.

Acceptance criteria (agree before starting, judge after ≤ 1 week)#

≥ half of the bookkeeping fields deleted (measured against the inventory).
The hand-mirrored rebuild predicate deleted, replaced by the generic stamp comparison.
All phase-0 characterization tests and the browser smoke tests pass unchanged (behavioral equivalence, not “tests updated to match”).
No new framework workaround was needed (a rewrite that trips a new Panel/Bokeh trap has negative value).
desired() has direct unit tests with plain data and no Panel imports.

Any criterion missed → stop, keep phases 0–1, record the reason in this document.

Test-suite impact and cleanup#

The proposal’s testing claim deserves the same scrutiny as its code claim, so here is what the suite looks like on each side of phase 2.

Today (~2100 lines across plot_grid_tabs_test.py and plot_grid_tabs_layout_test.py):

Every test pays the full integration price, including pure policy questions (“is this cell rebuilt when X changes?”): an eleven-fixture service chain (DataService → StreamManager → PlottingController → PlotOrchestrator → … → PlotGridTabs), a Bokeh/HoloViews extension load, and eight ad-hoc fake classes, several redefined per test class.
The _tick helper runs the pass only when _has_pending_work fires, so every behavioral test silently doubles as a guard on the mirrored predicate — deliberate and valuable today, but it means test intent (policy? mechanism? predicate?) is not readable from the test, and tests for documented predicate holes must bypass the helper and call the pass directly.

After phase 2, tests separate along the same seam as the code:

Policy tests become plain-data tests of desired(): construct topology, layer snapshots, and session view as literals; compare the returned plans. No fixtures, no Panel import, table-driven; a fake plotter reduces to a sentinel object compared by identity. This absorbs the bulk of the rebuild/visibility scenarios — including the #1216/#1220 gate scenarios, which today each need the full stack.
Differ tests are a small fixed set for the generic rules (build, rebuild on changed inputs, dispose, deferred no-op, failed build leaves the applied record unchanged → retry). Written once; future policy work never touches them.
Integration tests remain for what is genuinely integration — tab-strip reconcile and focus, widget insertion/disposal, toolbars, overlay composition, teardown — but stop re-litigating policy through the UI, so they shrink in count while keeping the eleven-fixture chain they legitimately need.
The predicate guard collapses to one property: work is pending iff input stamps differ from the applied stamp — testable directly, retiring both _tick’s dual role and the direct-call escape hatch.

Sequencing. During the spike, none of this happens: criterion 3 freezes the existing tests as the equivalence oracle. The reorganization is a follow-up after acceptance, and a characterization test is retired only when a named desired()/differ pair replaces it.

Outcome. The reorganization landed after acceptance and matched the sketch on three of four points: policy scenarios are plain-data tests of desired_cells, the surviving integration tests state which mechanism they are about (TestCellDiffer for the differ rules, TestCellActions for clicks and cross-session races, TestMaterializationWiring for the two things only the real chain can show), and the predicate guard collapsed into TestWakeGateContract — now the only place the gate is exercised, so it covers every stamped term explicitly and asserts the two deliberate holes (a foreign viewer token, and has_plot flipping without a transition) that were previously only prose.

The fourth point came out differently: the differ has no separate unit to test, because #1230 kept it inline in the applier — three comparisons over the plans, not a function returning actions. Its rules are therefore exercised through the real widgets, as a small fixed class. Extracting an action-emitting differ purely to make it unit-testable would add an intermediate representation nothing else needs; revisit only if a second applier ever appears.

Cleanup worth doing regardless of the go/no-go (fits phase 0): move the eight ad-hoc fakes into shared test helpers, and hoist the fixture chain into a conftest.py so the phase-0 characterization tests do not copy it a third time. Neither depends on the rewrite.

Phase 3 — fold in flush and freshness (optional)#

Model the data flush as part of the desired state (“grid G should display frame generation N”) and freshness as a derived output. Only worth it if phase 2 leaves flush/freshness as the odd ones out complicating the tick entry points; they are edge-triggered by nature and may legitimately stay a small imperative tail. Decide on the phase-2 outcome, same criteria style.

Decision (post phase 2): not pursued. The condition above is not met: after #1230 the flush/freshness tail is a compact block driven off the same plans (it iterates the active grid’s CellPlans and their snapshots), its three gates are already uniform stamps recorded like every other pass input, and the freshness stall term is inherently wall-clock. Modeling “grid G shows generation N” declaratively would add a second applied-state axis while deleting nothing. Revisit only if a future feature needs per-cell flush policy (e.g. pause).

Phase 4 — stable widget shell (speculative spike)#

Today a plotter swap rebuilds the whole CellWidget (titlebar, toolbars, panes, figure). If the shell were stable and only the figure pane swapped, rebuild cost would drop enough that much gating precision becomes unnecessary — the differ’s job shrinks further. Risk: Panel pane replacement has its own churn and known traps (reveal latch, reparenting); history in this codebase says prototype first. Strictly optional; nothing in phases 0–3 depends on it. The #1224 sever must survive this phase: swapping only the figure pane still discards a rendered plot, and the phase-0 subscriber-count invariant is the guard that it keeps being released.

Risks and mitigations#

Risk	Mitigation
Silently dropping a framework workaround	Phase 0 checklist + tests; criterion 4
Behavioral drift hidden by rewritten tests	Criterion 3 forbids test rewrites during the spike
`desired()` grows imperative side effects over time	Enforce purity: no Panel imports in its module; unit tests construct inputs as plain data
Differ becomes the new dumping ground	Differ rules are fixed and generic; policy changes go to `desired()` only — review rule to state in `.claude/rules/dashboard-widgets.md`
Spike overruns	Hard time box; abandoning keeps phases 0–1