ADR 0009: Generate the per-instrument stream registry at development time#

  • Status: accepted

  • Deciders: Simon

  • Date: 2026-07-22

This ADR is retroactive: it records a decision already implemented, written up after the fact.

Context#

NeXus-rich instruments (bifrost, dream, estia, loki, nmx, odin, tbl) each carry O(100) f144 log streams — motion, sample environment, choppers. Every one needs its NeXus path, Kafka topic, source, and units recorded as an F144Stream in Instrument.streams (src/ess/livedata/config/stream.py) before the corresponding workflow can be wired up. That data already exists, authoritatively, as group attributes in the instrument’s NeXus geometry file — the same file the filewriter and essreduce consume.

Both the backend services and the dashboard import an instrument’s config.instruments.<inst> package at startup to build Instrument. The dashboard in particular has no file-access path at that point: it runs wherever a user’s browser can reach it, not necessarily co-located with a NeXus geometry file or a Pooch cache.

Decision#

Extract the f144 stream declarations from a NeXus geometry file offline, via the CLI in nexus_helpers.py (python -m ess.livedata.nexus_helpers <geometry.nxs> --generate), and check the generated config/instruments/<inst>/streams_parsed.py into git as an ordinary Python module. The instrument’s hand-edited specs.py imports the generated PARSED_STREAMS dict and composes the final registry via name_streams, supplying renames and any hand-written synthetic entries.

Instruments with few streams (today: dummy) skip codegen entirely and hand-write the dict[str, F144Stream] literal.

Alternatives considered#

Option

Notes

Codegen from NeXus at development time, checked in (chosen)

Both backend and dashboard start from a plain importable module — no file I/O, no network, no cache-state dependency at import time. The generated diff is reviewable like any other code change against the previous geometry.

Parse the NeXus geometry file at process startup

The dashboard has no guaranteed file-access path to the geometry file; it would need a Pooch download, adding cold-start latency and a runtime dependency on cache state and network reachability that the dashboard does not otherwise have. Rejected.

Hand-maintain the registry (YAML or Python) for every instrument

O(100) entries per instrument, sourced from filewriter attributes nobody wants to retype. Selectively already done for dummy, where the entry count is small enough that hand-maintenance is cheaper than adding a geometry file just to run codegen. Rejected as the general approach.

Key design choices#

Generated output is a plain Python module, not a data file#

generate_streams_parsed_module emits a .py file with an F144Stream dict literal rather than YAML or JSON. It imports and type-checks like any other module, needs no bespoke parser in specs.py, and diffs the same way as hand-written code in review.

The generated dict is keyed by NeXus path, not instrument-facing name#

streams_parsed.py only extracts what the geometry file states directly; assigning instrument-facing names (auto-suggestion, collision handling, device detection, renames) is name_streams’s job in specs.py. This keeps the generator a thin, deterministic extraction step and puts every naming decision in one reviewable place per instrument.

Consequences#

  • streams_parsed.py is regenerated by rerunning the CLI against an updated geometry file and committing the diff; nothing currently detects staleness against the latest geometry automatically. An earlier drift test was dropped because it depended on a geometry file not available in CI’s Pooch cache, so it skipped in CI and gave false confidence rather than catching the realistic failure mode. A real check needs a stable geometry-file input pinned in Pooch first.

  • Stream is a frozen dataclass that round-trips through any serialisation layer, so a future move to runtime or dynamically-distributed stream data would only change who populates Instrument.streams, not the shape consumers see.