ADR 0001: In-process synthesis of merged device streams#

  • Status: accepted

  • Deciders: Simon

  • Date: 2026-05-19

Context#

Motorised devices on ESS instruments publish three independent f144 substreams per device with non-synchronised timestamps:

  • RBV — readback (actual measurement)

  • VAL — target (setpoint)

  • DMOV — “done moving” idle flag

Before this change only RBV was consumed anywhere in the stack — VAL and DMOV arrived from Kafka but no workflow or dashboard view used them. The system had no notion of a “device” as a unit, so a workflow that wanted, for example, to filter out readings taken while the motor was still moving had no way to access DMOV alongside RBV. The setpoint was similarly invisible to consumers.

Goal: make each device available as one logical stream and one DataArray — RBV as data, target/idle attached as time-dim coords on a shared time axis — so that any consumer (timeseries plotting, Sciline-based science workflows) can opt into using the extra fields without re-implementing cross-stream alignment. This change does not by itself introduce any consumer that filters on idle or overlays target; it unblocks those developments.

Decision#

Merge per-device substreams in-process via a MessageSource decorator (DeviceSynthesizer), matching the precedent of ChopperSynthesizer. The synthesizer:

  1. Maintains per-device state (last-seen value + time per substream).

  2. Emits LogData messages (with optional target / idle fields populated per device configuration) on a new StreamKind.DEVICE once bootstrap is complete (every configured substream observed at least once).

  3. Suppresses configured substream messages from forwarding — for configured devices, raw VAL/DMOV/RBV no longer reach downstream handlers.

  4. Emits on every input event for a configured substream (union-anchored).

  5. Stamps each emit with max(rbv_time, val_time, dmov_time) across the synthesizer’s last-seen substream times.

A new Device(Stream) record sits in Instrument.streams alongside its F144Stream substreams; name_streams auto-detects devices by EPICS source-suffix classification. The existing ToNXlog preprocessor (dispatched on StreamKind.DEVICE with has_target / has_idle flags set per device) grows a time-indexed sc.DataArray carrying RBV as data and optional target / idle coords.

Alternatives considered#

Layer

Device first-class?

Notes

MessageSource decorator (chosen)

Real stream

Matches ChopperSynthesizer; one merged stream visible to every consumer

Per-stream preprocessor

n/a

Abstraction violation — preprocessors are single-StreamId by construction

Sciline provider per workflow

Graph artifact

Per-workflow state replication; non-Sciline consumers (timeseries) get no merged view

Dedicated aggregator service → new Kafka topic

Real (observable in Kafka)

Heavy: new deployment + schema versioning per instrument

The cross-stream coupling belongs at the layer where streams are first-class. Anywhere else, the device is either a graph artifact (Sciline) or a multiplexed preprocessor (abstraction violation).

Key design choices#

Union-anchored emission#

Emit on every input event for a configured substream, not on RBV-only or change-only. RBV cadence dominates anyway (~10s–100s Hz during motion); VAL adds ~1 event per setpoint change and DMOV ~2 per motion. Union makes motion-start and settle transitions addressable as rows in the merged DataArray. Downsampling to RBV-only is trivial for any consumer that wants it; recovering transition rows after the fact is not.

max(rbv_time, val_time, dmov_time) timestamp policy#

Each row represents “device state as known up to time T”. Monotonic by construction — out-of-order substream messages fold into state without producing dropped emits. The alternative (“use the triggering input’s time”) would drop any emit whose triggering event has an older timestamp than the last, losing both the row and the state update.

Edge case: a substream message older than the current max updates state correctly but emits at the unchanged max-time, which ToNXlog drops as a duplicate. The new substream value becomes visible at the next emit that advances max. State is never lost; only the addressability of that one transition as its own row.

Bootstrap before first emit#

Suppress emits until every configured substream has been observed at least once. Worst-case latency is bounded by the f144 forwarder’s ~10s republish cadence. Producing partial samples with Optional-padded fields would force every downstream consumer to defensively handle missingness.

idle is a coord, not a mask#

A mask would silently exclude moving samples from sum/mean/etc., biasing reductions — but in-motion readings are not invalid, they measure a different thing. Whether to filter on idle is a workflow-level analysis choice, not a data-quality verdict.

Substream suppression is unconditional#

The synthesizer suppresses raw VAL/DMOV/RBV messages for configured devices, so downstream code sees only the merged Device stream. This keeps the routing model simple — the merged DataArray is the single canonical view of the device — and forecloses a class of confusion where a consumer might bind a workflow key to a raw substream when the merged view was intended. The merged DataArray carries every substream as a coord, so no information is lost.

Auto-detection by EPICS source suffix#

name_streams classifies each f144 substream by the suffix of its source attribute (the PV name written by the f144 forwarder): .RBV → readback, .VAL → target, .DMOV → idle. Substreams co-located under one NeXus parent group are then bundled into a Device if classified RBV is present and at least one of classified VAL / DMOV is present. Parent nx_class is not consulted — conforming devices exist on NXcollimator and NXlog parents at ESS; restricting to NXpositioner would miss them.

EPICS suffixes are the primary signal rather than NeXus child names because they are fixed by the EPICS motor-record convention while NeXus child names drift across instruments (e.g. tbl uses position_readback/position_setpoint while bifrost/loki use value/target_value). The NeXus filewriter template is per-instrument and operator-editable; the source attribute is what the f144 forwarder publishes for the PV, and the IOC-side .RBV / .VAL / .DMOV suffixes are stable across the facility.

Substreams whose source is None are not classifiable and are not grouped into Devices. Production f144 streams always carry a source; only synthesised or hand-crafted test fixtures can omit it. Test fixtures must set realistic sources to participate in device synthesis.

Non-motor readback variants (-PosReadback from piezo encoders, etc.) are deliberately not in the suffix allowlist. They either appear on their own (and remain plain streams) or appear as extra siblings of a canonical .RBV/.VAL/.DMOV group (in which case they remain plain streams alongside the synthesised Device).

State persists across RunStart / RunStop#

Run boundaries are file-writing markers, not physical events — a motor’s position survives the boundary. The synthesizer holds no run-lifecycle hooks; ToNXlog (the same accumulator used for plain f144 logs) carries no-clear-on-run behaviour through unchanged.

Consequences#

  • For configured devices, the dashboard exposes one merged per-device timeseries entry instead of an RBV-only stream entry; the underlying VAL and DMOV substreams remain inaccessible as standalone plots (they only ever surface as coords on the merged DataArray).

  • A Device entry in Instrument.streams is a valid LogContextBinding target via the same dict-membership check as F144Stream — this is the migration path Bifrost uses for InstrumentAngle / SampleAngle.

  • The extra target / idle coords ride along passively through sc.lookup and broadcasting; no downstream API changes were required.

  • Detection is conservative: lone-RBV groups (no setpoint / no idle flag) are left as plain streams. Future need for setpoint-less sensor-style devices can relax the rule.