ADR 0001: Relagate NXspe support#

  • Status: accepted

  • Deciders: Greg, Jan-Lukas

  • Date: 2024-10

Context#

Targeting community supported data formats will enable efficient use of existing analysis and visualization tools. A number of histogram formats have been listed for possible support:

SQW Prioritization and Challenges#

One of which, SQW, has been identified as higher importance because it will enable interfacing with the PACE tools.

The major drawback to the SQW format is that it is documented most thoroughly by its MATLAB serialization/deserialization routines, which are less than straightforward to replicate in part due to an extensive class hierarchy, single-source backwards compatibility over multiple file format versions, the ability to mix definition and implementation of functions and place either in at least three different locations (the rundata helper class seems to be spread over six), and the lack of a free IDE that can help sort out the runtime behavior.

Recently it was posited that, since Horace can read NXspe files and produce a valid SQW file from them, perhaps focusing on their output from essspectroscopy could be prudent.

Bending NXspe to Indirect-Geometry Time-of-Flight#

NXspe is the NeXus application definition implementation of the columns-of-text SPE data format.

SPE format files contain energy-transfer (or time) histograms of intensity, one per detector element in the instrument. Information about the positions and sizes of the detector elements was contained in a separate file, either PAR or PHX. The format was defined with Direct-Geometry (DG) Time-of-Flight (TOF) spectrometers in mind where a single incident energy applied to all data, and all detector elements shared the same energy-transfer histogram bin edges. This information was kept separately from the two text files, and put into analysis scripts by the user.

NXspe incorporates all information contained previously in SPE and PAR files plus minimal information that otherwise may have gone into a log book. It represents an appreciable improvement over the previous status-quo for DG-TOF.

For Indirect-Geometry (IG) TOF instruments with a single final neutron energy, like back-scattering instruments, it is also possible to histogram all detector energy spectra with a single set of bin edges. By means of a flag, Horace can exchange Ei and Ef in early data loading calculations, and then otherwise treat conversion more-or-less the same to produce intensity of (Q,E) observations binned, not unlike scipp, in a coarse 4-D grid.

The problems start, however, when not all detectors share a single final energy, as with BIFROST. A single incident wavelength range applies for all BIFROST detector elements, so one Ei bin-boundary set would naturally apply for all detectors. But this is far from fixed Ei or fixed Ef with a single energy transfer, E, bin-boundary set.

Imaginary observations#

The closest BIFROST data can get to ‘standard’ NXspe E binning is to stitch together the single Ei ranges offset by each Ef. This has three drawbacks

  1. The number of bins per detector element will consequently be larger

  2. The sizes of the bins may not match well to the ‘natural’ incident energy bin sizes for all detector elements

  3. Imaginary observations will be created for all detector elements.

Note: The term imaginary is used here to indicate that no measurement was performed for the incident neutron energies needed to produce the E in one or more bin.

Natural E binning#

Instead, by keeping per-detector-element E binning the adherence to the NXspe standard can be maintained but software like Horace struggle to handle such data — the required /entry/data/energy becomes 2-D but Horace expects (and reshapes) this data to be 1-D.

One symptom of this incompatibility is excessive memory usage. For the 13500 BIFROST detector elements, with energy transfer binned into 103 bins, in an early conversion step Horace requires a 13500 × 104 × 13500 array of double values, and while ∼ 141 GB is not an insurmountable value it’s also not necessary.

In testing, moving to a machine with 256 GB of memory was not sufficient for the conversion to proceed. And it seems likely there are more incompatible assumptions made about the content of the data for Horace to successfully produce an SQW object from the ‘BIFROST-NXspe’ files.

Decision#

It is now possible to make BIFROST data conform to the NXspe standard published by the NeXus Foundation, but community software that reads NXspe is likely to have made similar assumptions about the shape of required data as Horace. So the utility of NXspe as a common data format is likely limited.

Producing NXspe may still be useful, but it can not replace direct output of SQW data files.

∴ Relegate NXspe to lesser-supported output formats.

Consequences#

A fully-fledged SQW output method is still required, which will still pose the same problems of construction and maintainability.