NMX Reduction Workflow#

NMX does not expect users to use python interface directly. This documentation is mostly for instrument data scientists or instrument scientists.

Configuration#

essnmx provides a command line data reduction tool. The essnmx-reduce interface will reduce nexus file and save the results into NXlauetof(not exactly but very close) format for dials.

For conveniences and safety, all configuration options are wrapped in a nested pydantic model. Here is a python API you can use to build the configuration and turn it into command line arguments.

The configuration object is a pydantic model, and it thus enforces strict checks on the types of the arguments.

[2]:

from ess.nmx.configurations import (
    ReductionConfig,
    OutputConfig,
    InputConfig,
    WorkflowConfig,
    TimeBinCoordinate,
    Compression,
    to_command_arguments,
)

config = ReductionConfig(
    inputs=InputConfig(
        input_file=["PATH_TO_THE_NEXUS_FILE.hdf"],
        detector_ids=[0, 1, 2],  # Detector index to be reduced in alphabetical order.
    ),
    output=OutputConfig(output_file="scipp_output.hdf", skip_file_output=True),
    workflow=WorkflowConfig(
        time_bin_coordinate=TimeBinCoordinate.time_of_flight,
        nbins=10,
        tof_simulation_num_neutrons=1_000_000,
        tof_simulation_min_wavelength=1.8,
        tof_simulation_max_wavelength=3.6,
        tof_simulation_seed=42,
    ),
)

display(config)
print(to_command_arguments(config=config, one_line=True))

ReductionConfig(inputs=InputConfig(input_file=['PATH_TO_THE_NEXUS_FILE.hdf'], swmr=False, detector_ids=[0, 1, 2], iter_chunk=False, chunk_size_pulse=0, chunk_size_events=0), workflow=WorkflowConfig(time_bin_coordinate=<TimeBinCoordinate.time_of_flight: 'time_of_flight'>, nbins=10, min_time_bin=None, max_time_bin=None, time_bin_unit=<TimeBinUnit.ms: 'ms'>, tof_lookup_table_file_path=None, tof_simulation_num_neutrons=1000000, tof_simulation_min_wavelength=1.8, tof_simulation_max_wavelength=3.6, tof_simulation_min_ltotal=150.0, tof_simulation_max_ltotal=170.0, tof_simulation_seed=42), output=OutputConfig(verbose=False, skip_file_output=True, output_file='scipp_output.hdf', overwrite=False, compression=<Compression.BITSHUFFLE_LZ4: 'BITSHUFFLE_LZ4'>))

--input-file PATH_TO_THE_NEXUS_FILE.hdf \
--detector-ids 0 1 2 \
--chunk-size-pulse 0 \
--chunk-size-events 0 \
--time-bin-coordinate time_of_flight \
--nbins 10 \
--time-bin-unit ms \
--tof-simulation-num-neutrons 1000000 \
--tof-simulation-min-wavelength 1.8 \
--tof-simulation-max-wavelength 3.6 \
--tof-simulation-min-ltotal 150.0 \
--tof-simulation-max-ltotal 170.0 \
--tof-simulation-seed 42 \
--skip-file-output \
--output-file scipp_output.hdf \
--compression BITSHUFFLE_LZ4

Reduce Nexus File(s)#

OutputConfig has an option called skip_file_output if you want to reduce the file and use it only on the memory. Then you can use save_results function to explicitly save the results.

[3]:

from ess.nmx.executables import reduction
from ess.nmx.data import get_small_nmx_nexus

config = ReductionConfig(
    inputs=InputConfig(input_file=[get_small_nmx_nexus().as_posix()]),
    output=OutputConfig(skip_file_output=True),
)
results = reduction(config=config, display=display)
results

'Input file: /home/runner/.cache/ess/nmx/small_nmx_nexus.hdf.zip.unzip/small_nmx_nexus.hdf'

'Output file: /home/runner/work/essnmx/essnmx/docs/user-guide/scipp_output.h5'

/home/runner/work/essnmx/essnmx/.tox/docs/lib/python3.11/site-packages/ess/nmx/workflows.py:183: RuntimeWarning: No crystal rotation found in the Nexus file under 'entry/sample/crystal_rotation'. Returning zero rotation.
  warnings.warn(

[3]:

NMXLauetof(control=NMXMonitorMetadata(data=<scipp.DataArray>
Dimensions: Sizes[tof:50, ]
Coordinates:
* tof                       float64             [ns]  (tof [bin-edge])  [7.37443e+07, 7.50908e+07, ..., 1.39722e+08, 1.41069e+08]
Data:
                            float64             [ns]  (tof)  [1, 1, ..., 1, 1]

, tof_bin_coord='tof', mode=<ControlMode.monitor: 'monitor'>, preset=<scipp.Variable> ()    float64         [counts]  0), definitions='NXlauetof', instrument=NMXInstrument(detectors=DataGroup(sizes={}, keys=[
    detector_panel_0: NMXReducedDetector(data=<scipp.DataArray>
Dimensions: Sizes[y_pixel_offset:1280, x_pixel_offset:1280, tof:50, ]
Coordinates:
* Ltotal                    float64              [m]  (y_pixel_offset, x_pixel_offset)  [157.209, 157.208, ..., 157.208, 157.208]
* detector_number             int32        <no unit>  (y_pixel_offset, x_pixel_offset)  [0, 1, ..., 1638398, 1638399]
* position                  vector3              [m]  (y_pixel_offset, x_pixel_offset)  [(0.36382, 7.65002e-08, 0.334075), (0.363537, 0.000282919, 0.334077), ..., (-0.359399, -0.000282919, 0.33792), (-0.359682, -7.63807e-08, 0.337922)]
* tof                       float64             [ns]  (tof [bin-edge])  [7.37443e+07, 7.50908e+07, ..., 1.39722e+08, 1.41069e+08]
* x_pixel_offset            float64              [m]  (x_pixel_offset)  [-0.256, -0.2556, ..., 0.2552, 0.2556]
* y_pixel_offset            float64              [m]  (y_pixel_offset)  [-0.256, -0.2556, ..., 0.2552, 0.2556]
Data:
                            float32         [counts]  (y_pixel_offset, x_pixel_offset, tof)  [0, 0, ..., 0, 0]  [0, 0, ..., 0, 0]

, metadata=NMXDetectorMetadata(detector_name='detector_panel_0', x_pixel_size=<scipp.Variable> ()    float64              [m]  0.0004, y_pixel_size=<scipp.Variable> ()    float64              [m]  0.0004, origin=<scipp.Variable> ()    vector3              [m]  (0.00206884, 5.97659e-11, 0.335998), fast_axis=<scipp.Variable> ()    vector3  [dimensionless]  (-0.707097, 0.707107, 0.00375908), fast_axis_dim='x_pixel_offset', slow_axis=<scipp.Variable> ()    vector3  [dimensionless]  (-0.707097, -0.707107, 0.00375908), slow_axis_dim='y_pixel_offset', distance=<scipp.Variable> ()    float64              [m]  157.051, first_pixel_position=<scipp.Variable> ()    vector3              [m]  (0.36382, 7.65002e-08, 0.334075), polar_angle=<scipp.Variable> ()      int64            [deg]  0, azimuthal_angle=<scipp.Variable> ()      int64            [deg]  0)),
    detector_panel_1: NMXReducedDetector(data=<scipp.DataArray>
Dimensions: Sizes[y_pixel_offset:1280, x_pixel_offset:1280, tof:50, ]
Coordinates:
* Ltotal                    float64              [m]  (y_pixel_offset, x_pixel_offset)  [157.209, 157.209, ..., 157.209, 157.209]
* detector_number             int32        <no unit>  (y_pixel_offset, x_pixel_offset)  [1638400, 1638401, ..., 3276798, 3276799]
* position                  vector3              [m]  (y_pixel_offset, x_pixel_offset)  [(0.0556193, 0.419019, -0.256034), (0.0556215, 0.419019, -0.255634), ..., (0.420088, 0.0572634, 0.253236), (0.42009, 0.0572634, 0.253636)]
* tof                       float64             [ns]  (tof [bin-edge])  [7.37443e+07, 7.50908e+07, ..., 1.39722e+08, 1.41069e+08]
* x_pixel_offset            float64              [m]  (x_pixel_offset)  [-0.256, -0.2556, ..., 0.2552, 0.2556]
* y_pixel_offset            float64              [m]  (y_pixel_offset)  [-0.256, -0.2556, ..., 0.2552, 0.2556]
Data:
                            float32         [counts]  (y_pixel_offset, x_pixel_offset, tof)  [0, 0, ..., 0, 0]  [0, 0, ..., 0, 0]

, metadata=NMXDetectorMetadata(detector_name='detector_panel_1', x_pixel_size=<scipp.Variable> ()    float64              [m]  0.0004, y_pixel_size=<scipp.Variable> ()    float64              [m]  0.0004, origin=<scipp.Variable> ()    vector3              [m]  (0.237855, 0.238141, -0.00119925), fast_axis=<scipp.Variable> ()    vector3  [dimensionless]  (0.00531617, -4.11958e-08, 0.999986), fast_axis_dim='x_pixel_offset', slow_axis=<scipp.Variable> ()    vector3  [dimensionless]  (0.707097, -0.707107, -0.00375913), slow_axis_dim='y_pixel_offset', distance=<scipp.Variable> ()    float64              [m]  156.715, first_pixel_position=<scipp.Variable> ()    vector3              [m]  (0.0556193, 0.419019, -0.256034), polar_angle=<scipp.Variable> ()      int64            [deg]  0, azimuthal_angle=<scipp.Variable> ()      int64            [deg]  0)),
    detector_panel_2: NMXReducedDetector(data=<scipp.DataArray>
Dimensions: Sizes[y_pixel_offset:1280, x_pixel_offset:1280, tof:50, ]
Coordinates:
* Ltotal                    float64              [m]  (y_pixel_offset, x_pixel_offset)  [157.209, 157.209, ..., 157.209, 157.209]
* detector_number             int32        <no unit>  (y_pixel_offset, x_pixel_offset)  [3276800, 3276801, ..., 4915198, 4915199]
* position                  vector3              [m]  (y_pixel_offset, x_pixel_offset)  [(-0.0556194, 0.419019, 0.256034), (-0.0556215, 0.419019, 0.255634), ..., (-0.420088, 0.0572635, -0.253236), (-0.42009, 0.0572635, -0.253636)]
* tof                       float64             [ns]  (tof [bin-edge])  [7.37443e+07, 7.50908e+07, ..., 1.39722e+08, 1.41069e+08]
* x_pixel_offset            float64              [m]  (x_pixel_offset)  [-0.256, -0.2556, ..., 0.2552, 0.2556]
* y_pixel_offset            float64              [m]  (y_pixel_offset)  [-0.256, -0.2556, ..., 0.2552, 0.2556]
Data:
                            float32         [counts]  (y_pixel_offset, x_pixel_offset, tof)  [0, 0, ..., 0, 0]  [0, 0, ..., 0, 0]

, metadata=NMXDetectorMetadata(detector_name='detector_panel_2', x_pixel_size=<scipp.Variable> ()    float64              [m]  0.0004, y_pixel_size=<scipp.Variable> ()    float64              [m]  0.0004, origin=<scipp.Variable> ()    vector3              [m]  (-0.237855, 0.238141, 0.00119925), fast_axis=<scipp.Variable> ()    vector3  [dimensionless]  (-0.00531611, 4.14935e-08, -0.999986), fast_axis_dim='x_pixel_offset', slow_axis=<scipp.Variable> ()    vector3  [dimensionless]  (-0.707097, -0.707107, 0.00375903), slow_axis_dim='y_pixel_offset', distance=<scipp.Variable> ()    float64              [m]  156.716, first_pixel_position=<scipp.Variable> ()    vector3              [m]  (-0.0556194, 0.419019, 0.256034), polar_angle=<scipp.Variable> ()      int64            [deg]  0, azimuthal_angle=<scipp.Variable> ()      int64            [deg]  0)),
]), name='NMX', source=NMXSourceMetadata(position=<scipp.Variable> ()    vector3              [m]  (-0.491556, 0, -156.714), name='European Spallation Source', source_type=<SourceType.SpallationNeutronSource: 'Spallation Neutron Source'>, probe=<RadiationProbe.Neutron: 'neutron'>)), sample=NMXSampleMetadata(crystal_rotation=<scipp.Variable> ()    vector3            [deg]  (0, 0, 0), name='sample name', position=<scipp.Variable> ()    vector3              [m]  (0, 0, 0), orientation_matrix=<scipp.Variable> ()  linear_transform3  [dimensionless]  ((1, 0, 0), , (0, 1, 0), , (0, 0, 1), ), unit_cell_length=<scipp.Variable> ()    vector3  [dimensionless]  (1, 1, 1), unit_cell_angle=<scipp.Variable> ()    vector3            [deg]  (90, 90, 90)), lookup_table=TofLookupTable(array=<scipp.DataArray>
Dimensions: Sizes[distance:204, event_time_offset:287, ]
Coordinates:
* distance                  float64              [m]  (distance)  [149.8, 149.9, ..., 170, 170.1]
* event_time_offset         float64            [µs]  (event_time_offset)  [0, 249.75, ..., 71178.8, 71428.6]
Data:
                            float64            [µs]  (distance, event_time_offset)  [69739.3, 69948.4, ..., 140842, 141079]  [687152, 742069, ..., 782770, 784423]

, pulse_period=<scipp.Variable> ()    float64            [µs]  71428.6, pulse_stride=1, distance_resolution=<scipp.Variable> ()    float64              [m]  0.1, time_resolution=<scipp.Variable> ()    float64            [µs]  249.75, choppers=None), reducer=NMXProgram(program='essnmx'))

[4]:

from ess.nmx.executables import save_results

output_config = OutputConfig(
    output_file="scipp_output.hdf", overwrite=True, compression=Compression.GZIP
)
save_results(results=results, output_config=output_config)

Compression Modes#

There are multiple compression modes for detector counts data(other datasets are not compressed). The default mode is BITSHUFFLE_LZ4.

Here is the rough benchmark results with the small test dataset. With the result, users can decide which compression mode to use.

Compression Mode	Final Size [MB]	Compression Ratio	Writing Time [s]	Reading Time [s]
NONE	1_966	1	4	1
GZIP	5	370	18	5
BITSHUFFLE_LZ4	17	114	10	3

In the ESS standard VISA environment. (64 GB mem/6 VCPUs)

BITSHUFFLE_LZ4 showed almost twice faster speed for writing/reading the reduced file. GZIP and BITSHUFFLE could both compress the data more than 99% (when the histogram was very empty) but GZIP had 3 times better compression ratio than BITSHUFFLE for this particular dataset.

Note

Why BITSHUFFLE is the default compression mode? - Bitshuffle is compatible with DIALS and other crystallography packages. It is the primary compression mode of all data collected on DECTRIS EIGER detectors, which are the primary detectors used at synchrotron X-ray MX beamlines. Most of these packages can also read gzip data but the slow readout makes gzip less attractive than bitshuffle.

Warning

Bitshuffle may not be supported in cerntain environments, such as MacOS or Windows. It was accepted to be default because the first step of the reduction workflow (this workflow) is expected to be run by ESS in the specific environment that bitshuffle supports.

This Page

NMX Reduction Workflow#

TL;DR#

Configuration#

Reduce Nexus File(s)#

Loading Reduced File#

Compression Modes#