NMX Reduction Workflow#

NMX does not expect users to use python interface directly. This documentation is mostly for instrument data scientists or instrument scientists.

TL;DR#

[1]:

from ess.nmx.executables import reduction
from ess.nmx.data import get_small_nmx_nexus
from ess.nmx.configurations import (
    ReductionConfig,
    OutputConfig,
    InputConfig,
    WorkflowConfig,
    TimeBinCoordinate,
)

# Build Configuration
config = ReductionConfig(
    inputs=InputConfig(
        input_file=[get_small_nmx_nexus().as_posix()],
        detector_ids=[0, 1, 2],
    ),
    output=OutputConfig(
        output_file="scipp_output.hdf", skip_file_output=False, overwrite=True
    ),
    workflow=WorkflowConfig(
        time_bin_coordinate=TimeBinCoordinate.time_of_flight,
        nbins=10,
        tof_simulation_num_neutrons=1_000_000,
        tof_simulation_min_wavelength=1.8,
        tof_simulation_max_wavelength=3.6,
        tof_simulation_seed=42,
    ),
)

# Run Reduction
reduction(config=config, display=display)

Downloading file 'small_nmx_nexus.hdf.zip' from 'https://public.esss.dk/groups/scipp/ess/nmx/1/small_nmx_nexus.hdf.zip' to '/home/runner/.cache/ess/nmx'.
Unzipping contents of '/home/runner/.cache/ess/nmx/small_nmx_nexus.hdf.zip' to '/home/runner/.cache/ess/nmx/small_nmx_nexus.hdf.zip.unzip'

'Input file: /home/runner/.cache/ess/nmx/small_nmx_nexus.hdf.zip.unzip/small_nmx_nexus.hdf'

'Output file: /home/runner/work/essnmx/essnmx/docs/user-guide/scipp_output.hdf'

Downloading file 'ess/ess.h5' from 'https://github.com/scipp/tof-sources/raw/refs/heads/main/1/ess/ess.h5' to '/home/runner/.cache/tof'.
/home/runner/work/essnmx/essnmx/.tox/docs/lib/python3.11/site-packages/ess/nmx/workflows.py:176: RuntimeWarning: No crystal rotation found in the Nexus file under 'entry/sample/crystal_rotation'. Returning zero rotation.
  warnings.warn(

[1]:
  • scipp
    DataGroup
    (y_pixel_offset: 1280,
     x_pixel_offset: 1280,
     tof: 10)
      • detector_panel_0
        scipp
        DataArray
        (y_pixel_offset: 1280,
         x_pixel_offset: 1280,
         tof: 10)
        float32
        counts
        0.0, 0.0, ..., 0.0, 0.0
      • detector_panel_1
        scipp
        DataArray
        (y_pixel_offset: 1280,
         x_pixel_offset: 1280,
         tof: 10)
        float32
        counts
        0.0, 0.0, ..., 0.0, 0.0
      • detector_panel_2
        scipp
        DataArray
        (y_pixel_offset: 1280,
         x_pixel_offset: 1280,
         tof: 10)
        float32
        counts
        0.0, 0.0, ..., 0.0, 0.0
  • scipp
    DataGroup
    ()
      • detector_panel_0
        ess
        NMXDetectorMetadata
        ()
        NMXDetectorMetadata(detector_name='detector_panel_0', x_pixel_size=<scipp.Variab...
      • detector_panel_1
        ess
        NMXDetectorMetadata
        ()
        NMXDetectorMetadata(detector_name='detector_panel_1', x_pixel_size=<scipp.Variab...
      • detector_panel_2
        ess
        NMXDetectorMetadata
        ()
        NMXDetectorMetadata(detector_name='detector_panel_2', x_pixel_size=<scipp.Variab...
  • sample
    ess
    NMXSampleMetadata
    ()
    NMXSampleMetadata(crystal_rotation=<scipp.Variable> () vector3 [de...
  • source
    ess
    NMXSourceMetadata
    ()
    NMXSourceMetadata(source_position=<scipp.Variable> () vector3 [m...
  • monitor
    ess
    NMXMonitorMetadata
    ()
    NMXMonitorMetadata(monitor_histogram=<scipp.DataArray> Dimensions: Sizes[tof:10,...
  • lookup_table
    ess
    TofLookupTable
    ()
    TofLookupTable(array=<scipp.DataArray> Dimensions: Sizes[distance:203, event_tim...

Configuration#

essnmx provides a command line data reduction tool. The essnmx-reduce interface will reduce nexus file and save the results into NXlauetof(not exactly but very close) format for dials.

For conveniences and safety, all configuration options are wrapped in a nested pydantic model. Here is a python API you can use to build the configuration and turn it into command line arguments.

The configuration object is a pydantic model, and it thus enforces strict checks on the types of the arguments.

[2]:

from ess.nmx.configurations import (
    ReductionConfig,
    OutputConfig,
    InputConfig,
    WorkflowConfig,
    TimeBinCoordinate,
    Compression,
    to_command_arguments,
)

config = ReductionConfig(
    inputs=InputConfig(
        input_file=["PATH_TO_THE_NEXUS_FILE.hdf"],
        detector_ids=[0, 1, 2],  # Detector index to be reduced in alphabetical order.
    ),
    output=OutputConfig(output_file="scipp_output.hdf", skip_file_output=True),
    workflow=WorkflowConfig(
        time_bin_coordinate=TimeBinCoordinate.time_of_flight,
        nbins=10,
        tof_simulation_num_neutrons=1_000_000,
        tof_simulation_min_wavelength=1.8,
        tof_simulation_max_wavelength=3.6,
        tof_simulation_seed=42,
    ),
)

display(config)
print(to_command_arguments(config=config, one_line=True))

ReductionConfig(inputs=InputConfig(input_file=['PATH_TO_THE_NEXUS_FILE.hdf'], swmr=False, detector_ids=[0, 1, 2], iter_chunk=False, chunk_size_pulse=0, chunk_size_events=0), workflow=WorkflowConfig(time_bin_coordinate=<TimeBinCoordinate.time_of_flight: 'time_of_flight'>, nbins=10, min_time_bin=None, max_time_bin=None, time_bin_unit=<TimeBinUnit.ms: 'ms'>, tof_lookup_table_file_path=None, tof_simulation_num_neutrons=1000000, tof_simulation_min_wavelength=1.8, tof_simulation_max_wavelength=3.6, tof_simulation_min_ltotal=150.0, tof_simulation_max_ltotal=170.0, tof_simulation_seed=42), output=OutputConfig(verbose=False, skip_file_output=True, output_file='scipp_output.hdf', overwrite=False, compression=<Compression.BITSHUFFLE_LZ4: 'BITSHUFFLE_LZ4'>))

--input-file PATH_TO_THE_NEXUS_FILE.hdf \
--detector-ids 0 1 2 \
--chunk-size-pulse 0 \
--chunk-size-events 0 \
--time-bin-coordinate time_of_flight \
--nbins 10 \
--time-bin-unit ms \
--tof-simulation-num-neutrons 1000000 \
--tof-simulation-min-wavelength 1.8 \
--tof-simulation-max-wavelength 3.6 \
--tof-simulation-min-ltotal 150.0 \
--tof-simulation-max-ltotal 170.0 \
--tof-simulation-seed 42 \
--skip-file-output \
--output-file scipp_output.hdf \
--compression BITSHUFFLE_LZ4

Reduce Nexus File(s)#

OutputConfig has an option called skip_file_output if you want to reduce the file and use it only on the memory. Then you can use save_results function to explicitly save the results.

[3]:

from ess.nmx.executables import reduction
from ess.nmx.data import get_small_nmx_nexus

config = ReductionConfig(
    inputs=InputConfig(input_file=[get_small_nmx_nexus().as_posix()]),
    output=OutputConfig(skip_file_output=True),
)
results = reduction(config=config, display=display)
results

'Input file: /home/runner/.cache/ess/nmx/small_nmx_nexus.hdf.zip.unzip/small_nmx_nexus.hdf'

'Output file: /home/runner/work/essnmx/essnmx/docs/user-guide/scipp_output.h5'

/home/runner/work/essnmx/essnmx/.tox/docs/lib/python3.11/site-packages/ess/nmx/workflows.py:176: RuntimeWarning: No crystal rotation found in the Nexus file under 'entry/sample/crystal_rotation'. Returning zero rotation.
  warnings.warn(

[3]:
  • scipp
    DataGroup
    (y_pixel_offset: 1280,
     x_pixel_offset: 1280,
     tof: 50)
      • detector_panel_0
        scipp
        DataArray
        (y_pixel_offset: 1280,
         x_pixel_offset: 1280,
         tof: 50)
        float32
        counts
        0.0, 0.0, ..., 0.0, 0.0
      • detector_panel_1
        scipp
        DataArray
        (y_pixel_offset: 1280,
         x_pixel_offset: 1280,
         tof: 50)
        float32
        counts
        0.0, 0.0, ..., 0.0, 0.0
      • detector_panel_2
        scipp
        DataArray
        (y_pixel_offset: 1280,
         x_pixel_offset: 1280,
         tof: 50)
        float32
        counts
        0.0, 0.0, ..., 0.0, 0.0
  • scipp
    DataGroup
    ()
      • detector_panel_0
        ess
        NMXDetectorMetadata
        ()
        NMXDetectorMetadata(detector_name='detector_panel_0', x_pixel_size=<scipp.Variab...
      • detector_panel_1
        ess
        NMXDetectorMetadata
        ()
        NMXDetectorMetadata(detector_name='detector_panel_1', x_pixel_size=<scipp.Variab...
      • detector_panel_2
        ess
        NMXDetectorMetadata
        ()
        NMXDetectorMetadata(detector_name='detector_panel_2', x_pixel_size=<scipp.Variab...
  • sample
    ess
    NMXSampleMetadata
    ()
    NMXSampleMetadata(crystal_rotation=<scipp.Variable> () vector3 [de...
  • source
    ess
    NMXSourceMetadata
    ()
    NMXSourceMetadata(source_position=<scipp.Variable> () vector3 [m...
  • monitor
    ess
    NMXMonitorMetadata
    ()
    NMXMonitorMetadata(monitor_histogram=<scipp.DataArray> Dimensions: Sizes[tof:50,...
  • lookup_table
    ess
    TofLookupTable
    ()
    TofLookupTable(array=<scipp.DataArray> Dimensions: Sizes[distance:203, event_tim...
[4]:

from ess.nmx.executables import save_results

output_config = OutputConfig(
    output_file="scipp_output.hdf", overwrite=True, compression=Compression.GZIP
)
save_results(results=results, output_config=output_config)

Compression Modes#

There are multiple compression modes for detector counts data(other datasets are not compressed). The default mode is BITSHUFFLE_LZ4.

Here is the rough benchmark results with the small test dataset. With the result, users can decide which compression mode to use.

Compression Mode

Final Size [MB]

Compression Ratio

Writing Time [s]

Reading Time [s]

NONE

1_966

1

4

1

GZIP

5

370

18

5

BITSHUFFLE_LZ4

17

114

10

3

In the ESS standard VISA environment. (64 GB mem/6 VCPUs)

BITSHUFFLE_LZ4 showed almost twice faster speed for writing/reading the reduced file. GZIP and BITSHUFFLE could both compress the data more than 99% (when the histogram was very empty) but GZIP had 3 times better compression ratio than BITSHUFFLE for this particular dataset.

Note

Why BITSHUFFLE is the default compression mode? - Bitshuffle is compatible with DIALS and other crystallography packages. It is the primary compression mode of all data collected on DECTRIS EIGER detectors, which are the primary detectors used at synchrotron X-ray MX beamlines. Most of these packages can also read gzip data but the slow readout makes gzip less attractive than bitshuffle.

Warning

Bitshuffle may not be supported in cerntain environments, such as MacOS or Windows. It was accepted to be default because the first step of the reduction workflow (this workflow) is expected to be run by ESS in the specific environment that bitshuffle supports.