From tabular data to binned data#

Overview#

Binned data in scipp is conceptually equivalent to an array of tables. In other words, it represents an array of records (table rows) sorted into an (often multi-dimensional) array of “bins”. In this tutorial we begin by learning how to setup tabular data appropriate for histogramming and binning with scipp. The main focus will then be binning the tabular data and basic usage of the resulting binned data.

We will use a file of a simulated neutron-scattering experiment — at the powder diffractometer DREAM at the European Spallation Source. The approach and techniques displayed here are however applicable far more generally and not specific to this scientific area.

Loading tabular data#

We will use a file created by a simulation for a diamond sample using McStas and Geant4. We can use pandas.read_table to load the table as a pandas.Dataframe. pandas.read_table is much faster than numpy.loadtxt and also slightly more convenient:

[1]:

import pandas as pd

filename = 'https://public.esss.dk/groups/scipp/scipp/1/data_dream_diamond.zip'
df = pd.read_table(filename)
df

[1]:

	tof [ns]	wavelength [Angstrom]	module	segment	counter	wire	strip	x_pos [mm]	y_pos [mm]	z_pos [mm]	voxel_x [mm]	voxel_y [mm]	voxel_z [mm]
0	73035529	3.69426	12	6	2	22	20	956.990	-922.167	-1045.120	963.085	-920.410	-1049.580
1	72964368	3.70232	4	5	1	4	20	1106.620	255.398	-892.764	1103.670	248.977	-889.281
2	73056111	3.69120	14	5	2	27	20	703.154	-1204.100	-1102.690	704.337	-1196.640	-1094.100
3	73056111	3.69118	14	5	2	27	19	702.934	-1202.910	-1104.840	704.337	-1196.640	-1107.980
4	73024979	3.69372	10	1	1	21	19	1080.410	-766.272	-1051.010	1075.960	-768.968	-1053.870
...	...	...	...	...	...	...	...	...	...	...	...	...	...
999994	54289512	2.75436	2	5	2	31	154	1244.490	718.196	228.159	1239.600	720.253	226.852
999995	54250975	2.75877	10	2	1	15	153	1033.120	-722.267	192.328	1030.990	-715.544	190.343
999996	54261718	2.75799	14	2	2	19	154	569.039	-1165.440	201.670	573.491	-1164.860	205.394
999997	54261718	2.75797	14	2	2	19	153	572.311	-1164.580	200.893	573.491	-1164.860	197.211
999998	54254907	2.75829	13	3	2	17	153	726.378	-1053.590	193.689	727.408	-1048.180	193.777

999999 rows × 13 columns

scipp.compat.from_pandas can convert the pandas.Dataframe to a scipp.Dataset. The column names encode the physical units so we must extract them manually:

[2]:

import scipp as sc

ds = sc.compat.from_pandas(df)
del ds.coords['row']  # we have no use for this row index
for key in list(ds):
    name, *remainder = key.split(' ')
    ds[name] = ds.pop(key)
    ds[name].unit = remainder[0][1:-1] if remainder else None
sc.table(ds[:10])

This 1-D dataset represents the tabular data that was read from the file. In the above table, each row (record) describes an event, in this case the detection of a neutron, with its associated metadata such as the detector module or the x, y, and z position.

To histogram or bin data by a column, scipp must know which columns are metadata and which column holds data values. The table is actually a table of metadata values for events with an implicit data value of “1 count” each. To continue we convert this into a data array:

[3]:

table = sc.DataArray(sc.ones(sizes=ds.sizes, unit='counts'))
for name in ds:
    table.coords[name] = ds[name].data
table

Binning with edges and groups combined#

It is also possible to bin based on “edges” and “groups” at the same time. Since strips roughly correspond to scattering angle, a plot against wavelength and strip may be useful.

Exercise 5#

Use sc.bin to bin table by strip and wavelength.
Plot the result.

Solution#

[23]:

binned_strip_wavelength = sc.bin(table, groups=[groups['strip']], edges=[wavelength])
binned_strip_wavelength.bins.sum().plot(norm='log')

From tabular data to binned data

Contents

From tabular data to binned data#

Overview#

Loading tabular data#

Histogramming and binning#

Exercise 1#

Solution#

Multi-dimensional spatial binning#

Exercise 2#

Solution#

Multi-dimensional logical binning#

Exercise 3#

Solution#

From event-based metadata to bin-based metadata#

Exercise 4#

Solution#

Binning with edges and groups combined#

Exercise 5#

Solution#