GroupBy

“Group by” refers to an implementation of the “split-apply-combine” approach known from pandas and xarray. Scipp currently supports only a limited number of operations that can be applied.

Grouping based on label values

Suppose we have measured data for a number of parameter values, potentially repeating measurements with the same parameter multiple times:

import numpy as np
import scipp as sc

np.random.seed(0)

param = sc.Variable(dims=['x'], values=[1, 3, 1, 1, 5, 3])
values = sc.Variable(dims=['x', 'y'], values=np.random.rand(6, 16))
values += 1.0 + param

If we store this data as a data array we obtain the following plot:

data = sc.DataArray(
    values,
    coords={
        'x': sc.Variable(dims=['x'], values=np.arange(6)),
        'y': sc.Variable(dims=['y'], values=np.arange(16)),
    },
)
sc.plot(data)

Note that we chose the “measured” values such that the three distinct values of the underlying parameter are visible. We can now use the split-apply-combine mechanism to transform our data into a more useful representation. We start by storing the parameter values (or any value to be used for grouping) as a non-dimension coordinate:

data.coords['param'] = param

Next, we call scipp.groupby to split the data and call mean on each of the groups:

grouped = sc.groupby(data, group='param').mean('x')
sc.plot(grouped)

Apart from mean, groupby also supports sum, concat, and more. See GroupByDataArray and GroupByDataset for a full list.

Grouping based on binned label values

Grouping based on non-dimension coordinate values (also known as labels) is most useful when labels are strings or integers. If labels are floating-point values or cover a wide range, it is more convenient to group values into bins, i.e., all values within certain bounds are mapped into the same group. We modify the above example to use a contiuously-valued parameter:

param = sc.Variable(dims=['x'], values=np.random.rand(16))
values = sc.Variable(dims=['x', 'y'], values=np.random.rand(16, 16))
values += 1.0 + 5.0 * param

data = sc.DataArray(
    values,
    coords={
        'x': sc.Variable(dims=['x'], values=np.arange(16)),
        'y': sc.Variable(dims=['y'], values=np.arange(16)),
    },
)
sc.plot(data)

We create a variable defining the desired binning:

bins = sc.Variable(dims=["z"], values=np.linspace(0.0, 1.0, 10))

As before, we can now use groupby and mean to transform the data:

data.coords['param'] = param
grouped = sc.groupby(data, group='param', bins=bins).mean('x')
sc.plot(grouped)

The values in the white rows are NaN. This is the result of empty bins, which do not have a meaningful mean value.

Alternatively, grouping can be done based on groups defined as Variables rather than strings. This, however, requires bins to be specified, since bins define the new dimension label.

grouped = sc.groupby(data, group=param, bins=bins).mean(
    'x'
)  # note the lack of quotes around param!
sc.plot(grouped)