Histogramming, grouping, and binning#

Overview#

Histogramming (see sc.hist), grouping (using sc.groupby), and binning (see Binned data) all serve similar but slightly different purposes. Picking the optimal one of the three for a particular application may yield more natural code and better performance. Let us start by an example. Consider a table of scattered measurements:

[1]:

import numpy as np
import scipp as sc

N = 5000
values = 10 * np.random.rand(N)
table = sc.DataArray(
    data=sc.array(
        dims=['position'], unit=sc.units.counts, values=values, variances=values
    ),
    coords={
        'x': sc.array(dims=['position'], unit='m', values=np.random.rand(N)),
        'y': sc.array(dims=['position'], unit='m', values=np.random.rand(N)),
    },
)
table.values *= 1.0 / np.exp(5.0 * table.coords['x'].values)
sc.table(table['position', :5])

[1]:

Coordinates		Data
x [m]	y [m]	[counts]
0.873	0.247	0.101±2.823
0.109	0.792	3.006±2.279
0.054	0.834	4.577±2.450
0.395	0.909	0.617±2.112
0.750	0.439	0.157±2.582

We may now be interested in the total intensity (counts) as a function of 'x'. There are three ways to do this:

[2]:

xbins = sc.linspace('x', 0, 1, num=40, unit='m')
ds = sc.Dataset(
    {
        'histogram': table.hist(x=xbins),
        'groupby': table.groupby('x', bins=xbins).sum('position'),
        'bin': table.bin(x=xbins).bins.sum(),
    }
)
ds.plot()

[2]:

../../_images/user-guide_binned-data_histogramming-grouping-and-binning_3_0.svg

In the above plot we can only see a single line, since the three solutions yield exactly the same result (neglecting floating-point rounding errors):

hist sorts data points into ‘x’ bins, summing immediately.
groupby groups by ‘x’ and then sums (on-the-fly) all data points falling in the same ‘x’ bin.
bin sorts data points into ‘x’ bins. Summing all rows in a bin yields the same result as grouping and summing directly.

So in this case we get equivalent results, but the application areas differ, as described in more detail in the following sections.

Histogramming#

scipp.hist directly sums the data and is efficient. Limitations are:

When histogramming in more than one dimension, the implementation uses sc.bin internally, which may be less efficient and uses more memory.
Can only apply “sum” or “nansum” to accumulate into a bin. scipp.nanhist is currently implemented differently and uses sc.bin internally. It therefore uses more memory and may be less efficient.

We can also histogram binned data (since binning preserves the 'y' coord), to create 2-D (or N-D) histograms:

[3]:

binned = table.bin(x=xbins)
hist = binned.hist(y=30)
hist.plot()

[3]:

../../_images/user-guide_binned-data_histogramming-grouping-and-binning_5_0.svg

[4]:

hist

[4]:

scipp.DataArray (20.09 KB)

Dimensions:
- x: 39
- y: 30

Coordinates: (2)

(x [bin-edge])

float64

0.0, 0.026, ..., 0.974, 1.0

Values:
array([0.        , 0.02564103, 0.05128205, 0.07692308, 0.1025641 ,
       0.12820513, 0.15384615, 0.17948718, 0.20512821, 0.23076923,
       0.25641026, 0.28205128, 0.30769231, 0.33333333, 0.35897436,
       0.38461538, 0.41025641, 0.43589744, 0.46153846, 0.48717949,
       0.51282051, 0.53846154, 0.56410256, 0.58974359, 0.61538462,
       0.64102564, 0.66666667, 0.69230769, 0.71794872, 0.74358974,
       0.76923077, 0.79487179, 0.82051282, 0.84615385, 0.87179487,
       0.8974359 , 0.92307692, 0.94871795, 0.97435897, 1.        ])

(y [bin-edge])

float64

0.001, 0.034, ..., 0.967, 1.000

Values:
array([6.06004058e-04, 3.39184081e-02, 6.72308121e-02, 1.00543216e-01,
       1.33855620e-01, 1.67168024e-01, 2.00480428e-01, 2.33792832e-01,
       2.67105236e-01, 3.00417640e-01, 3.33730045e-01, 3.67042449e-01,
       4.00354853e-01, 4.33667257e-01, 4.66979661e-01, 5.00292065e-01,
       5.33604469e-01, 5.66916873e-01, 6.00229277e-01, 6.33541681e-01,
       6.66854085e-01, 7.00166489e-01, 7.33478893e-01, 7.66791297e-01,
       8.00103701e-01, 8.33416105e-01, 8.66728509e-01, 9.00040913e-01,
       9.33353317e-01, 9.66665721e-01, 9.99978125e-01])

Data: (1)

(x, y)

float64

counts

11.543, 12.004, ..., 0.173, 0.184

σ = 3.503, 3.594, ..., 4.829, 5.104

Values:
array([[1.15425936e+01, 1.20036873e+01, 2.65119958e+01, ...,
        3.55322427e+01, 2.86596889e+01, 1.64545172e+01],
       [1.51048737e+01, 1.17587308e+01, 1.59542835e+01, ...,
        1.56323445e+01, 1.71096955e+01, 2.14240240e+01],
       [1.51860697e+01, 1.91154263e+01, 1.40073122e+01, ...,
        1.50181730e+00, 2.13457187e+01, 1.33396470e+01],
       ...,
       [1.86119419e-01, 2.70130055e-01, 2.56578066e-01, ...,
        3.48401397e-01, 2.05931092e-01, 1.20652787e-01],
       [2.66116125e-02, 9.59008241e-02, 2.42074573e-01, ...,
        2.56943303e-01, 1.68540789e-01, 1.52756593e-01],
       [1.16641244e-01, 2.04345102e-01, 1.80672694e-01, ...,
        8.47414396e-02, 1.72689477e-01, 1.83647923e-01]])

Variances (σ²):
array([[12.26914818, 12.91825147, 27.94586173, ..., 38.66683113,
        30.11059852, 17.39351218],
       [17.74427355, 14.28100728, 19.45239352, ..., 18.79020555,
        20.76761409, 26.40602212],
       [21.14097296, 26.35503441, 18.50757458, ...,  1.94467103,
        29.63084007, 19.11635377],
       ...,
       [19.94200467, 29.9512705 , 28.09939694, ..., 37.90221902,
        22.35121893, 13.34101211],
       [ 3.36900383, 12.08484288, 28.88191493, ..., 31.17581055,
        20.85328706, 18.19329346],
       [16.95991685, 28.29955625, 25.476225  , ..., 11.45182507,
        23.32379799, 26.05357255]])

Another capability of hist is to histogram a dimension that has previously been binned with a different or higher resolution, i.e. different bin edges. Compare to the plot of the initial example:

[5]:

binned = table.bin(x=xbins)
binned.hist(x=100).plot()

[5]:

../../_images/user-guide_binned-data_histogramming-grouping-and-binning_8_0.svg

Histogramming, grouping, and binning#

Overview#

Histogramming#

Grouping#

Binning#