Histogramming, grouping, and binning#

Overview#

Histogramming (see sc.hist), grouping (using sc.groupby), and binning (see Binned data) all serve similar but slightly different purposes. Picking the optimal one of the three for a particular application may yield more natural code and better performance. Let us start by an example. Consider a table of scattered measurements:

[1]:
import numpy as np
import scipp as sc
N = 5000
values = 10*np.random.rand(N)
table = sc.DataArray(
    data=sc.array(dims=['position'], unit=sc.units.counts, values=values, variances=values),
    coords={
        'x':sc.array(dims=['position'], unit='m', values=np.random.rand(N)),
        'y':sc.array(dims=['position'], unit='m', values=np.random.rand(N))
    })
table.values *= 1.0/np.exp(5.0*table.coords['x'].values)
sc.table(table['position', :5])
[1]:
CoordinatesData
x [m]y [m] [counts]
0.2420.1471.272±2.065
0.2510.9781.225±2.073
0.3080.7020.267±1.117
0.7170.5140.143±2.277
0.5720.1900.076±1.152

We may now be interested in the total intensity (counts) as a function of 'x'. There are three ways to do this:

[2]:
xbins = sc.linspace('x', 0, 1, num=40, unit='m')
ds = sc.Dataset()
ds['histogram'] = table.hist(x=xbins)
ds['groupby'] = table.groupby('x', bins=xbins).sum('position')
ds['bin'] = table.bin(x=xbins).bins.sum()
ds.plot()
[2]:
../../_images/user-guide_binned-data_histogramming-grouping-and-binning_3_0.svg

In the above plot we can only see a single line, since the three solutions yield exactly the same result (neglecting floating-point rounding errors):

  • hist sorts data points into ‘x’ bins, summing immediately.

  • groupby groups by ‘x’ and then sums (on-the-fly) all data points falling in the same ‘x’ bin.

  • bin sorts data points into ‘x’ bins. Summing all rows in a bin yields the same result as grouping and summing directly.

So in this case we get equivalent results, but the application areas differ, as described in more detail in the following sections.

Histogramming#

scipp.hist directly sums the data and is efficient. Limitations are:

  • When histogramming in more than one dimension, the implementation uses sc.bin internally, which may be less efficient and uses more memory.

  • Can only apply “sum” or “nansum” to accumulate into a bin. scipp.nanhist is currently implemented differently and uses sc.bin interally. It therefore uses more memory and may be less efficient.

We can also histogram binned data (since binning preserves the 'y' coord), to create 2-D (or N-D) histograms:

[3]:
binned = table.bin(x=xbins)
hist = binned.hist(y=30)
hist.plot()
[3]:
../../_images/user-guide_binned-data_histogramming-grouping-and-binning_5_0.svg
[4]:
hist
[4]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (20.09 KB)
    • x: 39
    • y: 30
    • x
      (x [bin-edge])
      float64
      m
      0.0, 0.026, ..., 0.974, 1.0
      Values:
      array([0. , 0.02564103, 0.05128205, 0.07692308, 0.1025641 , 0.12820513, 0.15384615, 0.17948718, 0.20512821, 0.23076923, 0.25641026, 0.28205128, 0.30769231, 0.33333333, 0.35897436, 0.38461538, 0.41025641, 0.43589744, 0.46153846, 0.48717949, 0.51282051, 0.53846154, 0.56410256, 0.58974359, 0.61538462, 0.64102564, 0.66666667, 0.69230769, 0.71794872, 0.74358974, 0.76923077, 0.79487179, 0.82051282, 0.84615385, 0.87179487, 0.8974359 , 0.92307692, 0.94871795, 0.97435897, 1. ])
    • y
      (y [bin-edge])
      float64
      m
      0.000, 0.034, ..., 0.967, 1.000
      Values:
      array([2.84227383e-04, 3.36034426e-02, 6.69226579e-02, 1.00241873e-01, 1.33561088e-01, 1.66880304e-01, 2.00199519e-01, 2.33518734e-01, 2.66837949e-01, 3.00157165e-01, 3.33476380e-01, 3.66795595e-01, 4.00114810e-01, 4.33434026e-01, 4.66753241e-01, 5.00072456e-01, 5.33391671e-01, 5.66710887e-01, 6.00030102e-01, 6.33349317e-01, 6.66668532e-01, 6.99987748e-01, 7.33306963e-01, 7.66626178e-01, 7.99945393e-01, 8.33264609e-01, 8.66583824e-01, 8.99903039e-01, 9.33222254e-01, 9.66541470e-01, 9.99860685e-01])
    • (x, y)
      float64
      counts
      6.432, 8.534, ..., 0.162, 0.057
      σ = 2.581, 2.947, ..., 4.713, 2.838
      Values:
      array([[6.43225915e+00, 8.53408765e+00, 1.61357762e+01, ..., 9.61998875e+00, 2.63196186e+01, 8.88500848e+00], [7.28701741e+00, 2.77061058e+01, 1.71159994e+01, ..., 8.31204973e+00, 1.15054614e+01, 2.62398325e+01], [3.43283583e+01, 2.58602554e+01, 1.39362524e+01, ..., 1.11585774e+01, 1.13404384e+01, 7.42013579e+00], ..., [1.20073571e-01, 2.91334963e-01, 2.22833731e-01, ..., 1.45798778e-01, 1.87880921e-01, 2.82562548e-01], [8.91039474e-02, 1.22741412e-01, 2.47613120e-01, ..., 3.10903506e-02, 4.40345343e-02, 1.31482177e-01], [7.53692774e-02, 2.40346936e-01, 1.12981100e-01, ..., 1.11811506e-01, 1.62090451e-01, 5.74725161e-02]])

      Variances (σ²):
      array([[ 6.66324449, 8.68522441, 16.99271522, ..., 10.39135186, 27.80919825, 9.82037054], [ 8.88200024, 33.75808447, 20.26669155, ..., 9.76752459, 13.93988071, 31.55105085], [47.65110072, 35.48759563, 18.88186771, ..., 14.65859146, 15.87185355, 10.02006677], ..., [13.20166732, 32.60918699, 23.85709633, ..., 15.64305229, 20.20011259, 30.35711619], [11.10343282, 14.94530229, 29.85683051, ..., 3.58464119, 5.2600396 , 15.93900433], [10.86375711, 34.05767961, 15.48837449, ..., 16.38355044, 22.21536369, 8.0514598 ]])

Another capability of hist is to histogram a dimension that has previously been binned with a different or higher resolution, i.e. different bin edges. Compare to the plot of the initial example:

[5]:
binned = table.bin(x=xbins)
binned.hist(x=100).plot()
[5]:
../../_images/user-guide_binned-data_histogramming-grouping-and-binning_8_0.svg

Grouping#

groupby is more flexible in terms of operations than can be applied and may be the go-to solution when a quick one-liner is required. Limitations are:

  • Can only group along a single dimension.

  • Works best for small to medium-sized data, or if data is already mostly sorted along the grouping dimension. Slow if millions of small input slices contribute to each group.

groupby can also operate on binned data, combining bin contents by concatenation:

[6]:
binned = table.bin(x=xbins)
binned.coords['param'] = sc.array(dims=['x'], values=(np.random.random(39)*4).astype(np.int32))
grouped = binned.groupby('param').bins.concat('x')
grouped
/home/runner/work/scipp/scipp/.tox/docs/lib/python3.8/site-packages/scipp/core/bins.py:439: UserWarning: groupby(...).bins.concat(dim) is deprecated. Use `group` or `bin` instead
  warnings.warn(
[6]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (158.70 KB)
    • param: 4
    • param
      (param)
      int32
      𝟙
      0, 1, 2, 3
      Values:
      array([0, 1, 2, 3], dtype=int32)
    • (param)
      DataArrayView
      binned data [len=1248, len=1090, len=1076, len=1586]
      Values:
      [<scipp.DataArray> Dimensions: Sizes[position:1248, ] Coordinates: x float64 [m] (position) [0.00163507, 0.0180741, ..., 0.958057, 0.962814] y float64 [m] (position) [0.171556, 0.58443, ..., 0.721853, 0.999054] Data: float64 [counts] (position) [6.50914, 5.25295, ..., 0.0662346, 0.0712314] [6.56258, 5.74977, ..., 7.9704, 8.77799] , <scipp.DataArray> Dimensions: Sizes[position:1090, ] Coordinates: x float64 [m] (position) [0.0703357, 0.0530943, ..., 0.857427, 0.849979] y float64 [m] (position) [0.688504, 0.540069, ..., 0.778765, 0.760148] Data: float64 [counts] (position) [5.00704, 6.15776, ..., 0.0505985, 0.0497499] [7.11726, 8.03, ..., 3.68144, 3.48737] , <scipp.DataArray> Dimensions: Sizes[position:1076, ] Coordinates: x float64 [m] (position) [0.13676, 0.13164, ..., 0.94498, 0.924282] y float64 [m] (position) [0.433468, 0.23043, ..., 0.104499, 0.634286] Data: float64 [counts] (position) [3.42338, 1.22969, ..., 0.0582226, 0.0811027] [6.78307, 2.37492, ..., 6.56281, 8.24304] , <scipp.DataArray> Dimensions: Sizes[position:1586, ] Coordinates: x float64 [m] (position) [0.292743, 0.302205, ..., 0.99881, 0.98643] y float64 [m] (position) [0.115761, 0.358842, ..., 0.15545, 0.98124] Data: float64 [counts] (position) [1.29042, 1.83878, ..., 0.0574599, 0.029323] [5.57719, 8.33221, ..., 8.47723, 4.06644] ]

Each output bin is a combination of multiple input bins:

[7]:
grouped.values[0]
[7]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (40.25 KB out of 157.50 KB)
    • position: 1248
    • x
      (position)
      float64
      m
      0.002, 0.018, ..., 0.958, 0.963
      Values:
      array([0.00163507, 0.01807406, 0.00787948, ..., 0.95432664, 0.95805737, 0.96281377])
    • y
      (position)
      float64
      m
      0.172, 0.584, ..., 0.722, 0.999
      Values:
      array([0.17155621, 0.58443015, 0.02177361, ..., 0.03827492, 0.72185277, 0.99905439])
    • (position)
      float64
      counts
      6.509, 5.253, ..., 0.066, 0.071
      σ = 2.562, 2.398, ..., 2.823, 2.963
      Values:
      array([6.50914303, 5.25294699, 4.6224722 , ..., 0.02961367, 0.06623462, 0.0712314 ])

      Variances (σ²):
      array([6.56257581, 5.74976837, 4.80822044, ..., 3.49772903, 7.97040116, 8.77798887])

Binning#

scipp.bin actually reorders data and meta data such that all data contributing to a bin is in a contiguous block. Binning along multiple dimensions is supported. Of the three options it is the only solution that supports modifying data in the grouped/binned layout. A variety of operations on such binned data is available. Limitations are:

  • Requires copying and reordering the input data and can thus become expensive.

In the above example the 'y' information is dropped by hist and groupby, but bin preserves it:

[8]:
binned = table.bin(x=xbins)
binned.values[0]
[8]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (4.28 KB out of 157.50 KB)
    • position: 97
    • x
      (position)
      float64
      m
      0.002, 0.018, ..., 0.023, 0.020
      Values:
      array([0.00163507, 0.01807406, 0.00787948, 0.00471561, 0.00475924, 0.00926919, 0.02211219, 0.02231426, 0.02035678, 0.00657715, 0.00028212, 0.01483719, 0.02166934, 0.00324828, 0.00092215, 0.0076939 , 0.01942018, 0.00468691, 0.0168413 , 0.0032863 , 0.01365383, 0.00493771, 0.00348148, 0.02179481, 0.02550336, 0.01553187, 0.01126242, 0.01778987, 0.02398064, 0.01483494, 0.02157137, 0.01776288, 0.01945327, 0.01219393, 0.00850062, 0.00152977, 0.01263605, 0.01732134, 0.02322909, 0.00643027, 0.01650892, 0.00823884, 0.01503746, 0.00521161, 0.00581596, 0.00920671, 0.00816298, 0.0152451 , 0.00707722, 0.01087925, 0.01255156, 0.01018888, 0.01923751, 0.02412321, 0.01076648, 0.02341946, 0.02024841, 0.01808279, 0.0228824 , 0.00685432, 0.00234759, 0.02514075, 0.00286225, 0.00180934, 0.00483824, 0.01442335, 0.01126663, 0.00346538, 0.00427564, 0.01873888, 0.02386228, 0.00066803, 0.00665014, 0.01270846, 0.00557086, 0.01558542, 0.01706913, 0.00732633, 0.02187712, 0.01183466, 0.00302808, 0.00137219, 0.0117983 , 0.02175725, 0.00489752, 0.01229058, 0.01533855, 0.01354348, 0.02494784, 0.00852821, 0.00517309, 0.0028992 , 0.00719343, 0.0169242 , 0.01161465, 0.02302307, 0.02010524])
    • y
      (position)
      float64
      m
      0.172, 0.584, ..., 0.083, 0.721
      Values:
      array([0.17155621, 0.58443015, 0.02177361, 0.25580997, 0.29814468, 0.56306871, 0.94345908, 0.18544942, 0.43015385, 0.39093001, 0.94000263, 0.85367252, 0.97526313, 0.7255684 , 0.28770404, 0.70942209, 0.23161495, 0.86967337, 0.1120145 , 0.09718677, 0.18231586, 0.00590601, 0.74524847, 0.80089744, 0.49464224, 0.17893791, 0.39553804, 0.47683345, 0.23913301, 0.13235654, 0.75005764, 0.91041069, 0.22700141, 0.16485302, 0.72838041, 0.248998 , 0.29210531, 0.80977134, 0.62470416, 0.45689396, 0.54103293, 0.89127066, 0.91670346, 0.06711446, 0.72797447, 0.59136296, 0.0531766 , 0.55213088, 0.52778485, 0.17434822, 0.85459029, 0.10043656, 0.99262857, 0.23054087, 0.82411472, 0.75735532, 0.18169472, 0.13045737, 0.34710184, 0.62575337, 0.77135155, 0.47289075, 0.4552108 , 0.04813266, 0.70805666, 0.6826127 , 0.43391017, 0.94332372, 0.66121983, 0.15148953, 0.41633013, 0.32673091, 0.54237756, 0.16678583, 0.86363586, 0.81283264, 0.65674478, 0.3516015 , 0.70169407, 0.36072397, 0.90133463, 0.28048955, 0.39361615, 0.95543979, 0.21170578, 0.16175501, 0.81383721, 0.660121 , 0.12030602, 0.80297496, 0.76225236, 0.04921526, 0.96641565, 0.85801386, 0.84709303, 0.08298627, 0.72093129])
    • (position)
      float64
      counts
      6.509, 5.253, ..., 4.773, 7.893
      σ = 2.562, 2.398, ..., 2.314, 2.954
      Values:
      array([6.50914303, 5.25294699, 4.6224722 , 7.71597742, 3.82621342, 2.20429673, 8.67718453, 8.14624866, 3.41540998, 0.85114469, 1.99865881, 2.66702482, 2.84235384, 1.34092594, 7.94784914, 9.50650914, 4.81624711, 5.01026586, 2.59119278, 2.58207769, 0.62865803, 1.80978695, 0.72925291, 2.14768261, 6.95979803, 3.39863071, 0.36117256, 6.13511155, 4.28055379, 6.64449247, 4.3991737 , 2.66784497, 1.73024072, 3.01810624, 1.17524027, 1.87453657, 7.33745311, 6.62633593, 0.56115699, 6.41044725, 7.41894915, 3.87953011, 6.64519034, 8.78042131, 4.58243609, 6.01464091, 2.23225212, 6.02607251, 6.42009598, 0.82530766, 3.94862384, 4.27198766, 6.04265464, 1.88780447, 9.35409606, 1.62924447, 6.22153355, 0.18333663, 1.14071423, 4.81102939, 6.24877197, 2.20723015, 9.57614546, 6.14347406, 0.7468491 , 8.18597834, 8.33638579, 8.35326156, 4.16828554, 0.05831585, 2.09488267, 5.39549371, 9.56856181, 0.69323414, 1.39939406, 1.42379209, 3.29535915, 5.7473577 , 8.55942338, 5.22140247, 0.30695343, 5.54466672, 0.93040415, 0.75818756, 9.67835432, 8.22958085, 3.08656342, 0.77724986, 2.22043307, 9.4682102 , 4.85268077, 0.15836148, 6.53232619, 1.31431379, 7.80052438, 4.77327724, 7.89262363])

      Variances (σ²):
      array([6.56257581, 5.74976837, 4.80822044, 7.90006687, 3.91835471, 2.3088613 , 9.69158579, 9.10777874, 3.78135129, 0.87960057, 2.00148009, 2.87240454, 3.16761594, 1.36288229, 7.9845793 , 9.87934518, 5.30736736, 5.1290658 , 2.81883819, 2.62485554, 0.67307488, 1.85502405, 0.74205842, 2.39495267, 7.90635801, 3.67308514, 0.38209449, 6.70583201, 4.82584379, 7.15608464, 4.9001881 , 2.91562848, 1.90699158, 3.20784455, 1.22626838, 1.88892955, 7.81599301, 7.22580555, 0.63026866, 6.61990074, 8.05732831, 4.0426817 , 7.16408719, 9.01222892, 4.71764895, 6.29798773, 2.3252461 , 6.50337312, 6.6513456 , 0.87144476, 4.20437204, 4.49526056, 6.65275459, 2.12980531, 9.87144979, 1.83164358, 6.88440344, 0.20068529, 1.27898474, 4.97876897, 6.32255186, 2.50288025, 9.71417727, 6.19930451, 0.76513657, 8.79813232, 8.81948024, 8.49925901, 4.25835536, 0.06404387, 2.36034776, 5.41354572, 9.89207189, 0.73871346, 1.43892117, 1.53918165, 3.58895397, 5.96179648, 9.54882791, 5.53969451, 0.31163619, 5.58283933, 0.9869413 , 0.84532153, 9.91827979, 8.75117503, 3.33259425, 0.83170622, 2.51542424, 9.88067619, 4.97983502, 0.1606738 , 6.77155183, 1.43037362, 8.26693808, 5.35563075, 8.72728912])

If we omit the call to bins.sum in the original example, we can subsequently apply another histogramming or binning operation to the data:

[9]:
binned = binned.bin(y=100)
binned
[9]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (220.91 KB)
    • x: 39
    • y: 100
    • x
      (x [bin-edge])
      float64
      m
      0.0, 0.026, ..., 0.974, 1.0
      Values:
      array([0. , 0.02564103, 0.05128205, 0.07692308, 0.1025641 , 0.12820513, 0.15384615, 0.17948718, 0.20512821, 0.23076923, 0.25641026, 0.28205128, 0.30769231, 0.33333333, 0.35897436, 0.38461538, 0.41025641, 0.43589744, 0.46153846, 0.48717949, 0.51282051, 0.53846154, 0.56410256, 0.58974359, 0.61538462, 0.64102564, 0.66666667, 0.69230769, 0.71794872, 0.74358974, 0.76923077, 0.79487179, 0.82051282, 0.84615385, 0.87179487, 0.8974359 , 0.92307692, 0.94871795, 0.97435897, 1. ])
    • y
      (y [bin-edge])
      float64
      m
      0.000, 0.010, ..., 0.990, 1.000
      Values:
      array([2.84227383e-04, 1.02799920e-02, 2.02757565e-02, 3.02715211e-02, 4.02672857e-02, 5.02630503e-02, 6.02588148e-02, 7.02545794e-02, 8.02503440e-02, 9.02461085e-02, 1.00241873e-01, 1.10237638e-01, 1.20233402e-01, 1.30229167e-01, 1.40224931e-01, 1.50220696e-01, 1.60216461e-01, 1.70212225e-01, 1.80207990e-01, 1.90203754e-01, 2.00199519e-01, 2.10195283e-01, 2.20191048e-01, 2.30186813e-01, 2.40182577e-01, 2.50178342e-01, 2.60174106e-01, 2.70169871e-01, 2.80165635e-01, 2.90161400e-01, 3.00157165e-01, 3.10152929e-01, 3.20148694e-01, 3.30144458e-01, 3.40140223e-01, 3.50135987e-01, 3.60131752e-01, 3.70127517e-01, 3.80123281e-01, 3.90119046e-01, 4.00114810e-01, 4.10110575e-01, 4.20106339e-01, 4.30102104e-01, 4.40097869e-01, 4.50093633e-01, 4.60089398e-01, 4.70085162e-01, 4.80080927e-01, 4.90076692e-01, 5.00072456e-01, 5.10068221e-01, 5.20063985e-01, 5.30059750e-01, 5.40055514e-01, 5.50051279e-01, 5.60047044e-01, 5.70042808e-01, 5.80038573e-01, 5.90034337e-01, 6.00030102e-01, 6.10025866e-01, 6.20021631e-01, 6.30017396e-01, 6.40013160e-01, 6.50008925e-01, 6.60004689e-01, 6.70000454e-01, 6.79996218e-01, 6.89991983e-01, 6.99987748e-01, 7.09983512e-01, 7.19979277e-01, 7.29975041e-01, 7.39970806e-01, 7.49966570e-01, 7.59962335e-01, 7.69958100e-01, 7.79953864e-01, 7.89949629e-01, 7.99945393e-01, 8.09941158e-01, 8.19936922e-01, 8.29932687e-01, 8.39928452e-01, 8.49924216e-01, 8.59919981e-01, 8.69915745e-01, 8.79911510e-01, 8.89907274e-01, 8.99903039e-01, 9.09898804e-01, 9.19894568e-01, 9.29890333e-01, 9.39886097e-01, 9.49881862e-01, 9.59877626e-01, 9.69873391e-01, 9.79869156e-01, 9.89864920e-01, 9.99860685e-01])
    • (x, y)
      DataArrayView
      binned data [len=1, len=0, ..., len=1, len=2]
      Values:
      [<scipp.DataArray> Dimensions: Sizes[position:1, ] Coordinates: x float64 [m] (position) [0.00493771] y float64 [m] (position) [0.00590601] Data: float64 [counts] (position) [1.80979] [1.85502] , <scipp.DataArray> Dimensions: Sizes[position:0, ] Coordinates: x float64 [m] (position) [] y float64 [m] (position) [] Data: float64 [counts] (position) [] [] , ..., <scipp.DataArray> Dimensions: Sizes[position:1, ] Coordinates: x float64 [m] (position) [0.98643] y float64 [m] (position) [0.98124] Data: float64 [counts] (position) [0.029323] [4.06644] , <scipp.DataArray> Dimensions: Sizes[position:2, ] Coordinates: x float64 [m] (position) [0.988359, 0.976606] y float64 [m] (position) [0.991918, 0.999837] Data: float64 [counts] (position) [0.00647928, 0.000310945] [0.907236, 0.0410541] ]

As in the 1-D example above, summing the bins is equivalent to histogramming binned data:

[10]:
binned.bins.sum().plot()
[10]:
../../_images/user-guide_binned-data_histogramming-grouping-and-binning_18_0.svg