# Histogramming, grouping, and binning#

## Overview#

Histogramming (see sc.histogram), grouping (using sc.groupby), and binning (see Binned data) all serve similar but slightly different purposes. Picking the optimal one of the three for a particular application may yield more natural code and better performance. Let us start by an example. Consider a table of scattered measurements:

[1]:

import numpy as np
import scipp as sc
N = 5000
values = 10*np.random.rand(N)
data = sc.DataArray(
data=sc.Variable(dims=['position'], unit=sc.units.counts, values=values, variances=values),
coords={
'x':sc.Variable(dims=['position'], unit=sc.units.m, values=np.random.rand(N)),
'y':sc.Variable(dims=['position'], unit=sc.units.m, values=np.random.rand(N))
})
data.values *= 1.0/np.exp(5.0*data.coords['x'].values)
sc.table(data['position', :5])


We may now be interested in the total intensity (counts) as a function of 'x'. There are three ways to do this:

[2]:

xbins = sc.Variable(dims=['x'], unit=sc.units.m, values=np.linspace(0,1,num=40))
ds = sc.Dataset()
ds['histogram'] = sc.histogram(data, bins=xbins)
ds['groupby']   = sc.groupby(data, group='x', bins=xbins).sum('position')
ds['bin']       = sc.bin(data, edges=[xbins]).bins.sum()
ds.plot()


In the above plot we can only see a single line, since the three solutions yield exactly the same result (neglecting floating-point rounding errors):

• histogram sorts data points into ‘x’ bins, summing immediately.

• groupby groups by ‘x’ and then sums (on-the-fly) all data points falling in the same ‘x’ bin.

• bin sorts data points into ‘x’ bins. Summing all rows in a bin yields the same result as grouping and summing directly.

So in this case we get equivalent results, but the application areas differ, as described in more detail in the following sections.

## Histogramming#

histogram directly sums the data and is efficient. Limitations are:

• Can currently only histogram along a single dimension.

• Can currently only apply the “sum” to accumulate into a bin. Support for more operations, such as “mean”, is considered for the future.

While histogramming is only supported along a single dimension, we can histogram binned data (since binning preserves the 'y' coord), to create 2-D (or N-D) histograms:

[3]:

binned = sc.bin(data, edges=[xbins])
ybins = sc.linspace(dim='y', unit='m', start=0, stop=1, num=30)
hist = sc.histogram(binned, bins=ybins)
hist.plot()

[4]:

hist

[4]:

scipp.DataArray (18.22 KB)
• x: 39
• y: 29
• x
(x [bin-edge])
float64
m
0.0, 0.026, ..., 0.974, 1.0
Values:array([0.        , 0.02564103, 0.05128205, 0.07692308, 0.1025641 ,
0.12820513, 0.15384615, 0.17948718, 0.20512821, 0.23076923,
0.25641026, 0.28205128, 0.30769231, 0.33333333, 0.35897436,
0.38461538, 0.41025641, 0.43589744, 0.46153846, 0.48717949,
0.51282051, 0.53846154, 0.56410256, 0.58974359, 0.61538462,
0.64102564, 0.66666667, 0.69230769, 0.71794872, 0.74358974,
0.76923077, 0.79487179, 0.82051282, 0.84615385, 0.87179487,
0.8974359 , 0.92307692, 0.94871795, 0.97435897, 1.        ])
• y
(y [bin-edge])
float64
m
0.0, 0.034, ..., 0.966, 1.0
Values:array([0.        , 0.03448276, 0.06896552, 0.10344828, 0.13793103,
0.17241379, 0.20689655, 0.24137931, 0.27586207, 0.31034483,
0.34482759, 0.37931034, 0.4137931 , 0.44827586, 0.48275862,
0.51724138, 0.55172414, 0.5862069 , 0.62068966, 0.65517241,
0.68965517, 0.72413793, 0.75862069, 0.79310345, 0.82758621,
0.86206897, 0.89655172, 0.93103448, 0.96551724, 1.        ])
• (x, y)
float64
counts
8.396, 31.886, ..., 0.269, 0.076
σ = 2.914, 5.756, ..., 6.100, 3.261
Values:array([[ 8.39641242, 31.88568428, 22.27078085, ..., 22.66210417,
3.14250199, 22.83184781],
[23.52745841, 19.41053907,  4.19239471, ..., 16.62289778,
22.22532853,  9.93404436],
[22.75443472, 13.08811511, 17.00647428, ..., 10.41905606,
5.44561887,  0.        ],
...,
[ 0.14153241,  0.41908148,  0.23772608, ...,  0.37850046,
0.05528516,  0.15860482],
[ 0.25029185,  0.29483562,  0.13524347, ...,  0.28799866,
0.22157359,  0.04795045],
[ 0.38908606,  0.04054228,  0.30828497, ...,  0.35852876,
0.26914056,  0.07550346]])Variances (σ²):array([[ 8.49177297, 33.13435111, 24.22214775, ..., 24.48488183,
3.17700945, 24.19643904],
[28.36692163, 23.77902614,  5.20496392, ..., 20.37904945,
26.20180571, 12.29338566],
[31.64513587, 17.40798012, 22.61448634, ..., 13.88499471,
7.79034547,  0.        ],
...,
[15.64190923, 45.98802563, 26.05113627, ..., 40.69699665,
5.87197071, 16.79878647],
[30.48365751, 35.25998287, 16.14097141, ..., 34.25752089,
26.6426806 ,  5.60805823],
[54.06514934,  5.86011666, 42.78224913, ..., 50.21037111,
37.21274705, 10.63632898]])

Another capability of histogram is to histogram a dimension that has previously been binned with a different or higher resolution, i.e. different bin edges. Compare to the plot of the initial example:

[5]:

binned = sc.bin(data, edges=[xbins])
xbins_fine = sc.linspace(dim='x', unit='m', start=0, stop=1, num=100)
sc.histogram(binned, bins=xbins_fine).plot()


## Grouping#

groupby is more flexible in terms of operations than can be applied and may be the go-to solution when a quick one-liner is required. Limitations are:

• Can only group along a single dimension.

• Works best for small to medium-sized data, or if data is already mostly sorted along the grouping dimension. Slow if millions of small input slices contribute to each group.

groupby can also operate on binned data, combining bin contents by concatenation:

[6]:

binned = sc.bin(data, edges=[xbins])
binned.coords['param'] = sc.Variable(dims=['x'], values=(np.random.random(39)*4).astype(np.int32))
grouped = binned.groupby('param').bins.concat('x')
grouped

[6]:

scipp.DataArray (156.33 KB)
• param: 4
• param
(param)
int32
𝟙
0, 1, 2, 3
Values:array([0, 1, 2, 3], dtype=int32)
• (param)
DataArrayView
binned data [len=1286, len=1265, len=658, len=1791]
Values:[<scipp.DataArray>
Dimensions: Sizes[position:1286, ]
Coordinates:
x                         float64              [m]  (position)  [0.032882, 0.0428453, ..., 0.937061, 0.933097]
y                         float64              [m]  (position)  [0.850347, 0.927873, ..., 0.696978, 0.918954]
Data:
float64         [counts]  (position)  [1.22791, 7.03871, ..., 0.0596016, 0.0686324]  [1.44733, 8.72029, ..., 6.45743, 7.28995]

, <scipp.DataArray>
Dimensions: Sizes[position:1265, ]
Coordinates:
x                         float64              [m]  (position)  [0.123672, 0.114304, ..., 0.969074, 0.964263]
y                         float64              [m]  (position)  [0.68067, 0.0216048, ..., 0.389632, 0.0484379]
Data:
float64         [counts]  (position)  [4.86308, 3.88042, ..., 0.00110213, 0.0212314]  [9.02532, 6.87204, ..., 0.140136, 2.63543]

, <scipp.DataArray>
Dimensions: Sizes[position:658, ]
Coordinates:
x                         float64              [m]  (position)  [0.160522, 0.174295, ..., 0.772817, 0.786896]
y                         float64              [m]  (position)  [0.23835, 0.22451, ..., 0.873574, 0.298129]
Data:
float64         [counts]  (position)  [4.07403, 3.15008, ..., 0.0899545, 0.187831]  [9.09063, 7.53006, ..., 4.28719, 9.60482]

, <scipp.DataArray>
Dimensions: Sizes[position:1791, ]
Coordinates:
x                         float64              [m]  (position)  [0.00479115, 0.0178619, ..., 0.980332, 0.98015]
y                         float64              [m]  (position)  [0.0807446, 0.365795, ..., 0.541773, 0.178699]
Data:
float64         [counts]  (position)  [4.42218, 8.43824, ..., 0.0200944, 0.00111688]  [4.52939, 9.22653, ..., 2.70295, 0.150098]

]

Each output bin is a combination of multiple input bins:

[7]:

grouped.values[0]

[7]:

scipp.DataArray (40.19 KB out of 156.25 KB)
• position: 1286
• x
(position)
float64
m
0.033, 0.043, ..., 0.937, 0.933
Values:array([0.03288197, 0.0428453 , 0.0468578 , ..., 0.93262981, 0.93706097,
0.93309736])
• y
(position)
float64
m
0.850, 0.928, ..., 0.697, 0.919
Values:array([0.8503473 , 0.92787337, 0.48741375, ..., 0.04100729, 0.69697794,
0.91895394])
• (position)
float64
counts
1.228, 7.039, ..., 0.060, 0.069
σ = 1.203, 2.953, ..., 2.541, 2.700
Values:array([1.22790677, 7.03871368, 7.30632193, ..., 0.03434483, 0.05960158,
0.06863238])Variances (σ²):array([1.44733043, 8.72028527, 9.23526227, ..., 3.63949617, 6.45743106,
7.28994508])

## Binning#

bin actually reorders data and meta data such that all data contributing to a bin is in a contiguous block. Binning along multiple dimensions is supported. Of the three options it is the only solution that supports modifying data in the grouped/binned layout. A variety of operations on such binned data is available. Limitations are:

• Requires copying and reordering the input data and can thus become expensive.

In the above example the 'y' information is dropped by histogram and groupby, but bin preserves it:

[8]:

binned = sc.bin(data, edges=[xbins])
binned.values[0]

[8]:

scipp.DataArray (3.38 KB out of 156.25 KB)
• position: 108
• x
(position)
float64
m
0.005, 0.018, ..., 0.005, 0.001
Values:array([0.00479115, 0.0178619 , 0.00176121, 0.00883469, 0.01787603,
0.02292638, 0.01510055, 0.00358482, 0.00755353, 0.00435781,
0.00501771, 0.00297259, 0.02293475, 0.01855956, 0.01817986,
0.00077375, 0.00672185, 0.01312385, 0.01878268, 0.01446749,
0.00297089, 0.01659301, 0.01171055, 0.02108899, 0.02469007,
0.00070284, 0.00068422, 0.00651352, 0.01394589, 0.00376006,
0.0032306 , 0.02156602, 0.02352248, 0.02562007, 0.01619985,
0.00281634, 0.02458457, 0.02495147, 0.02323117, 0.00484756,
0.02275821, 0.0247923 , 0.01282138, 0.0159355 , 0.0070444 ,
0.02308238, 0.02507959, 0.00696116, 0.01229493, 0.00866675,
0.0139097 , 0.0193361 , 0.0057368 , 0.00130648, 0.00322855,
0.00182834, 0.00356437, 0.01732985, 0.0124581 , 0.00044772,
0.00536536, 0.02025856, 0.02302225, 0.02181273, 0.0040789 ,
0.01176556, 0.00833956, 0.00212167, 0.02460658, 0.01834536,
0.00536786, 0.01177272, 0.01778789, 0.01443792, 0.01425468,
0.0045913 , 0.00400323, 0.00900148, 0.01213252, 0.00927145,
0.01390285, 0.02258196, 0.00844765, 0.02077416, 0.01617647,
0.02247475, 0.00032323, 0.02137901, 0.01393531, 0.01496717,
0.0253427 , 0.00151336, 0.01371634, 0.01498715, 0.01067147,
0.00565903, 0.00218983, 0.02076028, 0.02412427, 0.00225866,
0.00398602, 0.00591638, 0.00792729, 0.02274039, 0.00288807,
0.02152807, 0.00476846, 0.00055879])
• y
(position)
float64
m
0.081, 0.366, ..., 0.172, 0.881
Values:array([0.08074462, 0.36579492, 0.68167464, 0.75715157, 0.72372741,
0.31568833, 0.10264264, 0.42227099, 0.99745613, 0.04281123,
0.81205975, 0.5435291 , 0.34680847, 0.60340273, 0.17591353,
0.72674447, 0.16890879, 0.69316613, 0.24649204, 0.03849636,
0.96305951, 0.51935559, 0.13254524, 0.53351782, 0.7001129 ,
0.93236874, 0.49258   , 0.18154978, 0.90339784, 0.82715618,
0.04466895, 0.77148283, 0.76317792, 0.22424065, 0.96932722,
0.14792856, 0.43768099, 0.84151654, 0.96657895, 0.28196434,
0.78339083, 0.83645045, 0.42524452, 0.88738301, 0.21515618,
0.8965987 , 0.56075522, 0.78966372, 0.65973799, 0.2580841 ,
0.66778693, 0.34480561, 0.37674847, 0.20003681, 0.13609741,
0.71240129, 0.68666214, 0.90510036, 0.90494684, 0.04471367,
0.58829569, 0.78333504, 0.43882549, 0.40629537, 0.72074519,
0.43722984, 0.54115033, 0.15571834, 0.33222578, 0.41537543,
0.50905842, 0.33760471, 0.65784149, 0.1558203 , 0.29821934,
0.82168551, 0.04580536, 0.24711427, 0.03997669, 0.37483739,
0.50672798, 0.07973377, 0.37755565, 0.75530604, 0.27680045,
0.56803422, 0.42402306, 0.06741243, 0.28902688, 0.92231143,
0.27689771, 0.85658157, 0.87338261, 0.3070665 , 0.62395348,
0.88528016, 0.39833425, 0.69487829, 0.99611271, 0.03159779,
0.97347738, 0.24577074, 0.42998278, 0.09722675, 0.1650194 ,
0.85012316, 0.17234382, 0.88115122])
• (position)
float64
counts
4.422, 8.438, ..., 3.580, 5.377
σ = 2.128, 3.038, ..., 1.915, 2.322
Values:array([4.42217521, 8.4382383 , 5.78214078, 0.20915081, 8.80342852,
1.1310264 , 7.07909612, 1.82876755, 2.45217409, 6.34564312,
5.51471013, 9.77688872, 5.63624677, 5.61047292, 2.59936092,
3.86860415, 5.13799156, 4.96109449, 2.24536426, 8.37704188,
2.04847394, 0.64517767, 1.14078052, 1.78721007, 1.72141047,
1.09402805, 7.95035499, 0.61116396, 6.69181975, 4.0803769 ,
7.00531958, 2.32734894, 0.89677482, 8.77591206, 8.09100942,
2.95240518, 5.40477897, 0.38325887, 1.16676063, 6.41438006,
6.19726674, 7.68165574, 8.79385832, 1.96308053, 9.1032632 ,
3.00434865, 7.30035284, 3.4686799 , 2.22452281, 4.59391855,
8.32046878, 3.94352952, 0.94162467, 6.19000289, 9.81871092,
5.08884979, 3.03579798, 1.3338164 , 3.9194303 , 5.78231152,
2.96435883, 7.54835936, 1.04839469, 5.09583309, 1.43907132,
3.42803364, 9.3693324 , 1.89996607, 6.41438854, 8.91364404,
3.95254544, 5.96050059, 3.41247531, 7.98290076, 3.13652232,
4.75382028, 0.75576987, 0.24241591, 1.39653866, 2.99151408,
5.51808797, 8.88144137, 8.39151258, 7.11019862, 6.67557737,
6.76168537, 7.72951596, 2.22305965, 5.52069038, 7.71268907,
7.03613983, 2.85108686, 2.70716163, 0.72750862, 6.14092378,
8.58487411, 6.48961738, 4.85462873, 2.03551452, 8.39641242,
9.08638915, 3.52589418, 4.54672216, 1.88806814, 4.99309792,
7.28866678, 3.58008484, 5.37697082])Variances (σ²):array([4.52939091, 9.22653035, 5.83328356, 0.21859681, 9.62651603,
1.26840147, 7.63428263, 1.86184205, 2.54655808, 6.48542605,
5.65481626, 9.92328721, 6.32109287, 6.15603461, 2.84671284,
3.88359972, 5.3136104 , 5.29755626, 2.46645315, 9.00547132,
2.07913002, 0.70098801, 1.20957061, 1.98595661, 1.94759316,
1.09787942, 7.97760066, 0.63139576, 7.17509001, 4.15781487,
7.11939525, 2.59233739, 1.00869953, 9.97529011, 8.77364837,
2.99427421, 6.11170875, 0.43418382, 1.31047157, 6.5717499 ,
6.94414867, 8.6954214 , 9.37606786, 2.12589402, 9.42961186,
3.37188749, 8.27567626, 3.59153567, 2.36556541, 4.79736643,
8.91974256, 4.34383077, 0.96902532, 6.23057083, 9.97849839,
5.13558393, 3.09038652, 1.45454557, 4.17133777, 5.79527034,
3.04495945, 8.35301921, 1.17629703, 5.68304305, 1.46872181,
3.63574706, 9.7682726 , 1.92022889, 7.25417084, 9.7699362 ,
4.06006546, 6.32188894, 3.72988492, 8.58049367, 3.36823231,
4.86421358, 0.77104988, 0.25357567, 1.4838787 , 3.13345705,
5.91532032, 9.94304843, 8.75354748, 7.88846049, 7.23795012,
7.56585922, 7.74201799, 2.47385955, 5.91907077, 8.31202099,
7.98666456, 2.87274227, 2.89933799, 0.78411962, 6.47748624,
8.83125359, 6.5610636 , 5.38562866, 2.29646269, 8.49177297,
9.26929833, 3.63175497, 4.73055734, 2.11542578, 5.06572301,
8.11700285, 3.66646792, 5.3920149 ])

If we omit the call to bins.sum in the original example, we can subsequently apply another histogramming or binning operation to the data:

[9]:

binned = sc.bin(binned, edges=[ybins])
binned

[9]:

scipp.DataArray (174.47 KB)
• x: 39
• y: 29
• x
(x [bin-edge])
float64
m
0.0, 0.026, ..., 0.974, 1.0
Values:array([0.        , 0.02564103, 0.05128205, 0.07692308, 0.1025641 ,
0.12820513, 0.15384615, 0.17948718, 0.20512821, 0.23076923,
0.25641026, 0.28205128, 0.30769231, 0.33333333, 0.35897436,
0.38461538, 0.41025641, 0.43589744, 0.46153846, 0.48717949,
0.51282051, 0.53846154, 0.56410256, 0.58974359, 0.61538462,
0.64102564, 0.66666667, 0.69230769, 0.71794872, 0.74358974,
0.76923077, 0.79487179, 0.82051282, 0.84615385, 0.87179487,
0.8974359 , 0.92307692, 0.94871795, 0.97435897, 1.        ])
• y
(y [bin-edge])
float64
m
0.0, 0.034, ..., 0.966, 1.0
Values:array([0.        , 0.03448276, 0.06896552, 0.10344828, 0.13793103,
0.17241379, 0.20689655, 0.24137931, 0.27586207, 0.31034483,
0.34482759, 0.37931034, 0.4137931 , 0.44827586, 0.48275862,
0.51724138, 0.55172414, 0.5862069 , 0.62068966, 0.65517241,
0.68965517, 0.72413793, 0.75862069, 0.79310345, 0.82758621,
0.86206897, 0.89655172, 0.93103448, 0.96551724, 1.        ])
• (x, y)
DataArrayView
binned data [len=1, len=7, ..., len=6, len=2]
Values:[<scipp.DataArray>
Dimensions: Sizes[position:1, ]
Coordinates:
x                         float64              [m]  (position)  [0.00225866]
y                         float64              [m]  (position)  [0.0315978]
Data:
float64         [counts]  (position)  [8.39641]  [8.49177]

, <scipp.DataArray>
Dimensions: Sizes[position:7, ]
Coordinates:
x                         float64              [m]  (position)  [0.00435781, 0.0144675, ..., 0.0121325, 0.021379]
y                         float64              [m]  (position)  [0.0428112, 0.0384964, ..., 0.0399767, 0.0674124]
Data:
float64         [counts]  (position)  [6.34564, 8.37704, ..., 1.39654, 2.22306]  [6.48543, 9.00547, ..., 1.48388, 2.47386]

, ..., <scipp.DataArray>
Dimensions: Sizes[position:6, ]
Coordinates:
x                         float64              [m]  (position)  [0.984623, 0.975783, ..., 0.97959, 0.991298]
y                         float64              [m]  (position)  [0.9475, 0.94378, ..., 0.934447, 0.949253]
Data:
float64         [counts]  (position)  [0.0501752, 0.0546013, ..., 0.0329245, 0.0485781]  [6.89558, 7.17942, ..., 4.41236, 6.90267]

, <scipp.DataArray>
Dimensions: Sizes[position:2, ]
Coordinates:
x                         float64              [m]  (position)  [0.985194, 0.996896]
y                         float64              [m]  (position)  [0.973102, 0.966568]
Data:
float64         [counts]  (position)  [0.047782, 0.0277214]  [6.58546, 4.05087]

]

As in the 1-D example above, summing the bins is equivalent to histogramming binned data:

[10]:

binned.bins.sum().plot()