Data Structures#
To keep this documentation generic we typically use dimensions x
or y
, but this should not be seen as a recommendation to use these labels for anything but actual positions or offsets in space.
Variable#
Basics#
scipp.Variable is a labeled multi-dimensional array. A variable has the following key properties:
values
: a multi-dimensional array of values, e.g., similar to a numpy.ndarrayvariances
: a (optional) multi-dimensional array of variances for the array valuesdims
: a list of dimension labels (strings) for each axis of the arrayunit
: a (optional) physical unit of the values in the array
Note that variables, unlike DataArray and its eponym xarray.DataArray, do not have coordinate dicts.
[1]:
import numpy as np
import scipp as sc
Variables should generally be created using one of the available creation functions. For example, we can create a variable from a NumPy array:
[2]:
var = sc.array(dims=['x', 'y'], values=np.random.rand(2, 4), unit='s')
Using a unit is optional but highly encouraged if the variable represents a physical quantity. See Creating Arrays and Datasets for an overview of the different methods for creating variables.
Note:
Internally Scipp does not use NumPy, so the above makes a copy of the numpy array of values into an internal buffer.
We can inspect the created variable as follows:
[3]:
sc.show(var)
[4]:
var
[4]:
- (x: 2, y: 4)float64s0.130, 0.002, ..., 0.358, 0.107
Values:
array([[0.13025082, 0.00247476, 0.07347377, 0.70248368], [0.41854659, 0.00367039, 0.35784183, 0.10728309]])
WARNING:
The above makes use of IPython’s rich output representation, but relying on this feature has a common pitfall:
IPython (and thus Jupyter) has an Output caching system. By default this keeps the last 1000 cell outputs. In the above case this is var
(not the displayed HTML, but the object itself). If such cell outputs are large then this output cache can consume enormous amounts of memory.
Note that del var
will not release the memory, since the IPython output cache still holds a reference to the same object. See this FAQ entry for clearing or disabling this caching.
[5]:
var.unit
[5]:
s
[6]:
var.values
[6]:
array([[0.13025082, 0.00247476, 0.07347377, 0.70248368],
[0.41854659, 0.00367039, 0.35784183, 0.10728309]])
0-D variables (scalars)#
A 0-dimensional variable contains a single value (and an optional variance).
[7]:
scalar = sc.scalar(1.2, unit='s')
sc.show(scalar)
scalar
[7]:
- ()float64s1.2
Values:
array(1.2)
Singular versions of the values
and variances
properties are provided:
[8]:
print(scalar.value)
print(scalar.variance)
1.2
None
An exception is raised from the value
and variance
properties if the variable is not 0-dimensional.
Note:
Scalar variables are distinct from arrays that contain a single value. For example, sc.scalar(1)
is equivalent to sc.array(dims=[], values=1)
. But all the following are distinct:
sc.array(dims=[], values=1)
sc.array(dims=['x'], values=[1])
sc.array(dims=['x', 'y'], values=[[1]])
In particular, the first is a scalar while the other two are not; they are arrays with an extent of one. Accessing the value
property of one of the latter two variables would raise an exception because this property requires a 0-dimensional variable.
DataArray#
Basics#
scipp.DataArray is a labeled array with associated coordinates. A data array is essentially a Variable object with attached dicts of coordinates and masks.
A data array has the following key properties:
data
: the variable holding the array data (its values, variances, dims, and unit).coords
: a dict-like container of coordinates for the array.masks
: a dict-like container of masks for the array.
All dict-likes are accessed using a string as key.
coords
can be seen as independent variables of the data array, while data
is the dependent variable. This means that coordinates must generally match in operations between data arrays. See the section on alignment in the computation guide for details.
masks
allows for storing boolean-valued masks alongside data.
data
as well as the individual values of the coords
and masks
dictionaries are of type Variable, i.e., they have a physical unit and can be used for computation.
**Deprecation**
Data arrays also have attrs
which are essentially unaligned coordinates. attrs
have been deprecated in version 23.09.0 and will be removed in version 24.12.0. See ADR 0016
A data array can be created from variables for its constituents as follows:
[9]:
da = sc.DataArray(
data=sc.array(dims=['y', 'x'], values=np.random.rand(2, 3)),
coords={
'y': sc.array(dims=['y'], values=np.arange(2.0), unit='m'),
'x': sc.array(dims=['x'], values=np.arange(3.0), unit='m'),
},
masks={
'm': sc.array(dims=['x'], values=[False, True, False]),
},
)
sc.show(da)
The dict-like coords
and masks
properties give access to the respective underlying variables:
[10]:
da.coords['x']
[10]:
- (x: 3)float64m0.0, 1.0, 2.0
Values:
array([0., 1., 2.])
[11]:
da.masks['m']
[11]:
- (x: 3)boolFalse, True, False
Values:
array([False, True, False])
Unlike values
when creating a variable, data
as well as entries in the metadata dicts (coords
and masks
) are not deep-copied on insertion into a data array. To avoid unwanted sharing, call the copy()
method. Compare:
[12]:
x2 = sc.zeros(dims=['x'], shape=[3])
da.coords['x2_shared'] = x2
da.coords['x2_copied'] = x2.copy()
x2 += 123
da
[12]:
- y: 2
- x: 3
- x(x)float64m0.0, 1.0, 2.0
Values:
array([0., 1., 2.]) - x2_copied(x)float64𝟙0.0, 0.0, 0.0
Values:
array([0., 0., 0.]) - x2_shared(x)float64𝟙123.0, 123.0, 123.0
Values:
array([123., 123., 123.]) - y(y)float64m0.0, 1.0
Values:
array([0., 1.])
- (y, x)float64𝟙0.383, 0.700, ..., 0.137, 0.012
Values:
array([[0.38274497, 0.69967414, 0.09481222], [0.67401721, 0.13731875, 0.01194942]])
- m(x)boolFalse, True, False
Values:
array([False, True, False])
Meta data can be removed in the same way as in Python dicts:
[13]:
del da.coords['x2_shared']
Alignment of coordinates can be queried with the aligned
property:
[14]:
da.coords['x'].aligned
[14]:
True
It can be set using the set_aligned
method of the coordinates:
[15]:
da.coords.set_aligned('x', False)
da.coords['x'].aligned
[15]:
False
Note that the alignment is encoded in Variable. It is, however, only meaningful in a coords
dict. Scipp ignores alignment in operations of plain variables and when handling masks.
The alignment flag is preserved when inserting variables into coords
dicts:
[16]:
da2 = sc.DataArray(sc.arange('x', 3), coords={'x': da.coords['x']})
da2.coords['x'].aligned
[16]:
False
Dataset#
scipp.Dataset is a dict-like container of data arrays. Individual items of a dataset (“data arrays”) are accessed using a string as a dict key.
In a dataset the coordinates of the sub-arrays are enforced to be aligned. That is, a dataset is not simply a dict of data arrays. Instead, the individual arrays share their coordinates. It is therefore not possible to combine arbitrary data arrays into a dataset. If, e.g., the extents in a certain dimension mismatch, or if coordinate values mismatch, insertion of the mismatching data array will fail.
Often a dataset is not created from individual data arrays. Instead we may provide a dict of variables (the data of the items), and dicts for coords:
[17]:
ds = sc.Dataset(
data={
'a': sc.array(dims=['y', 'x'], values=np.random.rand(2, 3)),
'b': sc.array(dims=['x', 'y'], values=np.random.rand(3, 2)),
},
coords={
'x': sc.array(dims=['x'], values=np.arange(3.0), unit='m'),
'y': sc.array(dims=['y'], values=np.arange(2.0), unit='m'),
'aux': sc.array(dims=['x'], values=np.random.rand(3)),
},
)
sc.show(ds)
[18]:
ds
[18]:
- y: 2
- x: 3
- aux(x)float64𝟙0.136, 0.439, 0.390
Values:
array([0.13645902, 0.43896798, 0.38958667]) - x(x)float64m0.0, 1.0, 2.0
Values:
array([0., 1., 2.]) - y(y)float64m0.0, 1.0
Values:
array([0., 1.])
- a(y, x)float64𝟙0.366, 0.626, ..., 0.726, 0.542
Values:
array([[0.36553556, 0.62580567, 0.31286863], [0.58489435, 0.72577026, 0.54174469]]) - b(x, y)float64𝟙0.809, 0.555, ..., 0.586, 0.025
Values:
array([[0.80894885, 0.5552055 ], [0.93817731, 0.68003091], [0.58550345, 0.02531963]])
[19]:
ds.coords['x'].values
[19]:
array([0., 1., 2.])
The name of a data item serves as a dict key. Item access returns a new data array which is a view onto the data in the dataset and its corresponding coordinates, i.e., no deep copy is made:
[20]:
sc.show(ds['a'])
ds['a']
[20]:
- y: 2
- x: 3
- aux(x)float64𝟙0.136, 0.439, 0.390
Values:
array([0.13645902, 0.43896798, 0.38958667]) - x(x)float64m0.0, 1.0, 2.0
Values:
array([0., 1., 2.]) - y(y)float64m0.0, 1.0
Values:
array([0., 1.])
- (y, x)float64𝟙0.366, 0.626, ..., 0.726, 0.542
Values:
array([[0.36553556, 0.62580567, 0.31286863], [0.58489435, 0.72577026, 0.54174469]])
Use the copy()
method to turn the view into an independent object:
[21]:
copy_of_a = ds['a'].copy()
copy_of_a += 17 # does not change d['a']
ds
[21]:
- y: 2
- x: 3
- aux(x)float64𝟙0.136, 0.439, 0.390
Values:
array([0.13645902, 0.43896798, 0.38958667]) - x(x)float64m0.0, 1.0, 2.0
Values:
array([0., 1., 2.]) - y(y)float64m0.0, 1.0
Values:
array([0., 1.])
- a(y, x)float64𝟙0.366, 0.626, ..., 0.726, 0.542
Values:
array([[0.36553556, 0.62580567, 0.31286863], [0.58489435, 0.72577026, 0.54174469]]) - b(x, y)float64𝟙0.809, 0.555, ..., 0.586, 0.025
Values:
array([[0.80894885, 0.5552055 ], [0.93817731, 0.68003091], [0.58550345, 0.02531963]])
Each data item is linked to its corresponding coordinates and masks. These are accessed using the coords
and masks
properties. The variable holding the data of the dataset item is accessible via the data
property:
[22]:
ds['a'].data
[22]:
- (y: 2, x: 3)float64𝟙0.366, 0.626, ..., 0.726, 0.542
Values:
array([[0.36553556, 0.62580567, 0.31286863], [0.58489435, 0.72577026, 0.54174469]])
For convenience, properties of the data variable are also properties of the data item:
[23]:
ds['a'].values
[23]:
array([[0.36553556, 0.62580567, 0.31286863],
[0.58489435, 0.72577026, 0.54174469]])
[24]:
ds['a'].variances
[25]:
ds['a'].unit
[25]:
dimensionless
All variables in a dataset must have consistent dimensions. Thanks to labeled dimensions, transposed data is supported. Compare items 'a'
and 'b'
:
[26]:
ds['a']
[26]:
- y: 2
- x: 3
- aux(x)float64𝟙0.136, 0.439, 0.390
Values:
array([0.13645902, 0.43896798, 0.38958667]) - x(x)float64m0.0, 1.0, 2.0
Values:
array([0., 1., 2.]) - y(y)float64m0.0, 1.0
Values:
array([0., 1.])
- (y, x)float64𝟙0.366, 0.626, ..., 0.726, 0.542
Values:
array([[0.36553556, 0.62580567, 0.31286863], [0.58489435, 0.72577026, 0.54174469]])
[27]:
ds['b']
[27]:
- x: 3
- y: 2
- aux(x)float64𝟙0.136, 0.439, 0.390
Values:
array([0.13645902, 0.43896798, 0.38958667]) - x(x)float64m0.0, 1.0, 2.0
Values:
array([0., 1., 2.]) - y(y)float64m0.0, 1.0
Values:
array([0., 1.])
- (x, y)float64𝟙0.809, 0.555, ..., 0.586, 0.025
Values:
array([[0.80894885, 0.5552055 ], [0.93817731, 0.68003091], [0.58550345, 0.02531963]])
When inserting a data array or variable into a dataset ownership is shared by default. Use the copy()
method to avoid this if undesirable:
[28]:
ds['da_shared'] = da
ds['da_copied'] = da.copy()
da += 1000
ds
[28]:
- y: 2
- x: 3
- aux(x)float64𝟙0.136, 0.439, 0.390
Values:
array([0.13645902, 0.43896798, 0.38958667]) - x(x)float64m0.0, 1.0, 2.0
Values:
array([0., 1., 2.]) - x2_copied(x)float64𝟙0.0, 0.0, 0.0
Values:
array([0., 0., 0.]) - y(y)float64m0.0, 1.0
Values:
array([0., 1.])
- a(y, x)float64𝟙0.366, 0.626, ..., 0.726, 0.542
Values:
array([[0.36553556, 0.62580567, 0.31286863], [0.58489435, 0.72577026, 0.54174469]]) - b(x, y)float64𝟙0.809, 0.555, ..., 0.586, 0.025
Values:
array([[0.80894885, 0.5552055 ], [0.93817731, 0.68003091], [0.58550345, 0.02531963]]) - da_copied(y, x)float64𝟙0.383, 0.700, ..., 0.137, 0.012
Values:
array([[0.38274497, 0.69967414, 0.09481222], [0.67401721, 0.13731875, 0.01194942]]) - da_shared(y, x)float64𝟙1000.383, 1000.700, ..., 1000.137, 1000.012
Values:
array([[1000.38274497, 1000.69967414, 1000.09481222], [1000.67401721, 1000.13731875, 1000.01194942]])
The usual dict
-like methods are available for Dataset
:
[29]:
for name in ds:
print(name)
a
b
da_shared
da_copied
[30]:
'a' in ds
[30]:
True
[31]:
'e' in ds
[31]:
False
DataGroup#
scipp.DataGroup is a dict-like container for arbitrary Scipp or Python objects. Unlike Dataset, DataGroup
does not have coords and does not enforce compatible dimensions of its items. A DataGroup
can contain other DataGroup
objects and thus allows for representing tree-like data. It can be created like a Python dict
:
[32]:
import numpy as np
import scipp as sc
dg = sc.DataGroup(
a=sc.arange('x', 4),
b=sc.arange('x', 6),
c=sc.arange('y', 2),
d=np.ones((2, 3)),
e='a string',
)
dg
[32]:
- ascippVariable(x: 4)int64𝟙0, 1, 2, 3
- bscippVariable(x: 6)int64𝟙0, 1, ..., 4, 5
- cscippVariable(y: 2)int64𝟙0, 1
- dnumpyndarray()shape=(2, 3), dtype=float64, values=1.0, ... , 1.0
- estr()a string
Just like DataArray, DataGroup
provides properties such as dims
, shape
, and sizes
:
[33]:
dg.dims
[33]:
('x', 'y')
[34]:
dg.shape
[34]:
(None, 2)
[35]:
dg.sizes
[35]:
{'x': None, 'y': 2}
The properties return the union of these properties over all the items in the data group. Non-Scipp objects are considered to have dims=()
and shape=()
. When items have inconsistent size along a dimension then shape
and sizes
report this as None
.
DataGroup
supports positional indexing if the shape along the indexed dimension is unique. Label-based indexing is supported if all items have a corresponding coordinate, even if the shape is not unique.
Most Scipp operations also work for DataGroup
, provided that the operation works for all items in the group. That is, operations will generally fail if the data group contains non-Scipp objects such as NumPy arrays or other Python objects such as integers or strings.