Data Structures#

To keep this documentation generic we typically use dimensions x or y, but this should not be seen as a recommendation to use these labels for anything but actual positions or offsets in space.

Variable#

Basics#

scipp.Variable is a labeled multi-dimensional array. A variable has the following key properties:

  • values: a multi-dimensional array of values, e.g., similar to a numpy.ndarray

  • variances: a (optional) multi-dimensional array of variances for the array values

  • dims: a list of dimension labels (strings) for each axis of the array

  • unit: a (optional) physical unit of the values in the array

Note that variables, unlike DataArray and its eponym xarray.DataArray, do not have coordinate dicts.

[1]:
import numpy as np
import scipp as sc

Variables should generally be created using one of the available creation functions. For example, we can create a variable from a numpy array:

[2]:
var = sc.array(dims=['x', 'y'], values=np.random.rand(2, 4), unit='s')

Using a unit is optional but highly encouraged if the variable represents a physical quantity. See Creating Arrays and Datasets for an overview of the different methods for creating variables.

Note:

Internally scipp does not use numpy, so the above makes a copy of the numpy array of values into an internal buffer.

We can inspect the created variable as follows:

[3]:
sc.show(var)
dims=('x', 'y'), shape=(2, 4), unit=s, variances=Falsevalues xy
[4]:
var
[4]:
Show/Hide data repr Show/Hide attributes
scipp.Variable (320 Bytes)
    • (x: 2, y: 4)
      float64
      s
      0.125, 0.378, ..., 0.562, 0.284
      Values:
      array([[0.12497594, 0.37798356, 0.93306891, 0.95658224], [0.9323779 , 0.93880056, 0.56175052, 0.28367602]])

WARNING:

The above makes use of IPython’s rich output representation, but relying on this feature has a common pitfall:

IPython (and thus Jupyter) has an Output caching system. By default this keeps the last 1000 cell outputs. In the above case this is var (not the displayed HTML, but the object itself). If such cell outputs are large then this output cache can consume enormous amounts of memory.

Note that del var will not release the memory, since the IPython output cache still holds a reference to the same object. See this FAQ entry for clearing or disabling this caching.

[5]:
var.unit
[5]:
s
[6]:
var.values
[6]:
array([[0.12497594, 0.37798356, 0.93306891, 0.95658224],
       [0.9323779 , 0.93880056, 0.56175052, 0.28367602]])

0-D variables (scalars)#

A 0-dimensional variable contains a single value (and an optional variance).

[7]:
scalar = sc.scalar(1.2, unit='s')
sc.show(scalar)
scalar
dims=(), shape=(), unit=s, variances=Falsevalues
[7]:
Show/Hide data repr Show/Hide attributes
scipp.Variable (264 Bytes)
    • ()
      float64
      s
      1.2
      Values:
      array(1.2)

Singular versions of the values and variances properties are provided:

[8]:
print(scalar.value)
print(scalar.variance)
1.2
None

An exception is raised from the value and variance properties if the variable is not 0-dimensional.

Note:

Scalar variables are distinct from arrays that contain a single value. For example, sc.scalar(1) is equivalent to sc.array(dims=[], values=1). But all the following are distinct:

  • sc.array(dims=[], values=1)

  • sc.array(dims=['x'], values=[1])

  • sc.array(dims=['x', 'y'], values=[[1]])

In particular, the first is a scalar while the other two are not; they are arrays with an extent of one. Accessing the value property of one of the latter two variables would raise an exception because this property requires a 0-dimensional variable.

DataArray#

Basics#

scipp.DataArray is a labeled array with associated coordinates. A data array is essentially a Variable object with attached dicts of coordinates, masks, and attributes.

A data array has the following key properties:

  • data: the variable holding the array data (its values, variances, dims, and unit).

  • coords: a dict-like container of coordinates for the array, accessed using a string as dict key.

  • masks: a dict-like container of masks for the array, accessed using a string as dict key.

  • attrs: a dict-like container of “attributes” for the array, accessed using a string as dict key.

The key distinction between coordinates (added via the coords property) and attributes (added via the attrs property) is that the former are required to match (“align”) in operations between data arrays whereas the latter are not.

masks allows for storing boolean-valued masks alongside data.

data as well as the individual values of the coords, masks, and attrs dictionaries are of type Variable, i.e., they have a physical unit and can be used for computation. A data array can be created from variables for its constituents as follows:

[9]:
da = sc.DataArray(
    data=sc.array(dims=['y', 'x'], values=np.random.rand(2, 3)),
    coords={
        'y': sc.array(dims=['y'], values=np.arange(2.0), unit='m'),
        'x': sc.array(dims=['x'], values=np.arange(3.0), unit='m'),
    },
    attrs={'aux': sc.array(dims=['x'], values=np.random.rand(3))},
)
sc.show(da)
(dims=('y', 'x'), shape=(2, 3), unit=dimensionless, variances=False)values yx yy(dims=('y',), shape=(2,), unit=m, variances=False)values y xx(dims=('x',), shape=(3,), unit=m, variances=False)values x auxaux(dims=('x',), shape=(3,), unit=dimensionless, variances=False)values x

Note how the 'aux' attribute is essentially a secondary coordinate for the x dimension. The dict-like coords and masks properties give access to the respective underlying variables:

[10]:
da.coords['x']
[10]:
Show/Hide data repr Show/Hide attributes
scipp.Variable (280 Bytes)
    • (x: 3)
      float64
      m
      0.0, 1.0, 2.0
      Values:
      array([0., 1., 2.])
[11]:
da.attrs['aux']
[11]:
Show/Hide data repr Show/Hide attributes
scipp.Variable (280 Bytes)
    • (x: 3)
      float64
      𝟙
      0.892, 0.307, 0.954
      Values:
      array([0.89209904, 0.30702198, 0.95408306])

Access to coords and attrs in a unified manner is possible with the meta property. Essentially this allows us to ignore whether a coordinate is aligned or not:

[12]:
da.meta['x']
[12]:
Show/Hide data repr Show/Hide attributes
scipp.Variable (280 Bytes)
    • (x: 3)
      float64
      m
      0.0, 1.0, 2.0
      Values:
      array([0., 1., 2.])
[13]:
da.meta['aux']
[13]:
Show/Hide data repr Show/Hide attributes
scipp.Variable (280 Bytes)
    • (x: 3)
      float64
      𝟙
      0.892, 0.307, 0.954
      Values:
      array([0.89209904, 0.30702198, 0.95408306])

Unlike values when creating a variable, data as well as entries in the meta data dicts (coords, masks, and attrs) are not deep-copied on insertion into a data array. To avoid unwanted sharing, call the copy() method. Compare:

[14]:
x2 = sc.zeros(dims=['x'], shape=[3])
da.coords['x2_shared'] = x2
da.coords['x2_copied'] = x2.copy()
x2 += 123
da
[14]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (2.16 KB)
    • y: 2
    • x: 3
    • x
      (x)
      float64
      m
      0.0, 1.0, 2.0
      Values:
      array([0., 1., 2.])
    • x2_copied
      (x)
      float64
      𝟙
      0.0, 0.0, 0.0
      Values:
      array([0., 0., 0.])
    • x2_shared
      (x)
      float64
      𝟙
      123.0, 123.0, 123.0
      Values:
      array([123., 123., 123.])
    • y
      (y)
      float64
      m
      0.0, 1.0
      Values:
      array([0., 1.])
    • (y, x)
      float64
      𝟙
      0.291, 0.978, ..., 0.613, 0.277
      Values:
      array([[0.29082429, 0.9775131 , 0.74153494], [0.19480703, 0.61330966, 0.27659611]])
    • aux
      (x)
      float64
      𝟙
      0.892, 0.307, 0.954
      Values:
      array([0.89209904, 0.30702198, 0.95408306])

Meta data can be removed in the same way as in Python dicts:

[15]:
del da.attrs['aux']

Distinction between dimension coords and non-dimension coords, and coords and attrs#

When the name of a coord matches its dimension, e.g., if da.coord['x'] depends on dimension 'x' as in the above example, we call this coord dimension coordinate. Otherwise it is called non-dimension coord. It is important to highlight that for practical purposes (such as matching in operations) dimension coords and non-dimension coords are handled equivalently. Essentially:

  • There is at most one dimension coord for each dimension, but there can be multiple non-dimension coords.

  • The dimension coordinate is the “primary” coordinate and will be used, e.g., when creating a plot (unless specified otherwise).

As mentioned above, the difference between coords and attrs is “alignment”, i.e., only the former are compared in operations. The concept of dimension coords is unrelated to the distinction between coords or attrs. In particular, dimension coords could be made attrs if desired, and non-dimension coords can (and often are) “aligned” coords.

Dataset#

scipp.Dataset is a dict-like container of data arrays. Individual items of a dataset (“data arrays”) are accessed using a string as a dict key.

In a dataset the coordinates of the sub-arrays are enforced to be aligned. That is, a dataset is not simply a dict of data arrays. Instead, the individual arrays share their coordinates. It is therefore not possible to combine arbitrary data arrays into a dataset. If, e.g., the extents in a certain dimension mismatch, or if coordinate values mismatch, insertion of the mismatching data array will fail.

Often a dataset is not created from individual data arrays. Instead we may provide a dict of variables (the data of the items), and dicts for coords:

[16]:
ds = sc.Dataset(
    data={
        'a': sc.array(dims=['y', 'x'], values=np.random.rand(2, 3)),
        'b': sc.array(dims=['y'], values=np.random.rand(2)),
        'c': sc.scalar(value=1.0),
    },
    coords={
        'x': sc.array(dims=['x'], values=np.arange(3.0), unit='m'),
        'y': sc.array(dims=['y'], values=np.arange(2.0), unit='m'),
        'aux': sc.array(dims=['x'], values=np.random.rand(3)),
    },
)
sc.show(ds)
aa(dims=('y', 'x'), shape=(2, 3), unit=dimensionless, variances=False)values yx yy(dims=('y',), shape=(2,), unit=m, variances=False)values y bb(dims=('y',), shape=(2,), unit=dimensionless, variances=False)values y cc(dims=(), shape=(), unit=dimensionless, variances=False)values auxaux(dims=('x',), shape=(3,), unit=dimensionless, variances=False)values x xx(dims=('x',), shape=(3,), unit=m, variances=False)values x
[17]:
ds
[17]:
Show/Hide data repr Show/Hide attributes
scipp.Dataset (3.34 KB)
    • y: 2
    • x: 3
    • aux
      (x)
      float64
      𝟙
      0.092, 0.282, 0.596
      Values:
      array([0.09200675, 0.28202088, 0.59622349])
    • x
      (x)
      float64
      m
      0.0, 1.0, 2.0
      Values:
      array([0., 1., 2.])
    • y
      (y)
      float64
      m
      0.0, 1.0
      Values:
      array([0., 1.])
    • a
      (y, x)
      float64
      𝟙
      0.057, 0.296, ..., 0.472, 0.909
      Values:
      array([[0.05736296, 0.29604705, 0.4594084 ], [0.44640387, 0.47249638, 0.9093595 ]])
    • b
      (y)
      float64
      𝟙
      0.092, 0.278
      Values:
      array([0.09200708, 0.27774053])
    • c
      ()
      float64
      𝟙
      1.0
      Values:
      array(1.)
[18]:
ds.coords['x'].values
[18]:
array([0., 1., 2.])

The name of a data item serves as a dict key. Item access returns a new data array which is a view onto the data in the dataset and its corresponding coordinates, i.e., no deep copy is made:

[19]:
sc.show(ds['a'])
ds['a']
(dims=('y', 'x'), shape=(2, 3), unit=dimensionless, variances=False)values yx yy(dims=('y',), shape=(2,), unit=m, variances=False)values y auxaux(dims=('x',), shape=(3,), unit=dimensionless, variances=False)values x xx(dims=('x',), shape=(3,), unit=m, variances=False)values x
[19]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (1.61 KB)
    • y: 2
    • x: 3
    • aux
      (x)
      float64
      𝟙
      0.092, 0.282, 0.596
      Values:
      array([0.09200675, 0.28202088, 0.59622349])
    • x
      (x)
      float64
      m
      0.0, 1.0, 2.0
      Values:
      array([0., 1., 2.])
    • y
      (y)
      float64
      m
      0.0, 1.0
      Values:
      array([0., 1.])
    • (y, x)
      float64
      𝟙
      0.057, 0.296, ..., 0.472, 0.909
      Values:
      array([[0.05736296, 0.29604705, 0.4594084 ], [0.44640387, 0.47249638, 0.9093595 ]])

Use the copy() method to turn the view into an independent object:

[20]:
copy_of_a = ds['a'].copy()
copy_of_a += 17  # does not change d['a']
ds
[20]:
Show/Hide data repr Show/Hide attributes
scipp.Dataset (3.34 KB)
    • y: 2
    • x: 3
    • aux
      (x)
      float64
      𝟙
      0.092, 0.282, 0.596
      Values:
      array([0.09200675, 0.28202088, 0.59622349])
    • x
      (x)
      float64
      m
      0.0, 1.0, 2.0
      Values:
      array([0., 1., 2.])
    • y
      (y)
      float64
      m
      0.0, 1.0
      Values:
      array([0., 1.])
    • a
      (y, x)
      float64
      𝟙
      0.057, 0.296, ..., 0.472, 0.909
      Values:
      array([[0.05736296, 0.29604705, 0.4594084 ], [0.44640387, 0.47249638, 0.9093595 ]])
    • b
      (y)
      float64
      𝟙
      0.092, 0.278
      Values:
      array([0.09200708, 0.27774053])
    • c
      ()
      float64
      𝟙
      1.0
      Values:
      array(1.)

Each data item is linked to its corresponding coordinates, masks, and attributes. These are accessed using the coords , masks, and attrs properties. The variable holding the data of the dataset item is accessible via the data property:

[21]:
ds['a'].data
[21]:
Show/Hide data repr Show/Hide attributes
scipp.Variable (304 Bytes)
    • (y: 2, x: 3)
      float64
      𝟙
      0.057, 0.296, ..., 0.472, 0.909
      Values:
      array([[0.05736296, 0.29604705, 0.4594084 ], [0.44640387, 0.47249638, 0.9093595 ]])

For convenience, properties of the data variable are also properties of the data item:

[22]:
ds['a'].values
[22]:
array([[0.05736296, 0.29604705, 0.4594084 ],
       [0.44640387, 0.47249638, 0.9093595 ]])
[23]:
ds['a'].variances
[24]:
ds['a'].unit
[24]:
dimensionless

Coordinates of a data item include only those that are relevant to the item’s dimensions, all others are hidden. For example, when accessing 'b', which does not depend on the 'y' dimension, the coord for 'y' as well as the 'aux' coord are not part of the item’s coords:

[25]:
sc.show(ds['b'])
(dims=('y',), shape=(2,), unit=dimensionless, variances=False)values y yy(dims=('y',), shape=(2,), unit=m, variances=False)values y

Similarly, when accessing a 0-dimensional data item, it will have no coordinates:

[26]:
sc.show(ds['c'])
(dims=(), shape=(), unit=dimensionless, variances=False)values
[27]:
ds['a'].variances
[28]:
ds['a'].unit
[28]:
dimensionless

Coordinates of a data item include only those that are relevant to the item’s dimensions, all others are hidden. For example, when accessing 'b', which does not depend on the 'y' dimension, the coord for 'y' as well as the 'aux' coord are not part of the item’s coords:

[29]:
sc.show(ds['b'])
(dims=('y',), shape=(2,), unit=dimensionless, variances=False)values y yy(dims=('y',), shape=(2,), unit=m, variances=False)values y

Similarly, when accessing a 0-dimensional data item, it will have no coordinates:

[30]:
sc.show(ds['c'])
(dims=(), shape=(), unit=dimensionless, variances=False)values

All variables in a dataset must have consistent dimensions. Thanks to labeled dimensions, transposed data is supported:

[31]:
ds['d'] = sc.array(dims=['x', 'y'], values=np.random.rand(3, 2))
sc.show(ds)
ds
dd(dims=('x', 'y'), shape=(3, 2), unit=dimensionless, variances=False)values x y aa(dims=('y', 'x'), shape=(2, 3), unit=dimensionless, variances=False)values yx yy(dims=('y',), shape=(2,), unit=m, variances=False)values y bb(dims=('y',), shape=(2,), unit=dimensionless, variances=False)values y cc(dims=(), shape=(), unit=dimensionless, variances=False)values auxaux(dims=('x',), shape=(3,), unit=dimensionless, variances=False)values x xx(dims=('x',), shape=(3,), unit=m, variances=False)values x
[31]:
Show/Hide data repr Show/Hide attributes
scipp.Dataset (4.14 KB)
    • y: 2
    • x: 3
    • aux
      (x)
      float64
      𝟙
      0.092, 0.282, 0.596
      Values:
      array([0.09200675, 0.28202088, 0.59622349])
    • x
      (x)
      float64
      m
      0.0, 1.0, 2.0
      Values:
      array([0., 1., 2.])
    • y
      (y)
      float64
      m
      0.0, 1.0
      Values:
      array([0., 1.])
    • a
      (y, x)
      float64
      𝟙
      0.057, 0.296, ..., 0.472, 0.909
      Values:
      array([[0.05736296, 0.29604705, 0.4594084 ], [0.44640387, 0.47249638, 0.9093595 ]])
    • b
      (y)
      float64
      𝟙
      0.092, 0.278
      Values:
      array([0.09200708, 0.27774053])
    • c
      ()
      float64
      𝟙
      1.0
      Values:
      array(1.)
    • d
      (x, y)
      float64
      𝟙
      0.907, 0.553, ..., 0.433, 0.091
      Values:
      array([[0.90696979, 0.55348889], [0.95114573, 0.99038345], [0.43300775, 0.09148638]])

When inserting a data array or variable into a dataset ownership is shared by default. Use the copy() method to avoid this if undesirable:

[32]:
ds['da_shared'] = da
ds['da_copied'] = da.copy()
da += 1000
ds
[32]:
Show/Hide data repr Show/Hide attributes
scipp.Dataset (6.31 KB)
    • y: 2
    • x: 3
    • aux
      (x)
      float64
      𝟙
      0.092, 0.282, 0.596
      Values:
      array([0.09200675, 0.28202088, 0.59622349])
    • x
      (x)
      float64
      m
      0.0, 1.0, 2.0
      Values:
      array([0., 1., 2.])
    • x2_copied
      (x)
      float64
      𝟙
      0.0, 0.0, 0.0
      Values:
      array([0., 0., 0.])
    • x2_shared
      (x)
      float64
      𝟙
      123.0, 123.0, 123.0
      Values:
      array([123., 123., 123.])
    • y
      (y)
      float64
      m
      0.0, 1.0
      Values:
      array([0., 1.])
    • a
      (y, x)
      float64
      𝟙
      0.057, 0.296, ..., 0.472, 0.909
      Values:
      array([[0.05736296, 0.29604705, 0.4594084 ], [0.44640387, 0.47249638, 0.9093595 ]])
    • b
      (y)
      float64
      𝟙
      0.092, 0.278
      Values:
      array([0.09200708, 0.27774053])
    • c
      ()
      float64
      𝟙
      1.0
      Values:
      array(1.)
    • d
      (x, y)
      float64
      𝟙
      0.907, 0.553, ..., 0.433, 0.091
      Values:
      array([[0.90696979, 0.55348889], [0.95114573, 0.99038345], [0.43300775, 0.09148638]])
    • da_copied
      (y, x)
      float64
      𝟙
      0.291, 0.978, ..., 0.613, 0.277
      Values:
      array([[0.29082429, 0.9775131 , 0.74153494], [0.19480703, 0.61330966, 0.27659611]])
    • da_shared
      (y, x)
      float64
      𝟙
      1000.291, 1000.978, ..., 1000.613, 1000.277
      Values:
      array([[1000.29082429, 1000.9775131 , 1000.74153494], [1000.19480703, 1000.61330966, 1000.27659611]])

The usual dict-like methods are available for Dataset:

[33]:
for name in ds:
    print(name)
a
b
c
d
da_shared
da_copied
[34]:
'a' in ds
[34]:
True
[35]:
'e' in ds
[35]:
False