Masking

The purpose of masks in Scipp is to exclude regions of data from analysis, or to select regions of interest (ROIs). For example, we may mask data from sensors we know to be broken, or we may mask everything outside our ROI in an image.

In some cases direct removal of the bad data or data outside the ROI may be preferred. Masking provides an alternative solution, which lets us, e.g., modify the mask or remove it later.

NumPy provides support for masked arrays in the numpy.ma module. Scipp’s masking feature is conceptually similar, but is based on a dictionary of masks. Each mask is a Variable, i.e., comes with explicit dimensions. Scipp can therefore store masks in a very space-efficient manner. For example, given an image stack with dimensions ('image', 'pixel_y', 'pixel_x') we may have a mask for “images” with dimensions ('image', ) and a second mask defining the ROI with dimensions ('pixel_y', 'pixel_x'). The support for multiple masks also enables Scipp to selectively apply or preserve masks. For example, a sum over the ‘image’ dimension can preserve the ROI mask.

import numpy as np
import scipp as sc

Creating and manipulating masks

Masks are simply variables with dtype=bool:

mask = sc.array(dims=['x'], values=[False, False, True])
mask

scipp.Variable (259 Bytes)

• • (x: 3)
    bool
    False, False, True

    Values:
    array([False, False,  True])

Boolean operators can be used to manipulate such variables:

print(~mask)
print(mask ^ mask)
print(mask & ~mask)
print(mask | ~mask)

<scipp.Variable> (x: 3)       bool        <no unit>  [True, True, False]
<scipp.Variable> (x: 3)       bool        <no unit>  [False, False, False]
<scipp.Variable> (x: 3)       bool        <no unit>  [False, False, False]
<scipp.Variable> (x: 3)       bool        <no unit>  [True, True, True]

Comparison operators such as ==, !=, <, or >= (see also the list of comparison functions) are a common method of defining masks:

var = sc.array(dims=['x'], values=np.random.random(5), unit='m')
mask2 = var < 0.5 * sc.Unit('m')
mask2

scipp.Variable (261 Bytes)

• • (x: 5)
    bool
    False, True, False, True, False

    Values:
    array([False,  True, False,  True, False])

Masks in data arrays and items of dataset

Data arrays and equivalently items of dataset can store arbitrary masks. Datasets themselves do not support masks. Masks are accessible using the masks keyword-argument and property, which behaves in the same way as coords:

a = sc.DataArray(
    data=sc.array(dims=['y', 'x'], values=np.arange(1.0, 7.0).reshape((2, 3))),
    coords={'y': sc.arange('y', 2.0, unit='m'), 'x': sc.arange('x', 3.0, unit='m')},
    masks={'x': sc.array(dims=['x'], values=[False, False, True])},
)
sc.show(a)

b = a.copy()
b.masks['x'].values[1] = True
b.masks['y'] = sc.array(dims=['y'], values=[False, True])

Note that setting a mask does not affect the data.

Masks of dataset items are accessed using the masks property of the item:

ds = sc.Dataset(data={'a': a})
ds['a'].masks['x']

scipp.Variable (259 Bytes)

• • (x: 3)
    bool
    False, False, True

    Values:
    array([False, False,  True])

Operations with masked objects

Element-wise binary operations

The result of operations between data arrays or dataset with masks contains the masks of both inputs. If both inputs contain a mask with the same name, the output mask is the combination of the input masks with an OR operation:

a + b

scipp.DataArray (1.84 KB)

• Dimensions:
  • y: 2
  • x: 3
• Coordinates: (2)
  • x
    (x)
    float64
    m
    0.0, 1.0, 2.0

    Values:
    array([0., 1., 2.])

  • y
    (y)
    float64
    m
    0.0, 1.0

    Values:
    array([0., 1.])

• Data: (1)
  • (y, x)
    float64
    𝟙
    2.0, 4.0, ..., 10.0, 12.0

    Values:
    array([[ 2.,  4.,  6.],
           [ 8., 10., 12.]])

• Masks: (2)
  • x
    (x)
    bool
    False, True, True

    Values:
    array([False,  True,  True])

  • y
    (y)
    bool
    False, True

    Values:
    array([False,  True])

Reduction operations

Operations like sum and mean over a particular dimension cannot preserve masks that depend on this dimension. If this is the case, the mask is applied during the operation and is not present in the output:

a.sum('x')

scipp.DataArray (1.03 KB)

• Dimensions:
  • y: 2
• Coordinates: (1)
  • y
    (y)
    float64
    m
    0.0, 1.0

    Values:
    array([0., 1.])

• Data: (1)
  • (y)
    float64
    𝟙
    3.0, 9.0

    Values:
    array([3., 9.])

The mean operation takes into account that masking is reducing the number of points in the mean, i.e., masked elements are not counted (in contrast to, e.g., treating them as 0):

a.mean('x')

scipp.DataArray (1.03 KB)

• Dimensions:
  • y: 2
• Coordinates: (1)
  • y
    (y)
    float64
    m
    0.0, 1.0

    Values:
    array([0., 1.])

• Data: (1)
  • (y)
    float64
    𝟙
    1.5, 4.5

    Values:
    array([1.5, 4.5])

If a mask does not depend on the dimension used for the sum or mean operation, it is preserved. Here b has two masks, one that is applied and one that is preserved:

b.sum('x')

scipp.DataArray (1.29 KB)

• Dimensions:
  • y: 2
• Coordinates: (1)
  • y
    (y)
    float64
    m
    0.0, 1.0

    Values:
    array([0., 1.])

• Data: (1)
  • (y)
    float64
    𝟙
    1.0, 4.0

    Values:
    array([1., 4.])

• Masks: (1)
  • y
    (y)
    bool
    False, True

    Values:
    array([False,  True])

Binning and resampling operations

Operations like bin and rebin over a particular dimension cannot preserve masks that depend on this dimension. In those cases, just like for reduction operations, masks are applied before the operation:

• bin treats masked bins as empty.

• rebin treats masked bins as zero.

In both cases the masks that are applied during the operation are not included in the masks of the output. Masks that are independent of the binning dimension(s) are unaffected and included in the output.