Masking#

The purpose of masks in Scipp is to exclude regions of data from analysis, or to select regions of interest (ROIs). For example, we may mask data from sensors we know to be broken, or we may mask everything outside our ROI in an image.

In some cases direct removal of the bad data or data outside the ROI may be preferred. Masking provides an alternative solution, which lets us, e.g., modify the mask or remove it later.

NumPy provides support for masked arrays in the numpy.ma module. Scipp’s masking feature is conceptually similar, but is based on a dictionary of masks. Each mask is a Variable, i.e., comes with explicit dimensions. Scipp can therefore store masks in a very space-efficient manner. For example, given an image stack with dimensions ('image', 'pixel_y', 'pixel_x') we may have a mask for “images” with dimensions ('image', ) and a second mask defining the ROI with dimensions ('pixel_y', 'pixel_x'). The support for multiple masks also enables Scipp to selectively apply or preserve masks. For example, a sum over the ‘image’ dimension can preserve the ROI mask.

[1]:

import numpy as np
import scipp as sc

Creating and manipulating masks#

Masks are simply variables with dtype=bool:

[2]:

mask = sc.array(dims=['x'], values=[False, False, True])
mask

[2]:
Show/Hide data repr Show/Hide attributes
scipp.Variable (259 Bytes)
    • (x: 3)
      bool
      False, False, True
      Values:
      array([False, False,  True])

Boolean operators can be used to manipulate such variables:

[3]:

print(~mask)
print(mask ^ mask)
print(mask & ~mask)
print(mask | ~mask)

<scipp.Variable> (x: 3)       bool        <no unit>  [True, True, False]
<scipp.Variable> (x: 3)       bool        <no unit>  [False, False, False]
<scipp.Variable> (x: 3)       bool        <no unit>  [False, False, False]
<scipp.Variable> (x: 3)       bool        <no unit>  [True, True, True]

Comparison operators such as ==, !=, <, or >= (see also the list of comparison functions) are a common method of defining masks:

[4]:

var = sc.array(dims=['x'], values=np.random.random(5), unit='m')
mask2 = var < 0.5 * sc.Unit('m')
mask2

[4]:
Show/Hide data repr Show/Hide attributes
scipp.Variable (261 Bytes)
    • (x: 5)
      bool
      True, True, False, False, True
      Values:
      array([ True,  True, False, False,  True])

Masks in data arrays and items of dataset#

Data arrays and equivalently items of dataset can store arbitrary masks. Datasets themselves do not support masks. Masks are accessible using the masks keyword-argument and property, which behaves in the same way as coords:

[5]:

a = sc.DataArray(
    data=sc.array(dims=['y', 'x'], values=np.arange(1.0, 7.0).reshape((2, 3))),
    coords={'y': sc.arange('y', 2.0, unit='m'), 'x': sc.arange('x', 3.0, unit='m')},
    masks={'x': sc.array(dims=['x'], values=[False, False, True])},
)
sc.show(a)
(dims=('y', 'x'), shape=(2, 3), unit=dimensionless, variances=False)values yx yy(dims=('y',), shape=(2,), unit=m, variances=False)values y xx(dims=('x',), shape=(3,), unit=m, variances=False)values x xx(dims=('x',), shape=(3,), unit=None, variances=False)values x

In the example above the mask was specified as the data array was created. However, similar to coords, masks can also be set on an existing DataArray instance:

[6]:

b = a.copy()
b.masks['x'].values[1] = True
b.masks['y'] = sc.array(dims=['y'], values=[False, True])
sc.show(b)
(dims=('y', 'x'), shape=(2, 3), unit=dimensionless, variances=False)values yx yy(dims=('y',), shape=(2,), unit=None, variances=False)values y yy(dims=('y',), shape=(2,), unit=m, variances=False)values y xx(dims=('x',), shape=(3,), unit=m, variances=False)values x xx(dims=('x',), shape=(3,), unit=None, variances=False)values x

A mask value of True means that the mask is on, i.e., the corresponding data value should be ignored. Note that setting a mask does not affect the data.

Masks of dataset items are accessed using the masks property of the item:

[7]:

ds = sc.Dataset(data={'a': a})
ds['a'].masks['x']

[7]:
Show/Hide data repr Show/Hide attributes
scipp.Variable (259 Bytes)
    • (x: 3)
      bool
      False, False, True
      Values:
      array([False, False,  True])

Operations with masked objects#

Element-wise binary operations#

The result of operations between data arrays or dataset with masks contains the masks of both inputs. If both inputs contain a mask with the same name, the output mask is the combination of the input masks with an OR operation:

[8]:

a + b

[8]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (1.71 KB)
    • y: 2
    • x: 3
    • x
      (x)
      float64
      m
      0.0, 1.0, 2.0
      Values:
      array([0., 1., 2.])
    • y
      (y)
      float64
      m
      0.0, 1.0
      Values:
      array([0., 1.])
    • (y, x)
      float64
      𝟙
      2.0, 4.0, ..., 10.0, 12.0
      Values:
      array([[ 2.,  4.,  6.],
       [ 8., 10., 12.]])
    • x
      (x)
      bool
      False, True, True
      Values:
      array([False,  True,  True])
    • y
      (y)
      bool
      False, True
      Values:
      array([False,  True])

Reduction operations#

Operations like sum and mean over a particular dimension cannot preserve masks that depend on this dimension. If this is the case, the mask is applied during the operation and is not present in the output:

[9]:

a.sum('x')

[9]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (921 Bytes)
    • y: 2
    • y
      (y)
      float64
      m
      0.0, 1.0
      Values:
      array([0., 1.])
    • (y)
      float64
      𝟙
      3.0, 9.0
      Values:
      array([3., 9.])

The mean operation takes into account that masking is reducing the number of points in the mean, i.e., masked elements are not counted (in contrast to, e.g., treating them as 0):

[10]:

a.mean('x')

[10]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (921 Bytes)
    • y: 2
    • y
      (y)
      float64
      m
      0.0, 1.0
      Values:
      array([0., 1.])
    • (y)
      float64
      𝟙
      1.5, 4.5
      Values:
      array([1.5, 4.5])

If a mask does not depend on the dimension used for the sum or mean operation, it is preserved. Here b has two masks, one that is applied and one that is preserved:

[11]:

b.sum('x')

[11]:
Show/Hide data repr Show/Hide attributes
scipp.DataArray (1.15 KB)
    • y: 2
    • y
      (y)
      float64
      m
      0.0, 1.0
      Values:
      array([0., 1.])
    • (y)
      float64
      𝟙
      1.0, 4.0
      Values:
      array([1., 4.])
    • y
      (y)
      bool
      False, True
      Values:
      array([False,  True])

Binning and resampling operations#

Operations like bin and rebin over a particular dimension cannot preserve masks that depend on this dimension. In those cases, just like for reduction operations, masks are applied before the operation:

  • bin treats masked bins as empty.

  • rebin treats masked bins as zero.

In both cases the masks that are applied during the operation are not included in the masks of the output. Masks that are independent of the binning dimension(s) are unaffected and included in the output.

Examples and common patterns using masks#

Masking where a coordinate is negative#

Example data:

[12]:

a = sc.DataArray(
    data=sc.ones(dims=['x'], shape=(6,)),
    coords={'x': sc.linspace('x', -1, 1, 6, unit='m')},
)

To mask entries of a data array where a coordinate is negative, create a boolean variable that is true where the coordinate is negative, and set it on the masks dictionary:

[13]:

a.masks['x is negative'] = a.coords['x'] < sc.scalar(0., unit='m')

Masking nan values#

It is common to mask data where the value of some coordinate is nan. To mask where a coordinate is nan-valued we need to create a boolean variable that is True where the coordinate is nan. That is exactly what `sc.isnan </generated/functions/scipp.isnan.html#scipp.isnan>`__ is for:

[14]:

a.masks['x is nan'] = sc.isnan(a.coords['x'])

Removing masks#

To remove one or several masks, use the .drop_masks method:

[15]:

a.masks

[15]:

<scipp.Dict>
  x is negative: <scipp.Variable> (x: 6)       bool        <no unit>  [True, True, ..., False, False]
  x is nan: <scipp.Variable> (x: 6)       bool        <no unit>  [False, False, ..., False, False]

[16]:

del a.masks['x is negative']

[17]:

a.masks

[17]:

<scipp.Dict>
  x is nan: <scipp.Variable> (x: 6)       bool        <no unit>  [False, False, ..., False, False]

To remove all masks from the data array, the .clear method can be used:

[18]:

a.masks.clear()
a.masks # now empty!

[18]:

<scipp.Dict>