Coordinate transformations#

Motivation#

In all fields of science we frequently encounter data that is represented in coordinates or coordinate systems that are not apt for certain operations or visualizations. In these cases we may thus need to compute new coordinates based on one or multiple existing coordinates. For simple cases this may just be done by hand. Consider:

[1]:

import scipp as sc
import plopp as pp

x = sc.linspace(dim='x', unit='m', start=1.0, stop=55.0, num=100)
da = sc.DataArray(data=x * x, coords={'x': x})
da.plot(figsize=(4, 3))

[1]:

../_images/user-guide_coordinate-transformations_1_0.svg

We may want to use \(x^2\) instead of \(x\) as a coordinate, to highlight the quadratic nature of our data:

[2]:

da2 = da.copy()
da2.coords['x_square'] = x * x
da2

[2]:

scipp.DataArray (3.60 KB)

Dimensions:
- x: 100

Coordinates: (2)

(x)

float64

1.0, 1.545, ..., 54.455, 55.0

Values:
array([ 1.        ,  1.54545455,  2.09090909,  2.63636364,  3.18181818,
        3.72727273,  4.27272727,  4.81818182,  5.36363636,  5.90909091,
        6.45454545,  7.        ,  7.54545455,  8.09090909,  8.63636364,
        9.18181818,  9.72727273, 10.27272727, 10.81818182, 11.36363636,
       11.90909091, 12.45454545, 13.        , 13.54545455, 14.09090909,
       14.63636364, 15.18181818, 15.72727273, 16.27272727, 16.81818182,
       17.36363636, 17.90909091, 18.45454545, 19.        , 19.54545455,
       20.09090909, 20.63636364, 21.18181818, 21.72727273, 22.27272727,
       22.81818182, 23.36363636, 23.90909091, 24.45454545, 25.        ,
       25.54545455, 26.09090909, 26.63636364, 27.18181818, 27.72727273,
       28.27272727, 28.81818182, 29.36363636, 29.90909091, 30.45454545,
       31.        , 31.54545455, 32.09090909, 32.63636364, 33.18181818,
       33.72727273, 34.27272727, 34.81818182, 35.36363636, 35.90909091,
       36.45454545, 37.        , 37.54545455, 38.09090909, 38.63636364,
       39.18181818, 39.72727273, 40.27272727, 40.81818182, 41.36363636,
       41.90909091, 42.45454545, 43.        , 43.54545455, 44.09090909,
       44.63636364, 45.18181818, 45.72727273, 46.27272727, 46.81818182,
       47.36363636, 47.90909091, 48.45454545, 49.        , 49.54545455,
       50.09090909, 50.63636364, 51.18181818, 51.72727273, 52.27272727,
       52.81818182, 53.36363636, 53.90909091, 54.45454545, 55.        ])

x_square

(x)

float64

m^2

1.0, 2.388, ..., 2965.298, 3025.000

Values:
array([1.00000000e+00, 2.38842975e+00, 4.37190083e+00, 6.95041322e+00,
       1.01239669e+01, 1.38925620e+01, 1.82561983e+01, 2.32148760e+01,
       2.87685950e+01, 3.49173554e+01, 4.16611570e+01, 4.90000000e+01,
       5.69338843e+01, 6.54628099e+01, 7.45867769e+01, 8.43057851e+01,
       9.46198347e+01, 1.05528926e+02, 1.17033058e+02, 1.29132231e+02,
       1.41826446e+02, 1.55115702e+02, 1.69000000e+02, 1.83479339e+02,
       1.98553719e+02, 2.14223140e+02, 2.30487603e+02, 2.47347107e+02,
       2.64801653e+02, 2.82851240e+02, 3.01495868e+02, 3.20735537e+02,
       3.40570248e+02, 3.61000000e+02, 3.82024793e+02, 4.03644628e+02,
       4.25859504e+02, 4.48669421e+02, 4.72074380e+02, 4.96074380e+02,
       5.20669421e+02, 5.45859504e+02, 5.71644628e+02, 5.98024793e+02,
       6.25000000e+02, 6.52570248e+02, 6.80735537e+02, 7.09495868e+02,
       7.38851240e+02, 7.68801653e+02, 7.99347107e+02, 8.30487603e+02,
       8.62223140e+02, 8.94553719e+02, 9.27479339e+02, 9.61000000e+02,
       9.95115702e+02, 1.02982645e+03, 1.06513223e+03, 1.10103306e+03,
       1.13752893e+03, 1.17461983e+03, 1.21230579e+03, 1.25058678e+03,
       1.28946281e+03, 1.32893388e+03, 1.36900000e+03, 1.40966116e+03,
       1.45091736e+03, 1.49276860e+03, 1.53521488e+03, 1.57825620e+03,
       1.62189256e+03, 1.66612397e+03, 1.71095041e+03, 1.75637190e+03,
       1.80238843e+03, 1.84900000e+03, 1.89620661e+03, 1.94400826e+03,
       1.99240496e+03, 2.04139669e+03, 2.09098347e+03, 2.14116529e+03,
       2.19194215e+03, 2.24331405e+03, 2.29528099e+03, 2.34784298e+03,
       2.40100000e+03, 2.45475207e+03, 2.50909917e+03, 2.56404132e+03,
       2.61957851e+03, 2.67571074e+03, 2.73243802e+03, 2.78976033e+03,
       2.84767769e+03, 2.90619008e+03, 2.96529752e+03, 3.02500000e+03])

Data: (1)

(x)

float64

m^2

1.0, 2.388, ..., 2965.298, 3025.000

Values:
array([1.00000000e+00, 2.38842975e+00, 4.37190083e+00, 6.95041322e+00,
       1.01239669e+01, 1.38925620e+01, 1.82561983e+01, 2.32148760e+01,
       2.87685950e+01, 3.49173554e+01, 4.16611570e+01, 4.90000000e+01,
       5.69338843e+01, 6.54628099e+01, 7.45867769e+01, 8.43057851e+01,
       9.46198347e+01, 1.05528926e+02, 1.17033058e+02, 1.29132231e+02,
       1.41826446e+02, 1.55115702e+02, 1.69000000e+02, 1.83479339e+02,
       1.98553719e+02, 2.14223140e+02, 2.30487603e+02, 2.47347107e+02,
       2.64801653e+02, 2.82851240e+02, 3.01495868e+02, 3.20735537e+02,
       3.40570248e+02, 3.61000000e+02, 3.82024793e+02, 4.03644628e+02,
       4.25859504e+02, 4.48669421e+02, 4.72074380e+02, 4.96074380e+02,
       5.20669421e+02, 5.45859504e+02, 5.71644628e+02, 5.98024793e+02,
       6.25000000e+02, 6.52570248e+02, 6.80735537e+02, 7.09495868e+02,
       7.38851240e+02, 7.68801653e+02, 7.99347107e+02, 8.30487603e+02,
       8.62223140e+02, 8.94553719e+02, 9.27479339e+02, 9.61000000e+02,
       9.95115702e+02, 1.02982645e+03, 1.06513223e+03, 1.10103306e+03,
       1.13752893e+03, 1.17461983e+03, 1.21230579e+03, 1.25058678e+03,
       1.28946281e+03, 1.32893388e+03, 1.36900000e+03, 1.40966116e+03,
       1.45091736e+03, 1.49276860e+03, 1.53521488e+03, 1.57825620e+03,
       1.62189256e+03, 1.66612397e+03, 1.71095041e+03, 1.75637190e+03,
       1.80238843e+03, 1.84900000e+03, 1.89620661e+03, 1.94400826e+03,
       1.99240496e+03, 2.04139669e+03, 2.09098347e+03, 2.14116529e+03,
       2.19194215e+03, 2.24331405e+03, 2.29528099e+03, 2.34784298e+03,
       2.40100000e+03, 2.45475207e+03, 2.50909917e+03, 2.56404132e+03,
       2.61957851e+03, 2.67571074e+03, 2.73243802e+03, 2.78976033e+03,
       2.84767769e+03, 2.90619008e+03, 2.96529752e+03, 3.02500000e+03])

While adding a new coordinate may often be done with a single line of code, the above example highlights the first shortcoming of this approach: To actually visualize da using this new coordinate we must additionally rename the dimension:

[3]:

da2 = da2.rename_dims({'x': 'x_square'})
da2.plot(figsize=(4, 3))

[3]:

../_images/user-guide_coordinate-transformations_5_0.svg

Further complications are:

The original coordinate is preserved and may get in the way in subsequent operations.
Event-coordinates of binned data are not handled.
Multi-step conversions with multiple inputs and multiple outputs may be required in practice.

To accommodate these recurring yet highly application-specific needs, Scipp provides a generic mechanism for transforming coordinates. This is described and exemplified in the following.

`transform_coords`#

Overview#

sc.transform_coords (also available as method of data arrays and datasets) is a tool for transforming one or more input coordinates into one or more output coordinates. It automatically handles:

Renaming of dimensions, if dimension-coordinates are transformed.
Making coordinates unaligned to avoid interference of coordinates consumed by the transformation in follow-up operations.
Conversion of event-coordinates of binned data, if present.

Basic example#

We start by revisiting the example given in Motivation. The building blocks transform_coords operates on are functions with named parameters. The parameter names define the names of the input coordinates to consume. Let us define x_square, which will consume x:

[4]:

def x_square(x):
    return x * x

Next, we create a dict, mapping from an output coord name to a function that can create this coordinate. The sc.show_graph helper is a convenient tool for visualizing the coordinate transformation defined by such as mapping:

[5]:

graph = {'x^2': x_square}
sc.show_graph(graph)

[5]:

../_images/user-guide_coordinate-transformations_9_0.svg

Here, the x coordinate can be consumed by the x_square function, creating the x^2 coordinate. Note that the function name and coordinate are unrelated. Next, we can call transform_coords. Apart from the graph, we also pass a list of desired output coordinates, here simply ['x^2']. transform_coords returns a new (shallow-copied) data array with added coordinates:

[6]:

transformed = da.transform_coords(['x^2'], graph=graph)
transformed

[6]:

scipp.DataArray (3.60 KB)

Dimensions:
- x^2: 100

Coordinates: (2)

(x^2)

float64

1.0, 1.545, ..., 54.455, 55.0

Values:
array([ 1.        ,  1.54545455,  2.09090909,  2.63636364,  3.18181818,
        3.72727273,  4.27272727,  4.81818182,  5.36363636,  5.90909091,
        6.45454545,  7.        ,  7.54545455,  8.09090909,  8.63636364,
        9.18181818,  9.72727273, 10.27272727, 10.81818182, 11.36363636,
       11.90909091, 12.45454545, 13.        , 13.54545455, 14.09090909,
       14.63636364, 15.18181818, 15.72727273, 16.27272727, 16.81818182,
       17.36363636, 17.90909091, 18.45454545, 19.        , 19.54545455,
       20.09090909, 20.63636364, 21.18181818, 21.72727273, 22.27272727,
       22.81818182, 23.36363636, 23.90909091, 24.45454545, 25.        ,
       25.54545455, 26.09090909, 26.63636364, 27.18181818, 27.72727273,
       28.27272727, 28.81818182, 29.36363636, 29.90909091, 30.45454545,
       31.        , 31.54545455, 32.09090909, 32.63636364, 33.18181818,
       33.72727273, 34.27272727, 34.81818182, 35.36363636, 35.90909091,
       36.45454545, 37.        , 37.54545455, 38.09090909, 38.63636364,
       39.18181818, 39.72727273, 40.27272727, 40.81818182, 41.36363636,
       41.90909091, 42.45454545, 43.        , 43.54545455, 44.09090909,
       44.63636364, 45.18181818, 45.72727273, 46.27272727, 46.81818182,
       47.36363636, 47.90909091, 48.45454545, 49.        , 49.54545455,
       50.09090909, 50.63636364, 51.18181818, 51.72727273, 52.27272727,
       52.81818182, 53.36363636, 53.90909091, 54.45454545, 55.        ])

x^2

(x^2)

float64

m^2

1.0, 2.388, ..., 2965.298, 3025.000

Values:
array([1.00000000e+00, 2.38842975e+00, 4.37190083e+00, 6.95041322e+00,
       1.01239669e+01, 1.38925620e+01, 1.82561983e+01, 2.32148760e+01,
       2.87685950e+01, 3.49173554e+01, 4.16611570e+01, 4.90000000e+01,
       5.69338843e+01, 6.54628099e+01, 7.45867769e+01, 8.43057851e+01,
       9.46198347e+01, 1.05528926e+02, 1.17033058e+02, 1.29132231e+02,
       1.41826446e+02, 1.55115702e+02, 1.69000000e+02, 1.83479339e+02,
       1.98553719e+02, 2.14223140e+02, 2.30487603e+02, 2.47347107e+02,
       2.64801653e+02, 2.82851240e+02, 3.01495868e+02, 3.20735537e+02,
       3.40570248e+02, 3.61000000e+02, 3.82024793e+02, 4.03644628e+02,
       4.25859504e+02, 4.48669421e+02, 4.72074380e+02, 4.96074380e+02,
       5.20669421e+02, 5.45859504e+02, 5.71644628e+02, 5.98024793e+02,
       6.25000000e+02, 6.52570248e+02, 6.80735537e+02, 7.09495868e+02,
       7.38851240e+02, 7.68801653e+02, 7.99347107e+02, 8.30487603e+02,
       8.62223140e+02, 8.94553719e+02, 9.27479339e+02, 9.61000000e+02,
       9.95115702e+02, 1.02982645e+03, 1.06513223e+03, 1.10103306e+03,
       1.13752893e+03, 1.17461983e+03, 1.21230579e+03, 1.25058678e+03,
       1.28946281e+03, 1.32893388e+03, 1.36900000e+03, 1.40966116e+03,
       1.45091736e+03, 1.49276860e+03, 1.53521488e+03, 1.57825620e+03,
       1.62189256e+03, 1.66612397e+03, 1.71095041e+03, 1.75637190e+03,
       1.80238843e+03, 1.84900000e+03, 1.89620661e+03, 1.94400826e+03,
       1.99240496e+03, 2.04139669e+03, 2.09098347e+03, 2.14116529e+03,
       2.19194215e+03, 2.24331405e+03, 2.29528099e+03, 2.34784298e+03,
       2.40100000e+03, 2.45475207e+03, 2.50909917e+03, 2.56404132e+03,
       2.61957851e+03, 2.67571074e+03, 2.73243802e+03, 2.78976033e+03,
       2.84767769e+03, 2.90619008e+03, 2.96529752e+03, 3.02500000e+03])

Data: (1)

(x^2)

float64

m^2

1.0, 2.388, ..., 2965.298, 3025.000

Values:
array([1.00000000e+00, 2.38842975e+00, 4.37190083e+00, 6.95041322e+00,
       1.01239669e+01, 1.38925620e+01, 1.82561983e+01, 2.32148760e+01,
       2.87685950e+01, 3.49173554e+01, 4.16611570e+01, 4.90000000e+01,
       5.69338843e+01, 6.54628099e+01, 7.45867769e+01, 8.43057851e+01,
       9.46198347e+01, 1.05528926e+02, 1.17033058e+02, 1.29132231e+02,
       1.41826446e+02, 1.55115702e+02, 1.69000000e+02, 1.83479339e+02,
       1.98553719e+02, 2.14223140e+02, 2.30487603e+02, 2.47347107e+02,
       2.64801653e+02, 2.82851240e+02, 3.01495868e+02, 3.20735537e+02,
       3.40570248e+02, 3.61000000e+02, 3.82024793e+02, 4.03644628e+02,
       4.25859504e+02, 4.48669421e+02, 4.72074380e+02, 4.96074380e+02,
       5.20669421e+02, 5.45859504e+02, 5.71644628e+02, 5.98024793e+02,
       6.25000000e+02, 6.52570248e+02, 6.80735537e+02, 7.09495868e+02,
       7.38851240e+02, 7.68801653e+02, 7.99347107e+02, 8.30487603e+02,
       8.62223140e+02, 8.94553719e+02, 9.27479339e+02, 9.61000000e+02,
       9.95115702e+02, 1.02982645e+03, 1.06513223e+03, 1.10103306e+03,
       1.13752893e+03, 1.17461983e+03, 1.21230579e+03, 1.25058678e+03,
       1.28946281e+03, 1.32893388e+03, 1.36900000e+03, 1.40966116e+03,
       1.45091736e+03, 1.49276860e+03, 1.53521488e+03, 1.57825620e+03,
       1.62189256e+03, 1.66612397e+03, 1.71095041e+03, 1.75637190e+03,
       1.80238843e+03, 1.84900000e+03, 1.89620661e+03, 1.94400826e+03,
       1.99240496e+03, 2.04139669e+03, 2.09098347e+03, 2.14116529e+03,
       2.19194215e+03, 2.24331405e+03, 2.29528099e+03, 2.34784298e+03,
       2.40100000e+03, 2.45475207e+03, 2.50909917e+03, 2.56404132e+03,
       2.61957851e+03, 2.67571074e+03, 2.73243802e+03, 2.78976033e+03,
       2.84767769e+03, 2.90619008e+03, 2.96529752e+03, 3.02500000e+03])

Note how x is now unaligned (not shown in bold), i.e., operations will not use it for alignment anymore. This is important since it will allow for operations combining transformed with other data that may have matching x^2 but not x.

Renaming of Dimensions#

This section is somewhat advanced and not required to understand the basic usage of transform_coords.

As shown above, transform_coords can rename dimensions of the data array if it processes dimension-coordinates. The rules controlling it are described in this section. They are generic and tend to favor not renaming a dimension if it is not entirely clear what the new name should be.

A dimension is only renamed if its dimension-coordinate can be uniquely associated with one output coordinate of transform_coords. In the example above, there are two dimension-coordinates in the input, datetime and location. The latter is used to compute two new coordinates, namely longitude and latitude. It is thus not possible to find a unique new name for the location dimension. datetime on the other hand is used to construct only a single coordinate, local_datetime, which is finally transformed into local_time. The datetime dimension is therefore renamed to local_time.

Identifying outputs with dimension-coordinates can be expressed as a graph-coloring problem. We assign a unique color to each dimension-coordinate (coordinate name matching a dimension of the coordinate); other coordinates are left uncolored (black). We then propagate colors through the directed graph using the following rules

If a node has exactly one colored parent, use that parent’s color (graph 1).
If a node has several colored parents, leave the node black (graph 2). The same happens if no parent is colored.
If a node has more than one child, it is not counted for rules 1 and 2; its color is not applied to its children (graph 3). Other colored nodes are free to apply their color to shared children (graph 4). It makes no difference if the children are computed by the same function or from multiple functions.

base rules

All graphs used by transform_coords are directed acyclic graphs, but they can still have undirected cycles. In those cases, it can be possible to identify an output with a dimension-coordinate even when it has multiple children as part of a cycle.

If a cycle has one and only one colored node as input (w.r.t. directions of the edges) and exactly one output, that output takes the color of the input. In example 1 below, a can be uniquely associated with c. But example 2, this is not possible because there are two final nodes that depend on a.

cycle rules

The following graphs illustrate how the rules interact in larger examples. In particular, it shows that dimensions are renamed to the ‘farthest away’ coordinate even if that is not a final result of the transformation.

In graph 1, a is renamed to c because of rule 1. d is renamed to f because of the second part of rule 3. g cannot be renamed because rule 2 applies to its only child, h.
In graph 2, a is renamed to h because c and h can be associated with each other through the cycle according to rule 4.
This is no longer the case when d is also a dimension-coordinate (graph 3). Like in example 1, a is renamed to c and d to f. But h depends on f directly and c through the cycle and can therefore not be associated with a single dimension.

larger examples

More details on how the algorithm works and the rationale behind it can be found in Architecture Decision Record 0011.

Coordinate transformations#

Motivation#

`transform_coords`#

Overview#

Basic example#

Example: Multi-step transform splitting and combining input coords#

Introduction#

Defining a transformation graph#

Sample data#

Performing a transformation#

Post-processing#

Renaming of Dimensions#

Coordinate transformations#

Motivation#

transform_coords#

Overview#

Basic example#

Example: Multi-step transform splitting and combining input coords#

Introduction#

Defining a transformation graph#

Sample data#

Performing a transformation#

Post-processing#

Renaming of Dimensions#

`transform_coords`#