Transforming data

Overview

Scipp can apply custom operations to the elements of one or more variables. This is essentially are more advanced version of std::transform.

Two alternatives are provided:

• transform_in_place transforms its first argument.

• transform returns a new Variable.

Both variants support:

• Automatic broadcasting and alignment based on dimension labels. This does also include event data, and operations mixing events and dense data are supported.

• Automatic propagation of uncertainties, provided that the user-provided operation is built from known existing operations.

• Operations between different data types, and operations that produce outputs with a new data type.

• Handling of units.

Basic usage

Transform requires:

• A list of types (or type-combinations) to support. Code is only instantiated for variables of those types.

• A functor, typically a lambda or an “overloaded” lambda. This can also be used to pass special flags, e.g., for disabling code generation for arguments that have variances, generating a runtime failure instead.

Example 1

Transform two variables with two type combinations:

• a of type double and b of type float

• a of type double and b of type double

Since + is defined for units::Unit the same lambda can be used for data and unit. If we needed to handle the unit manually (for example no operator which accepts it) we can use overloaded as shown in example 2.

This call to transform will add the two variables (or variable views) a and b and return a new variable.

auto var = transform<
    std::tuple<std::tuple<double, float>, std::tuple<double, double>>>(
    a, b, [](const auto &a_, const auto &b_) { return a_ + b_; });

Example 2

This call to transform_in_place accepts two variables (or variable views) a and b. a is modified in-place, no new variable is returned. The example also has special unit handling using overloaded, to define an “overloaded” lambda.

transform_in_place<std::tuple<bool>>(
    a, b,
    overloaded{[](auto &a_, const auto &b_) { a &= b; },
               [](const units::Unit &a, const units::Unit &b) {
                 if (a != b)
                   throw std::runtime_error("Unit must match");
               }});

transform_in_place modifies its first argument. Note that the template argument std::tuple<bool> is equivalent to std::tuple<std::tuple<bool, bool>>, i.e., both arguments are require have element type bool.

Recommended usage

• For improved testability and maintainability it is recommended to define the operator stand-alone instead of inline.

• Specifying types on the element wise operator allows them to be omitted from transform and transform_in_place.

• Use arg_list to define list of supported type combinations as shown below.

// This will accept two arguments of
// (a) double and bool,
// (b) float and int64_t,
// (c) int32_t and double, and
// (d) double and double.
// Note the special last argument to ``arg_list``, which could be written also as
// ``std::tuple<double, double>``: If all arguments are the same it suffices to list
// it directly instead of providing a tuple.
// More type tuples can be listed as required.
overloaded{arg_list<std::tuple<double, bool>,
                    std::tuple<float, int64_t>,
                    std::tuple<int32_t, double>,
                    double>
                    [](auto &a, auto &b) { /* Get double/bool...etc. pairs here */ },
                    [](const units::Unit &a, const units::Unit &b) { /* Unit handling */ }
           };

• Flags in transform_flags.h can be used to, e.g., prevent code generation if an argument has variances.

Example

If operation is added to namespace scipp::variable, define:

// In scipp/core/include/scipp/core/element/my_op.h:
namespace scipp::core::element {
constexpr auto my_op = overloaded{
    arg_list<std::tuple<double, int64_t>, std::tuple<double, int32_t>>,
    transform_flags::expect_no_variance_arg<0>,
    [](const auto &a, const auto &b) { return a + b; }};
};

// In scipp/core/include/scipp/variable/my_op.h:
namespace scipp::variable {
Variable my_op(const VariableConstView &a, const VariableConstView &b) {
    return transform(a, b, core::element::my_op);
}

• Here, variances for the first argument are disabled explicitly.

• Unit tests should be written independently for scipp::core::element::my_op.

• The lambdas for overloaded can be tested separately, i.e. test unit handling then test value handling for supported types

• scipp::variable::my_op should only have essential tests relying on correctness of transform and scipp::core::element::my_op.