{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Structures\n",
    "\n",
    "To keep this documentation generic we typically use dimensions `x` or `y`, but this should *not* be seen as a recommendation to use these labels for anything but actual positions or offsets in space.\n",
    "\n",
    "## Variable\n",
    "\n",
    "### Basics\n",
    "\n",
    "[scipp.Variable](../generated/classes/scipp.Variable.rst#scipp.Variable) is a labeled multi-dimensional array.\n",
    "A variable has the following key properties:\n",
    "\n",
    "- `values`: a multi-dimensional array of values, e.g., a [numpy.ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)\n",
    "- `variances`: a (optional) multi-dimensional array of variances for the array values\n",
    "- `dims`: a list of dimension labels (strings) for each axis of the array\n",
    "- `unit`: a (optional) physical unit of the values in the array\n",
    "\n",
    "Note that variables, unlike [DataArray](data-structures.ipynb#DataArray) and its eponym [xarray.DataArray](http://xarray.pydata.org/en/stable/user-guide/data-structures.html#dataarray), variables do *not* have coordinate dicts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipp as sc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variables should generally be created using one of the available [creation functions](../reference/api.rst#creation-functions).\n",
    "For example, we can create a variable from a numpy array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var = sc.array(dims=['x', 'y'], values=np.random.rand(2, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "**Note:**\n",
    "\n",
    "Internally scipp is not using numpy, so the above makes a *copy* of the numpy array of values into an internal buffer.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can inspect the created variable as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.show(var)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var.unit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(var.variances)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variances must have the same shape as values, and units are specified using the [scipp.units](../reference/units.rst) module or with a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var = sc.array(dims=['x', 'y'],\n",
    "               unit='m/s',\n",
    "               values=np.random.rand(2, 4),\n",
    "               variances=np.random.rand(2, 4))\n",
    "sc.show(var)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var.variances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0-D variables (scalars)\n",
    "\n",
    "A 0-dimensional variable contains a single value (and an optional variance).\n",
    "The most convenient way to create a scalar variable is by multiplying a value by a unit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scalar = 1.2 * sc.units.m\n",
    "sc.show(scalar)\n",
    "scalar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Singular versions of the `values` and `variances` properties are provided:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(scalar.value)\n",
    "print(scalar.variance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An exception is raised from the `value` and `variance` properties if the variable is not 0-dimensional.\n",
    "Note that a variable with one or more dimension extent(s) of 1 contains just a single value as well, but the `value` property will nevertheless raise an exception."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating scalar variables with variances or with custom `dtype` or variances is possible using [scipp.scalar](../generated/functions/scipp.scalar.rst#scipp.scalar):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var_0d = sc.scalar(value=1.0, variance=0.5, dtype=sc.dtype.float32, unit='kg')\n",
    "var_0d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var_0d.value = 2.3\n",
    "var_0d.variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataArray\n",
    "\n",
    "### Basics\n",
    "\n",
    "[scipp.DataArray](../generated/classes/scipp.DataArray.rst#scipp.DataArray) is a labeled array with associated coordinates.\n",
    "A data array is essentially a [Variable](../generated/classes/scipp.Variable.rst#scipp.Variable) object with attached dicts of coordinates, masks, and attributes.\n",
    "\n",
    "A data array has the following key properties:\n",
    "\n",
    "- `data`: the variable holding the array data.\n",
    "- `coords`: a dict-like container of coordinates for the array, accessed using a string as dict key.\n",
    "- `masks`: a dict-like container of masks for the array, accessed using a string as dict key.\n",
    "- `attrs`: a dict-like container of \"attributes\" for the array, accessed using a string as dict key.\n",
    "\n",
    "The key distinction between coordinates (added via the `coords` property) and attributes (added via the `attrs` property) is that the former are required to match (\"align\") in operations between data arrays whereas the latter are not.\n",
    "\n",
    "`masks` allows for storing boolean-valued masks alongside data.\n",
    "All four have items that are internally a [Variable](../generated/classes/scipp.Variable.rst#scipp.Variable), i.e., they have a physical unit and optionally variances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = sc.DataArray(\n",
    "    data = sc.array(dims=['y', 'x'], values=np.random.rand(2, 3)),\n",
    "    coords={\n",
    "        'y': sc.array(dims=['y'], values=np.arange(2.0), unit='m'),\n",
    "        'x': sc.array(dims=['x'], values=np.arange(3.0), unit='m')},\n",
    "    attrs={\n",
    "        'aux': sc.array(dims=['x'], values=np.random.rand(3))})\n",
    "sc.show(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the `'aux'` attribute is essentially a secondary coordinate for the x dimension.\n",
    "The dict-like `coords` and `masks` properties give access to the respective underlying variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.coords['x']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.attrs['aux']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access to coords and attrs in a unified manner is possible with the `meta` property.\n",
    "Essentially this allows us to ignore whether a coordinate is aligned or not:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.meta['x']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.meta['aux']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike `values` when creating a variable, `data` as well as entries in the meta data dicts (`coords`, `masks`, and `attrs`) are *not* deep-copied on insertion into a data array.\n",
    "To avoid unwanted sharing, call the `copy()` method.\n",
    "Compare:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x2 = sc.zeros(dims=['x'], shape=[3])\n",
    "a.coords['x2_shared'] = x2\n",
    "a.coords['x2_copied'] = x2.copy()\n",
    "x2 += 123\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Meta data can be removed in the same way as in Python dicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del a.attrs['aux']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distinction between dimension coords and non-dimension coords, and coords and attrs\n",
    "\n",
    "When the name of a coord matches its dimension, e.g., if `d.coord['x']` depends on dimension `'x'` as in the above example, we call this coord *dimension coordinate*.\n",
    "Otherwise it is called *non-dimension coord*.\n",
    "It is important to highlight that for practical purposes (such as matching in operations) **dimension coords and non-dimension coords are handled equivalently**.\n",
    "Essentially:\n",
    "\n",
    "- **Non-dimension coordinates are coordinates**.\n",
    "- There is at most one dimension coord for each dimension, but there can be multiple non-dimension coords.\n",
    "- Operations such as value-based slicing that accept an input dimension and require lookup of coordinate values will only consider dimension coordinates.\n",
    "\n",
    "As mentioned above, the difference between coords and attrs is \"alignment\", i.e., only the former are compared in operations.\n",
    "The concept of dimension coords is unrelated to the distinction between `coords` or `attrs`.\n",
    "In particular, dimension coords could be made attrs if desired, and non-dimension coords can (and often are) \"aligned\" coords."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "[scipp.Dataset](../generated/classes/scipp.Dataset.rst#scipp.Dataset) is a dict-like container of data arrays.\n",
    "Individual items of a dataset (\"data arrays\") are accessed using a string as a dict key.\n",
    "\n",
    "In a dataset the coordinates of the sub-arrays are enforced to be *aligned*.\n",
    "That is, a dataset is not actually just a dict of data arrays.\n",
    "Instead, the individual arrays share their coordinates.\n",
    "It is therefore not possible to combine *arbitrary* data arrays into a dataset.\n",
    "If, e.g., the extents in a certain dimension mismatch, or if coordinate values mismatch, insertion of the mismatching data array will fail.\n",
    "\n",
    "Often a dataset is not created from individual data arrays.\n",
    "Instead we may provide a dict of variables (the data of the items), and dicts for coords:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = sc.Dataset(\n",
    "            data={\n",
    "                'a': sc.array(dims=['y', 'x'], values=np.random.rand(2, 3)),\n",
    "                'b': sc.array(dims=['y'], values=np.random.rand(2)),\n",
    "                'c': sc.scalar(value=1.0)},\n",
    "             coords={\n",
    "                 'x': sc.array(dims=['x'], values=np.arange(3.0), unit='m'),\n",
    "                 'y': sc.array(dims=['y'], values=np.arange(2.0), unit='m'),\n",
    "                 'aux': sc.array(dims=['x'], values=np.random.rand(3))})\n",
    "sc.show(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d.coords['x'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The name of a data item serves as a dict key.\n",
    "Item access returns a new data array which is a view onto the data in the dataset and its corresponding coordinates, i.e., no deep copy is made:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.show(d['a'])\n",
    "d['a']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `copy()` method to turn the view into an independent object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "copy_of_a = d['a'].copy()\n",
    "copy_of_a += 17  # does not change d['a']\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each data item is linked to its corresponding coordinates, masks, and attributes.\n",
    "These are accessed using the `coords` , `masks`, and `attrs` properties.\n",
    "The variable holding the data of the dataset item is accessible via the `data` property:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['a'].data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For convenience, properties of the data variable are also properties of the data item:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['a'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['a'].variances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['a'].unit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Coordinates of a data item include only those that are relevant to the item's dimensions, all others are hidden.\n",
    "For example, when accessing `'b'`, which does not depend on the `'y'` dimension, the coord for `'y'` as well as the `'aux'` coord are not part of the item's `coords`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.show(d['b'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, when accessing a 0-dimensional data item, it will have no coordinates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.show(d['c'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All variables in a dataset must have consistent dimensions.\n",
    "Thanks to labeled dimensions, transposed data is supported:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['d'] = sc.array(dims=['x', 'y'], values=np.random.rand(3, 2))\n",
    "sc.show(d)\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When inserting a data array or variable into a dataset ownership is shared by default.\n",
    "Use the `copy()` method to avoid this if undesirable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d['a_shared'] = a\n",
    "d['a_copied'] = a.copy()\n",
    "a += 1000\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The usual `dict`-like methods are available for `Dataset`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name in d:\n",
    "    print(name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'a' in d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "'e' in d"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}