Reading and Writing Files#

HDF5#

Scipp supports writing variables, data arrays, and dataset to HDF5 files. Reading of HDF5 is supported only for these scipp-specific files. Other HDF5-based formats are not supported at this point. For reading the HDF5-based NeXus files, see scippneutron.

Warning

We do not recommend to use Scipp HDF5 files for archiving or as the sole means of storing valuable data. The current Scipp HDF5 schema is not a standard and will likely be subject to change due to the early development status of scipp. Future versions of Scipp may not be able to read older files.

That being said, the file format is quite simple and based on the HDF5 standard so it would still be possible to recover data from such files in such a case. Note that the Scipp version is stored as an HDF5 attribute of the saved objects.

[1]:

import numpy as np
import scipp as sc

x = sc.Variable(dims=['x'], values=np.arange(10))
var = sc.Variable(dims=['x', 'y'], values=np.random.rand(9, 3))
a = sc.DataArray(data=var, coords={'x': x})

a.save_hdf5(filename='test.hdf5')

/home/runner/work/scipp/scipp/.tox/docs/lib/python3.10/site-packages/scipp/io/hdf5.py:368: VisibleDeprecationWarning: sc.DataArray.attrs has been deprecated and will be removed in Scipp v24.12.0. The deprecation includes sc.DataArray.meta and sc.DataArray.drop_attrs. For unaligned coords, use sc.DataArray.coords and unset the alignment flag. For other attributes, use a higher-level data structure.
  views = [data.coords, data.masks, data.attrs]

[2]:

b = sc.io.load_hdf5(filename='test.hdf5')

[3]:

[3]:

scipp.DataArray (1.29 KB)

Dimensions:
- x: 9
- y: 3
Coordinates: (1)
- x
  (x [bin-edge])
  int64
  𝟙
  0, 1, ..., 8, 9
```
Values:
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
```

Data: (1)

(x, y)

float64

𝟙

0.538, 0.979, ..., 0.419, 0.294

Values:
array([[0.53803423, 0.97873883, 0.4425589 ],
       [0.35276343, 0.91329559, 0.14062536],
       [0.51161732, 0.51951101, 0.31656653],
       [0.44003449, 0.41156455, 0.72907397],
       [0.47249576, 0.73492772, 0.29793852],
       [0.82333468, 0.74304156, 0.45856649],
       [0.91417504, 0.90533124, 0.75811618],
       [0.83950051, 0.9020681 , 0.06992582],
       [0.5595541 , 0.41926006, 0.29380018]])

CSV#

Note

CSV support requires pandas which must be installed separately.

CSV files can be read into datasets with scipp.io.load_csv. For example, given the following CSV-encoded data can be read into a dataset as shown:

[4]:

csv_content = '''a [m],b [s],c
1,5,9
2,6,10
3,7,11
4,8,12'''

[5]:

from io import StringIO

ds = sc.io.load_csv(StringIO(csv_content), header_parser='bracket')
ds

This example uses StringIO to load the data directly from a string. But load_csv can also load from a file on your hard drive or even from a remote server. Simply pass the path or URL of the file as the first argument. See also pandas.read_csv.

See scipp.io.load_csv for more options to customize how the data is structured in the dataset.

Using pandas#

The CSV reader shown above is a wrapper around pandas.read_csv and provides commonly used functionality. But pandas supports many more file readers for, among others, Excel, JSON, and XML files. See pandas IO tools for a complete list.

It is possible to use pandas manually to load these files and then convert the result to a Scipp dataset using from_pandas. For example, JSON can be read as follows:

[6]:

json = '''{"A [m]": {"0": 1, "1": 3, "2": 5},
"B [m/s]": {"0": 2, "1": 4, "2": 6}}'''

[7]:

import pandas as pd

df = pd.read_json(json)
df

/tmp/ipykernel_7482/3504722790.py:3: FutureWarning: Passing literal json to 'read_json' is deprecated and will be removed in a future version. To read from a literal string, wrap it in a 'StringIO' object.
  df = pd.read_json(json)

[7]:

	A [m]	B [m/s]
0	1	2
1	3	4
2	5	6

[8]:

sc.compat.from_pandas(df, header_parser='bracket')

NeXus#

Scipp has no built-in support for loading NeXus files. However, the scippneutron package can internally use Mantid to load such files, or any other Mantid-supported file type, see scippneutron and in particular scippneutron.load_with_mantid.