LazyArrayDict#

class torch_brain.data.LazyArrayDict(**kwargs)[source]#

Bases: torch_brain.data.arraydict.ArrayDict

Lazy variant of ArrayDict. The data is not loaded until it is accessed. This class is meant to be used when the data is too large to fit in memory, and is intended to be intantiated via. LazyArrayDict.from_hdf5.

Note

To access an attribute without triggering the in-memory loading use self.__dict__[key] otherwise using self.key or getattr(self, key) will trigger the lazy loading and will automatically convert the h5py dataset to a numpy array as well as apply any outstanding masks.

select_by_mask(mask)[source]#

Index all arrays with a boolean mask and return a copy.

Lazy attributes will remain lazy, and masking will be applied to them upon access.

Parameters:

mask (ndarray | list | tuple | _SupportsArray) – Boolean array used for masking. The mask needs to be 1-dimensional, and of equal length as the object itself. Anything array-like (e.g. a list) is accepted and cast to a numpy array.

classmethod from_dataframe(df, unsigned_to_long=True)[source]#

Creates an ArrayDict object from a pandas DataFrame.

The columns in the DataFrame are converted to arrays when possible, otherwise they will be skipped.

Parameters:

  • df – DataFrame.

  • unsigned_to_long – If True, automatically converts unsigned integers to int64. Defaults to True.

to_hdf5(file)[source]#

Saves the data object to an HDF5 file.

Parameters:

file – HDF5 file.

import h5py
from torch_brain.data import ArrayDict

data = ArrayDict(
    unit_id=["unit01", "unit02"],
    brain_region=["M1", "M1"],
    waveform_mean=np.zeros((2, 48)),
)

with h5py.File("data.h5", "w") as f:
    data.to_hdf5(f)
classmethod from_hdf5(file)[source]#

Loads the data object from an HDF5 file.

Parameters:

file – HDF5 file.

import h5py
from torch_brain.data import ArrayDict

with h5py.File("data.h5", "r") as f:
    data = ArrayDict.from_hdf5(f)
__contains__(key)#

Returns True if the attribute key is present in the data.

Return type:

bool

__len__()#

Returns the first dimension shared by all attributes.

keys()#

Returns a list of all array attribute names.

Return type:

list[str]

materialize()#

Materializes the data object, i.e., loads into memory all of the data that is still referenced in the HDF5 file.

Return type:

ArrayDict