Data file

Data channel methods, unless specified should not be called directly.

class glitter2.storage.data_file.DataChannelBase(name: str, num: int, block: nixio.block.Block, data_file: glitter2.storage.data_file.DataFile, **kwargs)

Bases: object

Base class for data channels stored in a DataFile.

block: nixio.block.Block = None: The block containing the channel data.

property channel_config_dict: dict

Reads/writes the channel metadata from the current NixIO file into/from a dict.

Warning

This property reads/writes to the file when accessed.

See DataFile.set_file_data() for the metadata requirements if setting manually outside the GUI.

copy_data(channel: Union[glitter2.storage.data_file.DataChannelBase, glitter2.storage.data_file.TemporalDataChannelBase, glitter2.storage.data_file.EventChannelData, glitter2.storage.data_file.PosChannelData, glitter2.storage.data_file.ZoneChannelData]): Copies this channel’s data, if any, into the given channel.

create_initial_data(): Creates whatever initial data structures are needed for storing the channel data.

data_file: glitter2.storage.data_file.DataFile = None: The DataFile this channel belongs to.

metadata: nixio.section.Section = None: The metadata section that stores the channel metadata.

name: str = '': The name of the channel in the file. This is not the user facing name, but rather the internal NixIO name of the block containing the channel.

num: int = 0: The unique number used to identify the channel as stored in DataFile.event_channels, DataFile.pos_channels, or DataFile.zone_channels.

read_initial_data(reopen=False): Reads the channel data previously written to the file.

reopen_file(block: nixio.block.Block)

Repopulates the data links from the provided block.

Used after re-opening an unchanged nix file.

class glitter2.storage.data_file.DataFile(nix_file: nixio.file.File, unsaved_callback=<function _unsaved_callback>)

Bases: object

Data file interface to the NixIO file that stores the video file annotated data.

Data for the timestamps and all channels are similarly stored in a list of arrays. See timestamps_arrays for details.

Once a file is opened with open_file(), the notify_xxx form a state machine and must be called as appropriate.

Specifically, the first timestamp added from the video file with notify_add_timestamp() must be the very first frame’s timestamp and it must be followed by a call to notify_saw_first_timestamp().

Then each frame must be read sequentially with no skipping. If a frame is skipped at any time, e.g. by seeking, before we add the new timestamp with notify_add_timestamp(), you must call notify_interrupt_timestamps() to indicate the seek.

Whenever we read the last timestamp (even if we skipped some intermediate timestamps, e.g. due to a seek) notify_saw_last_timestamp() must called after adding that timestamp with notify_add_timestamp().

If we have seen the first and last timestamp and each consecutive pair of timestamps were seen with no notify_interrupt_timestamps() between them, that’s when we consider having seen all frames and saw_all_timestamps is automatically set to the True and timestamps becomes the singular timestamps array.

property app_config_dict: dict

Reads/writes the app config data into/from a dict.

It does not include the channel config data, just the application config. The app config can be generated most simply from get_app_config_data().

Warning

This property reads/writes to the file when accessed.

app_config_section: nixio.section.Section = None: Stores the overall application configuration that was used with this video file.

condition_timestamp(t: float) → Optional[float]

If timestamp t is within a interval of seen timestamps in timestamps_arrays, it returns the closest seen timestamp to t. Otherwise it returns None indicating the timestamp is outside of any timestamps interval.

When reading timestamps from the video file, different systems may read slightly different timestamps. Conditioning the timestamp before use ensures identical behavior when re-opening files on different systems.

create_channel(channel_type: str) → Union[glitter2.storage.data_file.DataChannelBase, glitter2.storage.data_file.TemporalDataChannelBase, glitter2.storage.data_file.EventChannelData, glitter2.storage.data_file.PosChannelData, glitter2.storage.data_file.ZoneChannelData]

Creates a channel of the given type.

Can be one of 'event', 'pos', or 'zone'.

delete_channel(i: int)

Deletes the channel corresponding to the ID as stored in event_channels, pos_channels, or zone_channels.

The channel ID is stored in DataChannelBase.num.

Creates a new channel with the same type and with the same data and metadata as the given channel and returns the new channel.

The name key’s value in each channel’s metadata should be unique across all channels, so the name value should be changed afterwards. E.g.:

new_channel = data_file.duplicate_channel(channel)
metadata = new_channel.channel_config_dict
metadata['name'] = 'new name'
new_channel.channel_config_dict = metadata

event_channels: Dict[int, glitter2.storage.data_file.EventChannelData] = {}

Map of all the EventChannelData instances.

Maps a globally unique ID, associated with the channel to the EventChannelData instance.

ffpyplayer_version: str = '': The ffpyplayer version used to create the file originally. Read only.

get_channel_from_id(i: int) → glitter2.storage.data_file.DataChannelBase

Given the unique channel ID as it’s stored in event_channels, pos_channels, or zone_channels it returns the corresponding channel.

The channel ID is stored in DataChannelBase.num.

static get_file_glitter2_version(filename) → Optional[str]

Gets the glitter version used to create the nixio file.

If the file or data is invalid, returns None.

static get_file_video_metadata(filename) → dict: Returns the video metadata dictionary of the nixio file.

glitter2_version: str = '': The glitter version used to create the file originally. Read only.

property has_content: Returns whether any video timestamps has yet been added to the file.

init_new_file()

Initializes a newly created empty NixIO file with all the required data structures.

Typically it’s used e.g.:

nix_file = nixio.File.open(filename, nixio.FileMode.Overwrite)
data_file = DataFile(nix_file=nix_file)
data_file.init_new_file()

Note

This method automatically class open_file().

is_end_timestamp(t: float) → bool: Returns whether the timestamp is the last timestamp of a interval.

nix_file: nixio.file.File = None: NixIO data file.

notify_add_timestamp(t: float) → int

Adds the next timestamp to the file.

We assume that this is called frame by frame with no skipping unless notify_interrupt_timestamps() was called.

Also, the timestamps t must be either outside any existing timestamp interval, in which case the timestamp must have been unseen before. Or, if it is within an existing interval, the timestamps must have been seen before. In both cases, the timestamp must be larger or equal than the first timestamp.

Parameters: t – the next timestamp.
Returns: The key in timestamps_arrays containing the timestamp.

notify_interrupt_timestamps(): Must be called whenever the video is seeked and the next timestamp to notify_add_timestamp() may not follow the last timestamp.

notify_saw_first_timestamp(): Must be called after we saw and added the timestamp of the first frame of the video file. By first we mean literally the first frame of the video file.

notify_saw_last_timestamp(): Must be called after we saw and added the timestamp of the last frame of the video file.

open_file()

Loads the data from the file and initializes the instance. Can be called multiple times.

Use after updating the file or opening an existing file. E.g.:

nix_file = nixio.File.open(filename, nixio.FileMode.ReadWrite)
data_file = DataFile(nix_file=nix_file)
data_file.upgrade_file()
data_file.open_file()

pad_all_channels_to_num_frames_interval(array_num=None)

Similar to pad_channel_to_num_frames_interval(), but for all the channels.

If array_num is specified, we pad that array, other wise we use _last_timestamps_n if it’s not None.

pad_channel_to_num_frames_interval(channel: glitter2.storage.data_file.TemporalDataChannelBase)

Pads the channel data arrays to the current number of timestamps.

When adding timestamps, we don’t increase all the channels data arrays with each new timestamp, but only do it when the channel data is updated or when the file is saved etc. This increases the data arrays size to the timestamps size.

pixels_per_meter: float = 0.0: The pixels per meter of the video, if known. Read only.

pos_channels: Dict[int, glitter2.storage.data_file.PosChannelData] = {}

Map of all the PosChannelData instances.

Maps a globally unique ID, associated with the channel to the PosChannelData instance.

read_channels_config() → Tuple[Dict[int, dict], Dict[int, dict], Dict[int, dict]]

Returns a tuple of the event, pos, and zone channel metadata.

Each item is a dictionary whose keys are the channel’s unique ID as e.g. in event_channels and whose values is a dict with the channel’s metadata.

reopen_file(nix_file: nixio.file.File)

Repopulates the data links with the provided file.

Used to reopen a nix file that has been closed after DataFile is initialized if no changes occurred to the file.

saw_all_timestamps = False

Whether we have seen all the timestamps of the video.

If we haven’t, it’s possible there are holes in the video and timestamps_arrays may have more than one array.

set_default_video_metadata(metadata: dict): Sets the video file metadata, but only for the metadata that has not yet been set.

set_file_data(video_file_metadata: Dict, saw_all_timestamps: bool, timestamps: List[Union[numpy.ndarray, List[float]]], event_channels: List[Tuple[dict, List[Union[numpy.ndarray, List[bool]]]]], pos_channels: List[Tuple[dict, List[Union[numpy.ndarray, List[Tuple[float, float]]]]]], zone_channels: List[dict])

Sets the data of the file at once.

The video metadata and timestamps can be read most easily using get_file_data().

For the channels, the metadata dict should contain a name keyword whose value is a globally unique name for the channel, unique across all the channels.

For zone_channels, the metadata should also contain a shape_config keyword whose value is a dict with the shape metadata as given by get_state(). To create a shape, use create_shape(). E.g.:

point = PaintCircle.create_shape([0, 0])
circle = PaintCircle.create_shape([0, 0], 5)
ellipse = PaintEllipse.create_shape([0, 0], 5, 10, 3.14)
polygon = PaintPolygon.create_shape(
    [0, 0, 300, 0, 300, 800, 0, 800], [0, 0])
polygon_metadata = polygon.get_state()

E.g. to create a new NixIO file with the timestamps of a video file:

# create file
nix_file = nixio.File.open(filename, nixio.FileMode.ReadWrite)
data_file = DataFile(nix_file=nix_file)
data_file.init_new_file()
# read timestamps
timestamps, metadata = GlitterPlayer.get_file_data(video_file)
# set just the timestamps
data_file.set_file_data(
    video_file_metadata=metadata, saw_all_timestamps=True,
    timestamps=[timestamps], event_channels=[], pos_channels=[],
    zone_channels=[])
# re-load data
data_file.open_file()

set_pixels_per_meter(value: float): Sets the pixels per meter of the video file.

timestamp_data_map: Dict[float, Tuple[int, int]] = {}

Maps timestamps to their (key, index) in timestamps_arrays.

For each known timestamp, it maps the timestamp to (key, index), where key` is the key in timestamps_arrays and index is the index in that array. Such that k, i = timestamp_data_map[t]; timestamps_arrays[k][i] == t.

timestamp_intervals_end: List[float] = []: List of timestamps_arrays end interval timestamps sorted by value.

timestamp_intervals_ordered_keys: List[int] = []: The keys of the timestamp arrays form timestamps_arrays sorted temporally such that the first timestamp in each array corresponding to the key is strictly increasing.

timestamp_intervals_start: List[float] = []: List of timestamps_arrays start interval timestamps sorted by value.

timestamps: nixio.data_array.DataArray = None

The first timestamps data array containing the very first timestamps of the video file, created when the data file is created. See timestamps_arrays.

If there’s only one array in timestamps_arrays (e.g. after we saw all the timestamps of the video file), this is referring to it.

timestamps_arrays: Dict[int, nixio.data_array.DataArray] = {}

Map of disjointed timestamps arrays representing known closed intervals of the video file.

Consider how data is coded. We play frame after frame and the user annotates each frame. And as we see each timestamps in a strictly-increasing order, we could just have a list of timestamps and corresponding state that is continuously increasing by one element as we see a new frame.

Unfortunately, a user may skip some frames by seeking ahead, thereby breaking continuity, because now we have missing timestamps in the middle and we don’t know what they are. Therefore we have a list, or mapping of timestamps arrays, each timestamp array represents a known interval of timestamps during which we didn’t seek and we have every timestamp in that interval.

The keys of timestamps_arrays are not meaningful, except for key 0 which is the same array as timestamps and is the array with the first timestamps of the video.

To get keys of the timestamp arrays ordered temporally, use timestamp_intervals_ordered_keys.

unsaved_callback: Callable = None: Callback that is called whenever the data file changes.

upgrade_file()

Upgrades file to add any missing data structures that may have been added in newer versions of glitter, since the file was created.

open_file() should be called after this.

property video_metadata_dict: dict

Reads/writes the video metadata from the current NixIO file into/from a dict.

This is the metadata from get_file_data().

Warning

This property reads/writes to the file when accessed.

video_metadata_section: nixio.section.Section = None: Stores the metadata of the video file.

write_channels_config(event_channels: Optional[Dict[int, dict]] = None, pos_channels: Optional[Dict[int, dict]] = None, zone_channels: Optional[Dict[int, dict]] = None)

Updates the metadata to the provided values for each channel.

xxx_channels is a dictionary whose keys are the channel’s unique ID as e.g. in event_channels and whose values is a dict with the channel’s new metadata.

zone_channels: Dict[int, glitter2.storage.data_file.ZoneChannelData] = {}

Map of all the ZoneChannelData instances.

Maps a globally unique ID, associated with the channel to the ZoneChannelData instance.

class glitter2.storage.data_file.EventChannelData(**kwargs)

Bases: glitter2.storage.data_file.TemporalDataChannelBase

Channel that stores event data.

For each timestamp we store a bool representing whether the event occurred at this timestamp.

Metadata: The only required metadata is the name key (unique across all channels).

create_data_array(n: int = 0, count: Optional[int] = None)

Creates a data array when a new timestamps interval is created.

Parameters

n – The data_arrays key to use for the array. This corresponds to the interval’s DatFile.timestamps_arrays key.
count – The size of the array to create, or empty if it’s None.

default_data_value = array([0], dtype=uint8)

The default value used before the user modifies the data.

It is False (0) for the event channel.

get_timestamp_value(t: float) → bool: Returns the value of the data array for the given timestamp t.

get_timestamps_modified_state() → Dict[float, bool]: Returns a dict whose keys are timestamps and values indicate whether the data for the timestamp has been changed from the default_data_value (True means it’s not the default).

set_timestamp_value(t: float, value: bool): Changes the value of the data array for the given timestamp t to value.

class glitter2.storage.data_file.PosChannelData(**kwargs)

Bases: glitter2.storage.data_file.TemporalDataChannelBase

Channel that stores position data.

For each timestamp we store a tuple of the (x, y) position at this timestamp.

Metadata: The only required metadata is the name key (unique across all channels).

create_data_array(n: int = 0, count: Optional[int] = None)

Creates a data array when a new timestamps interval is created.

Parameters

n – The data_arrays key to use for the array. This corresponds to the interval’s DatFile.timestamps_arrays key.
count – The size of the array to create, or empty if it’s None.

default_data_value = array([[-1., -1.]])

The default value used before the user modifies the data.

It is a tuple of (-1, -1) corresponding to x and y for the position channel.

get_timestamp_value(t: float) → Tuple[float, float]: Returns the value of the data array for the given timestamp t.

get_timestamps_modified_state() → Dict[float, bool]: Returns a dict whose keys are timestamps and values indicate whether the data for the timestamp has been changed from the default_data_value (True means it’s not the default).

set_timestamp_value(t: float, value: Union[Tuple[float, float], numpy.ndarray]): Changes the value of the data array for the given timestamp t to value.

class glitter2.storage.data_file.TemporalDataChannelBase(**kwargs)

Bases: glitter2.storage.data_file.DataChannelBase

Base class for data channels that have some kind of temporal dimension.

copy_data(channel: Union[glitter2.storage.data_file.DataChannelBase, glitter2.storage.data_file.TemporalDataChannelBase, glitter2.storage.data_file.EventChannelData, glitter2.storage.data_file.PosChannelData, glitter2.storage.data_file.ZoneChannelData]): Copies this channel’s data, if any, into the given channel.

create_data_array(n: int = 0, count: Optional[int] = None)

Creates a data array when a new timestamps interval is created.

Parameters

n – The data_arrays key to use for the array. This corresponds to the interval’s DatFile.timestamps_arrays key.
count – The size of the array to create, or empty if it’s None.

create_initial_data(): Creates whatever initial data structures are needed for storing the channel data.

data_array: nixio.data_array.DataArray = None

The data array storing the channel data, corresponding to DataFile.timestamps. If data_arrays has only one array, e.g. after seeing all frames, this is it.

See data_arrays.

data_arrays: Dict[int, nixio.data_array.DataArray] = {}

The channel data for the disjointed frame intervals.

This corresponds exactly to DatFile.timestamps_arrays with the arrays of the same keys containing the data for the corresponding intervals.

default_data_value = None: The per-channel type default data used before the user modifies the data. E.g. for an event channel it may default to False.

get_timestamp_value(t: float) → Any: Returns the value of the data array for the given timestamp t.

get_timestamps_modified_state() → Dict[float, bool]: Returns a dict whose keys are timestamps and values indicate whether the data for the timestamp has been changed from the default_data_value (True means it’s not the default).

merge_arrays(arr_num1: int, arr_num2: int)

Merges the data arrays into a single array.

The same as DatFile._merge_timestamp_channels_arrays, but for data arrays.

pad_channel_to_num_frames(array_num: int, size: int)

Pads the channel data arrays to the given size, if it’s smaller.

See DataFile.pad_channel_to_num_frames_interval.

read_initial_data(reopen=False): Reads the channel data previously written to the file.

reopen_file(block: nixio.block.Block)

Repopulates the data links from the provided block.

Used after re-opening an unchanged nix file.

reset_data_to_default(): Resets all the data array values to the default_data_value.

set_channel_data(data: Union[list, numpy.ndarray], mask: Optional[numpy.ndarray] = None)

Changes the values of the data array (optionally masked).

Must have saw_all_timestamps before this can be used.

Number of items in data must be the same as the number of true elements in mask, if provided.

set_timestamp_value(t: float, value: Any): Changes the value of the data array for the given timestamp t to value.

class glitter2.storage.data_file.ZoneChannelData(name: str, num: int, block: nixio.block.Block, data_file: glitter2.storage.data_file.DataFile, **kwargs)

Bases: glitter2.storage.data_file.DataChannelBase

Channel that stores a static zone.

Unlike the event and position channels, this contains no data and only metadata. The metadata stores the shape of the drawn zone as e.g. a circle, a polygon, an ellipse, etc.

Metadata: The required metadata is the name key (unique across all channels) and the shape_config key data. The shape data is the state of a PaintShape using get_state().

create_initial_data(): Creates whatever initial data structures are needed for storing the channel data.

read_initial_data(reopen=False): Reads the channel data previously written to the file.

glitter2.storage.data_file.read_nix_prop(prop)

Returns a nox property value across different nix versions.

Parameters: prop – the nix Property.
Returns: The value stored in the property.