Storage Controller

Handles all Ceed data aspects; file storage, loading and saving of configuration data, and acquisition and creation of experimental data.

When the Ceed GUI starts, it creates an instances of CeedDataWriterBase for storage management. Using the instance it loads, imports, and exports H5 and yaml files.

Experiment lifecycle

When an experiment starts, CeedDataWriterBase.prepare_experiment() is internally called to create a new section in the CeedDataWriterBase.nix_file where the streamed experiment data will be saved. It also starts an internal thread that saves the streamed data.

Then, during the experiment, Ceed sends data to be saved to that thread using a queue with the CeedDataWriterBase.add_frame(), CeedDataWriterBase.add_frame_flip(), CeedDataWriterBase.add_led_state(), and CeedDataWriterBase.add_debug_data() methods.

When the experiment ends, CeedDataWriterBase.stop_experiment() is called to finish writing any queued data and then stop the thread.

Projector-electrode synchronization

To support the temporal aligning of the projector data generated by Ceed with the electrode voltage data saved by MCS, Ceed stores some metadata with each frame that is recorded by MCS. See DataSerializerBase for details.

class ceed.storage.controller.CeedDataWriterBase(**kwargs)

Bases: kivy._event.EventDispatcher

Handles the loading and saving of Ceed H5 data files, importing and exporting of the app settings/stages/functions/shapes, and streaming live experiment data to the H5 data files.

root_path: The directory part of the path where the data is saved.

backup_interval

How frequently the backup file is flushed to disk.

The backup file is the one into which all new data is written to until saved explicitly in the GUI to the current data file. Until that happens, we write date to the backup file.

filename

The full filename of the h5 data file opened/saved in the GUI.

If the user hasn’t yet saved the data it is blank.

read_only_file: Whether the data file was opened in readonly mode.

backup_filename = ''

The filename of the backup file where live data file is written to.

The internal backup file always exists and all updates is done to this file, which is automatically created and managed by Ceed. When a user saves the data in the GUI, this file is flushed and copied to filename.

When the user discards the unsaved data in the GUI or if they open an existing file, filename (if there’s one) is copied and used as a base for the backup_filename.

nix_file: nixio.file.File = None: The nix.File object managing the open backup_filename.

compression

Whether the h5 data file should internally compress the data that it writes.

This is handled internally by the H5 library, with no external difference in how the file is loaded/saved/accessed, except that the file size may be smaller when compressed. Additionally, it may take a little more CPU when saving experiment data if compressions is enabled.

Valid values are "ZIP", "None", or "Auto".

has_unsaved: Indicates whether the file changed and needs to be saved or discarded by user before Ceed can exit.

config_changed: Indicates whether the app config changed and needs to be written again to the data file.

data_queue = None: Queue of incoming data to be saved by the data thread into the file.

data_thread = None: Second thread that is used to write live streamed experiment data to prevent main thread blocking.

data_lock = None: Lock used to safely write streamed experiment data to the file and to prevent the main thread from discarding the file while the second thread writes to it.

func_display_callback = None: Callback called when functions are loaded from a config file.

stage_display_callback = None: Callback called when stages are loaded from a config file.

clear_all_callback = None: Callback called when all the functions/stages/shapes are cleared due to re-loading the config.

backup_event = None: Clock trigger event that controls the periodic callback to write_changes() due to backup_interval.

property nix_compression: Gets the nix.Compression enum value that corresponds to the user selected compression.

get_function_plugin_contents(): Returns plugin_sources from the app’s FunctionFactoryBase

get_stage_plugin_contents(): Returns plugin_sources from the app’s StageFactoryBase

gather_config_data_dict(stages_only=False, use_cache=False)

Accumulates all the configuration data from all the classes into a single dict and returns it.

Parameters

stages_only – If True, it only returns the data required to be able to load the stages/functions/shapes, but not the app config, e.g. player config, mea/camera rotation etc.
use_cache – If True, it’ll get the state using the cache from previous times the state was read and cached, if the cache exists.

clear_existing_config_data(): Clears and resets all the user added data such as stages, functions, and shapes.

apply_config_data_dict(data, stages_only=False, requires_app_settings=True)

Applies the data from a config dict generated with gather_config_data_dict().

If stages_only, it’ll only import the data required to recreates the stages/functions/shapes but it won’t import the other app config.

If requires_app_settings, but the data only contains stage/shape/function config, it raises an exception.

get_filebrowser_callback(func, clear_data=False, ext=None, **kwargs): Called by the GUI when user browses for a file.

ui_close(app_close=False)

Called when the UI asked to close a file.

We create a new backup file immediately if the app doesn’t close.

If unsaved, will prompt if want to save

create_file(filename, overwrite=False)

Creates a new h5 data file and initializes it.

It creates the backup_filename and also copies it over to filename if one was specified.

static upgrade_file(nix_file)

Given an existing h5 data file, it’ll add any missing sections that have been added by newer Ceed versions since the file was created.

They are added with default values.

open_file(filename, read_only=False)

Loads the H5 file’s config and opens the file for usage.

It creates a backup_filename from on filename and saves filename.

close_file(force_remove_autosave=False): Closes the backup file without saving the data to the main file. But if data was unsaved it leaves the backup file, otherwise it removes it. If force_remove_autosave, it always removes it.

import_file(filename, stages_only=False)

Imports and applies the config data from the H5 file.

If stages_only, it only applies the stages/functions/shapes data, but not the app config, e.g. player config, mea/camera rotation etc.

discard_file(): Closes and discards the backup_filename and then re-opens the filename (if any) and creates a new backup_filename from it.

save_as(filename, overwrite=False): Saves the data in backup_filename to a new h5 file and opens that file.

save(filename=None, force=False): Saves any changes to backup_filename and copies the backup_filename to filename if it’s not empty. Otherwise, it saves it to filename if that is not empty.

write_changes(*largs, scheduled=False)

Writes all unsaved changes to backup_filename and flushes it.

Parameters: scheduled – Whether the function is called due to a scheduled saving of the config.

static save_config_to_yaml(filename, shape_factory=None, function_factory=None, stage_factory=None, overwrite=False)

Exports the shape/function/stage data to a yaml file so it can be imported from the GUI.

It can be used to create shapes, functions, and stages from a script, dump it to a file, and then import it from the GUI.

Parameters

filename – The filename to save to (should end with .yaml or .yml for ease of use).
shape_factory – The CeedPaintCanvasBehavior containing the shapes that will be dumped to the file.
function_factory – The FunctionFactoryBase containing the functions that will be dumped to the file.
stage_factory – The StageFactoryBase containing the stages that will be dumped to the file.
overwrite – Whether to overwrite the file if it exists.

E.g.:

from ceed.shape import CeedPaintCanvasBehavior, CeedPaintCircle
from ceed.function import FunctionFactoryBase,
    register_all_functions
from ceed.stage import StageFactoryBase, register_all_stages
from ceed.storage.controller import CeedDataWriterBase

shape_factory = CeedPaintCanvasBehavior()
function_factory = FunctionFactoryBase()
stage_factory = StageFactoryBase(
    function_factory=function_factory, shape_factory=shape_factory)
register_all_functions(function_factory)
register_all_stages(stage_factory)

stage_cls = stage_factory.get('CeedStage')
stage = stage_cls(
    function_factory=function_factory, stage_factory=stage_factory,
    shape_factory=shape_factory, name='best stage')
stage_factory.add_stage(stage)

circle = CeedPaintCircle.create_shape(
    center=(700, 300), radius=200, name='circle')
shape_factory.add_shape(circle)
stage.add_shape(circle)

func_cls = function_factory.get('ConstFunc')
func = func_cls(
    function_factory=function_factory, name='stable', duration=1,
    a=0.5)
stage.add_func(func)

CeedDataWriterBase.save_config_to_yaml(
    'config.yml', shape_factory=shape_factory,
    function_factory=function_factory, stage_factory=stage_factory)

See the example scripts directory or the docs for a complete and worked example.

write_yaml_config(filename, overwrite=False, stages_only=False)

Exports the config data to a yaml file.

Unlike a H5 file, the yaml file contains only the configuration data.

If stages_only, it only exports the stages/functions/shapes data, but not the app config, e.g. player config, mea/camera rotation etc.

read_yaml_config(filename, stages_only=False, requires_app_settings=True)

Imports and applies the config data from a yaml file that was previously exported with write_yaml_config().

If stages_only, it’ll only import the data required to recreates the stages/functions/shapes but it won’t import the other app config.

If requires_app_settings, but the data only contains stage/shape/function config, it raises an exception.

write_config(config_section=None, use_cache=False)

Writes the app config data and stages/functions/shapes to the backup_filename.

Parameters: use_cache – If True, it’ll get the state using the cache from previous times the config was read and cached, if the cache exists.

read_config(config_section=None) → dict: Reads the app config data and stages/functions/shapes from the backup_filename.

add_log_item(text): Adds the text to the currently open H5 file app log data.

add_app_log(text): Adds the text to the app log (indirectly) and shows it to the user in the app log screen.

get_log_data(): Gets the app log data, saved with add_log_item() from the currently open data file.

load_last_fluorescent_image(filename): Returns the last saved camera image from the H5 data file.

add_image_to_file(img, notes=''): Stores the image and the optional notes in the currently open H5 data file.

write_fluorescent_image(block, img, postfix=''): Writes the given image to the given experiment or image block in the currently open H5 data file.

get_num_fluorescent_images(): Same as get_file_num_fluorescent_images(), but for the currently open H5 data file.

static get_file_num_fluorescent_images(nix_file): Returns the total number of images saved to the H5 data file (not including images stored in each experiment).

get_experiment_numbers() → List[str]

Returns list of experiments saved in the currently open H5 data file.

They are each a number, even though they are returned as strings.

get_experiment_notes(experiment_block_number): Gets the experiment notes from the given experiment number stored in the currently open H5 data file.

set_experiment_notes(experiment_block_number, text): Sets the experiment notes for the given experiment number stored in the currently open H5 data file.

get_experiment_config(experiment_block_number) → dict: Gets the app config dict from the given experiment number stored in the currently open H5 data file.

get_config_mea_matrix_string(experiment_block_number=None)

Gets the yaml-string encoded mea_transform from the given experiment number stored in the currently open H5 data file.

If no experiment is provided, it gets it from the app config.

set_config_mea_matrix_string(experiment_block_number, config_string, new_config_string): Sets the yaml-string encoded mea_transform for the given experiment number stored in the currently open H5 data file.

get_experiment_metadata(experiment_block_number): Reads the experiment metadata (notes, time, etc.) from the given experiment in the currently open H5 data file.

get_saved_image(image_num) → dict: Reads and returns the requested image and its metadata from the currently open H5 data file.

static get_blocks_experiment_numbers(blocks, ignore_list=None) → List[str]

Returns list of experiments in the given nix blocks.

ignore_list, if provided, is a list of experiment numbers to skip.

Each experiment name is a number, represented as a str.

static get_experiment_block_name(experiment_num: Union[str, int]) → str: Converts the experiment number to the full name used to name the block that stores the experiment data.

collect_experiment(block, shapes, frame_bits, frame_counter, frame_time, frame_time_counter, queue, lock, led_state, event_data, event_data_count): Method that runs in its own thread, for each experiment, that gets incoming experiment data over the data_queue and saves it to the file.

prepare_experiment(stage_name, used_shapes)

Initializes the file with the current app config and prepares it to accept streamed experiment data for a new experiment using the given stage.

It also starts collect_experiment() in a new thread.

stop_experiment(): Stops the thread running the experiment data stream listener, collect_experiment(), and cleans up after the experiment.

add_frame(data)

Adds experiment data for a single video frame displayed on screen, to the data_queue.

data is a (counter_bits, shape_rgba) tuple. counter_bits is a dict whose "count" key is the frame global counter value (count) of the frame and "bits" is the 24-bit corner pixel value sent with the frame (DataSerializerBase.get_bits()).

shape_rgba is a dict whose keys are the names of shapes and whose values are the rgba 4-tuple intensity value of that shape for this frame.

add_frame_flip(data)

Adds experiment data for a single video frame displayed on screen, after it has been sent to the GPU and displayed, to the data_queue.

data is a dict whose "count" key is the frame global counter value (count) of the frame and "t" is the approximate time when the frame was finished being displayed.

add_led_state(count, r, g, b)

Adds experiment data about the projector LED state to the data_queue.

count is the count of the frame when it was set, (r, g, b) is each a bool indicating if the r, g, and b LEDs were set each ON or OFF.

add_debug_data(name, data)

Adds experiment debug data to the data_queue.

name is the name of the debug data. If an array with the name f'debug_{name}' already exists the data is appended to it, otherwise such an array is created and data appended. This allows streaming arbitrary data to the file.

data is a n-dimensional array. Subsequent data will be appended to the first axis of the array in the file and it must match the dimensionality of the original data.

add_event_data(data: bytes)

Adds event data logged during the experiment to the data_queue to be saved.

data is a json encoded bytes object, encoded with orjson. It contains a list of the most recent logged data. See event_data for details and for the decoded logs.

class ceed.storage.controller.DataSerializerBase

Bases: kivy._event.EventDispatcher

To facilitate temporal data alignment between the Ceed data (shape intensity) and MCS (electrode data), Ceed outputs a bit pattern in the corner pixel that is output by the projector controller as a bit pattern on its digital port, and is recorded by MCS.

Post-experiment we locate each Ceed frame in the recorded MCS data by locating the Ceed frame bit-pattern in the MCS data, thereby locating the sample index in the MCS data, corresponding to the Ceed frame.

While we have 24 bits to use from the Ceed side, MCS can only record upto 16 bits, possibly less depending on the config, so we have to be flexible on the bit-pattern to send. Our bit-pattern has three components:

A clock bit that alternates low/high for each frame, starting with high. The clock bit is set by clock_idx.
Some bits are dedicated to simple short counter that overflows back to zero at the top. E.g. if we dedicate 3-bits for the counter, then it’ll go 0,1,…,6,7,0,1,…,6,7,0,1,… etc.

The bits used by the short counter is set with short_count_indices. The length of the list determines the size of the counter, as explained.

As explained in #3 below, there’s a global counter that increments for each frame. Potentially, this increment could be larger than one (although currently it’s always one). The short counter increments the counter with this frame increment, so it could potentially go 0,1,3,…,6,7,0,1,2,… if the third frame incremented the global counter by two instead of one.
Some bits are dedicated to a long term counter (and initial handshaking). E.g. if the counter is a 32-bit int, then the counter will just count up until 2 ^ 32 when it overflows back to zero.

However, the counter could be 32-bits wide, but the number of bits available on the projector is much less. So, we split up the counter across multiple frames. E.g. if counter_bit_width is 32 and count_indices has only, say, 4 bits available. Then we split the 32 bits into 32 / 4 transactions. However, each transaction is sent twice, the value and its one’s complement (aligning with the clock being low), except for the first transaction of each number and for the initial handshake bytes sent, where it’s sent twice, identically, without inverting to one’s complement. So, the total number of frames required is 32 / 4 * 2 = 16 frames.

So, overall for this example, if the first count value starts at zero and increments with each frame, then the first 16 frames sends the number 0. When it’s done, the counter is at 16, so we send the number 16 over the next 16 frames, then the number 32 over the next 16 frames etc.

The counter is broken and sent starting from the least significant (lower) bits to the most significant (upper) bits.

At the start, we also optionally send an arbitrary sequence of bytes to help identify experimental specific metadata as described in get_bits(). The counter then starts sending its current value when the bytes are done. The length of the bytes is sent before the bytes (zero is sent if empty).