Data and IO

This section presents the classes used to read and write data that is acquired by the SSDAQ system. Data is written using two file formats; slow signal data received from the camera is written to hard drive using the hdf5 fileformat, while all other data use the custom indexed container format (icf) using their on wire formats for serialization.

In general, to read data the DataReader handles all data except for slow signal data which can be read using SSDataReader.

ssdaq.data.io

class ssdaq.data.io.DataReader(filename: str)

__init__(filename: str)

Open a Streamed Object File to read.

Parameters:filename (str) – path and filename Raises:ValueError – Is raised if no suitable unpacker is found

close()

closes file handle

filesize

the file size on disk in bytes

Returns:file size Return type:int

n_entries

number of objects in file.

Returns:file object count Return type:int

read()

Reads one object at the position of the file pointer.

Returns:Bytes that represent the object Return type:bytes

read_at(ind: int)

Reads one object at the index indicated by ind

Parameters:ind (int) – the index of the object to be read Returns:that represent the object Return type:bytes Raises:IndexError – if index out of range

reload()

Reload the index table. Useful if the file is being written too when read

resetfp()

Resets file pointer to the first object in file

timestamp

unix timestamp of the file creation time

Returns:creation datetime Return type:datetime.datetime

class ssdaq.data.io.SSDataReader(filename, mapping='ssl2asic_ch')

A reader for Slow Signal data

__init__(filename, mapping='ssl2asic_ch')

Parameters:filename (str) – path to file

Kwargs:

mapping (str): determines how the pixels are mapped. Two mappings

are availabe: ‘ssl2colrow’ and ‘ssl2asic_ch’ which correspond to a 2D col-row layout and ASIC-channel which is the same ordering used for the fast signal data.

close_file()

Closes file handle

load_all_data(tm, calib=None, mapping=None)

Loads all rows of data for a particular target moduel into memory (in the future a selection of modules)

Parameters:tm (int) – The slot number of the target module

Kwargs:

calib (arraylike): an array with calibration coefficient that should be applied to the data mapping (str or arraylike): a string to select a mapping or an array with the mapping

[‘ssl2colrow’,’ssl2asic_ch’,’raw’]

read(start=None, stop=None, step=None)

A data file iterator for reading data rows.

Parameters:

• start (int) – starting row number
• stop (int) – stopping row number
• step (int) – size of the step at each iteration

class ssdaq.data.io.SSDataWriter(filename: str, attrs: dict = None, filters: tables.filters.Filters = None, buffer: int = 1000, tel_table=True)

A writer for Slow Signal data

__init__(filename: str, attrs: dict = None, filters: tables.filters.Filters = None, buffer: int = 1000, tel_table=True)

Initialize self. See help(type(self)) for accurate signature.

close_file()

Closes file handle

ssdaq.core

class ssdaq.core.io.RawObjectWriterBase(filename: str, protocol: int = 1, compressor='bz2', **kwargs)

Base class of a file object for writing indexable chunks of serialized data to file.

__init__(filename: str, protocol: int = 1, compressor='bz2', **kwargs)

Summary

Parameters:

• filename (str) – Description
• protocol (int, optional) – Description
• compressor (str, optional) – Description
• **kwargs – Description

close()

Closes the file handle

data_counter

Counter of the number of objects written to file

Returns:int

write(data: bytes)

Writes a stream of bytes to file

Parameters:data (bytes) – bytes to be writen to file

class ssdaq.core.io.RawObjectReaderBase(filename: str)

This class

Variables:

• file (TYPE) – Description
• filename (TYPE) – Description
• metadata (dict) – Description

__init__(filename: str)

Reads Streamed Object Files

Parameters:filename (str) – filename and path to the file Raises:TypeError – Raised if the file is not recognized as SOF

close()

closes file handle

filesize

the file size on disk in bytes

Returns:file size Return type:int

n_entries

number of objects in file.

Returns:file object count Return type:int

read() → bytes

Reads one object at the position of the file pointer.

Returns:Bytes that represent the object Return type:bytes

read_at(ind: int) → bytes

Reads one object at the index indicated by ind

Parameters:ind (int) – the index of the object to be read Returns:that represent the object Return type:bytes Raises:IndexError – if index out of range

reload()

Reload the index table. Useful if the file is being written too when read

resetfp()

Resets file pointer to the first object in file

timestamp

unix timestamp of the file creation time

Returns:creation datetime Return type:datetime.datetime

Fileformat Protocols

Both the custom protocols used by the RawObjectWriterBase and RawObjectReaderBase are indexed containers meaning that they just provide the framework and file format to index objects that are written to file. Methods of serializing and deserializing the objects must be provided to these classes.

protocol 0

This protocol provides a basic format for indexing objects. However, it has limitations as there is no efficient way of compressing the data saved in this format unless the whole file is compressed. The protocol is defined as:

Data chunks of serialized data are
sequentially writter to file, with
each chunk being prepended with:
chunk length in bytes (4 bytes)
and a crc32 hash      (4 bytes)

The file header is 24 bytes long and has the following layout

    bytes:      field:
    0-7         Custom field for file format specifications (set by the header parameter)
    8-11        Protocol version
    12-15       Not used
    16-19       Not used
    20-23       Not used

General file structure:
    +-------------+
    | File Header |
    +-------------+
    | Chunk Header|
    +-------------+
    |     Data    |
    +-------------+
    | Chunk Header|
    +-------------+
    |     Data    |
    +-------------+
          ...
          ...

protocol 1

Protocol 1 is more complicated than protocol 0 as it supports compression, includes a filecreation timestamp (redundancy of the unix timestamp which can be accidentally changed while copying) and an extendible header which can be used by custom formats to store meta-data. The protocol is defined as:

Code structure