Recorders

We support the following recorders beside the parent class Recorder:

SSDRecorder (Pulse data)
SSDParser (Pulse data read in chunks)
PMTRecorder (Pulse data)
FileRecorder (Finds files)
FileParser (Finds files and reads in chunks)
ImageFileRecorder (Finds image files and their metadata)
CoilRecorder (Records the coil)
IonRecorder (Records the Ionizer)
LaserRecorder (Records the laser frequencies)
GaugeRecorder (…)
HeaterRecorder (Records the heating of Yttrium foil
CycleRecorder (Records the analyses results)
ParameterRecorder (Records the parameters used in the analysis)
SSDResultsRecorder (Records the result of the SSDAnalysis)
StaticRecorder (Turns a pandas dataframe into a Recorder)

class data_eng_utokyo.recorders.Recorder(filepath: str, has_metadata: bool = True, delimiter: str = ',', always_update: bool = False, encoding='utf-8')[source]

Bases: object

Base class for mapping a csv file to a pandas dataframe in real-time.

Parameters:

filepath (str) – Full path to the csv file.
has_metadata (bool) – Whether or not the csv file has metdata.
delimiter (str) – Delimiter used in the csv file.
always_update (bool) – Should the loading of new data be forced.
encoding (str) – Encoding used in the csv file.

filepath[source]

Full path to the csv file.

Type:: str

has_metadata[source]

Whether or not the csv file has metdata.

Type:: bool

delimiter[source]

Delimiter used in the csv file.

Type:: str

always_update[source]

Should the loading of new data be forced.

Type:: bool

encoding[source]

Encoding used in the csv file.

Type:: str

read_data_lines[source]

How many lines corresponding to data have been read.

Type:: int

last_updated[source]

Time in human readable format as string.

Type:: str

get_table() → DataFrame[source]

Get the full table consisting of data and metdata.

Returns:: Pandas dataframe.
Return type:: DataFrame

get_data() → DataFrame[source]

Get just the data.

Returns:: Pandas dataframe.
Return type:: DataFrame

get_metadata() → DataFrame[source]

Get just the metadata

Returns:: Pandas dataframe.
Return type:: DataFrame

is_up_to_date() → bool[source]

Returns true if the csv has not been modified since the last loading.

Returns:: bool
Return type:: bool

_get_mod_time()[source]: Get the modification time of the csv file.

_update()[source]: Update both the data and the metadata with the csv file.

_update_data()[source]: Updates self._data_df and self.data_last_updated incrementally.

Note

Makes use of _load_new_data().

_timestamp_to_datetimes(df: DataFrame)[source]

Takes a dataframe with a timestamp column (int) and adds datetime.

Parameters:: df (pd.DataFrame) – Datarame which should be manipulated.

abstract _load_initial_data() → DataFrame[source]

Returns all data up to now and defines the data columns.

Returns:: Pandas dataframe.
Return type:: DataFrame

abstract _load_new_data() → DataFrame[source]

Gets the rows which have not been loaded so far.

Returns:: Pandas dataframe.
Return type:: DataFrame

abstract _load_metadata() → DataFrame[source]

Reloads all metadata.

Returns:: Pandas dataframe.
Return type:: DataFrame

abstract _harmonize_time()[source]: Converts the time format of the csv file to a standard time format.

class data_eng_utokyo.recorders.SSDRecorder(filepath: str, always_update: bool = False, lines_per_update: int = 100000.0)[source]

Bases: Recorder

Records all the SSD2 data at once.

Note

In most cases, there is too much data incoming at once. In this case, we recommend to use the SSDParser, which
reads the data in chunks.

Parameters:

filepath (str)
always_update (bool)
lines_per_update (int)

is_up_to_date() → bool[source]

Returns true if the csv has not been modified since the last loading.

Returns:: bool
Return type:: bool

_load_new_data() → DataFrame[source]

Just load the new part.

Return type:: DataFrame

_load_metadata()[source]: Overwrite the metadata with the new version.

_harmonize_time()[source]: Convert the relative time and start time to the real time.

class data_eng_utokyo.recorders.SSDParser(filepath: str, always_update: bool = False, lines_per_update: int = 100000.0)[source]

Bases: SSDRecorder

Records the SSD data in chunks.

Acts as a parser in the sense that it forgets about the old data upon reloading. This keeps the table size small.

Parameters:

filepath (str)
always_update (bool)
lines_per_update (int)

class data_eng_utokyo.recorders.PMTRecorder(filepath: str, has_metadata: bool = False, always_update: bool = False)[source]

Bases: Recorder

Parameters:

filepath (str)
has_metadata (bool)
always_update (bool)

class data_eng_utokyo.recorders.FileRecorder(filepath: str, always_update: bool = False, match: str = '')[source]

Bases: Recorder

Tracks the filenames which match a format in a specific folder.

Parameters:

filepath (str) – Path to the folder in which the files are stored.
always_update (bool) – Should the recorder always check for new data.
match (str) – Regex with which the filenames are compared. Only matching strings are tracked.

filepath[source]

Path to the folder in which the files are stored.

Type:: str

always_update[source]

Should the recorder always check for new data.

Type:: bool

match[source]

Regex with which the filenames are compared. Only matching strings are tracked.

Type:: str

filepath_set[source]

Set of filepaths that were found.

Type:: set

_load_initial_data() → DataFrame[source]

Gets all data (filepath and metadata of images).

Returns:: Data as a pandas dataframe.
Return type:: DataFrame

_load_new_data() → DataFrame[source]

Gets all data (filepath and metadata of images) which are new.

Returns:: New data as a pandas dataframe.
Return type:: DataFrame

_load_metadata() → DataFrame[source]

Reloads all metadata.

Returns:: Empty dataframe. There is no metadata.
Return type:: DataFrame

_harmonize_time()[source]: Reads the time and adds it to the table.

class data_eng_utokyo.recorders.FileParser(filepath: str, always_update: bool = False, match: str = '')[source]

Bases: FileRecorder

Works like the FileRecorder, but just returns new data each time.

Note

The parser forgets the old data and just returns the new one. It is convenient when we want to laod many files in chunks. We can just call the FileParser multiple times, and each time it tells us which files we should load.

Parameters:

filepath (str) – Path to the folder in which the files are stored.
always_update (bool) – Should the recorder always check for new data.
match (str) – Regex with which the filenames are compared. Only matching strings are tracked.

filepath[source]

Path to the folder in which the files are stored.

Type:: str

always_update[source]

Should the recorder always check for new data.

Type:: bool

match[source]

Regex with which the filenames are compared. Only matching strings are tracked.

Type:: str

filepath_set[source]

Set of filepaths that were found.

Type:: set

_update_data()[source]: Loads new data.

is_up_to_date() → bool[source]

Returns whether all data has already been returned.

Return type:: bool

class data_eng_utokyo.recorders.ImageFileRecorder(filepath: str, always_update: bool = False, match: str = '')[source]

Bases: Recorder

Variation of the FileRecorder which works online of Google Colab. It just works for camera images (.csv), because they contain a metadata file (all_data.csv) in the next-higher folder. This metadata file allows us to read of the timestamp.

Parameters:

filepath (str)
always_update (bool)
match (str)

_load_initial_data() → DataFrame[source]

Returns all data (filepath and metadata of images) which are new.

Return type:: DataFrame

_load_new_data() → DataFrame[source]

Returns all data (filepath and metadata of images) which are new.

Return type:: DataFrame

_load_metadata() → DataFrame[source]

Reloads all metadata.

Return type:: DataFrame

_filepath_to_nr(fp: str)[source]

Takes something of the form “…cmos_000043.csv” and returns 43.

Parameters:: fp (str)

class data_eng_utokyo.recorders.CoilRecorder(filepath: str, always_update: bool = False)[source]

Bases: Recorder

Tracks the whether the current of the MOT coil is on or off.

Parameters:

filepath (str) – Path to the text file.
always_update (bool)

filepath[source]

Path to the text file.

Type:: str

always_update[source]

Should the recorder always check for new data.

Type:: bool

_harmonize_time()[source]: Reads the time from the text file and adds it to the table.

class data_eng_utokyo.recorders.IonRecorder(filepath: str, always_update: bool = False)[source]

Bases: Recorder

Parameters:

filepath (str)
always_update (bool)

class data_eng_utokyo.recorders.LaserRecorder(filepath: str, always_update: bool = False)[source]

Bases: Recorder

Parameters:

filepath (str)
always_update (bool)

_aggregate_laser_rows(original_df: DataFrame)[source]

Originally, one measurement of the six laser wavelengths is distributed over six rows. We aggregate these rows into one row. The only tradeoff is that we have to approximate the time with the time of the last measurement.

Parameters:: original_df (DataFrame)

_combine(entries: list)[source]

If entries has length 1, then it returns the entry. Otherwise, it converts the list to a comma separated string.

Parameters:: entries (list)

class data_eng_utokyo.recorders.GaugeRecorder(filepath: str, always_update: bool = False)[source]

Bases: Recorder

Tracks the Rb dispenser and the neutralizer current.

Parameters:

filepath (str)
always_update (bool)

class data_eng_utokyo.recorders.HeaterRecorder(filepath: str, always_update: bool = False)[source]

Bases: Recorder

Class for data engineering of the heater data.

Parameters:

filepath (str)
always_update (bool)

_load_new_data() → DataFrame[source]

Returns the rows which have not been loaded so far.

Return type:: DataFrame

class data_eng_utokyo.recorders.CycleRecorder(filepath: str, has_metadata: bool = True, delimiter: str = ',', always_update: bool = False, encoding='utf-8')[source]

Bases: Recorder

Tracks all results and combines them into one dataframe.

Parameters:

filepath (str)
has_metadata (bool)
delimiter (str)
always_update (bool)

class data_eng_utokyo.recorders.ParameterRecorder(filepath: str, always_update: bool = False)[source]

Bases: Recorder

Tracks the parameter settings of each cycle.

Parameters:

filepath (str)
always_update (bool)

class data_eng_utokyo.recorders.SSDResultsRecorder(filepath: str, always_update: bool = False)[source]

Bases: Recorder

Tracks the result file of the SSD Analysis.

Parameters:

filepath (str)
always_update (bool)

class data_eng_utokyo.recorders.StaticRecorder(df: DataFrame)[source]

Bases: object

Represents a static pandas dataframe as a recorder.

Parameters:: df (DataFrame)

get_table() → DataFrame[source]

Return type:: DataFrame

get_data() → DataFrame[source]

Return type:: DataFrame

get_metadata() → DataFrame[source]

Return type:: DataFrame

is_up_to_date() → bool[source]

Return type:: bool