Module util 

Structs

EpochMetadata

RootManifest

Constants

EPOCH_METADATA_FILENAME

MANIFEST_FILENAME

SUCCESS_MARKER

Marker file written to an epoch directory in the store once all files for that epoch have been written.

Functions

copy_file

delete_files

delete_recursively

find_all_dirs_with_epoch_prefix

This function will find all child directories in the input store which are of the form “epoch_num” and return a map of epoch number to the directory path

find_all_files_with_epoch_prefix

This function will find all child directories in the input store which are of the form “epoch_num” and return a map of epoch number to the directory path

find_missing_epochs_dirs

This function will find missing epoch directories in the input store and return a list of such epoch numbers. If the highest epoch directory in the store is epoch_N then it is expected that the store will have all epoch directories from epoch_0 to epoch_N. Additionally, any epoch directory should have the passed in marker file present or else that epoch number is already considered as missing. The returned list will contain epoch_N+1.

get

get_path

list_all_epochs

Finds all epochs in the store and returns them as a sorted list, paired with each epoch’s end timestamp in ms when its metadata file is present, or 0 otherwise.

path_to_filesystem

put

Writes bytes in the store with specified path.

run_manifest_update_loop

Writes the epochs existed in the store to the root MANIFEST (contains only a list of epochs in the store) every 300 seconds.