Struct HistoricalReader

pub struct HistoricalReader { /* private fields */ }

Implementations

impl HistoricalReader

pub fn new(config: HistoricalReaderConfig) -> Result<Self>

pub fn verify_and_get_manifest_files( &self, manifest: Manifest, ) -> Result<Vec<FileMetadata>>

This function verifies the manifest and returns the file metadata sorted by the starting sequence number.

More specifically it verifies that the files in the remote store cover the entire range of checkpoints from sequence number 0 until the latest available checkpoint with no missing checkpoint.

pub async fn verify_file_consistency( &self, files: Vec<FileMetadata>, ) -> Result<()>

This function downloads checkpoint data files and ensures their computed checksum matches the one in manifest.

pub async fn stream_blobs_for_range( &self, checkpoint_range: Range<CheckpointSequenceNumber>, ) -> Result<impl Stream<Item = Result<Bytes>> + Send + use<'_>>

Stream blobs of [Bytes] that include checkpoint data for the specified range.

This method retrieves files with batches of serialized checkpoint data from the remote store, and streams the respective contents as blobs.

Errors

Returns an error if resolving the files that need to be fetched from the remote store fails.

Additionally the stream may fail if fetching the file from the remote store fails.

Examples

use futures::StreamExt;

let range = 100..200;
let mut stream = historical_reader.stream_blobs_for_range(range.clone()).await?;
while let Some(Ok(blob)) = stream.next().await {
    // we can now iterate over the checkpoint data
    for data in make_blob_iterator_for_range(blob, range.clone())? {
        println!("Received checkpoint data: {data:?}");
    }
}

pub async fn iter_for_range( &self, checkpoint_range: Range<CheckpointSequenceNumber>, ) -> Result<impl Iterator<Item = CheckpointData>>

Construct an Iterator over CheckpointData for the specified range.

This method eagerly consumes the stream of blobs returned from Self::stream_blobs_for_range and holds the data in memory until the iterator is consumed.

For lazy processing of the blobs use directly Self::stream_blobs_for_range along with make_blob_iterator_for_range.

pub async fn iter_for_file( &self, file_path: Path, ) -> Result<impl Iterator<Item = CheckpointData>>

Iterate CheckpointData from the given remote file.

This method retrieves the file with batches of serialized checkpoint data from the remote store, decodes the raw data, and streams the deserialized values.

Errors

Returns an error in the following cases:

• If fetching the file from the remote store fails.
• If the file is corrupted and fails to decode.

pub async fn latest_available_checkpoint( &self, ) -> Result<CheckpointSequenceNumber>

Return latest available checkpoint in archive.

pub fn remote_store_identifier(&self) -> String

pub async fn sync_manifest_once(&self) -> Result<()>

Syncs the Manifest from remote store.

pub async fn get_manifest(&self) -> Manifest

Trait Implementations

impl Clone for HistoricalReader

fn clone(&self) -> HistoricalReader

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.