Trait Worker

pub trait Worker: Send + Sync {
    type Error: Debug + Display;
    type Message: Send + Sync;

    // Required method
    fn process_checkpoint<'life0, 'async_trait>(
        &'life0 self,
        checkpoint: Arc<CheckpointData>,
    ) -> Pin<Box<dyn Future<Output = Result<Self::Message, Self::Error>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;

    // Provided method
    fn preprocess_hook(&self, _: Arc<CheckpointData>) -> Result<(), Self::Error> { ... }
}

Processes individual checkpoints and produces messages for optional batch processing.

The Worker trait defines the core processing logic for checkpoint data. Workers run in parallel within a WorkerPool to process checkpoints and generate messages that can optionally be batched and processed by a Reducer.

Processing Modes

Workers support two processing modes:

• Direct Processing: Messages are handled immediately without batching.
• Batch Processing: Messages are accumulated and processed in batches by a Reducer.

The processing mode is determined by the presence of a Reducer in the WorkerPool configuration.

Concurrency

Multiple instances of a worker can run in parallel in the worker pool. The implementation must be thread-safe and handle checkpoint processing efficiently.

Integration with Optional Reducer

Messages produced by the worker can be:

• Processed directly without batching.
• Accumulated and passed to a Reducer for batch processing.

The worker’s Message type must match the reducer’s input type when batch processing is enabled.

Required Associated Types

type Error: Debug + Display

type Message: Send + Sync

Required Methods

fn process_checkpoint<'life0, 'async_trait>( &'life0 self, checkpoint: Arc<CheckpointData>, ) -> Pin<Box<dyn Future<Output = Result<Self::Message, Self::Error>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Processes a single checkpoint and returns a message.

This method contains the core logic for processing checkpoint data.

Note

• Checkpoints are processed in order when a single worker is used.
• Parallel processing with multiple workers does not guarantee checkpoint order.

Provided Methods

fn preprocess_hook(&self, _: Arc<CheckpointData>) -> Result<(), Self::Error>

A hook that allows preprocessing a checkpoint before it’s fully processed.

This method can be used to perform actions like validation or data transformation before the main process_checkpoint logic is executed.

Default implementation

By default it returns Ok(()).

Implementors