Crate iota_data_ingestion_core

This library provides an easy way to create custom indexers.

Graceful shutdown

The shutdown sequence in the data ingestion system ensures clean termination of all components while preserving data integrity. It is initiated via a CancellationToken, which triggers a hierarchical and graceful shutdown process.

The shutdown process follows a top-down hierarchy:

1. Worker: Individual workers within a WorkerPool detect the cancellation signal, completes current checkpoint processing, sends final progress updates and signals completion to parent WorkerPool via WorkerStatus::Shutdown message.
2. WorkerPool: Coordinates worker shutdowns, ensures all progress messages are processed, waits for all workers’ shutdown signals and notifies IndexerExecutor with WorkerPoolStatus::Shutdown message when fully terminated.
3. IndexerExecutor: Orchestrates the shutdown of all worker pools and and finalizes system termination.

Modules

history

Handle historical checkpoint data.

reader

Structs

DataIngestionMetrics

FileProgressStore

Manages persistent progress information stored in a JSON file.

IndexerExecutor

The Executor of the main ingestion pipeline process.

ReaderOptions

Options for configuring how the checkpoint reader fetches new checkpoints.

ShimProgressStore

A simple, in-memory progress store primarily used for unit testing.

WorkerPool

A pool of Worker’s that process checkpoints concurrently.

Enums

IngestionError

Constants

MAX_CHECKPOINTS_IN_PROGRESS

Traits

ProgressStore

A trait defining the interface for persistent storage of checkpoint progress.

Reducer

Processes and commits batches of messages produced by workers.

Worker

Processes individual checkpoints and produces messages for optional batch processing.

Functions

create_remote_store_client

Creates a remote store client without any retry mechanism.

create_remote_store_client_with_ops

Creates a remote store client with configurable retry behavior and options.

setup_single_workflow

Sets up a single workflow for data ingestion.

Type Aliases

IngestionResult