NewIngestionPipeline creates an IngestionPipeline that processes all pulsar messages.

WithRetry executes the supplied action until it either completes successfully or it returns false, indicating that the error is fatal.

Batcher batches up events from a channel.

IngestionPipeline is a pipeline that reads message from pulsar and inserts them into a sink.

HasPulsarMessageIds should be implemented by structs that can store a batch of pulsar message ids This is needed so we can pass message Ids down the pipeline and ack them at the end.

InstructionConverter should be implemented by structs that can convert a batch of eventsWithIds into an object suitable for passing to the sink.

Sink should be implemented by the struct responsible for putting the data in its final resting place, e.g.

BatchMerger merges together events within the batch, where possible.

BatchMetricPublisher logs a summary of the batching process.

EventCounter determines the true count of events, as some utils.ArmadaEvent can contain nested events.

MessageUnmarshaller converts consumed pulsar messages to the intermediate type, utils.EventsWithIds.