Package postprocessors is basically the same as package transformers, but used at a different stage of the RAG pipeline.

DedupeByFileMetadata is a deduplication function that checks if the document is a duplicate based on the file metadata.

DummyDedupe is a dummy deduplication function that always returns false (i.e.

GetDefaultDSNs returns the paths for the datastore and vectorstore databases.

ErrDBDocumentNotFound is returned when a document is not found in the database.

ErrDBFileNotFound is returned when a file is not found.

IsDuplicateFuncs is a map of deduplication functions by name.

IsDuplicateFunc is a function that determines whether a document is a duplicate or if it should be ingested.