BatchedEmbed creates embeddings for the given input texts, batching them into batches of batchSize if needed.

BatchTexts splits strings by the length batchSize.

CreateDocuments creates documents from texts and metadatas with a text splitter.

DefaultTsOptions returns the default options for all text splitter.

NewEmbedder creates a new Embedder from the given EmbedderClient, with some options that affect how embedding will be done.

New returns a new OpenAI client.

NewText creates a new text loader with an io.Reader.

SplitDocuments splits documents using a textsplitter.

ToRetriever takes a vector store and returns a retriever using the vector store to retrieve documents.

WithAllowedSpecial sets the allowed special tokens for a text splitter.

WithBatchSize is an option for specifying the batch size.

WithChunkOverlap sets the chunk overlap for a text splitter.

WithChunkSize sets the chunk size for a text splitter.

WithCodeBlocks sets whether indented and fenced codeblocks should be included in the output.

WithDeduplicater returns an Option for setting the deduplicater that could be used when adding documents.

WithDisallowedSpecial sets the disallowed special tokens for a text splitter.

WithEmbedder returns an Option for setting the embedder that could be used when adding documents or doing similarity search (instead the embedder from the Store context) this is useful when we are using multiple LLMs with single vectorstore.

WithEncodingName sets the encoding name for a text splitter.

WithFilters searches can be limited based on metadata filters.

WithLenFunc sets the lenfunc for a text splitter.

WithModelName sets the model name for a text splitter.

WithNameSpace returns an Option for setting the name space.

WithReferenceLinks sets whether reference links (i.e.

WithSecondSplitter sets the second splitter for a text splitter.

WithSeparators sets the separators for a text splitter.

WithStripNewLines is an option for specifying the should it strip new lines.

ErrEmptyResponse is returned when the OpenAI API returns an empty response.

ErrMismatchMetadatasAndText is returned when the number of texts and metadatas given to CreateDocuments does not match.

Completion is a completion.

EmbeddingRequest is a request to create an embedding.

OpenAIClient is a client for the OpenAI API.

Options is a set of options for similarity search and add documents.

Retriever is a retriever for vector stores.

Text loads text data from an io.Reader.

TokenSplitter is a text splitter that will split texts by tokens.

TsOptions is a struct that contains options for a text splitter.

Doer performs a HTTP request.

Embedder is the interface for creating vector embeddings from texts.

EmbedderClient is the interface LLM clients implement for embeddings.

Loader is the interface for loading and splitting documents from a source.

TextSplitter is the standard interface for splitting texts.

VectorStore is the interface for saving and querying documents in the form of vector embeddings.

EmbedderClientFunc is an adapter to allow the use of ordinary functions as Embedder Clients.

Option is a function that configures an Options.

TsOption is a function that can be used to set options for a text splitter.