CreateDocuments creates documents from texts and metadatas with a text splitter.

DefaultOptions returns the default options for all text splitter.

NewMarkdownTextSplitter creates a new Markdown text splitter.

NewRecursiveCharacter creates a new recursive character splitter with default values.

SplitDocuments splits documents using a textsplitter.

WithAllowedSpecial sets the allowed special tokens for a text splitter.

WithChunkOverlap sets the chunk overlap for a text splitter.

WithChunkSize sets the chunk size for a text splitter.

WithCodeBlocks sets whether indented and fenced codeblocks should be included in the output.

WithDisallowedSpecial sets the disallowed special tokens for a text splitter.

WithEncodingName sets the encoding name for a text splitter.

WithHeadingHierarchy sets whether the hierarchy of headings in a document should be persisted in the resulting chunks.

WithJoinTableRows sets whether tables should be split by row or not.

WithKeepSeparator sets whether the separators should be kept in the resulting split text or not.

WithLenFunc sets the lenfunc for a text splitter.

WithModelName sets the model name for a text splitter.

WithReferenceLinks sets whether reference links (i.e.

WithSecondSplitter sets the second splitter for a text splitter.

WithSeparators sets the separators for a text splitter.

ErrMismatchMetadatasAndText is returned when the number of texts and metadatas given to CreateDocuments does not match.

MarkdownTextSplitter markdown header text splitter.

Options is a struct that contains options for a text splitter.

RecursiveCharacter is a text splitter that will split texts recursively by different characters.

TokenSplitter is a text splitter that will split texts by tokens.

TextSplitter is the standard interface for splitting texts.

Option is a function that can be used to set options for a text splitter.