DefaultBpeDecoder create a new BpeDecoder with default suffix (`</w>`).

DefaultBpeDecoder create a new BpeDecoder with default suffix (`</w>`).

NewBpeDecoder creates a new BpeDecoder.

NewBpeDecoder creates a new BpeDecoder.

Allows decoding Original BPE by joining all the tokens and then replacing the suffix used to identify end-of-words by whitespaces.

Fuse constructs Fuse decoder It's simply fuses all tokens into one big string.

WordPieceDecoder takes care of decoding a list of wordpiece tokens back into a readable string.