BertBaseUncase loads pretrained BERT tokenizer.

BertLargeCasedWholeWordMaskingSquad loads pretrained BERT large case whole-word masking tokenizer finetuned on SQuAD dataset.

CreateNormalizer creates Normalizer from config data.

FromFile constructs a new Tokenizer from json data file (normally 'tokenizer.json').

GPT2 loads GPT2 (small) tokenizer from vocab and merges files.

RobertaBase loads pretrained RoBERTa tokenizer.

RobertaBaseSquad2 loads pretrained RoBERTa fine-tuned SQuAD Question Answering tokenizer.