Tokenizers: text to numbers

Models do not see text. They see sequences of integers produced by a tokenizer. Understanding tokenization explains many quirks of model behaviour, cost and limits.

Subword tokenization

Modern tokenizers split text into subwords. Common words become single tokens, while rare words are split into pieces:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
tok.tokenize("The refund was unprocessable")
# ['the', 'ref', '##und', 'was', 'un', '##process', '##able']

The exact pieces depend on the tokenizer. Methods such as WordPiece, BPE and Unigram are trained on a corpus and learn a vocabulary of frequent pieces. This lets a model handle any word, including typos and new product names, with a fixed vocabulary size.

From tokens to ids

enc = tok("Where is my order?", truncation=True, max_length=128)
enc["input_ids"]       # integers, including special tokens
enc["attention_mask"]  # 1 for real tokens, 0 for padding

Tokenizers add special tokens, such as a classification token at the start and a separator at the end, which the model was pretrained to expect. Always use the tokenizer that matches the model. Mixing them silently produces garbage.

Batches, padding and truncation

Sequences in a batch must have the same length. The tokenizer pads short ones and truncates long ones:

batch = tok(["Hi", "My parcel arrived damaged and I want a refund"],
            padding=True, truncation=True, max_length=64, return_tensors="pt")

The attention mask tells the model to ignore padding. Dynamic padding, padding each batch only to its longest sequence, saves a lot of computation compared with padding everything to the maximum length.

Context length

Each model has a maximum sequence length. Text beyond it is cut off when truncating, so if the key information in long support emails comes at the end, a model limited to 512 tokens may never see it. Check the distribution of token lengths in your data before choosing a model:

lengths = [len(tok(t)["input_ids"]) for t in texts]

If many examples exceed the limit, consider a model with a longer context, splitting documents into chunks, or keeping the most relevant part.

Languages and tokens

Tokenizers trained mostly on English split other languages into many more tokens per word. That affects speed, cost and sometimes quality. For multilingual data, choose a multilingual model and compare token counts across your languages.

Tokenize your project data, plot the length distribution and pick a sensible maximum length for the support ticket task.