The pretrained model revolution

For a long time, building a text classifier meant collecting tens of thousands of labelled examples and training a model from scratch. Since the transformer architecture appeared in 2017 and models like BERT followed, the standard approach has been transfer learning: start from a model pretrained on huge amounts of text, then adapt it to your task.

Two kinds of pretraining

• Encoder models, such as BERT and its successors, are trained to understand text by predicting masked words. They read the whole input at once and produce a vector for each token. They are excellent for classification, entity extraction and embeddings, and they are relatively small and fast.
• Decoder models, the family behind modern chat assistants, are trained to predict the next token. They generate text and can follow instructions, but they are larger and slower to run.

There are also encoder decoder models, such as T5, used for tasks like translation and summarisation.

A common mistake is reaching for a large generative model for every text problem. For routing tickets into ten categories, a fine tuned encoder with a hundred million parameters is often more accurate, much cheaper and far faster than prompting a large model. We will compare both approaches on the same task later in the course.

The Hugging Face ecosystem

• transformers: model architectures, tokenizers and training utilities.
• datasets: loading, processing and caching datasets efficiently.
• evaluate: standard metrics.
• peft: parameter efficient fine tuning methods such as LoRA.
• sentence-transformers: models and tools for embeddings.
• The Hub: a large public collection of models and datasets, each with a model card describing training data, intended use, limitations and licence.

Reading a model card

Before using a model, read its card. Check:

1. Task and training data: was it trained on text similar to yours, in your language?
2. Licence: some models restrict commercial use or require attribution.
3. Size: parameters and memory needs, which determine where you can run it.
4. Evaluation and limitations: known weaknesses and biases.

A model that tops a leaderboard but was trained on a different domain or forbids your use case is the wrong model.

Our approach

Every project starts with a simple baseline, such as TF-IDF with logistic regression. If a transformer cannot beat it clearly, the extra complexity is not justified. This habit keeps the work honest and often surprises people.