Most data science work touches text eventually — customer feedback, support tickets, descriptions, logs. NLP turns text into features or predictions.
This page covers the practical pipeline from raw text to models.
Each step has choices that affect downstream quality.
Text data is messy:
Decisions:
Context-dependent. Sentiment analysis on tweets cares about emojis; legal document analysis doesn't.
Splitting text into units (tokens).
Split on whitespace, handle punctuation. Simple but breaks for languages without word boundaries.
BPE, WordPiece, SentencePiece — break words into subword units.
Used by all modern transformers. Handles rare words and morphology.
Each character is a token. Maximum vocabulary efficiency, longer sequences.
Use the tokenizer matching your model. Don't custom-tokenize for pretrained models.
Classical preprocessing:
For modern transformer models: skip these. The model handles them.
For classical models (TF-IDF + logistic regression): may help.
Each document is a vector of word counts. Simple, interpretable, sparse.
Word counts weighted by inverse document frequency. Common words get lower weight.
Strong baseline for many text tasks.
Dense vectors per word: word2vec, GloVe, fastText.
Words with similar meanings have similar vectors.
Pre-deep-learning innovation; still useful.
BERT, RoBERTa, etc. Embeddings depend on context.
"Bank" in "river bank" vs "bank account" gets different vectors.
Sentence-transformers, OpenAI ada-002 — turn whole text into single vector.
Useful for similarity, clustering, retrieval.
For most tasks: use a pretrained transformer's embedding layer. Sentence-transformers for sentence-level work.
Sentiment, topic, intent, spam.
Modern: fine-tune a transformer on labeled data. Or use embeddings + classifier.
Find names, places, organizations, dates.
Pretrained models (spaCy, BERT-based) work well out of the box.
Part-of-speech tagging, chunking, parsing.
Mostly solved problems with pretrained models.
Extract structured info from text. Often combines NER + relation extraction.
Discover themes in document collections.
LDA (classical), BERTopic (modern with embeddings).
Abstract or extract summaries.
LLMs handle this well.
Extract answers from documents (extractive) or generate answers (generative).
Find relevant documents for a query.
BM25 (classical), dense retrieval (modern), hybrid (best in practice).
TF-IDF + logistic regression is a baseline that works surprisingly well. Establish before going complex.
Fine-tuning a pretrained model beats training from scratch with limited data.
For many tasks: pre-compute embeddings, train a classifier on top.
Cheaper than fine-tuning. Often comparable quality.
For prototyping: zero-shot or few-shot LLM may answer the question.
For production: usually fine-tune a smaller model for cost.
Be wary of "English-trained model on other language" — quality drops.
Using one tokenizer in training and another in inference. Subtle, often silent.
Spam detection has 99% non-spam. Accuracy 99% by predicting "not spam" always.
Aggressive regularization, data augmentation, or smaller models.
Training on news; deploying on tweets. Doesn't transfer well.
Especially for retrieval evaluation: ensure test queries aren't in training data.
NLP works well for English on common domains. Specialized text (medical, legal, code) needs domain adaptation.
Beyond accuracy: precision, recall, F1.
For text generation: BLEU, ROUGE, METEOR (imperfect), and human evaluation (gold standard).
For retrieval: NDCG, MRR, recall@k.
Don't trust automated metrics blindly. Sample outputs and read them.
LLMs:
Traditional ML:
Hybrid: use LLMs to label data; train smaller model.