Natural language processing is teaching computers to understand and generate human language. The field has gone through dramatic shifts, with the latest (LLMs) reshaping what's possible.
This page surveys the field.
Hand-written rules for syntax and semantics. Brittle but interpretable. Worked for limited domains.
Hidden Markov Models, Conditional Random Fields, Support Vector Machines. Required hand-engineered features.
Machine translation, named entity recognition, sentiment analysis worked acceptably.
word2vec, GloVe — dense vector representations. Captured semantic similarity.
Enabled deep learning to flourish in NLP.
RNNs, LSTMs, GRUs handled sequences. Became standard for many NLP tasks.
"Attention is All You Need" introduced transformers. Replaced RNNs by 2019.
Architecture choice for nearly all modern NLP.
BERT, GPT-2 — pretrain on huge text corpora, fine-tune for tasks.
Set new state-of-the-art across NLP tasks.
GPT-3, GPT-4, Claude, Gemini. Few-shot and zero-shot capabilities.
Reshaped how NLP problems are approached.
Assign categories to text:
Approaches:
Find entity mentions: people, places, organizations, dates, etc.
Pretrained models (spaCy, BERT-NER) are strong out of the box.
Extract structured information from text.
Often combines NER with relation extraction.
LLMs handle this well with prompting.
Per-token labels: part-of-speech, chunking, dependency parsing.
Mostly solved problems.
Determine which expressions refer to the same entity. ("Alice... she...")
Hard; getting better with modern models.
Translate between languages.
Transformer-based (Google Translate, DeepL) dominates. LLMs are competitive.
Condense long text into shorter form.
Extractive: pick important sentences. Abstractive: generate new text.
LLMs handle this naturally.
Extractive: find span in document that answers question. Generative: produce free-form answer.
Modern: retrieval + LLM (RAG).
Multi-turn interaction. Once specialized; now general LLM capability.
Produce text from prompt or condition.
LLMs dominate.
Find relevant documents for a query.
Approaches:
Speech recognition and synthesis are NLP-adjacent. Now end-to-end deep learning.
Splitting text into units. Modern: subword (BPE, WordPiece, SentencePiece).
Different tokenizers give different segmentations. Match the tokenizer to the model.
Dense vectors for tokens, words, sentences, or documents.
Semantic similarity = vector similarity (cosine).
Weighted combination of representations. Allows the model to focus on relevant parts.
The breakthrough enabling transformers.
Train on huge unlabeled text. Predict masked tokens (BERT) or next tokens (GPT).
Captures language patterns; transferable to many tasks.
Adapt pretrained model to a specific task with labeled data.
Less data needed than training from scratch.
Quality has caught up to closed models for many use cases.
Best raw quality; pay per use; no data privacy guarantees by default.
For most retrieval, modern open embeddings are competitive.
Retrieve relevant docs; pass to LLM. Standard for question answering on private data.
Show examples in the prompt; LLM generalizes. Powerful for prototyping.
For high-volume tasks: fine-tune small specialized model rather than calling large API per request. Cheaper, faster.
Use LLM to label training data; train smaller model.
For information extraction: ask LLM for JSON; parse.
Tools like Outlines, Instructor help reliability.
LLMs that can call tools. Useful for complex multi-step workflows.
Subtle bugs from different tokenizers in training vs inference.
LLMs generate plausible falsehoods. Always verify factual claims.
User input that overrides instructions. Major security concern for production.
Models trained on general English may fail on specialized text (medical, legal, code).
NLP success often depends on data quality more than model choice.
LLMs are expensive. For high-volume simple tasks, smaller models work.
Metrics depend on task:
Don't trust automated generation metrics. Always sample outputs and read.