The open source LLM ecosystem has matured rapidly. Open models are competitive with closed APIs for many use cases. Tooling is solid. The question for most teams is when open makes sense, not whether it does.
This page maps the landscape.
At high volume, self-hosted open models cost less than API calls.
Data stays on your infrastructure. No API provider sees it.
You choose the model, configuration, deployment. No surprise deprecations or behavior changes.
Fine-tuning, embeddings, full weight access. Things APIs can't offer.
Some deployments must run without internet (regulated industries, offline products).
Working with open models builds organizational ML capabilities.
The dominant open family.
License allows commercial use with conditions.
Apache 2.0 for older models; custom for newer.
Apache 2.0 for many variants.
Custom Google license.
MIT license.
Strong technical capability.
For specialized tasks, specialized models often beat large generalists.
C++ library. CPU-friendly. Aggressive quantization.
The standard for running quantized LLMs on consumer hardware.
Python/CUDA. Continuous batching, PagedAttention.
Standard for high-throughput GPU serving.
Hugging Face's serving framework. Similar capabilities to vLLM.
Simplified deployment for local LLM use. Built on llama.cpp.
GUI for local inference. Good for non-technical users.
Memory-efficient GPU inference, especially for quantized models.
Compiles models for many backends (mobile, browser, edge).
Format and quantization for CPU/edge inference.
Quants: Q2 (smallest, lowest quality) → Q8 (largest, highest quality).
Q4_K_M is a common balance.
GPU-friendly quantization. Common in HF model hub.
Activation-aware. Often slightly better quality at same precision than GPTQ.
For exllamav2. Variable per-layer precision.
Easiest path; native PyTorch support. 4-bit, 8-bit.
Update all weights. Memory-intensive; not always best.
Train small adapter weights. 100x fewer trainable parameters; quality nearly matches full fine-tuning for many tasks.
QLoRA: LoRA on quantized base. Runs on consumer GPUs.
Open source:
Commercial: Pinecone, Vespa.
Python, JavaScript. Many integrations. Some criticism for over-abstraction.
Focused on RAG and indexing.
Mature, modular pipelines.
Programming model for LLM applications. Compile prompts.
Often the right choice. Frameworks add abstraction overhead.
The picture in late 2025/early 2026:
For many production tasks, a 70B open model is sufficient.
Hardware + electricity + ops time.
A100 instance: ~$1-2/hour.
Throughput at 70B: ~1000-3000 tokens/sec with batching.
Breakeven: depends on volume. Often 10M+ tokens/day favors self-hosting if you have the ops capability.
Self-hosting LLMs is harder than running APIs.
Benchmark performance doesn't always match your task.
Q2 ruins quality. Q8 wastes memory. Q4-Q5 is usually right.
Reaching for LangChain when you need 50 lines of Python.
Open models need evaluation on your specific task.
Ecosystem moves fast. 6-month-old advice may be stale.
Use open source when:
Use closed APIs when:
Many teams run hybrid: closed for hard tasks, open for high-volume ones.