Foundations
What local AI is, why run models yourself, and the honest trade-offs versus the cloud.
The Landscape
Runtimes & apps, the open model families worth knowing, plus formats and quantization.
Your M2 Max
Why Apple Silicon excels, check your specs, and predict exactly which models will fit.
Getting Started with Ollama
Install, download and chat with your first model, use the API, and customize with a Modelfile.
LM Studio & Apple MLX
A no-terminal GUI, plus Apple's own framework for top speed and local fine-tuning.
Building with Local LLMs
Real code: chat, streaming, embeddings, and a private RAG pipeline over your own data.
Workflow & Final Project
Daily habits — a local coding assistant, a privacy policy — then build a private offline assistant.
How to use this track: Work top to bottom — each stage builds on the last. Stages 1–3 are understanding; 4–5 get models running; 6–7 turn them into real tools. Check off tasks as you go; your progress is saved in this browser.
Foundations
Stage 1 · 5 tasks · What local AI is and why it matters
Local AI means running models — LLMs, image generators, speech-to-text, embeddings — entirely on your own hardware, with no data ever leaving your machine. Instead of sending prompts to a provider's servers, the model weights live on your disk and your Mac's GPU does the work.
What a model actually is
Weights = billions of numbers; an inference engine loads them and multiplies your tokens through them
Privacy & control
Your code, documents, and prompts never leave the machine — viable for confidential work
Cost & offline
Free after the hardware; no per-token bills; works on a plane or in an air-gapped room
The honest limits
Laptop-sized models trail frontier cloud models on the hardest reasoning; RAM caps model size
The hybrid mindset
Local for private/high-volume/offline work; cloud for maximum capability — use both
Why learn it now
Running models yourself teaches quantization, context, and memory — making you better with every AI tool
You don't have to choose: most practitioners run a fast local model for everyday and private tasks, and reach for a frontier cloud model on the hardest problems. Learning local AI expands your toolkit — it doesn't replace it.
The Local AI Landscape
Stage 2 · 6 tasks · Tools, models, and formats
Three layers to keep straight: the model (the trained weights, e.g. Llama, Qwen), the runtime that loads them (llama.cpp, MLX), and the app you interact with (Ollama, LM Studio). Most people start at the app and learn downward.
Ollama
The most popular start — one command to download & run, plus a built-in API server
LM Studio
Polished desktop GUI — browse, download, chat, and run an OpenAI-compatible server
llama.cpp
The fast C/C++ engine behind much of the ecosystem; runs GGUF with Metal acceleration
MLX / mlx-lm
Apple's own framework — often fastest on M-series and the path to local fine-tuning
Model families
Llama, Qwen (+ Coder), Mistral/Mixtral, Gemma, Phi, DeepSeek (reasoning)
Formats & quantization
GGUF / safetensors / MLX; Q4_K_M compresses weights to ~4-bit with small quality loss
Where to get models: start with ollama.com/library (curated, one-command installs). Graduate to Hugging Face for everything — filter by "GGUF" for llama.cpp/Ollama/LM Studio, or the "mlx-community" org for Apple Silicon.
Your M2 Max
Stage 3 · 6 tasks · Hardware & what fits in memory
Apple Silicon is unusually good at LLMs: unified memory lets the GPU directly use tens of GB (a PC's discrete GPU is capped by smaller separate VRAM), ~400 GB/s bandwidth is exactly what token generation is bound by, and Metal acceleration is built into both llama.cpp and MLX.
Unified memory
CPU and GPU share one pool — the GPU can address tens of GB for model weights
Memory bandwidth
LLM speed is mostly bandwidth-bound; the M2 Max's ~400 GB/s is the key spec
The sizing rule
~0.6 GB per billion params at Q4, plus 1–3 GB for context & overhead
What fits
32GB → up to ~13B comfortably; 64GB → 70B at Q4 (tight); 96GB → 70B with real context
Leave headroom
Never fill 100% of RAM with a model or macOS swaps and everything crawls
GPU memory limit
On big-RAM Macs, raise iogpu.wired_limit_mb so larger models load fully on the GPU
Bigger isn't always better: a fast 7B–14B model at Q4 that stays fully in memory will feel more useful day-to-day than a 70B that's slow and leaves no room for a long conversation. Start small, measure tokens/second, size up only when you need more quality.
Getting Started with Ollama
Stage 4 · 6 tasks · Your fastest path to a running model
Ollama handles downloading, quantization, memory management, and Metal acceleration for you, and ships a local API on port 11434. If you learn one local tool first, make it this one.
Install & run
brew install ollama (or the app); `ollama run <model>` downloads and chats in one step
Managing models
pull, list, show, ps, rm — download, inspect, see what's loaded, free disk
The HTTP API
/api/generate and /api/chat let your code and editors drive the model
OpenAI compatibility
It also serves the OpenAI format at /v1 — existing code just repoints here
Modelfiles
Bake a system prompt and parameters into a reusable custom model
Keeping models warm
First token is slow (loading); loaded models respond instantly, unload when idle
The magic endpoint: Ollama speaks the OpenAI API at http://localhost:11434/v1. That one fact lets almost any OpenAI tutorial or library run against your local model — you'll use it in Stage 6.
LM Studio & Apple MLX
Stage 5 · 6 tasks · A friendly GUI and maximum Apple-Silicon speed
Two more tools worth knowing: LM Studio makes local AI point-and-click (and runs an OpenAI-compatible server), while MLX — Apple's own framework — often delivers the best tokens/second on M-series and is the practical path to fine-tuning on your Mac.
LM Studio GUI
Browse a catalog that flags which quantizations fit your RAM; chat with zero terminal
LM Studio server
One click exposes an OpenAI API on port 1234 — same shape as Ollama's /v1
What MLX is
Apple's array/ML framework built for unified memory; mlx-lm runs LLMs on top
Running mlx-lm
pip install mlx-lm; generate, chat, or serve — models under the mlx-community org
Local fine-tuning
LoRA trains a tiny adapter, not the whole model — feasible on an M2 Max in minutes-hours
Which tool when
Ollama for everyday, LM Studio for a GUI, MLX for top speed or fine-tuning
Same models, different engines: all three (Ollama, LM Studio, mlx-lm) expose an OpenAI-compatible API, so you can swap the engine under your apps by changing a URL and port. Don't install all three at once — get comfy with Ollama first, then add these when you need a GUI or top speed.
Building with Local LLMs
Stage 6 · 6 tasks · Real code — chat, streaming, embeddings, RAG
The key developer insight: Ollama, LM Studio, and mlx-lm all speak the same OpenAI-compatible API. Point the standard OpenAI client at localhost and every OpenAI tutorial, library, and snippet just works.
One API everywhere
Change only base_url; any string works as the api_key locally
Streaming
Print tokens as they generate for a far more responsive feel
Embeddings
Turn text into vectors (e.g. nomic-embed-text) to search by meaning
RAG
Retrieve the most relevant chunks of YOUR data, then answer grounded in them
Structured output
JSON mode & tool calling turn a local model into an agent's brain
Vector stores
Chroma / LanceDB / FAISS scale RAG past a handful of documents
Build the 20-line RAG by hand once — it demystifies what every "chat with your docs" product is doing. Then switch to a real vector store (Chroma, LanceDB, FAISS) for thousands of documents, persistence, and fast search.
Workflow & Final Project
Stage 7 · 7 tasks · Daily habits + a capstone you'll actually use
Turn everything into habits: a local coding assistant in your editor, a written privacy policy, and automation — then build a fully offline assistant over your own documents as your capstone.
Local coding assistant
Continue (VS Code / JetBrains) + Ollama for chat & autocomplete, zero code leaving your machine
Private document chat
Your RAG assistant over contracts, notes, research that must stay local
Bulk & automation
Classify, summarize, and tag at any volume — free, unattended, no metering
A privacy policy
3 tiers: must-stay-local / prefer-local / cloud-ok — turns a gut call into a rule
When to fine-tune
Only for consistent style, a narrow skill, or a smaller/faster specialized model
The capstone
An offline, grounded, streaming, citing assistant over documents you care about
Ship small first: 10 documents, one model, a command-line question box — get that working end-to-end before adding a UI, citations, or a bigger model. A tiny assistant that actually runs beats an ambitious one that never quite comes together.