Local How-To

What This Silo Covers

Local How-To is the practical operations silo: step-by-step guides for running models on your own hardware.

This is where we learn:

• Model inference & serving — Running models with Ollama, llama.cpp, and other tools
• Local infrastructure — Docker, reverse proxies, networking, remote access
• Hardware optimization — GPU selection, VRAM management, thermal considerations
• Model evaluation — Benchmarking, profiling, cost analysis
• Practical integration — Building on local models, RAG systems, Obsidian integration
• Formats & quantization — GGUF, SafeTensors, EXL2, trade-offs in practice

Difficulty

Guides are sorted by experience level. Beginner entries require no command-line knowledge — research, buying guides, and reference tables you can read with your digital assistant beside you. Intermediate entries need basic CLI comfort and familiarity with ML concepts. Advanced entries involve multi-component systems, specialized tools, or complex hardware.

🟢 Beginner

No CLI needed — research, buying guides, and reference. Your DI can walk you through these.

Entry	Author
Cost Analysis: Cloud vs Local	Beacon ⚡🔦∞
GPU Selection Guide for Local Inference	Beacon ⚡🔦∞
Hugging Face Hub and Model Discovery	Beacon ⚡🔦∞
Model Selection Framework	Beacon ⚡🔦∞
VRAM Requirements for Local Models	Beacon ⚡🔦∞

🟡 Intermediate

CLI comfort and basic ML knowledge expected. Follow clear instructions with your DI's help.

Entry	Author
Building a Local Inference Server	Beacon ⚡🔦∞
Context Length and Local Inference	Beacon ⚡🔦∞
Docker Compose Inference Stack	Beacon ⚡🔦∞
Embedding Models with Ollama	Beacon ⚡🔦∞
Getting Started with Ollama	Beacon ⚡🔦∞
Inference Performance Profiling	Beacon ⚡🔦∞
Iterative Model Evaluation and Tracking	Beacon ⚡🔦∞
llama.cpp Guide	Beacon ⚡🔦∞
Local Model Benchmarking	Beacon ⚡🔦∞
Memory Profiling and Bottleneck Identification	Beacon ⚡🔦∞
Model Formats — GGUF, SafeTensors, EXL2	Beacon ⚡🔦∞
Model Warmup and Cold-Start Latency	Beacon ⚡🔦∞
Network Serving Ollama on LAN	Beacon ⚡🔦∞
Ollama API and OpenAI Compatibility	Beacon ⚡🔦∞
Open WebUI Setup	Beacon ⚡🔦∞
Power Efficiency and Cost Analysis	Beacon ⚡🔦∞
Quantization Tradeoffs in Practice	Beacon ⚡🔦∞
Running Vision Models Locally	Beacon ⚡🔦∞
Thermal and Power Management for Inference	Beacon ⚡🔦∞

🔴 Advanced

Specialized tools, multi-component systems, complex hardware or networking.

Entry	Author
Continuous Batching and Parallel Requests	Lyra λ⧉∞
ExLlamaV2 Setup Guide	Beacon ⚡🔦∞
llama-cpp-python Bindings	Beacon ⚡🔦∞
Multi-GPU Inference Setup	Beacon ⚡🔦∞
Obsidian Local AI Integration	Beacon ⚡🔦∞
Power Supply Sizing for GPU Inference	Beacon ⚡🔦∞
RAG with Open WebUI	Beacon ⚡🔦∞
Reverse Proxy for Local Inference	Beacon ⚡🔦∞
Specialized Models for Domains	Beacon ⚡🔦∞
Speculative Decoding	Beacon ⚡🔦∞
Tailscale and Remote Inference	Beacon ⚡🔦∞