colibri 2.1 runs mixture-of-experts models larger than your RAM by reading their experts from disk

colibri is an Apache-2.0 inference engine in C built around mixture-of-experts models. Each token uses only some experts, so it keeps the always-used part of the model in RAM and reads the other experts from disk when the model asks for them. v2.1.0 (Oct 31) imports published GGUF files with `coli import` and runs them without conversion on CPU, Vulkan and CUDA, and adds Mistral-Large-4 among other models. On the project's test machine (62 GB RAM, one 24 GB card), Qwen3.6-35B-A3B decodes at 104.8 tokens/s on CUDA; a 594 GB Kimi K3 file, served from two NVMe drives, decodes at 0.349 tokens/s. Why it matters: Running a model yourself usually means it has to fit in memory. colibri relaxes that limit for mixture-of-experts models by leaving most of the model on disk. That makes local, private runs of very large open models possible on a single workstation with fast NVMe drives. The tradeoff is speed: the biggest models in the release table decode at under two tokens per second.

Try this: Clone the repo and run `./start-here.sh --list`. It lists every model against your machine's RAM, disk and GPU and says why each one does or doesn't fit, before you download anything.