What Colibrì Actually Does

GLM-5.2 is a Mixture-of-Experts model: instead of activating all 744 billion parameters for every token, it routes each token through a small subset of "expert" sub-networks. Only about 40 billion parameters actually fire per token, and of those, roughly 11GB worth are the "routed" experts that change from token to token.

Colibrì exploits that structure directly:

  • Dense layers stay resident in RAM. Attention, shared experts, and embeddings — about 17B parameters — are quantized to int4 and kept in memory at all times (~9.9GB resident).
  • Routed experts stream from disk on demand. The remaining ~21,504 expert chunks (roughly 19MB each at int4) live on disk — around 370GB total — and are streamed in per-layer with an LRU cache, rather than pre-loaded.
  • No external dependencies at inference time. The engine itself is about 1,300 lines of C. Python is only used for the one-time model conversion step and an optional API gateway.

In effect, Colibrì treats VRAM, RAM, and disk as one managed memory hierarchy instead of requiring the entire model to fit in RAM or VRAM at once.

Setup Requirements

Before trying this yourself, know what you're signing up for:

RequirementDetails
RAM~25GB (int4 dense layers, resident)
Disk space~400GB free (converted expert weights)
Compilergcc + OpenMP
Conversion stepRequires Python: torch, safetensors, huggingface_hub, numpy
LicenseApache 2.0

Basic setup:

cd c
./setup.sh                              # builds with gcc + OpenMP, runs self-test
./coli convert --model /nvme/glm52_i4   # one-time conversion, ~400GB free, resumable
# or download prebuilt weights instead of converting:
# https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4

COLI_MODEL=/nvme/glm52_i4 ./coli chat

Other useful commands:

./coli plan     # inspect planned VRAM/RAM/disk placement before running
./coli doctor   # read-only readiness check on your hardware
./coli web      # API + web dashboard on one port
./coli serve    # OpenAI-compatible API only, no dashboard

The Catch: Speed

This is the part most write-ups gloss over. Colibrì's whole design is a tradeoff: it makes a 744B model runnable on modest hardware, not fast.

  • Single spinning or SATA SSD — expect well under 1 token per second.
  • Faster NVMe storage — community-reported numbers land in the low single digits of tokens per second.
  • High-end setups — the project's own demo uses 6× RTX 5090 GPUs with full expert residency, reaching around 4 tok/s with sub-2-second time-to-first-token.

In other words: on a single consumer laptop, this is not a chat replacement for a hosted assistant. It's closer to a research curiosity you queue a task on and check back on later, not something you'd use for interactive conversation.

🤖

DevOpsArsenal AI Prompt Library

If you're experimenting with local or hosted LLMs, browse ready-made prompt templates for debugging, code review, incident writeups, and more — free, no signup.

Try Prompt Library Free →

Is Colibrì Right for You?

Colibrì is worth trying if:

  • You want to run frontier-scale open weights without renting cloud GPUs.
  • You have spare NVMe storage (~400GB) and are comfortable with a CLI workflow.
  • You're experimenting rather than running a production inference service.

It's not a fit if you need real-time responses, are RAM-constrained below ~25GB, or want a polished GUI experience — Colibrì is unapologetically a CLI-first, systems-level project.

Try It Yourself

GitHub repo: github.com/JustVugg/colibri

Prebuilt int4 weights: huggingface.co/jlnsrk/GLM-5.2-colibri-int4

If you try it on your own hardware, the project's maintainer is actively collecting community benchmark numbers via GitHub issues — worth contributing your results if you test it.

Frequently Asked Questions

What is Colibrì? ▼
Colibrì is an open-source, pure-C inference engine (about 1,300 lines of code) that runs GLM-5.2, a 744-billion-parameter Mixture-of-Experts model, on a consumer machine with roughly 25GB of RAM. It works by keeping the model's dense layers resident in memory and streaming the much larger set of routed expert weights from disk on demand, so the full model never has to fit in RAM at once.
How much disk space and RAM does Colibrì need? ▼
Colibrì needs about 25GB of RAM to hold the quantized dense layers resident, plus roughly 400GB of free disk space for the converted int4 expert weights. You'll also need gcc with OpenMP to build the engine, and Python with torch, safetensors, huggingface_hub, and numpy only for the one-time model conversion step.
Is Colibrì fast enough for everyday chat use? ▼
No, not on typical consumer hardware. On a laptop with a single SATA or NVMe SSD, expect well under 1 token per second, since routed experts have to be fetched from disk for every token. Community benchmarks on faster NVMe storage reach low single-digit tokens per second, and the project's own high-end demo (6x RTX 5090 GPUs with full expert residency) hits around 4 tokens per second. It's a proof of concept, not a replacement for a hosted chat assistant.
Does Colibrì change the model's precision or accuracy to save memory? ▼
The dense layers and experts are quantized to int4 as part of the one-time conversion step, which is a deliberate, fixed tradeoff made upfront. Colibrì's memory-management policy — deciding what stays in RAM versus what streams from disk — is designed to never silently change model precision or routing behavior at runtime based on available memory; insufficient fast storage affects speed, not correctness.
Average response time is a comfortable metric — easy to display on a dashboard and easy to celebrate. But it's a poor proxy for user experience. P99 is uncomfortable because it forces you to look at your worst cases. Colibrì is the same kind of uncomfortable-but-honest engineering: it doesn't pretend a 744B model is fast on a laptop. It just makes it possible, and tells you exactly what it costs.