tiny-duo-infer

A learning-first LLM inference engine built from scratch on Apple Silicon, with an OpenAI-compatible local API

View on GitHub Learn more ↓

About

tiny-duo-infer is a pure-Python LLM inference engine that teaches how large language models work under the hood — from tokenization to the final sampled token. Every inference concept is visible in readable code rather than hidden behind Transformers, mlx-lm, or vLLM.

Each building block — prefill, decode, KV cache updates, grouped-query attention, RoPE rotary embeddings, SwiGLU feed-forward networks — is explicitly implemented so that you can trace exactly what happens between a prompt and a generated token.

Inspired by nano-vllm and tiny-llm, the project follows a progressive learning roadmap: from single-user Llama inference to multi-model support, HTTP serving, per-request observability, and INT4/INT8 weight-only quantization.

Read the code. Understand the engine. Six core inference concepts — fully explicit, no black boxes.

Prefill & Decode Loop
Trace every token generation step — from KV cache population through the decode loop to EOS detection.
Grouped-Query Attention (GQA)
See how K/V heads are expanded to match Q heads, and how the causal mask is built and applied at each step.
Rotary Position Embeddings (RoPE)
Follow frequency precomputation and the explicit rotation formula applied pair-wise across the head dimension.
KV Cache Management
Understand the update() / advance() protocol that keeps K/V positions consistent across all transformer layers.
Weight-Only Quantization
Learn how INT4/INT8 quantized matmul replaces full-precision forward passes, and how memory savings are measured per run.
Sampling Strategies
Step through greedy argmax, temperature scaling, top-k filtering, and top-p nucleus selection — all in plain Python.
Prompt Tokenizer Prefill · Decode Engine KV Cache Sampler Output

Explore the source on GitHub →

10
CORE CONCEPTS BUILT FROM SCRATCH
RoPE · GQA · KV Cache · SwiGLU · Quantization · ···
2
MODEL FAMILIES
Llama-3.2-1B · Qwen3-0.6B
589
FAST-SUITE TESTS
unit · integration · serving · OpenAI API
0
FRAMEWORK WRAPPERS
no mlx-lm, no transformers core, pure Python

Capabilities

Explicit Prefill → Decode Pipeline
The full generation loop — prefill, KV cache population, decode step, EOS detection — is implemented in readable Python with no framework wrapper.
GQA · RoPE · SwiGLU from Scratch
Grouped-query attention, rotary position embeddings, and SwiGLU feed-forward networks are each implemented explicitly so every computation is traceable.
INT4 / INT8 Weight-Only Quantization
MLX-native quantized matmul with per-group affine scales. Memory accounting reports full-precision vs runtime linear-weight bytes for every run.
Loopback HTTP + OpenAI-Compatible Chat
FastAPI keeps native JSON/NDJSON endpoints and adds /v1/models plus Chat Completions with JSON and SSE streaming. Qwen3 supports chat; Llama remains on native endpoints. One generation runs at a time across every route.
Per-request Observability
Time-to-first-token, decode throughput, KV-cache memory, context-budget policy outcome, and quantization stats — reported on every request via CLI or HTTP.
Multi-model Portability
Supports Llama-3.2-1B and Qwen3-0.6B on the same engine, demonstrating model-family portability before backend portability.

Tech Stack

Language Python 3.12 — no C++ or custom kernels
Backend MLX (Apple Silicon unified memory)
Models meta-llama/Llama-3.2-1B · Qwen/Qwen3-0.6B
Tokenizer HuggingFace tokenizers package
Weights safetensors shard loading with HF key mapping
Serving FastAPI — loopback by default; native JSON/NDJSON + OpenAI Chat Completions (JSON/SSE)
Tooling uv · pytest (589 tests) · GitHub Actions CI

CLI & API

Run inference locally from the command line, start a local OpenAI-compatible server, or profile generation across quantization modes — all with a single uv run. 

# Plain completion (Llama-3.2-1B)
uv run python -m tiny_duo_infer.cli \
  --model-path ./models/llama-3.2-1b \
  --prompt "The capital of France is" \
  --max-new-tokens 32 --temperature 0.0

# Qwen3 chat mode
uv run python -m tiny_duo_infer.cli \
  --model-path ./models/qwen3-0.6b \
  --prompt "What is attention in transformers?" \
  --chat --max-new-tokens 64

# INT4 quantization with full stats
uv run python -m tiny_duo_infer.cli \
  --model-path ./models/llama-3.2-1b \
  --prompt "The capital of France is" \
  --quantization int4 --quant-group-size 64 --show-stats

# Start a loopback server for OpenAI-compatible Qwen3 chat
uv run python -m tiny_duo_infer.serving.api \
  --model-path ./models/qwen3-0.6b \
  --served-model-name qwen3-0.6b --max-seq-len 2048

# OpenAI Chat Completions with SSE streaming
curl -s -N http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-0.6b","messages":[{"role":"user","content":"Explain KV cache."}],"stream":true}'

Full documentation on GitHub →

Interested in the source?

View on GitHub