LLM inference & serving
PageServe
A vLLM-style inference engine, grown from a naive generate() loop in 11 phases.
Continuous batching, a paged KV cache and CPU swap. Under load it slows down instead of falling over.
- −98.5%
- mean TTFT at 4-way load
1,418 → 20.9 ms - −26.5%
- wall-clock latency
vs sequential - 11
- build phases,
each runnable