Nikhil Mourya. Reverse-engineering LLM inference and serving. Driving LLMs at 780 tok/s.

一About

The math decides when to stop, not the model's confidence.

temperature

Retrieval scores over self-reported certainty. Pruned weights over bloated checkpoints. Shipped code over benchmark screenshots.

So I go one layer down: I write the scheduler instead of calling generate(), and the autograd instead of importing it.

BIT Mesra. Still in a staring contest with one code block. We have beef.

languages
  • Python
  • C++20
  • TypeScript
  • SQL
modeling
  • PyTorch
  • HuggingFace
  • PEFT / LoRA
  • scikit-learn
inference
  • Continuous batching
  • Paged KV cache
  • Pruning
  • ONNX
  • Ollama
agents · rag
  • LangGraph
  • ChromaDB
  • FAISS
  • RAGAS
  • LangSmith
infra
  • FastAPI
  • Docker
  • AWS
  • Redis
  • OpenTelemetry
肖像 · nikhil.exe online
Jaipur, IN train/loss0.000

full precision

名刺 nikhil --infov2026
location
Jaipur, India
college
BIT Mesra
focus
Multi-agent systems
cf
1400+ Specialist
builds
SIH '24 · IIIT-D '23
reading
The Humans, Matt Haig

Ask this page

Searches this page in your browser. No LLM. If it doesn't know, it says so.

Right now

live, while you read
Jaipur · --:-- IST Probably building something Replies usually within a day

二Experience

Where it runs in production.

Where my code met real users.

  1. Now

    Rasen AI

    Machine Learning Engineer – AI Infrastructure · Remote

    • Building real-time STT → LLM → TTS voice agents for an Accel-backed lab on Pipecat, FastAPI and GCP/Kubernetes; 13 PRs merged in two weeks.
    • Cut call handshake latency 6 s → 2.5 s (58%+); shipped tool-calling runtime, versioned knowledge store, and data-ingestion pipeline from scratch.
  2. 2025

    HireBuddy

    Software Engineer, ML

    • Replaced regex resume parsing with a RAG + Transformers matcher.
    • 10K+ resumes a day through FastAPI, Docker and AWS, measured with RAGAS and LangSmith.
    +35%shortlist accuracy
    −28%chain latency
    +40%user engagement
  3. Founder

    DevPath

    Live product · dev-path.site

    Roadmaps and one focused task a day, for developers stuck in tutorial hell. Built and shipped solo.

  4. Research

    IIT Kharagpur

    Model compression collaboration

    Pruned a U-Net for biomedical segmentation: 97.3% fewer parameters, IoU still above 0.95. Most of it was along for the ride.

三Work10 projects

Rebuilt from first principles. No black boxes I didn't open.

Two systems, end to end. One serves LLMs, one trains agents.

LLM inference & serving

★ 97 ⑂ 10

PageServe

A vLLM-style inference engine, grown from a naive generate() loop in 11 phases.

Continuous batching, a paged KV cache and CPU swap. Under load it slows down instead of falling over.

−98.5%
mean TTFT at 4-way load
1,418 → 20.9 ms
−26.5%
wall-clock latency
vs sequential
11
build phases,
each runnable
kv pool · 64 blocks × 16 tokens live
pageserve-phi.vercel.app open ↗

Reinforcement learning · C++20

★ 11

RLForge

A reinforcement-learning library with no import torch, down to its own autograd.

Tensors, autograd, DQN and PPO in C++20, zero dependencies, and the diamond-graph gradient that most from-scratch engines get wrong.

PPO, threading and CUDA backends co-built with aprv10.

148/148
tests passing
warnings as errors
0
runtime
dependencies
−100→3
GridWorld eval return
3 = proven optimal
backward pass · diamond dependency tensor.cpp
x ax · 2 bx · 3 ca + b lossmean(c) forward ↑ grad ↓ ∂x += 2/n ∂x += 3/n → 5/n ✓
$ cmake --build build -j && ctest
100% tests passed, 0 failed out of 148
$ ./train --env gridworld --steps 20000
eval return -100.0 → 3.0 # optimal
rl-forge.vercel.app open ↗
  • A 5-agent pipeline (Planner → Researcher → Critic → Writer → Evaluator) that loops until a deterministic confidence threshold is met, computed from mean retrieval score, not the LLM's self-report. The same rule "Ask this page" uses up there.

    Agents & RAGFastAPI · Redis · ChromaDB · OpenTelemetry · Docker

  • A VS Code extension that chunks code by AST with tree-sitter, embeds it locally through Ollama, and jumps you to the exact line from a plain-English query. No internet, no API keys. It reindexes on save and beat plain grep on every query in a 50-query test.

    ToolsTypeScript · tree-sitter · Ollama · VS Code API

  • Retrieval by LLM-guided tree traversal instead of embeddings. I wanted to know whether the vector-DB default was actually necessary, so I built the version without one.

    Agents & RAG

  • Fine-tuned a 767M summariser with 1.57M trainable params. Full fine-tuning fell apart on unseen domains; LoRA didn't.

    ModelsPyTorch · HuggingFace · PEFT

  • Research collaboration with IIT Kharagpur. Structured pruning of U-Net for biomedical segmentation: 97.3% fewer parameters, 92% fewer FLOPs, IoU still above 0.95 on MoNuSeg.

    Research

  • The full Transformer from the paper, multi-head attention to training loop. Not "used a framework."

    ModelsPyTorch

  • Skin-lesion classification across 7 classes with a 57:1 imbalance. Focal loss so the rare classes count, Grad-CAM so you can see what it looked at, ONNX export so it runs anywhere.

    Models

  • A senior engineer code-reviews your brain. Six-stage multimodal pipeline, real models, no affirmations.

    Tools

四Lab5 toys

Ideas I keep explaining, so now you can poke them.

Speculative decoding how fast inference cheats, honestly

A small model drafts 4 tokens; the big one checks them in one pass and keeps what matches.

target passes
…
tokens / pass
…
expected (1−αk+1)/(1−α)
…
vs one token per pass
…

五Writing20 articles

I build it, then I write it down. Usually in that order.

What I cannot create, I do not understand.
Richard Feynman

六Contact

If you've made it this far, you might as well say hi.

Email is fastest. DMs on X work too.

It's --:-- in Jaipur (IST)

鯉 · tap the water to feed them They come when you tap the water. I come when you send an email.

Writing · visible only to you

Add an article

Checked against the server once, then remembered in this browser only.