Writing
Notes I write to understand how large language models, retrieval systems and AI infrastructure work, from the math to the engineering trade-offs.
- Why Chunking Matters in RAG Pipelines: Beyond Splitting Documents
Chunking decides what a retriever can find and what a generator can see. A look at fixed, recursive, semantic and structure-aware chunking, the precision/recall trade-off behind chunk size, parent-child and contextual retrieval, and how to evaluate a chunking configuration instead of guessing.
- How Large Language Models Actually Work: From Tokenization to Autoregressive Generation
A walk through the full path of a decoder-only transformer, from bytes to tokens to embeddings, through RoPE, attention, feed-forward layers and residual streams, to the training objective, sampling, and the arithmetic that sets the cost of training and inference.
- Knowledge Distillation in LLMs: Transferring Intelligence into Smaller Models
Why a small model trained on a large model's outputs learns more than one trained on the data alone. Soft targets and temperature, the distillation loss and its T² factor, forward versus reverse KL for generative models, feature-based and on-policy distillation, and how distillation combines with pruning and quantization in production.
- Why KV Cache Matters in LLM Inference: Attention Optimization, Memory Trade-offs, and Serving Efficiency
The KV cache turns quadratic recomputation into linear work, then becomes the main memory cost of serving LLMs. Where K and V come from, prefill versus decode, how to size the cache, and the techniques that shrink or manage it, from GQA and MLA to PagedAttention, prefix caching, quantization and eviction.
- LLM Alignment: Understanding Human Preferences, RLHF, DPO, and the Alignment Problem
Why a model trained to predict text is not a model that does what we want, and how post-training closes part of that gap. Supervised fine-tuning, preference data and the Bradley–Terry reward model, RLHF with PPO and a KL penalty, the derivation of DPO, GRPO and verifiable rewards, AI feedback, and the failure modes such as reward hacking and sycophancy that remain open.
- Dense Retrieval - How It Actually Works: Embeddings, Contrastive Training, and Approximate Nearest Neighbor Search
How dense retrievers turn queries and documents into vectors that can be compared, why they are trained with contrastive losses and hard negatives, the bi-encoder versus cross-encoder trade-off, how ANN indexes like HNSW, IVF and product quantization make search over millions of vectors fast, and where dense retrieval still loses to BM25.
- BM25 - How It Actually Works: Probabilistic Relevance, Term Saturation, and Length Normalization
Why a ranking function from the 1990s is still the baseline that neural retrievers have to beat. The inverted index behind it, where IDF and term-frequency saturation come from in the probabilistic relevance framework, how length normalization changes rankings, a worked example, BM25F and learned sparse variants, and where BM25 fails.