LLM Project

This project brings together my work through the CS336 language-model assignments: understanding the building blocks of a model, then exploring the cost of training it.

Language-model foundations

The first repository contains implementations of a byte-pair encoding (BPE) tokenizer and Transformer components, including embeddings, RMSNorm, rotary positional embeddings, multi-head self-attention, and SwiGLU feed-forward layers. These are assembled into a language model, alongside an AdamW optimizer and training and inference notebooks.

I’m interested in understanding how these pieces fit together, rather than treating the model as a single black box.

Training systems

The second repository explores forward passes, backward passes, and full training steps through a configurable benchmark. It includes GPU profiling annotations, warmup comparisons, mixed-precision experiments, and a tiled attention forward-pass prototype.

The tiled attention prototype currently covers the forward pass. Completing the backward pass and evaluating its behavior are further steps in this learning project.

What I want to write about

I plan to use this space for notes on tokenization, model architecture, memory use, and the difference between implementing an algorithm and making it efficient. Related essays will appear here as I publish them.