Building a 0.04B (36M) LLM from Scratch: End-to-End Full-Stack Overview

🎯 Engineering Context, Dataset Profiles, and Full Lifecycle Targets

1. Task Background and Core Engineering Challenges

This project implements a domain-specific classical Chinese poetry model completely from scratch. Unlike fine-tuning pre-existing weights, every component—from raw byte tokenization to tensor operators, dual-track dataset processing, pretraining, SFT alignment, and autoregressive streaming inference—is custom built.

Short-form classical poetry introduces a unique industrial challenge: the hyper-dense metadata crisis:

• Standard web corpora: Contexts span thousands of tokens where titles and authors occupy less than $1\%$, making raw concatenation harmless.
• Classical poetry stanzas: A five-character quatrain has only 20 content characters, while title, author, and dynasty span 8–15 characters—metadata accounts for 35% to 45% of total text!
• Naive training causes small 0.04B models to overfit on author names and regurgitate structural markers. We solve this via a three-part defense: 8 atomic special tokens + label loss masking (-100) + 25% metadata dropout.

2. Dataset Profile

• Raw Data Source: Chinese Classical Poetry Corpus (aggregating 484 poet databases with 476,329 raw poems spanning pre-Qin, Han, Tang, Song, Ming, and Qing dynasties).
• Three-Stage Purification Funnel:
  1. Formatting & numbering cleanup: Strip extraneous annotations, volume indices, and damaged glyph placeholders (□, 〇), purging 15,993 corrupted entries;
  2. MinHash deduplication: Remove 2,058 duplicate poems with identical title and content;
  3. Meter & rhyme inspection: Validate line lengths between 4 and 8 characters with at least two stanzas.
• Curated Asset Manifest:
  • Primary metadata corpus: data/poetry_meta.jsonl (460,336 clean poems, 29,013,877 characters, formatted as {"title": "...", "author": "...", "dynasty": "...", "content": "..."}), powering both pretraining and SFT;
  • Dedicated tokenizer corpus: data/raw.txt (83.89 MB of pure verse stripped of metadata tags to prevent label contamination during BPE vocabulary induction);
  • Packed binary data streams: train_tokens.bin (59.37 MB, 31,126,208 tokens) + train_labels.bin (59.37 MB with -100 prefix masking), with 5% split for validation.

3. Historical Version Evolution and Architecture Trade-offs

[!NOTE] Legacy Checkpoint Cleanup Notice: Early exploratory 0.1B checkpoints (~3GB) and pre-bugfix runs (such as anomalous runs prior to fixing the double-shift issue) have been archived. All active workflows use the validated checkpoints_0.04b/ directory.

4. Full Lifecycle Milestone Targets

📖 Comprehensive Curriculum Directory and Guide Index

Chapter 00 ｜ 0.04B Model Architecture Design and Intuition

• 0.1 Foundations: Understanding Large Language Models through Real-World Metaphors (Mixing console knobs, building blocks, and carpenter workbench memory models)
• 0.2 0.04B Hyperparameter Blueprint (Vocab 4096, 512 dim, 12 layers, SwiGLU 1408 dim, GQA 8Q/2KV)
• 0.3 Mathematical Breakdown: Where Are the 35.93M Parameters Allocated? (Layer-by-layer accounting and shared weight tying)
• 0.4 Training Volume: How Much Data Is Needed to Saturate a Small Model? (Chinchilla scaling laws and 460k poetry corpus saturation)
• 0.5 Learning Mindset and Roadmap Navigation

---

Chapter 01 ｜ Implementing a Byte-Level BPE Tokenizer from Scratch

• 1.1 First Principles: How Silicon Hardware Understands Text (Word-level OOV nightmares vs. character-level sequence explosion)
• 1.2 BPE Algorithm by Hand: A 3-Minute Paper Derivation (Manual pair frequency counting on toy examples)
• 1.3 Pre-tokenization Regex: Why Punctuation Cleavers Are Essential (Fused punctuation regex eliminating rogue hanging punctuation)
• 1.4 Pure Python BPE Tokenizer Implementation (Special tokens, 256 base byte atoms, inverted index pair merging, serialization)
• 1.5 Industrial Pitfall 1: Structured Special Tokens and Atomic Protection (Preventing multi-token subword fragmentation)
• 1.6 Practical Lab: Verifying Lossless Reconstruction (Spot-checking 4096 vocabulary with punctuation ratio dropping to 0.53%)

---

Chapter 02 ｜ Building the Modern Transformer Core from Scratch

• 2.1 Modern Core Operator Deconstruction and Intuition
  • RMSNorm (Automatic volume line-level balancer)
  • RoPE (Relative wristwatch angular distance encoding)
  • SwiGLU (Dual-channel intelligent feature valve)
  • GQA (8Q : 2KV grouped-query attention)
• 2.2 Assembling 0.04B Transformer Blocks (TransformerBlock and MiniLLaMAForCausalLM)
• 2.3 Verification: Architecture Diagnostic Check and Parameter Count (35,926,528 parameters)
• 🌟 Chapter 02 Appendix ｜ High School Math Derivation of Modern Transformers (Dot products, complex rotations, Euler's formula, variance scaling $\frac{1}{\sqrt{d_k}}$, cross-entropy)

---

Chapter 03 ｜ High-Performance Binary Data Pipelines and Memory Mapping

• 3.1 Beginner Pitfalls: The Two Fatal Traps in Data Pipelines (On-the-fly tokenization starving GPUs vs. in-memory loading causing OOM)
• 3.2 Production Solution: Offline Pre-baking and the Card Catalog Pattern (uint16 binary packing and np.memmap virtual memory paging)
• 3.3 Causal Alignment Standard (Isomorphic input and label alignment where shift operations occur strictly inside the model)
• 3.4 Data Anatomy: Why Metadata Is Toxic in Short Stanzas (Quatrain metadata comprising 40% of sequence length inducing mode collapse)
• 3.5 Three Architecture Trade-offs and Decision Matrix (Raw text vs. loss masking vs. synthetic encyclopedia wrappers)
• 3.6 Industrial Pitfall 2: Double-Shift Bug and Metadata Loss Masking (Resolving double-shift stagnation, -100 target masks, and 25% metadata dropout)
• 3.7 Offline Pre-baking and Zero-Copy Dataset Loader in PyTorch (prepare_data.py and src/dataset.py dual-stream loading)

---

Chapter 04 ｜ Engineering-Grade Pretraining Engine and Optimization

• 4.1 Intuition: Loss and Perplexity as a Multiple-Choice Exam (Shrinking search space from 4,096 candidates to 7 sharp choices)
• 4.2 Conditional Pretraining Loss Masking (ignore_index=-100 isolating prefix backpropagation gradients)
• 4.3 Optimizer Tuning: Precision Bonsai Pruning (Decaying 2D weight matrices while strictly exempting 1D normalization vectors)
• 4.4 Learning Rate Schedules: Warmup and Cosine Decay
• 4.5 Production Speed and Memory Dual Guards (AMP mixed precision and gradient accumulation simulating large batch dynamics)
• 4.6 Complete Pretraining Engine Source Code: src/trainer.py (Checkpointing and seamless training resumption)

---

Chapter 05 ｜ Autoregressive Inference and KV-Cache Acceleration

• 5.1 What Is Autoregressive Generation? Demystifying Word Continuation
• 5.2 The Fatal Trap of Naive Inference: The $O(N^2)$ Goldfish Dilemma
• 5.3 Breaking Through: Understanding KV-Cache via Sticky-Note Memory (Compressing sequence state into ~6MB memory footprints)
• 5.4 Sampling Strategies: Temperature, Top-K, and Top-P Shift Logic
• 5.5 Rhythmic Constrained Decoding (Enforcing exact 5-character and 7-character metric symmetries via state machines)
• 5.6 Hands-on Streaming Generator: Crafting Your Interactive Classical Poet

---

Chapter 06 ｜ End-to-End Hands-on Training Run and Failure Recovery

• 6.1 Hands-on Lab: Running the End-to-End Workflow Locally (Five-act experiment and stateful training resumption)
• 6.2 Dual Generation Modes: Prompted Topic Generation and Freeform Continuation
• 6.3 Troubleshooting Clinic: Field Recovery for Six Common Failures (Double-shift loss floor at 5.18, metadata looping, and punctuation domination)
• 6.4 Future Horizons: The Three-Stage LLM Evolution Roadmap (Pretraining $\to$ SFT $\to$ DPO $\to$ Quantized Deployment)

---

Chapter 07 ｜ Supervised Fine-Tuning (SFT) and Instruction Alignment

• 7.1 The Two-Stage Leap: From Conditional Completion to Instruction Following (Bridging base models to helpful assistants)
• 7.2 The Secret Sauce of SFT: Prompt Masking and ignore_index=-100 (Penalizing output tokens while ignoring user prompts)
• 7.3 Industrial Chat Templates and Structural Tokens (<|im_start|>, <|im_end|>, preventing repetition loops)
• 7.4 Hands-on Lab: Synthesizing 30,000 Alpaca-Style Instructions from Poetry Metadata (Dynamic template variation across 5 dimensions)
• 7.5 Writing the SFT Dataset Loader and Tensor Collation (SFTDataset, dynamic padding, and collate routines)

---

Chapter 08 ｜ Parameter-Efficient Fine-Tuning: LoRA from Scratch

• 8.1 Why PEFT Matters: Overcoming Consumer VRAM Constraints
• 8.2 Mathematical Aesthetics of LoRA: The Art of Low-Rank Decomposition ($W = W_0 + \frac{\alpha}{r} B \cdot A$)
• 8.3 Implementing LoRA Linear Layers in PyTorch (LoRALinear and LinearWithLoRA)
• 8.4 Adapter Injection and Parameter Freezing Pipeline (Fine-tuning only 0.5% of total parameters)
• 8.5 Weight Merging for Zero-Latency Deployment

---

Chapter 09 ｜ Full Training Lifecycle Review and Golden Rules

• 9.1 Production Environment and Hyperparameter Telemetry Card
• 9.2 Pretraining Diagnostics and Convergence Milestones
• 9.3 SFT Instruction Tuning Lab
• 9.4 LoRA Parameter-Efficient Adaptation Run
• 9.5 Head-to-Head Model Evaluation Across Three Stages
• 9.6 Industrial Golden Rules: Eight Hard Principles for LLM Engineering
• 7.6 SFT Fine-Tuning Golden Rules (Low learning rates preventing catastrophic forgetting, fast convergence)
• 7.7 Final Demonstration: Base Model vs. SFT Behavioral Evolution

---

Chapter 08 ｜ Standing on the Shoulders of Giants: Industrial LoRA and Qwen2.5 Practice

• 8.1 VRAM Crisis: Why Consumer GPUs Cannot Handle Full Fine-Tuning (120GB VRAM ledger for full 7B parameter tuning)
• 8.2 Mathematical Aesthetics of LoRA (Intrinsic rank hypothesis, $\Delta W = B \times A$, zero-perturbation initialization)
• 8.3 Modern Base Model Selection: Why Qwen2.5-1.5B Hits the 16GB Sweet Spot (Qwen2.5 model family landscape from 0.5B to 72B)
• 8.4 Handcrafting a Minimal LoRA Layer from Scratch (Demystifying low-rank adapters in under 30 lines of pure PyTorch)
• 8.5 Weight Merging and Zero-Cost Inference (Offline weight fusion adding zero latency and zero VRAM footprint in production)
• 8.6 Track 1: Native 0.04B Local LoRA Fast Adaptation Lab (Pure PyTorch, 319k low-rank parameters, 1.2MB adapter checkpoint)
• 8.7 Track 2: Qwen2.5-1.5B Industrial Model Track (ChatML prompt masking and real-time streaming dialogue)
• 8.8 Conclusion: Full-Stack Large Language Model Mastery

---

Chapter 09 ｜ Full Training Lifecycle Review: From Pretraining to SFT and LoRA

• 9.1 Production Telemetry and Full Configuration Blueprint (Intel Arc 130T 16GB, 460k poems, 34.63M tokens, 0.04B architecture)
• 9.2 Act One: Pretraining Diagnostics and Telemetry (Console traces, JIT warm-up, and PPL convergence curves)
• 9.3 Act Two: SFT Instruction Tuning in Practice (Dataset automation, prompt loss masking, and 10-minute alignment)
• 9.4 Act Three: LoRA Low-Rank Adaptation Sprint (0.48% parameter fine-tuning and one-click lossless weight merging)
• 9.5 Act Four: Head-to-Head Evaluation (Pretrained base vs. SFT instruction model showdown)
• 9.6 Master Debrief Lab: Five Golden Rules of LLM Training

---

Happy learning! Begin your journey to building language models from the ground up! 🚀