Supervised Fine-Tuning (SFT) and Instruction Alignment

7.2 The Core Secret of SFT: Prompt Masking and ignore_index=-100

To enable a base model to follow instructions, the universal industry technique is Supervised Fine-Tuning (SFT).

Within SFT engineering lies the most critical and frequently overlooked mechanism: Prompt Loss Masking.

7.2.1 The Fatal Mistake: Naive Full-Sequence Fine-Tuning

Suppose we prepare instruction pairs:

• Prompt (Human Instruction): "Write a five-character quatrain about autumn."
• Response (Model Output): "After fresh rain on empty mountains, the evening weather brings autumn."

Amateurs concatenate both strings directly:

"Write a five-character quatrain about autumn. After fresh rain on empty mountains, the evening weather brings autumn."

Then they compute cross-entropy across the entire sequence from start to finish as if pretraining.

The Catastrophic Consequence: The model not only learns to answer, it expends precious gradient updates memorizing human questioning styles! Idiosyncrasies and typos in user questions degrade model weights, leading it to interrupt users or question itself in endless soliloquies.

7.2.2 The Elegant Solution: Penalizing Responses, Never Penalizing Prompts

We must instruct the network:

"Humans may ask however they please; you do not need to memorize their questions. Your sole objective is producing accurate responses given their query!"

In PyTorch, this is achieved via ignore_index = -100 in cross-entropy loss:

===================================================================================
                     SFT 训练中的 Prompt 掩码对齐示意图
===================================================================================

完整输入序列 (X):
  [BOS]  请  以  秋  天  写  首  诗  [SEP]  空  山  新  雨  后  [EOS]

目标标签 (Target Labels Y):
  原始标签:   请   以   秋   天   写   首   诗  [SEP]   空   山   新   雨   后  [EOS]
  SFT掩码:  -100 -100 -100 -100 -100 -100 -100 -100   空   山   新   雨   后  [EOS]
            \_______________________________/        \___________________________/
                     人类提问部分 (Prompt)                  模型回答部分 (Response)
                     全部填入 -100！                        保留真实 Token ID！
                     损失梯度 = 0！                         正常反向传播计算 Loss！

When PyTorch evaluates torch.nn.functional.cross_entropy(logits, targets, ignore_index=-100): any target label set to -100 is hard-skipped with strictly 0.0 gradient!

Thus, 100% of backpropagation compute focuses entirely on generating beautiful answers conditioned on given instructions!

7.3 Industrial Chat Templates and Special Tokens

In production models (Qwen, LLaMA), conversational turns are delimited by structured Chat Templates rather than crude colon separators.

7.3.1 Special Control Tokens as Traffic Lights

Why do amateur models fail to stop generating? Because they lack an explicit boundary indicating when to conclude an answer.

In SFT, we deploy designated Special Tokens:

| Special Token | Operational Role | Physical Meaning | | <|im_start|> | Turn Start Token | Signals to the network: "Pay attention, a conversational turn begins!" | | <|im_end|> (or <eos>) | Turn End Token | The vital brake pedal! Signals: "This speaker has finished, cease generation immediately!" | | system | System Prompt Role | Establishes persona (e.g. "You are a master of classical Tang and Song poetry"). | | user | Human User Role | Contains the human's explicit request or query. | | assistant | AI Assistant Role | The response content the model must learn to generate. |

A standard ChatML sequence formats as follows:

<|im_start|>system
你是一位精通古典诗词创作的文学导师。<|im_end|>
<|im_start|>user
请以《春夜喜雨》为题，为我作一首诗。<|im_end|>
<|im_start|>assistant
好雨知时节，当春乃发生。随风潜入夜，润物细无声。<|im_end|>

| During training, only the response tokens Good rain knows its season...<|im_end|> incur loss; when generating, emitting <|im_end|> terminates inference instantly! |

7.4 Hands-on Practice: Transforming Poetry Corpora into SFT Instruction Pairs

We need not manually annotate hundreds of thousands of dialog turns. Using our 460,000 verified poems, an automated pipeline synthesizes diverse instructions.

7.4.1 Generating Diverse Prompts (Avoiding Stiff Book Title Boilerplate)

Beginners often wrap every poem title rigidly in book brackets 《》:

❌ "Please compose a poem with the title 《Quiet Night Thoughts》."

This constitutes a severe design flaw in LLM engineering!

• Human User Habits: In casual mobile typing, switching keyboards to enter book brackets is tedious. Over 90% of real users never type brackets!
• Prompt Overfitting Hazard: If 100% of training data encases titles in 《》, the model develops tunnel vision, failing when prompted naturally without brackets.

The industrial best practice is a Mixed Distribution: the vast majority employ colloquial phrasing without brackets, with a minority retaining brackets:

PROMPT_TEMPLATES = [
    # 1. 现代自然口语（不带任何书名号，接地气、容错率高）
    "请以{title}为题，创作一首古风诗词。",
    "帮我写一首关于{title}的诗，要有{author}的意境风格。",
    "请模仿{author}的笔触，写一首{title}。",
    "我想读一首关于{title}的诗词，作者偏向{author}。",
    "请围绕{title}赋诗一首，作者风格：{author}。",
    "请为我创作一首古诗，题目叫{title}。",
    "写一首{author}的{title}。",

    # 2. 双引号 / 括号等日常标点
    "请以“{title}”为题作诗一首。",
    "请帮我以【{title}】为题，写一首古诗，风格仿照{author}。",

    # 3. 正式书名号（兼顾严谨公文体裁）
    "请以《{title}》为题，创作一首古典诗词。",
    "假若你是古代诗人{author}，请为我挥毫作一首《{title}》。",

    # 4. 键值格式
    "题目：{title}，作者：{author}",
]

A model trained on this distribution reliably recognizes user intent whether brackets are present or omitted!

7.4.2 SFT JSON Dataset Generation Code

In the scripts/ directory, we implement an automated synthesis tool: scripts/build_sft_data.py:

import json
import random

def build_sft_dataset(raw_poetry_list, output_path, max_samples=30000):
    """
    将原始诗词转换为标准的 Alpaca/ChatML 指令微调格式
    """
    sft_data = []
    
    # 随机采样 30,000 首精品诗词（做 SFT 质量远重于数量）
    sampled_poems = random.sample(raw_poetry_list, min(len(raw_poetry_list), max_samples))
    
    for poem in sampled_poems:
        title = poem.get("title", "无题")
        author = poem.get("author", "无名氏")
        dynasty = poem.get("dynasty", "古")
        content = poem.get("content", "")
        
        template = random.choice(PROMPT_TEMPLATES)
        instruction = template.format(title=title, author=author, dynasty=dynasty)
        
        sft_data.append({
            "instruction": instruction,
            "input": "",
            "output": content
        })
        
    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(sft_data, f, ensure_ascii=False, indent=2)
        
    print(f"✅ 成功生成 {len(sft_data)} 条高质量指令微调样本！")

7.5 Writing the SFT Dataset and Tensor Collation

The core engineering implementation in SFT: concatenating Prompt and Response cleanly inside PyTorch Dataset and setting prompt labels precisely to -100.

7.5.1 SFTDataset Class Implementation

import torch
from torch.utils.data import Dataset

class SFTDataset(Dataset):
    def __init__(self, data_list, tokenizer, max_seq_len=512):
        self.tokenizer = tokenizer
        self.max_seq_len = max_seq_len
        self.samples = []
        
        for item in data_list:
            # 1. 编码人类提示词与模型回答
            prompt = f"问：{item['instruction']}\n答："
            response = item['output']
            
            prompt_ids = tokenizer.encode(prompt, add_bos=True, add_eos=False)
            response_ids = tokenizer.encode(response, add_bos=False, add_eos=True)
            
            input_ids = prompt_ids + response_ids
            # 2. 构造目标标签：Prompt 区域全置为 -100，Response 区域保留原词！
            labels = [-100] * len(prompt_ids) + list(response_ids)
            
            # 3. 截断至最大长度
            if len(input_ids) > max_seq_len:
                input_ids = input_ids[:max_seq_len]
                labels = labels[:max_seq_len]
                
            self.samples.append((input_ids, labels))

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, idx):
        input_ids, labels = self.samples[idx]
        return torch.tensor(input_ids, dtype=torch.long), torch.tensor(labels, dtype=torch.long)

7.5.2 Dynamic Padding and Collate Function

Because poem lengths vary, collating batches requires dynamic padding:

• Padding input_ids: Fill with <pad> or <eos> token IDs;
• Padding labels: Must fill strictly with -100! Padding tokens must never generate gradient updates!

def sft_collate_fn(batch, pad_token_id=0):
    input_ids_list, labels_list = zip(*batch)
    max_len = max(len(x) for x in input_ids_list)
    
    batch_input_ids = []
    batch_labels = []
    
    for input_ids, labels in zip(input_ids_list, labels_list):
        pad_len = max_len - len(input_ids)
        # 输入张量用 pad_token 补齐
        batch_input_ids.append(torch.cat([input_ids, torch.full((pad_len,), pad_token_id, dtype=torch.long)]))
        # 标签张量必须用 -100 补齐！
        batch_labels.append(torch.cat([labels, torch.full((pad_len,), -100, dtype=torch.long)]))
        
    return torch.stack(batch_input_ids), torch.stack(batch_labels)

7.6 Golden Rules of SFT Alignment (Hyperparameters & Pitfalls)

Compared to multi-thousand-step pretraining, SFT follows distinctly different physical laws:

| Hyperparameter / Metric | Pre-training Phase | SFT Alignment Phase | Mentor Engineering Justification | | Initial Weights | Pure random initialization (Gaussian distribution) | Must load pre-trained checkpoint | Never train from scratch! SFT merely teaches conversational etiquette to a model that already understands language. | | Base Learning Rate | Moderate: $3\times 10^{-4} \sim 5\times 10^{-4}$ | Very Small: $2\times 10^{-5} \sim 5\times 10^{-5}$ | Excessive learning rates trigger Catastrophic Forgetting, erasing the linguistic common sense acquired in pretraining! | | Training Steps & Time | Thousands of steps (13 Epochs) | Hundreds of steps (usually 200 ~ 500 steps) | A 0.04B model completes alignment in just 8 ~ 12 minutes! Over-training risks rigid repetitive recitation. | | Loss Trajectory | Plummets from 8.3+ to 2.02.5 | Decreases gently from 2.5 to 1.5~1.8 | Measures precision in outputting target responses given clear user instructions. |

7.6.2 Industrial Trap: Synthetic Template Poisoning and the LIMA Principle ⭐

In post-training, amateurs expanding synthetic instruction datasets often write simplistic random slot scripts:

# ❌ 反面教材：充满机械套话与分布漂移的随机插槽
GREETINGS = ["AI助手，", "劳驾，", "大师，", "诗仙，", "请问你能否"]
SUFFIXES = ["，谢谢！", "？", "。"]
# 生成结果形如：“AI助手，吟诵一首以‘阌乡寓居木塔偶题’为主题的七言律诗，谢谢！”

🚨 This artificial boilerplate inflicts fatal template poisoning on compact models:

• Severe Test-Time Distribution Drift (OOD):
• Real human questions are concise: "Write a five-character poem in Li Bai's style", "Write a farewell seven-character verse";
• If training data is saturated with verbose formalities, the model stumbles on direct prompts lacking filler trigger words.
• Cross-Domain Tokens Corrupting Classical Context:
• Our 4096 classical poetry vocabulary does not contain modern English AI; it gets chopped into disjoint byte tokens that corrupt classical embeddings.
• Conflicting Metric Genre Labels:
• Mislabelling a five-character quatrain with a "seven-character" prompt sends conflicting length signals, destroying rhythmic intuition.

💡 Meta LIMA Principle (Less Is More for Alignment):

Quality vastly outweighs quantity in alignment! A few thousand clean, natural, diverse golden instructions far surpass tens of thousands of rigid robotic templates.

In scripts/build_sft_data.py, we rebuilt the instruction generator:

• Natural Conversational Imperatives (35%): Write a five-character Deng Guanque Lou, Compose a poem on "Spring Snow";
• Poet Stylistic Emulation (25%): Write a five-character poem in Li Bai's style, Emulate Du Fu's poetic tone;
• Realistic Thematic Imagery (20%): Adaptively extract themes (Farewell, Homesickness, Spring Snow, Bright Moon) reflecting natural speech;
• Strict Metric Consistency: Dynamically detect five-character vs. seven-character meters to ensure 100% prompt-response alignment.

7.7 Empirical Evaluation: Behavioral Evolution from Base to SFT Model

After 300 steps of SFT, you witness the defining breakthrough of modern NLP—the model begins following human intent!

===================================================================================
                       同一提示词下：基座模型 vs SFT 模型的真实表现
===================================================================================

用户输入Prompt:
  "请帮我模仿李白的风格，写一首关于秋天喝酒的五言绝句。"

【阶段一：预训练 Base 模型输出】
  "请帮我模仿李白的风格，写一首关于秋天喝酒的五言绝句。李白字太白，号青莲居士，
  唐代伟大的浪漫主义诗人，被后人誉为诗仙。他的代表作有《望庐山瀑布》《行路难》
  《蜀道难》……（无休止百度百科式胡话）"

【阶段二：SFT 指令微调模型输出】
  "秋风吹落叶，把酒对青山。
  举盏邀明月，浮生尽醉欢。<eos>
  [模型成功击中结束符，优雅停机]"

The model no longer rambles or repeats questions, but:

• Recognizes that the user is requesting a poem;
• Identifies the style as "Li Bai", theme as "autumn and wine", and meter as "five-character quatrain";
• Emits 20 structured, antithetical characters, terminating cleanly with <eos>!

7.8 Summary and Forward Outlook

You have conquered the first two stages of the LLM three-stage rocket:

• Chapters 01~06 (Stage 1: Pre-training): Built the engine from scratch, absorbing linguistic likelihoods into a foundation model;
• Chapter 07 (Stage 2: SFT): Applied Prompt Masking and instruction alignment, transforming an unsupervised continuation generator into an obedient assistant.

Now a realistic engineering question confronts us:

What if we wish to fine-tune modern foundation models like Qwen-2.5 with larger parameter counts (1.5B, 7B) on consumer GPUs without running out of memory?

In the next chapter, we dive into the crowning jewel of parameter-efficient adaptation—PEFT & Handcrafted LoRA, empowering you to adapt billion-parameter foundation models on a single consumer GPU! 🚀