A purely pre-trained base model is merely a statistical completion engine. Prompted with “What is the capital of France?”, a base model may respond with “What is the capital of Spain? What is the capital of Italy?”, continuing the pattern of a geographic quiz. Supervised Fine-Tuning (SFT) and Multi-Task Instruction Tuning bridge the chasm between raw text prediction and user alignment, imbuing the model with conversational protocols, instruction following, and role discipline.
1. The ChatML Standard & Structured Delimiters
To prevent malicious prompt injections and distinguish user commands from system directives, conversational training wraps plain text in structured markup languages such as ChatML:
<|im_start|>system
You are a helpful mathematical assistant.<|im_end|>
<|im_start|>user
Solve $x^2 - 4x + 4 = 0$.<|im_end|>
<|im_start|>assistant
$(x - 2)^2 = 0 implies x = 2$.<|im_end|>Special reserved tokens (<|im_start|>, <|im_end|>) are assigned dedicated embedding IDs and prevented from being generated by raw user prompts, establishing immutable permission boundaries between roles.
2. Prompt-Response Loss Masking
A fatal pitfall during supervised fine-tuning is calculating loss over the entire sequence. The model should never be penalized for the user’s prompt; it must optimize exclusively for generating the target response.
Given sequence tokens $x_1, dots, x_N$, where tokens $1 dots T$ represent the system prompt and user instruction, and tokens $T+1 dots N$ represent the target assistant response, the masked SFT loss is formulated as:
$mathcal{L}_{text{SFT}}(theta) = – sum_{t=T+1}^N log P_theta(x_t mid x_1, dots, x_{t-1})$
The cross-entropy loss for tokens $t le T$ is zeroed out by applying a label mask $M_t = mathbb{I}(t > T)$. Backpropagation gradients flow exclusively through the assistant’s reasoning and answer tokens.
3. Multi-Task Instruction Generalization
SFT transforms language models through multi-task diversity rather than mere scale (Chung et al., 2022, Flan). By training on thousands of diverse tasks—spanning code synthesis, logical deduction, creative writing, summarization, and translation—the model learns to map instruction semantics into latent function execution. Even zero-shot tasks never encountered during training trigger appropriate response patterns due to the shared compositional manifold established during pre-training.
