## Diagram: Training Process with DPO and Distillation ### Overview The image is a diagram illustrating a training process that incorporates Direct Preference Optimization (DPO) and token-level distillation. It shows the flow of data from a query, through student and teacher models, and into a loss function that updates the student model. ### Components/Axes * **Top-Left:** Input "Query" * **Models:** * "Student Model (NBG4-3B-SFT)" * "Teacher Model (NBG3.5-Pro)" * "Student Model (NBG4-3B)" * **Samples:** * "Negative Sample" * "Positive Sample" * **Filter:** "Pair-wise Preference Filter [Ensure Positive >>> Negative]" * **Output:** "Training Data" * **Losses (Right Side):** * "Sequence-level DPO Loss" * "Positive Token-level Distillation" * "Negative Token-level Distillation" * **Final Loss:** "Joint Loss: Ldpo + Lpos + Lneg" * **Update:** Feedback loop from "Joint Loss" to "Student Model (NBG4-3B)" ### Detailed Analysis 1. **Query Processing:** * A "Query" is input. * The query is fed into both a "Student Model (NBG4-3B-SFT)" and a "Teacher Model (NBG3.5-Pro)". * "Teacher Logits" are output from the Teacher Model and fed into the "Sequence-level DPO Loss" block. * "Negative Sample" and "Positive Sample" are generated. 2. **Preference Filtering:** * The "Negative Sample" and "Positive Sample" are passed to a "Pair-wise Preference Filter [Ensure Positive >>> Negative]". * The output of the filter is "Training Data". 3. **Loss Calculation:** * The "Training Data" is fed back into the "Student Model (NBG4-3B)". * "Student Logits" are output from the Student Model and fed into the "Positive Token-level Distillation" and "Negative Token-level Distillation" blocks. * **Sequence-level DPO Loss:** * "Positive Score" is represented by a smiling face emoji and a horizontal bar. * "Negative Score" is represented by a frowning face emoji and a horizontal bar. * The goal is to "Maximize" the "Margin" between the positive and negative scores. * **Positive Token-level Distillation:** * "Teacher Prob" is represented by a gray bell curve. * "Student Prob" is represented by a blue bell curve. * The goal is to "Minimize" the Kullback-Leibler divergence ("KL") between the teacher and student probabilities. * **Negative Token-level Distillation:** * "Teacher Prob" is represented by a gray bell curve. * "Student Prob" is represented by a blue bell curve. * The goal is to "Minimize" the Kullback-Leibler divergence ("KL") between the teacher and student probabilities. * The outputs of the three loss components ("Sequence-level DPO Loss", "Positive Token-level Distillation", and "Negative Token-level Distillation") are combined into a "Joint Loss: Ldpo + Lpos + Lneg". 4. **Model Update:** * The "Joint Loss" is used to "Update" the "Student Model (NBG4-3B)" in a feedback loop. ### Key Observations * The diagram illustrates a training process that combines DPO and token-level distillation. * The process involves a teacher model, a student model, and a preference filter. * The loss function consists of three components: sequence-level DPO loss, positive token-level distillation, and negative token-level distillation. * The joint loss is used to update the student model. ### Interpretation The diagram describes a training methodology aimed at improving the performance of a student model by leveraging both a teacher model and a preference-based filtering mechanism. The use of DPO at the sequence level encourages the model to generate outputs that are preferred over others, while token-level distillation ensures that the student model learns to mimic the behavior of the teacher model at a finer granularity. The combination of these techniques likely leads to a more robust and accurate student model. The feedback loop from the joint loss back to the student model signifies an iterative training process where the model continuously learns and improves based on the calculated loss.

\n ## Diagram: Model Training Pipeline ### Overview This diagram illustrates a model training pipeline involving a teacher model, a student model, and a preference filter, culminating in a joint loss function. The pipeline appears to be focused on refining a student model using knowledge distillation and direct preference optimization (DPO). ### Components/Axes The diagram consists of several interconnected blocks representing different stages of the training process. Key components include: * **Query:** The initial input to the system. * **Teacher Model (NBG3.5-Pro):** A pre-trained model providing guidance. * **Student Model (NBG4-3B-SFT):** The model being trained. * **Student Model (NBG4-3B):** The model being updated. * **Negative Sample:** A sample representing an undesirable output. * **Positive Sample:** A sample representing a desired output. * **Pair-wise Preference Filter:** A component ensuring the positive sample is preferred over the negative sample. * **Training Data:** The data used to update the student model. * **Teacher Logits:** The output of the teacher model. * **Student Logits:** The output of the student model. * **Sequence-level DPO Loss:** A loss function optimizing the model based on sequence-level preferences. * **Positive Token-level Distillation:** A distillation process focusing on positive samples. * **Negative Token-level Distillation:** A distillation process focusing on negative samples. * **Joint Loss:** The combined loss function. * **Update:** An arrow indicating the direction of model updates. ### Detailed Analysis or Content Details The diagram shows a flow of information starting with a "Query" which is fed into both the "Teacher Model (NBG3.5-Pro)" and the "Student Model (NBG4-3B-SFT)". The teacher model outputs "Teacher Logits". The student model generates both "Negative Sample" and "Positive Sample". These samples are then passed through a "Pair-wise Preference Filter" which ensures "Ensure Positive >> Negative". The output of this filter is "Training Data". The "Training Data" and "Teacher Logits" are then used in two distillation processes and a DPO loss calculation. * **Sequence-level DPO Loss:** This block shows a horizontal bar with "Maximize Margin" labeled above it. The bar is divided into two sections: a light gray section labeled "Positive Score" with a smiling face icon, and a dark gray section labeled "Negative Score" with a frowning face icon. * **Positive Token-level Distillation:** This block shows two overlapping bell curves. The left curve is gray and labeled "Teacher Prob". The right curve is blue and labeled "Student Prob". The space between the curves is labeled "KL". The text "Minimize" is above the block. * **Negative Token-level Distillation:** Similar to the positive distillation, this block shows two overlapping bell curves. The left curve is gray and labeled "Teacher Prob". The right curve is blue and labeled "Student Prob". The space between the curves is labeled "KL". The text "Minimize" is above the block. Finally, these three loss components are combined into a "Joint Loss" represented by the equation: `L_DPO + L_pos + L_neg`. An arrow labeled "Update" points from the "Joint Loss" block back to the "Student Model (NBG4-3B)" block. ### Key Observations The diagram highlights a multi-faceted training approach combining direct preference optimization (DPO) at the sequence level with token-level distillation for both positive and negative samples. The use of a teacher model suggests a knowledge distillation strategy. The preference filter is crucial for ensuring the training data reflects desired model behavior. ### Interpretation This diagram depicts a sophisticated model training pipeline designed to align a student model with a teacher model and human preferences. The DPO loss encourages the student model to generate outputs preferred by humans, while the distillation losses help the student model mimic the teacher model's behavior at the token level. The preference filter ensures that the training data is consistent with the desired preference ordering. The joint loss function balances these different objectives, leading to a refined student model. The use of both sequence-level and token-level approaches suggests a focus on both overall coherence and fine-grained accuracy. The model names (NBG3.5-Pro, NBG4-3B-SFT, NBG4-3B) indicate specific versions or configurations of the models involved. The diagram doesn't provide quantitative data, but rather illustrates the *process* of training. It suggests a focus on reinforcement learning from human feedback (RLHF) or a similar preference-based learning paradigm.

\n ## System Architecture Diagram: Student Model Training via Teacher Distillation and DPO ### Overview This image is a technical flowchart illustrating a machine learning training pipeline. It depicts a process where a smaller "Student Model" (NBG4-3B) is trained using outputs from a larger "Teacher Model" (NBG3.5-Pro) and preference data. The system combines Sequence-level Direct Preference Optimization (DPO) loss with token-level distillation losses to update the student model. ### Components/Axes The diagram is organized into three main regions from left to right: **Data Preparation**, **Model Forward Pass**, and **Loss Calculation & Update**. **1. Data Preparation (Left Region):** * **Input:** A "Query" box at the top-left. * **Models:** Two parallel boxes below the query: * "Student Model (NBG4-3B-SFT)" * "Teacher Model (NBG3.5-Pro)" * **Samples:** Each model generates a sample: * Student Model → "Negative Sample" * Teacher Model → "Positive Sample" * **Filtering:** Both samples feed into a "Pair-wise Preference Filter" box with the annotation: "[Ensure Positive >> Negative]". * **Output:** The filter outputs to a "Training Data" box. **2. Model Forward Pass (Center Region):** * The "Training Data" and the original "Query" are used as input to the core "Student Model (NBG4-3B)". * This model produces two outputs: * An arrow labeled "Student Logits" points to the right. * An arrow labeled "Teacher Logits" originates from the "Teacher Model (NBG3.5-Pro)" in the left region and points to the right, bypassing the student model. **3. Loss Calculation & Update (Right Region):** This region is enclosed in a dashed yellow box and contains three parallel loss computation modules, all receiving "Teacher Logits" and "Student Logits". * **Top Module: "Sequence-level DPO Loss"** * Contains two score bars: "Positive Score" (orange) and "Negative Score" (blue). * An arrow between them is labeled "Maximize Margin". * **Middle Module: "Positive Token-level Distillation"** * Shows two probability distribution curves: "Teacher Prob" (gray) and "Student Prob" (blue). * An arrow between them is labeled "Minimize KL" (Kullback-Leibler divergence). * **Bottom Module: "Negative Token-level Distillation"** * Identical structure to the middle module: "Teacher Prob" (gray) and "Student Prob" (blue) curves with a "Minimize KL" arrow. * **Final Output:** The outputs of all three modules converge into a final box labeled "Joint Loss" with the formula: `L_Dpo + L_pos + L_neg`. * **Update Loop:** An arrow labeled "Update" flows from the "Joint Loss" box back to the "Student Model (NBG4-3B)" in the center, completing the training loop. ### Detailed Analysis The diagram specifies a multi-objective training strategy: 1. **Preference Learning:** The "Pair-wise Preference Filter" ensures the training data consists of query-response pairs where the teacher's response (positive) is explicitly preferred over the student's initial response (negative). 2. **Hybrid Loss Function:** The student model is updated to minimize a composite loss: * **`L_Dpo` (Sequence-level):** Optimizes the model to assign a higher score (margin) to the preferred (positive) sequence over the dispreferred (negative) one. * **`L_pos` (Token-level):** Uses knowledge distillation (minimizing KL divergence) to make the student's token-level probability distribution for the *positive* sample match the teacher's distribution. * **`L_neg` (Token-level):** Similarly distills knowledge for the *negative* sample, aligning the student's distribution with the teacher's on the less preferred output. ### Key Observations * **Model Naming Convention:** The student model is referred to as "NBG4-3B-SFT" (likely Supervised Fine-Tuned) during data generation and as "NBG4-3B" during the core training loop, suggesting the SFT version is a starting point. * **Asymmetric Role:** The teacher model (NBG3.5-Pro) is only used to generate the positive sample and provide logits for distillation; it is not updated. * **Dual Distillation:** The system performs distillation on both positive and negative samples, which is a nuanced approach to transfer the teacher's behavior comprehensively. * **Spatial Flow:** The layout clearly separates the one-time data preparation (left) from the iterative training loop (center and right). ### Interpretation This diagram represents a sophisticated **knowledge distillation and alignment pipeline** for training a compact language model (3B parameters). The core innovation is the **joint optimization** of three distinct learning signals: 1. **Preference Alignment (DPO):** Teaches the model *what* is better (positive vs. negative responses). 2. **Behavioral Cloning (Distillation):** Teaches the model *how* the teacher thinks, by mimicking its internal probability distributions at the token level for both good and bad responses. The "Pair-wise Preference Filter" is a critical component, acting as a gatekeeper to ensure the training signal is clean and the positive sample is indeed superior. The overall goal is to produce a student model that not only prefers the teacher's outputs but also internalizes the teacher's reasoning patterns, leading to a more capable and aligned smaller model. The use of "NBG" in model names suggests this may be part of a specific model family or project.

## Diagram: Machine Learning Model Training Pipeline with Knowledge Distillation ### Overview This diagram illustrates a technical pipeline for training a student machine learning model (NBG4-3B) using knowledge distillation from a teacher model (NBG3.5-Pro). The process involves query processing, sample selection, preference filtering, logit generation, and multi-stage loss optimization to update the student model. ### Components/Axes 1. **Input/Output Flow**: - **Query**: Top-left entry point. - **Training Data**: Bottom-left output after filtering. - **Update**: Bottom-right output after loss optimization. 2. **Key Components**: - **Models**: - Student Model (NBG4-3B-SFT) - Teacher Model (NBG3.5-Pro) - Student Model (NBG4-3B) - **Samples**: - Negative Sample - Positive Sample - **Filters**: - Pair-wise Preference Filter (green box) - **Logits**: - Teacher Logits - Student Logits - **Loss Functions**: - Sequence-level DPO Loss - Positive Token-level Distillation - Negative Token-level Distillation - Joint Loss (L_DPO + L_pos + L_neg) 3. **Visual Elements**: - Arrows indicate data flow direction. - Color-coded blocks (blue for models, green for filters, orange for loss functions). - Distributions (bell curves) for token-level distillation. - Emoji-based scoring (😊 for positive, 😞 for negative). ### Detailed Analysis 1. **Query Processing**: - Queries split into **Negative Sample** (left) and **Positive Sample** (right) for both student and teacher models. 2. **Pair-wise Preference Filter**: - Ensures **Positive >> Negative** samples are prioritized for training data. 3. **Logit Generation**: - Teacher Model generates **Teacher Logits** from positive samples. - Student Model (NBG4-3B) generates **Student Logits** from positive samples. 4. **Loss Optimization**: - **Sequence-level DPO Loss**: Maximizes positive scores (😊) and minimizes negative scores (😞) with a margin. - **Token-level Distillation**: - Positive: Minimizes KL divergence between teacher and student probabilities. - Negative: Same minimization for negative samples. - **Joint Loss**: Combines DPO, positive, and negative losses for model updates. ### Key Observations - The pipeline emphasizes **positive sample prioritization** via the preference filter. - **KL divergence** is used to align student and teacher token-level probabilities. - **DPO Loss** introduces a margin-based scoring system for sequence-level optimization. - The **joint loss** integrates multiple objectives for holistic model updates. ### Interpretation This diagram represents a **knowledge distillation framework** where the student model learns from the teacher model's outputs while incorporating preference-based alignment (DPO) and token-level probability matching. The use of both sequence-level (DPO) and token-level (KL) losses suggests a multi-granularity approach to model improvement. The green "Pair-wise Preference Filter" acts as a quality control mechanism, ensuring the student model focuses on high-quality positive examples. The emoji-based scoring system visually reinforces the optimization goals, making the pipeline's objectives intuitive despite its technical complexity.