## Flowchart: Model Alignment Pipeline with Data Processing ### Overview The image depicts a technical flowchart illustrating a model alignment pipeline that integrates supervised fine-tuning (SFT), direct preference optimization (DPO), and data processing components. The diagram emphasizes iterative refinement through data augmentation, rejection sampling, and rule-based filtering, culminating in an "Aligned Ling" output. ### Components/Axes **Legend (Right Panel):** - **Model pipeline**: Orange boxes (SFT, DPO, Aligned Ling) - **Data pipeline**: Blue arrows and processes - **Model**: Orange rectangles - **Dataset**: Gray rectangles - **Data process**: Light blue rectangles **Key Components:** 1. **Post-training Phase** (Orange box at top) - Contains SFT and DPO subcomponents 2. **Pre-trained Checkpoint** (Orange box, leftmost) - Initial model state 3. **Data Processing Loop** (Blue arrows/cycles) - Data deduplication → Data Augmentation & Synthesis → Rejection Sampling → Rule-based Filtering → LLM-based Judge 4. **Data Sources**: - SFT Data (gray box) - Preference Data (gray box) 5. **Output**: - Best aligned policy (orange box) - Aligned Ling (orange box, final output) **Flow Direction:** - Left-to-right progression from Pre-trained Checkpoint to Aligned Ling - Circular feedback loop for data refinement - Vertical connections between data processes and model components ### Detailed Analysis **Model Pipeline (Orange):** - **SFT (Supervised Fine-Tuning)**: Receives input from Pre-trained Checkpoint - **DPO (Direct Preference Optimization)**: Receives input from SFT - **Aligned Ling**: Final output combining SFT and DPO outputs **Data Pipeline (Blue):** 1. **Data Deduplication**: Initial processing step 2. **Data Augmentation & Synthesis**: Generates code/math/logical reasoning data 3. **Rejection Sampling**: Filters low-quality outputs 4. **Rule-based Filtering**: Applies predefined criteria 5. **LLM-based Judge**: Evaluates data quality using large language models **Dataset Connections:** - SFT Data feeds into Data Augmentation - Preference Data connects to DPO - Both datasets cycle through the data processing loop ### Key Observations 1. **Iterative Refinement**: The circular data processing loop suggests continuous improvement of training data quality 2. **Hybrid Approach**: Combines traditional SFT/DPO with modern data engineering techniques 3. **Quality Control**: Multiple filtering stages (rejection sampling + rule-based + LLM judge) ensure high-quality training data 4. **Policy Optimization**: The "Best aligned policy" emerges from integrating SFT/DPO with refined data ### Interpretation This pipeline demonstrates a sophisticated approach to model alignment that goes beyond standard SFT/DPO by: 1. **Enhancing Data Quality**: The multi-stage data processing ensures training data is both diverse (through augmentation) and high-quality (through multiple filtering stages) 2. **Balancing Approaches**: Combines rule-based filtering (deterministic) with LLM-based judging (context-aware) for comprehensive data evaluation 3. **Policy Optimization**: The final "Best aligned policy" likely represents a Pareto-optimal solution balancing multiple alignment objectives 4. **Iterative Nature**: The circular data flow implies continuous model improvement through refined training data The diagram suggests this pipeline could be particularly effective for complex alignment tasks requiring both technical precision (code/math) and logical reasoning capabilities.