## Flowchart: Model Training Data Composition and Evolution
### Overview
The image depicts a two-phase model training pipeline with data composition changes across stages. It shows percentage distributions of data types (General, Reasoning, Context, CoT) and absolute data sizes (10T, 150B, 600B) across four key substages.
### Components/Axes
- **Main Sections**:
- **Pre-Training (4K)**: Left-side blue boxes
- **Mid-Training (32K)**: Right-side orange/red boxes
- **Substages**:
1. Pre-Training Substage 1 (68% General, 32% Reasoning)
2. Pre-Training Substage 2 (54% General, 46% Reasoning)
3. Long Context Extension (54% General, 46% Reasoning + 150B Context Data)
4. Reasoning Pre-Activation (55% General, 45% Reasoning + 600B CoT Data)
- **Data Size Labels**:
- 10T (10 trillion)
- 150B (150 billion)
- 600B (600 billion)
- **Color Coding**:
- Blue: Pre-Training
- Orange: Long Context Extension
- Red: Reasoning Pre-Activation
### Detailed Analysis
1. **Pre-Training Phase**:
- Substage 1: Dominated by General Data (68%) with Reasoning Data at 32% (10T total)
- Substage 2: Balanced shift to 54% General and 46% Reasoning (10T total)
- *Trend*: Gradual increase in Reasoning Data proportion
2. **Mid-Training Phase**:
- Long Context Extension: Maintains 54% General/46% Reasoning ratio but adds 150B Long Context Data
- Reasoning Pre-Activation: Slightly more Reasoning (45%) than General (55%) with 600B CoT Data
- *Trend*: Increased focus on specialized Reasoning data and context
### Key Observations
- **Data Size Progression**:
- Pre-Training: 10T total data
- Mid-Training: 150B + 600B = 750B additional data
- **Reasoning Data Growth**:
- Pre-Training: 32% → 46%
- Mid-Training: 46% → 45% (with CoT specialization)
- **General Data Stability**:
- Maintains ~54-55% across Mid-Training phases despite increased specialization
### Interpretation
This pipeline demonstrates a strategic shift from broad pre-training to specialized mid-training:
1. **Pre-Training**: Establishes foundational knowledge with general data dominance
2. **Mid-Training**:
- **Long Context Extension**: Enhances model's ability to handle extended reasoning chains
- **Reasoning Pre-Activation**: Focuses on chain-of-thought capabilities with massive CoT data
3. **Data Composition**: Maintains general data as a baseline while progressively increasing reasoning specialization
4. **Scale**: Shows 75x increase in data size from Pre-Training (10T) to Mid-Training (750B additional)
The progression suggests a deliberate architectural choice to balance general knowledge with specialized reasoning capabilities, using increasingly complex data types and sizes across training phases.
