## Training Data Flow Diagram ### Overview The image is a diagram illustrating the data flow and stages of a training process, likely for a large language model (LLM). It outlines four distinct phases: Text Pre-training, ViT Training, Joint Pre-training, Joint Cooldown, and Joint Long-context. Each phase is represented by a blue rounded rectangle containing information about the data used. The diagram also indicates the resumption of the learning rate (LR) scheduler between certain phases. ### Components/Axes * **Blue Rounded Rectangles:** Represent the different training phases. * **Text within Rectangles:** Describes the phase name, the amount of data used (in terabytes), and additional details about the data or training process. * **Green Arrows:** Indicate the flow of the training process and the resumption of the LR scheduler. ### Detailed Analysis or ### Content Details **1. Text Pre-training:** * Data: 5.2T data * Data Type: Pure Text Data **2. ViT Training:** * Data: 2.0T -> 0.1T data * Details: CoCa-loss with tiny language decoder -> align to LLM **3. Joint Pre-training:** * Data: 1.4T data * Details: Up to 40% Multimodal Data, Progressive Multimodal Ratio * Arrow: A green arrow indicates that the LR scheduler resumes after this phase. **4. Joint Cooldown:** * Data: 0.6T data * Details: High-quality Text & Multimodal Data, Re-warmup to higher LR **5. Joint Long-context:** * Data: 0.3T data * Details: Long Text & Long Video & Long Doc, RoPE base: 50,000 -> 800,000 * Arrow: A green arrow indicates that the LR scheduler resumes after this phase. ### Key Observations * The amount of data used decreases as the training progresses from Text Pre-training (5.2T) to Joint Long-context (0.3T). * The training process transitions from pure text data to multimodal data. * The ViT Training phase significantly reduces the amount of data used (2.0T -> 0.1T). * The LR scheduler is resumed after the Joint Pre-training and Joint Long-context phases. ### Interpretation The diagram illustrates a multi-stage training process for a model, likely a large multimodal model. The initial phase focuses on pre-training with a large amount of pure text data. Subsequent phases incorporate multimodal data and fine-tune the model for specific tasks or contexts, such as long-context understanding. The reduction in data size and the use of techniques like CoCa-loss and RoPE suggest a focus on efficiency and specialized training as the process evolves. The resumption of the LR scheduler indicates adjustments to the learning rate during training, likely to optimize convergence and performance. The progression from pure text to multimodal data suggests an effort to build a model capable of processing and understanding diverse types of information.

\n ## Diagram: Training Pipeline Stages ### Overview The image depicts a sequential training pipeline consisting of four main stages: Text Pre-training, Joint Pre-training, Joint Cooldown, and Joint Long-context. Below these stages is a separate stage for Vision Transformer (ViT) Training. Each stage is represented by a colored rectangle containing information about the data used, the training process, and relevant parameters. Arrows indicate the flow of the training process. ### Components/Axes The diagram consists of five rectangular blocks arranged horizontally. Each block represents a training stage. The blocks are colored as follows: - Text Pre-training: Blue - Joint Pre-training: Green - Joint Cooldown: Yellow - Joint Long-context: Orange - ViT Training: Light Blue Each block contains text labels describing the stage, data size, and specific training details. There are also two circular icons with checkmarks and text "resumes LR scheduler" positioned above the Joint Pre-training and Joint Long-context stages. ### Detailed Analysis or Content Details **1. Text Pre-training (Blue)** - Data: 5.2T data - Data Type: Pure Text Data **2. Joint Pre-training (Green)** - Data: 1.4T data - Data Composition: Up to 40% Multimodal Data - Training Approach: Progressive Multimodal Ratio - Icon: "resumes LR scheduler" (top-left of the block) **3. Joint Cooldown (Yellow)** - Data: 0.6T data - Data Quality: High-quality Text & Multimodal Data - Training Approach: Re-warmup to higher LR **4. Joint Long-context (Orange)** - Data: 0.3T data - Data Type: Long Text & Long Video & Long Doc - Parameter: RoPE base: 50,000 -> 800,000 - Icon: "resumes LR scheduler" (top-left of the block) **5. ViT Training (Light Blue)** - Data: 0.0T -> 0.1T data - Training Method: CoCa-loss with tiny language decoder -> align to LLM ### Key Observations - The data size decreases as the training progresses from Text Pre-training to Joint Long-context. - The training process transitions from pure text data to increasingly multimodal data. - The "resumes LR scheduler" icon suggests a learning rate scheduling strategy is employed in the Joint Pre-training and Joint Long-context stages. - The ViT training is a separate process, potentially running concurrently or as a pre-processing step for the multimodal data. - The RoPE base parameter in the Joint Long-context stage indicates a focus on handling long sequences. ### Interpretation This diagram illustrates a multi-stage training pipeline for a large language model (LLM) that incorporates vision capabilities. The pipeline begins with pre-training on a massive corpus of text data, then gradually introduces multimodal data (images, videos, documents) during the Joint Pre-training phase. The Joint Cooldown stage likely fine-tunes the model after the initial multimodal pre-training. Finally, the Joint Long-context stage focuses on extending the model's ability to process long sequences, potentially using techniques like RoPE (Rotary Positional Embedding). The separate ViT training suggests that visual features are extracted using a Vision Transformer and then integrated into the LLM. The decreasing data size across stages could indicate a focus on higher-quality data or more efficient training methods in later stages. The "resumes LR scheduler" icon suggests a dynamic learning rate adjustment strategy to optimize training performance. The overall pipeline aims to create a powerful multimodal LLM capable of understanding and generating both text and visual content.

## Diagram: Multimodal AI Model Training Pipeline ### Overview The image is a horizontal flowchart illustrating a multi-stage training pipeline for a multimodal AI model. The process flows from left to right, beginning with two parallel initial training phases that converge into a joint training sequence. The diagram uses color-coded blocks (shades of blue) and annotated green arrows to depict stages, data volumes, and key procedural notes. ### Components/Axes The diagram consists of four primary rectangular blocks arranged horizontally, with one additional block stacked vertically on the far left. Green curved arrows with text annotations connect specific stages. **Block 1 (Top-Left):** * **Title:** Text Pre-training * **Data Volume:** 5.2T data * **Description:** Pure Text Data **Block 2 (Bottom-Left, stacked below Block 1):** * **Title:** ViT Training * **Data Volume:** 2.0T -> 0.1T data * **Description:** CoCa-loss with tiny language decoder -> align to LLM **Block 3 (Center-Left):** * **Title:** Joint Pre-training * **Data Volume:** 1.4T data * **Description:** Up to 40% Multimodal Data / Progressive Multimodal Ratio **Block 4 (Center-Right):** * **Title:** Joint Cooldown * **Data Volume:** 0.6T data * **Description:** High-quality Text & Multimodal Data / Re-warmup to higher LR **Block 5 (Far-Right):** * **Title:** Joint Long-context * **Data Volume:** 0.3T data * **Description:** Long Text & Long Video & Long Doc / RoPE base: 50,000 -> 800,000 **Connecting Elements:** * **Arrow 1:** A green, curved arrow originates from the top-right corner of the "Text Pre-training" block and points to the top-left corner of the "Joint Pre-training" block. The text above the arrow reads: `resumes LR scheduler`. * **Arrow 2:** A green, curved arrow originates from the top-right corner of the "Joint Pre-training" block and points to the top-left corner of the "Joint Cooldown" block. The text above the arrow reads: `resumes LR scheduler`. ### Detailed Analysis The pipeline describes a sequential training regimen with distinct phases characterized by data type, volume, and learning rate (LR) schedule. 1. **Initial Parallel Phase:** * **Text Pre-training:** This is the largest single data phase, using 5.2 trillion (`5.2T`) tokens of pure text data. * **ViT Training:** This phase shows a data reduction, starting with 2.0 trillion (`2.0T`) tokens and ending with 0.1 trillion (`0.1T`) tokens. It uses a CoCa-loss function with a tiny language decoder, with the explicit goal to "align to LLM." 2. **Joint Training Sequence:** The outputs of the initial phases feed into a joint training sequence. * **Joint Pre-training:** Uses 1.4 trillion (`1.4T`) data tokens. The multimodal data ratio is not fixed; it increases progressively up to a maximum of 40%. * **Joint Cooldown:** Uses a smaller, curated dataset of 0.6 trillion (`0.6T`) tokens described as "High-quality Text & Multimodal Data." A key procedural step is a "Re-warmup to higher LR," indicating a deliberate adjustment of the learning rate schedule. * **Joint Long-context:** The final phase uses the smallest dataset of 0.3 trillion (`0.3T`) tokens. It focuses on extending the model's context window for "Long Text & Long Video & Long Doc." A technical specification notes the RoPE (Rotary Positional Embedding) base is increased from 50,000 to 800,000. ### Key Observations * **Data Volume Trend:** The total data volume decreases significantly across the joint training phases (1.4T -> 0.6T -> 0.3T), suggesting a shift from broad pre-training to specialized fine-tuning. * **Learning Rate (LR) Management:** The LR scheduler is explicitly "resumed" when transitioning from the initial text pre-training to joint pre-training, and again from joint pre-training to joint cooldown. The cooldown phase itself involves a "re-warmup to a higher LR," indicating active and nuanced management of this hyperparameter. * **Specialization of Phases:** Each joint phase has a clear, distinct purpose: general multimodal integration (Pre-training), quality refinement (Cooldown), and context window extension (Long-context). * **Architectural Alignment:** The ViT (Vision Transformer) training phase has the explicit goal of aligning its output to the LLM (Large Language Model), which is a critical step for effective multimodal fusion. ### Interpretation This diagram outlines a sophisticated, staged approach to building a capable multimodal AI. The process begins by separately establishing strong unimodal foundations in text (LLM) and vision (ViT). The critical "alignment" step in ViT training ensures the vision encoder's output is compatible with the language model's representation space. The subsequent joint phases represent a deliberate curriculum. The model first learns to process mixed text and image/video data (Joint Pre-training). It then refines this ability on a smaller, higher-quality dataset while adjusting the learning rate to escape potential local minima (Joint Cooldown). Finally, it specializes in handling very long sequences of text and visual data, which is essential for understanding documents, videos, and complex narratives (Joint Long-context). The progressive increase in multimodal data ratio and the final extension of the RoPE base are technical strategies to efficiently build a model that is not just multimodal, but also capable of deep, long-form reasoning across modalities. The decreasing data volumes across joint stages imply a focus on precision and specialization over raw scale as the model matures.

## Flowchart: Multimodal Model Training Pipeline ### Overview The image depicts a four-stage training pipeline for a multimodal AI model, visualized as a horizontal flowchart with blue rectangles connected by green arrows. Each stage represents a distinct training phase with specific data requirements and objectives. The flow progresses from left to right, with green arrows labeled "resumes LR scheduler" indicating iterative optimization steps. ### Components/Axes 1. **Stages (Left to Right):** - Text Pre-training - ViT Training - Joint Pre-training - Joint Cooldown - Joint Long-context 2. **Data Sizes:** Expressed in terabytes (T), with ranges (e.g., 2.0T -> 0.1T) indicating progressive reduction. 3. **Key Attributes:** - Data type (text, multimodal, long-context) - Technical specifications (e.g., CoCa-loss, RoPE base) - Optimization strategies (e.g., LR scheduler resumption) ### Detailed Analysis 1. **Text Pre-training** - **Data:** 5.2T pure text - **Purpose:** Foundational language understanding - **Color:** Light blue 2. **ViT Training** - **Data:** 2.0T -> 0.1T (text-to-image alignment) - **Technique:** CoCa-loss with tiny language decoder - **Objective:** Align vision-language representations - **Color:** Dark blue 3. **Joint Pre-training** - **Data:** 1.4T (40% multimodal) - **Feature:** Progressive multimodal ratio - **Color:** Medium blue 4. **Joint Cooldown** - **Data:** 0.6T (high-quality text/multimodal) - **Strategy:** Re-warmup to higher learning rate (LR) - **Color:** Dark blue 5. **Joint Long-context** - **Data:** 0.3T (long text/video/docs) - **Technical:** RoPE base increased from 50k -> 800k - **Color:** Light blue ### Key Observations - **Data Progression:** Total data decreases from 5.2T to 0.3T, but complexity increases (text → multimodal → long-context). - **Optimization Cycles:** LR scheduler resumption between stages suggests iterative refinement. - **Multimodal Focus:** 40% multimodal data in Joint Pre-training highlights hybrid training emphasis. - **Context Scaling:** RoPE base expansion (50k → 800k) indicates specialized long-sequence handling. ### Interpretation This pipeline demonstrates a staged approach to multimodal model development: 1. **Foundation Building:** Start with massive text pre-training (5.2T) to establish language understanding. 2. **Vision Integration:** Use ViT training (2.0T→0.1T) with CoCa-loss to bridge text-image gaps. 3. **Hybrid Optimization:** Joint Pre-training (1.4T) introduces multimodal data with progressive ratio adjustments. 4. **Efficiency Phase:** Cooldown (0.6T) focuses on high-quality data while re-warming LR for stability. 5. **Specialization:** Final stage (0.3T) targets long-context understanding via RoPE base expansion. The decreasing data sizes with increasing technical complexity suggest a shift from broad data collection to targeted, high-quality training. The LR scheduler resumption between stages implies careful learning rate management to avoid overfitting while maintaining model adaptability. The 40% multimodal ratio in Joint Pre-training indicates a balanced approach to hybrid training before final specialization in long-context scenarios.