# Technical Document Extraction: Computation and Communication Timeline
## 1. Document Overview
This image is a horizontal timeline diagram illustrating the interleaving of **Computation** and **Communication** tasks over **Time**. It specifically details the execution flow of a neural network layer (likely a Transformer block, given the MLP and ATTN labels) during distributed training.
## 2. Legend and Symbols
The diagram uses specific symbols to denote the phase of the data chunk:
* **$\triangle$ (White Triangle):** Forward chunk
* **$\blacktriangle$ (Black Triangle):** Backward chunk
## 3. Component Isolation
### A. Y-Axis Labels (Left Side)
* **Computation:** The top row of the timeline, representing processing tasks.
* **Communication:** The bottom row of the timeline, representing data transfer tasks.
### B. X-Axis (Bottom)
* **Time:** Indicated by a label and a right-pointing arrow ($\rightarrow$), showing that the sequence progresses from left to right.
### C. Color Coding
* **Lime Green:** Backward-related computation/communication.
* **Light Blue:** Weight-related computation.
* **Orange/Yellow:** Forward-related computation/communication.
* **Purple:** Pipeline Parallelism (PP) overhead.
* **Red (Vertical Bars):** Synchronization points or barriers.
---
## 4. Sequence of Operations (Timeline Flow)
The timeline is divided into two main phases: the **MLP (Multi-Layer Perceptron)** phase and the **ATTN (Attention)** phase, separated by a red synchronization bar.
### Phase 1: MLP Operations
| Row | Task Label | Symbol | Color | Description |
| :--- | :--- | :--- | :--- | :--- |
| **Computation** | MLP(B) | $\blacktriangle$ | Lime Green | Backward pass computation for MLP. |
| **Computation** | MLP(W) | $\blacktriangle$ | Light Blue | Weight gradient computation for MLP. |
| **Computation** | MLP(F) | $\triangle$ | Orange | Forward pass computation for MLP. |
| **Communication** | DISPATCH(F) | $\triangle$ | Orange | Forward dispatch communication (overlaps with MLP(B) and MLP(W)). |
| **Communication** | DISPATCH(B) | $\blacktriangle$ | Lime Green | Backward dispatch communication (overlaps with MLP(F)). |
### Phase 2: ATTN Operations
| Row | Task Label | Symbol | Color | Description |
| :--- | :--- | :--- | :--- | :--- |
| **Computation** | ATTN(B) | $\blacktriangle$ | Lime Green | Backward pass computation for Attention. |
| **Computation** | ATTN(W) | $\blacktriangle$ | Light Blue | Weight gradient computation for Attention. |
| **Computation** | ATTN(F) | $\triangle$ | Orange | Forward pass computation for Attention. |
| **Communication** | COMBINE(F) | $\triangle$ | Orange | Forward combine communication (overlaps with ATTN(B)). |
| **Communication** | PP | N/A | Purple | Pipeline Parallelism communication/stall. |
| **Communication** | COMBINE(B) | $\blacktriangle$ | Lime Green | Backward combine communication (overlaps with ATTN(W) and ATTN(F)). |
---
## 5. Key Trends and Technical Observations
* **Concurrency/Overlapping:** The diagram demonstrates a high degree of overlap between computation and communication. For example, while the system is performing `MLP(F)` (Forward computation), it is simultaneously performing `DISPATCH(B)` (Backward communication).
* **Symmetry:** The structure of the MLP phase and the ATTN phase are similar, both involving a sequence of Backward (B), Weight (W), and Forward (F) computations.
* **Synchronization:** Red vertical bars indicate strict boundaries where computation and communication must synchronize before proceeding to the next major block (e.g., between MLP and ATTN).
* **Pipeline Overhead:** The `PP` (Purple) block in the communication row represents a specific interval where pipeline-related communication occurs, notably occurring between the Forward and Backward combine steps in the Attention phase.
