# Technical Document Extraction: Distributed Training Process Diagram
## Legend
- **Colors/Labels**:
- Dark Green: `Parameters not updated`
- Light Green: `Parameters updated`
- **Placement**: Top-left corner of the diagram.
## Key Components
1. **Devices**:
- **Device 1**:
- `Forward` (Blue)
- `Backward` (Dark Green)
- `Optimizer Step` (Orange)
- `Forward` (Blue)
- **Device 2**:
- `Forward` (Blue)
- `Backward` (Dark Green)
- `Optimizer Step` (Orange)
- `Forward` (Blue)
- **Device 3**:
- `Forward` (Blue) (only visible in failure scenario)
2. **Controller**:
- Central red oval labeled `Controller`.
- Handles:
- `Gradient Synchronization` (Red text)
- `Barrier` (Red text)
- `Stop+Clean+Reset` (White text)
3. **Failure Handling**:
- Triggered during `i_th step` (subscript notation).
- Actions:
- `Restoration+Rollback` (Black text)
- Leads to `(i+1)_th step` (subscript notation).
## Flow Description
1. **Training Cycle**:
- Devices alternate between `Forward` (blue) and `Backward` (dark green) phases.
- Parameters are **not updated** during Forward/Backward (dark green).
- `Optimizer Step` (orange) updates parameters (light green).
2. **Synchronization**:
- `Gradient Synchronization` and `Barrier` occur between devices before the Optimizer Step.
3. **Failure Scenario**:
- If a failure occurs during the `i_th step` (Device 3):
- Controller initiates `Stop+Clean+Reset`.
- Devices roll back to the previous state (`Restoration+Rollback`).
- Resumes at `(i+1)_th step`.
## Spatial Grounding
- **Legend**: Top-left corner (`x=0, y=0` relative to diagram origin).
- **Controller**: Central position, connecting all devices.
- **Failure Path**: Right side of the diagram, branching from Device 3.
## Notes
- No numerical data or charts present; the diagram is a schematic representation of a distributed training workflow.
- All text is in English; no additional languages detected.
