## Diagram: NVIDIA H100 GPU Architecture
### Overview
The diagram illustrates the hierarchical structure of an NVIDIA H100 GPU, showcasing its memory hierarchy, compute units, and interconnects. Key components include GPU memory, HBM3 modules, memory buses, L2 cache, streaming multiprocessors (SMs), and compute instances. The layout emphasizes spatial relationships between hardware elements and their capacities.
### Components/Axes
1. **Vertical Hierarchy**:
- **Top**: GPU Memory (96 GB) divided into six HBM3 modules (16 GB each).
- **Middle**: Memory Bus (6144 bits) with Memory Controllers (MC) labeled beneath each HBM3 module.
- **Lower Middle**: L2 Cache (60 MB total, 7.5 MB per instance) with eight L2 blocks.
- **Bottom**: SMs (Streaming Multiprocessors) arranged in compute instances, with labels for GPU Instance configurations (4g.48gb and 2g.24gb).
2. **Legend**:
- **CE**: Compute Engine (colored in light gray).
- **MC**: Memory Controller (colored in dark gray).
- **SM**: Streaming Multiprocessor (colored in white with black text).
- **Unused**: Grayed-out section labeled "Unused in this configuration."
3. **Spatial Grounding**:
- **Legend**: Positioned at the top-right corner, adjacent to the HBM3 modules.
- **Compute Instances**: Labeled at the bottom, with configurations differentiated by GPU Instance size (4g.48gb vs. 2g.24gb).
- **Unused Section**: Highlighted in the bottom-right GPU Instance (2g.24gb) with a triangular annotation.
### Detailed Analysis
1. **Memory Hierarchy**:
- **HBM3**: Six modules totaling 96 GB (16 GB each), connected via a 6144-bit memory bus.
- **L2 Cache**: Eight instances (7.5 MB each) totaling 60 MB, positioned between HBM3 and SMs.
- **SMs**: 132 total SMs (×132 on H100), organized into compute instances. Each compute instance contains 32 SMs (2c = 2 compute instances).
2. **GPU Instance Configurations**:
- **4g.48gb**: Full utilization of SMs (132 SMs).
- **2g.24gb**: Partial utilization (112 SMs active, 20 SMs grayed as "Unused").
3. **Memory Bus**:
- 6144-bit width (768 bytes per cycle) connects HBM3 modules to Memory Controllers (MC).
### Key Observations
1. **Hierarchical Scaling**:
- HBM3 modules dominate the memory hierarchy, followed by L2 cache and SMs.
- Compute instances scale with SM count, reflecting parallel processing capabilities.
2. **Unused Resources**:
- The 2g.24gb GPU Instance configuration leaves 20 SMs unused, suggesting modular scalability for different workloads.
3. **Memory Bandwidth**:
- 6144-bit bus implies high bandwidth for data transfer between HBM3 and SMs.
### Interpretation
The diagram highlights the H100's optimization for high-throughput computing. The hierarchical memory structure (HBM3 → L2 Cache → SMs) minimizes latency while maximizing bandwidth. The unused SMs in the 2g.24gb configuration suggest flexibility for smaller workloads or cost-efficient deployments. The 6144-bit memory bus underscores the GPU's ability to handle large datasets simultaneously, critical for AI/ML and HPC tasks. The spatial arrangement of components emphasizes modularity, allowing customization based on performance requirements.
