## Diagram: Hybrid Neural Network Inference Process with Crossbar Arrays and Off-Chip DRAM
### Overview
The diagram illustrates a two-phase hybrid approach for neural network inference, combining static and dynamic operations. It emphasizes time savings by optimizing data movement between on-chip crossbar arrays and off-chip DRAM. The process involves matrix multiplications, softmax scoring, and result computation, with a focus on reducing latency through on-chip computation.
### Components/Axes
- **Static Phase (a)**:
- **Components**: Off-Chip DRAM, Crossbar Arrays (W_Q, W_K, W_V), Store/Fetch operations.
- **Flow**: Weights (W_Q, W_K, W_V) are statically stored in DRAM once. Input X is multiplied by weight transposes (W_Q^T, W_K^T, W_V^T) to compute Q, K, V. K^T is fetched from DRAM for dynamic computation of Out = Q·K^T. Softmax scoring and result computation (Score·V) follow.
- **Dynamic Phase (b)**:
- **Components**: Crossbar Arrays (W_Q), on-chip operations (R1, R2, Scaling, Softmax).
- **Flow**: Input X is multiplied by W_Q^T with scaling (1/√d_k). R1 and R2 are computed via crossbar reads (FeFET/Trilinear). Softmax (LUT-based) and result computation (Score·W_V^T) occur on-chip, avoiding DRAM access for K^T.
- **Timeline**:
- **Color-coded blocks** represent operations:
- **Blue**: Crossbar Write (static phase).
- **Green**: Crossbar Read (Regular/FeFET).
- **Orange**: Crossbar Read (Trilinear/FeFET).
- **Yellow**: Scaling (Digital).
- **Purple**: Softmax (LUT-Digital).
- **Gray**: Off-Chip DRAM.
### Detailed Analysis
1. **Static Phase (a)**:
- **Weights (W_Q, W_K, W_V)**: Stored once in Off-Chip DRAM (gray).
- **Operations**:
- Q = X·W_Q^T, K = X·W_K^T, V = X·W_V^T (green crossbar reads).
- Out = Q·K^T (requires fetching K^T from DRAM via red arrows).
- Score = Softmax(Out/√d_k) (purple).
- Result = Score·V (green).
2. **Dynamic Phase (b)**:
- **On-Chip Computations**:
- R1 = X·W_Q^T·(1/√d_k) (orange crossbar read with scaling).
- R2 = R1·W_K·X^T (orange crossbar read).
- Score = Softmax(R2) (purple LUT).
- Result = Score·W_V^T (orange crossbar read).
- **Time Saved**: Red dashed arrow highlights reduced latency by avoiding DRAM access for K^T.
### Key Observations
- **Time Efficiency**: Dynamic phase eliminates repeated DRAM fetches for K^T, saving time (red arrow).
- **Hybrid Architecture**: Combines static DRAM storage (for weights) with dynamic on-chip computation (crossbar arrays).
- **FeFET Integration**: Trilinear/FeFET crossbar reads (orange) enable analog computations, reducing digital overhead.
- **Scaling**: Digital scaling (1/√d_k) is applied during crossbar reads to maintain numerical stability.
### Interpretation
The diagram demonstrates a **memory-efficient neural network inference pipeline** optimized for low latency. By offloading initial weight storage to DRAM (static phase) and leveraging on-chip crossbar arrays for repeated computations (dynamic phase), the system minimizes data movement bottlenecks. The use of FeFET-based crossbar reads and LUT-based softmax further enhances energy efficiency. This hybrid approach is critical for real-time applications like edge AI, where balancing computational speed and memory access costs is paramount. The "Time Saved" annotation underscores the architectural advantage of avoiding redundant DRAM accesses in the dynamic phase.
