## Mathematical Expression Table: RWKV Mechanisms (RWKV-4, Eagle, Finch)
### Overview
The image contains three technical tables describing mathematical operations for different recurrent neural network mechanisms (RWKV-4, Eagle/RWKV-5, Finch/RWKV-6). Each table defines parameter relationships across time steps (t=0 to t=3) with matrix operations, element-wise products, and diagonal matrix manipulations. The tables use LaTeX notation for mathematical expressions.
### Components/Axes
- **Primary Table Structure**:
- **Left Column**: Time index `t` (0-3)
- **Right Column**: Mechanism-specific expressions
- **Key Variables**:
- `u, w, k_t, v_t`: Matrix/vector parameters (RWKV-4: `ℝ^D`, Eagle/Finch: `ℝ^64×64`)
- `σ()`: Non-linear activation function
- `diag()`: Diagonal matrix operation
- `⊗`: Element-wise product
- `·`: Matrix multiplication
- `T`: Matrix transpose
- **Head Sizes**:
- RWKV-4: Head size 1
- Eagle (RWKV-5): 64 heads, head size 64
- Finch (RWKV-6): 64 heads, head size 64
### Detailed Analysis
#### RWKV-4 Mechanism
| t | Expression |
|----|-------------------------------------------------------------------------------------------------------------------------------------------|
| 0 | σ(r₀) ⊗ ( (u⊙k₀⊙v₀) / (u⊙k₀) ) |
| 1 | σ(r₁) ⊗ ( (u⊙k₁⊙v₁ + k₀⊙v₀) / (u⊙k₁ + k₀) ) |
| 2 | σ(r₂) ⊗ ( (u⊙k₂⊙v₂ + k₁⊙v₁ + w⊙k₀⊙v₀) / (u⊙k₂ + k₁ + w⊙k₀) ) |
| 3 | σ(r₃) ⊗ ( (u⊙k₃⊙v₃ + k₂⊙v₂ + w⊙k₁⊙v₁ + w²⊙k₀⊙v₀) / (u⊙k₃ + k₂ + w⊙k₁ + w²⊙k₀) ) |
#### Eagle (RWKV-5) Mechanism
| t | Expression |
|----|----------------------------------------------------------------------------|
| 0 | r₀ · (diag(u) · k₀ᵀ · v₀) |
| 1 | r₁ · (diag(u) · k₁ᵀ · v₁ + k₀ᵀ · v₀) |
| 2 | r₂ · (diag(u) · k₂ᵀ · v₂ + k₁ᵀ · v₁ + diag(w) · k₀ᵀ · v₀) |
| 3 | r₃ · (diag(u) · k₃ᵀ · v₃ + k₂ᵀ · v₂ + diag(w) · k₁ᵀ · v₁ + diag(w²) · k₀ᵀ · v₀) |
#### Finch (RWKV-6) Mechanism
| t | Expression |
|----|-------------------------------------------------------------------------------------------------------------------------------------------|
| 0 | r₀ · (diag(u) · k₀ᵀ · v₀) |
| 1 | r₁ · (diag(u) · k₁ᵀ · v₁ + diag(w₁) · k₀ᵀ · v₀) |
| 2 | r₂ · (diag(u) · k₂ᵀ · v₂ + diag(w₂) · k₁ᵀ · v₁ + diag(w₂⊙w₁) · k₀ᵀ · v₀) |
| 3 | r₃ · (diag(u) · k₃ᵀ · v₃ + diag(w₃) · k₂ᵀ · v₂ + diag(w₃⊙w₂) · k₁ᵀ · v₁ + diag(w₃⊙w₂⊙w₁) · k₀ᵀ · v₀) |
### Key Observations
1. **Progressive Complexity**:
- RWKV-4: Simple element-wise operations with single time-step dependencies
- Eagle: Introduces diagonal matrix operations and multi-time-step accumulation
- Finch: Adds weighted diagonal combinations and recursive dependencies
2. **Dimensionality**:
- RWKV-4 uses generic `ℝ^D` dimensions
- Eagle/Finch specify 64×64 matrices for head-based parallelism
3. **Operation Patterns**:
- All mechanisms use `diag(u)` as a base operation
- Time dependencies increase with higher `t` values
- Weight matrices (w₁, w₂, w₃) appear in later mechanisms
### Interpretation
This appears to document variations of the RWKV recurrent neural network architecture, showing how different versions (4, 5, 6) implement increasingly complex attention-like mechanisms. The progression suggests:
- **RWKV-4**: Basic temporal modeling with element-wise operations
- **Eagle (RWKV-5)**: Introduces parallel attention heads via diagonal matrices
- **Finch (RWKV-6)**: Implements multi-scale attention through weighted diagonal combinations
The mathematical patterns indicate a design philosophy of:
1. **Temporal Dependency Chaining**: Each time step incorporates information from all previous steps
2. **Weighted Accumulation**: Later mechanisms use weighted sums of past operations
3. **Matrix Decomposition**: Use of diagonal matrices for efficient computation
The increasing complexity across versions suggests an optimization for handling longer-term dependencies while maintaining computational efficiency through matrix sparsity (via diagonal operations).
