2305.13048v2

# RWKV: Reinventing RNNs for the Transformer Era > Equal first authorship. Others listed alphabetically. noitemsep noitemsep ## Abstract Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence length. In contrast, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to match the same performance as Transformers due to limitations in parallelization and scalability. We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. Our approach leverages a linear attention mechanism and allows us to formulate the model as either a Transformer or an RNN, thus parallelizing computations during training and maintains constant computational and memory complexity during inference. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers, suggesting future work can leverage this architecture to create more efficient models. This work presents a significant step towards reconciling trade-offs between computational efficiency and model performance in sequence processing tasks. Code at: https://github.com/BlinkDL/RWKV-LM ## 1 Introduction Deep learning has greatly advanced artificial intelligence, impacting a range of scientific and industrial uses. These often involve complex sequential data processing tasks such as natural language understanding, conversational AI, time-series analysis, and indirectly sequential formats like images and graphs Brown et al. (2020); Ismail Fawaz et al. (2019); Wu et al. (2020); Albalak et al. (2022). Predominant among these techniques include RNNs and Transformers Vaswani et al. (2017), each with specific benefits and drawbacks. RNNs require less memory, particularly for handling long sequences. However, they suffer from the vanishing gradient problem and non-parallelizability in the time dimension during training, limiting their scalability Hochreiter (1998); Le and Zuidema (2016). <details> <summary>x1.png Details</summary>  ### Visual Description ## Line Graph: Model Accuracy vs. Compute (exaFLOP) ### Overview The image is a line graph comparing the accuracy of four language models (BLOOM, Pythia, OPT, RWKV) across varying computational resources (exaFLOP). The x-axis represents compute power on a logarithmic scale (10² to 10³ exaFLOP), while the y-axis shows accuracy (40–55). Four distinct data series are plotted with unique symbols and colors. ### Components/Axes - **X-axis (Compute)**: Labeled "Compute (exaFLOP)" with a logarithmic scale (10², 10³). - **Y-axis (Accuracy)**: Labeled "Accuracy" with a linear scale (40–55). - **Legend**: Located in the bottom-right corner, mapping: - **Orange circles**: BLOOM - **Blue squares**: Pythia - **Green triangles**: OPT - **Red stars**: RWKV ### Detailed Analysis 1. **BLOOM (Orange Circles)**: - Starts at ~42.5 accuracy at 10² exaFLOP. - Increases steadily to ~51 accuracy at 10³ exaFLOP. - Slope: Gentle upward trend. 2. **Pythia (Blue Squares)**: - Begins at ~41 accuracy at 10² exaFLOP. - Rises sharply to ~54 accuracy at 10³ exaFLOP. - Slope: Steeper than BLOOM. 3. **OPT (Green Triangles)**: - Starts at ~42 accuracy at 10² exaFLOP. - Increases steadily to ~56 accuracy at 10³ exaFLOP. - Slope: Moderate upward trend. 4. **RWKV (Red Stars)**: - Begins at ~41 accuracy at 10² exaFLOP. - Rises sharply to ~58 accuracy at 10³ exaFLOP. - Slope: Steepest among all models. ### Key Observations - **All models improve accuracy with increased compute**, but rates vary. - **RWKV** achieves the highest accuracy (~58) at 10³ exaFLOP, surpassing others. - **BLOOM** has the lowest final accuracy (~51) despite similar starting points. - **Pythia** and **RWKV** show the steepest performance gains. - At 10² exaFLOP, all models cluster between 41–42.5 accuracy. ### Interpretation The graph demonstrates a clear relationship between computational resources and model accuracy. RWKV exhibits the most efficient scaling, achieving the highest accuracy with minimal compute. Pythia and OPT follow closely, while BLOOM lags despite comparable initial performance. The logarithmic x-axis emphasizes exponential scaling benefits, suggesting that even small increases in compute (e.g., 10² to 10³ exaFLOP) yield significant accuracy improvements. This highlights the importance of resource allocation for model training and deployment. </details> Figure 1: Average performance of RWKV models compared to transformers across twelve NLP tasks. For further details, see section 5. Transformers emerged as a powerful alternative, adept at managing local and long-range dependencies and supporting parallelized training Tay et al. (2022). Models such as GPT-3 Brown et al. (2020), ChatGPT OpenAI (2022); Kocoń et al. (2023), LLaMA Touvron et al. (2023), and Chinchilla Hoffmann et al. (2022) showcase the potential of Transformers in NLP. However, the self-attention mechanism’s quadratic complexity makes it computationally and memory intensive for tasks involving long sequences and constrained resources. This has stimulated research to enhance Transformers’ scalability, sometimes sacrificing some of their effectiveness Wang et al. (2020); Zaheer et al. (2020); Dao et al. (2022a). | Transformer Reformer Performer | $O(T^2d)$ $O(T\log{T}d)$ $O(Td^2\log{d})$ | $O(T^2+Td)$ $O(T\log{T}+Td)$ $O(Td\log{d}+d^2\log{d})$ | | --- | --- | --- | | Linear Transformers | $O(Td^2)$ | $O(Td+d^2)$ | | AFT-full | $O(T^2d)$ | $O(Td)$ | | AFT-local | $O(Tsd)$ | $O(Td)$ | | MEGA | $O(cTd)$ | $O(cd)$ | | RWKV (ours) | $O(Td)$ | $O(d)$ | Table 1: Inference complexity comparison with different Transformers. Here $T$ denotes the sequence length, $d$ the feature dimension, $c$ is MEGA’s chunk size of quadratic attention, and $s$ is the size of a local window for AFT. To tackle these challenges, we introduce the Receptance Weighted Key Value (RWKV) model, combining the strengths of RNNs and Transformers while circumventing key drawbacks. RWKV alleviates memory bottleneck and quadratic scaling associated with Transformers Katharopoulos et al. (2020) with efficient linear scaling, while maintaining the expressive properties of the Transformer, such as parallelized training and robust scalability. RWKV reformulates the attention mechanism with a variant of linear attention, replacing traditional dot-product token interaction with more effective channel-directed attention. This implementation, without approximation, offers the lowest computational and memory complexity; see Table 1. The motivation behind RWKV is to balance computational efficiency with expressive capacity in neural networks. It offers a solution for handling large-scale models with billions of parameters, exhibiting competitive performance at a reduced computational cost. Experiments suggest RWKV addresses scaling and deployment challenges in AI, especially for sequential data processing, pointing towards more sustainable and efficient AI models. Our contributions in this paper are as follows: - The introduction of RWKV, a novel architecture combining RNNs and Transformer advantages while mitigating their limitations. - Detailed experiments demonstrating RWKV’s performance and efficiency on benchmark datasets for large-scale models. - The release of pretrained models, from 169 million to 14 billion parameters, trained on the Pile Gao et al. (2020); Biderman et al. (2022). https://huggingface.co/RWKV ## 2 Background Here we briefly review the fundamentals of RNNs and Transformers. ### 2.1 Recurrent Neural Networks (RNNs) Popular RNN architectures such as LSTM Hochreiter and Schmidhuber (1997) and GRU Chung et al. (2014) are characterized by the following formulation (shown for LSTM, others can be reasoned similarly): $$ \displaystyle f_t \displaystyle=σ_g(W_fx_t+U_fh_t-1+b_f), \displaystyle i_t \displaystyle=σ_g(W_ix_t+U_ih_t-1+b_i), \displaystyle o_t \displaystyle=σ_g(W_ox_t+U_oh_t-1+b_o), \displaystyle\tilde{c}_t \displaystyle=σ_c(W_cx_t+U_ch_t-1+b_c), \displaystyle c_t \displaystyle=f_t\odot c_t-1+i_t\odot\tilde{c}_t, \displaystyle h_t \displaystyle=o_t\odotσ_h(c_t). \tag{1} $$ Although RNNs can be factored into two linear blocks ( $W$ and $U$ ) and an RNN-specific block (1)–(6), as noted by Bradbury et al. (2017), the data dependency relying on previous time steps prohibits parallelizing these typical RNNs. ### 2.2 Transformers and AFT Introduced by Vaswani et al. (2017), Transformers are a class of neural networks that have become the dominant architecture for several NLP tasks. Instead of operating on sequences step-by-step like RNNs, Transformers rely on attention mechanisms to capture relationships between all input and all output tokens: $$ \displaystyleAttn(Q,K,V)=softmax(QK^⊤)V, \tag{7} $$ where the multi-headness and scaling factor $\frac{1}{√{d_k}}$ is omitted for convenience. The core $QK^⊤$ multiplication is an ensemble of pairwise attention scores between each token in a sequence, which can be decomposed as vector operations: $$ \displaystyleAttn(Q,K,V)_t \displaystyle=\frac{∑_i=1^Te^q^{⊤_tk_i}\odot v_i}{∑_i=1 ^Te^q^{⊤_tk_i}}. \tag{8} $$ AFT Zhai et al. (2021), alternately formulates $$ \displaystyleAttn^+(W,K,V)_t=\frac{∑_i=1^te^w_t,i+k_i \odot v_i}{∑_i=1^te^w_t,i+k_i}, \tag{9} $$ where $\{w_t,i\}∈ R^T× T$ is the learned pair-wise position biases, and each $w_t,i$ is a scalar. Inspired by AFT, RWKV takes a similar approach. However, for simplicity, it modifies the interaction weights so that it can be transformed into an RNN. Each $w_t,i$ in RWKV is a channel-wise time decay vector multiplied by the relative position and traced backward from current time as it decays: $$ \displaystyle w_t,i=-(t-i)w, \tag{10} $$ where $w∈(R_≥ 0)^d$ , with $d$ the number of channels. We require $w$ to be non-negative to ensure that $e^w_t,i≤ 1$ and the per-channel weights decay backwards in time. ## 3 RWKV The RWKV model architecture is defined by four fundamental elements that are intrinsic to the time-mixing and channel-mixing blocks: - $R$ : The Receptance vector acts as the receiver of past information. - $W$ : The Weight signifies the positional weight decay vector, a trainable parameter within the model. - $K$ : The Key vector performs a role analogous to $K$ in traditional attention mechanisms. - $V$ : The Value vector functions similarly to $V$ in conventional attention processes. These core elements interact multiplicatively at each timestep, as depicted in Figure 2. <details> <summary>x2.png Details</summary>  ### Visual Description ## Diagram: Neural Network Architecture with Channel and Time Mixing ### Overview The diagram illustrates a neural network architecture with two primary processing streams: **Channel Mixing** (top) and **Time Mixing** (bottom). Both streams involve parameterized operations (σ, μ), normalization (LayerNorm), and attention-like mechanisms (WKV). Arrows indicate data flow, with cross symbols (⊗) representing element-wise multiplication and plus symbols (+) representing addition. ### Components/Axes - **Channel Mixing (Top Block)**: - **σ (sigma)**: Orange square, likely a sigmoid or scaling function. - **R', K', V'**: Orange squares, representing modified "query," "key," and "value" vectors. - **μ'**: Yellow rectangle, possibly a mean or bias term. - **LayerNorm**: Yellow rectangle, layer normalization applied after mixing. - **Flow**: σ → (R', K', V') → μ' → LayerNorm. - **Time Mixing (Bottom Block)**: - **σ (sigma)**: Orange square, same as in Channel Mixing. - **R, K, V**: Orange squares, unmodified "query," "key," and "value" vectors. - **WKV**: Red rectangle, a specialized operation combining R, K, V. - **μ**: Yellow rectangle, mean or bias term. - **LayerNorm**: Yellow rectangle, normalization after mixing. - **Out**: Light blue rectangle, final output. - **Flow**: σ → (R, K, V) → WKV → μ → LayerNorm → Out. - **Legend**: - Orange: σ, R, K, V, R', K', V'. - Red: WKV. - Yellow: μ, μ', LayerNorm. - Light Blue: Out. ### Detailed Analysis 1. **Channel Mixing**: - σ modulates R', K', V' (possibly attention weights). - μ' is added to the mixed vectors, followed by LayerNorm. - Output feeds into Time Mixing via a cross symbol (⊗), suggesting element-wise interaction. 2. **Time Mixing**: - σ scales R, K, V before WKV processing. - WKV combines R, K, V into a unified representation (e.g., attention output). - μ is added, followed by LayerNorm and final output (Out). 3. **Connections**: - The cross symbol (⊗) between Channel and Time Mixing implies multiplicative interaction between their outputs. - Arrows indicate sequential processing within each block. ### Key Observations - **Symmetry**: Both blocks use σ, μ, and LayerNorm, suggesting modular design. - **WKV Specialization**: The red WKV block in Time Mixing implies a custom attention mechanism. - **Normalization**: LayerNorm appears after mixing in both blocks, ensuring stable gradients. ### Interpretation This architecture resembles a **transformer-based model** with specialized attention mechanisms. - **Channel Mixing** likely processes spatial or feature-level interactions (e.g., in vision transformers). - **Time Mixing** handles temporal or sequential dependencies (e.g., in language models). - The cross symbol (⊗) between blocks suggests **multi-modal integration** or **cross-attention**, where outputs from Channel Mixing modulate Time Mixing. - WKV’s red color and central role in Time Mixing highlight its importance in combining query, key, and value vectors, possibly with learnable weights. The design emphasizes **modularity** (reusable components like σ, μ, LayerNorm) and **efficiency** (shared operations across blocks). The use of LayerNorm after mixing aligns with practices in modern transformers to mitigate internal covariate shift. </details> <details> <summary>x3.png Details</summary>  ### Visual Description ## Diagram: RWKV-LM Neural Network Architecture ### Overview The diagram illustrates the architecture of an RWKV-LM (Recurrent Weighted Key Value Long Memory) neural network model. It shows the flow of data through various components, including input embedding, normalization layers, mixing operations, and output processing. The architecture combines recurrent and feed-forward elements with specialized mixing mechanisms. ### Components/Axes 1. **Input Section (Bottom)** - Pink rectangle labeled "Input Embedding" - Orange rectangle labeled "LayerNorm" (Layer Normalization) 2. **Middle Processing Layers** - Gray rectangle labeled "Time Mixing" (horizontal data flow) - Gray rectangle labeled "Channel Mixing" (vertical data flow) - Orange rectangles labeled "LayerNorm" between mixing layers 3. **Output Section (Top)** - Blue rectangle labeled "RWKV-LM Head" - Green rectangle labeled "Softmax" (output probabilities) - Light blue rectangle labeled "Out" (final output) - Orange rectangle labeled "LayerNorm" before output 4. **Flow Indicators** - Black arrows showing data flow direction - Circular nodes representing data merging points ### Detailed Analysis - **Input Path**: - Input Embedding (pink) → LayerNorm (orange) - Data flows upward through Time Mixing (gray) - **Time Mixing Path**: - Time Mixing (gray) → LayerNorm (orange) - Data splits: part flows to Channel Mixing, part returns to Input path - **Channel Mixing Path**: - Channel Mixing (gray) → LayerNorm (orange) - Data merges with Time Mixing output at circular node - **Output Path**: - Final LayerNorm (orange) → RWKV-LM Head (blue) - Output Probabilities → Softmax (green) → Out (light blue) ### Key Observations 1. **Recurrent Architecture**: The circular data flow between Time Mixing and Channel Mixing suggests recurrent processing capabilities. 2. **Normalization**: LayerNorm appears at multiple stages, indicating its importance for training stability. 3. **Specialized Mixing**: Separate Time and Channel Mixing layers imply dimensionality-specific processing. 4. **Output Structure**: Softmax layer confirms this is a language modeling architecture producing token probabilities. ### Interpretation This architecture combines elements of recurrent neural networks (RNNs) with transformer-like mixing mechanisms. The Time Mixing layer likely processes sequential dependencies, while Channel Mixing handles feature interactions. The multiple LayerNorm applications suggest careful attention to gradient flow and training stability. The RWKV-LM Head appears to be the final processing stage before probability calculation, indicating a hybrid approach between traditional RNNs and attention-based models. The architecture's design suggests optimization for both sequential processing and parallel computation through its mixing layers. </details> Figure 2: Elements within an RWKV block (left) and the complete RWKV residual block, equipped with a final head for language modeling (right). ### 3.1 Architecture The RWKV model is composed of stacked residual blocks. Each block consists of a time-mixing and a channel-mixing sub-block, embodying recurrent structures to leverage past information. This model uses a unique attention-like score update process, which includes a time-dependent softmax operation improving numerical stability and mitigating vanishing gradients (for rigorous proof, see Appendix H). It ensures that the gradient is propagated along the most relevant path. Additionally, layer normalization Ba et al. (2016) incorporated within the architecture aids in stabilizing the gradients, effectively addressing both vanishing and exploding gradient issues. These design elements not only enhance the training dynamics of deep neural networks but also facilitate the stacking of multiple layers, leading to superior performance over conventional RNN models by capturing complex patterns across different levels of abstraction (see also Appendix I). <details> <summary>x4.png Details</summary>  ### Visual Description ## Diagram: Transformer-Based Language Model Architecture ### Overview The diagram illustrates a transformer-based language model architecture processing three input tokens ("name," "is," "Bob") through sequential layers. Each token undergoes transformations via "LM Head," "Channel Mix," "Layer Norm," and "Time Mix" components, with residual connections (dashed arrows) and token shifts. The output tokens ("My," "name," "is") represent the model's generated sequence. ### Components/Axes - **Input Tokens**: - Top row: "name," "is," "Bob" (blue hexagons). - **Processing Layers**: - **LM Head**: Orange rectangles (first layer for all tokens). - **Channel Mix**: Green rectangles (applied post-LM Head). - **Layer Norm**: Yellow rectangles (applied after Channel Mix and Time Mix). - **Time Mix**: Purple diamonds (introduces "States" and "Token shift"). - **Output Tokens**: - Bottom row: "My," "name," "is" (pink hexagons). - **Arrows**: - Solid arrows: Data flow direction. - Dashed arrows: Residual connections or additional pathways. ### Detailed Analysis 1. **Token Flow**: - Each input token ("name," "is," "Bob") is first processed by an "LM Head" (orange). - Outputs are fed into "Channel Mix" (green), followed by "Layer Norm" (yellow). - Residual connections (dashed arrows) bypass "Channel Mix" and "Layer Norm" to later layers. - "Time Mix" (purple) introduces "States" (token shifts) and connects to another "Layer Norm." - Final outputs ("My," "name," "is") emerge after the second "Layer Norm." 2. **Key Patterns**: - **Residual Connections**: Dashed arrows ensure information from earlier layers is preserved. - **Token Shifts**: "Time Mix" modifies token positions (e.g., "name" → "My"). - **Layer Normalization**: Applied after both "Channel Mix" and "Time Mix" to stabilize gradients. ### Key Observations - The architecture combines **spatial processing** ("Channel Mix") and **temporal processing** ("Time Mix"), typical of transformer models. - "States" in "Time Mix" likely represent positional or contextual embeddings. - Output tokens differ from inputs, indicating the model generates new sequences (e.g., "My" instead of "Bob"). ### Interpretation This diagram represents a transformer-based language model where: - **LM Head** initializes token embeddings. - **Channel Mix** captures contextual relationships between tokens. - **Time Mix** introduces sequential dependencies via "States." - **Layer Norm** ensures stable training by normalizing activations. - Residual connections prevent vanishing gradients, enabling deeper networks. The model likely performs tasks like text generation or translation, where input tokens ("name," "is," "Bob") are transformed into coherent outputs ("My," "name," "is"). The architecture balances local context (Channel Mix) and global sequence structure (Time Mix), a hallmark of modern NLP systems. </details> Figure 3: RWKV architecture for language modeling. #### 3.1.1 Token Shift In this architecture, all linear projection vectors ( $R$ , $K$ , $V$ in time-mixing, and $R^\prime$ , $K^\prime$ in channel-mixing) involved in computations are produced by linear interpolation between current and previous timestep inputs, facilitating a token shift. The vectors for time-mixing computation are linear projections of linear combinations of the current and previous inputs of the block: $$ \displaystyle r_t \displaystyle=W_r·(μ_r\odot x_t+(1-μ_r)\odot x_t-1), \displaystyle k_t \displaystyle=W_k·(μ_k\odot x_t+(1-μ_k)\odot x_t-1), \displaystyle v_t \displaystyle=W_v·(μ_v\odot x_t+(1-μ_v)\odot x_t-1), \tag{11} $$ as are the channel-mixing inputs: $$ \displaystyle r^\prime_t \displaystyle=W^\prime_r·(μ^\prime_r\odot x_t+(1-μ^\prime_r)\odot x_t-1), \displaystyle k^\prime_t \displaystyle=W^\prime_k·(μ^\prime_k\odot x_t+(1-μ^\prime_k)\odot x_t-1). \tag{14} $$ The token shift is implemented as a simple offset in the temporal dimension at each block using the PyTorch (Paszke et al., 2019) library as nn.ZeroPad2d((0,0,1,-1)). #### 3.1.2 WKV Operator The computation of the $WKV$ operator in our model parallels the method used in Attention Free Transformer (AFT) Zhai et al. (2021). However, unlike AFT where $W$ is a pairwise matrix, our model treats $W$ as a channel-wise vector that is modified by relative position. In our model, this recurrent behavior is defined by the time-dependent update of the $WKV$ vectors, formalized in the following equation: $$ \displaystyle wkv_t \displaystyle=\frac{∑_i=1^t-1e^-(t-1-i)w+k_i\odot v_i+e^u+k_t \odot v_t}{∑_i=1^t-1e^-(t-1-i)w+k_i+e^u+k_t}. \tag{16} $$ To circumvent any potential degradation of $W$ , we introduce a vector $U$ that separately attends to the current token. More information about this can be found in Appendix I. #### 3.1.3 Output Gating Output gating is implemented in both time-mixing and channel-mixing blocks using the sigmoid of the receptance, $σ(r)$ . The output vector $o_t$ post the $WKV$ operator is given by: $$ \displaystyle o_t \displaystyle=W_o·(σ(r_t)\odot wkv_t). \tag{17} $$ In the channel-mixing block, a similar operation is performed: $$ \displaystyle o^\prime_t \displaystyle=σ(r^\prime_t)\odot(W^\prime_v·\max({k^\prime_t},0)^2), \tag{18} $$ where we adopt the squared ReLU activation function So et al. (2021). ### 3.2 Transformer-like Training RWKV can be efficiently parallelized using a technique called time-parallel mode, reminiscent of Transformers. The time complexity of processing a batch of sequences in a single layer is $O(BTd^2)$ , primarily consisting of matrix multiplications $W_λ$ , where $λ∈\{r,k,v,o\}$ (assuming $B$ sequences, $T$ maximum tokens, and $d$ channels). In contrast, updating attention scores $wkv_t$ involves a serial scan (see Appendix D for more detail) and has complexity $O(BTd)$ . The matrix multiplications can be parallelized similarly to $W_λ$ , where $λ∈\{Q,K,V,O\}$ in conventional Transformers. The element-wise $WKV$ computation is time-dependent but can be readily parallelized along the other two dimensions Lei et al. (2018) For extremely long sequences, more sophisticated methods such as Martin and Cundy (2017) that parallelize over sequence length could be used.. ### 3.3 RNN-like Inference Recurrent networks commonly utilize the output at state $t$ as input at state $t+1$ . This usage is also observed in the autoregressive decoding inference of language models, where each token must be computed before being passed to the next step. RWKV takes advantage of this RNN-like structure, known as time-sequential mode. In this context, RWKV can be conveniently formulated recursively for decoding during inference, as demonstrated in Appendix D. ### 3.4 Additional Optimizations Custom Kernels To address inefficiencies in the $WKV$ computation arising from the sequential nature of the task when using standard deep learning frameworks, we have developed a custom CUDA kernel. This kernel enables the execution of a single compute kernel on training accelerators, while all other parts of the model, such as matrix multiplications and point-wise operations, are already inherently parallelizable and efficient. Small Init Embedding During the initial stage of training a transformer model Vaswani et al. (2017), we observe that the embedding matrix undergoes slow changes, presenting a challenge for the model to move away from its initial noisy embedding state. To address this issue, we propose an approach that involves initializing the embedding matrix with small values and subsequently applying an additional LayerNorm operation. This accelerates and stabilizes the training process, allowing for the training of deep architectures with post-LN components. The effectiveness of this approach is demonstrated in Figure 9, illustrating improved convergence by enabling the model to quickly transition away from the initially small embedding. This is achieved through small changes occurring in a single step, which subsequently lead to substantial alterations in directions and further notable changes after the LayerNorm operation. Custom Initialization Building on principles from previous works (He et al., 2016; Jumper et al., 2021), we adopt an initialization strategy where parameters are set to values resembling an identity mapping while breaking symmetry to establish a clear information flow. The majority of weights are initialized to zero, and linear layers do not employ biases. Detailed formulas are given in Appendix E. We observe that the choice of initialization plays a crucial role in both the speed and quality of convergence (refer to Appendix F for further details). ### 3.5 Implementation RWKV is implemented using the PyTorch Deep Learning Library Paszke et al. (2019). We integrate additional optimization strategies inspired by DeepSpeed Rasley et al. (2020) into the system, improving its efficiency and scalability. The model begins with an embedding layer, as detailed in Section 3.4. Following this are several identical residual blocks arranged sequentially. These are depicted in Figures 2 and 3 and adheres to the principles outlined in Section 3.1.1. After the last block, a simple output projection head, consisting of a LayerNorm Ba et al. (2016) and a linear projection, is employed for logits generation for next-token prediction and computation of the cross-entropy loss during training. ## 4 Trained Models and Computing Costs To demonstrate the scalability of RWKV, we train six models ranging from 169 million to 14 billion parameters as shown in Table 2. All models are trained for one epoch (330 billion tokens) on the Pile (Gao et al., 2020; Biderman et al., 2022). Name Layers Model Dimension Parameters FLOP per token $169 M$ 12 768 $1.693×{10}^08$ $2.613×{10}^08$ $430 M$ 24 1024 $4.304×{10}^08$ $7.573×{10}^08$ $1.5 B$ 24 2048 $1.515×{10}^09$ $2.823×{10}^09$ $3 B$ 32 2560 $2.985×{10}^09$ $5.710×{10}^09$ $7 B$ 32 4096 $7.393×{10}^09$ $1.437×{10}^10$ $14 B$ 40 5120 $1.415×{10}^10$ $2.778×{10}^10$ Table 2: RWKV model architectures and FLOP counts. Further details of these hyperparameters are elaborated upon in Appendix G. The number of parameters for each model is computed using the formula: $\# parameters=2VD+13D^2L+D(11L+4)$ where $V$ = 50277 is the vocabulary size, $D$ represents the Model Dimension and $L$ corresponds to the number of layers. FLOPs is for a forward pass for one token. It was calculated as $2(2VD+13D^2L)$ , which is the twice (add and multiply) the number of parameters in linear layers. The backwards pass FLOPs can be approximated as twice that of the forward pass, giving a total of $6(2VD+13D^2L)$ FLOP per token. Notably, this matches the standard formula for FLOP calculations in transformers Kaplan et al. (2020): $FLOP=6·[\# tokens]·[\# parameters]$ . ### 4.1 Additional Training Details For training, we use the standard Adam optimizer without weight decay, use bfloat16 precision, and train with a context length of 1024 tokens. Further details on hyperparameters are in Appendix G. Diverting from standard practice for transformers, we apply exponential decay to our learning rate. We also incorporate the auxiliary loss introduced by PaLM Chowdhery et al. (2022), supplementing the standard cross-entropy loss function. This auxiliary loss encourages the softmax normalizer to approximate zero closely. As for the learning rate schedule, it remains constant for the initial iterations, and subsequently decays exponentially. ### 4.2 Scaling Laws Scaling laws Kaplan et al. (2020); Henighan et al. (2020); Hoffmann et al. (2022); Muennighoff et al. (2023) in language models refer to the mathematical relationships that describe how the performance of a language model changes with respect to various factors. These factors can include the model size ( $N$ ), dataset size ( $D$ ), or the optimally allocated compute budget ( $C_\rm min$ ). Scaling laws are important for two primary reasons: they allow us to make predictions and plans regarding the costs and performance of large models before they are trained via interpolation and extrapolation Black et al. (2022); Le Scao et al. (2022) and the contexts in which they fail provides rich feedback on important areas for future research Wei et al. (2022a); Biderman et al. (2023a). Previous work on scaling laws for RNNs has claimed that LSTMs do not strictly follow the same log-log linear scaling that transformers do Kaplan et al. (2020). We train 45 RWKV models for a variety of pairs (dataset, parameters) and find that RWKV does follow the same general form of the scaling law that is well established for transformers. Figure 4 shows our results for loss as a function of compute, with the linear fit to the Pareto optimal points holding an $r^2$ value of $0.994$ . Even when we extrapolate our curve an additional order of magnitude (blue), we find an extremely good fit with an $r^2$ of $0.875$ . <details> <summary>x5.png Details</summary>  ### Visual Description ## Scatter Plot: Compute vs. Loss Efficiency ### Overview The image depicts a scatterplot analyzing the relationship between computational resources (exaFLOP) and loss efficiency. A downward-trending blue line represents the expected relationship, with data points categorized as "Non-Optimal" (black), "Optimal" (red), and "Test" (blue). The plot uses logarithmic scales for both axes. ### Components/Axes - **X-axis (Compute)**: Labeled "Compute (exaFLOP)" with values ranging from 10⁻¹ to 10³ (logarithmic scale). - **Y-axis (Loss)**: Labeled "Loss" with values from 2×10⁰ to 3×10⁰ (logarithmic scale). - **Legend**: Located in the top-right corner, with: - Black circles = Non-Optimal - Red circles = Optimal - Blue circles = Test - Blue line = Trend Line ### Detailed Analysis 1. **Non-Optimal (Black Points)**: - Positioned primarily above the trend line. - Spread across the entire x-axis range (10⁻¹ to 10³). - Loss values consistently higher than the trend line prediction for equivalent compute. 2. **Optimal (Red Points)**: - Clustered tightly around the trend line. - Distributed across the full x-axis range. - Loss values closely match the trend line, suggesting efficient resource utilization. 3. **Test (Blue Points)**: - Located below the trend line, especially at higher compute values (>10¹ exaFLOP). - Positioned at 10² and 10³ exaFLOP on the x-axis. - Loss values significantly lower than the trend line prediction. 4. **Trend Line**: - Straight diagonal line from top-left (3×10⁰ loss at 10⁻¹ exaFLOP) to bottom-right (2×10⁰ loss at 10³ exaFLOP). - Represents a power-law relationship (y ∝ x⁻¹) between compute and loss. ### Key Observations - **Efficiency Gradient**: Loss decreases by ~1 log unit for every 10x increase in compute (per trend line). - **Test Case Anomaly**: Blue points deviate sharply from the trend at high compute, suggesting exceptional efficiency or unique optimization. - **Non-Optimal Variance**: Black points show 20–50% higher loss than the trend line across all compute ranges. - **Optimal Consistency**: Red points maintain <5% deviation from the trend line, indicating reliable performance. ### Interpretation The data demonstrates a clear inverse relationship between compute and loss, with optimal configurations achieving near-ideal efficiency. The Test points (blue) represent a notable outlier, performing 30–40% better than the trend line at 10²–10³ exaFLOP. This could indicate: 1. Specialized hardware/software optimizations 2. Different problem characteristics 3. Measurement artifacts Non-Optimal configurations (black) consistently underperform, with losses 2–3× higher than the trend line suggests. The logarithmic scale emphasizes exponential scaling effects, showing that doubling compute reduces loss by ~30% in optimal scenarios. The Test case's deviation warrants further investigation to determine if it represents a new efficiency benchmark or an outlier requiring validation. </details> Figure 4: Scaling laws curves for RWKV models ## 5 Evaluations Having demonstrated the scalability of RWKV models in the previous section, we now turn our attention to their competitiveness with traditional transformers. We focus on two questions: Competitiveness Is RWKV competitive against quadratic transformer architectures with the same amount of compute? Long Context Does increasing the context length of RWKV yield better language modeling loss when RWKV models are trained for context lengths that most open-sourced quadratic transformers cannot efficiently process? ### 5.1 NLP Evaluations <details> <summary>x6.png Details</summary>  ### Visual Description ## Line Chart: Model Accuracy vs. Compute (exaFLOP) ### Overview The chart compares the accuracy of four language models (BLOOM, Pythia, OPT, RWKV) across varying compute levels (exaFLOP). Accuracy is plotted on a linear y-axis (17.5–37.5), while compute is on a logarithmic x-axis (10²–10³). Lines represent model performance trends, with RWKV showing the steepest growth. ### Components/Axes - **X-axis**: "Compute (exaFLOP)" (logarithmic scale: 10², 10³) - **Y-axis**: "Accuracy" (linear scale: 17.5–37.5) - **Legend**: Located in the bottom-right corner, mapping: - **BLOOM**: Orange circles - **Pythia**: Blue squares - **OPT**: Green triangles - **RWKV**: Red stars ### Detailed Analysis 1. **BLOOM** (orange circles): - Starts at ~18.5 accuracy at 10² exaFLOP. - Gradual increase to ~30.5 accuracy at 10³ exaFLOP. - Slope: Moderate upward trend. 2. **Pythia** (blue squares): - Begins at ~19 accuracy at 10² exaFLOP. - Sharp rise to ~32 accuracy at 10³ exaFLOP. - Slope: Steeper than BLOOM but less than RWKV. 3. **OPT** (green triangles): - Starts at ~19 accuracy at 10² exaFLOP. - Accelerates to ~33 accuracy at 10³ exaFLOP. - Slope: Steeper than BLOOM/Pythia, approaching RWKV. 4. **RWKV** (red stars): - Begins at ~19.5 accuracy at 10² exaFLOP. - Rapid ascent to ~37 accuracy at 10³ exaFLOP. - Slope: Steepest, outpacing all models. ### Key Observations - **RWKV dominates** in accuracy at higher compute levels, achieving ~37 accuracy vs. ~33 for OPT and ~32 for Pythia at 10³ exaFLOP. - **Convergence at low compute**: All models start near 18.5–19 accuracy at 10² exaFLOP. - **Logarithmic scale effect**: RWKV’s exponential growth is visually emphasized, while BLOOM’s linear progression appears flatter. ### Interpretation The data suggests **RWKV is the most compute-efficient model**, delivering superior accuracy gains per additional exaFLOP. This makes it ideal for high-resource environments. In contrast, **BLOOM and Pythia** show diminishing returns at scale, while **OPT balances efficiency and performance**. The logarithmic x-axis highlights RWKV’s scalability advantage, critical for applications requiring high accuracy with limited compute budgets. No outliers or anomalies are observed; trends align with model design philosophies (e.g., RWKV’s architecture optimized for parallelization). </details> (a) ARC (Challenge) <details> <summary>x7.png Details</summary>  ### Visual Description ## Line Graph: Relationship Between Compute (exaFLOP) and Accuracy ### Overview The image depicts a line graph illustrating the relationship between computational resources (measured in exaFLOP) and model accuracy. Four distinct data series are plotted, each represented by a unique color and marker. The x-axis (Compute) uses a logarithmic scale, while the y-axis (Accuracy) ranges from 25 to 55. All data series exhibit upward trends, suggesting a positive correlation between compute and accuracy. ### Components/Axes - **X-Axis (Compute)**: Labeled "Compute (exaFLOP)" with a logarithmic scale spanning 10^2 to 10^3. - **Y-Axis (Accuracy)**: Labeled "Accuracy" with a linear scale from 25 to 55. - **Legend**: Located on the right side of the graph, associating colors and markers with four data series: - Green triangles - Orange stars - Blue squares - Yellow circles ### Detailed Analysis 1. **Green Triangles**: - Starts at ~28 accuracy at 10^2 exaFLOP. - Rises steeply to ~52 accuracy at 10^3 exaFLOP. - **Trend**: Sharpest upward slope among all series. 2. **Orange Stars**: - Begins at ~29 accuracy at 10^2 exaFLOP. - Increases to ~51 accuracy at 10^3 exaFLOP. - **Trend**: Steady but less aggressive growth compared to green. 3. **Blue Squares**: - Starts at ~26 accuracy at 10^2 exaFLOP. - Reaches ~50 accuracy at 10^3 exaFLOP. - **Trend**: Consistent linear progression. 4. **Yellow Circles**: - Begins at ~32 accuracy at 10^2 exaFLOP. - Ends at ~46 accuracy at 10^3 exaFLOP. - **Trend**: Slower growth, with a plateau near 10^3 exaFLOP. ### Key Observations - All data series show a positive correlation between compute and accuracy. - The green line (triangles) demonstrates the most efficient scaling, achieving the highest accuracy gain per exaFLOP. - The yellow line (circles) exhibits diminishing returns, with accuracy plateauing despite increased compute. - At 10^3 exaFLOP, the green and orange lines converge near 50–52 accuracy, while the yellow line lags at ~46. ### Interpretation The data suggests that computational resources significantly enhance model accuracy, but the efficiency of this relationship varies across methods or models. The green line’s steep slope implies superior algorithmic or architectural optimization, maximizing accuracy per exaFLOP. In contrast, the yellow line’s slower growth may indicate inherent limitations, such as suboptimal scaling or resource bottlenecks. The convergence of green and orange lines at higher compute levels highlights potential trade-offs between resource allocation and performance gains. Notably, the yellow line’s higher starting accuracy at 10^2 exaFLOP (~32 vs. ~26–29 for others) suggests it may be pre-optimized for lower-resource environments but struggles to scale effectively. This analysis underscores the importance of balancing computational investment with algorithmic efficiency to achieve optimal performance. </details> (b) HellaSwag <details> <summary>x8.png Details</summary>  ### Visual Description ## Line Graph: Model Accuracy vs. Compute (exaFLOP) ### Overview The graph compares the accuracy of four language models (BLOOM, Pythia, OPT, RWKV) against computational resources (exaFLOP). Accuracy increases with compute, with all models showing upward trends. RWKV and Pythia achieve the highest accuracy at maximum compute, while BLOOM lags slightly behind. ### Components/Axes - **X-axis (Compute)**: Labeled "Compute (exaFLOP)", logarithmic scale from 10² to 10³. - **Y-axis (Accuracy)**: Labeled "Accuracy", linear scale from 20 to 70. - **Legend**: Located at bottom-right, mapping colors to models: - **Yellow circle**: BLOOM - **Blue square**: Pythia - **Green triangle**: OPT - **Orange star**: RWKV ### Detailed Analysis 1. **Pythia (Blue Square)**: - Starts at ~20 accuracy at 10² exaFLOP. - Sharp increase to ~50 at 10².⁵ exaFLOP. - Reaches ~70 accuracy at 10³ exaFLOP. - *Trend*: Steep upward slope, fastest improvement. 2. **BLOOM (Yellow Circle)**: - Begins at ~35 accuracy at 10² exaFLOP. - Gradual rise to ~45 at 10².⁵ exaFLOP. - Ends at ~58 accuracy at 10³ exaFLOP. - *Trend*: Slowest initial growth but consistent improvement. 3. **OPT (Green Triangle)**: - Starts at ~38 accuracy at 10² exaFLOP. - Moderate increase to ~55 at 10².⁵ exaFLOP. - Reaches ~68 accuracy at 10³ exaFLOP. - *Trend*: Steady linear growth. 4. **RWKV (Orange Star)**: - Begins at ~33 accuracy at 10² exaFLOP. - Sharp rise to ~55 at 10².⁵ exaFLOP. - Ends at ~70 accuracy at 10³ exaFLOP. - *Trend*: Matches Pythia’s steepness, highest final accuracy. ### Key Observations - **Compute-Accuracy Correlation**: All models show positive correlation between compute and accuracy. - **Efficiency Differences**: Pythia and RWKV achieve similar accuracy gains with less compute compared to BLOOM. - **Final Performance**: At 10³ exaFLOP, RWKV and Pythia tie at ~70 accuracy, outpacing OPT (~68) and BLOOM (~58). ### Interpretation The data demonstrates that increased computational resources improve model accuracy across all architectures. However, efficiency varies significantly: - **Pythia and RWKV** maximize accuracy per exaFLOP, suggesting optimized architectures or training methods. - **BLOOM** requires more compute to reach comparable accuracy, potentially due to architectural limitations or data inefficiency. - The logarithmic x-axis highlights diminishing returns at higher compute levels, as accuracy plateaus near 70 for most models. This graph underscores the trade-off between computational cost and performance, guiding resource allocation for model development. </details> (c) LAMBADA (OpenAI) <details> <summary>x9.png Details</summary>  ### Visual Description ## Line Graph: Model Accuracy vs. Compute (exaFLOP) ### Overview The graph illustrates the relationship between computational resources (exaFLOP) and model accuracy for four distinct models (A, B, C, D). Accuracy improves with increased compute, but the rate of improvement varies significantly across models. The x-axis uses a logarithmic scale (10² to 10³ exaFLOP), while the y-axis is linear (12.5–30.0 accuracy). ### Components/Axes - **X-axis (Compute)**: Logarithmic scale (10² to 10³ exaFLOP), labeled "Compute (exaFLOP)". - **Y-axis (Accuracy)**: Linear scale (12.5–30.0), labeled "Accuracy". - **Legend**: Located in the top-right corner, mapping colors to models: - **Orange (★)**: Model A - **Green (▲)**: Model B - **Blue (■)**: Model C - **Yellow (●)**: Model D ### Detailed Analysis 1. **Model A (Orange ★)**: - **Trend**: Sharp initial increase in accuracy, followed by a plateau. - **Data Points**: - ~16.5 exaFLOP → ~17.5 accuracy - ~21 exaFLOP → ~21 accuracy - ~1000 exaFLOP → ~29.5 accuracy - **Key Insight**: Achieves near-peak accuracy (~29.5) at the highest compute but shows diminishing returns after ~21 exaFLOP. 2. **Model B (Green ▲)**: - **Trend**: Steady, linear increase in accuracy. - **Data Points**: - ~17 exaFLOP → ~17 accuracy - ~100 exaFLOP → ~22.5 accuracy - ~1000 exaFLOP → ~27.5 accuracy - **Key Insight**: Maintains consistent improvement but lags behind Model A at lower compute levels. 3. **Model C (Blue ■)**: - **Trend**: Slow initial growth, followed by accelerated improvement. - **Data Points**: - ~18 exaFLOP → ~18 accuracy - ~100 exaFLOP → ~22 accuracy - ~1000 exaFLOP → ~26 accuracy - **Key Insight**: Outperforms Model B at higher compute but underperforms at lower compute. 4. **Model D (Yellow ●)**: - **Trend**: Gradual increase with a slight dip mid-range. - **Data Points**: - ~19 exaFLOP → ~19 accuracy - ~100 exaFLOP → ~21 accuracy - ~1000 exaFLOP → ~25 accuracy - **Key Insight**: Least efficient model, requiring the most compute for marginal gains. ### Key Observations - **Efficiency Variance**: Model A achieves the highest accuracy (~29.5) with the least compute (~1000 exaFLOP), while Model D requires similar compute for ~25 accuracy. - **Diminishing Returns**: Model A plateaus near its peak accuracy, suggesting architectural limits. - **Compute Thresholds**: All models show significant accuracy jumps between 10² and 10³ exaFLOP, but Model A’s early gains are most pronounced. ### Interpretation The data highlights a trade-off between computational cost and accuracy. Model A demonstrates superior efficiency, achieving near-optimal performance with minimal resources. In contrast, Model D’s gradual improvement suggests suboptimal design or training. The logarithmic x-axis emphasizes that even modest compute increases (e.g., 10x) yield disproportionate accuracy gains, particularly for Models A and B. This underscores the importance of architectural optimization over brute-force scaling. The plateau in Model A’s performance raises questions about hardware/software co-design limits, while Model D’s inefficiency may indicate overfitting or poor resource utilization. </details> (d) OpenBookQA <details> <summary>x10.png Details</summary>  ### Visual Description ## Line Chart: Model Accuracy vs. Compute (exaFLOP) ### Overview The image depicts a line chart comparing the accuracy of four language models (BLOOM, Pythia, OPT, RWKV) across varying computational resources (exaFLOP). All models exhibit upward trends, with OPT consistently achieving the highest accuracy. The x-axis (Compute) spans 10² to 10³ exaFLOP, while the y-axis (Accuracy) ranges from 50 to 90. ### Components/Axes - **X-axis (Compute)**: Labeled "Compute (exaFLOP)", logarithmic scale from 10² (100) to 10³ (1000). - **Y-axis (Accuracy)**: Labeled "Accuracy", linear scale from 50 to 90. - **Legend**: Located in the bottom-right corner, associating colors/markers with models: - **BLOOM**: Orange circles - **Pythia**: Blue squares - **OPT**: Green triangles - **RWKV**: Red stars - **Gridlines**: Horizontal and vertical, aiding data point alignment. ### Detailed Analysis 1. **BLOOM (Orange Circles)**: - Starts at ~68 accuracy at 100 exaFLOP. - Ends at ~85 accuracy at 1000 exaFLOP. - Trend: Steady, moderate upward slope. 2. **Pythia (Blue Squares)**: - Starts at ~50 accuracy at 100 exaFLOP. - Ends at ~87 accuracy at 1000 exaFLOP. - Trend: Sharp initial rise, then gradual increase. 3. **OPT (Green Triangles)**: - Starts at ~71 accuracy at 100 exaFLOP. - Ends at ~89 accuracy at 1000 exaFLOP. - Trend: Consistent, linear growth with the highest final accuracy. 4. **RWKV (Red Stars)**: - Starts at ~62 accuracy at 100 exaFLOP. - Ends at ~86 accuracy at 1000 exaFLOP. - Trend: Gradual, near-linear increase. ### Key Observations - **OPT** maintains the highest accuracy across all compute levels, suggesting superior efficiency or architecture. - **Pythia** demonstrates the steepest initial improvement, indicating strong sensitivity to increased compute. - **RWKV** and **BLOOM** converge at higher compute levels (~85–86 accuracy), implying diminishing returns. - All models show diminishing marginal gains as compute approaches 1000 exaFLOP. ### Interpretation The data suggests that computational resources significantly impact model accuracy, with OPT achieving the best performance per exaFLOP. Pythia’s rapid early growth highlights its potential for optimization under resource constraints. The convergence of RWKV and BLOOM at higher compute levels implies that further efficiency gains may require architectural innovations rather than scaling alone. The logarithmic x-axis emphasizes that even modest increases in compute (e.g., 10x) yield substantial accuracy improvements, particularly for Pythia. This trend underscores the trade-off between resource investment and performance in large language models. </details> (e) ReCoRD <details> <summary>x11.png Details</summary>  ### Visual Description ## Line Chart: Accuracy vs. Compute (exaFLOP) ### Overview The chart illustrates the relationship between computational resources (measured in exaFLOP) and model accuracy across four distinct data series. Each series is represented by a unique color and marker type, with accuracy plotted on the y-axis (50–66) and compute on the x-axis (10²–10³). ### Components/Axes - **X-axis (Compute)**: Labeled "Compute (exaFLOP)" with logarithmic scale markers at 10² (100), 10³ (1000). - **Y-axis (Accuracy)**: Labeled "Accuracy" with linear scale markers from 50 to 66. - **Legend**: Positioned on the right, associating colors and markers: - Green triangles (Line A) - Orange stars (Line B) - Blue squares (Line C) - Yellow diamonds (Line D) ### Detailed Analysis #### Line A (Green Triangles) - **Trend**: Starts at ~50.1 (10²) and rises sharply to ~65.0 (10³). - **Key Points**: - 10²: ~50.1 - 10³: ~65.0 #### Line B (Orange Stars) - **Trend**: Gradual upward slope from ~50.5 (10²) to ~64.3 (10³). - **Key Points**: - 10²: ~50.5 - 10³: ~64.3 #### Line C (Blue Squares) - **Trend**: Steeper increase after 10², reaching ~63.8 (10³). - **Key Points**: - 10²: ~52.8 - 10³: ~63.8 #### Line D (Yellow Diamonds) - **Trend**: Sharp rise from ~50.3 (10²) to ~64.5 (10³). - **Key Points**: - 10²: ~50.3 - 10³: ~64.5 ### Key Observations 1. **Line A (Green)** exhibits the steepest initial increase, suggesting rapid accuracy gains with compute. 2. **Line D (Yellow)** starts lowest but surpasses others by 10³, indicating strong scalability. 3. **Line B (Orange)** and **Line C (Blue)** show more gradual improvements, with Line C outperforming Line B at higher compute levels. 4. All lines converge near 64–65 accuracy at 10³ exaFLOP, suggesting diminishing returns at extreme compute levels. ### Interpretation The data demonstrates a clear positive correlation between compute resources and model accuracy. Line A’s sharp rise implies it may leverage more efficient algorithms or architectures, while Line D’s late surge highlights potential for underperforming models to catch up with sufficient resources. The convergence at high compute levels suggests that beyond a certain threshold, additional compute yields minimal accuracy improvements. This could inform resource allocation strategies for model training, prioritizing models with higher scalability (e.g., Line A or D) for optimal performance. **Note**: Values are approximate, with uncertainty due to visual estimation from the chart. No textual content or non-English elements are present. </details> (f) Winogrande Figure 5: Zero-Shot Performance of RWKV on common language modeling evaluation benchmarks. Additional plots can be found in Appendix J. To demonstrate that RWKV is competitive with traditional transformers at NLP tasks, we compare with similarly sized models trained for a similar number of tokens (Pythia Biderman et al. (2023b), OPT Zhang et al. (2022) and BLOOM Scao et al. (2022)). All RWKV models were trained for one epoch on the Pile (330B tokens), which is close but not identical to the amount of tokens the Pythia, OPT, and BLOOM models were trained for. Consequently, we compare our models on a FLOP-matched basis. We avoid comparing with model trained in the Chinchilla-optimal regime (Hoffmann et al., 2022) or the overtrained regime (Touvron et al., 2023) to ensure the most equitable comparison. We report results on ARC (both Easy and Challenge) Clark et al. (2018), BoolQ Clark et al. (2019), COPA Roemmele et al. (2018), HeadQA Vilares and Gómez-Rodríguez (2019), HellaSwag Zellers et al. (2019), LAMBADA Paperno et al. (2016), OpenBookQA Mihaylov et al. (2018), PIQA Bisk et al. (2020), ReCoRD Zhang et al. (2018), SciQ Johannes Welbl Nelson F. Liu (2017), and Winogrande Zellers et al. (2020). Figure 1 shows the average results across all benchmarks. Some individual benchmarks are shown in Fig 5, with the rest in Appendix J. Additionally, we carried out comparative studies on RWKV and ChatGPT / GPT-4, see Appendix L. They revealed that RWKV is very sensitive to prompt engineering. When the prompts were adjusted (re-ordered) from the ones used for GPT to more suitable for RWKV, the performance (F1) increased even from 44.2% to 74.8%. For sarcasm detection, RWKV outperformed ChatGPT, but was still slightly worse than the SOTA solution. ### 5.2 Extended Context Finetuning Unlike transformers, RNNs do not have a pre-defined sequences length when they are created. However in order to efficient make use of compute we nevertheless need to preprocess the training data into contexts of the same length. We find that we are able to teach the model how to efficiently handle substantially larger batch sizes by finetuning with progressively increasing sequence length. Specifically, we first double the sequence length from 1024 to 2048 and finetune for 10B tokens from the original pretraining corpus, then we double again to 4096 for 100B tokens from the same corpus, and finally double to 8192 tokens for another 100B tokens from the same corpus. In Fig. 6 we show that increasing context length leads to lower test loss on the Pile, an indication that RWKV can make effective use of long contextual information. <details> <summary>x12.png Details</summary>  ### Visual Description ## Line Chart: Pile Test Loss vs. Context Length ### Overview The image depicts a logarithmic line chart comparing pile test loss across different context lengths for two model configurations: 7B 8k (blue dashed line) and 14B 8k (orange dashed line). Both axes use logarithmic scales, with context length ranging from 2¹ to 2¹¹ and pile test loss from 2¹ to 2². ### Components/Axes - **X-axis (Context Length)**: Logarithmic scale from 2¹ to 2¹¹, labeled "Context Length." - **Y-axis (Pile Test Loss)**: Logarithmic scale from 2¹ to 2², labeled "Pile test loss." - **Legend**: Located in the top-right corner, with: - Blue dashed line: "7B 8k" - Orange dashed line: "14B 8k" ### Detailed Analysis #### 7B 8k (Blue Line) - **Data Points**: - 2¹: ~2.8 - 2³: ~1.8 - 2⁵: ~1.3 - 2⁷: ~1.1 - 2⁹: ~1.05 - 2¹¹: ~1.02 - **Trend**: Steep initial decline (2¹ to 2³), followed by gradual flattening. Loss stabilizes near 1.0 at 2¹¹. #### 14B 8k (Orange Line) - **Data Points**: - 2¹: ~2.9 - 2³: ~1.9 - 2⁵: ~1.4 - 2⁷: ~1.2 - 2⁹: ~1.1 - 2¹¹: ~1.05 - **Trend**: Similar trajectory to 7B 8k but consistently higher loss values. Converges with 7B 8k at larger context lengths (2⁹–2¹¹). ### Key Observations 1. **Initial Disparity**: At 2¹ context length, 14B 8k exhibits ~3.6% higher loss than 7B 8k. 2. **Convergence**: Both models show near-identical performance (loss ~1.0–1.1) at 2⁹–2¹¹ context lengths. 3. **Logarithmic Scaling**: Exponential context length increases correlate with sub-linear reductions in pile test loss. ### Interpretation The data suggests that larger model size (14B vs. 7B) initially incurs higher computational costs (higher loss) but achieves comparable performance to smaller models at extended context lengths. The logarithmic scaling reveals diminishing returns for increasing context length beyond 2⁵, as both models stabilize near baseline loss thresholds. This implies that model efficiency plateaus at larger scales, with context length having a more pronounced impact on early-stage performance than on stabilized outcomes. </details> Figure 6: RWKV shows decreasing mean test loss as a function of context length on the Pile Gao et al. (2020) ### 5.3 Long Context Benchmarks Additionally, we evaluate our model’s ability to handle very long sequences by comparing to state-of-the-art long sequence models on the Long-Range Arena (LRA) benchmark Tay et al. (2021). LRA is designed to assess the performance of models in handling lengthy context situations. It includes a collection of tasks with sequences ranging from 1,000 to 16,000 tokens, covering various types of data like text, natural language, synthetic images, and mathematical expressions. We apply RWKV on the LRA benchmark and the results are in Appendix J.2. The results show that RWKV performs second only to the S4 model in five datasets. ## 6 Inference Experiments We benchmark inference requirements according to size and family. Specifically, we evaluate text generation speed and memory requirements on typical compute platforms including CPU (x86) and GPU (NVIDIA A100 80 GB). For all of our inference experiments we use float32 precision and the HuggingFace Transformers Wolf et al. (2020). We include all model parameters in the parameter count, including both embedding and non-embedding layers. Performance under different quantization setups is left to further work. See Appendix K for more results. <details> <summary>x13.png Details</summary>  ### Visual Description ## Line Chart: Cumulative Text Generation Time vs. Token Count ### Overview The image displays a line chart comparing the cumulative time (in seconds) required for text generation across five different AI models as a function of token count (0-1000 tokens). The chart shows five distinct data series with varying performance characteristics. ### Components/Axes - **X-axis**: "# Tokens" (0 to 1000 in increments of 200) - **Y-axis**: "Cumulative time (s) in text generation" (0 to 60 seconds) - **Legend**: Located in top-left corner with five entries: - Blue: facebook/opt-2.7b - Orange: EleutherAI/gpt-neo-2.7B - Green: bigscience/bloom-3b - Red: EleutherAI/pythia-2.8b - Purple: rwkv-4-pile-3b ### Detailed Analysis 1. **facebook/opt-2.7b** (Blue line): - Starts at (0,0) - Curves upward with moderate slope - Reaches ~52s at 1000 tokens 2. **EleutherAI/gpt-neo-2.7B** (Orange line): - Starts at (0,0) - Curves upward with steep slope - Reaches ~55s at 1000 tokens 3. **bigscience/bloom-3b** (Green line): - Starts at (0,0) - Curves upward with steepest slope - Reaches ~60s at 1000 tokens 4. **EleutherAI/pythia-2.8b** (Red line): - Starts at (0,0) - Curves upward with steep slope - Reaches ~58s at 1000 tokens 5. **rwkv-4-pile-3b** (Purple line): - Starts at (0,0) - Linear progression with shallow slope - Reaches ~10s at 1000 tokens ### Key Observations - All models show increasing cumulative time with token count - rwkv-4-pile-3b demonstrates significantly better performance (lowest time) - bigscience/bloom-3b shows worst performance (highest time) - Lines exhibit different curvature patterns: - rwkv-4-pile-3b: Most linear - Others: Increasingly concave upward - At 600 tokens, performance gap widens: - rwkv-4-pile-3b: ~7s - facebook/opt-2.7b: ~22s - EleutherAI/gpt-neo-2.7B: ~24s - EleutherAI/pythia-2.8b: ~26s - bigscience/bloom-3b: ~28s ### Interpretation The data reveals significant performance disparities between models. The rwkv-4-pile-3b model demonstrates exceptional efficiency, maintaining near-linear scaling with minimal time increase per token. In contrast, the bigscience/bloom-3b model shows diminishing returns with steeper time increases per additional token. This suggests architectural differences in model design, with rwkv-4-pile-3b potentially optimized for faster inference. The chart highlights important considerations for deployment scenarios where real-time text generation is critical, with rwkv-4-pile-3b appearing most suitable for high-throughput applications. </details> Figure 7: Cumulative time on text generation for LLMs. Unlike transformers, RWKV exhibits linear scaling. ## 7 Future Work There are several promising directions for future work on the RWKV architecture. Work can be done to increase model expressivity by enhancing the time-decay formulations and exploring initial model states while maintaining efficiency. The RWKV computational efficiency can be further improved by applying a parallel scan in the $wkv_t$ step to reduce the computational cost to $O(B\log(T)d)$ . The mechanisms used in RWKV can be applied to encoder-decoder architectures, potentially replacing the cross-attention mechanism. This could be applicable in seq2seq or multimodal settings, thereby enhancing efficiency during both training and inference. RWKV’s state (or context) can be leveraged for interpretability, predictability in sequence data, and safety. Manipulating the hidden state could also guide behavior and allow greater customizability through prompt tuning. The RWKV architecture is not perfect, and can be improved via many aspects, such as modifying the formulae or implementing larger internal states. Larger states can enhance the model’s memory to previous context and improve performance over various tasks. ## 8 Conclusions We introduced RWKV, a new approach to RNN models exploiting the potential of time-based mixing components. RWKV introduces several key strategies that allow it to capture locality and long-range dependencies while addressing limitations of current architectures by: (1) replacing the quadratic QK attention with a scalar formulation at linear cost, (2) reformulating recurrence and sequential inductive biases to enable efficient training parallelization and efficient inference, and (3) enhancing training dynamics using custom initializations. We benchmark the proposed architecture in a wide variety of NLP tasks and show comparable performance to SoTA with reduced cost. Further experiments on expressivity, interpretability, and scaling showcase the model capabilities and draw parallels in behavior between RWKV and other LLMs. RWKV opens a new route for scalable and efficient architectures to model complex relationships in sequential data. While many alternatives to Transformers have been proposed with similar claims, ours is the first to back up those claims with pretrained models with tens of billions of parameters. ## 9 Limitations While our proposed RWKV model has demonstrated promising results regarding training and memory efficiency during inference, some limitations should be acknowledged and addressed in future work. First, the linear attention of RWKV leads to significant efficiency gains but still, it may also limit the model’s performance on tasks that require recalling minutiae information over very long contexts. This is due to the funneling of information through a single vector representation over many time steps, compared with the full information maintained by the quadratic attention of standard Transformers. In other words, the model’s recurrent architecture inherently limits its ability to “look back” at previous tokens, as opposed to traditional self-attention mechanisms. While learned time decay helps prevent the loss of information, it is mechanistically limited compared to full self-attention. Another limitation of this work is the increased importance of prompt engineering in comparison to standard Transformer models. The linear attention mechanism used in RWKV limits the information from the prompt that will be carried over to the model’s continuation. As a result, carefully designed prompts may be even more crucial for the model to perform well on tasks. The above RWKV property was confirmed by studies on prompt engineering presented in Appendix L. By changing the order of the information pieces, we were even able to almost double the RWKV performance for some tasks. ## 10 Ethics Statement In this paper, we present a novel architecture for sequential data processing and prove its effectiveness by building a series of LLMs trained on publicly released pretraining data Gao et al. (2020); Biderman et al. (2022) and later fine-tuned on publicly available instructions Taori et al. (2023); Chaudhary (2023); Cheung (2023); Anand et al. (2023); Anonymous (2023); Yang (2023); Ji et al. (2023a, b). As a novel architecture for sequential data, RWKV has the potential to improve sequence-based models across different applications ranging from natural language processing to biomedical data processing or climate modelling. Since the training code is released open source, RWKV contributes to the democratization of AI, levels the playing field, and empowers members of the Open Source community to inspect, study, and finetune RWKV in particular tasks. Moreover, it contributes to advancing the understanding of LLMs capabilities and limitations. A significant amount of work has been devoted to increasing the efficiency of RWKV training so as to minimize its cost and promote accessibility. As LLMs trained on public data, RWKV’s lower inference cost compared to Transformer alternatives makes it more suitable for deployment in consumer and edge hardware, which is a step towards the democratization and distribution of LLMs to the general public, creating better privacy and ownership incentives. It also lowers the resource barrier to Chat assistants and text generation for small and/or underrepresented communities. PreTrained model weights for different sizes ranging from 0.1B to 14B parameters trained on multiple languages are released to increase ease of adoption and allow for the study of emergent phenomena. On the other hand, with lower resource barriers, the spreading of AI-generated text might become more prevalent. Current RWKV LLMs may exhibit and/or reproduce biases and potentially harmful content present in the data used for training. Nonetheless, mitigation and finetuning strategies discussed for other, large Transformer models should be applicable to RWKV as well. ## Acknowledgements We thank StabilityAI for the compute used to train our models and for technical support in development of RWKV. We also thank the members of the RWKV and EleutherAI Discord servers for their help and work on further extending the applicability of RWKV to different domains. ## References - Alam et al. (2023) Mohammad Mahmudul Alam, Edward Raff, Stella Biderman, Tim Oates, and James Holt. 2023. Recasting self-attention with holographic reduced representations. arXiv preprint arXiv:2305.19534. - Albalak et al. (2022) Alon Albalak, Yi-Lin Tuan, Pegah Jandaghi, Connor Pryor, Luke Yoffe, Deepak Ramachandran, Lise Getoor, Jay Pujara, and William Yang Wang. 2022. FETA: A benchmark for few-sample task transfer in open-domain dialogue. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10936–10953, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. - Anand et al. (2023) Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. 2023. Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo. https://github.com/nomic-ai/gpt4all. - Anonymous (2023) Anonymous. 2023. Sharegpt_vicuna_unfiltered. - Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. - Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650, Online. Association for Computational Linguistics. - Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150. - Biderman et al. (2022) Stella Biderman, Kieran Bicheno, and Leo Gao. 2022. Datasheet for the pile. arXiv preprint arXiv:2201.07311. - Biderman et al. (2023a) Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. 2023a. Emergent and predictable memorization in large language models. arXiv preprint arXiv:2304.11158. - Biderman et al. (2023b) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023b. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373. - Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence. - Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. URL: https://doi. org/10.5281/zenodo, 5297715. - Black et al. (2022) Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-source autoregressive language model. In Proceedings of BigScience Episode $\backslash$ # 5–Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95–136. - Bradbury et al. (2017) James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2017. Quasi-recurrent neural networks. In ICLR. - Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. - Bulatov et al. (2023) Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. 2023. Scaling transformer to 1m tokens and beyond with rmt. - Bulatov et al. (2022) Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:11079–11091. - Chaudhary (2023) Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca. - Cheung (2023) Joseph Cheung. 2023. Guanacodataset. - Choromanski et al. (2020) Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2020. Rethinking attention with performers. - Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311. - Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Deep Learning and Representation Learning Workshop. - Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044. - Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. In arXiv:1803.05457. - Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. In arXiv, volume abs/2110.14168. - Dao et al. (2022a) Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. 2022a. Flashattention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems. - Dao et al. (2022b) Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. 2022b. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052. - Demszky et al. (2020) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan S. Cowen, Gaurav Nemade, and Sujith Ravi. 2020. Goemotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4040–4054. Association for Computational Linguistics. - Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. - Gu et al. (2021) Albert Gu, Karan Goel, and Christopher Ré. 2021. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396. - Gu et al. (2022) Albert Gu, Karan Goel, and Christopher Ré. 2022. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR). - Guo et al. (2022) Mandy Guo, Joshua Ainslie, David C Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. Longt5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 724–736. - Gupta et al. (2022) Ankit Gupta, Albert Gu, and Jonathan Berant. 2022. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982–22994. - He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. - Henighan et al. (2020) Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701. - Hochreiter (1998) Sepp Hochreiter. 1998. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02):107–116. - Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. - Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. - Ismail Fawaz et al. (2019) Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. 2019. Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4):917–963. - Jaegle et al. (2021) Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. 2021. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651–4664. PMLR. - Jang et al. (2019) Hanhwi Jang, Joonsung Kim, Jae-Eon Jo, Jaewon Lee, and Jangwoo Kim. 2019. Mnnfast: A fast and scalable system architecture for memory-augmented neural networks. In Proceedings of the 46th International Symposium on Computer Architecture, pages 250–263. - Ji et al. (2023a) Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Baochang Ma, and Xiangang Li. 2023a. Belle: Be everyone’s large language model engine. https://github.com/LianjiaTech/BELLE. - Ji et al. (2023b) Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. 2023b. Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases. arXiv preprint arXiv:2303.14742. - Johannes Welbl Nelson F. Liu (2017) Matt Gardner Johannes Welbl Nelson F. Liu. 2017. Crowdsourcing multiple choice science questions. In DOI:10.18653/v1/W17-4413. - Jumper et al. (2021) John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, and et al. 2021. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589. - Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. - Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156–5165. PMLR. - Kitaev et al. (2020) Nikita Kitaev, L. Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. ArXiv, abs/2001.04451. - Kocoń et al. (2023) Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocoń, Bartłomiej Koptyra, Wiktoria Mieleszczenko-Kowszewicz, Piotr Miłkowski, Marcin Oleksy, Maciej Piasecki, Łukasz Radliński, Konrad Wojtasik, Stanisław Woźniak, and Przemysław Kazienko. 2023. Chatgpt: Jack of all trades, master of none. Information Fusion, page 101861. - Kocoń et al. (2019) Jan Kocoń, Piotr Miłkowski, and Monika Zaśko-Zielińska. 2019. Multi-level sentiment analysis of polemo 2.0: Extended corpus of multi-domain consumer reviews. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 980–991. - Le and Zuidema (2016) Phong Le and Willem Zuidema. 2016. Quantifying the vanishing gradient and long distance dependency problem in recursive neural networks and recursive lstms. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 87–93. - Le Scao et al. (2022) Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Jason Phang, Ofir Press, et al. 2022. What language model to train if you have one million gpu hours? In Proceedings of BigScience Episode #5–Workshop on Challenges & Perspectives in Creating Large Language Models. - Lei et al. (2018) Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. 2018. Simple recurrent units for highly parallelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4470–4481, Brussels, Belgium. Association for Computational Linguistics. - Liu et al. (2021) Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to mlps. - Ma et al. (2021) Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. Luna: Linear unified nested attention. Advances in Neural Information Processing Systems, 34:2441–2453. - Ma et al. (2023) Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. 2023. Mega: Moving average equipped gated attention. In ICLR. - Martin and Cundy (2017) Eric Martin and Chris Cundy. 2017. Parallelizing linear recurrent neural nets over sequence length. ArXiv, abs/1709.04057. - Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36. - Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP. - Muennighoff et al. (2023) Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling data-constrained language models. arXiv preprint arXiv:2305.16264. - OpenAI (2022) OpenAI. 2022. Introducing chatgpt. https://openai.com/blog/chatgpt. - Orvieto et al. (2023) Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023. Resurrecting recurrent neural networks for long sequences. arXiv preprint arXiv:2303.06349. - Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525–1534, Berlin, Germany. Association for Computational Linguistics. - Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. - Poli et al. (2023) Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. 2023. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866. - Price et al. (2020) Ilan Price, Jordan Gifford-Moore, Jory Flemming, Saul Musker, Maayan Roichman, Guillaume Sylvain, Nithum Thain, Lucas Dixon, and Jeffrey Sorensen. 2020. Six attributes of unhealthy conversations. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 114–124, Online. Association for Computational Linguistics. - Rabe and Staats (2022) Markus N. Rabe and Charles Staats. 2022. Self-attention does not need $o(n^2)$ memory. - Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 3505–3506, New York, NY, USA. Association for Computing Machinery. - Roemmele et al. (2018) Melissa Roemmele, Cosmin Adrian Bejan, , and Andrew S. Gordon. 2018. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI. - Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. - Siddiqui (2019) Ramsha Siddiqui. 2019. SARCASMANIA: Sarcasm Exposed! http://www.kaggle.com/rmsharks4/sarcasmania-dataset. [Online; accessed 02-February-2023]. - So et al. (2021) David R. So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. 2021. Primer: Searching for efficient transformers for language modeling. CoRR, abs/2109.08668. - Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. - Tay et al. (2020) Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. 2020. Synthesizer: Rethinking self-attention in transformer models. - Tay et al. (2021) Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations. - Tay et al. (2022) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1–28. - Tolstikhin et al. (2021) Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. 2021. Mlp-mixer: An all-mlp architecture for vision. CoRR, abs/2105.01601. - Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. - Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. - Vilares and Gómez-Rodríguez (2019) David Vilares and Carlos Gómez-Rodríguez. 2019. Head-qa: A healthcare dataset for complex reasoning. In ACL. - Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. - Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. - Wang et al. (2020) Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. - Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. ArXiv, abs/2206.07682. - Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. - Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. pages 38–45. Association for Computational Linguistics. - Wu et al. (2020) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4–24. - Wulczyn et al. (2017) Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pages 1391–1399. ACM. - Yang (2023) Jianxin Yang. 2023. Firefly. https://github.com/yangjianxin1/Firefly. - Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33. - Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In ACL. - Zellers et al. (2020) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In ACL. - Zhai et al. (2021) Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. 2021. An attention free transformer. - Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. In arXiv:1810.12885. - Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. ## Appendix A Author Contributions All authors contributed to the drafting of this paper. Eric Alcaide and Quentin Anthony organized the paper and its experiments and were involved in all phases of the development process. Model Design and Development Bo Peng (lead), Matteo Grella, Xuzheng He, Haowen Hou, Jiaming Kong, Johan S. Wind Model Training Bo Peng Scaling Laws Analysis Stella Biderman, Bo Peng Benchmark Evaluations Stella Biderman (lead), Kranthi Kiran GV, Krishna Sri Ipsit Mantri, Atsushi Saito, Qihang Zhao, Peng Zhou, Rui-Jie Zhuåç Long Context Experiments Xingjian Du, Rui-Jie Zhu, Bolun Wang, Ruichong Zhang, Jian Zhu, Rui-Jie Zhu Inference Speed Experiments Samuel Arcadinho, Przemysław Kazienko, Qinghua Zhou Information Flow Experiments Huanqi Cao, Michael Chung, Matteo Grella, Ferdinand Mom, Zhenyuan Zhang Chat Experiments Jan Kocoń (lead), Przemysław Kazienko, Bartłomiej Koptyra, Hayden Lau, Xiangru Tang, Stanisław Woźniak, Zhenyuan Zhang Ethics and Broader Impacts Stella Biderman, Guangyu Song ## Appendix B Author Contributions Bo Peng Original RWKV idea, original code, performance optimizations, original experiments, and trained RWKV models from 0.1B to 14B. Eric Alcaide Manuscript (initial draft sections 1, C; sections 3, 7 and 8; revision and proofreading; final version ). Figures (2, 3, 3, 8). Experiments section 6. Appendices E, K. Contributions to Appendix M. Quentin Anthony Manuscript (organization, initial draft sections 1, C, 2; revision and proofreading; final version). Alon Albalak Manuscript (abstract and sections 1, 9, and 7; proofreading and revision). Samuel Arcadinho Contributions to Figures 7, 13, and 14. Contributions to Appendix K. Stella Biderman Performed the scaling laws analysis and evaluated competitor models on benchmark tasks. Huanqi Cao Manuscript (contributions to 3.2 and 3.3; proofreading and revision). Experiments for Appendix I. Xin Cheng Manuscript (proofreading and revision). Contributions to Appendix M, J. Michael Chung Manuscript (contributions to section I; proofreading and revision). Xingjian Du Evaluation on Long Range Arena Benchmark (TBD until 5.31). Matteo Grella Manuscript (sections H, I, 8; contributions to sections 1, 7 and 9; proofreading and revision). Contributions to Appendix D. Kranthi Kiran GV Manuscript (sections C and 5; contributions to section 2; revision and proofreading). Tables K and K. Appendix 4. Xuzheng He Manuscript (contributions to section 2; proofreading and revision). Contributions to Figure 8. Appendix I. Contributions to appendix H. Haowen Hou Figure 9. Appendix F. Jiaju Lin RWKV on LRA benchmarking Przemysław Kazienko Manuscript (proofreading and revision). Contributions to Section 6, 9, and Appendix L. Jan Kocon Manuscript (Section 1; proofreading and revision). Contributions to Appendix L. Jiaming Kong Manuscript (revision and proofreading). Appendix H. Bartłomiej Koptyra Manuscript (revision and proofreading) Contributions to Appendix L. Hayden Lau Manuscript (contributions to section 1 and 9; proofreading and revision). Contributions to Appendix M. Krishna Sri Ipsit Mantri Figure 12 Ferdinand Mom Manuscript (contributions to section 1, C, 3.3, I; proofreading and revision). Contributions to Appendix D. Atsushi Saito Manuscript (sections 2 and 5; contributions to section C). Contributions to Appendix J Guangyu Song Manuscript (rewrote section 3; final version). Initial draft Ethics Statement). Xiangru Tang Manuscript (sections C and 2; contributions to abstract; revision and proofreading). Contributions to Appendix M. Bolun Wang Contributions to Tables 1. Johan S. Wind RWKV performance optimizations (CUDA), Contributions to Appendix 4. Stanisław Woźniak Contributions to Appendix L. Ruichong Zhang Manuscript (proofreading and revision); Contributions to Figure 6 and Appendix M. Zhenyuan Zhang Manuscript (revision and proofreading). Figure 3. Experiments Appendix I. Contributions to Appendices D and M. Qihang Zhao Manuscript (proofreading and revision). Contributions to Table 5. Peng Zhou Contributions to Tables 1 and Table 5. Qinghua Zhou Manuscript (Proofreading and revision of section 3; Add missing citations in 3.3). Revision of Figures 2 and 12. Jian Zhu Manuscript (section C; proofreading and revision). Figures 3 and 6. Rui-Jie Zhu Tables 1 and 5. Experiments for table 5. ## Appendix C Additional Related Work Recently, a number of techniques have been proposed to address the limitations of transformers. Optimizing Attention Mechanism Many transformer variants (“x-formers”) have been introduced to reduce the complexity of transformers Tay et al. (2022), including sparse attention Beltagy et al. (2020); Kitaev et al. (2020); Guo et al. (2022), approximating the full attention matrix Wang et al. (2020); Ma et al. (2021); Choromanski et al. (2020), combining chunked attention with gating Ma et al. (2023) and other efficient methods Katharopoulos et al. (2020); Jaegle et al. (2021). Some recent works like FlashAttention Dao et al. (2022a) and others Rabe and Staats (2022); Jang et al. (2019) share similarities with RWKV’s chunked computation scheme. Despite being memory-efficient, their time complexity remains quadratic or contains chunk size as a hidden factor. In contrast, RWKV achieves better space and time complexity during inference by formulating a linear attention as an RNN. Attention Free Models Another line of research replaces the attention mechanism with other modules to scale to long sequences. MLP-Mixer and others Tolstikhin et al. (2021); Liu et al. (2021) propose replacing attention by Multi-Layer Perceptrons (MLPs) in computer vision tasks. The Attention Free Transformer (AFT) (Zhai et al., 2021) and HrrFormer (Alam et al., 2023) replaces dot-product self-attention with a computationally efficient alternative. None of these models have been successfully scaled to the point where drawing comparisons with transformer-based large language models makes sense. There has also been substantial research into state space models (SSM) (Gu et al., 2021) and its variants (Dao et al., 2022b; Gupta et al., 2022; Poli et al., 2023). In contrast to the preceding models, SSM and its successors have shown substantial progress towards efficient scaling. Simultaneously with this work, Poli et al. (2023) train SSM-based models with 125 million and 355 million parameters and show that the performance is on-par with a transformer that uses a mix of local and global attention (Black et al., 2021). Advances in RNNs Inspired by the success of transformers, RNN-style Hochreiter and Schmidhuber (1997); Chung et al. (2014) recursive components have also been modified to increase context length, such as the Recurrent Memory Transformer Bulatov et al. (2022, 2023) and Linear Recurrent Units Orvieto et al. (2023). Most similar to our work, the Quasi-Recurrent neural network (QRNN) Bradbury et al. (2017) uses both convolutional layers and recurrent pooling functions across timesteps and channels. While QRNN utilizes convolutional filters with fixed sizes, RWKV employs a time-mixing module as an attention mechanism with time-decaying factors. Different from the element-wise pooling in QRNN, RWKV includes a parametrized channel-mixing module that is parallelizable. ## Appendix D Time-Mixing Block as an RNN Cell As stated in 3.3, the RWKV time-mixing block can be formulated as an RNN, as the $WKV$ computation can be written in such a recursive form: $$ \displaystyle a_0,b_0 \displaystyle=0, \displaystyle wkv_t \displaystyle=\frac{a_t-1+e^u+k_t\odot v_t}{b_t-1+e^u+k_t}, \displaystyle a_t \displaystyle=e^-w\odot a_t-1+e^k_t\odot v_t, \displaystyle b_t \displaystyle=e^-w\odot b_t-1+e^k_t. \tag{19} $$ <details> <summary>x14.png Details</summary>  ### Visual Description ## Diagram: Recurrent Neural Network (RNN) Cell Architecture ### Overview This diagram illustrates the computational flow of a standard RNN cell, showing how hidden states (`h_t-1`, `h_t`), inputs (`x_t-1`, `x_t`), and outputs (`o_t`) interact through gating mechanisms. The diagram includes activation functions, weights, and element-wise operations. ### Components/Axes - **Inputs**: - `x_t-1` (previous input) and `x_t` (current input), represented as yellow and orange circles, respectively. - **Hidden States**: - `h_t-1` (previous hidden state) and `h_t` (current hidden state), labeled in black text. - **Gates**: - **Input Gate (1)**: Red circle, computes `σ(wkv_t * [h_t-1; x_t])`. - **Forget Gate (2)**: Blue circle, computes `σ(wkv_t * [h_t-1; x_t])`. - **Output Gate (3)**: Pink circle, computes `σ(wkv_t * [h_t-1; x_t])`. - **Activation Functions**: - `σ` (sigmoid, orange circle), `μ` (mean, yellow circle), `e` (element-wise, purple circle). - **Weights**: - `wkv_t` (shared weights for gates), labeled in black text. - **Output**: - `o_t` (current output), labeled in black text. ### Detailed Analysis 1. **Flow of Information**: - `h_t-1` and `x_t-1` are inputs to the RNN cell. - `x_t` is combined with `h_t-1` via element-wise operations (`e`) and weighted sums (`wkv_t`). - Gates (1, 2, 3) modulate the flow of information using sigmoid (`σ`) and mean (`μ`) activation functions. - The output `o_t` is derived from the weighted combination of gate outputs. 2. **Key Equations**: - Input Gate: `σ(wkv_t * [h_t-1; x_t])` - Forget Gate: `σ(wkv_t * [h_t-1; x_t])` - Output Gate: `σ(wkv_t * [h_t-1; x_t])` ### Key Observations - **Gating Mechanism**: The three gates (input, forget, output) control the retention and update of information in the hidden state. - **Activation Functions**: - `σ` (sigmoid) ensures outputs are between 0 and 1, critical for gate operations. - `μ` (mean) and `e` (element-wise) handle input transformations. - **Weight Sharing**: `wkv_t` is shared across all gates, reducing parameter count. ### Interpretation This diagram represents a **Gated Recurrent Unit (GRU)** or **Long Short-Term Memory (LSTM)** cell, depending on the specific gating logic. The gates address the vanishing gradient problem by regulating information flow: - The **forget gate** (`2`) decides what information to discard from the cell state. - The **input gate** (`1`) determines what new information to store. - The **output gate** (`3`) controls what parts of the cell state to output. The use of shared weights (`wkv_t`) and activation functions (`σ`, `μ`, `e`) ensures efficient computation while maintaining temporal dependencies. This architecture is foundational for sequence modeling tasks like language processing and time-series prediction. **Note**: No numerical values or trends are present in the diagram; it is a structural representation of an RNN cell. </details> Figure 8: RWKV time-mixing block formulated as an RNN cell. Color codes: yellow ( $μ$ ) denotes the token shift, red (1) denotes the denominator, blue (2) denotes the numerator, and pink (3) denotes the fraction computations in 16. $h$ denotes the numerator-denominator tuple. The dataflow of the RNN-like time-mixing is shown in Fig. 8, where the hidden states $h$ is the numerator-denominator tuple $(a,b)$ . To avoid overflow in calculating $e^k_t$ , a numerical trick is used in the official implementation. Noticing that $a_1=e^k_1\odot v_1$ and $b_1=e^k_1$ , we set $a^\prime_1=v_1,b^\prime_1=1,p_1=k_1$ , where $p_t$ stores the shared exponents of $a_t$ and $b_t$ . Now the above recursion can be converted into a numerical safe version, for each time step $t>1$ : $$ \displaystyle q \displaystyle:=\max(p_t-1,u+k_t), \displaystyle wkv_t \displaystyle=\frac{e^p_t-1-q\odot a^\prime_t-1+e^u+k_t-q\odot v_ t}{e^p_t-1-q\odot b^\prime_t-1+e^u+k_t-q}. \tag{23} $$ The update to $a^\prime_t,b^\prime_t$ , and their shared exponent is also carried out in a similar fashion: $$ \displaystyle q^\prime \displaystyle:=\max(p_t-1-w,k_t), \displaystyle a^\prime_t \displaystyle=e^p_t-1-w-q^{\prime}\odot a^\prime_t-1+e^k_t-q^{ \prime}\odot v_t, \displaystyle b^\prime_t \displaystyle=e^p_t-1-w-q^{\prime}\odot b^\prime_t-1+e^k_t-q^{ \prime}, \displaystyle p_t \displaystyle=q^\prime. \tag{25} $$ The RWKV model has an internal state that stores some previous information. In each layer, the internal state consists five parts, each of which is a vector with $D$ numbers, where $D$ is the model dimension. The five parts are: - The current input of the Time-mix block $x_t$ ; - The current input of the Channel-mix block $y_t$ ; - The numerator of the $WKV$ value $a^\prime_t$ , as defined in equation (26); - The denominator of the $WKV$ value $b^\prime_t$ , as defined in equation (27); - An auxiliary state $p_t$ in (28), which is used for $WKV$ computation to maintain numerical precision. Which yields a total size of $5DL$ parameters. It is worth noting that in an algebraic context with infinite precision, the helper state $p_t$ can be ignored, and the $WKV$ numerator and denominator can be computed directly using equations (21) and (22), reducing the size of the internal state to $4DL$ . ## Appendix E Parameter initializations We describe the specific parameter initializations below and motivate the design choices. Parameters belonging to residual blocks are often adjusted by layer depth and total number of layers. Let $\#$ denote the vocabulary size, $s$ denote the embedding dimension, $d$ denote the hidden size (we use $d=4s$ ), $L$ the number of layers, $l$ the layer index (from 0 to $L-1$ ), we use the following initializations: - Embeddings are initialized to $U$ ( $±$ $1×{10}^-4$ ) as explained in 3.4 - For the time-mixing blocks (11, 12, 13), initializations are $μ_k_{i}=(\frac{i}{s})^1-\frac{l{L}}$ , $μ_v_{i}=(\frac{i}{s})^1-\frac{l{L}}+\frac{0.3l}{L-1}$ and $μ_r_{i}=\frac{1}{2}·(\frac{i}{s})^1-\frac{l{L}}$ - For the channel-mixing blocks (14, 15), $μ_k_{i}$ and $μ_r_{i}$ are initialized to $(\frac{i}{s})^1-\frac{l{L}}$ - $w_i$ (16), also known as “time decay”, is initialized to $-5+8·(\frac{i}{d-1})^0.7+\frac{1.3l{L-1}}$ . Intuitively, it is the discount factor applied to previous tokens over time. - $u_i$ (16), also known as “bonus”, is set to $0.5·(((i+1)\mod 3)-1)+\log 0.3$ . It is the special weighting applied to the current token in equation 16. The alternating zigzag pattern initially creates subtle variations in the tensor elements, which are intended to help the model treat different dimensions of the embedding distinctively. - $W_o$ (17) (time-mixing) and $W_v$ (channel-mixing) are initialized to $N(0,√{\frac{d}{s}}=2)$ - All other $W_r,W_k,W_v$ weights are initialized to 0 so the model can start learning from the beginning without noisy signals. - All LayerNorm weights start from 1 and biases from 0. ## Appendix F Small Init Embedding This section presents the experimental validation of small initialization embedding. The experimental setup is as follows. In the baseline configuration, the parameters are initialized using a normal distribution with a mean of 0.0 and a standard deviation of 0.02, which is a commonly used initialization method in models like BERT and GPT. On the other hand, in the small initialization of the embedding (small init emb) experiment, the parameters are initialized using a uniform distribution with a range of 1e-4, which is slightly different from RWKV where a normal distribution with a standard deviation of 1e-4 is used. However, this difference is negligible and does not affect our conclusions. The experiments were conducted with a batch size of 400. As depicted in Figure 9, the loss curve for the small init emb exhibits a faster rate of decrease and convergence compared to the traditional initialization using a normal distribution. <details> <summary>x15.png Details</summary>  ### Visual Description ## Line Graph: Loss vs.Training Steps ### Overview The image depicts a line graph comparing the loss reduction of two training methods ("Baseline" and "Small Init Emb") over 50,000 training steps. Both lines show a decreasing trend, with the "Small Init Emb" method consistently achieving lower loss values than the "Baseline" method. ### Components/Axes - **X-axis (Horizontal)**: Labeled "Step," ranging from 0 to 50,000 in increments of 10,000. - **Y-axis (Vertical)**: Labeled "Loss," ranging from 4 to 11 in increments of 1. - **Legend**: Located in the top-right corner, with: - **Blue line**: "Baseline" - **Orange line**: "Small Init Emb" ### Detailed Analysis 1. **Baseline (Blue Line)**: - Starts at approximately **10.8 loss** at step 0. - Declines sharply initially, reaching ~7.5 loss by 10,000 steps. - Gradually decreases to ~4.7 loss by 50,000 steps. - Exhibits minor fluctuations (e.g., ~5.3 loss at 30,000 steps). 2. **Small Init Emb (Orange Line)**: - Starts slightly higher at ~10.9 loss at step 0. - Declines more steeply than the Baseline, reaching ~6.5 loss by 10,000 steps. - Continues to drop, stabilizing around ~4.2 loss by 50,000 steps. - Shows smoother convergence with fewer fluctuations compared to Baseline. ### Key Observations - Both methods show a **monotonic decrease** in loss, but "Small Init Emb" achieves lower loss values at every step. - The gap between the two lines narrows over time, suggesting convergence in performance at later stages. - The "Small Init Emb" method demonstrates **~20-30% faster initial loss reduction** compared to Baseline. ### Interpretation The data suggests that initializing embeddings with smaller dimensions ("Small Init Emb") improves training efficiency, as evidenced by faster and more consistent loss reduction. This could indicate that smaller initial embeddings help the model generalize better or reduce overfitting early in training. The convergence of the two lines at later steps implies that both methods eventually reach similar performance levels, but "Small Init Emb" maintains a persistent advantage. The trends align with common machine learning principles where initialization strategies significantly impact training dynamics. </details> Figure 9: Effect of small initialization embedding. ## Appendix G Hyperparameters | Model Init LR Warmup Mini-Epochs | 169M 0.0006 361 | 430M 0.0004 411 | 1.5B 0.0003 443 | 3B 0.00015 451 | 7B 0.00015 465 | 14B 0.0001 544 | | --- | --- | --- | --- | --- | --- | --- | | End LR | 0.00001 | 0.00001 | 0.00001 | 0.00001 | 0.00001 | 0.000007 | Table 3: Hyperparameters for our learning rate (LR) schedule of the pretrained models. To train the models mentioned, we use $ε=(0.9,0.99)$ without weight decay for the Adam optimizer, and switch batch size dynamically between 128 or 256 sequences, each of 1024 tokens. We further organize the training into multiple mini-epochs, each of 40320 samples, to guide our learning rate schedule. The training process takes 8043 mini-epochs to make one pass over the Pile. The initial warming up mini-epochs have a constant learning rate of “Init LR”. After the warming up mini-epochs, the learning rate exponentially decays until in the last mini-epoch, in which the model finishes training on the entire Pile, the learning rate arrives at the “End LR”. The related hyperparameters are shown in Table 3. ## Appendix H Gradient Stability in RWKV In this section, we present a mathematical description of the gradient stability property in RWKV, focusing specifically on the time-mixing block. By gradient stability we mean that if the inputs $x_t$ are bounded and the model parameters are fixed, then the gradients with respect to $W_k$ and $W_v$ are uniformly bounded for all $T$ (thus not exploding). Consequently, we can control the amount each $x_t$ contributes to the gradient at $T$ in a naturally decaying fashion by the weight decay mechanism $w$ (thus not vanishing unless desired). First, we make the simplification that there are no token shifts, this will not affect the final conclusion. In this scenario, $wkv_T$ can be written as $$ \displaystyle wkv_T=\frac{∑_t=1^TK^e_t\odot v_t}{∑_t=1^T K^e_t}=E(v_t)=\frac{S(v_t)}{S(1)}, \tag{1} $$ where $$ v_t=W_vx_t, \frac{∂(v_t)_i}{∂(W_v)_i,j}=(x_t )_j, $$ $$ K^e_t=e^W_kx_t+w_T,t, \frac{∂(K^e_t)_i}{∂( W_k)_i,j}=(x_t)_j(K^e_t)_i, $$ and $S(·)$ and $E(·)$ are shorthand for denoting sums and averages over weights $K^e_t$ . The loss function at position $T$ can be written as $$ \displaystyle L_T=l(f(wkv_T),y_T). \tag{30} $$ Because $wkv_T$ relates to $(W_k)_i,j$ and $(W_v)_i,j$ only through the $i$ -th channel $(wkv_T)_i$ , we have $$ \displaystyle\frac{∂ L_T}{∂(W_v)_i,j}=\frac{∂ L_T }{∂(wkv_T)_i}\frac{∂(wkv_T)_i}{∂(W_v)_i,j}. \tag{31} $$ The first part of the above equation contains trivial operations like output layers, and other layers of time-mixing, which can be proven inductively. The second part of the above equation can be bounded as $$ \displaystyle≤ft|\frac{∂(wkv_T)_i}{∂(W_v)_i,j}\right| \displaystyle=≤ft|\frac{∂E_i[(v_t)_i]}{∂(W_v) _i,j}\right| \displaystyle=|E_i[(x_t)_j]|≤\max_t|(x_t)_j|, \tag{32} $$ which is irrelevant to $T$ . Similarly, $$ \displaystyle\frac{∂(wkv_T)_i}{∂(W_k)_i,j} \displaystyle=∂\frac{S_i[(v_t)_i]}{S_i(1)}/ ∂(W_k)_i,j \displaystyle=\frac{S_i[(x_t)_j(v_t)_i]}{S_i(1)} -\frac{S_i[(x_t)_j]S_i[(v_t)_i]}{S_i( 1)^2} \displaystyle=E_i[(x_t)_j(v_t)_i]-E_i[(x_t)_j ]E_i[(v_t)_i] \displaystyle=cov_i((x_t)_j,(v_t)_i) \tag{1} $$ can also be bounded. Note that $wkv$ ’s softmax operation contains at least two non-zero terms ( $u$ and $w$ ), so the above “covariance” will not degenerate into 0. ## Appendix I Model Behavior Visualization <details> <summary>x16.png Details</summary>  ### Visual Description ## Line Chart:Time Decay (sorted along channel axis) ### Overview The chart displays time decay trends across 12 layers (Layer 1 to Layer 12) as a function of channel index (0–800). Each layer is represented by a distinct colored line, with decay values normalized between 0.0 and 1.0. Lines are sorted along the channel axis, with Layer 1 (dark purple) at the top and Layer 12 (yellow) at the bottom. ### Components/Axes - **X-axis (Channel)**: Linear scale from 0 to 800, labeled "Channel." - **Y-axis (Time Decay)**: Linear scale from 0.0 to 1.0, labeled "Time Decay." - **Legend**: Positioned in the bottom-left corner, listing layers 1–12 with corresponding colors (dark purple to yellow). - **Lines**: 12 distinct lines, each representing a layer. Colors transition from dark purple (Layer 1) to yellow (Layer 12). ### Detailed Analysis - **Layer 1 (dark purple)**: Starts at 1.0, decays gradually to ~0.2 by channel 800. Minimal decay in channels 0–200, sharp drop between 200–400, and gradual decline thereafter. - **Layer 2 (medium purple)**: Similar to Layer 1 but steeper decay. Reaches ~0.4 by channel 800. - **Layers 3–9 (blue to teal)**: Intermediate decay rates. Layers 3–5 decay to ~0.6–0.8 by channel 800; Layers 6–9 decay to ~0.4–0.6. - **Layer 10 (light green)**: Steeper decay than Layers 3–9. Drops to ~0.3 by channel 800. - **Layer 11 (lime green)**: Rapid decay; reaches ~0.2 by channel 800. - **Layer 12 (yellow)**: Sharpest decay. Drops to ~0.1 by channel 800. ### Key Observations 1. **Decay Rate Correlation**: Higher-numbered layers (10–12) exhibit significantly faster decay than lower-numbered layers (1–9). 2. **Channel Threshold**: All layers show minimal decay (<0.2) in channels 0–100. A critical threshold occurs between channels 300–500, where decay accelerates sharply. 3. **Layer Ordering**: Lines are sorted by decay rate, with Layer 12 (yellow) having the steepest slope and Layer 1 (dark purple) the gentlest. 4. **Normalization**: All lines start at 1.0, indicating 100% time decay at channel 0. ### Interpretation The data suggests that time decay is inversely proportional to layer number, with higher layers (10–12) experiencing rapid degradation in channel performance. The sharp decay threshold between channels 300–500 implies a structural or operational limit in the system, beyond which additional channels contribute minimally to time decay. Layer 12’s extreme sensitivity (yellow line) may indicate a critical failure mode or resource exhaustion point. The sorting of layers along the channel axis could reflect prioritization of channels by decay resilience, with Layer 1 (dark purple) representing the most stable configuration. </details> Figure 10: Model behavior visualizations of RWKV. The right plot illustrates the time decays ( $e^-w$ ) in each layer of the RWKV-169M model, sorted along the channel axis. Notably, several decays in the last layers are very close or equal to one, implying that certain information is preserved and propagated throughout the model’s temporal context. Meanwhile, many decays in the initial layer are close to zero, which corresponds to local operations in $wkv$ (16), likely to be associated with tasks such as text parsing or lexical analysis. (Note that the local operations in $wkv$ are due to the extra parameter $u$ , when $e^-w$ is degenerated into 0.) These patterns of time decays are partly learned, but also come from parameter initialization as it speeds up training. The plot below shows the information retrieval and propagation path in the RWKV-430M model. The experiment follows the causal trace method introduced by Meng et al. (2022), where we 1. Run the model once, and record all states and activation of each layer during the computation; 1. Corrupt the input embeddings of the subject using noise (“The Eiffel Tower” in this example); 1. Restore the states and activation of a certain layer at a certain token during the computation, and record the log-probability of the model outputting the correct answer (“Paris”). Unlike transformers, RWKV relies on the recursive propagation of information in the time dimension. In this case, the fact that the Eiffel Tower is located in Paris is retrieved in layer 4 just after the model sees “The Eiffel”. It is then passed down to the subsequent layers. In layer 20, mostly, the information is propagated through time until reaching where it is needed. Finally, at the token “of”, it is passed down to the last layer for outputting the answer. <details> <summary>x17.png Details</summary>  ### Visual Description ## Heatmap: Information propagation path ### Overview The image is a heatmap visualizing the log-probability of the word "Paris" across different layers and linguistic components. Darker purple shades represent lower log-probability values (more negative), while lighter shades indicate higher values (closer to zero). ### Components/Axes - **X-axis (Layer)**: Discrete numerical values at positions 1, 6, 11, 16, and 21. Represents sequential processing layers. - **Y-axis (Linguistic components)**: Phrases stacked vertically: 1. "The" 2. "E" 3. "iff" 4. "el" 5. "Tower" 6. "is" 7. "located" 8. "in" 9. "the" 10. "city" 11. "of" - **Color scale (Log-probability of "Paris")**: Ranges from -1 (lightest purple) to -7 (darkest purple), with intermediate values at -2, -3, -4, -5, -6. ### Detailed Analysis - **Layer 1**: All cells are light purple, indicating log-probabilities near -1. - **Layer 6**: Gradual darkening from "The" to "el," with "el" reaching -3. - **Layer 11**: Darkest cells in "el" (-4) and "Tower" (-3). - **Layer 16**: "Tower" (-4), "is" (-3), and "located" (-2). - **Layer 21**: Darkest cells in "city of" (-7), "the" (-6), and "located" (-5). ### Key Observations 1. **Trend**: Log-probability decreases (becomes more negative) as layers increase, particularly in later layers (16–21). 2. **Critical regions**: - "city of" consistently shows the lowest values (-7 at layer 21). - "el" and "Tower" exhibit significant drops in layers 6–11. 3. **Anomalies**: - "iff" and "E" remain relatively light across all layers. - "is" and "in" show moderate darkening only in later layers. ### Interpretation The heatmap suggests that the word "Paris" becomes increasingly improbable as linguistic components are processed in deeper layers. The darkest cells in "city of" (layer 21) imply that phrases describing spatial relationships (e.g., "located in the city of") are least likely to include "Paris" in later processing stages. This could reflect a model's prioritization of local context over global semantic relationships in later layers. The persistence of light shades in "iff" and "E" indicates these components retain higher uncertainty about "Paris" throughout processing. The structured darkening pattern aligns with typical attention mechanisms in transformer models, where early layers capture syntactic features while later layers focus on semantic disambiguation. </details> Figure 11: Model behavior visualizations of the RWKV model. ## Appendix J Additional Evaluations ### J.1 Further details on NLP tasks We evaluate on the following tasks: ARC Clark et al. (2018) A dataset designed for multiple-choice question answering, encompassing science exam questions ranging from third grade to ninth grade. It has Easy and Challenge subsets that we report results on separately. BoolQ Clark et al. (2019) A binary yes/no question answering benchmark. COPA Roemmele et al. (2018) A dataset to evaluate achievement in open-domain commonsense causal reasoning. HeadQA Vilares and Gómez-Rodríguez (2019) A benchmark consisting of graduate-level questions encompassing various fields such as medicine, nursing, biology, chemistry, psychology, and pharmacology. HellaSwag Zellers et al. (2019) A novel benchmark for commonsense Natural Language Inference (NLI) which is build by adversarial filtering against transformer models. LAMBADA Paperno et al. (2016) A benchmark dataset that evaluates the model’s contextual reasoning and language comprehension abilities by presenting context-target pairs, where the objective is to predict the most probable target token. We follow standard practice and use the untokenized version created by OpenAI (Brown et al., 2020). OpenBookQA Mihaylov et al. (2018) A QA dataset to evaluate human comprehension of a subject by incorporating open book facts, scientific knowledge, and perceptual common sense, drawing inspiration from open book exams. PIQA Bisk et al. (2020) A benchmark for the task of physical common sense reasoning, which consists of a binary choice task that can be better understood as a set of two pairs, namely (Goal, Solution). ReCoRD Zhang et al. (2018) A benchmark for evaluating commonsense reasoning in reading comprehension by generating queries from CNN/Daily Mail news articles and requiring text span answers from corresponding summarizing passages. SciQ Johannes Welbl Nelson F. Liu (2017) A multiple-choice QA dataset which was created using an innovative approach to gather well-crafted multiple-choice questions that are focused on a specific domain. Winogrande Zellers et al. (2020) A dataset designed to evaluate the acquisition of common sense reasoning by neural language models, aiming to determine whether we are accurately assessing the true capabilities of machine common sense. <details> <summary>x18.png Details</summary>  ### Visual Description ## Line Chart: Model Accuracy vs. Compute (exaFLOP) ### Overview The chart compares the accuracy of four language models (BLOOM, Pythia, OPT, RWKV) across varying compute levels (exaFLOP). Accuracy is plotted on the y-axis (0–100), while compute is logarithmic on the x-axis (10²–10³). All models show upward trends, with RWKV achieving the highest accuracy at higher compute levels. ### Components/Axes - **X-axis**: "Compute (exaFLOP)" with logarithmic scale (10², 10³). - **Y-axis**: "Accuracy" (0–100, linear scale). - **Legend**: - Orange circle: BLOOM - Blue square: Pythia - Green triangle: OPT - Red star: RWKV - **Legend Position**: Bottom-right corner. ### Detailed Analysis 1. **BLOOM (Orange)**: - Starts at ~18 accuracy at 10² exaFLOP. - Gradual increase to ~30 accuracy at 10³ exaFLOP. - Slope: Moderate upward trend. 2. **Pythia (Blue)**: - Begins at ~18 accuracy at 10² exaFLOP. - Rises to ~32 accuracy at 10³ exaFLOP. - Slope: Steeper than BLOOM but less than RWKV. 3. **OPT (Green)**: - Starts at ~19 accuracy at 10² exaFLOP. - Increases to ~31 accuracy at 10³ exaFLOP. - Slope: Similar to Pythia but slightly flatter. 4. **RWKV (Red)**: - Begins at ~20 accuracy at 10² exaFLOP. - Sharp rise to ~38 accuracy at 10³ exaFLOP. - Slope: Steepest among all models. ### Key Observations - All models show **positive correlation** between compute and accuracy. - **RWKV** consistently outperforms others at higher compute levels. - **BLOOM** has the lowest accuracy across all compute ranges. - No model exhibits a decline in accuracy; all trends are monotonically increasing. ### Interpretation The data suggests that **compute efficiency** significantly impacts model accuracy, with RWKV demonstrating the most efficient scaling. BLOOM’s lower performance may indicate architectural limitations or training data differences. The logarithmic x-axis emphasizes that even small increases in compute (e.g., 10² to 10³) yield substantial accuracy gains, highlighting the importance of computational resources in model development. The absence of overlapping lines confirms no model overtakes another in this range, suggesting inherent differences in scalability. </details> (a) ARC (Challenge) <details> <summary>x19.png Details</summary>  ### Visual Description ## Line Graph: Model Accuracy vs. Compute (exaFLOP) ### Overview The image depicts a line graph comparing the accuracy of four computational models (A, B, C, D) across varying levels of compute power (exaFLOP). Accuracy is plotted on a linear y-axis (0–100%), while compute power is plotted on a logarithmic x-axis (10² to 10⁴ exaFLOP). All models show upward trends, with accuracy increasing as compute power grows. ### Components/Axes - **Y-Axis (Accuracy)**: Labeled "Accuracy" with ticks at 0, 20, 40, 60, 80, 100. - **X-Axis (Compute)**: Labeled "Compute (exaFLOP)" with logarithmic ticks at 10², 10³, and 10⁴. - **Legend**: Located in the top-right corner, associating: - Blue squares: Model A - Red stars: Model B - Green triangles: Model C - Orange circles: Model D ### Detailed Analysis 1. **Model A (Blue Squares)**: - Starts at ~40% accuracy at 10² exaFLOP. - Rises steadily to ~70% at 10³ exaFLOP. - Reaches ~75% at 10⁴ exaFLOP. - **Trend**: Consistent linear growth with the steepest slope. 2. **Model B (Red Stars)**: - Begins at ~45% accuracy at 10² exaFLOP. - Accelerates to ~65% at 10³ exaFLOP. - Peaks at ~72% at 10⁴ exaFLOP. - **Trend**: Slightly concave upward, with faster gains at higher compute levels. 3. **Model C (Green Triangles)**: - Starts at ~42% accuracy at 10² exaFLOP. - Gains ~18% to reach ~60% at 10³ exaFLOP. - Ends at ~68% at 10⁴ exaFLOP. - **Trend**: Gradual, near-linear improvement. 4. **Model D (Orange Circles)**: - Begins at ~40% accuracy at 10² exaFLOP. - Increases to ~55% at 10³ exaFLOP. - Reaches ~65% at 10⁴ exaFLOP. - **Trend**: Slowest growth, with diminishing returns at higher compute levels. ### Key Observations - All models improve accuracy with increased compute power, but efficiency varies. - **Model A** and **Model B** achieve the highest accuracy, outperforming others by ~5–10% at 10⁴ exaFLOP. - **Model D** lags behind, gaining only ~25% accuracy across the compute range. - The logarithmic x-axis emphasizes that even modest compute increases (e.g., 10² to 10³) yield significant accuracy gains. ### Interpretation The data suggests that computational efficiency and architectural design (Model A/B) are critical for maximizing accuracy. While all models benefit from increased compute, the diminishing returns for Model D imply potential limitations in scalability or optimization. The steepest gains for Models A and B highlight their superior resource utilization, making them preferable for high-stakes applications. The logarithmic scale underscores the exponential cost-benefit tradeoff: doubling compute power (e.g., 10³ to 10⁴ exaFLOP) yields smaller relative accuracy improvements, suggesting diminishing returns at extreme scales. </details> (b) ARC (Easy) <details> <summary>x20.png Details</summary>  ### Visual Description ## Line Graph: Model Accuracy vs. Compute (exaFLOP) ### Overview The image is a line graph comparing the accuracy of four machine learning models (A, B, C, D) across varying computational resources (exaFLOP). The y-axis represents accuracy (0–100%), and the x-axis represents compute power on a logarithmic scale (10² to 10³ exaFLOP). Four distinct data series are plotted with unique markers and colors. ### Components/Axes - **Y-Axis**: "Accuracy" (0–100%, linear scale) - **X-Axis**: "Compute (exaFLOP)" (logarithmic scale: 10², 10².5, 10³) - **Legend**: Located on the right, associating: - Blue squares: Model A - Green triangles: Model B - Orange circles: Model C - Red stars: Model D ### Detailed Analysis 1. **Model A (Blue Squares)**: - Starts at ~50% accuracy at 10² exaFLOP. - Gradually increases to ~60% at 10³ exaFLOP. - Trend: Steady upward slope with minimal fluctuation. 2. **Model B (Green Triangles)**: - Begins at ~55% at 10² exaFLOP. - Dips slightly to ~58% at 10².5 exaFLOP. - Rises to ~62% at 10³ exaFLOP. - Trend: Initial dip followed by consistent growth. 3. **Model C (Orange Circles)**: - Starts at ~52% at 10² exaFLOP. - Peaks at ~61% at 10².5 exaFLOP. - Slightly declines to ~60% at 10³ exaFLOP. - Trend: Early rapid improvement, then plateau. 4. **Model D (Red Stars)**: - Begins at ~60% at 10² exaFLOP. - Drops to ~58% at 10².5 exaFLOP. - Sharp rise to ~65% at 10³ exaFLOP. - Trend: Initial decline followed by steep recovery. ### Key Observations - **Model D (Red Stars)** demonstrates the most dramatic improvement, gaining ~7% accuracy between 10².5 and 10³ exaFLOP. - **Model C (Orange Circles)** shows diminishing returns after 10².5 exaFLOP. - **Model B (Green Triangles)** exhibits a potential "learning curve" dip before stabilizing. - All models converge near ~60–65% accuracy at 10³ exaFLOP, suggesting diminishing marginal gains at higher compute levels. ### Interpretation The data suggests that computational power correlates with accuracy, but efficiency varies significantly across models. Model D (red stars) achieves the highest performance at scale, implying superior optimization for large-scale compute. Model C’s plateau indicates potential architectural limitations despite early gains. The dip in Model B’s performance at mid-range compute may reflect training instability or resource bottlenecks. Notably, no model exceeds ~65% accuracy even at 10³ exaFLOP, highlighting a possible ceiling effect or dataset constraints. The logarithmic x-axis emphasizes that compute improvements beyond 10³ exaFLOP would yield progressively smaller accuracy gains, per the observed trends. </details> (c) BoolQ <details> <summary>x21.png Details</summary>  ### Visual Description ## Line Graph: Model Accuracy vs. Compute (exaFLOP) ### Overview The image is a line graph comparing the accuracy of four language models (BLOOM, Pythia, OPT, RWKV) across varying computational budgets (exaFLOP). The x-axis uses a logarithmic scale (10² to 10³ exaFLOP), while the y-axis represents accuracy as a percentage (0–100%). All models show upward trends in accuracy with increased compute, though performance plateaus or diverges at higher compute levels. ### Components/Axes - **X-axis (Horizontal)**: "Compute (exaFLOP)" with logarithmic scale markers at 10², 10³. - **Y-axis (Vertical)**: "Accuracy" with linear scale from 0 to 100. - **Legend**: Located in the bottom-right corner, mapping colors to models: - Orange (●): BLOOM - Blue (■): Pythia - Green (▲): OPT - Red (★): RWKV ### Detailed Analysis 1. **BLOOM (Orange)**: - Starts at ~60% accuracy at 60 exaFLOP. - Increases to ~70% at 700 exaFLOP. - Slight dip to ~72% at 1000 exaFLOP. - *Trend*: Gradual improvement with diminishing returns. 2. **Pythia (Blue)**: - Begins at ~65% accuracy at 60 exaFLOP. - Rises steadily to ~80% at 1000 exaFLOP. - *Trend*: Consistent upward trajectory. 3. **OPT (Green)**: - Starts at ~65% accuracy at 60 exaFLOP. - Peaks at ~85% at 1000 exaFLOP. - *Trend*: Strong linear growth with no plateau. 4. **RWKV (Red)**: - Begins at ~65% accuracy at 60 exaFLOP. - Reaches ~85% at 1000 exaFLOP. - *Trend*: Similar to OPT but with a steeper initial rise. ### Key Observations - **Performance Trade-offs**: - BLOOM shows the weakest scaling, with accuracy plateauing near 72% despite doubling compute. - OPT and RWKV achieve ~85% accuracy at 1000 exaFLOP, outperforming Pythia. - **Efficiency**: - Pythia maintains steady gains but lags behind OPT/RWKV at higher compute. - All models require at least 60 exaFLOP to exceed 60% accuracy. - **Anomalies**: - BLOOM’s accuracy dip at 1000 exaFLOP is unusual compared to other models. ### Interpretation The graph highlights the relationship between computational resources and model performance. OPT and RWKV demonstrate superior scalability, achieving higher accuracy with increased compute, while BLOOM’s performance plateaus, suggesting inefficiencies in resource utilization. Pythia offers a balanced trade-off but underperforms at the highest compute levels. The logarithmic x-axis emphasizes that even modest compute increases (e.g., 60→700 exaFLOP) yield significant accuracy gains for most models. The data underscores the importance of architectural efficiency in large-scale language models, with OPT and RWKV appearing more cost-effective for high-accuracy applications. </details> (d) COPA <details> <summary>x22.png Details</summary>  ### Visual Description ## Line Graph: Model Accuracy vs. Compute (exaFLOP) ### Overview The graph compares the accuracy of four machine learning models (A, B, C, D) across varying compute resources (exaFLOP). Accuracy is measured on a 0-100 scale, while compute resources are plotted logarithmically from 10² to 10³ exaFLOP. All models show upward trends, with Model B demonstrating the steepest improvement. ### Components/Axes - **Y-axis (Accuracy)**: Labeled "Accuracy" with increments of 20 (0, 20, 40, 60, 80, 100). - **X-axis (Compute)**: Labeled "Compute (exaFLOP)" with logarithmic markers at 10², 10³, and 10⁴ (though data only extends to 10³). - **Legend**: Located in the top-right corner, associating: - Blue squares: Model A - Orange circles: Model B - Red stars: Model C - Green triangles: Model D ### Detailed Analysis 1. **Model A (Blue Squares)**: - Starts at ~22 accuracy at 10² exaFLOP. - Gradually increases to ~26 at 10³ exaFLOP. - Slope: Moderate upward trend. 2. **Model B (Orange Circles)**: - Begins at ~21 accuracy at 10² exaFLOP. - Sharp rise to ~30 accuracy at 10³ exaFLOP. - Slope: Steepest among all models. 3. **Model C (Red Stars)**: - Starts at ~20 accuracy at 10² exaFLOP. - Increases to ~24 at 10³ exaFLOP. - Slope: Slightly less steep than Model A. 4. **Model D (Green Triangles)**: - Starts at ~19 accuracy at 10² exaFLOP. - Minimal improvement to ~21 at 10³ exaFLOP. - Slope: Nearly flat compared to others. ### Key Observations - All models improve accuracy with increased compute, but **Model B** scales most efficiently (+9 accuracy from 10² to 10³ exaFLOP). - **Model D** shows the weakest compute dependency, with only +2 accuracy gain. - The logarithmic x-axis emphasizes exponential compute growth, yet accuracy gains are linear for most models. ### Interpretation The data suggests that **Model B** is optimized for high-compute environments, achieving disproportionate accuracy gains relative to resource investment. This could indicate architectural advantages (e.g., parallelization efficiency) or algorithmic optimizations. Conversely, **Model D**’s plateau implies diminishing returns or inherent limitations in leveraging additional compute. The consistent upward trends across models confirm that compute resources are a critical factor for performance, but efficiency varies significantly between implementations. The absence of data beyond 10³ exaFLOP leaves uncertainty about long-term scalability, particularly for Model B, which might face saturation or diminishing returns at higher scales. </details> (e) HeadQA <details> <summary>x23.png Details</summary>  ### Visual Description ## Line Graph: Accuracy vs Compute (exaFLOP) ### Overview The graph illustrates the relationship between computational power (exaFLOP) and accuracy for four models (A, B, C, D). All models show an upward trend in accuracy as compute increases, with varying rates of improvement. ### Components/Axes - **X-axis (Compute)**: Labeled "Compute (exaFLOP)" with logarithmic scale markers at 10², 10³, and 10⁴. - **Y-axis (Accuracy)**: Labeled "Accuracy" with linear scale from 0 to 100. - **Legend**: Located in the top-left corner, mapping colors to models: - Blue squares: Model A - Green triangles: Model B - Orange circles: Model C - Red stars: Model D ### Detailed Analysis - **Model A (Blue Squares)**: - 10² exaFLOP: ~25% accuracy - 10³ exaFLOP: ~35% accuracy - 10⁴ exaFLOP: ~45% accuracy - 10⁵ exaFLOP: ~50% accuracy - **Model B (Green Triangles)**: - 10² exaFLOP: ~30% accuracy - 10³ exaFLOP: ~40% accuracy - 10⁴ exaFLOP: ~48% accuracy - 10⁵ exaFLOP: ~52% accuracy - **Model C (Orange Circles)**: - 10² exaFLOP: ~28% accuracy - 10³ exaFLOP: ~33% accuracy - 10⁴ exaFLOP: ~42% accuracy - 10⁵ exaFLOP: ~47% accuracy - **Model D (Red Stars)**: - 10² exaFLOP: ~32% accuracy - 10³ exaFLOP: ~37% accuracy - 10⁴ exaFLOP: ~45% accuracy - 10⁵ exaFLOP: ~52% accuracy ### Key Observations 1. **Upward Trend**: All models show increased accuracy with higher compute, confirming the positive correlation between computational power and performance. 2. **Efficiency Differences**: - Model D achieves the highest accuracy at 10⁵ exaFLOP (~52%) despite starting with lower compute than Model B. - Model A exhibits diminishing returns after 10⁴ exaFLOP, with minimal improvement (45% → 50%). 3. **Steepest Slope**: Model D demonstrates the most rapid improvement between 10³ and 10⁴ exaFLOP (~37% → 45%). ### Interpretation The data suggests that computational efficiency varies significantly across models. Model D outperforms others in both absolute accuracy and cost-effectiveness, achieving comparable results to Model B at lower compute levels. The plateau observed in Model A at higher compute levels implies potential architectural or algorithmic limitations. These findings highlight the importance of optimizing model design for compute efficiency rather than solely increasing hardware resources. The logarithmic x-axis emphasizes that even modest increases in compute (e.g., 10³ → 10⁴ exaFLOP) can yield disproportionate accuracy gains for certain models. </details> (f) HellaSwag <details> <summary>x24.png Details</summary>  ### Visual Description ## Line Graph: Accuracy vs. Compute (exaFLOP) ### Overview The image is a line graph comparing the accuracy of four computational models (BLOOM, Pythia, OPT, RWKV) as a function of computational resources (exaFLOPs). The x-axis represents compute in exaFLOPs on a logarithmic scale (10² to 10³), and the y-axis represents accuracy as a percentage (0–100). All four models show upward trends, with accuracy increasing as compute increases. The graph includes a legend in the bottom-right corner to distinguish the models by color and marker. --- ### Components/Axes - **X-axis (Horizontal)**: - Label: "Compute (exaFLOP)" - Scale: Logarithmic (10² to 10³) - Tick marks: 10², 10³ - **Y-axis (Vertical)**: - Label: "Accuracy" - Scale: Linear (0–100) - Tick marks: 0, 20, 40, 60, 80, 100 - **Legend**: - Position: Bottom-right corner - Entries: - **BLOOM**: Orange circles - **Pythia**: Blue squares - **OPT**: Green triangles - **RWKV**: Red stars --- ### Detailed Analysis 1. **BLOOM (Orange Circles)**: - Starts at ~30 exaFLOPs with ~30% accuracy. - Increases to ~60 exaFLOPs at ~60% accuracy. - Slope: Moderate upward trend. 2. **Pythia (Blue Squares)**: - Starts at ~20 exaFLOPs with ~20% accuracy. - Rises sharply to ~70 exaFLOPs at ~70% accuracy. - Slope: Steepest among all models. 3. **OPT (Green Triangles)**: - Starts at ~40 exaFLOPs with ~40% accuracy. - Increases to ~70 exaFLOPs at ~70% accuracy. - Slope: Moderate upward trend. 4. **RWKV (Red Stars)**: - Starts at ~50 exaFLOPs with ~50% accuracy. - Rises to ~70 exaFLOPs at ~70% accuracy. - Slope: Steeper than BLOOM and OPT but less than Pythia. All lines converge near 70% accuracy at 10³ exaFLOPs, suggesting diminishing returns at higher compute levels. --- ### Key Observations - **Universal Trend**: All models improve accuracy with increased compute, but efficiency varies. - **Efficiency Differences**: - Pythia achieves the highest accuracy at the lowest compute (~70% at ~70 exaFLOPs). - BLOOM and OPT require more compute to reach similar accuracy levels. - **Convergence**: At 10³ exaFLOPs, all models plateau near 70% accuracy, indicating diminishing returns. - **Logarithmic Scale Impact**: The x-axis’s logarithmic scale emphasizes performance differences at lower compute ranges (e.g., 10² vs. 10³). --- ### Interpretation The graph demonstrates that computational efficiency is critical for model performance. Pythia and RWKV outperform BLOOM and OPT in terms of accuracy per exaFLOP, suggesting architectural or training advantages. The convergence at 10³ exaFLOPs implies that beyond a certain compute threshold, further resource investment yields minimal accuracy gains. This highlights trade-offs between resource allocation and model design, with implications for cost-effective deployment in resource-constrained environments. </details> (g) LAMBADA (OpenAI) <details> <summary>x25.png Details</summary>  ### Visual Description ## Line Graph: Accuracy vs Compute (exaFLOP) ### Overview The graph illustrates the relationship between computational power (measured in exaFLOP) and accuracy across four distinct algorithms or models. Accuracy is plotted on the y-axis (0–100%), while compute is logarithmic on the x-axis (10² to 10⁴ exaFLOP). Four data series are represented by colored lines with unique markers. ### Components/Axes - **X-axis (Compute)**: Labeled "Compute (exaFLOP)" with logarithmic scale markers at 10², 10³, and 10⁴. - **Y-axis (Accuracy)**: Labeled "Accuracy" with linear scale markers at 20, 40, 60, 80, and 100. - **Legend**: Located in the top-right corner, associating: - Blue squares (Line 1) - Green triangles (Line 2) - Orange circles (Line 3) - Red stars (Line 4) ### Detailed Analysis 1. **Line 1 (Blue Squares)**: - Starts at ~10 exaFLOP with ~15% accuracy. - Gradually increases to ~25% at 10⁴ exaFLOP. - Slope is relatively flat compared to other lines. 2. **Line 2 (Green Triangles)**: - Begins at ~10 exaFLOP with ~18% accuracy. - Rises to ~27% at 10⁴ exaFLOP. - Shows moderate upward trend. 3. **Line 3 (Orange Circles)**: - Starts at ~10 exaFLOP with ~16% accuracy. - Peaks at ~25% at 10³ exaFLOP, then plateaus. - Shows early saturation. 4. **Line 4 (Red Stars)**: - Begins at ~10 exaFLOP with ~17% accuracy. - Reaches ~30% at 10⁴ exaFLOP. - Steepest slope, highest final accuracy. ### Key Observations - All lines exhibit upward trends, confirming that increased compute correlates with higher accuracy. - Line 4 (red stars) achieves the highest accuracy (~30%) but plateaus earlier than others. - Lines 1 and 2 show similar performance, with Line 2 slightly outperforming Line 1. - Line 3 (orange circles) saturates fastest, reaching its peak at 10³ exaFLOP. ### Interpretation The graph demonstrates that computational power improves model accuracy, but diminishing returns occur at higher exaFLOP thresholds. Line 4’s algorithm appears most efficient, achieving the highest accuracy with moderate compute. Lines 1 and 2 suggest scalability with fewer diminishing returns, while Line 3’s early saturation implies architectural or algorithmic limitations. The data underscores the trade-off between resource allocation and performance gains in high-performance computing systems. </details> (h) OpenBookQA <details> <summary>x26.png Details</summary>  ### Visual Description ## Line Graph: Accuracy vs. Compute (exaFLOP) ### Overview The image is a line graph comparing the accuracy of four models (A, B, C, D) across varying computational resources (exaFLOP). The x-axis represents compute (exaFLOP) on a logarithmic scale (10² to 10³), while the y-axis shows accuracy (0–100%). Four distinct lines with unique markers and colors represent each model, showing how accuracy improves with increased compute. ### Components/Axes - **X-axis (Compute)**: Labeled "Compute (exaFLOP)" with a logarithmic scale (10² to 10³). Ticks are spaced logarithmically, with approximate values at 10², 10².5, and 10³. - **Y-axis (Accuracy)**: Labeled "Accuracy" with a linear scale (0–100%). - **Legend**: Located on the right side of the graph. Entries: - **Model A**: Blue squares - **Model B**: Red stars - **Model C**: Green triangles - **Model D**: Yellow circles - **Lines**: Four continuous lines connect data points for each model, with markers at specific x-axis intervals. ### Detailed Analysis - **Model A (Blue Squares)**: - At 10² exaFLOP: ~60% accuracy. - At 10³ exaFLOP: ~70% accuracy. - Trend: Steady upward slope, moderate improvement. - **Model B (Red Stars)**: - At 10² exaFLOP: ~62% accuracy. - At 10³ exaFLOP: ~75% accuracy. - Trend: Slightly steeper slope than Model A, highest accuracy at 10³. - **Model C (Green Triangles)**: - At 10² exaFLOP: ~63% accuracy. - At 10³ exaFLOP: ~76% accuracy. - Trend: Similar to Model B but slightly lower final accuracy. - **Model D (Yellow Circles)**: - At 10² exaFLOP: ~61% accuracy. - At 10³ exaFLOP: ~72% accuracy. - Trend: Slowest improvement, lowest final accuracy. ### Key Observations 1. **Positive Correlation**: All models show increased accuracy with higher compute, confirming the general trend of computational efficiency improving performance. 2. **Model B Dominance**: Model B (red stars) achieves the highest accuracy (~75%) at 10³ exaFLOP, outperforming others. 3. **Model D Lag**: Model D (yellow circles) shows the smallest improvement (~11% gain) and lowest final accuracy (~72%). 4. **Logarithmic Scale Impact**: The x-axis compression emphasizes differences in performance gains at higher compute levels (e.g., 10³ exaFLOP). ### Interpretation The data suggests that computational resources significantly impact model accuracy, but efficiency varies across models. Model B demonstrates the most effective scaling, achieving higher accuracy with relatively lower compute compared to others. Model D’s slower improvement highlights potential inefficiencies in its architecture or training. The logarithmic x-axis underscores that gains at higher compute levels (e.g., 10³ exaFLOP) are more pronounced, suggesting diminishing returns at lower compute ranges. This analysis could guide resource allocation for model training, prioritizing models with better compute-to-accuracy ratios. </details> (i) PiQA <details> <summary>x27.png Details</summary>  ### Visual Description ## Line Graph: Model Accuracy vs. Compute (exaFLOP) ### Overview The image depicts a line graph comparing the accuracy of four AI models (BLOOM, Pythia, OPT, RWKV) across varying computational budgets (exaFLOP). Accuracy is plotted on the y-axis (0–100%), and compute is plotted on the x-axis (logarithmic scale: 10² to 10³ exaFLOP). All models show upward trends, with OPT achieving the highest accuracy at the highest compute level. ### Components/Axes - **X-axis (Compute)**: Labeled "Compute (exaFLOP)" with a logarithmic scale (10² to 10³). - **Y-axis (Accuracy)**: Labeled "Accuracy" with a linear scale (0–100%). - **Legend**: Located in the bottom-right corner, mapping colors to models: - Orange (●): BLOOM - Blue (■): Pythia - Green (▲): OPT - Red (★): RWKV ### Detailed Analysis 1. **BLOOM (Orange)**: - Starts at ~70 exaFLOP with ~70% accuracy. - Increases to ~80 exaFLOP with ~75% accuracy. - Reaches ~1000 exaFLOP with ~85% accuracy. 2. **Pythia (Blue)**: - Begins at ~50 exaFLOP with ~50% accuracy. - Rises to ~100 exaFLOP with ~75% accuracy. - Achieves ~1000 exaFLOP with ~85% accuracy. 3. **OPT (Green)**: - Starts at ~100 exaFLOP with ~75% accuracy. - Peaks at ~1000 exaFLOP with ~88% accuracy. 4. **RWKV (Red)**: - Begins at ~100 exaFLOP with ~70% accuracy. - Reaches ~1000 exaFLOP with ~85% accuracy. ### Key Observations - All models improve accuracy with increased compute, but OPT demonstrates the steepest efficiency gain. - Pythia starts with the lowest accuracy at lower compute levels but converges with BLOOM and RWKV at higher compute. - OPT consistently outperforms others at the highest compute (1000 exaFLOP). - RWKV’s accuracy plateaus slightly below Pythia and OPT at 1000 exaFLOP. ### Interpretation The data suggests that **compute efficiency** significantly impacts model performance. OPT achieves the highest accuracy with the least compute, indicating superior optimization. Pythia and RWKV require more compute to reach comparable accuracy levels, while BLOOM’s performance is mid-tier. The logarithmic x-axis highlights that even small increases in compute (e.g., 10x) yield diminishing returns for some models. This underscores the importance of architectural efficiency in AI development. </details> (j) ReCoRD <details> <summary>x28.png Details</summary>  ### Visual Description ## Line Graph: Model Accuracy vs. Compute (exaFLOP) ### Overview The graph illustrates the relationship between computational resources (measured in exaFLOP) and model accuracy for four distinct models (A, B, C, D). Accuracy is plotted on a linear scale (0–100%), while compute is represented on a logarithmic scale (10² to 10³). All models show increasing accuracy with higher compute, with convergence observed at higher compute levels. ### Components/Axes - **Y-Axis (Accuracy)**: Labeled "Accuracy" with ticks at 0, 20, 40, 60, 80, 100. - **X-Axis (Compute)**: Labeled "Compute (exaFLOP)" with logarithmic ticks at 10², 10².⁵, 10³. - **Legend**: Located on the right, associating: - Blue squares → Model A - Orange stars → Model B - Green triangles → Model C - Yellow circles → Model D ### Detailed Analysis 1. **Model A (Blue Squares)**: - Starts at ~60% accuracy at 10² exaFLOP. - Increases to ~75% at 10².⁵ exaFLOP. - Reaches ~90% at 10³ exaFLOP. 2. **Model B (Orange Stars)**: - Begins at ~75% accuracy at 10² exaFLOP. - Rises to ~85% at 10².⁵ exaFLOP. - Stabilizes near ~90% at 10³ exaFLOP. 3. **Model C (Green Triangles)**: - Starts at ~75% accuracy at 10² exaFLOP. - Increases to ~85% at 10².⁵ exaFLOP. - Peaks at ~90% at 10³ exaFLOP. 4. **Model D (Yellow Circles)**: - Initial accuracy ~80% at 10² exaFLOP. - Gains to ~88% at 10².⁵ exaFLOP. - Maintains ~90% at 10³ exaFLOP. ### Key Observations - **Convergence**: All models achieve ~90% accuracy at 10³ exaFLOP, suggesting diminishing returns beyond this compute threshold. - **Initial Disparity**: Model A lags significantly at lower compute (10² exaFLOP) compared to others. - **Efficiency**: Models B, C, and D show steeper initial improvements in accuracy relative to compute increases. ### Interpretation The data demonstrates that computational scaling improves model accuracy, but the marginal gains diminish as compute increases. Model A’s lower initial performance may indicate architectural inefficiencies or suboptimal training. The convergence at 10³ exaFLOP implies that beyond this point, further compute offers limited accuracy benefits, highlighting the importance of algorithmic optimization over brute-force scaling. This trend aligns with Pareto principles in resource allocation, where efficiency often outweighs raw capacity in practical applications. </details> (k) SciQ <details> <summary>x29.png Details</summary>  ### Visual Description ## Line Graph: Accuracy vs Compute (exaFLOP) ### Overview The graph illustrates the relationship between computational power (measured in exaFLOP) and accuracy across four models. Accuracy is plotted on the y-axis (0–100%), while compute is on the x-axis (logarithmic scale: 10² to 10³ exaFLOP). Four data series are represented by distinct markers and colors, with a legend on the right. ### Components/Axes - **X-axis**: "Compute (exaFLOP)" with logarithmic scaling (10² to 10³). - **Y-axis**: "Accuracy" (0–100%). - **Legend**: - Blue squares: Model A - Green triangles: Model B - Orange circles: Model C - Red stars: Model D - **Title**: "Accuracy vs Compute (exaFLOP)" (top-center). ### Detailed Analysis 1. **Model A (Blue Squares)**: - Starts at ~50% accuracy at 10² exaFLOP. - Gradually increases to ~60% at 10³ exaFLOP. - Slope: Moderate upward trend. 2. **Model B (Green Triangles)**: - Begins at ~45% accuracy at 10² exaFLOP. - Sharp rise to ~65% at 10³ exaFLOP. - Slope: Steepest upward trajectory. 3. **Model C (Orange Circles)**: - Starts at ~50% accuracy at 10² exaFLOP. - Dips to ~45% at ~10².⁵ exaFLOP (midpoint). - Recovers to ~60% at 10³ exaFLOP. - Slope: Initial decline, then steady increase. 4. **Model D (Red Stars)**: - Begins at ~50% accuracy at 10² exaFLOP. - Consistent rise to ~62% at 10³ exaFLOP. - Slope: Steady upward trend. ### Key Observations - All models show improved accuracy with increased compute. - **Model B** achieves the highest accuracy (~65%) at 10³ exaFLOP. - **Model C** exhibits a temporary dip at ~10².⁵ exaFLOP but recovers. - **Model D** demonstrates the most consistent scaling. - Accuracy plateaus near 60–65% for all models at 10³ exaFLOP. ### Interpretation The data suggests a positive correlation between computational power and accuracy, with diminishing returns as compute approaches 10³ exaFLOP. **Model B** is the most efficient, achieving the highest accuracy with the steepest improvement. **Model C**'s dip may indicate a temporary inefficiency or optimization challenge at intermediate compute levels. The convergence of accuracy near 60–65% at 10³ exaFLOP implies a potential ceiling for these models, highlighting the need for architectural or algorithmic innovations beyond this compute threshold. </details> (l) Winogrande Figure 12: Zero-Shot Performance of RWKV on common language modeling evaluation benchmarks. ### J.2 Evaluation on Long Range Arena The Long-Range Arena (LRA) benchmark (Tay et al., 2021) is designed to assess the performance of models in handling lengthy context situations. It includes a collection of tasks with sequences ranging from 1,000 to 16,000 tokens, covering various types of data like text, natural language, synthetic images, and mathematical expressions. We apply RWKV on the LRA benchmark and the report results are in Table 4. Other models’ performances are directly cited from Gu et al. (2022); Alam et al. (2023). Table 4: Evaluation on Long Range Arena. Other models reported in the literature (Gu et al., 2022; Alam et al., 2023). Bolded values are the best. | Model | ListOps | Text | Retrieval | Image | Pathfinder | Path-X | Avg | | --- | --- | --- | --- | --- | --- | --- | --- | | Transformer | 36.37 | 64.27 | 57.46 | 42.44 | 71.40 | ✗ | 53.66 | | Reformer | 37.27 | 56.10 | 53.40 | 38.07 | 68.50 | ✗ | 50.56 | | BigBird | 36.05 | 64.02 | 59.29 | 40.83 | 74.87 | ✗ | 54.17 | | Linear Trans. | 16.13 | 65.90 | 53.09 | 42.34 | 75.30 | ✗ | 50.46 | | Performer | 18.01 | 65.40 | 53.82 | 42.77 | 77.05 | ✗ | 51.18 | | FNet | 35.33 | 65.11 | 59.61 | 38.67 | 77.80 | ✗ | 54.42 | | Nyströmformer | 37.15 | 65.52 | 79.56 | 41.58 | 70.94 | ✗ | 57.46 | | Luna-256 | 37.25 | 64.57 | 79.29 | 47.38 | 77.72 | ✗ | 59.37 | | Hrrformer | 39.98 | 65.38 | 76.15 | 50.45 | 72.17 | ✗ | 60.83 | | S4 | 59.60 | 86.82 | 90.90 | 88.65 | 94.20 | 96.35 | 86.09 | | RWKV | 55.88 | 86.04 | 88.34 | 70.53 | 58.42 | ✗ | 72.07 | The results show that RWKV performs second only to the S4 model in five datasets. While RWKV substantially underpreforms S4 on Image, Pathfinder, and Path-X, on the problems related to natural language and computer code processing RWKV performs on par with S4 or nearly so. ### J.3 Enwik8 Perplexity We also evaluate our model in terms of perplexity on the Enwik8 dataset. Baseline comparisons are made with Reformer (Kitaev et al., 2020), Synthesizer (Tay et al., 2020) (the best performing dense version), Linear Transformer (Katharopoulos et al., 2020), Performer (Choromanski et al., 2020). $L,d,$ and $T$ denote the number of blocks (network depth), dimension of features, and sequence length, respectively. Both Linear Transformer and Performer are implemented with customized CUDA kernels (github.com/idiap/fast-transformers), and all other models are implemented in native Pytorch. ${}^1$ No weight decay nor dropout was used. ${}^2$ Trained with AdamW and weight decay set to 0.1, dropout of 0.1, batch size of 16, and initial learning rate of 6e-4. | Method | L | d | T | Train bpc | Test bpc | Time Complexity | Space Complexity | | --- | --- | --- | --- | --- | --- | --- | --- | | Transformer | 12 | 512 | 1024 | 0.977 | 1.137 | $O(T^2d)$ | $O(T^2+Td)$ | | Transformer | 24 | 256 | 1024 | 1.039 | 1.130 | $O(T^2d)$ | $O(T^2+Td)$ | | Reformer | 12 | 512 | 1024 | 1.040 | 1.195 | $O(T\log{T}d)$ | $O(T\log{T}+Td)$ | | Synthesizer | 12 | 512 | 1024 | 0.994 | 1.298 | $O(T^2d)$ | $O(T^2+Td)$ | | Linear Transformer | 12 | 512 | 1024 | 0.981 | 1.207 | $O(Td^2)$ | $O(Td+d^2)$ | | Performer | 12 | 512 | 1024 | 1.002 | 1.199 | $O(Td^2\log{d})$ | $O(Td\log{d}+d^2\log{d})$ | | AFT-simple | 12 | 512 | 1024 | 1.046 | 1.209 | $O(Td)$ | $O(Td)$ | | RWKV-RNN ${}^1$ | 6 | 512 | 1024 | 0.720 | - | $O(Td)$ | $O(d)$ | | RWKV-RNN ${}^2$ | 12 | 512 | 1024 | 1.010 | 1.178 | $O(Td)$ | $O(d)$ | Table 5: Enwik8 results, measured in bits per character (bpc). ## Appendix K Inference results Figures 13 and 14 illustrate, respectively, the results on time (s) and memory (RAM, VRAM) requirements for LLM inference in float32 precision. We benchmark the following model families and sizes: - RWKV: 169m, 430m, 1.4b, 3b, 7b, 14b - Bloom Scao et al. (2022): 560m, 1b, 3b - OPT Zhang et al. (2022): 125m, 350m, 1.3b, 2.7b, 6.7b, 13b - GPT-Neo Black et al. (2021): 125m, 1.3b, 2.7b - Pythia Biderman et al. (2023b): 160m, 410m, 1.4b, 2.8b, 6.7b, 12b <details> <summary>x30.png Details</summary>  ### Visual Description ## Line Chart: Peak Memory Usage vs. Number of Parameters for 512 Tokens ### Overview The chart compares the peak memory consumption (in gigabytes) required to process 512 tokens for five different AI models, plotted against the number of parameters (in scientific notation). The y-axis represents memory usage, while the x-axis represents model size. Each model is represented by a distinct line style and color. ### Components/Axes - **X-axis**: "# Params" (Number of Parameters), logarithmic scale from 10⁸ to 10¹⁰. - **Y-axis**: "Peak Memory (Gb) for 512 tokens (excluding model params)", linear scale from 0 to 12. - **Legend**: Located in the top-left corner, with five entries: - **bloom-cpu**: Red "X" marker. - **gpt-neo-cpu**: Blue "+" marker. - **opt-cpu**: Green star marker. - **pythia-cpu**: Cyan triangle marker. - **rwkv-4-pile-cpu**: Black triangle marker. ### Detailed Analysis 1. **bloom-cpu (Red X)**: - **Trend**: Steep upward slope, starting at ~1.5 Gb for 10⁸ params, rising to ~4 Gb at 10⁹ params, and peaking at ~12 Gb for 10¹⁰ params. - **Key Data Points**: - 10⁸ params: ~1.5 Gb. - 10⁹ params: ~4 Gb. - 10¹⁰ params: ~12 Gb. 2. **gpt-neo-cpu (Blue +)**: - **Trend**: Moderate upward slope, starting at ~1.8 Gb for 10⁸ params, reaching ~6 Gb at 10⁹ params, and ~11 Gb at 10¹⁰ params. - **Key Data Points**: - 10⁸ params: ~1.8 Gb. - 10⁹ params: ~6 Gb. - 10¹⁰ params: ~11 Gb. 3. **opt-cpu (Green Star)**: - **Trend**: Gradual upward slope, starting at ~1.2 Gb for 10⁸ params, rising to ~5 Gb at 10⁹ params, and ~11.5 Gb at 10¹⁰ params. - **Key Data Points**: - 10⁸ params: ~1.2 Gb. - 10⁹ params: ~5 Gb. - 10¹⁰ params: ~11.5 Gb. 4. **pythia-cpu (Cyan Triangle)**: - **Trend**: Steeper than opt-cpu but less than bloom-cpu, starting at ~1 Gb for 10⁸ params, reaching ~4.5 Gb at 10⁹ params, and ~9.5 Gb at 10¹⁰ params. - **Key Data Points**: - 10⁸ params: ~1 Gb. - 10⁹ params: ~4.5 Gb. - 10¹⁰ params: ~9.5 Gb. 5. **rwkv-4-pile-cpu (Black Triangle)**: - **Trend**: Gentle upward slope, starting at ~0.2 Gb for 10⁸ params, rising to ~0.8 Gb at 10⁹ params, and ~3 Gb at 10¹⁰ params. - **Key Data Points**: - 10⁸ params: ~0.2 Gb. - 10⁹ params: ~0.8 Gb. - 10¹⁰ params: ~3 Gb. ### Key Observations - **Efficiency Variance**: rwkv-4-pile-cpu demonstrates the lowest memory usage across all parameter scales, while bloom-cpu is the most memory-intensive. - **Scaling Behavior**: All models show increasing memory usage with parameter count, but bloom-cpu and gpt-neo-cpu exhibit steeper scaling. - **Outliers**: No anomalies detected; all lines follow consistent upward trends. ### Interpretation The chart highlights significant differences in memory efficiency among models. rwkv-4-pile-cpu’s minimal memory footprint suggests architectural optimizations (e.g., sparse attention mechanisms), while bloom-cpu’s high memory demands may stem from dense parameterization or less efficient computation. The divergence in scaling rates implies trade-offs between model complexity and resource efficiency, critical for deployment in resource-constrained environments. The logarithmic x-axis emphasizes how memory requirements grow disproportionately with parameter count, underscoring the importance of model design for scalability. </details> <details> <summary>x31.png Details</summary>  ### Visual Description ## Line Graph: Peak Memory Usage vs. Model Parameters ### Overview The graph illustrates the relationship between the number of parameters (# Params) and peak memory usage (excluding model parameters) for five different CUDA-optimized models. The x-axis uses a logarithmic scale (10⁸ to 10¹⁰ parameters), while the y-axis ranges from 0 to 20 GB. Each model is represented by a distinct line style and color, with trends showing how memory requirements scale with model complexity. ### Components/Axes - **X-axis**: "# Params" (logarithmic scale: 10⁸, 10⁹, 10¹⁰) - **Y-axis**: "Peak Memory (GB) for 512 tokens (excluding model params)" (linear scale: 0–20 GB) - **Legend**: Located in the top-left corner, mapping colors and markers to models: - Red crosses: bloom-cuda - Blue pluses: gpt-neo-cuda - Green stars: opt-cuda - Cyan triangles: pythia-cuda - Black triangles: rwkv-4-pile-cuda ### Detailed Analysis 1. **bloom-cuda (Red Cross)**: - Starts at ~2 GB for 10⁸ params, rising steeply to ~12.5 GB at 10⁹ params, then plateauing. - At 10¹⁰ params, memory usage spikes to ~17.5 GB. 2. **gpt-neo-cuda (Blue Plus)**: - Begins at ~1.5 GB for 10⁸ params, increasing to ~6 GB at 10⁹ params, then ~10 GB at 10¹⁰ params. 3. **opt-cuda (Green Star)**: - Starts at ~1 GB for 10⁸ params, rising to ~5 GB at 10⁹ params, then ~11 GB at 10¹⁰ params. 4. **pythia-cuda (Cyan Triangle)**: - Begins at ~0.5 GB for 10⁸ params, increasing to ~4 GB at 10⁹ params, then ~11.5 GB at 10¹⁰ params. 5. **rwkv-4-pile-cuda (Black Triangle)**: - Remains flat at ~0.1 GB for 10⁸ params, rising slightly to ~0.5 GB at 10⁹ params, then ~2.5 GB at 10¹⁰ params. ### Key Observations - **Scaling Trends**: All models except rwkv-4-pile-cuda exhibit exponential memory growth with parameter count. bloom-cuda and pythia-cuda show the steepest increases. - **Efficiency Outlier**: rwkv-4-pile-cuda maintains significantly lower memory usage across all parameter ranges. - **Divergence at 10¹⁰ Params**: bloom-cuda and pythia-cuda surpass other models in memory consumption, while rwkv-4-pile-cuda remains the most efficient. ### Interpretation The data suggests that memory efficiency varies dramatically between models. rwkv-4-pile-cuda’s flat curve implies architectural optimizations (e.g., sparse attention, quantization) that reduce memory overhead, making it suitable for resource-constrained deployments. In contrast, bloom-cuda and pythia-cuda’s steep curves highlight their unsuitability for large-scale models without specialized hardware. The divergence at 10¹⁰ params underscores the trade-off between model size and practicality, with rwkv-4-pile-cuda offering a viable alternative for memory-constrained environments. </details> Figure 13: Text generation inference memory (CPU RAM, GPU VRAM) for LLMs. Model parameters are not accounted. <details> <summary>x32.png Details</summary>  ### Visual Description ## Line Chart: CPU Time vs. Number of Parameters for 512 Tokens ### Overview The chart compares the CPU time (in seconds) required to process 512 tokens across five different systems as the number of parameters increases from 10⁸ to 10¹⁰. The y-axis represents CPU time, and the x-axis represents the number of parameters on a logarithmic scale. Five systems are compared: bloom-cpu, gpt-neo-cpu, opt-cpu, pythia-cpu, and rwkv-4-pile-cpu. ### Components/Axes - **X-axis**: "# Params" (logarithmic scale: 10⁸, 10⁹, 10¹⁰) - **Y-axis**: "CPU time (s) for 512 tokens" (linear scale: 0 to 2000) - **Legend**: Located in the top-left corner, with line styles and colors: - **bloom-cpu**: Blue dashed line with "×" markers - **gpt-neo-cpu**: Orange dash-dot line with "+" markers - **opt-cpu**: Green dash-dot line with "★" markers - **pythia-cpu**: Red dashed line with "▲" markers - **rwkv-4-pile-cpu**: Purple dash-dot line with "●" markers ### Detailed Analysis 1. **bloom-cpu** (blue): Starts at ~50s for 10⁸ params, rises to ~500s at 10⁹, and ~800s at 10¹⁰. 2. **gpt-neo-cpu** (orange): Begins at ~50s for 10⁸, increases to ~400s at 10⁹, and ~700s at 10¹⁰. 3. **opt-cpu** (green): Starts at ~50s for 10⁸, rises sharply to ~400s at 10⁹, and explodes to ~2200s at 10¹⁰. 4. **pythia-cpu** (red): Starts at ~50s for 10⁸, increases to ~350s at 10⁹, and ~700s at 10¹⁰. 5. **rwkv-4-pile-cpu** (purple): Remains flat at ~10s for 10⁸ and 10⁹, then jumps to ~900s at 10¹⁰. ### Key Observations - **opt-cpu** exhibits the steepest growth, with CPU time increasing **5.5x** between 10⁹ and 10¹⁰ parameters. - **rwkv-4-pile-cpu** maintains the lowest CPU time until 10¹⁰, where it suddenly spikes. - All systems except rwkv-4-pile-cpu show gradual increases between 10⁸ and 10⁹ parameters. - At 10¹⁰ parameters, opt-cpu's CPU time surpasses all others by a factor of ~3x. ### Interpretation The data suggests that **opt-cpu** scales poorly with increasing model size, making it impractical for large parameter counts. In contrast, **rwkv-4-pile-cpu** demonstrates superior efficiency across most parameter ranges, though its performance degrades dramatically at 10¹⁰ parameters. The divergence in trends highlights trade-offs between system design and scalability. The sudden spike in rwkv-4-pile-cpu at 10¹⁰ params may indicate an architectural limitation or optimization threshold. These results emphasize the importance of selecting hardware/software configurations aligned with the target model size. </details> <details> <summary>x33.png Details</summary>  ### Visual Description ## Line Graph: GPU Time vs. Number of Parameters for CUDA Implementations ### Overview The graph compares GPU execution time (in seconds) for processing 512 tokens across five CUDA-based implementations (bloom-cuda, gpt-neo-cuda, opt-cuda, pythia-cuda, rwkv-4-pile-cuda) as a function of model parameter count. The x-axis spans 10⁸ to 10¹⁰ parameters, while the y-axis ranges from 0 to 250 seconds. Lines exhibit distinct trends, with some implementations showing exponential scaling behavior. ### Components/Axes - **X-axis**: "# Params" (logarithmic scale: 10⁸, 10⁹, 10¹⁰) - **Y-axis**: "GPU time (s) for 512 tokens" (linear scale: 0–250) - **Legend**: Top-left corner, color-coded with markers: - Blue crosses: bloom-cuda - Orange pluses: gpt-neo-cuda - Green stars: opt-cuda - Red triangles: pythia-cuda - Purple circles: rwkv-4-pile-cuda ### Detailed Analysis 1. **bloom-cuda** (blue crosses): - Flat line at ~0 seconds across all parameter ranges. - No visible deviation from baseline. 2. **gpt-neo-cuda** (orange pluses): - Slight upward trend, peaking at ~10 seconds at 10¹⁰ parameters. - Minimal variance between 10⁸–10⁹ parameters. 3. **opt-cuda** (green stars): - Stable at ~0 seconds until 10⁹ parameters. - Sharp exponential increase: ~150s at 10⁹ → 270s at 10¹⁰. 4. **pythia-cuda** (red triangles): - Gradual rise from ~5s at 10⁸ → 30s at 10⁹. - Steep ascent to ~240s at 10¹⁰. 5. **rwkv-4-pile-cuda** (purple circles): - Consistently flat at ~2–5 seconds across all ranges. ### Key Observations - **Exponential Scaling**: opt-cuda and pythia-cuda show dramatic GPU time increases (>1000% between 10⁹–10¹⁰ parameters). - **Efficiency Outliers**: bloom-cuda and rwkv-4-pile-cuda maintain near-constant performance regardless of parameter count. - **Intermediate Performance**: gpt-neo-cuda exhibits moderate scaling (~10x increase from 10⁸–10¹⁰). ### Interpretation The data suggests significant architectural differences in how these implementations handle parameter scaling: - **Efficient Designs**: bloom-cuda and rwkv-4-pile-cuda likely use optimized kernels or quantization techniques to maintain constant GPU time. - **Scaling Challenges**: opt-cuda and pythia-cuda may lack memory optimization or parallelization strategies, leading to quadratic/linear scaling penalties. - **Intermediate Case**: gpt-neo-cuda’s moderate scaling implies partial optimization, possibly with trade-offs in model complexity. The stark divergence between implementations highlights the importance of algorithmic efficiency in large language model deployment, particularly for parameter counts exceeding 10⁹. </details> Figure 14: Text generation inference time for LLMs. ## Appendix L Importance of prompt construction and comparison to GPT models Inspired by Kocoń et al. (2023), we compared the zero-shot performance of the RWKV-4-Raven-14B with ChatGPT (access in February 2023) and GPT-4 using several known NLP tasks, i.e., recognizing textual entailment (RTE), Winograd Natural Language Inference (WNLI), and recognizing emotions elicited in readers (GoEmotions and PolEmo2). Each model got the same prompts manually chosen to receive proper responses from the ChatGPT model. As shown in Tab. 6, RWKV performs significantly worse than ChatGPT and GPT-4 in several specific tasks. We suspect that this disparity is likely caused by the choice of prompts used to generate the answers since the prompts are written in natural language and do not take into account that RWKV, as an RNN, is unable to look back inside an instruction. | RTE WNLI GoEmotions | F1 Macro Accuracy F1 Macro | 88.1 81.7 25.6 | 91.3 91.6 23.1 | 44.2 47.9 7.9 | 74.8 49.3 7.9 | 92.1 97.9 52.8 | | --- | --- | --- | --- | --- | --- | --- | | PolEmo2 | F1 Macro | 44.1 | 41.0 | 38.2 | 40.9 | 76.4 | Table 6: ChatGPT, GPT-4 and RWKV-4-Raven-14B reasoning performance comparison in RTE Wang et al. (2019), WNLI Wang et al. (2018), GoEmotions Demszky et al. (2020), and PolEmo2 Kocoń et al. (2019) benchmarks. RWKV GPT prompts were primarily used for ChatGPT in Kocoń et al. (2023). SOTA is provided as a supplementary reference. When the instruction style was adapted (re-ordered) to respect that RNNs are not capable of "retrospective processing", the quality may significantly change, e.g., for RTE Wang et al. (2019) F1 Macro increased from 44.2% to 74.8%. We hypothesize that RWKV models are more sensitive to the position of the components in the context, as RNN-based architectures cannot look back and readjust the weight of previous information. For better performance, the desired information should be placed after the main question. Can you tell me if the hypothesis is entailment or is not entailment to the premise? premise: <here is a premise> hypothesis: <here is a hypothesis> | Aggression MathQA Sarcasm | F1 Macro Accuracy F1 Macro | 69.10 71.40 49.88 | 56.66 5.43 50.96 | 74.45 83.20 53.57 | | --- | --- | --- | --- | --- | | TweetSent | F1 Macro | 63.32 | 52.50 | 72.07 | | Unhealthy | F1 Macro | 45.21 | 43.30 | 50.96 | Table 7: ChatGPT and RWKV-4-Raven-14B performance comparison in Aggresion Wulczyn et al. (2017), Sarcasm Siddiqui (2019), Unhealthy Price et al. (2020), MathQA Cobbe et al. (2021), and TweetSent Barbieri et al. (2020) benchmarks. SOTA is provided as a supplementary reference. While separating the instruction from the input is relatively easy to do, some other aspects of prompt engineering are harder to quantify. For that purpose, we also tested the approach of stating the input after the question on multiple other tasks, i.e., aggression and sarcasm detection, classification of unhealthy (offensive) texts, mathematical Q&A, and sentiment analysis, see Tab. 7. The results suggest that better prompts might reduce the disparity between models. Raven achieves comparable results to ChatGPT on unhealthy conversation detection and even surpasses it on the sarcasm detection dataset. While such an approach to prompting looks necessary, it is not enough in itself to replace the capability of having free access to the whole context. Therefore, prompt engineering seems to be significantly more important for the RNN models rather than for standard transformers. It is entirely possible that good prompts to RNN models do not mean additional restrictions, but should simply be constructed using completely different guidelines. The authors of the aforementioned paper Kocoń et al. (2023) This is in line with the idea discussed in Wei et al. (2022b) perform chain-of-thought to improve results on the MathQA dataset. Even including this approach, the Raven model achieved a very low accuracy of 5.43%. Without it, the model performed even worse, performing only very basic and simple calculations and achieving 4.13% accuracy. Raven struggled with questions that required intermediate results. It is likely that the order of information presented in the math questions inside the dataset poses a challenge for the RWKV model. It is yet to be seen if prompt engineering can address this issue. This further emphasizes the importance of the order of information the model receives. Write the answer to the math question in the format: "Answer: ". The question is: <here is a question> ## Appendix M Cases In this part, we present a few instances of outputs produced by RWKV model using a Chat interface ttps://github.com/BlinkDL/ChatRWKV https://huggingface.co/spaces/BlinkDL/ChatRWKV-gradio. Tell me about ravens. Ravens are large, black birds with a distinctive white head and neck. They are found in most parts of the world, including North America, Europe, Asia, and Australia. Ravens are known for their intelligence and problem-solving abilities. They are also considered to be symbols of death and transformation in many cultures. Ravens are often associated with the afterlife or death because they have been known to eat carrion or even other birds. In some cultures, ravens are also believed to be messengers of the gods or guardians of treasure. Explain the concept of machine learning to a 10-year-old. Machine learning is like a super-smart computer that can learn on its own. It can recognize patterns and make decisions based on what it has learned. Just like how you might remember something from the past, the computer can remember what it’s learned and use it to make predictions or decisions in the future. It’s like having a smart friend who always knows what to do! Write a short poem describing nature’s beauty in springtime. Nature’s beauty in springtime is rare, A symphony of colors, a gentle surprise. The flowers bloom, the birds sing and sing, As the sun shines down, warming our skin. The trees sway in the gentle breeze, As we take in their beauty with ease. The sky is blue, the clouds are white, A sight to behold, it’s truly divine. Below is the numerical stable formula for updating the state variable in RWKV: $\displaystyle a_t^*$ $\displaystyle=e^p-qa_t-1+e^u+k_t-qv_t$ $\displaystyle b_t^*$ $\displaystyle=e^p-qb_t-1+e^u+k_t-q$ $\displaystyle wkv_t$ $\displaystyle=\frac{a_t^*}{b_t^*}$ Please write out the corresponding pytorch code to do so, where both $a_t$ and $b_t$ are torch.Tensor([n]) Here is the PyTorch code to update the state variable in RWKV using the given formula: ⬇ import torch def update_state (a, b, u, v, p, q): a_t = torch. exp (p - q) * a + torch. exp (u + k - q) * v b_t = torch. exp (p - q) * b + torch. exp (u + k - q) return a_t, b_t a, b = torch. randn (2, 3) u = torch. randn (2) v = torch. randn (2) wkv = torch. randn (2) a_t, b_t = update_state (a, b, u, v, wkv, torch. randn (2)) print ("a_t:", a_t) print ("b_t:", b_t) This code defines a function update_state that takes in the state variables a and b, the control variables u and v, and the parameters wkv and torch.randn. It then uses the PyTorch torch.exp function to compute the exponential of the control variables and the log of the normalization factor. Finally, it uses these values to update the state. [h] Explain the following metaphor: Life is like cats. The metaphor "Life is like cats" suggests that life can be unpredictable and full of unexpected twists and turns, just like a cat’s behavior. It implies that we cannot fully control or predict the outcome of our lives, just as we cannot control the movements of a cat. The metaphor also implies that life is complex and multifaceted, with many different variables and factors at play. Just as a cat can have multiple personalities or moods, life can have many twists and turns that are difficult to predict or understand. Overall, the metaphor suggests that life is complex and unpredictable, much like a cat’s behavior. [ht] Test the convergence of $∑_i=0^∞2^-i$ . The given series is a geometric series with the first term $a_0=2$ and common ratio $r=\frac{1}{2}$ . We can find the sum of this series using the formula for a geometric series: ∑_i=0^∞a_i = a 0 1-r = 1 2 1- 1 2 = 11- 1 2 = 21 = 2 So, the sum of the given series is $\boxed{2}$ . I would like to cook some Chinese food at home. Do you have any suggestions on the meal I could choose? Yes, of course! If you are not very experienced in cooking Chinese food, you could start with making noodles or rice, then make a dish using one of those. I also think making stir-fry with some meat is an excellent idea. I want to make something more special. What about red braised pork belly? Yes, red braised pork belly is also very popular and tasty. Tell me the steps to cook. You will need a few ingredients such as pork belly, star anise, sugar, ginger, scallions and wine. Put all of these in a pot, cook on medium heat and simmer until the meat is tender. After that, remove the meat from the pot and reserve the sauce. Slice the meat and add some ginger slices into it, then boil until it’s almost done. What do I need to be careful with the most during the cooking process? To cook pork belly properly, it is very important to monitor the heat of the stove and be patient with the cooking process. Overcooking it can cause the pork to be tough, dry and difficult to chew. You can add a few ingredients like soy sauce or black beans to give the pork more flavor. Tell me more about the proper heat and timing. If the temperature of the water is too high, it can damage the collagen and connective tissues, resulting in tough pork belly. It is essential to control the temperature carefully, especially when braising the pork. Additionally, keep the water simmering but not boiling to maintain a low simmer, as high heat will dry out the pork belly and result in tough and unpalatable meat. Lastly, monitor the heat during cooking and make sure the heat is medium, as this is what is most appropriate for slow cooking pork belly. I see. Thank you for your advice. You’re welcome! It was a pleasure chatting with you. Let me know if you need any further assistance in the future. [h] If all apples are fruits and all fruits are sweet, are all apples sweet? Please explain your answer in detail. Yes, all apples are fruits, and all fruits are sweet. Therefore, it is safe to assume that all apples are sweet. If birds can fly. Penguins are birds. Can penguins fly? Please explain your answer in detail. No, penguins cannot fly. Penguins are birds, but they are not capable of flight like other birds. They have wings and can glide for short distances, but they cannot take off or fly long distances like other birds. The day before two days after the day before tomorrow is Saturday. What day is it today? Please explain your answer in detail. Today is Friday. Two days after tomorrow is Sunday. Tomorrow is Saturday. If five cats can catch five mice in five minutes, how long will it take one cat to catch one mouse? Please explain your answer in detail. If five cats can catch five mice in five minutes, it means that each cat can catch one mouse in one minute. To find out how long it will take one cat to catch one mouse, we can use the concept of proportionality. Let’s assume that it takes x minutes for a cat to catch a mouse. Then, the number of mice caught by a cat in one minute is equal to the number of cats multiplied by the number of mice caught by each cat: x = 5 * 1 = 5 So, it will take 5 minutes for one cat to catch one mouse. Summarize the following paragraph. <The Abstract of this paper goes here> The paper presents a novel architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of Transformers with the efficient inference of RNNs. The model is able to scale to tens of billions of parameters and exhibits linear computational complexity during training and inference, making it a promising alternative to Transformers for sequence processing tasks. Please further summarize it in one sentence. The researchers have developed a new architecture that combines the efficient parallelizable training of Transformers with the efficient inference of RNNs, which allows for linear scaling in memory and computational requirements.