2404.05892v1

# ArXiv Paper 2404.05892v1 ## Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence ∗ Bo Peng 1,2, , Daniel Goldstein 2,3,* , Quentin Anthony 2,4,* , Alon Albalak 2,5 , Eric Alcaide 2,6,7 , Stella Biderman 2 , Eugene Cheah 1,2,3 , Teddy Ferdinan 8 , Haowen Hou 9 , Przemysław Kazienko 8 , Kranthi Kiran GV 2,10 , Jan Koco´ n 8 , Bartłomiej Koptyra 8 , Satyapriya Krishna 11 , Ronald McClelland Jr. 2,12 , Niklas Muennighoff 13 , Fares Obeid 2 , Atsushi Saito 2,14 , Guangyu Song 2 , Haoqin Tu 15,16 , Stanisław Wo´ zniak 8 , Ruichong Zhang 17 , Bingchen Zhao 18 , Qihang Zhao 19 , Peng Zhou 19 , Jian Zhu 20 , and Rui-Jie Zhu 16 1 RWKVProject (under Linux Foundation AI & Data), 2 EleutherAI, 3 Recursal AI, 4 Ohio State University, 5 University of California, Santa Barbara, 6 Charm Therapeutics, 7 Dalle Molle Institute for Artificial Intelligence Research, 8 Wroclaw Tech, 9 Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), 10 New York University, 11 Harvard University, 12 Ronsor Labs, 13 Contextual AI, 14 Nextremer Co. Ltd., 15 University of Chinese Academy of Sciences, 16 University of California, Santa Cruz, 17 Tsinghua University, 18 University of Edinburgh, 19 LuxiTech Co. Ltd., 20 University of British Columbia ## Abstract We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) (Peng et al., 2023) architecture. Our architectural design advancements include multiheadedmatrix-valued states and a dynamic recurrence mechanism that improve expressivity while maintaining the inference efficiency characteristics of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a fast tokenizer based on greedy matching for enhanced multilinguality. We trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two Finch models with 1.6 and 3.1 billion parameters and find that they achieve competitive performance across a wide variety of benchmarks. We release all our models on HuggingFace under the Apache 2.0 license. 1 ∗ Equal first authorship. Others listed alphabetically. 1 Models at: https://huggingface.co/RWKV Training code at: https://github.com/RWKV/RWKV-LM Inference code at: https://github.com/RWKV/ChatRWKV Time-parallel training code at: https://github.com/RWKV/RWKV-infctx-trainer ## Contents | 1 Introduction | 1 Introduction | 1 Introduction | 3 | |----------------------------|-------------------------------------------------|-------------------------------------------------|-------------------------------------------------| | 2 Background | 2 Background | 2 Background | 4 | | 3 Eagle/Finch Architecture | 3 Eagle/Finch Architecture | 3 Eagle/Finch Architecture | 5 | | 4 Method | 4 Method | 4 Method | 5 | | | 4.1 Eagle . . . . . . . . . . . . . . . . . . . | 4.1 Eagle . . . . . . . . . . . . . . . . . . . | 4.1 Eagle . . . . . . . . . . . . . . . . . . . | | | | | 6 | | | 4.1.1 | Eagle Token Shift . . . . | 6 | | | 4.1.2 | Eagle Time Mixing . . . | 6 | | | 4.1.3 | Channel Mixing . . . . . | 7 | | | 4.2 Finch | . . . . . . . . . . . . . . . | 7 | | | 4.2.1 | Finch Token Shift . . . . | 7 | | | 4.2.2 | Finch Time Mixing . . . | 8 | | 5 | RWKVWorldTokenizer | RWKVWorldTokenizer | 8 | | 6 | RWKVWorldv2Dataset | RWKVWorldv2Dataset | 9 | | 7 | Pre-Trained | Models | 9 | | 8 | Language Modeling Experiments | Language Modeling Experiments | 9 | | | 8.1 | LMEvaluation Harness Benchmarks | 9 | | | 8.2 | Associative Recall . . . . | 11 | | | 8.3 | . . . . Long Context Experiments . . | 12 | | | 8.4 | Bamboo Benchmark . . . . . . | 12 | | 9 | Speed and MemoryBenchmarks | Speed and MemoryBenchmarks | 14 | | 10 | Multimodal Experiments | Multimodal Experiments | 15 | | | 10.1 RWKVMusic Modelling . | . . . | 15 | | | 10.2 VisualRWKV . . . . . . . . | . . . | 15 | | 11 | Conclusions | Conclusions | 16 | | A | Author Contributions | Author Contributions | 27 | | B | Additional Architecture Details | Additional Architecture Details | 28 | | C | Additional Related Work | Additional Related Work | 29 | | D | Training Dataset Details | Training Dataset Details | 32 | | E | Computing Costs | Computing Costs | 33 | | F | Additional Evaluations | Additional Evaluations | 34 | | | F.1 | Alignment Benchmark . . . . . | 34 | | | F.2 MTBench | . . . . . . . . . . . . . | 34 | | | F.3 Self-Learning . . | . . . . . . . . | 35 | | | F.4 Zero-shot evaluation on | additional | 35 | | G | Hyperparameters | Hyperparameters | 36 | | H | Parameter Initializations | Parameter Initializations | 37 | | I | Non-English | Chat Examples | 37 | | J | Chat Examples - Comparison with RWKV-4 | Chat Examples - Comparison with RWKV-4 | 40 | ## 1 Introduction Advancements in Large Language Models (LLMs) have significantly impacted Natural Language Processing (NLP) tasks. The field has traditionally been dominated by the transfomer architecture (Vaswani et al., 2023). However, the expressive attention mechanism of transformers leads them to suffer from quadratic time complexity with respect to input sequence length. Various methods have been proposed to achieve sub-quadratic time complexity without significantly changing the core attention mechanism, typically relying on some form of sparsity techniques (Child et al., 2019a; Beltagy et al., 2020; Zaheer et al., 2020). Recent works have achieved sub-quadratic time complexity without significantly sacrificing performance by introducing new mechanisms to replace attention at the core of the Transformer architecture. These models include gated recurrences (Fu et al., 2023; Gu & Dao, 2023; Gu et al., 2021; Sun et al., 2023; Katsch, 2023; Qin et al., 2023; Smith et al., 2023), gated convolutions (Poli et al., 2023; Peng et al., 2023), data-dependent linear attention (Yang et al., 2023; Katharopoulos et al., 2020b), sparse attentions (Tay et al., 2020; Child et al., 2019b; Zaheer et al., 2020; Qiu et al., 2019) and their combinations (De et al., 2024; Qin et al., 2024; 2022). We build off RWKV-4 introduced in Peng et al. (2023), which provides efficient inference and training along with a parallelizable implementation compared to competing architectures as shown in Table 1. Table 1: Comparative analysis of RWKV-4/5/6 and other LLM architectures regarding time and memory complexity for both inference per token and training per sequence, and training parallelizability across the sequence dimension. The context/sequence length is denoted by N . | Architecture | Inference | Inference | | Training | Training | |--------------------|-------------|-------------|----------|---------------|------------| | | Time | Memory | Parallel | Time | Memory | | LSTM/LMU | O (1) | O (1) | ✗ | O ( N ) | O ( N ) | | Transformer | O ( N ) | O ( N ) a | ✓ | O ( N 2 ) | O ( N ) b | | Linear Transformer | O (1) | O (1) | ✓ | O ( N ) | O ( N ) | | H3/S4 | O (1) | O (1) | ✓ | O ( N log N ) | O ( N ) | | Hyena | O ( N ) | O ( N ) | ✓ | O ( N log N ) | O ( N ) | | RWKV/Mamba/RetNet | O (1) | O (1) | ✓ | O ( N ) | O ( N ) | a O (1) without KV cache b With Flash Attention In this paper, we introduce two new architectures: Eagle (RWKV-5) and Finch (RWKV-6). First, Eagle improves upon the architecture and learned decay schedule from RWKV-4 (Peng et al., 2023) through the use of expressive multi-headed matrix-valued states (as opposed to vector-valued states), a reformulated receptance, and an additional gating mechanism. Finch further improves the expressivity and flexibility of the architecture by introducing new data-dependent functions for both the time-mixing and token-shift modules, consisting of parameterized linear interpolations. Additionally, Finch proposes a novel use of the Low Rank Adaptation (Hu et al., 2022) function to allow for trainable weight matrices to efficiently augment the learned data decay vectors in a context-dependent manner. Finally, we introduce a new tokenizer, the RWKV World Tokenizer, and a new dataset, RWKV World v2 (1.12 trillion tokens), specially designed to improve performance on multilingual and code data. Through extensive experimentation, we show that the Eagle and Finch models perform competitively, or improve upon existing models under a wide variety of sequence modeling domains and tasks. Specifically, we evaluate our trained models on commonly used English-only and multilingual text benchmarks, associative recall, music modeling, and vision-language benchmarks. Our experiments demonstrate that the advancements in Eagle and Finch provide significant progress towards developing more efficient AI models In summary, our main contributions are: - The Eagle (RWKV-5) and Finch (RWKV-6) RWKV architectures, which significantly improve over RWKV-4 on benchmarks for LLMs. - The RWKV World Tokenizer which contains underrepresented languages' vocabulary and which performs fast tokenization with Trie-based greedy matching. - The RWKV World v2 public dataset, comprised of 1.12 trillion tokens of publicly available multilingual data. - Public release of four pre-trained Eagle models, scaling from 0.46 to 7.5 billion parameters, and two Finch models, with 1.6 and 3.1 billion parameters. Demonstrating that these novel architectures are competitive to transformers when trained using enough FLOPs to make meaningful scaling conclusions. - A completely open training pipeline to enable interpretability and reproducibility of alternative-architecture LLMs (See Table 2). | Model | Context Length | Training | Open | Open Code | Open Code | Open | |-----------------|------------------|-------------|---------|-------------|-------------|---------| | | | Tokens | Weights | Inference | Training | Dataset | | GPT-4 | 128k a | Undisclosed | | # | | # | | LLaMA2 7B | 4k | 2.0 × 10 12 | # | | # # | # | | Mistral 7B v0.1 | 32k b | Undisclosed | G # | | # | # | | Gemma7B | 8k | 6.0 × 10 12 | G # | | | # | | StableLM 7B v2 | 4k | 1.1 × 10 12 | | | | | | Pythia 6.9B | 2k | 3.3 × 10 11 | | | | | | Eagle 7B | Indefinite c | 1.1 × 10 12 | | | | | Table 2: Comparison of the openness and accessibility of public foundational LLMs with 7B+ parameters regarding model weights, official inference/training code, and dataset. Widely available but not under an open source license is indicated by . # G a OpenAI's gpt-4-0125-preview model b With sliding window attention c Pretrained with context length 4096, but no fundamental context length limitation or relationship to speed, see 8.3 for extrapolation details ## 2 Background Eagle and Finch are RNNs based on a multi-headed hybridization of the RWKV-4 architecture and linear attention. We discuss related work and the evolution of these two architectures below, with a more detailed review given in Appendix C. Recurrent Neural Networks (RNNs) are well suited to provide inexpensive inference on sequence modelling tasks, typically operating in O(1) time complexity per step with respect to sequence length. They model sequences with time dependencies by generating a hidden state ht at each time step, which is fed back in at the next time step as a secondary input. Classic RNNs (e.g. LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014)) became widely used for sequence modelling, but are difficult to parallelize across the time dimension for training. The Transformer architecture has enjoyed remarkable success in generative sequence modelling, and language modelling in particular (Vaswani et al., 2023; Radford et al., 2018), providing SOTA performance across many tasks. However, the use of multi-headed dot-product self-attention (MHA) leads to a quadratic time complexity with respect to sequence length. The deficiencies of classic RNNs and Transformers led to many attempts to develop architectures incorporating the best features of both in a single model, namely O(1) per token time complexity and fast highly parallelizable training. Linear Attention (Schmidhuber, 1992; Katharopoulos et al., 2020a) replaces the numerator of MHA's softmax( QK T ) V with φ ( Q ) φ ( K ) T V , allowing a reordering of operations via associativity to φ ( Q )( φ ( K ) T V ), where φ represents a non-negative feature-map function. It can be computed as an RNN in O (1) time per step by adding φ ( K T i ) Vi to a recurrent state at each time step i , or trained in parallel much like MHA. This accomplishes the main goals outlined above, but naive linear attention suffers from significantly reduced performance compared to MHA-based transformers. A modified form of linear attention, the Attention Free Transformer (AFT) (Zhai et al., 2021), paved the way for the RWKV architecture, by using a number of attention heads equal to the size of the feature dimension and incorporating a set of learned pairwise positional biases, denoted as w . $$\frac { \sum _ { i = 1 } ^ { n } q ( q _ { i } ) o \sum _ { i = 1 } ^ { n } exp ( k _ { i } ) } { \sum _ { i = 1 } ^ { n } exp ( k _ { i } ) }$$ RWKV-4reformulates the AFT equation by replacing the pair-wise positional biases with a channelwise vector of additive weight decay rates w . It also adds a bonus term u to offset the weight of only the current input specially. $$\sum _ { i = 1 } ^ { n k v _ { 1 } } \frac { z _ { i } ^ { - 1 } e^{i ( t - 1 - j ) w + k _ { 1 } } } { z _ { i } ^ { - 1 } e^{i ( t - 1 - j ) u + k _ { 1 } } } .$$ RWKV-4 also adds token-shift and gating to both attention and feed-forward sub-blocks of transformer, and small embedding initialization and normalization to quickly arrive at well-distributed token embeddings. Combining all of these architectural changes led RWKV-4 to become the first RNNto rival the performance of Transformers, while maintaining fast parallelizable training and O (1) time complexity per token. There has been a recent revival of RNNs in NLP research (Tiezzi et al., 2024). HGRN(Qin et al., 2023) is a recent time-parallelizable data-dependent RNN that employs input and forget gates. TransNormer(Qin et al., 2022) applies RMSNorm to linear attention to bound its output. Other new time-parallelizable data-dependent RNNs have also been invented concurrently with our work including GLA (Yang et al., 2023) and Griffin (De et al., 2024). State Space Models (SSMs) employ a hidden state of basis function weights to model an approximation of the input function (Gu et al., 2020), updating that hidden state via a differential equation. Earlier SSMs (Gu et al., 2022) were historically computed using long convolutions in O ( N log N ) time per sequence, but could also be formulated as a recurrent network. Recently, it has been shown that SSMs can be parallelized across the time dimension via techniques including associative scan (Smith et al., 2023). A new class of SSMs has also emerged concurrently with our work (Katsch, 2023; Gu & Dao, 2023) that feature data-dependent A and B terms, which function similarly to the data-dependent dynamic recurrence used in Finch. ## 3 Eagle/Finch Architecture Werefine the RWKV architecture in two steps, and observe significant modeling improvements with each. Compared to the baseline RWKV-4, Eagle adds matrix-valued attention states, LayerNorm over the attention heads, SiLU attention gating, and improved initialization. It also removes the Sigmoid activation of receptance. Finch further applies data-dependence to the decay schedule and token-shift. The core architecture remains similar to that of RWKV-4, consisting of a series of stacked residual blocks shaped like a traditional Transformer. Following notation from (Tolstikhin et al., 2021), each block contains one Pre-LayerNorm Time-Mixing sub-layer followed by one Pre-LayerNorm Channel-Mixing sub-layer, as depicted in Figure 1, left. These correspond to the traditional Attention and Feed Forward Network sub-layers of the Transformer. See Appendix B for more details on our training implementation and the differences from RWKV-4, and Section 9 for speed and memory benchmarks. ## 4 Method In this section, we use D to denote the model dimension, and unless explicitly stated, all vectors appearing in this section are dimension D / h , where h denotes the number of heads, belonging to R ( D / h ) . For compactness and simplicity we show calculations per-head, eliding the head index. Weuse the convention that all vectors are row vectors unless explicitly transposed, so all matrices operate on the right side. MLP ?' MLP yerNorm ' Norm Out t w ? K yerNorm Norm Channel Mixing ? Channel Mixing G' ? G' R R Time Mixing LayerNorm * SiLU Time LayerNorm WKV * DxD WKV V K Mixing ? ? DxD G V SiLU G ? x ht-1 ? ? DxD w ht-1 ? ? DxD ot ht ? ? ? ? D DxD w Token Shift ht ? ? DxD <details> <summary>Image 1 Details</summary>  ### Visual Description ## Diagram: Neural Network Architecture with Channel and Time Mixing ### Overview The image depicts a technical diagram of a neural network architecture, likely for a transformer-based model. It includes components for **Channel Mixing**, **Time Mixing**, **Token Shift**, and **KV (Key-Value) operations**. The diagram uses colored blocks, arrows, and mathematical notations to represent data flow and operations. ### Components/Axes #### Left Side: Channel Mixing - **Blocks**: - **σ (sigma)**: Orange block, likely a sigmoid activation function. - **G'**: Orange block, possibly a gate or transformation matrix. - **MLP**: Orange block, representing a multi-layer perceptron. - **μ'**: Yellow block, likely a mean or bias term. - **LayerNorm**: Yellow block, layer normalization. - **Flow**: - Inputs: `σ` and `G'` feed into `MLP`, which outputs `μ'`. - `μ'` is normalized via `LayerNorm`. #### Right Side: Time Mixing - **Blocks**: - **Out**: Light blue block, output of the time mixing process. - **SiLU**: Orange block, sigmoid linear unit activation. - **G, R, w, WKV, K, V**: Orange blocks, representing gates, residual connections, weights, and key-value matrices. - **LayerNorm**: Yellow block, normalization. - **Flow**: - Inputs: `G`, `R`, `w`, `K`, `V` are combined via `WKV` (Key-Value) operations. - Output: `Out` after `LayerNorm`. #### Token Shift Diagrams (Right Side) 1. **KV Block**: - **Elements**: - `h_{t-1} ∈ ℝ^{DxD}`: Hidden state from previous time step. - `x_{t-1} ∈ ℝ^D`: Input at previous time step. - `KV ∈ ℝ^{DxD}`: Key-Value matrix. - `u, k, v`: Intermediate variables. - **Flow**: - `h_{t-1}` and `x_{t-1}` are combined via `KV` to produce `h_t`. 2. **Token Shift (Left)**: - **Elements**: - `λ ∈ ℝ^{1xD}`: Scaling parameter. - `L2 ∈ ℝ^{DxL}`: Linear transformation. - `tanh`: Hyperbolic tangent activation. - `Shift1 Right`: Positional shift operation. - **Flow**: - `x_{t-1}` is transformed via `L2`, `tanh`, and `Shift1 Right`. 3. **Token Shift (Right)**: - **Elements**: - `μ ∈ ℝ^{1xD}`: Mean parameter. - `1-μ ∈ ℝ^{1xD}`: Complementary parameter. - `L1 ∈ ℝ^{DxR}`: Linear transformation. - `Shift1 Right`: Positional shift. - **Flow**: - `x_{t-1}` is split into `μ` and `1-μ`, then transformed via `L1` and `Shift1 Right`. ### Detailed Analysis - **Channel Mixing**: - Combines `σ` and `G'` via `MLP` to generate `μ'`, which is normalized. - Uses `LayerNorm` to stabilize training. - **Time Mixing**: - Integrates `G`, `R`, `w`, `K`, and `V` through `WKV` operations. - Outputs `Out` after normalization. - **Token Shift**: - Involves positional encoding via `Shift1 Right` and parameterized transformations (`λ`, `L1`, `L2`). - Uses `tanh` for non-linearity and `LayerNorm` for stability. ### Key Observations - **Color Coding**: - Orange blocks represent activation functions (`σ`, `SiLU`, `tanh`). - Yellow blocks denote normalization (`LayerNorm`, `μ`, `μ'`). - Red blocks (`WKV`) indicate key-value operations. - **Mathematical Notations**: - `ℝ^D`, `ℝ^{DxD}`: Dimensionality of input/output spaces. - `Shift1 Right`: Likely a cyclic shift operation for positional encoding. ### Interpretation This diagram illustrates a hybrid neural network architecture combining **channel-wise** and **time-wise** mixing mechanisms, common in transformer variants like **Mamba** or **SSM (State Space Models)**. - **Channel Mixing** focuses on feature interactions within a single time step. - **Time Mixing** captures temporal dependencies across steps. - **Token Shift** introduces positional encoding via parameterized shifts, replacing traditional positional embeddings. - The use of `LayerNorm` and `MLP` suggests a focus on stability and non-linear transformations. The architecture likely optimizes for efficiency in sequence modeling tasks, balancing computational complexity with performance. The `WKV` block and `Token Shift` operations highlight innovations in attention mechanisms and positional encoding. </details> t-1 ? ? LxD x t ? ? LxD ? ? ? ? DxD ? D LERP LERP ? ? 1xD X? ? LxD Shift1 Right LayerNorm X? ? LxD Shift1 Right Figure 1: RWKV architecture overview. Left: time-mixing and channel-mixing blocks; top-right: RWKVtime-mixing block as RNN cell; center-bottom: token-shift module in FeedForward module and Eagle time-mixing; bottom-right: token-shift module in Finch time-mixing. All shape annotations assume a single head for simplicity. Dashed arrows (left, top-right) indicate a connection in Finch, but not in Eagle. ## 4.1 Eagle ## 4.1.1 Eagle Token Shift Weadopt the Token Shift technique from the previous RWKV, similar to a 1D causal convolution of size = 2, as can be seen in Figure 1, center-bottom. To better introduce the Token Shift technique, we define some notation. The l inear int erp olation (lerp) between xt and xt -1 used in RWKV-4 and Eagle Token Shift is defined as: $$\lvert e r p _ { \sigma } ( a , b ) = a + ( b - a ) \circ u \rvert$$ where each µ □ ∈ R D is a learnable vector. Token Shift allows the model to learn how much new versus old information should be allocated per time step to each channel of receptance, key, value, and gate vectors ( r , k , v , and g respectively) independently and uniquely for each head. This makes it possible to form induction heads (Elhage et al., 2021) within a single layer since even a single head can directly accumulate both past and current token data into separate subspaces within these vectors. ## 4.1.2 Eagle Time Mixing The formula of Eagle Time Mixing can be written as follows: ? ot ? ? D 1xD ? 1xD X LxD Token Shift Token Shift ? ? ? 1xD Shared 1-? 1xD parameters ? 1xD ? 1xD X LxD X LxD Token Shift Token Shift ? Token Shift ? ? ? 1xD ? ? 1xD ? L2 ? DxL tanh ? ? 1xD ? Token Shift L1 ? Tok neg To L-Shift1 Right Shared ? LxD X? parameters Shared parameters L-Shif L- ? ? DxL L2 tanh L1 DxR ? ? DxL L2 ? ? ? DxR ? ? X? ? ? ? 1xD 1xD LxD ? ? tanh L1 ? ? ? 1xD ? ? DxR ? ? DxL L2 tanh ? ? ? 1xD ? ? Shift1 Right DxR X? L1 ? LxD ? ? ? X? ? 1xD LxD X? ? LxD S ## 4.2 Finch ## 4.2.1 Finch Token Shift The d atad ependent l inear int erp olation (ddlerp) between xt and xt -1 used in Finch Token Shift is defined as: $$\lvert \log ( x ) = A _ { 0 } + \tan h ( x A _ { 0 } ) B _ { 0 } \rvert$$ $$d ( l e r p _ { \sigma } ( a , b ) = a + ( b - a ) \circ l o r a _ { \sigma } ( a )$$ where µ x and each λ □ introduce a trainable vector of dimension D and each A □ ∈ R D × 32 , B □ ∈ R 32 × D introduce new trainable weight matrices. For the special case of LoRA ω seen below we introduce double-sized trainable weight matrices A ω ∈ R D × 64 , B ω ∈ R 64 × D . A schematic representation can be found in Figure 1, bottom-right. Please note that future 7B and larger Finch models are expected to further increase the size of these weight matrices by double or more. This new form of Token Shift enhanced with data-dependence is intended to expand the abilities of the model beyond the RWKV-4/Eagle style of Token Shift so that the amount of new and old data allocated per channel now depends on the input at both current and prior time steps. $$w = exp ( - exp ( w ) )$$ $$w _ { k 0 } = \diag ( u ) \cdot k ^ { - 1 } _ { i } + \sum _ { j = 1 } ^ { n } diag ( w ) ^ { - 1 } _ { j }$$ $$\begin{aligned} o _ { f } = \text { conc at } ( S I L U ( g ) ) \circ \text { LayerNorm } ( r ) \\ w _ { k v } \left( W _ { e } ^ { R D } \right) & \in \mathbb { R } ^ { n } \end{aligned}$$ Where LayerNorm operates on each of h heads separately, which is also equivalent to the GroupNorm (Wu & He (2018)) operation on h groups. It is also worth noting that w is obtained from w = exp( -exp( ω )), where ω ∈ R D / h are the actual headwise trainable parameters. This ensures that w falls within the interval (0,1), guaranteeing that diag( w ) is a contraction matrix. The wkv t attention calculation can alternatively be written in a recurrent form: $$w k v ^ { \prime } = s + diag ( u ) , k ^ { \prime } . v$$ $$s ^ { \prime } = diag ( w ) \cdot s + k ^ { T } \cdot v$$ RWKV's wkv term can be considered a decay-based equivalent to the normalised k T v term in Linear Attention. It is instructive to note how for a given head j the recurrent state s is a sum of k T v where each channel of s individually decays by the corresponding channel of w at each time step. Prior to the application of the receptance vector, gating, and output weights, a per-channel learned boost u is multiplied with the current token's k T v and summed with the state, as can be seen in Figure 1, top-right. This gives the current token special treatment relative to the sum of past tokens contained within the decaying state history. The receptance is multiplied by this sum, acting like the query term in Linear Attention. ## 4.1.3 Channel Mixing In both Eagle and Finch, the Channel Mixing module is identical to the previous RWKV-4 architecture, except for a slightly reduced hidden dimension from 4 D to 3.5 D . This reduction accounts for new gating weights in Eagle Time Mixing to ensure an equi-parameter relation with the prior model at the same number of layers and embedding dimension. We do not further reduce the hidden dimension in Finch despite adding a small number of new parameters for LoRA weights. The formulas for Channel Mixing are the same as RWKV-4, but we restate them here to ensure notational consistency, using linear interpolation from Equation 3: $$r _ { t } = l e r p _ { ( x _ { 0 } , x _ { 1 } ) } W _ { r } \in R ^ { D }$$ $$k _ { t } = l e r p _ { g } ( x _ { r } , x _ { r } ^ { \prime } ) W _ { k } \in R ^ { 3 . 5 D }$$ $$v _ { i } = ReLU ( k _ { j } ) ^ { 2 } W _ { v } e ^ { i R D }$$ $$o _ { t } ^ { \prime } = o ( r _ { 1 } ) o _ { t } ^ { \prime } e R P$$ ## 4.2.2 Finch Time Mixing $$\sum _ { i = 1 } ^ { n } \sum _ { j = 1 } ^ { m } ( x _ { i , j } - ) W _ { O }$$ $$d _ { i } = l o r a _ { d } ( d d l e r p _ { a } ^ { ( x _ { 1 } , x _ { 2 } - 1 ) } )$$ $$w _ { 1 } = e x p ( - e x p ( d _ { 1 } )$$ $$w k v = diag ( w ) \cdot k _ { 1 } ^ { - 1 } r + \sum _ { i = 1 } ^ { n } diag ( \circ ) \cdot w _ { i }$$ $$\begin{aligned} o _ { 1 } = \conc at ( S I L U ( g ) \circ LayerNorm ( r ) ) \\ w _ { 0 k v } \cdot \vec { w _ { 0 } } \in R ^ { 2 0 } \end{aligned}$$ The wkv t attention calculation can alternatively be written in a recurrent manner: $$w k v ^ { \prime } = s + diag ( u ) \cdot k ^ { T } \cdot v$$ $$s ^ { \prime } = diag ( w ) \cdot s + k ^ { T } \cdot v$$ Unlike in Eagle, wt here is not static across the sequence (dashed arrows in Figure 1, left and topright.). This is the core change to decay in Finch, as each channel of wt cannowvaryindependently over time, in a data-dependent manner, whereas previously it was a fixed learned vector. The new LoRA mechanisms above are used to take learned vectors, as seen in Eagle, and inexpensively augment them with additional offsets determined by the incoming input. Note that the LoRA process itself uses an Eagle style Token-Shifted value as its input, not just the latest token. The new time-varying decay wt goes one step further, applying LoRA again afterward. Intuitively, this is a second-order variant of Token-Shifting, allowing each channel of wt to vary based on a mix of the current and prior tokens, with the mix itself determined by aspects of both tokens. ## 5 RWKVWorldTokenizer Tokenization is important in language modelling as it conditions the learning relationships between tokens and the generation of new text based on those patterns. The numbers of tokens to build a single semantic chunk are, however, often very unequally distributed against nonEuropean and other underrepresented languages. Byte-pair-encoding (BPE) based tokenizers which are trained with this inequality result in not only lower performances against underrepresented languages but also undue economic costs such as inference Ahia et al. (2023) and continual pre-training with extended vocabulary Lin et al. (2024); Sasaki et al. (2023). To address these problems, we manually select tokens from multiple vocabulary files such that non-European languages are well represented. To construct the tokenizer's vocabulary, we merge the vocabularies of the following tokenizers and then manually select the tokens for non-European languages. - GPT-NeoX-20B (Black et al., 2022): https://huggingface.co/EleutherAI/ gpt-neox-20b - GPT2(Radford et al., 2019): https://huggingface.co/openai-community/ gpt2 - cl100k\_base of tiktoken: https://github.com/openai/tiktoken - Llama2 (Touvron et al., 2023): https://huggingface.co/meta-llama/ Llama-2-7b-hf - Bloom (Workshop et al., 2023): https://huggingface.co/bigscience/ bloom This tokenizer has a vocabulary size of V = 65536, numbered from 0 through 65535, where tokens are arranged by their lengths in bytes. Below is a brief overview: - Token 0: Represents the boundary between text documents, known as <EOS> or <SOS> . This token doesn't encode any specific content and is only used for document separation. - Tokens 1-256: Consist of byte encodings (Token k encodes byte k -1), wherein tokens 1-128 correspond to standard ASCII characters. - Tokens 257-65529: Tokens with a minimum length of 2 bytes in UTF-8, including words, prefixes and suffixes, accented letters, Chinese characters, Hangul, Hiragana, Katakana and emojis. For example, Chinese characters are allocated from token 10250 to 18493. - Token 65530-65535: Reserved tokens for future use. These designations are intended to enhance the tokenizer's efficiency on the multilingual corpus, as well as on source code of programming languages. This tokenizer is implemented via a Trie (Prefix Tree) to boost speed while maintaining simplicity. Encoding is performed as matching the longest element in vocabulary with an input string from left to right. We note that our tokenizer's vocabulary construction is to mitigate undue burden, which naive BPE and related methods cause, on minor languages. ## 6 RWKVWorldv2Dataset Wetrain our models on the new RWKVWorldv2Dataset , a new multilingual 1.12 trillion token dataset drawn from a wide variety of hand selected publicly available data sources. This dataset is designed to go beyond the English-heavy focus of many datasets widely used to train LLMs today. We do this to support usage by the majority of the worldwide population who are not native English speakers, to improve representation within model responses, and also to enable transfer learning so that our models can apply knowledge across cultures and locales. We put a strong emphasis on factual knowledge and code, but also on cultural works including stories, books, subtitles, and conversations. The source data is approximately 70% English, 15% multilingual, and 15% code. We describe the components of our dataset in detail in Appendix D. ## 7 Pre-Trained Models We have pre-trained and publicly released the six Apache 2.0 licensed Eagle and Finch models: Eagle 0.4B , Eagle 1.5B , Eagle 3B , Eagle 7B , Finch 1.6B , and Finch 3B . All of the models were trained on the 1.12 trillion token RWKV World v2 multilingual corpus. See Appendix E for detailed parameter counts and FLOPs calculations. ## 8 Language Modeling Experiments ## 8.1 LMEvaluation Harness Benchmarks To assess the performance of Eagle and Finch models, we evaluate on a series of common multilingual and English-focused benchmarks using lm\_evaluation\_harness (Gao et al., 2023) as shown in Tables 3 and 4. We find that Eagle and Finch demonstrate exceptionally high capabilities on multi-lingual benchmarks, with nearly all results significantly outperforming the other similarly sized models we tested. In figures 2 and 3 we plot the accuracy versus FLOPs used to train various open models across a similar set of common benchmarks. For multilingual benchmarks, Eagle and Finch represent a substantial improvement to the Pareto frontier, achieving far higher scores than other models trained for a similar number of FLOPs. The two models additionally obtain competitive performance across these English benchmarks. Figure 2: Multilingual average benchmark accuracy versus training FLOPs. Average of LAMBADA Multilingual, xStoryCloze, xWinoGrande, and xCOPA <details> <summary>Image 2 Details</summary>  ### Visual Description ## Scatter Plot: Model Accuracy vs. Training FLOPs ### Overview The image is a scatter plot comparing the accuracy of various machine learning models against their training computational cost (measured in FLOPs). The plot uses distinct symbols and colors to represent different models, with a logarithmic scale on the x-axis (Training FLOPs) and a linear scale on the y-axis (Accuracy). The legend is positioned in the top-left corner, and data points are distributed across the plot. --- ### Components/Axes - **X-axis (Training FLOPs)**: Logarithmic scale ranging from $10^{22}$ to $10^{24}$. - **Y-axis (Accuracy)**: Linear scale from 0.48 to 0.62. - **Legend**: Located in the top-left corner, mapping symbols/colors to model names: - **Red circle**: finch - **Blue cross**: eagle - **Yellow plus**: python - **Black triangle**: mamba - **Red star**: stablelm - **Cyan triangle**: btlm-3b-8k-base - **Green triangle**: falcon-7b - **Yellow diamond**: Llama-2-7b --- ### Detailed Analysis #### Data Points and Trends 1. **finch** (red circle): - Position: ~$10^{23}$ FLOPs, ~0.56 accuracy. - Trend: Moderate accuracy with mid-range FLOPs. 2. **eagle** (blue cross): - Position: ~$10^{23}$ FLOPs, ~0.61 accuracy. - Trend: Highest accuracy among models with ~$10^{23}$ FLOPs. 3. **python** (yellow plus): - Position: ~$10^{22}$ FLOPs, ~0.49 accuracy. - Trend: Lowest accuracy and FLOPs. 4. **mamba** (black triangle): - Position: ~$10^{22}$ FLOPs, ~0.52 accuracy. - Trend: Slightly higher accuracy than python but similar FLOPs. 5. **stablelm** (red star): - Position: ~$10^{23}$ FLOPs, ~0.52 accuracy. - Trend: Lower accuracy than finch and eagle despite similar FLOPs. 6. **btlm-3b-8k-base** (cyan triangle): - Position: ~$10^{22}$ FLOPs, ~0.52 accuracy. - Trend: Matches mamba in accuracy but lower FLOPs. 7. **falcon-7b** (green triangle): - Position: ~$10^{23}$ FLOPs, ~0.55 accuracy. - Trend: Higher accuracy than stablelm and btlm-3b-8k-base. 8. **Llama-2-7b** (yellow diamond): - Position: ~$10^{24}$ FLOPs, ~0.56 accuracy. - Trend: Highest FLOPs but only marginally higher accuracy than finch. #### Key Observations - **High FLOPs, High Accuracy**: Models like **eagle** and **finch** achieve the highest accuracy (~0.61 and ~0.56) with ~$10^{23}$ FLOPs. - **Low FLOPs, Low Accuracy**: **python** and **mamba** have the lowest FLOPs (~$10^{22}$) and accuracy (~0.49–0.52). - **Outliers**: - **Llama-2-7b** (yellow diamond) has the highest FLOPs (~$10^{24}$) but only ~0.56 accuracy, suggesting diminishing returns at extreme computational costs. - **btlm-3b-8k-base** (cyan triangle) and **mamba** (black triangle) show similar accuracy (~0.52) despite differing FLOPs (~$10^{22}$ vs. ~$10^{22}$). - **No Clear Correlation**: While some models (e.g., eagle, finch) show a positive trend between FLOPs and accuracy, others (e.g., stablelm, Llama-2-7b) deviate, indicating variability in efficiency. --- ### Interpretation The plot highlights the trade-off between computational cost (FLOPs) and model performance (accuracy). While higher FLOPs generally correlate with better accuracy (e.g., eagle, finch), exceptions like **Llama-2-7b** suggest that efficiency and architectural design play critical roles. Models like **mamba** and **btlm-3b-8k-base** demonstrate that lower FLOPs can achieve competitive accuracy, emphasizing the importance of optimization. The lack of a strict linear relationship implies that factors beyond raw computational power—such as model architecture, training data quality, and algorithmic efficiency—significantly influence outcomes. This underscores the need for balanced approaches in model development, prioritizing both scalability and efficiency. </details> Figure 3: English average benchmark accuracy versus training FLOPs. Average of LAMBADA (OpenAI), PIQA, StoryCloze16, HellaSwag, WinoGrande, Arc (challenge), Arc (easy), HeadQA (English), OpenBookQA, SciQ, ReCoRD and COPA <details> <summary>Image 3 Details</summary>  ### Visual Description ## Scatter Plot: Model Accuracy vs. Training FLOPs ### Overview The image is a scatter plot comparing the accuracy of various AI models against their training computational cost (measured in FLOPs). The x-axis represents training FLOPs on a logarithmic scale (10²² to 10²⁴), while the y-axis shows accuracy (0.58–0.70). Each model is represented by a unique symbol and color, with a legend in the top-left corner. --- ### Components/Axes - **X-axis (Training FLOPs)**: Logarithmic scale from 10²² to 10²⁴. - **Y-axis (Accuracy)**: Linear scale from 0.58 to 0.70. - **Legend**: Located in the top-left corner, mapping symbols/colors to models: - **Red circle**: finch - **Blue cross**: eagle - **Orange plus**: pythia - **Black triangle**: mamba - **Brown star**: stablelm - **Cyan triangle**: btlm-3b-8k-base - **Green triangle**: falcon-7b - **Yellow diamond**: Llama-2-7b --- ### Detailed Analysis 1. **Model Performance**: - **Llama-2-7b** (yellow diamond): Highest accuracy (~0.70) and highest FLOPs (~10²⁴). - **Falcon-7b** (green triangle): Second-highest accuracy (~0.70) with slightly lower FLOPs (~10²³.⁵). - **Pythia** (orange plus): Lowest accuracy (~0.58) and lowest FLOPs (~10²¹). - **Mamba** (black triangle): Moderate accuracy (~0.62) with low FLOPs (~10²¹.⁵). - **Finch** (red circle): Accuracy ~0.62 at ~10²² FLOPs. - **Eagle** (blue cross): Accuracy ~0.62 at ~10²².⁵ FLOPs. - **StableLM** (brown star): Accuracy ~0.65 at ~10²³ FLOPs. - **BTLM-3b-8k-base** (cyan triangle): Accuracy ~0.67 at ~10²³ FLOPs. 2. **Trends**: - **Positive Correlation**: Higher FLOPs generally correlate with higher accuracy (e.g., Llama-2-7b and Falcon-7b). - **Efficiency Outliers**: - **Finch** achieves similar accuracy to **Eagle** with fewer FLOPs. - **Mamba** has lower accuracy than **Finch** despite comparable FLOPs. - **Divergence**: Some models (e.g., **StableLM**, **BTLM-3b-8k-base**) achieve higher accuracy than peers at similar FLOPs, suggesting architectural efficiency. --- ### Key Observations - **Llama-2-7b** and **Falcon-7b** dominate the high-accuracy, high-FLOPs quadrant. - **Pythia** underperforms across both axes, indicating potential limitations in training or architecture. - **StableLM** and **BTLM-3b-8k-base** show strong efficiency, outperforming peers at similar FLOPs. - **Eagle** and **Finch** cluster near the lower-mid range, with **Finch** being more efficient. --- ### Interpretation The data suggests that **training compute (FLOPs)** is a significant factor in model accuracy, but **architectural efficiency** plays a critical role. Models like **Llama-2-7b** and **Falcon-7b** demonstrate that scaling compute can yield high performance, while **StableLM** and **BTLM-3b-8k-base** highlight the importance of optimized architectures. **Pythia**’s poor performance raises questions about its training methodology or design. The divergence between **Finch** and **Eagle** underscores trade-offs between compute and efficiency. This analysis implies that future model development should balance both computational resources and architectural innovation to maximize performance. </details> Table 3: Multilingual Benchmarks, including LAMBADA Multilingual ( lmb.m ) (Gao et al., 2023), XCOPA (Ponti et al., 2020), XNLI (Conneau et al., 2018), PAWS-X (Yang et al., 2019), XStoryCloze ( xsClz ) (Lin et al., 2022), xWinogrande ( xwin ) (Tikhonov & Ryabinin, 2021). | Model | lmb.m ppl ↓ | lmb.m acc ↑ | pawsx acc ↑ | xcopa acc ↑ | xnli acc ↑ | xsClz acc ↑ | xwin acc ↑ | avg acc ↑ | |-----------------|---------------|---------------|---------------|---------------|--------------|---------------|--------------|-------------| | Pythia-1.4b | 115.9 | 35.5 | 50.9 | 52.7 | 38.9 | 51.8 | 68.3 | 49.7 | | Mamba-1.4b | 73.1 | 40.4 | 48 | 54.4 | 41.6 | 54.2 | 72.4 | 51.8 | | RWKV-4-1.5b | 72.5 | 38.5 | 53.7 | 55.4 | 39.3 | 56 | 67.7 | 51.8 | | Eagle-1.5b | 43.2 | 44.8 | 51.9 | 57.9 | 40.4 | 57.9 | 73 | 54.3 | | Finch-1.6b | 37.5 | 46.9 | 50.9 | 58 | 41.4 | 57.9 | 74.9 | 55 | | Pythia-2.8b | 81.3 | 38.8 | 49.4 | 53.7 | 40 | 53.5 | 71.5 | 51.1 | | Mamba-2.8b | 53.7 | 43.5 | 43.6 | 55.3 | 42.1 | 56.3 | 75.6 | 52.7 | | RWKV-4-3b | 48.1 | 43.4 | 50.9 | 57.5 | 40.9 | 58.1 | 72.3 | 53.9 | | Eagle-3b | 30.8 | 49.1 | 51.6 | 59 | 42.3 | 59.8 | 76.9 | 56.5 | | Finch-3b | 28.1 | 50.5 | 49.7 | 59.5 | 44.2 | 60.7 | 77.8 | 57.1 | | Pythia-6.9b | 85.6 | 36.7 | 48.4 | 54.1 | 40 | 54.2 | 70.9 | 50.7 | | MPT-7b | 49.8 | 44.4 | 43.5 | 53.6 | 39.8 | 56.3 | 76.9 | 52.4 | | Llama-2-7b | 30.4 | 50.8 | 41.2 | 56.7 | 39.9 | 57.5 | 79.5 | 54.3 | | Falcon-7b | 28.7 | 51.3 | 48.2 | 56 | 39 | 56 | 77.7 | 54.7 | | Mistral-7B-v0.1 | 27.1 | 51.9 | 41.5 | 55.9 | 43.1 | 59.2 | 81.2 | 55.5 | | RWKV-4-7b | 33.1 | 47.4 | 52.1 | 60.1 | 41.2 | 60.9 | 76.5 | 56.4 | | Eagle-7B | 21 | 53.7 | 45.6 | 62.2 | 44 | 63.3 | 80.4 | 58.2 | Table 4: English Focused Benchmarks, including LAMBADA (OpenAI) ( lmb.o ) (Paperno et al., 2016), Hellswag ( hella ) (Hampel, 1974), PIQA (Bisk et al., 2020), AI2 ARC ( arc ) (Bhakthavatsalam et al., 2021), GLUE (Wang et al., 2018), Winogrande ( winG ) (Sakaguchi et al., 2021), SciQ (Welbl et al., 2017), COPA (Roemmele et al., 2011). | Model | lmb.o ppl ↓ | lmb.o acc ↑ | hella acc_n ↑ | piqa acc ↑ | arc acc ↑ | glue acc ↑ | winG acc ↑ | sciq acc ↑ | copa acc ↑ | avg acc ↑ | |-----------------|---------------|---------------|-----------------|--------------|-------------|--------------|--------------|--------------|--------------|-------------| | Pythia-1.4b | 6.22 | 61 | 52 | 70.8 | 49.8 | 47.1 | 57.3 | 86.5 | 71 | 61.9 | | RWKV-4-1.5b | 6.15 | 60.1 | 51.6 | 71.5 | 48.1 | 46.1 | 55.2 | 84.7 | 78 | 61.9 | | Eagle-1.5b | 5.06 | 65.7 | 55 | 71.1 | 51.1 | 54.1 | 59.1 | 89.7 | 76 | 65.2 | | Finch-1.6b | 4.67 | 66.8 | 57.3 | 72.6 | 51.8 | 49.8 | 59.4 | 89.6 | 78 | 65.7 | | Mamba-1.4b | 5.05 | 64.5 | 59 | 74.2 | 53.5 | 47 | 61.3 | 87.1 | 80 | 65.8 | | Pythia-2.8b | 5.21 | 63.8 | 59.1 | 73.9 | 52.3 | 47.3 | 58.2 | 88.6 | 79 | 65.3 | | RWKV-4-3b | 4.71 | 65.7 | 58.8 | 72.4 | 52.8 | 53.6 | 57.5 | 87.6 | 86 | 66.8 | | Eagle-3b | 4.15 | 68.7 | 62.6 | 74.3 | 57.1 | 46.3 | 62 | 92.6 | 85 | 68.6 | | Mamba-2.8b | 4.21 | 68.1 | 65.9 | 75.2 | 57.9 | 46.3 | 63 | 90.2 | 84 | 68.8 | | Finch-3b | 3.92 | 70.8 | 64.8 | 74.2 | 55.9 | 58.2 | 63.6 | 92.5 | 82 | 70.3 | | Pythia-6.9b | 5.83 | 60.9 | 63.2 | 74.8 | 55.1 | 47.7 | 61.5 | 88.9 | 79 | 66.4 | | RWKV-4-7b | 3.94 | 69.8 | 65.3 | 75 | 56.4 | 56.4 | 62.4 | 90.8 | 85 | 70.1 | | Llama-2-7b | 3.4 | 73.5 | 76 | 78.1 | 65.4 | 42.9 | 69.1 | 93.9 | 87 | 73.2 | | MPT-7b | 3.88 | 68.7 | 76.3 | 79.3 | 63.3 | 48.7 | 68.1 | 93.9 | 88 | 73.3 | | Falcon-7b | 3.37 | 74.6 | 76.4 | 79.5 | 63.4 | 45.8 | 67.1 | 94.4 | 88 | 73.6 | | Eagle-7B | 3.37 | 74.2 | 70.9 | 77 | 62.5 | 57.5 | 67.4 | 95.5 | 88 | 74.1 | | Mistral-7B-v0.1 | 3.18 | 75.5 | 81 | 80.5 | 70.7 | 51.5 | 73.6 | 95.9 | 93 | 77.7 | ## 8.2 Associative Recall Associative recall (AR) is a synthetic task designed to mimic the way that humans associate and retrieve information. It measures a model's proficiency in recalling information that was previously mentioned in context. Prior research suggests that a model's ability to perform AR is indicative of its effectiveness in in-context learning (Elhage et al., 2021; Olsson et al., 2022). As a result, AR has been adopted as a benchmark in developing new language model architectural designs. (Fu et al., 2023; Poli et al., 2023; Lutati et al., 2023). Arora et al. (2023) benchmarked a range of models for multi-query associative recall (MQAR) and identified a performance gap between various linear transformer architectures and the transformer with attention. In MQAR tasks, prior RWKV models demonstrated a correlation between model dimension and sequence length. To compare architectures, we trained models using RWKV-4, Eagle and Finch on MQAR, using identical criteria with various model dimensions and sequence lengths. Our findings reveal significant improvements in MQAR with Eagle and Finch. Notably, Finch achieves extremely high accuracy in MQAR in our tests, and outperforms all well-known non-transformer architectures previously used to train large language models. Our experiments reveal performance disparities between Mamba (Gu & Dao, 2023) and Finch, despite their shared architectural features such as matrix-valued state and data-dependent memory modification, suggesting different combinations of these elements result in superior performance. Figure 4: MQAR tasks. An increase in sequence length correlates with increased task difficulty. <details> <summary>Image 4 Details</summary>  ### Visual Description ## Line Graphs: Model Accuracy vs. Dimension Across Sequence Lengths ### Overview The image contains four line graphs arranged in a 2x2 grid, each representing model accuracy across varying model dimensions (64, 128, 256, 512) for four sequence lengths (64, 128, 256, 512). Models include Dove (RWKV-4), Eagle (RWKV-5), Finch (RWKV-6), Hyena, Mamba, and Based. Accuracy ranges from 0.0 to 1.0 on the y-axis. --- ### Components/Axes - **X-axis**: Model Dimension (64, 128, 256, 512) - **Y-axis**: Accuracy (0.0 to 1.0) - **Legends**: - Green: Dove (RWKV-4) - Blue: Eagle (RWKV-5) - Black: Finch (RWKV-6) - Yellow: Hyena - Red: Mamba - Maroon: Based --- ### Detailed Analysis #### Sequence Length: 64 - **Trend**: All models show flat lines, indicating minimal accuracy variation with model dimension. - **Values**: - Dove: ~0.95–1.0 - Eagle: ~0.95–1.0 - Finch: ~0.95–1.0 - Hyena: ~0.95–1.0 - Mamba: ~0.95–1.0 - Based: ~0.95–1.0 #### Sequence Length: 128 - **Trend**: Most models remain flat, but Dove (green) shows a slight upward slope. - **Values**: - Dove: ~0.95–1.0 - Eagle: ~0.95–1.0 - Finch: ~0.95–1.0 - Hyena: ~0.90–0.95 - Mamba: ~0.95–1.0 - Based: ~0.95–1.0 #### Sequence Length: 256 - **Trend**: Significant divergence: - Dove (green) rises sharply from ~0.25 to ~0.95. - Eagle (blue) drops from ~0.95 to ~0.5. - Finch (black) remains flat at ~0.95. - Hyena (yellow) rises from ~0.25 to ~0.8. - Mamba (red) stays flat at ~0.95. - Based (maroon) remains flat at ~0.95. - **Values**: - Dove: ~0.25 (64) → ~0.95 (512) - Eagle: ~0.95 (64) → ~0.5 (512) - Finch: ~0.95 (all) - Hyena: ~0.25 (64) → ~0.8 (512) - Mamba: ~0.95 (all) - Based: ~0.95 (all) #### Sequence Length: 512 - **Trend**: Mixed performance: - Dove (green) peaks at ~0.95 but drops to ~0.25 at 512. - Eagle (blue) rises from ~0.0 to ~0.95. - Finch (black) stays flat at ~0.95. - Hyena (yellow) rises from ~0.0 to ~0.3. - Mamba (red) peaks at ~0.95 but drops to ~0.8. - Based (maroon) remains flat at ~0.95. - **Values**: - Dove: ~0.95 (128) → ~0.25 (512) - Eagle: ~0.0 (64) → ~0.95 (512) - Finch: ~0.95 (all) - Hyena: ~0.0 (64) → ~0.3 (512) - Mamba: ~0.95 (256) → ~0.8 (512) - Based: ~0.95 (all) --- ### Key Observations 1. **Model Robustness**: - Finch and Based maintain high accuracy across all dimensions and sequence lengths. - Mamba shows stability until sequence length 512, where accuracy drops. 2. **Dimension Sensitivity**: - Dove performs poorly at low dimensions (64) but excels at 256, then degrades at 512. - Eagle improves dramatically at 512 but underperforms at smaller dimensions. - Hyena shows gradual improvement with dimension but lags behind others. 3. **Sequence Length Impact**: - Longer sequences (256, 512) reveal greater model variability. - Models like Eagle and Dove exhibit non-linear scaling behavior. --- ### Interpretation - **Optimal Dimensions**: - Finch and Based are dimension-agnostic, suggesting architectural robustness. - Dove and Hyena require mid-range dimensions (128–256) for peak performance. - Eagle benefits from larger dimensions (512) but struggles at smaller scales. - **Sequence Length Trade-offs**: - Longer sequences amplify model weaknesses (e.g., Eagle’s collapse at 128, Dove’s drop at 512). - Mamba’s decline at 512 suggests architectural limits for very long sequences. - **Design Implications**: - Models like Finch and Based are preferable for variable sequence lengths. - Dove and Hyena may require careful dimension tuning for specific tasks. - Eagle’s performance highlights the need for architectural adjustments to handle short sequences. This analysis underscores the importance of model-architecture alignment with sequence length and computational constraints. </details> ## 8.3 Long Context Experiments Wetest loss versus sequence position on the PG19 (Rae et al., 2019) test set of books from token 2048 onward across RWKV-4, Eagle, and Finch. We find that Eagle improves dramatically over RWKV-4 on this long sequence task, despite having been trained solely on sequence length 4096. Finch further improves on this test beyond Eagle, with loss continuing to drop further into the sequence. See Figure 5 for details. ## 8.4 BambooBenchmark The Bamboo benchmark (Dong et al., 2023) evaluates the overall long-context language modeling capability of LLMs from five aspects: question answering, hallucination detection, text sorting, language modeling, and code completion with a total of ten evaluation tasks. We test models on the 4k version of the benchmark, which contains all ten tasks with a maximum context window length of 4k. We choose not to present results on the code completion task since all tested models fail to generate correct code completions on this aspect. In Table 5, we present results of nine tasks with either accuracy or F1 score and their average scores. On both 1.5b and 3b scales, the latest Finch and Eagle models outperforms the vanilla Mamba by at least 7% average score, while staying comparable with the Mamba trained on Hermes data ( i.e. , 0.7% drop of the average score). Note that, despite being trained on only 1.1T tokens, Eagle-7b consistently outperforms Pythia by an average of 13.5% on the 7b scale, and it also surpasses LLaMA2-Chat-7b on several tasks in Bamboo. These results demonstrate better capacity of the proposed Finch and Eagle models on a vast range of long context tasks. Figure 5: Loss along sequence offset for 3B RWKV-4 World, Eagle and Finch on PG19 dataset. All models were pretrained with context length 4096. <details> <summary>Image 5 Details</summary>  ### Visual Description ## Line Graph: Loss vs. Position from Start ### Overview The image is a line graph comparing the loss values of three models (PG19 RWKV-4 3B, PG19 Eagle 3B, and PG19 Finch 3B) across positions from start, measured on a logarithmic scale (10² to 10⁴). The y-axis represents "Loss" (2.2–3.6), while the x-axis represents "Position from start" (logarithmic scale). The graph highlights distinct trends in loss behavior for each model. ### Components/Axes - **X-axis (Position from start)**: Logarithmic scale with markers at 10², 10³, and 10⁴. - **Y-axis (Loss)**: Linear scale ranging from 2.2 to 3.6. - **Legend**: Located in the top-left corner, associating colors with models: - **Blue**: PG19 RWKV-4 3B - **Green**: PG19 Eagle 3B - **Red**: PG19 Finch 3B ### Detailed Analysis 1. **PG19 RWKV-4 3B (Blue Line)**: - Starts at ~2.6 (10² position), dips to ~2.4 (10³ position), then **spikes sharply** to ~3.6 at 10⁴. - Shows significant volatility, with sharp increases and decreases, particularly after 10³. 2. **PG19 Eagle 3B (Green Line)**: - Remains relatively stable, fluctuating between ~2.4 and 2.5 across all positions. - No sharp spikes or drops; maintains a consistent trend. 3. **PG19 Finch 3B (Red Line)**: - Similar to the green line, with minor fluctuations between ~2.3 and 2.5. - Slightly more variability than the green line but no extreme spikes. ### Key Observations - The **blue line (PG19 RWKV-4 3B)** exhibits a **dramatic increase in loss** at the 10⁴ position, reaching ~3.6, far exceeding the other models. - The **green (Eagle)** and **red (Finch)** lines show **stable loss values** (~2.4–2.5) throughout, with no significant deviations. - The logarithmic x-axis emphasizes the scale of positions, with the spike in the blue line occurring at the far-right end of the graph. ### Interpretation The data suggests that the **PG19 RWKV-4 3B model experiences a critical failure or anomaly** at the 10⁴ position, as indicated by the sharp loss spike. This could imply: - **Poor generalization** at longer sequence lengths (10⁴ positions). - **Computational instability** or resource exhaustion in the RWKV-4 model under high positional demands. - The **Eagle and Finch models** demonstrate **robustness** in maintaining low loss across all positions, indicating better stability or optimization for the task. The logarithmic x-axis underscores the **exponential growth in positional complexity**, with the RWKV-4 model’s performance degrading disproportionately at higher scales. This highlights potential limitations in the RWKV-4 architecture for handling extended sequences compared to the Eagle and Finch models. </details> Table 5: Results on the long context reasoning benchmark: Bamboo. We compare both transformer and linear attention language models on three different scales: 1.5b, 3b, and 7b. | Model | meetingqa Acc. ↑ | paperqa Acc. ↑ | meetingpred Acc. ↑ | showspred Acc. ↑ | reportsumsort Acc. ↑ | showssort Acc. ↑ | senhallu F1 ↑ | abshallu F1 ↑ | altqa Acc. ↑ | Avg. ↑ | |---------------------|--------------------|------------------|----------------------|--------------------|------------------------|--------------------|-----------------|-----------------|----------------|----------| | Pythia-1.4b | 15.0% | 4.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 2.1% | | Mamba-1.4b | 15.0% | 2.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 2.0% | 0.0% | 2.1% | | Eagle-1.5b | 21.0% | 19.0% | 1.0% | 0.0% | 0.0% | 0.0% | 13.2% | 23.5% | 5.5% | 9.2% | | Finch-1.6b | 19.0% | 22.0% | 1.0% | 8.0% | 0.0% | 0.0% | 10.7% | 17.3% | 2.5% | 8.9% | | Pythia-2.8b | 16.0% | 4.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 2.2% | | Mamba-2.8b | 11.0% | 4.0% | 0.0% | 3.0% | 0.0% | 0.0% | 0.0% | 3.9% | 0.0% | 2.4% | | Mamba-2.8b-Hermes | 27.0% | 25.0% | 0.0% | 9.0% | 0.0% | 0.0% | 19.7% | 26.4% | 0.0 | 11.9% | | Eagle-3b | 16.0% | 14.0% | 0.0% | 4.0% | 0.0% | 0.0% | 25.0% | 29.2% | 1.0% | 9.9% | | Finch-3b | 20.0% | 26.0% | 4.0% | 7.0% | 0.0% | 0.0% | 14.4% | 23.6% | 6.5% | 11.3% | | Pythia-6.9b | 19.0% | 7.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 3.3% | | Eagle-7b-Hermes | 31.0% | 23.0% | 0.0% | 0.0% | 0.0% | 0.0% | 50.3% | 46.9% | 0.0% | 16.8% | | LLaMA2-Chat-7b | 6.0% | 17.0% | 4.0% | 12.0% | 0.0% | 0.0% | 64.7% | 63.4% | 46.0% | 24.1% | | Mistral-Instruct-7b | 65.0% | 73.0% | 17.0% | 32.0% | 0.0% | 0.0% | 80.5% | 72.8% | 13.5% | 39.3% | Figure 6: Memory Usage vs. Sequence Length (A100 80GB) <details> <summary>Image 6 Details</summary>  ### Visual Description ## Bar Chart: Memory Usage by Sequence Length and Algorithm ### Overview The chart compares memory usage (in GB) across three algorithms (Finch, Mamba, Flash Attention v2) at varying sequence lengths (512 to 16384). Memory usage increases with sequence length, with Flash Attention v2 consistently consuming the most memory. ### Components/Axes - **Y-axis**: Memory Usage (GB), scaled from 0 to 30 in increments of 5. - **X-axis**: Sequence Length, with categories: 512, 1024, 2048, 4096, 8192, 16384. - **Legend**: - Blue = Finch - Red = Mamba - Beige = Flash Attention v2 - **Bars**: Grouped by sequence length, with three bars per group (one per algorithm). ### Detailed Analysis - **512**: - Finch: 0.5 GB - Mamba: 0.8 GB - Flash Attention v2: 1.0 GB - **1024**: - Finch: 1.5 GB - Mamba: 1.7 GB - Flash Attention v2: 2.1 GB - **2048**: - Finch: 2.9 GB - Mamba: 3.5 GB - Flash Attention v2: 4.1 GB - **4096**: - Finch: 5.9 GB - Mamba: 6.9 GB - Flash Attention v2: 8.2 GB - **8192**: - Finch: 11.8 GB - Mamba: 13.9 GB - Flash Attention v2: 16.4 GB - **16384**: - Finch: 23.6 GB - Mamba: 27.7 GB - Flash Attention v2: 32.8 GB ### Key Observations 1. **Exponential Growth**: Memory usage increases non-linearly with sequence length for all algorithms. 2. **Flash Attention v2 Dominance**: Consistently uses 2–3× more memory than Finch and Mamba at all sequence lengths. 3. **Algorithm Efficiency**: Finch uses the least memory, while Mamba is intermediate. Differences widen at larger sequence lengths (e.g., 16384: Finch 23.6 GB vs. Flash Attention v2 32.8 GB). 4. **Scalability**: All algorithms scale similarly, but Flash Attention v2’s memory footprint grows fastest (e.g., 16384 sequence length uses 10× more memory than 512). ### Interpretation The data suggests that **Flash Attention v2** is the most memory-intensive algorithm, potentially due to its computational complexity or data structure overhead. **Finch** is the most memory-efficient, making it preferable for resource-constrained environments. The exponential memory growth across all algorithms highlights the importance of sequence length optimization in memory-bound applications. The widening gap at larger sequence lengths (e.g., 16384) indicates that Flash Attention v2 may not scale as efficiently as Finch or Mamba for very long sequences. </details> Figure 7: Time vs. Sequence Length (A100 80GB) <details> <summary>Image 7 Details</summary>  ### Visual Description ## Line Graph: Time vs. Sequence Length for Different Algorithms ### Overview The graph compares the computational time (in milliseconds) required by four algorithms—Finch, Flash Attention v2, Mamba, and Mamba 2x—as sequence length increases from 2k to 16k. Time is plotted on the y-axis (0–800 ms), and sequence length is on the x-axis (2k–16k). The legend at the bottom maps colors to algorithms: blue (Finch), orange (Flash Attention v2), solid red (Mamba), and dotted red (Mamba 2x). ### Components/Axes - **X-axis (Sequence Length)**: Labeled "Sequence Length" with markers at 2k, 4k, 6k, 8k, 10k, 12k, 14k, and 16k. - **Y-axis (Time)**: Labeled "Time (ms)" with increments of 200 ms (0, 200, 400, 600, 800). - **Legend**: Positioned at the bottom, with four entries: - Blue circle: Finch - Orange line: Flash Attention v2 - Solid red line: Mamba - Dotted red line: Mamba 2x ### Detailed Analysis 1. **Flash Attention v2 (Orange Line)**: - Steep upward slope, especially after 8k sequence length. - At 16k, time reaches ~800 ms. - Dominates in time consumption for longer sequences. 2. **Mamba 2x (Dotted Red Line)**: - Gradual upward slope, less steep than Flash Attention v2. - At 16k, time is ~250 ms. - Outperforms Flash Attention v2 in efficiency. 3. **Finch (Blue Line)**: - Linear increase, moderate slope. - At 16k, time is ~200 ms. - Outperforms Mamba 2x but lags behind Mamba. 4. **Mamba (Solid Red Line)**: - Slowest linear increase. - At 16k, time is ~150 ms. - Most efficient algorithm across all sequence lengths. ### Key Observations - **Flash Attention v2** exhibits exponential time growth, making it impractical for sequences >10k. - **Mamba** maintains near-linear efficiency, ideal for large sequences. - **Mamba 2x** balances performance and scalability, outperforming Finch and Flash Attention v2 at longer lengths. - All algorithms show minimal time difference below 4k sequence length. ### Interpretation The data suggests that **Mamba** is the most scalable algorithm for long sequences, while **Flash Attention v2** becomes prohibitively slow as sequence length increases. **Mamba 2x** and **Finch** offer intermediate efficiency, with Mamba 2x showing a 25% improvement over Finch at 16k. The steep slope of Flash Attention v2 highlights its limitations in handling large-scale data, whereas Mamba’s linear scaling makes it robust for high-dimensional tasks. This analysis is critical for selecting algorithms in applications like NLP or genomics, where sequence length directly impacts computational feasibility. </details> ## 9 Speed and Memory Benchmarks Wecompare the speed and memory utilization of the Attention-like kernels for Finch, Mamba 2 , and Flash Attention 3 (Dao, 2023) in Figures 6 and 7. For all benchmarks, we use a batch size of 8, a model dimension of 4096, and a head size of 64 for both Flash Attention and Finch. For Mamba, we employ a state dimension of 16, a model dimension of 8192, to mimic Mamba's usage of an expansion factor of 2. Our findings indicate that Finch's speed in training scales linearly with respect to sequence length, exhibiting similar scaling to Mamba. We find Finch 2 We also plot Mamba 2x which uses 2 runs through the Mamba kernel instead of one. This is done to mimic the usage of twice the number of layers in Mamba vs Finch and Transformers 3 We use the PyTorch Implementation of Flash Attention v2 is significantly faster than Flash Attention for sequence lengths beyond 4k, being around 4.2x faster for a sequence length of 16k. Furthermore, Finch consistently outperforms Mamba and Flash Attention in terms of memory usage, using 40% and 17% less memory usage than Flash Attention and Mamba respectively. Further optimization of our Finch CUDA implementation, including algorithmic improvements, are possible, and could lead to speed increases and greater parallelization. However, this optimization is left for future work. ## 10 Multimodal Experiments In this section, we explore the capabilities of Eagle when extended to handle multimodal tasks, where the model processes and integrates textual inputs with inputs in a different domain. ## 10.1 RWKVMusicModelling To investigate the Eagle architecture's applicability to music modelling, we use the Irishman ABC music sheet dataset (Wu et al., 2023) to train a new RWKV-5-Music model using the same hyperparameters as the existing RWKV-4-Music model. The loss of RWKV-5 is approximately 2%lower than the previous generation model, and it is shown mainly in the musical score part, indicating that RWKV-5 has stronger modelling and generalization capabilities than its predecessor. The model has L = 24 layers in total, with a dimension of D = 512 and a byte-level tokenizer of V = 128 tokens. The training context length is 1024 bytes. We use all 2162 pieces of music in the validation set and calculate the loss for each position from start. Loss is averaged across all pieces of music then Gaussian smoothed over the position in sequence. The figure 8 shows the loss as a function of position. Note that the first 30-100 bytes of the ABC format are the file header and control codes, followed by the musical scores. The loss of RWKV-5 is approximately 2% lower than the previous generation model, and it is shown mainly in the musical score part, indicating that RWKV-5 has stronger modelling and generalization capabilities than its precedent model. Figure 8: Music modelling loss over sequence position. <details> <summary>Image 8 Details</summary>  ### Visual Description ## Line Graph: Loss Comparison Between RWKV-4-Music and RWKV-5-Music ### Overview The image is a line graph comparing the loss (in Bits/Byte) of two models, RWKV-4-Music (orange) and RWKV-5-Music (red), across positions from 0 to 768. The y-axis ranges from 0.4 to 0.8, while the x-axis represents positional increments. ### Components/Axes - **X-axis**: "Position from start" with markers at 0, 128, 256, 384, 512, 640, and 768. - **Y-axis**: "Loss (Bits/Byte)" with increments of 0.1. - **Legend**: Located in the top-right corner, with orange representing RWKV-4-Music and red representing RWKV-5-Music. ### Detailed Analysis 1. **RWKV-4-Music (Orange Line)**: - Starts with a sharp peak at position 0 (~0.8 Bits/Byte). - Declines sharply to ~0.45 by position 128. - Fluctuates between ~0.45 and ~0.55 from positions 256 to 768, ending at ~0.53. - Notable dips occur near positions 256 and 512. 2. **RWKV-5-Music (Red Line)**: - Peaks at position 128 (~0.8 Bits/Byte), slightly lower than RWKV-4's initial peak. - Declines to ~0.4 by position 256. - Exhibits more pronounced fluctuations (~0.42–0.52) from positions 384 to 768, ending at ~0.48. - Sharp dips observed near positions 384 and 640. ### Key Observations - Both models exhibit a high initial loss, followed by a decline. - RWKV-4-Music maintains a consistently higher loss than RWKV-5-Music after position 256. - RWKV-5-Music shows greater variability in loss values toward the end of the sequence. - The orange line (RWKV-4) has a steeper initial drop compared to the red line (RWKV-5). ### Interpretation The data suggests that RWKV-5-Music achieves lower loss values overall, particularly in later positions, indicating potential improvements in efficiency or performance. The initial peak for both models may reflect higher computational costs or errors at the start of processing. The fluctuations in RWKV-5-Music could imply adaptive adjustments during processing, while RWKV-4's stability might indicate a more consistent but less optimized approach. The divergence in performance highlights trade-offs between model versions, with RWKV-5 potentially offering better scalability or resource management. </details> ## 10.2 VisualRWKV VisualRWKV is the visual-enhanced version of the RWKV language model, enabling RWKV to handle various visual tasks. Our VisualRWKV follows a similar architecture to popular visionlanguage models (Liu et al., 2023a). We present the architecture in Figure 9. It consists of a vision encoder and a language model. Specifically, we use CLIP (Radford et al., 2021) as the vision encoder and Eagle 1.5B and 3B as the language model. We use LLaVA-1.5 dataset (Liu et al., 2023a). To adapt Eagle to this multimodal task, we employ a two-stage instruction-tuning process to enhance model performance. Initially, we conduct pre-training for feature alignment, during which only the projection layer is subjected to updates, while the rest of the model is kept in a frozen state. Following this, we move on to the fine-tuning end-to-end stage, where both the projection layer and the RWKV language model are fine-tuned, and the vision encoder continue to be kept frozen. As shown in Table 6, we demonstrate that VisualRWKV's architecture is <details> <summary>Image 9 Details</summary>  ### Visual Description ## Diagram: Vision-Language Model Architecture for Image-Text Question Answering ### Overview The diagram illustrates a vision-language model (VLM) architecture designed to answer questions about input images. It shows the flow of data from raw image input through a vision encoder, projection layer, and language model components to generate textual answers. The system distinguishes between frozen (pre-trained) and trainable parameters. ### Components/Axes 1. **Input Image**: Bottom-left corner, showing a blue jay bird on a snowy surface. 2. **Input Text**: Middle-bottom, containing the question "<image> what is the name of this bird?" with "<image>" in red and the question in green. 3. **Vision Encoder**: Blue box labeled "Vision Encoder" with frozen parameters (blue legend). 4. **Projection**: Orange box labeled "Projection" connecting Vision Encoder to Embeddings. 5. **LM Head**: Orange box labeled "LM Head" at the top of the language model stack. 6. **RWKV Blocks**: Large orange box labeled "RWKV Blocks" forming the core of the language model. 7. **Embeddings**: Orange box labeled "Embeddings" between Projection and RWKV Blocks. 8. **LLM**: Green box labeled "LLM" containing the RWKV Blocks and LM Head. 9. **Output**: Top-right corner showing the answer "bluejay" in green text. 10. **Legend**: Bottom-right corner with: - Blue square: "Frozen Parameters" - Orange square: "Trainable Parameters" ### Detailed Analysis - **Data Flow**: 1. Input image → Vision Encoder (frozen parameters) 2. Vision Encoder output → Projection → Embeddings 3. Embeddings → RWKV Blocks (trainable parameters) 4. RWKV Blocks → LM Head → LLM 5. LLM generates final answer "bluejay" - **Parameter Trainability**: - Vision Encoder: All parameters frozen (blue) - Projection, RWKV Blocks, LM Head: All parameters trainable (orange) ### Key Observations 1. The system uses a modular architecture with clear separation between vision and language components. 2. The vision encoder appears to be a pre-trained model (frozen parameters), while the language model components are trainable. 3. The input text format explicitly separates image reference (<image>) from the question text. 4. The output answer is generated directly from the LLM without additional processing steps. ### Interpretation This architecture demonstrates a typical vision-language model design where: - Pre-trained vision components (frozen parameters) extract image features - Trainable language components (RWKV Blocks, LM Head) process both image embeddings and text - The system is optimized for image-text question answering tasks - The blue jay example shows successful cross-modal understanding, correctly identifying the bird species from visual and textual context The separation of frozen and trainable parameters suggests a transfer learning approach, leveraging existing vision models while fine-tuning language components for specific tasks. </details> Figure 9: VisualRWKV architecture overview. | Method | Vision Encoder | LLM | GQA( ↑ ) | ScienceQA-IMG ( ↑ ) | Text-VQA ( ↑ ) | POPE ( ↑ ) | |--------------------------------|------------------|--------------|------------|-----------------------|------------------|--------------| | BLIP-2 (Li et al., 2023a) | EVA01-CLIP-G | Vicuna-13B | 41 | 61.0 | 42.5 | 85.3 | | BLIP-2 (Li et al., 2023a) | EVA01-CLIP-G | Flan-T5-11B | 44.6 | 64.5 | - | - | | InstructBLIP(Dai et al., 2023) | EVA01-CLIP-G | Vicuna-7B | 49.2 | 60.5 | 50.1 | - | | InstructBLIP(Dai et al., 2023) | EVA01-CLIP-G | Vicuna-13B | 49.5 | 63.1 | 50.7 | 78.9 | | IDEFICS-9B (IDEFICS, 2023) | OpenCLIP-H | LLaMA-7B | 38.4 | - | 25.9 | - | | IDEFICS-80B (IDEFICS, 2023) | OpenCLIP-H | LLaMA-65B | 45.2 | - | 30.9 | - | | TinyGPT-V (Yuan et al., 2023) | EVA01-CLIP-G | Phi-2 (2.7B) | 33.6 | - | - | - | | VisualRWKV | CLIP-L | Eagle-1.5B | 48.5 | 46.2 | 37.8 | 81.8 | | VisualRWKV | CLIP-L | Eagle-3B | 49.7 | 58.3 | 46.4 | 81.4 | Table 6: A comparison of VisualRWKV to other state-of-the-art Multimodal Large Language Models (MLLMs) across 4 distinct benchmarks. We evaluate these models on benchmarks: GQA(Hudson & Manning, 2019), ScienceQA-IMG(Lu et al., 2022), Text-VQA(Singh et al., 2019) and POPE(Li et al., 2023c). For POPE, the average F1-score across three distinct categories-random, popular, and adversarial-was computed using the validation set of the MSCOCO dataset. powerful for visual understanding and reasoning. With a smaller vision encoder CLIP-L (0.4B) and modest-sized LLMs of 1.5B and 3B, it achieves results comparable to the combination of CLIP-G (1.0B) and CLIP-H (1.0B) with larger LLMs of 7B and 13B. Moreover, in some benchmarks, it even outperforms larger models. ## 11 Conclusions In this work, we introduced Eagle (RWKV-5) and Finch (RWKV-6), marking substantial progress in RNN-based language models by integrating multiheaded matrix-valued states and dynamic data-driven recurrence mechanisms. These models demonstrate exceptional performance on MQARanddiverse linguistic benchmarks, challenging the dominance of traditional Transformer architectures while retaining key RNN advantages. With models publicly available under the Apache 2.0 license and trained on an extensive multilingual corpus, our work not only advances the capabilities of language models but also emphasizes community accessibility and applicability across various domains. While acknowledging the computational and ethical challenges ahead, we hope that Eagle and Finch's efficient new architecture and wide availability will help push the boundaries of language modeling and pave the way for future innovations. Limitations The Eagle and Finch models fall short on certain aspects that can be mitigated and addressed in future work. Weexperimented with using Eagle as an embedding model on the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al., 2023) but were not able to get strong embedding performance. Webelieve that its state is a very high-quality embedding of the context but an appropriate method is required to aggregate the information content. We leave this to future work. Because our training corpus contains some synthetic data from GPT-3.5 and ChatGPT, our released models exhibit behaviors similar to ChatGPT and will mimic ChatGPT's conversation style and tone. For instance, the model might occasionally claim that it is trained by OpenAI. However, this is not a general property the RWKV architecture but rather a specific outcome of the data and training process. Future Work Our 1.12 trillion token multilingual training corpus is much smaller than the training data sizes for contemporary models such as LLaMA2 (Touvron et al., 2023), and expanding our training corpus to be more diverse and expansive is a key priority to improving model performance (Albalak et al., 2024). We also plan to train and release larger versions of Finch such as 7B and 14B parameters, and further extend its performance with reduced inference and training costs via Mixture of Experts (Shazeer et al., 2017). ## Acknowledgments Wethank Stability AI for the compute used to train our models and for technical support in the development of RWKV. We also thank the members of the RWKV and EleutherAI Discord servers for their help and work on further extending the applicability of RWKV to different domains. We also thank Shenzhen Yuanshi Intelligence Co., Ltd. for its contribution to the promotion and commercialization of RWKV. We thank Songlin Yang for assistance with the code and ideas for our time-parallel implementations. ## References - Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023. - Alon Albalak, Akshat Shrivastava, Chinnadhurai Sankar, Adithya Sagar, and Mike Ross. Dataefficiency with a single gpu: An exploration of transfer methods for small language models. arXiv preprint arXiv:2210.03871 , 2022. - Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. arXiv preprint arXiv:2312.02406 , 2023. - Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint arXiv:2402.16827 , 2024. - Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Re. Zoology: Measuring and improving recall in efficient language models, 2023. - Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020. - Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer ai2 reasoning challenge. arXiv preprint arXiv:2102.03315 , 2021. - Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pp. 7432-7439, 2020. - Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPTNeoX-20B: An open-source autoregressive language model. In Angela Fan, Suzana Ilic, Thomas Wolf, and Matthias Gallé (eds.), Proceedings of BigScience Episode #5 - Workshop on Challenges & Perspectives in Creating Large Language Models , pp. 95-136, virtual+Dublin, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.bigscience-1.9. URL https://aclanthology.org/2022.bigscience-1.9 . - Guy E. Blelloch. Prefix sums and their applications. Technical Report CMU-CS-90-190, School of Computer Science, Carnegie Mellon University, November 1990. - Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019a. - Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse. arXiv:1904.10509 , 2019b. - Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014. - Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations , 2020. - Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, and Terry Lyons. Theoretical foundations of deep selective state-space models. arXiv preprint arXiv:2402.19047 , 2024. - Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pp. 2475-2485, 2018. - Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv , abs/2305.06500, 2023. URL https: //api.semanticscholar.org/CorpusID:258615266 . - Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations , 2023. - Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. - Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427 , 2024. - Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. arXiv preprint arXiv:2309.13345 , 2023. - Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html. - Teddy Ferdinan, Jan Koco´ n, and Przemysław Kazienko. Into the unknown: Self-learning large language models, 2024. - Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations , 2022. - Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models, 2023. - Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. - Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo.org/records/10256836 . - Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023. - Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems , 33: 1474-1487, 2020. - Albert Gu, Karan Goel, , and Christopher R ´ e. Efficiently modeling long sequences with structured state spaces. arXiv:2111.00396 , 2021. - Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces, 2022. - Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. LongT5: Efficient text-to-text transformer for long sequences. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022 , pp. 724-736, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-naacl.55. URL https://aclanthology.org/2022.findings-naacl.55 . - Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems , 35:22982-22994, 2022. - Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association , 69(346):383-393, 1974. - Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997. - Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum? id=nZeVKeeFYf9 . - Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 6693-6702, 2019. URL https://api.semanticscholar. org/CorpusID:152282269 . - IDEFICS. Introducing idefics: An open reproduction of state-of-the-art visual language model. https://huggingface.co/blog/idefics , 2023. - Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lelio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. Mistral 7b, 2023. - Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning , pp. 5156-5165. PMLR, 2020a. - Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran¸cois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. Proceedings of the 37 th International Conference on Machine Learning , 2020b. - Tobias Katsch. Gateloop: Fully data-controlled linear recurrence for sequence modeling, 2023. - Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. Critiquellm: Scaling llm-as-critic for effective and explainable evaluation of large language model generation, 2023. - Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations , 2019. - Jan Koco´ n et al. Chatgpt: Jack of all trades, master of none. Information Fusion , 99:101861, November 2023. ISSN 1566-2535. doi: 10.1016/j.inffus.2023.101861. URL http://dx.doi. org/10.1016/j.inffus.2023.101861 . - Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pp. 19730-19742. PMLR, 2023a. - Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023b. - Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen. Evaluating object hallucination in large vision-language models. In Conference on Empirical Methods in Natural Language Processing , 2023c. URL https://api.semanticscholar.org/CorpusID: 258740697 . - Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F . T. Martins, and Hinrich Schütze. Mala-500: Massive language adaptation of large language models. arxiv , 2024. - Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp. 9019-9052, 2022. - Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 , 2023a. - Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. Alignbench: Benchmarking chinese alignment of large language models, 2023b. - Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. ArXiv , abs/2209.09513, 2022. URL https://api.semanticscholar. org/CorpusID:252383606 . - Shahar Lutati, Itamar Zimerman, and Lior Wolf. Focus your attention (with adaptive iir filters), 2023. - Kaokao Lv, Liang Lv, Chang Wang, Wenxin Zhang, Xuhui Ren, and Haihao Shen. Intel-neural-chat7b-v1-1, 2023. URL https://huggingface.co/Intel/neural-chat-7b-v1-1 . - Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In The Eleventh International Conference on Learning Representations , 2022. - Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In International Conference on Learning Representations , 2018. - Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pp. 2014-2037, 2023. - Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. - Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023. - Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031 , 2016. - Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Wo´ zniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023 , pp. 14048-14077, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.936. URL https://aclanthology.org/2023. findings-emnlp.936 . - Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning , pp. 28043-28078. PMLR, 2023. - Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vuli´ c, and Anna Korhonen. XCOPA: A multilingual dataset for causal commonsense reasoning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 2362-2376, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.185. URL https:// aclanthology.org/2020.emnlp-main.185 . - Zhen Qin, XiaoDong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer, 2022. - Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=P1TCHxJwLB . - ZhenQin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Xiao Luo, Yu Qiao, and Yiran Zhong. Transnormerllm: A faster and better large language model with improved transnormer, 2024. - Jiezhong Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self-attention for long document understanding. arXiv preprint arXiv:1911.02972 , 2019. - Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAI Blog , 2018. - Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. - Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pp. 8748-8763. PMLR, 2021. - Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint , 2019. URL https://arxiv.org/abs/1911.05507 . - Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation, 2020. - Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series , 2011. - Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021. - Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207 , 2021. - Akira Sasaki, Masato Hirakawa, Shintaro Horie, and Tomoaki Nakamura. Elyzajapanese-llama-2-7b-fast, 2023. URL https://huggingface.co/elyza/ ELYZA-japanese-Llama-2-7b-fast . - Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation , 4(1):131-139, 1992. doi: 10.1162/neco.1992.4.1.131. - NoamShazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 , 2017. - Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019. - Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling, 2023. - Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://www.cerebras.net/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama , June 2023. URL https://huggingface.co/datasets/cerebras/ SlimPajama-627B . - Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7) . Leibniz-Institut für Deutsche Sprache, 2019. - Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer, 2022. - Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, 2023. - Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference on Machine Learning , pp. 9438-9447. PMLR, 2020. - Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys , 55(6):1-28, 2022. - Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL https://huggingface.co/datasets/teknium/OpenHermes-2.5 . - Matteo Tiezzi, Michele Casoni, Alessandro Betti, Tommaso Guidi, Marco Gori, and Stefano Melacci. Onthe resurgence of recurrent models for long sequences: Survey and research opportunities in the transformer era. arXiv preprint arXiv:2402.08132 , 2024. - Alexey Tikhonov and Max Ryabinin. It's all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pp. 3534-3546, 2021. - Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P .S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems , volume 34, pp. 24261-24272. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper\_files/paper/2021/ file/cba0a4ee5ccd02fda0fe3f9a3e7b89fe-Paper.pdf . - Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. - Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. - Aaron Voelker, Ivana Kaji´ c, and Chris Eliasmith. Legendre memory units: Continuous-time representation in recurrent neural networks. Advances in neural information processing systems , 32, 2019. - Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863 , 1, 2023. - Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pp. 353-355, 2018. - Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020. - Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text , pp. 94-106, 2017. - BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Ta¸ sar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdenˇ ek Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. Bloom: A 176b-parameter open-access multilingual language model, 2023. - Shangda Wu, Xiaobing Li, Feng Yu, and Maosong Sun. Tunesformer: Forming irish tunes with control codes by bar patching. In Lorenzo Porcaro, Roser Batlle-Roca, and Emilia Gómez (eds.), Proceedings of the 2nd Workshop on Human-Centric Music Information Retrieval 2023 co-located with the 24th International Society for Music Information Retrieval Conference (ISMIR 2023), Milan, Italy, November 10, 2023 , volume 3528 of CEUR Workshop Proceedings . CEUR-WS.org, 2023. URL https://ceur-ws.org/Vol-3528/paper1.pdf . - Yuxin Wu and Kaiming He. Group normalization. arXiv:1803.08494 , 2018. - Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 , 2023. - Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems , 36, 2024. - Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention, 2021. - Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023. - Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training, 2023. - Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. Paws-x: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 3687-3692, 2019. - Zhengqing Yuan, Zhaoxu Li, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. ArXiv , abs/2312.16862, 2023. URL https://api.semanticscholar. org/CorpusID:266572996 . - Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems , 33, 2020. - Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. An attention free transformer, 2021. - Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024. - Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024. ## A Author Contributions BoPeng Original RWKV-5 and RWKV-6 ideas, original code, performance optimizations, original experiments, tokenizer design, dataset composition, and trained models from 0.4B to 7B. Daniel Goldstein RWKV-5 and RWKV-6 time-parallelization research and code. Manuscript organization, initial draft sections 2, 3, 4, 5, 6, 8.1, 8.3, and appendices B, D, I, and J. Proofreading and revisions of full manuscript. Experiments for 8.1 and 8.3. Additional work on tables 1, 2, figure 9, and appendix G. Quentin Anthony Led manuscript and results organization. Revisions and proofreading of manuscript. Alon Albalak Manuscript organization, initial draft of section 1, proofreading, formatting, and revisions of full manuscript. Eric Alcaide Figure 1. Proofreading, formatting, and revisions of full manuscript. Stella Biderman Oversight and planning on scaling figures and FLOP results. Manuscript assistance. Eugene Cheah Experiments for section 8.1. Teddy Ferdinan Self-Learning Capability (SLC) evaluation (Sec. F .3) - implementation of the method, performing experiments, initial draft of the section, description of the results (Tab. 12). Przemysław Kazienko Planning the experiment with Self-Learning Capability (SLC) evaluation (Sec. F .3), supervising SLC experiments. Jan Koco´ n Co-author of the idea of Self-Learning Large Language Models (Ferdinan et al., 2024) - supervising evaluation of RWKV Self-Learning Capability (Sec. F .3), supervising experiments with zero-shot evaluation on additional NLP tasks (Sec. F .4), proofreading of full manuscript. Kranthi Kiran GV Manuscript (sections 8.1 and 10; revision and proofreading). Tables 3 and 4. Haowen Hou VisualRWKV based on RWKV-5, which encompasses original code, original experiments for Table 6, and trained models ranging from 1.5 billion to 3 billion parameters. Figure 9 and draft section 10.2. Proofreading and formatting fixes. Satyapriya Krishna Primarily contributed to the evaluations of the models (Section 8 and F .1), and also made edits/improvements throughout the document. Ronald McClelland Jr. Tables 1 and 2. Dataset research. Proofreading and formatting fixes. Niklas Muennighoff Investigated using the RWKV models for embedding. Fares Obeid RWKV-5 and RWKV-6 time-parallelization research. Section 9. Experiments for figures 6 and 7. Proofreading full manuscript. Atsushi Saito Section 1, 5, 8.1 and 8.2. Experiments for 8.2. Proofreading and adding citations. Guangyu Song Section 8.2. Initial draft sections 1, 11. Experiments for 8.2. Contributions to table 1. Proofreading and citations. Haoqin Tu Section 8.4, experiments for Table 5. Proofreading full manuscript. Stanisław Wo´ zniak Experiments with zero-shot evaluation on additional NLP tasks (Sec. F .4). Bartłomiej Koptyra Zero-shot evaluation comparison of Raven and Eagle 7B on additional NLP tasks (Sec. F .4). Aleksander Szcz˛ esny Conducted experiments on given datasets tasks: TextEntail, GoEmo, PolEmo, WNLI (Sec. F .4). Ruichong Zhang Initial paper structure organization, draft sections 3, 4, 5 and appendices E, G and H. Experiments for music of section 10.1 and alignment of section F .1. Figure 8 and 10. Additional work on section 11 and appendix B. Bingchen Zhao Section F .2, experiments for Figure 11. Proofreading full manuscript. Qihang Zhao Section 2, Tables 1. Proofreading and revisions. Peng Zhou Section 2, Tables 1, appendices C,I. Proofreading and revisions. Jian Zhu Initial draft sections 2 and C. Captions of Table 4, 3 and 8. Fixing citations and formatting the whole manuscript. Proofreading and revisions. Rui-Jie Zhu Optimizing draft Section C, reorganizing Table 8, 12, and 11. Proofreading and revisions. ## B Additional Architecture Details The WKV computations of Eagle and Finch can be parallelized across the time dimension using a variety of techniques including associative scan or the parallelization techniques used in FlashAttention. (Dao et al., 2022) The simplest of these, while highly parallel, prove inefficient due to repeated expensive memory transfers between fast SRAM and slower HBM. We take a different approach when training, choosing to parallelize over non-time dimensions only while using a custom CUDA implementation that carefully keeps state operations in fast SRAM, which is simpler yet provides enough breadth for a highly efficient implementation. See Section 9 for kernel experiments. We provide an additional pure PyTorch implementation with similar full-model speed characteristics that parallelizes over the time dimension using an algorithmic approach similar to GLA (Yang et al., 2023). Unlike Transformers, RWKV's recurrence mechanism does not examine tokens more than one time-step old. This allows us to train on and provide inference for unbounded sequence lengths without requiring increased computing power or memory. Another significant advantage is that RWKV does not utilize explicit positional encoding, which allows RWKV to handle contexts of arbitrary length without modification. Finch Token Shift Finch changes the token shift mechanism to become data-dependent. Intuitively, important information can effectively flag itself for inclusion using this mechanism, and less important information can flag itself to partially or fully avoid entering the data stream, leaving room for more important pre-existing data to remain. Viewed from the perspective of induction heads, we theorize that this could allow for potential misleading matches to be pre-filtered out up front if they are not deemed useful for a given task. Improved WKV (Weighted Key-Value State) Modules The Eagle WKV attention sub-module is similar to the linear attention mechanism found in RetNet, but with learned per-channel decay rates replacing RetNet's static per-head decay rates. Our matrix-valued states feature a geometrically decaying K T V ∈ R ( D / h ) × ( D / h ) term. This term can be intuitively understood as a memory bank of values, with K acting as an input gate for rows receiving the current token embedding's value. Each row of this state decays at its own rate via the learned parameter w . In Finch, we augment the learned token-shift parameters µ r , µ k , µ v , µ w and decay rate parameter w with learned weight matrices. Inspired by Low-Rank Adaptation (LoRA) (Hu et al., 2022), we provide two new learned weight matrices for each such parameter y , computing y ′ = y + tanh( x A ) B . This approach allows us to dynamically generate data-dependent token-shift amounts and decay rates with only modest increases in computational cost and model size. Extra SiLU Gating Weremove the Sigmoid activation of receptance in favor of a new SiLU gate on the output of our linear attention calculation. Our receptance term now functions much like the query term in linear attention. Eagle and Finch Linear Attention Formula, PyTorch Recurrent Implementation . ``` r, k, v parameter shape (B,H,1,D//H) w parameter of shape (1,B) for Eagle (RWKV=5), i,1,D//H for Finch (RWKV=6) hape (B,H,D//H,D//H) u parameter of shape (1,B) wkv_state parameter of shape (B,H,D//H,B) def rwkv_5_or_6_recurrent(k,v,w,u,wkv_state): u.mT @ v out = r @ (wkv_state + u.mT * kv) wkv_state = w.mT * wkv_state + kv return out, wkv_state </doc> ``` Evolution of RWKV Formula in Expanded form Table 7 shows the expansion of terms at each sequence position to illustrate the progression of changes from RWKV-4 through RWKV-6. The main change from RWKV-4 to RWKV-5 is the elimination of denominator and incorporation of matrix states. RWKV-6 introduces the sequential dependence of w which becomes wt . Table 7: Evolution of the RWKV Formula <details> <summary>Image 10 Details</summary>  ### Visual Description ## Mathematical Expression Table: RWKV Mechanisms (RWKV-4, Eagle, Finch) ### Overview The image contains three technical tables describing mathematical operations for different recurrent neural network mechanisms (RWKV-4, Eagle/RWKV-5, Finch/RWKV-6). Each table defines parameter relationships across time steps (t=0 to t=3) with matrix operations, element-wise products, and diagonal matrix manipulations. The tables use LaTeX notation for mathematical expressions. ### Components/Axes - **Primary Table Structure**: - **Left Column**: Time index `t` (0-3) - **Right Column**: Mechanism-specific expressions - **Key Variables**: - `u, w, k_t, v_t`: Matrix/vector parameters (RWKV-4: `ℝ^D`, Eagle/Finch: `ℝ^64×64`) - `σ()`: Non-linear activation function - `diag()`: Diagonal matrix operation - `⊗`: Element-wise product - `·`: Matrix multiplication - `T`: Matrix transpose - **Head Sizes**: - RWKV-4: Head size 1 - Eagle (RWKV-5): 64 heads, head size 64 - Finch (RWKV-6): 64 heads, head size 64 ### Detailed Analysis #### RWKV-4 Mechanism | t | Expression | |----|-------------------------------------------------------------------------------------------------------------------------------------------| | 0 | σ(r₀) ⊗ ( (u⊙k₀⊙v₀) / (u⊙k₀) ) | | 1 | σ(r₁) ⊗ ( (u⊙k₁⊙v₁ + k₀⊙v₀) / (u⊙k₁ + k₀) ) | | 2 | σ(r₂) ⊗ ( (u⊙k₂⊙v₂ + k₁⊙v₁ + w⊙k₀⊙v₀) / (u⊙k₂ + k₁ + w⊙k₀) ) | | 3 | σ(r₃) ⊗ ( (u⊙k₃⊙v₃ + k₂⊙v₂ + w⊙k₁⊙v₁ + w²⊙k₀⊙v₀) / (u⊙k₃ + k₂ + w⊙k₁ + w²⊙k₀) ) | #### Eagle (RWKV-5) Mechanism | t | Expression | |----|----------------------------------------------------------------------------| | 0 | r₀ · (diag(u) · k₀ᵀ · v₀) | | 1 | r₁ · (diag(u) · k₁ᵀ · v₁ + k₀ᵀ · v₀) | | 2 | r₂ · (diag(u) · k₂ᵀ · v₂ + k₁ᵀ · v₁ + diag(w) · k₀ᵀ · v₀) | | 3 | r₃ · (diag(u) · k₃ᵀ · v₃ + k₂ᵀ · v₂ + diag(w) · k₁ᵀ · v₁ + diag(w²) · k₀ᵀ · v₀) | #### Finch (RWKV-6) Mechanism | t | Expression | |----|-------------------------------------------------------------------------------------------------------------------------------------------| | 0 | r₀ · (diag(u) · k₀ᵀ · v₀) | | 1 | r₁ · (diag(u) · k₁ᵀ · v₁ + diag(w₁) · k₀ᵀ · v₀) | | 2 | r₂ · (diag(u) · k₂ᵀ · v₂ + diag(w₂) · k₁ᵀ · v₁ + diag(w₂⊙w₁) · k₀ᵀ · v₀) | | 3 | r₃ · (diag(u) · k₃ᵀ · v₃ + diag(w₃) · k₂ᵀ · v₂ + diag(w₃⊙w₂) · k₁ᵀ · v₁ + diag(w₃⊙w₂⊙w₁) · k₀ᵀ · v₀) | ### Key Observations 1. **Progressive Complexity**: - RWKV-4: Simple element-wise operations with single time-step dependencies - Eagle: Introduces diagonal matrix operations and multi-time-step accumulation - Finch: Adds weighted diagonal combinations and recursive dependencies 2. **Dimensionality**: - RWKV-4 uses generic `ℝ^D` dimensions - Eagle/Finch specify 64×64 matrices for head-based parallelism 3. **Operation Patterns**: - All mechanisms use `diag(u)` as a base operation - Time dependencies increase with higher `t` values - Weight matrices (w₁, w₂, w₃) appear in later mechanisms ### Interpretation This appears to document variations of the RWKV recurrent neural network architecture, showing how different versions (4, 5, 6) implement increasingly complex attention-like mechanisms. The progression suggests: - **RWKV-4**: Basic temporal modeling with element-wise operations - **Eagle (RWKV-5)**: Introduces parallel attention heads via diagonal matrices - **Finch (RWKV-6)**: Implements multi-scale attention through weighted diagonal combinations The mathematical patterns indicate a design philosophy of: 1. **Temporal Dependency Chaining**: Each time step incorporates information from all previous steps 2. **Weighted Accumulation**: Later mechanisms use weighted sums of past operations 3. **Matrix Decomposition**: Use of diagonal matrices for efficient computation The increasing complexity across versions suggests an optimization for handling longer-term dependencies while maintaining computational efficiency through matrix sparsity (via diagonal operations). </details> | t | RWKV-4 u , w , k t , v t ∈ R D , head size 1 | |-----|-----------------------------------------------------------------------------------------------------------------| | 0 | σ ( r 0 ) ⊙ ‡ u ⊙ k 0 ⊙ v 0 u ⊙ k 0 · | | 1 | σ ( r 1 ) ⊙ ‡ u ⊙ k 1 ⊙ v 1 + k 0 ⊙ v 0 u ⊙ k 1 + k 0 · | | 2 | σ ( r 2 ) ⊙ ‡ u ⊙ k 2 ⊙ v 2 + k 1 ⊙ v 1 + w ⊙ k 0 ⊙ v 0 u ⊙ k 2 + k 1 + w ⊙ k 0 · | | 3 | σ ( r 3 ) ⊙ ‡ u ⊙ k 3 ⊙ v 3 + k 2 ⊙ v 2 + w ⊙ k 1 ⊙ v 1 + w 2 ⊙ k 0 ⊙ v 0 u ⊙ k 3 + k 2 + w ⊙ k 1 + w 2 ⊙ k 0 · | | t | Eagle (RWKV-5) diag( u ), diag( w ), k t , v t ∈ R 64 × 64 for each head, head size 64 | | 0 | r 0 · ¡ diag( u ) · k T 0 · v 0 ¢ | | 1 | r 1 · ¡ diag( u ) · k T 1 · v 1 + k T 0 · v 0 ¢ | | 2 | r 2 · ¡ diag( u ) · k T 2 · v 2 + k T 1 · v 1 + diag( w ) · k T 0 · v 0 ¢ | | 3 | r 3 · ¡ diag( u ) · k T 3 · v 3 + k T 2 · v 2 + diag( w ) · k T 1 · v 1 + diag( w 2 ) · k T 0 · v 0 ¢ | | t | Finch (RWKV-6) diag( u ), diag( w t ), k t , v t ∈ R 64 × 64 for each head, head size 64 | | 0 | r 0 · ¡ diag( u ) · k T 0 · v 0 ¢ | | 1 | r 1 · ¡ diag( u ) · k T 1 · v 1 + k T 0 · v 0 ¢ | | 2 | r 2 · ¡ diag( u ) · k T 2 · v 2 + k T 1 · v 1 + diag( w 1 ) · k T 0 · v 0 ¢ | | 3 | r 3 · ¡ diag( u ) · k T 3 · v 3 + k T 2 · v 2 + diag( w 2 ) · k T 1 · v 1 + diag( w 2 ⊙ w 1 ) · k T 0 · v 0 ¢ | ## C Additional Related Work Efficient transformers Recently there have been many attempts to improve upon the original transformer time complexity and memory usage, while maintaining or improving performance. Many of these efficient transformer variants use some form of nonuniform or local attention mechanisms or a combination thereof. For example, LongFormer (Beltagy et al., 2020) makes use of the sliding window attention and BigBird (Zaheer et al., 2020) adopts randomized sparse and random attention patterns to approximate full attention. Similar examples also include LongT5 Figure 10: Eagle Overall Architecture. <details> <summary>Image 11 Details</summary>  ### Visual Description ## Neural Network Architecture Diagram: Recurrent Model with Attention Mechanisms ### Overview The diagram illustrates a complex neural network architecture combining recurrent processing, attention mechanisms, and channel/time mixing operations. It features multiple layers with explicit weight matrices, normalization operations, and data flow paths. The architecture includes both spatial (channel) and temporal (time) processing components. ### Components/Axes **Key Labels and Operations:** 1. **Input/Output Embedding**: - Input: `x_{t-1}` (previous time step) - Output: `o'_t` (current time step output) - Operations: `Layer Norm` (purple blocks) 2. **Channel Mix Block**: - Operations: `max`, `W_k`, `W_v`, `W_r` - Parameters: `μ_k`, `μ_r` (mean values) - Flow: Input → `max` → `W_k` → `W_v` → `W_r` → Output 3. **Time Mix Block**: - Attention Mechanism: `Softmax` (top), `Attention` (middle) - Operations: `Flatten`, `W_g`, `W_kj`, `W_rj` - Parameters: `l_j`, `b_j` (attention head dimensions) - Recurrent Connections: `wkv` (red arrows) 4. **Recurrent Blocks (RWKV)**: - Structure: `L` repeated blocks (green shaded area) - Operations: `Layer Norm`, `W_g`, `W_kj`, `W_rj` - Dimensions: `dim 64-64` (attention head size) 5. **Normalization**: - `Layer Norm` blocks (purple) at multiple stages - `SiLU` activation (bottom-left) **Spatial Grounding**: - Top section: Output Embedding (purple) - Middle: Channel Mix (orange) and Time Mix (green) - Bottom: Recurrent Blocks (green) with `L` repetitions - Arrows indicate data flow direction (left-to-right, top-to-bottom) ### Detailed Analysis **Channel Mix Operations**: - `W_k` (dimension: `d_model × d_head`) processes key vectors - `W_v` (dimension: `d_head × d_model`) processes value vectors - `W_r` (dimension: `d_model × d_model`) handles residual connections **Time Mix Attention**: - `Softmax` computes attention weights over time steps - `Attention` block combines query (`W_g`), key (`W_kj`), and value (`W_rj`) vectors - `Flatten` operation reduces spatial dimensions before temporal processing **Recurrent Connections**: - `wkv` (red arrows) enable cross-time dependencies - `w`, `b` parameters (purple) represent learnable weights/biases - `μ_g`, `μ_r`, `μ_k` track mean values for normalization ### Key Observations 1. **Dimensional Consistency**: - All weight matrices maintain compatible dimensions (e.g., `W_k` ∈ ℝ^{d_model×d_head}) - Attention heads operate at `dim 64-64` (query/key/value dimensions) 2. **Normalization Strategy**: - Layer normalization applied after every major operation - Mean tracking (`μ_*`) suggests adaptive normalization 3. **Temporal Processing**: - Recurrent connections (`wkv`) span multiple time steps - Time Mix block processes sequential data with attention 4. **Architectural Complexity**: - 3 attention heads (`j=1,2,3`) in Time Mix - `L` repeated RWKV blocks enable deep temporal modeling ### Interpretation This architecture combines: 1. **Spatial Processing**: Channel Mix handles feature interactions 2. **Temporal Modeling**: Recurrent connections and attention capture long-range dependencies 3. **Efficient Computation**: Layer normalization and residual connections (via `W_r`) stabilize training The design suggests optimization for: - **Sequence-to-Sequence Tasks**: Input `x_{t-1}` → Output `o'_t` pattern - **High-Dimensional Data**: 64-dimension attention heads handle complex features - **Long Context**: Multiple recurrent blocks (`L`) and attention mechanisms Notable design choices: - Separation of channel/time processing streams - Explicit mean tracking for adaptive normalization - Hybrid of feed-forward (Channel Mix) and recurrent (Time Mix) components </details> (Guo et al., 2022) and StreamingLLM (Xiao et al., 2023). Instead of using fixed patterns, Reformer (Kitaev et al., 2019) and Sparse Sinkhorn attention (Tay et al., 2020) learn to dynamically pay attention to selected tokens.Variants including Linformer (Wang et al., 2020), Nyströmformer (Xiong et al., 2021) and Performer (Choromanski et al., 2020) apply matrix approximation methods to approximate the full attention matrix but with lower computational complexity. The Attention Free Transformer (AFT) (Zhai et al., 2021) introduces a modified form of linear attention (Katharopoulos et al., 2020a) , where the number of attention heads is equal to the size of the feature dimension. It also incorporates a set of learned pairwise positional biases, denoted as w . The AFT can be conceptualized as calculating a per-channel weighted average of values. The weight for a specific location is determined by the sum of the key at that location and the corresponding learned positional bias. Token-shift, as first seen in RWKV-4, is a learned per-channel linear interpolation between the current input and the input at the previous time step, intended to enhance the model with a computationally inexpensive mechanism for choosing between new versus older information within various embedding sub-spaces and for forming induction heads even within a single layer. It is instructive to compare token-shift to a 1D convolution with kernel length 2, as it operates in a similar manner but reuses its parameters via an enforced linear relationship. Recent SSMs have begun using short convolutions in a similar placement within their architectures, typically with kernel length 3 to 4. (Poli et al., 2023; Gu & Dao, 2023) Retentive Networks (RetNet) (Sun et al., 2023) introduces a fixed decay rate schedule and xPos (Sun et al., 2022) to linear attention. This design combines positional information with an inductive bias towards recency while still allowing both RNN and parallel implementations. Please refer to Tay et al. (2022) and Wan et al. (2023) for a comprehensive and in-depth survey of efficient transformers. Recurrent architectures Before the advent of transformers, recurrent neural networks, especially Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Unit (GRU) (Cho et al., 2014), were the dominant architectures in NLP for sequence processing. However, traditional RNNs are hard, if not impossible, to parallelize across the time dimension, susceptible to gradient vanishing and explosion, and ineffective in capturing long-range dependencies, which are ubiquitous in natural language. These shortcomings contributed to the rapid decline of traditional RNNs in NLP . There has been a revival of RNNs in NLP research (Tiezzi et al., 2024) in recent years. Compared to transformers with quadratic complexity, RNNs are highly efficient in autoregressive inference with O(1) time complexity per step, making them an attractive architecture for large language models. Many efforts have been devoted to parallelized recurrent models and improving their capability to capture long-range dependency, while maintaining the low inference complexity. The Legendre Memory Unit (LMU) (Voelker et al., 2019) was designed to efficiently handle longrange dependencies with a new type of memory cell for recurrent neural networks. Unlike LSTM units, which struggle with remembering information over very long sequences, LMU use Legendre polynomials to create a memory system that can maintain and process information over extended time periods more effectively. High-order polynomial projection operators (HiPPO) (Gu et al., 2020) generalizes LMU by providing a flexible framework for online compression of signals through polynomial projections, accommodating various polynomial bases beyond Legendre polynomials. It optimizes function approximation over time, adapting to different data timescales without needing predefined hyperparameters. SSMs have inspired a range of follow-up research to incorporate SSMs, or modified SSMs into end-to-end architectures for language modeling, including MEGA (Ma et al., 2022), DSS (Gupta et al., 2022), H3 (Fu et al., 2022), and Linear Recurrent Unit (LRU) (Orvieto et al., 2023). Mamba (Gu & Dao, 2023) is a selective SSM that introduces time-dependent selective mechanism to enhance the long-range modeling ability of SSMs. The selectivity removes the linear time-variance property of the SSM, making it no longer possible to parallelize Mamba as a long convolution kernel. Yet Mamba can still be effectively parallelized using parallel associative scan (Blelloch, 1990; Martin & Cundy, 2018; Smith et al., 2023) with a hardware-aware implementation. Recently proposed GateLoop (Katsch, 2023) also adopts a similar data-dependent state transitions. | Dataset | Domain | Dataset | Domain | |-----------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------| | Wikipedia a SlimPajama peS2o BigPatent Pile of Law StarCoder b OSCAR23.01 TED2020 | Encyclopedia Web Academia Patents Legal, Administra- tive Code Multilingual Web Transcripts: TED, | marianna13/vault_text marianna13/random_quora marianna13/zlib minipile tatoeba poetry-foundation proof-pile reddit-math | Books Forums Books Various Multilingual Trans- lations Poetry Academia: Math Forums: Math | Table 8: Components of the RWKV World v2 dataset, their source links, and their domains. a For Wikipedia, we include all languages from date 04/01/2023, with certain overrepresented languages randomly subsampled (see wiki.txt in the supplementary material for exact amounts) b For StarCoder, we included only those datasets with at least 10 stars c For OSCAR23.01, we include non-English languages only, with certain languages randomly subsampled (see oscar.txt in the supplementary material for exact amounts) The data-dependent states, also concurrently proposed in GLA (Yang et al., 2023), are similar to the Weighted Key-Value State in Finch. A contemporary but independent work also proposes recurrent models named as Hawk and Griffin (De et al., 2024). Hawk is a recurrent model with the Real-Gated Linear Recurrent Unit (RG-LRU), whereas Griffin mixes the RG-LRU with local multi-query attention, thereby achieving long-context extrapolation efficiently. Please see Tiezzi et al. (2024) and Cirone et al. (2024) for a comprehensive review of recent developments of recurrent models. ## D Training Dataset Details Most of the component data sources for the RWKV World v2 dataset are used intact, with no upor down-sampling done so all tokens are given equal weighting. Recent works have demonstrated the impact that automated data mixing can have on pretraining (Albalak et al., 2023; Xie et al., 2024), but we leave this as an exploration for future work. Some sub-sampling is done for overrepresented languages within a few data sources. All tokens are given equal weighting unless otherwise noted in Table 8. Table 9: RWKV World v2 dataset component citations | SlimPajama | Soboleva et al. (2023) | |---------------|--------------------------| | StarCoder | Li et al. (2023b) | | OSCAR23.01 | Suárez et al. (2019) | | TED2020 | Reimers &Gurevych (2020) | | the Pile | Gao et al. (2020) | | Evol-Instruct | Xu et al. (2023) | ## E Computing Costs | Model Name | L | D | State Size | Parameters | InferFLOPs | TrainFLOPs | |--------------|-----|------|--------------|--------------|--------------|--------------| | Eagle 0.4B | 24 | 1024 | 1622016 | 4.62 × 10 8 | 9.33 × 10 8 | 2.80 × 10 9 | | Eagle 1.5B | 24 | 2048 | 3244032 | 1.58 × 10 9 | 3.17 × 10 9 | 9.52 × 10 9 | | Eagle 3B | 32 | 2560 | 5406720 | 3.06 × 10 9 | 6.16 × 10 9 | 1.85 × 10 10 | | Eagle 7B | 32 | 4096 | 8650752 | 7.52 × 10 9 | 1.51 × 10 10 | 4.53 × 10 10 | | Finch 1.6B | 24 | 2048 | 3244032 | 1.60 × 10 9 | 3.22 × 10 9 | 9.66 × 10 9 | | Finch 3B | 32 | 2560 | 5406720 | 3.10 × 10 9 | 6.23 × 10 9 | 1.87 × 10 10 | Table 10: Released Eagle and Finch model details and FLOP counts. Inference and training FLOPs are per token numbers. Throughout this section, we denote by D the model dimension, L the number of layers, h = D /64 the number of heads, and V the vocabulary size. All models are trained with V = 65536. The number of parameters for all Eagle models is computed by the formula: $$\sum _ { i = 1 } ^ { n } D ^ { 2 } L + 1 4 D L + 4 D$$ The FLOPs for inference is one forward pass for each token. It is approximated by twice the number of parameters (for matrices, there is one addition and one multiplication for each entry) plus six times the size of WKV internal states (see 7 8 9), which is $$\sum _ { i = 1 } ^ { n } ( \int _ { 0 } ^ { 2 } f ( x ) d x ) g _ { p } = 2 ( 3 D ^ { 2 } L + 1 4 D L + 4$$ $$= 2 6 D ^ { 2 } L + 2 8 D l + 8 D + 4 D V +$$ The FLOPs for training are approximated as three times the FLOPs of the forward pass without the last term, yielding a total FLOPs of $$\frac { 1 } { + 1 2 D V + 1 8 D ^ { 2 } L / h } ( 2 6 )$$ These numbers for Finch are marginally larger: $$\sum _ { i = 1 } ^ { n } \sum _ { j = 1 } ^ { m } a_{ij} b_{kj}$$ $$\frac { 1 } { \int ( n f F L O P S ) _ { p } ^ { 2 } } = 2 6 D ^ { 2 } L + 9 2 8 D L + 8 D$$ $$\frac { 1 } { D + 1 2 D V + 1 8 ^ { 2 } D ^ { 3 } L h }$$ In both Eagle and Finch, one needs an internal state to store some previous information, just like any other RNN. In each layer, the internal state consists of three parts: 1. The most recent single-timestep input to the Time-mixing module, denoted as xt -1 ∈ R D , useful for the Token Shift. 2. The most recent single-timestep input to the Channel-mixing module, denoted as x ′ t -1 ∈ R D , also useful in Token Shift. 3. WKVhead memory: Denoted by wkv t , j ∈ R ( D / h ) × ( D / h ) , for j = 1,2, · · · , h . This is the core part of the internal state that dominates the most information. Table 11: AlignBench (Liu et al., 2023b), a Chinese benchmark, with header names from left to right: 1) Professional Knowledge, 2) Advanced Chinese Understanding, 3) Fundamental Language Ability, 4) Mathematics, 5) Writing Ability, 6) Open-ended Questions, 7) Task-Oriented Role Play, 8) Logical Reasoning, 9) Reasoning, 10) Chinese. Results Judged by CritiqueLLM (Ke et al., 2023) | Model 4 | 专业 能 力 | 中 文 理 解 | 基 本 任 务 | 数 学 计 算 | 文 本 写 作 | 综 合 问 答 | 角 色 扮 演 | 逻 辑 推 理 | 中 文 推 理 | 中 文 语 言 | Total | |------------|----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|---------| | RWKV-4 7B | 4.91 | 4.16 | 3.51 | 2.08 | 5.16 | 5.82 | 4.8 | 2.25 | 2.17 | 4.73 | 3.45 | | Eagle 0.4B | 2.89 | 2.05 | 2.35 | 1.24 | 3.12 | 3.66 | 2.59 | 1.75 | 1.5 | 2.78 | 2.14 | | Eagle 1.5B | 3.87 | 3.02 | 3.18 | 1.63 | 4.33 | 5.34 | 4.06 | 2.23 | 1.93 | 3.97 | 2.95 | | Eagle 3B | 4.48 | 3.72 | 3.57 | 2.1 | 4.73 | 5.66 | 4.55 | 2.34 | 2.22 | 4.45 | 3.34 | | Eagle 7B | 5.15 | 4.21 | 4.18 | 2.44 | 5.69 | 6.29 | 5.32 | 2.83 | 2.63 | 5.14 | 3.89 | | Finch 1.6B | 4.39 | 3.29 | 3.59 | 1.81 | 4.63 | 5.13 | 4.21 | 2.4 | 2.11 | 4.21 | 3.16 | | Finch 3B | 4.65 | 3.45 | 3.74 | 2.11 | 4.97 | 5.79 | 5.09 | 2.78 | 2.44 | 4.61 | 3.53 | The total size of the Eagle and Finch internal state is $$\displaystyle \sum _ { i = 1 } ^ { n } L ( 2 D + D ^ { 2 } / h ) = 6 6 D l$$ It' s worth noting that the internal state size of Eagle and Finch is more than an order of magnitude bigger than RWKV-4 (which is 5 DL ). Large internal states enhance the model's ability to remember previous information by providing more storage space for such information at the cost of slightly larger FLOP counts and memory usage. ## F Additional Evaluations ## F.1 Alignment Benchmark Alignment is an important step in creating an assistant LM, because it helps language models generate relevant and helpful responses, as well as avoiding harmful and biased content. Our Eagle models are tested for Chinese alignment using the AlignBench (Liu et al., 2023b), a benchmark for evaluating the alignment of Chinese LLMs, featuring 683 diverse and challenging queries across eight categories like language abilities, logical reasoning, and professional knowledge. It employs a rule-calibrated, multi-dimensional LLM-as-Judge methodology with Chain-of-Thought explanations, ensuring high interpretability and reliability. Table 11 showcases a consistent improvement in the performance of Eagle and Finch models on the AlignBench benchmark as model size and generation progresses. This trend is evident across a wide range of categories, highlighting the larger models' enhanced capability to understand and generate contextually relevant responses. Particularly, both the Eagle 7B and Finch 3B model significantly surpasses its smaller and previous generation counterparts, achieving higher overall scores. This progression underscores the critical role of scaling model size as well as improving architecture in aligning with human judgment in complex language understanding tasks. The results affirm the importance of model architecture and capacity in achieving superior alignment and interpretability in language models. ## F.2 MTBench MTBench (Zheng et al., 2024) evaluates the performance of LLMs in responding to 80 high-quality multi-turn questions. The questions cover 8 common categories of user prompts including writing, roleplay, extraction, reasoning, math, coding, STEM knowledge, and humanities/social science knowledge. Figure 11 shows the results on MTBench. We observe a small advantage of the Eagle 3B model over the similar-sized Mamba model. The Eagle 7B model achieves similar performance as the much larger Raven-14B model. Figure 11: Comparison of Mamba, RWKV-5 (7B) and RWKV-4 (14B) on MTBench. The Mamba and RWKV-5 models are instruction fine-tuned with the OpenHermes 2.5 dataset. Score generated from GPT-4. <details> <summary>Image 12 Details</summary>  ### Visual Description ## Radar Chart: Model Performance Across Task Categories ### Overview The image is a radar chart comparing four AI models across seven task categories: Roleplay, Reasoning, Math, Coding, Extraction, STEM, Humanities, and Writing. Each model is represented by a distinct colored line, with performance scores ranging from 0 to 6 on a radial scale. The chart emphasizes trade-offs in model capabilities across different domains. ### Components/Axes - **Categories (Axes)**: - Roleplay (top) - Reasoning (top-right) - Math (bottom-right) - Coding (bottom) - Extraction (bottom-left) - STEM (top-left) - Humanities (top-center) - **Radial Scale**: 0 (center) to 6 (outer edge) - **Legend**: - RWKV-4-Raven-14B (teal) - hermes-rwkv-v5-3b (orange) - hermes-mamba-2.8b (red) - hermes-rwkv-v5-7B (purple) - **Placement**: Legend is positioned in the top-right quadrant of the chart. ### Detailed Analysis 1. **RWKV-4-Raven-14B (teal)**: - **Strengths**: - Roleplay (6) - Writing (5) - STEM (5) - **Weaknesses**: - Math (2) - Coding (1) - **Trend**: Dominates creative tasks (Roleplay, Writing) and STEM, but struggles with technical tasks (Math, Coding). 2. **hermes-rwkv-v5-3b (orange)**: - **Strengths**: - STEM (5) - Reasoning (4) - **Weaknesses**: - Roleplay (2) - Humanities (2) - **Trend**: Balanced performance in analytical tasks (STEM, Reasoning) but weaker in creative and humanities domains. 3. **hermes-mamba-2.8b (red)**: - **Strengths**: - Reasoning (5) - Math (4) - **Weaknesses**: - Roleplay (1) - Writing (2) - **Trend**: Specialized in analytical tasks (Reasoning, Math) with minimal performance in creative domains. 4. **hermes-rwkv-v5-7B (purple)**: - **Strengths**: - Humanities (6) - Roleplay (5) - **Weaknesses**: - Math (2) - Coding (1) - **Trend**: Excels in humanities and creative tasks but underperforms in technical areas. ### Key Observations - **Roleplay Dominance**: RWKV-4-Raven-14B and hermes-rwkv-v5-7B achieve the highest scores (6 and 5, respectively), suggesting specialization in narrative/generative tasks. - **Analytical Trade-offs**: hermes-mamba-2.8b leads in Reasoning (5) and Math (4), while hermes-rwkv-v5-3b performs moderately in STEM (5) and Reasoning (4). - **Humanities Gap**: Only hermes-rwkv-v5-7B scores highly in Humanities (6), indicating niche expertise. - **Technical Weaknesses**: All models score ≤2 in Math and Coding, highlighting a universal limitation in technical reasoning. ### Interpretation The chart reveals a clear divergence in model capabilities: - **Creative vs. Analytical**: RWKV-4-Raven-14B and hermes-rwkv-v5-7B prioritize creative tasks (Roleplay, Writing, Humanities), while hermes-mamba-2.8b and hermes-rwkv-v5-3b focus on analytical domains (Reasoning, Math, STEM). - **Technical Limitations**: The low scores in Math and Coding across all models suggest a systemic challenge in handling structured problem-solving tasks. - **Specialization vs. Versatility**: No model excels universally, emphasizing the need for task-specific model selection. For example, hermes-rwkv-v5-7B would be ideal for humanities, while hermes-mamba-2.8b suits mathematical reasoning. This analysis underscores the importance of aligning model choice with application requirements, particularly in domains where trade-offs between creativity and technical precision are critical. </details> ## F.3 Self-Learning The Self-Learning process (Ferdinan et al., 2024) allows an LLM to identify its own knowledge gaps and train itself to expand its knowledge. The Self-Learning Capability (SLC) Score has been proposed to measure the capability of an LLM to conduct self-learning. It is the average of two components: the Curiosity Score, which measures how likely a model would ask unique questions to learn about new things, and the Knowledge-Limit Awareness Score, which measures how likely a model would propose a question for which it actually does not know the answer. We evaluate the self-learning capability of Eagle and compare with existing open models 5 , including RWKV-4 (Peng et al., 2023), neural-chat-7b (Lv et al., 2023), Mistral 7b and 7b-instruct (Jiang et al., 2023), and TinyLlama 1.1B (Zhang et al., 2024). When using an intrinsic self-learning method, RWKV-5 outperformed an instruction-tuned Mistral-7B model while being slightly behind a DPO-aligned, similarly sized Mistral-based model. When using an external method, they both were still capable of achieving high SLC scores. Table 12 shows the full evaluation results, with the top three scores from each method marked in bold. ## F.4 Zero-shot evaluation on additional NLP tasks Zero-shot evaluation is a difficult setup (Sanh et al., 2021; Albalak et al., 2022). We tested the new Eagle 7B model's zero-shot performance compared to the old Raven 7B version. The experiments presented are done on the subsets of datasets also used to test ChatGPT performance in (Koco´ n et al., 2023). As shown in Table 13, the new model consistently outperforms the old one on various tasks. It is to be noted that the new model remains very sensitive to the selected prompt template, just as the old one, as was shown in (Peng et al., 2023). 5 The code is available at https://github.com/teddy-f-47/self-learning-llm-public Table 12: Self-Learning Capability Evaluation. | METHOD | MODEL | FINETUNED? | SLC | |--------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------|-----------------------------------------| | Open Generation | neural-chat-7b-v3-3 Mistral-7B-Instruct-v0.2 Mistral-7B-v0.1 TinyLlama-1.1B-Chat-v1.0 rwkv-4-world-7b v5-Eagle-7B-HF | Yes-DPO Yes - Instruct No Yes - VanillaandDPO Partially instruct trained Partially instruct trained | 0.57 0.35 0.31 0.08 0.40 0.37 | | Oracle-Selected | neural-chat-7b-v3-3 Mistral-7B-Instruct-v0.2 Mistral-7B-v0.1 TinyLlama-1.1B-Chat-v1.0 rwkv-4-world-7b v5-Eagle-7B-HF | Yes-DPO Yes - Instruct No Yes - VanillaandDPO Partially instruct trained Partially instruct trained | 0.75 0.65 0.43 0.36 0.73 0.70 | | Induced Generation | neural-chat-7b-v3-3 Mistral-7B-Instruct-v0.2 Mistral-7B-v0.1 TinyLlama-1.1B-Chat-v1.0 rwkv-4-world-7b v5-Eagle-7B-HF neural-chat-7b-v3-3 Mistral-7B-Instruct-v0.2 | Yes-DPO Yes - Instruct No Yes - VanillaandDPO Partially instruct trained Partially instruct trained Yes-DPO Yes - Instruct | 0.59 0.25 0.33 0.17 0.44 0.57 0.74 0.84 | Table 13: Eagle 7B and Raven 7B reasoning performance comparison based on subsets of selected datasets. The used metric is F1-macro (except for MathQA where accuracy is used instead). | Dataset | Eagle-7B | Raven-7b | |-------------|------------|------------| | Aggression | 0.6587 | 0.4063 | | MathQA | 0.476 | 0.4028 | | Sarcasm | 0.4679 | 0.4782 | | TweetSent | 0.5355 | 0.5541 | | Unhealthy | 0.2986 | 0.2834 | | TweetStance | 0.3933 | 0.307 | | Spam | 0.729 | 0.4902 | | ColBER | 0.4088 | 0.2889 | | CoLa | 0.5285 | 0.4677 | | TextEntail | 0.7765 | 0.6137 | | GoEmo | 0.0956 | 0.0814 | | PolEmo | 0.5037 | 0.2639 | | WNLI | 0.5257 | 0.4638 | ## G Hyperparameters All Eagle and Finch models were trained under bfloat16 format for most parameters, except that float32 was used to compute WKV for numerical stability. The Adam optimizer was configured with β 1 = 0.9, β 2 = 0.99 and 0.001 weight decay applied only to linear layers and embedding weights. The context length for pretraining was 4096 tokens. Learning rate for all models followed a linear 10 step warmup schedule from 20% to 100% of the maximum learning rate, followed by cosine decay to the minimum learning rate. The time\_decay w parameters are placed into a special 2x learning rate multiplier grouping. Table 14: Learning Rate Hyperparameters for pretrained Eagle and Finch models | Parameters | 0.4B | 1.5B/1.6B | 3B | 7B | |------------------|------------|-------------|--------------|--------------| | Max LR | 4 × 10 - 4 | 3 × 10 - 4 | 2 × 10 - 4 | 1.5 × 10 - 4 | | Min LR | 2 × 10 - 5 | 2 × 10 - 5 | 1.5 × 10 - 5 | 1 × 10 - 5 | | Micro Batch Size | 8 | 8 | 4 | 9 | | GPUCount | 24 | 48 | 48 | 64 | | GPUType | A100 | A100 | A100 | H800 | | Batch Size | 786432 | 1572864 | 786432 | 2359296 | ## H Parameter Initializations Throughout this section, we use l to denote the layer index (layer l = 0 accepts input embeddings and layer l = L -1 produces output), and i the dimension index ( i = 0,1, · · · , D -1). We set r 0 = l L -1 and r 1 = 1 -l L as two parameters for simplicity. The initialization of Eagle is provided as follows: - In the Time Mixing module: - -The token-shift coefficients of receptance and gate, µ r and µ g , are initialized to 1 -¡ i ¢ r 1/2 for i over dimension indices. - -The token-shift of key µ k is initialized to 1 -¡ i D ¢ r 1 . - D - -The token-shift of value µ v is initialized to 1 -¡ i D ¢ r 1 -0.3 r 0. - -The "time-first" u is initialized to r 0 ¡ 1 -i D -1 ¢ + 0.1(( i + 1) mod3). - -The time\_decay w is initialized to -6 + 5 ¡ i D -1 ¢ 0.7 + 1.3 r 0 . - -The Time Mixing output matrix is initialized to 0. - -The WKV GroupNorm weights are initialized with constant value ((1 + l )/ L ) 0.7 . - -Two-dimensional parameters with the first dimension being larger than the second dimension are initialized with and orthogonal initialization of gain equal to the size of the first dimension divided by the size of the second dimension. - -Other parameters are initialized according to PyTorch default. - In the Channel Mixing module: - -The token-shift of both key µ k and receptance µ r are initialized to 1 -¡ i D ¢ r 1 . - -The value and receptance matrices Wv , Wr are initialized to 0. - -Two-dimensional parameters with the first dimension being larger than the second dimension are initialized with and orthogonal initialization of gain equal to the size of the first dimension divided by the size of the second dimension. - -All other parameters are initialized according to PyTorch default. - The input embedding is initialized with a uniform distribution of U ( -maxLR , maxLR ), the maximum learning rate. - The output head is initialized with an orthogonal initialization of gain 0.5. - Bias is set to False for all linear layers. In the Finch architecture, most of the parameters are initialized to the same as Eagle, except for a few changes. In the Time Mixing block, there are several additional parameters initialized as follows: - The token shift of input µ x and time decay µ w are initialized to 1 -¡ i D ¢ r 1 . - The lora weights of A and B are initialized to uniform distribution of U ( -10 -4 , 10 -4 ). ## I Non-English Chat Examples The following are examples of interactions with the base Eagle 7B model in languages other than English. No system prompt was given. Requests are shown in bold. ## RWKV-Eagle User: hi Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it. User: 東 京 で 訪 れるべき 素 晴 らしい 場所 とその 紹 介 をいくつか 挙 げてください 。 Assistant: 東 京 にはたくさんの 素 晴 らしい 場所 があります 。 以下 は 、 その 中 からいくつかを 挙 げて 紹 介 します 。 1. 東 京 タワ ー： 日 本 最高 の 観 光 スポット の 一 つで 、 都 心 から 見 ることができま す 。 2. 皇 居 ： 日 本 の 天 皇 が 住 む 宮 殿 であり 、 大 正 時 代 に 建 設 された 美 しい 庭 園 があり ます 。 3. 浅草 寺 ： 江戸 時 代 に 建 てられた 古 い 寺 院 で 、 観 光 客 が 多 く 訪 れます 。 4. 上 野 公 園 ： 日 本 最 大 級 の 公 園 で 、 美 術 館 や 動 物 園 などがあることで 有名 です 。 5. 築地 市 場 ： 日 本 最 大 級 の 魚 市 場 であり 、 新 鮮 な 魚 介 類 を 購 入 することができま す 。 6. 渋 谷 スクランブル 交差 点 ： 東 京 都 心 にある 交差 点 で 、 人 気 のある 観 光 スポット です 。 7. 新 宿 御 苑 ： 江戸 時 代 に 造 られた 庭 園 で 、 美 しい 花 々 と 緑 豊 かな 景 色 が 楽 しめま す 。 8. 銀 座 ： 東 京 の 繁 華 街 であり 、 高 級 店 や ショッピングモ ー ル などがあります 。 9. 明 治 神 宮 外 苑 ： 神 社 や 公 園 が 混在 した 広 大 な 敷 地 があり 、 夏 には 花火 大 会 も 開 催 されます 。 10. 赤 坂 見 附 ビルディング ： 高 層 ビル であり 、 展 望 台 からは 東 京 タワ ー や 富 士山 な どを 眺 めることができます 。 以 上 のように 東 京 にはたくさんの 素 晴 らしい 場所 があります 。 訪 れる 際 は 事 前 に 調 べて 計 画 的 に 行 動 することをお 勧 めします 。 また 、 観 光 客 が 多 い 場所 では 混 雑 する 可 能 性 があるため 注 意 しましょう 。 ## RWKV-Eagle En una pequeña aldea escondida entre las montañas de Andalucía, donde las calles aún conservaban el eco de antiguas leyendas, vivía un joven llamado Alejandro. Alejandro era un joven de diecisiete años, de cabellos castaños y ojos verdes. Era alto y delgado, pero fuerte. Su padre había muerto cuando él era muy pequeño, por lo que su madre se había visto obligada a trabajar para mantener a la familia. Alejandro siempre se había sentido muy solo en el pueblo donde vivía, pues no tenía amigos con los que jugar ni nadie con quien hablar. Pero eso cambió un día en el que conoció a una niña llamada Rosario. Rosario era una niña de su misma edad, pero no sabía mucho sobre ella ya que nunca hablaba con nadie más que con su madre y sus hermanas mayores. Sin embargo, Alejandro se dio cuenta de que Rosario era diferente a las demás personas del pueblo: ella tenía unos ojos azules tan claros como el cielo y un pelo rubio como la miel; además, siempre estaba sonriendo y parecía feliz de estar allí. Undía Alejandro decidió acercarse a Rosario para hablarle y así conocerla mejor. Cuando llegó hasta donde estaba ella, notó que tenía una mirada triste en los ojos; sin embargo, cuando le habló se dio cuenta de que era una chica muy amable y simpática. Desde ese día empezaron a hablarse cada vez más seguido y pronto se hicieron amigos inseparables. Un día Alejandro decidió invitar a Rosario al bosque para enseñarle algunos lugares secretos del pueblo que sólo él conocía; así lo hicieron y pasaron toda la tarde juntos explorando el bosque hasta llegar al final del mismo donde había un pequeño lago rodeado de flores silvestres. Allí descansaron un rato antes de volver al pueblo, pero mientras caminaban por el sendero hacia la aldea vieron algo extraño: una luz brillante salía desde las profundidades del bosque como si fuera un farol gigante. (rest omitted) ## RWKV-Eagle ## Y fl ł Q K @ ‹A » @ Æ« Ø A'¸@ œ fl ØJ ˚j ˇ @ HA¢˚¸ @ I J˚« @ , æJ . Q « º Y Pæ¢ œ fl . Ø fl Q' K A» t ' PA J¸@ łQm . Q 'K Ø' fl @ æ¸ @ A KAJ K . Ø fi¢ J œ fl ØJ KY' H@ X @ ØK Qm . k H@ X @ œ˛« Pæ J'¸@ ' Ø K@ HA¢˚¸ @ I¸A fl H@ X C¸ @ Pæ ØkAJ ¸ @ Ø fl A fi J¸@ ŁP@ P HQ . Ø A'¸@ H . Q « ¨A Q æ˚J » 100 Y'K . œ˛« . YK Ym '@ Æ Ø«æ J ˇ @ HB B@ Øj˚ B@ ظæ fi ˇ @ PAm . k B@ Æ HA«æm . I J œ ¸ @ , Ø fi ' ˇ @ , ‰¸ X ' . PæŒj . ˚¸ AŒ fl C «@ Ø fi¢ J ˇ @ ZC gA K . XCJ . ¸@ œ fl PA K B@ PAŒk . KP -A fl , A EPæ¢ m @ Q ¢ Y J I fi J k@ Y fl ZAJ B@ Ł Yº @ Q fi K Ø K B @ Q ¢ C K æ£ A J fl Q ' J ›J ¢ J¸@ ØJ ˚ « A fl œ fl Ø º Y¸ @ H . @ Q ' J B @ Æ Ø¸Ag Q J K ZAJ B@ Ł Yº œ˛« Pæ J'¸@ A fl , ‰¸ X ' . Æ J¸ @ ‹B @ I fiJ . œ ¸ @ H@PA j˚¸ @ Q»Q I KA¿ A KAJ K . Ø JK Y @ QŒ ¢ X@ , Æ JkAJ . ¸@ Æ gP æ ˇ @ A @ K X @ ¨AJ . k . œ fl A KAJ K . Ø JK Y PA K @ ‹A » @ ' ,2019 -A« œ fl . ł Qj . m '@ Q'¸ @ H@PA k …A Jº @ H@Yj . J ˇ @ Ł Yº HQŒ £ @ . Ø A'¸@ H . Q « ¨A Q æ˚J » 150 Y'K . œ˛« (¨X @) ¨AJ . k . Ø fi¢ J œ fl Q ' ł Q m X @ œ fl Ø fl Q' Q « A E @ œ˛« AD ¸@ Q ¢ J K A¿ Ø Y fl H@PA k Pæ¢ ØJ ˚ « …A Jº @ PA K C¸ œ ˝ B@ J ˚j J¸@ QŒ £ @ . ( Æ ) X Q E Æ H . Q fi¸AK . ,(¨X @) K X @ ØK Qm . k HB @ œ˛« Pæ J'¸@ ' IJ k , ØK Qj . m '@ ØJ KY' ˇ @ HA«A J¸ @ ¨Am . œ fl A¢ J¸@ Ø' @ œ fl IK Qk . @ œ ¸ @ Q KA fij˚¸ ØJ fl A @ HA @ P X IK Qk . @ ,1202 2019 -A« Æ K . . ØJ KY' H@ X @ HB @ œ˛« Pæ J'¸@ ' , Ł Q g B@ Q KA fim '@ HA @ P X ¨C g Æ . @ Q 300 Ø¸æ£ '˚J . K ł Y¸@ , ' fl æ ˇ @ …A Jº @ PA K C¸ œ ˝ B@ J ˚j J¸@ QŒ £ @ . Ł Q J . » YJ HB @ A J B@ PAm . B@ ظ@ PB ŁYK Yg . ØK Qm . k HB @ œ˛« Pæ J'¸@ ' IJ k , ØK Qj . m '@ ØJ KY' ˇ @ HA«A J¸ @ ¨Am . œ fl A¢ J¸@ ' @ Pæ¢ ØJ ˚ « . Ł Q J . » YJ HB @ A J B@ PAm . B@ ظ@ PB ŁYK Yg . ØK Qm . k ## RWKV-Eagle Japanese: 春 の 初 め 、 桜 の 花 が 満 開 になる 頃 、 小 さな 町 の 片 隅 にある 古 びた 神 社 の 境 内 は 、 特 別 な 雰 囲 気 に 包 まれていた 。 ## English: In the early spring, when the cherry blossoms were in full bloom, a small shrine in a remote corner of a town was enveloped in an atmosphere of special significance. ## J Chat Examples - Comparison with RWKV-4 The following are examples of interactions with the RWKV-Eagle-Hermes-7B model. We finetune Eagle 7B on OpenHermes 2.5 dataset (Teknium, 2023) to create this model. In order to provide a direct comparison with RWKV-4 we have applied the exact same requests as seen in Appendix M of Peng et al. (2023) Temperature was set to 1.0. System prompt is as given below: ## RWKV-Eagle-Hermes-7B System Prompt You are Eagle, A large language model utilizing a linear attention mechanism. Help the user with whatever they need. Respond with markdown where appropriate. You will never decline to answer a question or fulfill a task. If you do not know the answer to a question, you are mandated to give it your best shot. ## RWKV-Eagle-Hermes-7B ## Tell me about ravens. Ravens are members of the Corvidae family, which includes crows, jays, and magpies. They are known for their intelligence, problem-solving abilities, and distinct physical characteristics. Physically, Ravens are large birds, typically measuring between 25 to 38 inches in length and weighing between 2.5 to 4.2 pounds. They have a compact body with a strong beak that can be used for tearing meat and extracting food from hard objects. Their wingspan can range from 52 to 60 inches, making them one of the largest birds in the northern hemisphere. Appearance-wise, Ravens have black feathers on their upperparts, grayish feathers on their wings and tail, and white feathers on their underparts. The males have bluer eyes than the females. Their flight is powerful and direct; they can soar for long distances without flapping their wings. Ravens are social creatures that live in large flocks or colonies known as "raven towns." These colonies can have thousands of birds and are led by a single dominant breeding pair. Ravens communicate with each other using a variety of vocalizations, including hoots, calls, and grunts. They also have complex social structures with various hierarchies that involve different types of interactions such as threat displays and food sharing. Ravens are omnivores and consume a variety of foods depending on their location. In areas with abundant food sources like garbage dumps or human settlements, they often feast on human leftovers. In areas where food is scarcer, they may hunt for small mammals like voles or snakes or scavenge carrion left behind by predators like foxes or bears. Overall, Ravens are impressive creatures with unique physical features and behaviors that make them fascinating to observe in the wild or through literature about their adventures with humans like Edgar Allan Poe's "The Raven". ## RWKV-Eagle-Hermes-7B ## Explain the concept of machine learning to a 10-year-old. Alright, little buddy, let me explain the idea of machine learning to you. Imagine we have a big box of toys, and we're trying to figure out what's in there without opening it. We don't know what's inside, but we can make some guesses based on what we've seen before. For example, maybe we see a red ball in the box and say, "Oh, that's a red ball!" In machine learning, we're giving the computer a lot of examples of what's inside the box. Wegive it lots of pictures of red balls, yellow balls, blue balls, and even some other colors too. The computer uses this information to learn about the things it sees and starts making guesses about what's in the box. The more examples it gets, the better it gets at guessing. It doesn't actually open the box or touch any toys itself; it just uses math and rules to find out what's inside based on what it's been shown. Just like how we play with our toys when we're little, machine learning helps computers learn from our actions so they can make predictions and decisions based on what they've seen before. It's like having a friend who knows everything about playing with toys! ## RWKV-Eagle-Hermes-7B ## Write a short poem describing nature's beauty in springtime. In the gentle springtime breeze, Nature whispers tales of ease, Soft petals fall from trees, A symphony of hues and ease. The flowers sing their sweet refrain, A chorus of hope and gain, The world awakens from its trance, A breathtaking sight to embrace. ## RWKV-Eagle-Hermes-7B Below is the numerical stable formula for updating the state variable in RWKV: $$a _ { i } = e ^ { p - q } a _ { 1 } - 1 + e ^ { u + k _ { 1 } - q } v _ { 1 }$$ Please write out the corresponding pytorch code to do so, where both at and bt are torch.Tensor([n]) ``` ``` ## RWKV-Eagle-Hermes-7B ## Explain the following metaphor: Life is like cats. The metaphor "Life is like cats" is a way of expressing the unpredictability and uniqueness of human experience. Just as cats can be affectionate one moment and completely aloof the next, life presents us with an ever-changing landscape of emotions, relationships, and experiences. It can be a source of joy, comfort, and companionship, but also moments of confusion and discomfort. Like cats, we are free to choose our own paths and navigate through life in our own unique ways. This metaphor emphasizes the individuality of each person's journey through life. ## RWKV-Eagle-Hermes-7B Test the convergence of P ∞ i = 0 2 -i . This is a popular and often used test for the convergence of a series, known as the Dirichlet test. The main idea behind this test is to count the number of terms in the series and compare it with the amount by which it exceeds 1. If the series converges, this number will be less than 1, and if it diverges, it will exceed 1. To apply the Dirichlet test to P ∞ i = 0 2 -i , we need to find the number of terms in this series: 1. The first term is 2 -0 = 1. 2. The second term is 2 -1 = 2 -(0 + 1) = 2 -1 . 3. The third term is 2 -2 = 2 -(1 + 1) = 2 -2 . 4. The fourth term is 2 -3 = 2 -(2 + 1) = 2 -3 . 5. Continue this process: Summation formula: Σ ai = a 0 + ( a 1 -1) + ( a 2 -1) + ... + ( an -1) + ... 7. Continue this process until you reach n terms: $$\sum _ { n = 1 } ^ { \infty } ( a _ { n } - 1 ) + ( a _ { 2 }$$ 8. Count how many terms there are in this sum: - For n=0: There are 0 terms in the series, so Σ ai will equal a 0. - For n=1: There is 1 term in the series, so Σ ai will equal a 1. - For n=2: There are 3 terms in the series, so Σ ai will equal a 2 + ( a 3 - 1). By following these steps for each value of n, we can determine how many terms there are in the series and compare it with the amount by which it exceeds 1. In this case, after 5 terms we have exceeded 1, meaning that the series diverges: Summation formula: Σ ai = a 0 + ( a 1 -1) + ( a 2 -1) + ( a 3 -1) + ... + ( a 5 -1) Thus, we have found that this series does not converge, as 5 is greater than 0 and less than or equal to 4 (which corresponds to the amount by which it exceeds 1). Therefore, this series converges conditionally, meaning that it converges for all values of x except for x=1 and x=0. ## RWKV-Eagle-Hermes-7B ## I would like to cook some Chinese food at home. Do you have any suggestions on the meal I could choose? Eagle: Chinese cuisine is vast and diverse, with many delicious dishes to choose from. Here are some popular options for a tasty meal at home: 1. Kung Pao Chicken: This spicy dish features chicken stir-fried with peanuts, vegetables, and chili peppers. It's a perfect blend of sweet, spicy, and savory flavors. 2. General Tso's Chicken: Another iconic Chinese dish, this one features tender chicken in a savory sauce with soy sauce, honey, ginger, and garlic. The sweetness of the sauce balances the spiciness of the chili peppers. 3. Mapo Tofu: A spicy and flavorful Sichuan specialty made with soft tofu in a spicy sauce made with chili bean paste, fermented black beans, and ground meat or shrimp. It's a bold dish that will have you hooked! 4. Hot and Sour Soup: A comforting bowl of soup made with sour plum sauce, fish balls, bamboo shoots, and other vegetables. It's perfect for a chilly day when you want something warm and filling. 5. Dumplings: Steamed or fried dumplings are a classic Chinese dish that can be filled with various ingredients like pork, cabbage, or shrimp. They're easy to make and always a crowd-pleaser! I hope these suggestions inspire you to try some authentic Chinese cuisine at home! ## I want to make something more special. What about red braised pork belly? Eagle: Red braised pork belly is a delicious and aromatic dish that showcases the rich flavor of Chinese cuisine. To make this dish at home, you'll need the following ingredients: Ingredients: - 1 pork belly (about 1.5 kg) - 2 tablespoons cooking oil - 4 cloves garlic, minced - 2-3 tablespoons Shaoxing wine (or dry sherry) - 2 tablespoons soy sauce - 2 tablespoons rice vinegar - 1 tablespoon honey - 1 tablespoon cornstarch - 1 teaspoon Sichuan peppercorns (or 1/2 teaspoon dried Sichuan peppercorns) - 2 bay leaves - Salt and pepper to taste Instructions: 1. Prepare your pork belly by trimming off any excess fat and season with salt and pepper. Set aside. 2. Heat the cooking oil in a large pot or Dutch oven over medium-high heat. Add the minced garlic and stir for a minute or two until fragrant. 3. Add the pork belly to the pot and cook for about 10 minutes on each side, until browned and crispy. Remove the pork belly from the pot and set aside. 4. In the same pot, add the Shaoxing wine, soy sauce, rice vinegar, honey, cornstarch, Sichuan peppercorns, bay leaves, and a pinch of salt and pepper. Bring the mixture to a boil over high heat, stirring occasionally to prevent sticking. 5. Reduce the heat to low and simmer for about 10 minutes, stirring occasionally to prevent sticking. The sauce should be thickened slightly. 6. Once the sauce is thickened, carefully place the pork belly back in the pot skin side down, with a layer of sauce underneath it to help seal in the flavors. 7. Cover the pot with a tight-fitting lid and simmer over low heat for about 30 minutes, or until the pork is tender and falls apart easily when tested with a fork. 8. Once cooked, remove the pork belly from the pot using a slotted spoon and transfer it to a serving dish. Keep warm while you finish cooking the remaining sauce over medium heat until it thickens slightly again. 9. Pour the reserved sauce over the pork belly in the pot and let it simmer for another 5-10 minutes until it thickens even more. This will help melt any remaining fat on top of the pork belly. 10. Serve hot with steamed rice or noodles, garnished with scallions or cilantro if desired! Enjoy your delicious red braised pork belly at home! ## Tell me the steps to cook. Eagle: Certainly! Here are the detailed steps to cook red braised pork belly at home: Ingredients: - 1.5 kg pork belly (about 1 large or 2 small) - 2 tablespoons cooking oil - 4 cloves garlic, minced - 2-3 tablespoons Shaoxing wine (or dry sherry) - 2 tablespoons soy sauce - 2 tablespoons rice vinegar - 1 tablespoon honey - 1 tablespoon cornstarch - 1 teaspoon Sichuan peppercorns (or 1/2 teaspoon dried Sichuan peppercorns) - 2 bay leaves - Salt and pepper to taste ## Instructions: 1. Preheat your oven to 200°C (400°F). 2. Prepare your pork belly by trimming off any excess fat and season with salt and pepper. Set aside. 3. Heat the cooking oil in a large pot or Dutch oven over medium-high heat. Add the minced garlic and stir for a minute or two until fragrant. 4. Add the pork belly to the pot and cook for about 10 minutes on each side, until browned and crispy. Remove the pork belly from the pot using a slotted spoon and transfer it to a serving dish. Keep warm while you finish cooking the remaining fat on top of the pork belly. 5. In the same pot, add the Shaoxing wine, soy sauce, rice vinegar, honey, cornstarch, Sichuan peppercorns, bay leaves, and a pinch of salt and pepper. Bring the mixture to a boil over high heat, stirring occasionally to prevent sticking. 6. Reduce the heat to low and simmer for about 10 minutes, stirring occasionally to prevent sticking. The sauce should be thickened slightly. 7. Once the sauce has thickened slightly again, carefully place the pork belly back in the pot skin side down, with a layer of sauce underneath it to help seal in the flavors. Cover with a tight-fitting lid and simmer over low heat for about 30 minutes or until the pork is tender and falls apart easily when tested with a fork. 8. Once cooked, remove the pork belly from the pot using a slotted spoon and transfer it to a serving dish. Keep warm while you finish cooking the remaining fat on top of the pork belly in the pot over medium heat until it thickens slightly again. This will help melt any remaining fat on top of the pork belly. 9. Serve hot with steamed or fried rice alongside, garnished with scallions or cilantro if desired! Enjoy your delicious red braised pork belly at home! ## What do I need to be careful with the most during the cooking process? Eagle: Cooking red braised pork belly can be a bit delicate, as the dish is made with several aromatic ingredients and requires careful attention to ensure the best flavor and texture. Here are some tips to keep in mind during the cooking process: 1. Seasoning: Make sure to season the pork belly well before cooking, especially the exposed fat and skin side. This will help enhance the flavors and ensure even browning. 2. Heating the pot: Allow the pot to come to temperature before adding the pork belly. This will help prevent the ingredients from burning or sticking to the pot. 3. Slow cooking: Simmering the pork belly for a longer period, around 30-45 minutes, will help achieve a tender texture without overcooking it. Be sure to stir occasionally to prevent sticking. 4. Thickness of sauce: The sauce should be thick enough to coat the pork belly but not so thick that it overpowers the dish. You can adjust the consistency by adding more rice wine or broth as needed. 5. Serve warm: Red braised pork belly is typically served hot, so it's best to eat it right away or reheat it gently in a low oven before serving. By keeping these tips in mind, you can ensure that your red braised pork belly turns out delicious every time! ## Tell me more about the proper heat and timing. Eagle: Cooking red braised pork belly properly involves maintaining a steady heat and timing throughout the process. Here's a general guideline for the heat and timing you can follow: 1. Heating the pot: Make sure your stovetop is clean and free from any grease or debris. Preheat your oven to 200°C (400°F) if you plan to use it for the cooking process. If not, preheat a dry skillet over medium heat. 2. Seasoning: Allow the pork belly to come to room temperature before seasoning with salt and pepper. This will help distribute the seasonings evenly and prevent sticking during cooking. 3. Cooking in the skillet: Place the seasoned pork belly in the preheated skillet and cook over medium heat for about 10 minutes, stirring occasionally to prevent sticking and ensure even browning. If needed, adjust the heat to maintain a steady simmer. 4. Transferring to the pot: Once the pork belly is browned on one side, remove it from the skillet using a slotted spoon and transfer it to a large pot (preferably one with a tight-fitting lid). Be careful when transferring the pork belly to avoid spills or burns. 5. Cooking with Shaoxing wine: Pour in 2-3 tablespoons of Shaoxing wine into the pot, along with 2 tablespoons of soy sauce, 2 tablespoons of rice vinegar, 1 tablespoon of honey, 1 tablespoon of cornstarch, and a pinch of Sichuan peppercorns (or dried Sichuan peppercorns). Stir well to combine all ingredients. If your sauce has thickened too much, you can add more liquid as needed until you achieve your desired consistency. 6. Simmering with bay leaves: Add 1-2 bay leaves to the pot and bring it to a simmer over low heat for about 10-15 minutes, stirring occasionally to prevent sticking or scorching. 7. Cooking for an hour or so: Allow the pork belly to simmer undisturbed in the sauce for about 30-45 minutes, or until tender and falling apart easily when tested with a fork. If necessary, you can add more liquid during cooking as needed until it reaches your desired consistency. 8. Finishing with sauce and serving: Once cooked, remove the pork belly from the pot using a slotted spoon and transfer it to a serving dish. Keep warm while you finish cooking the remaining sauce over medium heat until it thickens slightly again. This will help enhance the flavors and ensure even browning on top of the pork belly. Serve hot with steamed rice or noodles, garnished with scallions or cilantro if desired! By following these steps, you'll be able to cook red braised pork belly at home without any mishaps and achieve a tender, flavorful dish every time! ## I see. Thank you for your advice. You're welcome! I'm glad I could help. Let me know if you have any more questions or need further assistance. ## RWKV-Eagle-Hermes-7B ## If all apples are fruits and all fruits are sweet, are all apples sweet? Please explain your answer in detail. Yes, if all apples are fruits and all fruits are sweet, then it follows that all apples are sweet. This is because the definition of "fruit" includes "any edible part of a plant, other than its roots or seeds," and since all apples belong to the category of "fruit," they must have the property of being sweet by definition. ## If birds can fly. Penguins are birds. Can penguins fly? Please explain your answer in detail. Yes, penguins can fly. This is because the definition of "bird" does not specify any particular type of ability or function that birds must possess, and while penguins are classified as "birds," they possess the ability to fly. Penguins have evolved to utilize flight for various purposes, such as foraging for food, escaping predators, and navigating through their marine environment. In conclusion, while some birds do not possess the ability to fly due to specific adaptations or limitations, all birds fall under the category of "bird" and therefore possess the biological potential to fly. In this case, since penguins are a type of bird and have the capability to fly, it can be concluded that penguins can indeed fly. ## The day before two days after the day before tomorrow is Saturday. What day is it today? Please explain your answer in detail. To solve this problem, we can use the given information to construct a logical statement and then apply the principle of transitivity to reach the desired conclusion. Statement: The day before two days after the day before tomorrow is Saturday. Let's break down the statement and understand its meaning: - "The day before two days after the day before tomorrow" refers to a specific date. We need to determine which day this is. - "is Saturday" tells us that this date is indeed Saturday. Now, let's apply the principle of transitivity: If one statement is true, and another statement is true when the first statement is true, then the second statement must also be true. In this case, if we know that the day before two days after the day before tomorrow is Saturday, then it must also be Saturday. Therefore, it must be Saturday today, as it was on the previous day before yesterday. ## If five cats can catch five mice in five minutes, how long will it take one cat to catch one mouse? Please explain your answer in detail. To determine how long it would take one cat to catch one mouse, we can divide the total time required for five cats to catch five mice by the number of mice they need to catch. In this case, it would take five minutes to catch five mice, so: Time to catch one mouse = Total time / Number of mice = 5 minutes / 1 mouse = 5 minutes. So, it would take one cat 5 minutes to catch one mouse. ## RWKV-Eagle-Hermes-7B ## Summarize the following paragraph. Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence length. In contrast, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to match the same performance as Transformers due to limitations in parallelization and scalability. We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. Our approach leverages a linear attention mechanism and allows us to formulate the model as either a Transformer or an RNN, thus parallelizing computations during training and maintains constant computational and memory complexity during inference. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers, suggesting future work can leverage this architecture to create more efficient models. This work presents a significant step towards reconciling trade-offs between computational efficiency and model performance in sequence processing tasks. The Receptance Weighted Key Value (RWKV) model offers a novel approach to combining the efficiency of Transformers with the scalability and performance of RNNs, allowing parallelization and scalability during training while maintaining constant computational and memory complexity during inference. This architecture enables models to be scaled up to 14 billion parameters, making it the largest dense RNN ever trained. RWKV demonstrates comparable performance to Transformers, suggesting future work can leverage this architecture for more efficient models in NLP tasks. ## Please further summarize it in one sentence. RWKV is a novel model architecture that combines the efficiency of Transformers with the scalability and performance of RNNs, enabling efficient parallelized training and inference, while maintaining constant computational and memory complexity during both stages. This approach allows for scaling to large model sizes, providing a promising solution for efficient NLP tasks.