# A Survey of RWKV
**Authors**: Zhiyuan Li , Tingyu Xia, Yi Chang and Yuan Wu
> Zhiyuan Li and Tingyu Xia are equal contributions. E-mail: Yuan Wu is the corresponding author. E-mail: All authors are with School of Artificial Intelligence, Jilin University. Yi Chang is with Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China E-mail:
## Abstract
The Receptance Weighted Key Value (RWKV) model offers a novel alternative to the Transformer architecture, merging the benefits of recurrent and attention-based systems. Unlike conventional Transformers, which depend heavily on self-attention, RWKV adeptly captures long-range dependencies with minimal computational demands. By utilizing a recurrent framework, RWKV addresses some computational inefficiencies found in Transformers, particularly in tasks with long sequences. RWKV has recently drawn considerable attention for its robust performance across multiple domains. Despite its growing popularity, no systematic review of the RWKV model exists. This paper seeks to fill this gap as the first comprehensive review of the RWKV architecture, its core principles, and its varied applications, such as natural language generation, natural language understanding, and computer vision. We assess how RWKV compares to traditional Transformer models, highlighting its capability to manage long sequences efficiently and lower computational costs. Furthermore, we explore the challenges RWKV encounters and propose potential directions for future research and advancement. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/RWKV-Survey.
Index Terms: RWKV, Large Language Model, Foundation Models, Deep Learning.
## 1 Introduction
As an important branch of machine learning, deep learning originated from the perceptron model [1], which laid the foundation for the development of neural networks. Over the past decade, deep learning has rapidly advanced and been widely applied, profoundly impacting various fields. Classic deep learning models, including Multilayer Perceptron (MLP), Convolutional Neural Networks (CNN), and Recurrent Neural Networks (RNN), have achieved significant success in areas such as image recognition [2, 3, 4], speech recognition [5, 6, 7], and natural language processing [8, 9]. For example, CNNs have become central to computer vision [10, 11, 12], with applications ranging from facial recognition to automatic driving [13, 14, 15]. Meanwhile, RNNs and their variants, particularly long-short-term memory (LSTM) networks [16], have proven essential for natural language processing tasks, especially in speech recognition and machine translation [17, 18, 19].
Proposed by Vaswani et al. in 2017, the Transformer model [20] has since become a dominant architecture in natural language processing (NLP) [21, 22, 23] and various other fields [24, 25, 26]. Transformer-based models such as BERT [27] and GPT [28] have demonstrated exceptional performance in tasks such as text classification [29], language generation [30], and question answering [31]. The Transformer utilizes a self-attention mechanism to adeptly capture long-distance dependencies in input sequences, providing a more robust approach for modeling intricate data relationships than earlier models like RNNs and LSTMs [32]. Furthermore, the parallelization of both training and inference processes enhances the scalability of the Transformer, significantly improving processing speed and model performance. However, despite these advantages, the Transformer faces significant challenges. The computational cost of the attention mechanism grows quadratically with the sequence length, leading to inefficient memory usage and slower processing of long sequences [33]. This issue becomes particularly problematic in tasks that require handling large amounts of data. Despite numerous endeavors to explore methods for reducing the complexity of the Transformer model, unfortunately, significant progress in substantially decreasing its complexity has yet to be achieved [34, 35, 36].
To address the limitations of the Transformer model, recent advancements have introduced state-space models [37], which provide a promising solution for efficiently handling long sequences while maintaining strong contextual understanding. Models such as Mamba [38] and RetNet [39] have been proposed to reduce computational complexity in sequence data processing. Mamba aims to enhance the efficiency of long-sequence processing by modifying the attention mechanism and reducing memory usage in the process. Similarly, RetNet employs a recurrent structure to improve memory retention and extend the model’s ability to capture long-range dependencies, all while avoiding the quadratic time complexity that affects traditional Transformers. One of the most notable innovations in recent times is the Receptance Weighted Key Value (RWKV) model [40], which effectively combines the strengths of RNNs with the attention-based mechanism of Transformers. The RWKV model utilizes a unique key-value approach to manage dependencies sequentially, significantly reducing computational overhead while preserving the ability to model long-term dependencies with minimal memory requirements. RWKV has already demonstrated its potential across a wide range of applications, from NLP tasks such as text generation [41], machine translation [42], and sentiment analysis to time series prediction tasks [43].
Inspired by the exceptional efficiency of RWKV in handling long sequences and its growing influence in various fields, there has been a surge in research exploring its capabilities and potential applications. Given the increasing interest in RWKV, this paper aims to provide a comprehensive overview of the model, its evolution, and future directions. As depicted in Figure 1, the remainder of this paper is organized as follows: Section 2 presents a systematic review of foundational concepts, covering RNNs, Transformer architectures, and Attention-Free Transformer (AFT). Section 3 delves into the principles and implementations of RWKV, providing a comparative analysis with Transformer and Mamba, along with a detailed discussion of its various implementations. Section 4 thoroughly explores various applications of RWKV, including Natural Language Generation (NLG), Natural Language Understanding (NLU), other NLP tasks, computer vision, web applications, and evaluation methods. Finally, Section 5 provides the key challenges RWKV faces and its future research directions.
for tree= grow=east, reversed=true, anchor=base west, parent anchor=east, child anchor=west, base=left, font=, rectangle, draw, rounded corners,align=left, minimum width=2.5em, inner xsep=4pt, inner ysep=1pt, , where level=1text width=5em,fill=blue!10, where level=2text width=5em,font=,fill=pink!30, where level=3font=,yshift=0.26pt,fill=yellow!20, [A Survey of RWKV,fill=green!20 [Background (Sec. 2),text width=9em [Recurrent Neural Networks,text width=11em] [Transformers,text width=5em] [Attention Free Transformer,text width=11em] ] [RWKV (Sec. 3),text width=6em [RWKV-4,text width=4em] [RWKV-5(Eagle),text width=6em] [RWKV-6(Finch),text width=6em] [Compare RWKV with Transformer or mamba,text width=9em] [Various Implementations,text width=7em [Official Version [44] / ChatRWKV [45] / C [46] [47] / C++ [48] [49] [50] Rust [51, 52, 53, 54, 55] / TensorFlow [56, 57, 58, 59, 60, 61, 62] Fortran [63] / Julia [64] / Zig [65] / Go [66, 67, 68, 69, 70, 71] Keras [72, 73] / PaddlePaddle [74, 75] / JAX [76, 77, 78] Node.js [79, 80] / MLX [81] / LoRA [82, 83] Others [84] [85] [86] [87] [88] [89] [90] [91] [92]] ] ] [Applications of the RWKV Model(Sec. 4),text width=9em [Natural Language Generation,text width=7em [Chatbot [93, 94, 95, 96, 97, 98, 99, 100, 101, 102] Role-playing [103, 104] / Writing robot [41] Others [105] [106] [107] [108]] ] [Natural Language Understanding,text width=7em [Virtual Assistant [109] / PDF Query Systems [110] Machine Translation [42] / OnlySports [111] Classification [112] / Write novel [113] Others [114, 115, 116, 117, 118]] ] [Other NLP tasks,text width=4em [CLOPPS [119] / MSRWKV-2DTCN [43] / Stock Price Prediction [120, 121, 122] RWKVCC [123] / Code Completion [124] / Mod-RWKV [125] / GoldFinch [126] RWKV-TS [127] / TIM [128] / RRWKV [129] / LKPNR [130] Decision-RWKV [131] / ProSG [132] / SpikeGPT [133] Others [134] [135] [136] [137] ] ] [Computer Vision,text width=4em [BSBP-RWKV [138] / Restore-RWKV [139] / RWKV-based MIL [140] / LION [141] PointRWKV [142] / OccRWKV [143] / TLS-RWKV [144] / RuleVM [145] LineRWKV [146] / RWKV-SAM [147] / Vision-RWKV [148] / VisualRWKV [149] LSTM CrossRWKV [150] / RWKV-CLIP [151] / SDiT [152] / Social-CVAE [153] Diffusion-RWKV [154] / RWKV-IR [155] / StyleRWKV [156] Others [157] [158]] ] [Audio and Music Tasks,text width=5em [Android RWKV MIDI [159] / RWKV-v4 MIDI [160] / RWKV-v4 music models [161] Music-Diff [162] / Voice Activity Detection [163] / RWKV-Transducer [164] RWKV-ASR [165] / AudioRWKV [166] / RWKV-MusicGenerator [167] Music-Genre-Classification-RWKV [168] RWKV-SpeechChat [169]] ] [Web Application,text width=5em [MoZi-RWKV [170] / RWKV-3B-7B-Webui-GPT-SoVITS [171] / AI00-Server [172] RWKV-v4-Web [173] / RWKV-Role-Playing [174] / RWKV-QQBot-BackEnd [175] Web-RWKV-Axum [176] / WebChatRWKVstic [177] / Chatux-Server-RWKV [178] ChatRWKV-Flask-Api [179] / ChatRWKV-Webui [180] / RWKV-Api [181] RWKV.cpp-Webui-Macos [182] / RWKV-Role-Playing-UI [183] RWKV.cpp-Webui-GPT-SoVITS [184] ] ] [Evaluation of RWKV Models,text width=6em [REGBENCH [185] / LongctxBench [186] / LooGLE [187] / RULER [188] S3EVAL [189] / MAGNIFICO [190] / MANGO [191] / SuperCLUE [192] Zhujiu [193] / BIPIA [194] / CoDI-Eval [195] / VasE [196] / CMATH [197] Head-to-Tail [198] / PRE [199] / Freshbench [200] / LongICLBench [201] LongBench [202] / Long Range Arena [203] / MT-bench [204] Others [205] [206] [207] [208] [209] [210] [211] [212] ] ] [Others,text width=3em [Spiking Mixer [213] / MiniRWKV-4 [214] / RWKV-RAG [100] VisualRWKV [215] / Dlip-RWKV [216] / RWKV-LM-EXT [217] RWKV-Reward-Enhanced [218] / RWKV-Twitter-Bot-Detection [219] UnifiedImplicitAttnRepr [220] / RWKV-PEFT [221] [222] / RWKV5-InfctxLM [223] DecisionRWKV [224] / RWKV-Nonogram [225] / RWKV-RUL [226] TrainChatGalRWKV [227] / TinyRWKV [228] / Simple-RWKV [229] RWKV-Runner [230] / Llama-node [231] / Godot-rwkv [232] RWKVRaven-Alpaca [233] / RWKV-Reward [234] / Decision-RWKV [235] RWKV-LM-RLHF [236] / RWKV-LM-State [237] / MARWKV [238] RWKV-Finetune [239] / RWKVWorld-LoraAlpaca [240] RWKV-LM-LoRA [241] / RWKV-Light-API [242] / RWKV-StateTuning [243] ChunkRWKV6 [244] / RWKV-LM-Interpretability [245] web-RWKV-Inspector [246] / RWKV-Umap [247] / RWKV-Tensorrt [248] LLMFarm [249] / RWKV-Chains [250] / RWKV-Ouroboros [251] AltaeraAI [252] / RWKVinLLAMA [253] / GPTQ-for-RWKV [254] RWKV-LM-AMD-Radeon-ROCm [255] / InnerDNN [256] / EasyChat-Server [257] ChatRWKV-PC [258] / Generative Calibration [259] / RWKV- DPA [260] RWKV-edge [261] / PCF-RWKV [262] / L3TC [263] /PlantFlowerDatasets [264]] ] ] [Future Directions and Challenges (Sec. 5),text width=9em [Expanding RWKV’s Capabilities in Future Applications,text width=12em [(1) Long sequence processing capability (2) Multi-modal and Cross-modal Learning (3) Parameter-efficient Finetuning] ] [Security and Stability,text width=5em [(1) Safety&Robustness (2) Bias and fairness (3) Privacy] ] [Optimizing RWKV for Hardware Acceleration,text width=9em ] ] ]
Figure 1: Structure of this paper.
## 2 Background
### 2.1 Recurrent Neural Networks (RNNs)
Recurrent Neural Network (RNN) [265] is a powerful tool for processing sequence data. It is capable of internally maintaining a state, which enables it to capture the dynamic features of time series data. The fundamental concept underlying RNN is to leverage the dependency among consecutive elements within a sequence and transmit information via cyclic connections. This allows the network to retain prior information and subsequently utilize the prior information for further computations, thereby endowing it with the ability to handle sequential data in a more effective and contextually aware manner.
The working principle of RNN can be described by the following mathematical model:
$$
\displaystyle h_t=\tanh(W_hh· h_t-1+W_hx· x_t+b_h) \displaystyle y_t=W_ho· h_t+b_o \tag{1}
$$
Where $h_t$ represents the hidden layer state at time $t$ , $x_t$ represents the input at time $t$ , and $y_t$ represents the output at time $t$ . These two equations demonstrate the process of merging the current input with the preceding hidden state to derive the present hidden state. Subsequently, they also illustrate how this hidden state is transformed into an output.
While RNN models are effective at handling time series data, they struggle with recognizing long-term dependencies. This issue arises primarily due to the phenomena of vanishing or exploding gradients. In RNN models, the issue of gradient disappearance arises when gradients are multiplied by the weight matrix across time steps. When the eigenvalue of this weight matrix is less than 1, it results in the exponential diminishment of gradient values over time, leading to vanishing gradients. Conversely, when the eigenvalue exceeds 1, the gradient values increase exponentially with time, resulting in gradient explosion.
The structure of RNN models imposes a restriction on their capability to learn long-term dependencies. In typical RNN models, each time step’s hidden state is influenced solely by the hidden state from the preceding time step. Consequently, RNN models struggle to effectively capture and understand dependencies that span across extended time periods.
In order to solve the problem of gradient disappearance or gradient explosion that may occur when RNN processes long sequences, researchers have proposed several variants of RNN: Long short-term memory network (LSTM) [16]: By introducing a gating mechanism to control the flow of information, it can capture long-term dependencies. Gated recurrent unit (GRU) [266]: The structure is simpler than LSTM, but it can provide similar performance. Bidirectional recurrent neural network (Bi-RNN) [267]: By processing the forward and reverse information of the sequence at the same time, it can more comprehensively understand the patterns in the sequence.
Recently, with increasing attention on overcoming the scalability limitations of Transformer-based architectures, the potential of RNN-based variants has garnered renewed interest. Models such as minLSTMs and minGRUs [268], xLSTM [269], and Aaren [268] show that addressing the limitations of traditional RNNs can make these models perform as well as or even better than Transformers in various tasks. These developments highlight the enduring significance and promising potential of RNN-based approaches in advancing modern deep learning.
### 2.2 Transformers
In RNNs, the computation of each time step is based on the output of the previous step, which requires sequential processing and prohibits parallel computation. Conversely, the Transformer architecture employs the Self Attention mechanism to process all time steps simultaneously, enabling parallel computation. Unlike conventional recurrent neural network models like LSTM and GRU, the Transformer [20] eschews the recursive architecture entirely. Instead, it leverages self-attention to establish dependencies across all positions within the input sequence, thereby enhancing both training speed and efficiency.
Attention mechanisms have the capability to identify and record the relationships across all input and output tokens. In Self-Attention, the input is denoted by the matrix $x$ . This undergoes a linear transformation using the matrices $W_q$ , $W_k$ , and $W_v$ , allowing the calculation of the Query $Q$ , Key $K$ , and Value $V$ . It is important to highlight that each row within the matrices $x$ , $Q$ , $K$ , and $V$ corresponds to an individual word.
$$
\displaystyle Q=x· W_q,K=x· W_k,V=x· W_v \tag{3}
$$
where $W_q$ , $W_k$ , $W_v$ are the trainable parameters. After obtaining the matrices $Q$ , $K$ , $V$ , it can calculate the output of Self-Attention. The calculation formula is shown as follows:
$$
\displaystyle Attn(Q,K,V)=softmax(\frac{QK^T}{√{d_k}})V \tag{4}
$$
where $d_k$ is the dimension of the key vectors. It scales the result by $√{d_k}$ in order to prevent the inner product from being too large.
In a single Attention mechanism, a singular representation space is derived. When utilizing multiple attentions, it is possible to derive several distinct representation spaces. This concept leads to the development of Multi-Head Attention by integrating several Self-Attentions. Each Attention employs unique weight matrices for Query, Key, and Value, with these matrices initialized in a random manner. Training subsequently projects word embeddings into diverse representation spaces. The mathematical expression representing the multi-head attention mechanism is as follows:
$$
\displaystyle Q_i=x· W_q_{i},K_i=x· W_k_{i},V_i=x· W_v_
{i}, \displaystyle head_i=Attention(Q_i,K_i,V_i),i∈[1,n], \tag{5}
$$
where $n$ means the number of attention heads, $Concat$ means concatenate and $W_o$ is the projection matrix which is a trainable parameter. Multi-head attention enables the simultaneous processing of input sequences from various perspectives, enhancing the model’s capacity to comprehend and identify intricate dependencies. This capability substantially boosts the Transformer model’s effectiveness in handling natural language processing tasks as well as other types of sequence data processing.
### 2.3 Attention Free Transformer (AFT)
The attention mechanism in Transformers excels at capturing long-term dependencies and emphasizing pertinent elements within the input sequence. Nevertheless, the self-attention mechanism demands matrix multiplication involving the transpose of the query and key. This operation exhibits quadratic space complexity relative to sequence length, rendering it inefficient for handling extended sequences effectively. Attention Free Transformer (AFT) [270], represents a Transformer variant that does not rely on matrix multiplication. This change leads to a reduction in memory complexity—from $O(T^2d+Td)$ to $O(Td)$ , as noted by [40] —and a decrease in time complexity—from $O(T^2d)$ to $O(Tsd)$ , according to [40]. Here, $T$ denotes the sequence length, $d$ stands for the embedding dimension, and $s$ represents the size of a local window specific to AFT. Two primary AFT variants exist: AFT-full and AFT-local. The latter, AFT-local, arose from the author’s observation that attention typically exhibits pronounced local patterns; thus, utilizing a reduced attention window was deemed advantageous.
Same as Transformer, for the input feature $x$ , AFT-full uses three different linear layers to calculate Query $Q$ , Key $K$ , and Value $V$ , $Q∈ℝ^T× d$ , $K∈ℝ^T× d$ , $V∈ℝ^T× d$ , where $T$ is the maximum length of the sequence, $d$ is the number of hidden layer nodes. Unlike Transformer, AFT adds a learnable positional encoding $w_t^\prime,t$ to $K_t^\prime$ , where $1<t^\prime<T$ . And use softmax to normalize $K$ with position encoding.
$$
\displaystyleWeighted(K_t^\prime)=\frac{\exp(K_t^\prime+w_t,t^\prime)}{∑_t^\prime=1^T\exp(K_t^
\prime+w_t,t^\prime)} \tag{8}
$$
Subsequently, the sigmoid activation function is applied to normalize $Q_t$ , resulting in the weight matrix $α_t^\prime$ , which is derived by conducting a dot multiplication with $Weighted(K_{t^\prime})$ :
$$
\displaystyleα_t^\prime \displaystyle=σ_Q(Q_t)\odotWeighted(K_t^\prime) \displaystyle=σ_Q(Q_t)\odot\frac{\exp(K_t^\prime+w_t,t^\prime)}{∑_t^\prime=1^T\exp(K_t^\prime+w_t,t^\prime
)} \tag{9}
$$
Finally, the AFT-full’s attention mechanisms $Attn(Q,K,V)_t=Y_t$ is calculated as follow:
$$
\displaystyleY_t \displaystyle=∑_t^\prime=1^Tα_t^\primeV_t^\prime \displaystyle=∑_t^\prime=1^Tσ_Q(Q_t)\odot\frac{\exp
(K_t^\prime+w_t,t^\prime)}{∑_t^\prime=1^T\exp(
K_t^\prime+w_t,t^\prime)}\odotV_t^\prime \displaystyle=σ_Q(Q_t)\odot\frac{∑_t^\prime=1^T\exp(
K_t^\prime+w_t,t^\prime)\odot V_t^\prime}{∑_t^\prime=1^T
\exp(K_t^\prime+w_t,t^\prime)} \tag{11}
$$
Analyzing equation 13 along with the implementation of AFT-full reveals that the complexity of AFT-full is $O(T^2d)$ , similar to that of the Transformer. However, the distinct feature of AFT-full is that this complexity arises from the incorporation of $w$ into the computations, which allows for modifying the style of $w$ in order to enhance AFT’s speed. The authors of AFT introduced AFT-local as a refinement. In this variant, any position embedding values that extend beyond the window $s$ , with the condition $s≤ T$ , are assigned a value of zero.
$$
\displaystyle≤ft.w_t,t^\prime=≤ft\{\begin{array}[]{ll}w_t,t^\prime
,&if~{}|t-t^\prime|<s\\
0,&otherwise\end{array}\right.\right. \tag{16}
$$
AFT-local assigns a value of 0 to the position embedding outside the window. However, since these zeroed values remain part of the entire computational process, there is no reduction in the computational load. Thus, the time complexity for AFT-local remains $O(Tsd)$ .
## 3 RWKV
The presence of a cyclical architecture in RNNs [265] is responsible for their elevated computational complexity and reduced training efficiency. This architecture also makes them susceptible to issues such as gradient vanishing and gradient exploding, thereby limiting the scalability of RNNs. In contrast, the Transformer [20] employs a self-attention mechanism, enabling it to handle entire sentences at once. This allows for improved training efficiency and makes parallel processing possible. Nonetheless, Transformers are characterized by high computational complexity, with their time complexity being $O(T^2d)$ , making them less suitable for handling long sequences efficiently. Grounded in the classic Transformer framework, AFT [270] replaces matrix multiplication with a technique akin to Residual Connection. This change allows AFT to bypass the attention mechanism, thereby lessening the space requirements. Consequently, AFT is capable of computing the attention weight with substantially reduced space complexity compared to the Transformer model; however, it does not achieve major improvements in reducing time complexity.
RWKV [40] was developed in response to the challenges posed by RNNs, Transformers, and AFT issues, primarily focusing on enhancing the linear attention mechanism. This innovation aims to address the difficulty of parallelizing RNNs. The model maintains a time complexity comparable to that of RNNs while achieving performance effects akin to Transformers.
Currently, the official versions of RWKV mainly include three types: RWKV-4, RWKV-5 (Eagle), and RWKV-6 (Finch). Among them, the commonly used version is RWKV-4. In addition, the latest RWKV-7 (Goose) has released a preview version and the corresponding code, but the specific document has not yet been made public yet https://www.rwkv.com .
### 3.1 RWKV-4
RWKV-4 [40] represents the first version officially made available to the public, with RWKV-1, RWKV-2, and RWKV-3 existing as experimental predecessors. Referring to figure 2, the architecture of the RWKV-4 model includes a series of stacked residual blocks. Each of these blocks contains sub-blocks dedicated to time-mixing and channel-mixing, which efficiently leverage previous information through an incorporated recursive framework.
<details>
<summary>x1.png Details</summary>
### Visual Description
## Diagram: Transformer Neural Network Architecture
### Overview
The diagram illustrates a multi-layer Transformer architecture with two primary processing pathways: **Channel Mixing** and **Time Mixing**. The model processes input embeddings through stacked layers (N×), incorporating residual connections, layer normalization, and attention mechanisms.
### Components/Axes
1. **Input Embedding**: Starting point for the data flow.
2. **LayerNorm**: Layer normalization applied after each mixing block.
3. **Add**: Residual connections (skip connections) for gradient stability.
4. **Channel Mixing Block**:
- **R'**, **K'**, **V'**: Modified attention components (likely for channel-wise operations).
- **σ (sigma)**: Non-linear activation function.
- **μ (mu)**: Mean or bias term.
5. **Time Mixing Block**:
- **R**, **K**, **V**: Attention components for temporal operations.
- **WKV**: Weighted key-value mechanism (possibly a variant of attention).
6. **Softmax**: Final output normalization for classification.
7. **Out**: Final output layer.
### Detailed Analysis
- **Channel Mixing**:
- Inputs **R'** and **K'** are combined via cross-attention (⊗ symbol), modulated by **σ**.
- **V'** is derived from the interaction, with **μ** added as a bias.
- **Time Mixing**:
- **R**, **K**, **V** are processed through **WKV**, with **σ** applied to the output.
- Residual connections (Add) merge outputs from both mixing blocks.
- **Stacked Layers (N×)**:
- Each layer repeats the **Channel Mixing** → **Time Mixing** → **Add** → **LayerNorm** sequence.
- Final output is passed through **Softmax** for probabilistic classification.
### Key Observations
- **Residual Connections**: Enable deep network training by mitigating vanishing gradients.
- **Layer Normalization**: Stabilizes training by normalizing activations.
- **Dual Mixing Pathways**: Suggests a hybrid approach combining spatial (channel) and temporal (time) attention.
- **WKV Block**: Likely a specialized attention mechanism for temporal dependencies.
### Interpretation
This architecture combines **channel-wise** and **temporal** attention mechanisms, typical of modern Transformers (e.g., Vision Transformers or Time Series Transformers). The **Channel Mixing** block processes spatial features, while **Time Mixing** handles sequential dependencies. Residual connections and layer normalization ensure efficient training. The **WKV** block may optimize memory usage compared to standard attention. The final **Softmax** layer indicates a classification task, though the exact application (e.g., NLP, computer vision) depends on the input embedding type.
**Note**: No numerical data or uncertainty ranges are present in the diagram. The structure emphasizes modular design and multi-path processing for complex data.
</details>
Figure 2: The structure of the RWKV model consists of stacked residual blocks, where each block is made up of a time-mixing sub-block and a channel-mixing sub-block, incorporating recurrent elements to capture past information.
The time-mixing block is designed for global interaction, akin to how self-attention functions in a Transformer. Initially, the term ”RWKV” is composed of the following four letters:
- $R$ : Receptance vector, responsible for gauging the amount of past information that is permitted;
- $W$ : Weight vector, includes a position weight decay vector and contains the model’s trainable parameters;
- $K$ : Key vector, analogous to the $K$ in the traditional attention mechanism;
- $V$ : Value vector, analogous to the $V$ present in the traditional attention mechanism.
For time $t$ , given word $x_t$ and the previous word $x_t-1$ , the Time-Mix module formulas are as follows:
$$
\displaystyle r_t \displaystyle=W_r·(μ_r\odot x_t+(1-μ_r)\odot x_t-1), \displaystyle k_t \displaystyle=W_k·(μ_k\odot x_t+(1-μ_k)\odot x_t-1), \displaystyle v_t \displaystyle=W_v·(μ_v\odot x_t+(1-μ_v)\odot x_t-1), \displaystyle wkv_t \displaystyle=\scalebox{1.0}{$\frac{∑_i=1^t-1exp(-(t-1-i)w+k_i)\odot v
_i+exp(u+k_t)\odot v_t}{∑_i=1^t-1exp(-(t-1-i)w+k_i)+exp(u+k_t
)}$} \displaystyle o_t \displaystyle=W_o·(σ(r_t)\odot wkv_t) \tag{17}
$$
The terms $r_t$ , $k_t$ , and $v_t$ utilized in this context are analogous to the $Q$ , $K$ , and $V$ components found in AFT or Transformer architectures. A notable distinction in the computation of $R$ , $K$ , and $V$ as compared to the Transformer model is that the input $x$ is not merely the embedding of the current token. Instead, it represents the weighted sum of the current token’s embedding and that of the preceding token. Additionally, the computation of $wkv_t$ is linked to the attention mechanism’s implementation. This implementation represents a linear interpolation utilizing both historical and present moment data, and it is important to note its exponential nature. The relationship between the current token and all preceding tokens is characterized by an exponential decay sum. This specific aspect endows $wkv_t$ with the features of linear attention. Notably, there exist two primary distinctions between equation 20 and equation 21 as well as AFT’s equation 13:
1. AFT modifies the bias $w_t^\prime,t$ from being based on absolute position to being based on relative position, resulting in the need to train only a single parameter $w$ vector.
2. The parameter $u$ is introduced to handle the current position independently within the process.
The RWKV model employs Time Mixing to capture the relationship between tokens, and the Channel Mixing module models RWKV by exploring the dimensions of the hidden layer that correspond to individual tokens. The channel-mixing block utilizes equations 17 and 18 again to determine a fresh set of $r_t^\prime$ and $k_t^\prime$ using the result from the time-mixing block. Subsequently, the final output is computed in the following manner:
$$
\displaystyle r_t^\prime \displaystyle=W_r^\prime·(μ_r^\prime\odot x_t+(1-μ_r^
\prime)\odot x_t-1), \displaystyle k_t^\prime \displaystyle=W_k^\prime·(μ_k^\prime\odot x_t+(1-μ_k^
\prime)\odot x_t-1), \displaystyle o_t^\prime \displaystyle=σ(r_t^\prime)\odot(W_v^\prime·\max(k_t^
\prime,0)^2). \tag{22}
$$
Channel Mixing conducts integration within the feature dimension. Given that the feature vector has a dimension of $d$ , each element in these dimensions must gather data from other dimensions to refresh its own value. In this context, each feature vector dimension is referred to as a ”channel”.
The RWKV architecture combines the strengths of both Transformers and RNNs. Unlike conventional RNNs, it offers the benefit of having a stable gradient conducive to deeper architecture, similar to that of Transformers. Additionally, RWKV excels in reasoning efficiency. RWKV redefines the attention framework by adopting linear attention, substituting the usual dot-product interaction between tokens with a more efficient attention directed at channels. This adaptation results in a computational complexity of $O(Td)$ and decreases the memory complexity down to $O(d)$ .
### 3.2 RWKV-5 (Eagle)
The sequential models Eagle (RWKV-5) and Finch (RWKV-6) [271] have been developed as advancements of the RWKV (RWKV-4) architecture. In these models, enhancements have been incorporated into the architecture design, such as multi-headed matrix-valued states and dynamic recursive mechanisms, which bolster the models’ expressive capabilities while preserving the inference efficiency typical of RNNs. Compared with RWKV-4, RWKV-5 and RWKV-6 mainly made changes to the time mixing block.
Eagle (RWKV-5) [271] enhances the architecture and learning decay strategy derived from RWKV-4 [40] through the adoption of expressive multi-head matrix-valued states rather than the traditional vector-valued states. It also includes a reconfiguration of receptive states and incorporates supplementary gating mechanisms.
Eagle introduced a new notation $lerp$ , which means the linear interpolation between $x_t$ and $x_t-1$ used in RWKV-4 and it is defined as:
$$
\displaystyle_\square(a,b)=a+(b-a)\odotμ_\square \tag{25}
$$
where each $μ_\square∈ℝ^D$ is a learnable vector. The formula of Eagle Time Mixing can be written as follows:
$$
\displaystyle\square_t=_\square(x_t,x_t-1)
\boldsymbol{W}_\square, \square∈\{r,k,v,g\} \displaystyle w=\exp(-\exp(ω)) \displaystyle\boldsymbol{w}\boldsymbol{k}\boldsymbol{v}_t=(u)· k_t^T· v_t \displaystyle +∑_i=1^t-1(w)^t-1-i
· k_i^T· v_i∈ℝ^(D/h)×(D/h), \tag{26}
$$
LayerNorm is applied individually to each of the $h$ heads, which can be seen as analogous to the Group-Norm applied over $h$ groups. Additionally, it should be highlighted that $w$ is derived using the formula $w=\exp(-\exp(ω))$ , with $ω∈ℝ^D/h$ representing the trainable parameters for each head. This computation ensures that $w$ remains within the range $(0,1)$ , thus assuring that $$ functions as a contraction matrix. Then, The $\boldsymbol{w}\boldsymbol{k}\boldsymbol{v}_t$ attention calculation can alternatively be written in a recurrent form:
$$
\displaystyle\boldsymbol{wkv}^\prime=s+diag(u)· k^T·ν \displaystyle s^\prime=diag(u)· s+k^T·ν \tag{30}
$$
### 3.3 RWKV-6 (Finch)
Finch (RWKV-6) [271] enhances the architecture’s expressiveness and adaptability by incorporating innovative data-driven functions into the time mixing and token shifting modules, such as parameterized linear interpolation. Furthermore, Finch (RWKV-6) introduces a novel application of low-rank adaptation functions [272], allowing the weight matrices to be trainable and effectively augment the learned data decay vectors in a way that is sensitive to the context.
The token shift of Finch is $ddlerp$ , which means data dependent linear interpolation between $x_t$ and $x_t-1$ , and it is defined as:
$$
\displaystylelora_\square(x)=λ_\square+\tanh(xA_\square)B_\square \tag{32}
$$
In this setup, both $μ_x$ and each $λ_\square$ bring in a trainable vector with dimension $D$ , while each $A_\square∈ℝ^D× 32$ and $B_\square∈ℝ^32× D$ represent new trainable weight matrices. The LoRA mechanisms [272] described above enable the cost-effective enhancement of learned vectors, similar to those in Eagle, by incorporating additional offsets that are defined by the incoming input. By integrating this innovative form of Token Shift with data-dependence, Finch aims to extend the model’s capabilities beyond the Token Shift method utilized in RWKV-4/Eagle. The model now computes the distribution of new and existing data per channel based on the input at both the present and preceding time steps. Thus, Finch’s time mixing block is defined accordingly:
$$
\displaystyle\square_t=ddlerp_\square(x_t,x_t-1)W_\square,
\square∈\{r,k,ν,g\} \displaystyle d_t=lora_d(ddlerp_d(x_t,x_t-1)) \displaystyle w_t=\exp(-\exp(d_t)) \displaystyle\boldsymbol{wkv}_t=diag(u)· k_t^T
·ν_t \displaystyle +∑_i=1^t-1diag≤ft(∑_j=i+1^
t-1w_j\right)· k_i^T·ν_i∈ℝ^(D/h)×
(D/h) \tag{34}
$$
Unlike in Eagle, $w_t$ of Finch is not static across the sequence. The fundamental transformation in Finch involves a shift to decay, allowing each component of $w_t$ to independently fluctuate over time in accordance with data, unlike the prior model which employed a static learned vector. The repetitive structure of the $\boldsymbol{wkv}_t$ attention computation remains identical to that of Eagle.
### 3.4 Compare RWKV with Other improved models based on Transformer
Linear Transformers [36] represent a variant of the Transformer architecture. While the conventional Transformer relies on the Self-Attention mechanism, Linear Transformers focus on approximating or enhancing this mechanism through a more efficient linear approach. Their primary goal is to decrease computational complexity and boost the model’s training and inference efficiency without sacrificing the transformative sequence processing capabilities inherent to Transformer models. Linear Transformers employ kernel methods to approximate self-attention. For instance, in linear self-attention, they utilize a low-rank matrix to approximate the query-key ( $Q$ - $K$ ) product found in standard self-attention mechanisms. Consider $Q$ as the query matrix and $K$ as the key matrix. In standard self-attention, the expression $QK^T$ involves a quadratic computation. Linear transformers can incorporate a low-rank matrix $M$ , altering the operation to $QMK^T$ , where $M$ has a lower rank. Selecting a suitable $M$ allows the computational complexity of this operation to decrease to a linear level. Utilizing kernel methods, linear transformers achieve a computational complexity of $O(Td^2)$ . In contrast, the computational complexity of RWKV is $O(Td)$ , indicating that linear transformers require more computation. Additionally, as linear transformers approximate the typical self-attention mechanism, there could be a diminution in the model’s representational capabilities.
In addition to RWKV, Mamba [273] is another frequently encountered and notable improved model developed from the Transformer architecture. Mamba leverages state-space models (SSMs), setting itself apart by effectively incorporating Structured State Space (S4) models into a large language model (LLM) framework. This inventive approach allows Mamba to achieve linear scalability concerning sequence length in complexity, marking a significant advancement compared to the quadratic complexity typically associated with traditional Transformer-based models. The carefully crafted architecture of Mamba, which incorporates selective SSM layers, enhances both computational efficiency and flexibility. Compared to RWKV, Mamba achieves better results in sequence segmentation. However, it demands greater computational resources. Conversely, RWKV excels by offering greater efficiency, quicker inference times, and reduced memory usage during operation [147].
Retentive Network (RetNet) [39] can simultaneously meet the three requirements of parallel training, low-cost inference, and good performance. By introducing the Retention mechanism, RetNet effectively solves the problems of computational complexity and memory occupation that Transformer has when dealing with long sequence data. Compared with the Attention mechanism, the Retention mechanism is more efficient during the calculation process and can significantly reduce the consumption of computational resources and memory. When processing sequence data, RetNet combines the parallel representation and the recurrent representation to achieve a balance between efficiency and performance. In addition, RetNet can generate multi-scale sequence representations. This means that it can understand sequence data from different granularities or levels. As a work emerging after RWKV, compared with RWKV, RetNet’s retention mechanism preserves high-dimensional states to encode sequence information, thereby further enhancing the expressive ability and leading to better performance.
Hyena [274] is a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. Unlike traditional RNNs and some Transformer-based architectures, the Hyena Hierarchy processes data through a hierarchical structure. The Hyena hierarchy consists of two operators (long convolution and element-wise multiplication gating) that are recursively defined by efficient quadratic primitives. Among them, the long convolution can effectively extract local features in sequences and, through the sliding window method, can capture similar patterns at different positions. The element-wise multiplication gating is generally located after the long convolution or between different layers in the Hyena hierarchy. Its main function is to screen and adjust the features extracted by the long convolution or the features transferred between different layers. The time complexity of Hyena is $O(ndt(\log t+d))$ , and the space complexity is $O(td)$ , where $n$ is the number of projections, $d$ is the dimension, and $t$ is the sequence length. In the case of long sequences, the growth of its computational complexity is relatively slow, enabling it to handle longer sequence data without a computational explosion. As for RWKV, both the time complexity and the space complexity are $O(td)$ . Compared with the traditional Transformer architecture, the computational efficiency has been significantly improved. Especially in the long-sequence inference scenario, its linear time complexity allows it to handle large-scale sequence data more efficiently.
### 3.5 Various Implementations
The RWKV authors offer an official PyTorch implementation as detailed in version [44], encompassing the implementation specifics for RWKV-1 through RWKV-6. Their efforts are further extended with the ongoing development of a laboratory version of RWKV-7 (Goose). Additionally, they introduced a Chatbot framework called ChatRWKV [45], designed to enable users to construct their own RWKV inference engines.
Beyond the primary implementation of PyTorch, RWKV has motivated a host of third-party versions developed in an array of programming languages and frameworks, thus showcasing its adaptability and wide-ranging utility. These implementations cover lightweight options in languages such as C [46, 47], C++ [48, 49, 50], and Fortran [63]. Additionally, RWKV has been integrated into modern programming languages including Julia [64] and Zig [65]. Notably, the ecosystems for Go [66, 67, 68, 69, 70, 71] and Rust [51, 52, 53, 54, 55] exhibit strong support, offering models, tokenizers, and tailored platform optimizations. Moreover, RWKV has been integrated into popular deep learning frameworks such as TensorFlow [56, 57, 58, 59, 60, 61, 62], Keras [72, 73], and PaddlePaddle [74, 75], allowing advanced features such as multi-GPU training and inference. Additional contributions in JAX [76, 77, 78], Node.js [79, 80], and MLX [81] further highlight the model’s versatility, establishing RWKV as a prominent tool for various computational and application needs.
RWKV’s capabilities have been notably improved in various areas, including training, inference, and applications, through additional implementations. In the realm of training, time-parallel methods have been implemented, with more sophisticated options incorporating LoRA, 4-bit quantization, as well as support for platforms like CUDA and ROCm [84, 82, 83]. For inference purposes, enhancements include Flash-Linear-Attention in RWKV v6 [85], execution support on Android CPUs using ONNX [86], compatibility with the NCNN framework [87], and deployment on Qualcomm HTP with the QNN SDK [88]. Moreover, a Java/JNI wrapper for RWKV.cpp has been developed to enable CPU-based inference with quantization, eliminating the need for Python [89]. Additionally, a kernel operator has been integrated into the bert4keras3 library to improve compatibility with RWKV6 models [90]. Implementations focused on specific applications include an API using the Flask framework, specifically tailored for role-playing scenarios [91]. Furthermore, inspired by NanoGPT, there exists an implementation utilizing RWKV’s RNN architecture to deliver LLM performance on par with GPT [92].
## 4 Applications of the RWKV Model
The RWKV model integrates the advantages of recurrent mechanisms with those of attention-based approaches. This unique architectural design enhances its efficiency and scalability, enabling it to adeptly tackle a wide range of problems. This chapter provides an overview of the practical applications of the RWKV model, focusing on its deployment in areas like natural language generation, natural language understanding, computer vision, audio and music tasks, and web-related tasks. Figure 3 illustrates examples of downstream tasks that make use of RWKV-based models. Additionally, the discussion extends to the use of RWKV in other NLP tasks, where its performance is evaluated across diverse assessment frameworks.
<details>
<summary>x2.png Details</summary>
### Visual Description
## Diagram: RWKV Model Architecture and Applications
### Overview
The diagram illustrates the RWKV (Recurrent Transformer) model family, showcasing its architecture, key applications, and technical advantages. It is structured into three horizontal sections:
1. **Top**: Domain-specific applications (ChatBot, Image Segmentation, Point Cloud Learning, Machine Translation)
2. **Middle**: Core RWKV variants (Point RWKV, Diffusion RWKV, Vision RWKV)
3. **Bottom**: Technical descriptions of the RWKV model's design and strengths
### Components/Axes
- **Top Section**:
- **ChatBot**: Text box with question about RWKV vs. Transformer architecture, efficiency, and applications.
- **Image Segmentation**: Four medical imaging examples (MRI/CT scans) with segmentation overlays.
- **Point Cloud Learning**: 3D point cloud visualization of a mechanical structure.
- **Machine Translation**: Text box with Chinese description (translated below) about RWKV's hybrid architecture.
- **Middle Section**:
- **Title**: "RWKV Family" in bold.
- **Subsections**:
- **Point RWKV**: Blue box with description of RWKV as a neural network combining RNNs and Transformers.
- **Diffusion RWKV**: Green box with medical imaging examples (segmented organs/tissues).
- **Vision RWKV**: Orange box with color-coded point cloud visualization.
- **Bottom Section**:
- **ChatBot**: Text box with robot icon and question about RWKV's design.
- **Image Segmentation**: Four segmented medical images (e.g., brain, spine).
- **Point Cloud Learning**: Color-coded 3D point cloud of a mechanical object.
- **Machine Translation**: Text box with Chinese description (translated below) about RWKV's computational efficiency.
### Content Details
- **Technical Descriptions**:
- **RWKV Model**:
- Combines RNNs and Transformers to address computational bottlenecks.
- Designed for efficiency and practical applications.
- **Applications**:
- **ChatBot**: Focus on architectural differences and efficiency.
- **Image Segmentation**: Medical imaging use cases (e.g., tumor detection).
- **Point Cloud Learning**: 3D object reconstruction.
- **Machine Translation**: Hybrid architecture for sequence processing.
- **Text Boxes**:
- **Chinese Text (Machine Translation)**:
- "RWKV模型是一种结合了Transformer与循环神经网络优势的新型架构。它的设计目标是解决Transformer在处理长序列时的计算瓶颈，同时保留RNN在序列建模方面的优势。"
- Translation: "The RWKV model is a novel architecture that combines the strengths of Transformers and recurrent neural networks (RNNs). Its design goal is to address the computational bottlenecks of Transformers in handling long sequences while retaining the advantages of RNNs in sequence modeling."
### Key Observations
1. **Hybrid Architecture**: RWKV explicitly bridges RNNs and Transformers, emphasizing efficiency and scalability.
2. **Domain-Specific Use Cases**: Applications span NLP (ChatBot, Translation), medical imaging (Segmentation), and 3D reconstruction (Point Cloud).
3. **Visual Consistency**: Color coding (blue, green, orange) aligns with RWKV variants and their respective applications.
4. **Technical Focus**: Descriptions highlight computational advantages over traditional models.
### Interpretation
The diagram positions RWKV as a versatile, efficient alternative to Transformers, particularly for tasks requiring long-sequence processing (e.g., translation) or resource-constrained environments. By integrating RNN-like recurrence with Transformer strengths, it aims to reduce computational overhead while maintaining performance. The medical and 3D applications suggest practical utility in fields requiring precise spatial or temporal modeling. The emphasis on efficiency aligns with trends in optimizing large language models for real-world deployment.
</details>
Figure 3: Examples of downstream tasks utilizing RWKV-based models.
### 4.1 Natural Language Generation
Natural Language Generation holds significant importance as an application for RWKV models, showcasing their ability to generate text that mirrors human-like communication. By leveraging the architecture of the RWKV model, it adeptly captures contextual subtleties and generates coherent narratives across diverse formats. The model excels at preserving contextual relevance while producing text that is fluid and captivating. Additionally, RWKV’s adaptability enables it to adjust to various tones and styles, rendering it especially beneficial for tasks such as automated storytelling, content creation, and engaging chatbots.
Several investigations underscore the wide range of uses for RWKV in the field of natural language generation. As an example, RWKV has been effectively utilized in producing web novels spanning numerous genres, such as fantasy and romance [41]. Additionally, the model has been integrated into multiple chatbot projects [93, 94, 95, 96], such as a QQ chatbot [97], a lightweight local chat AI [98], and a chatbot designed to respond to inquiries regarding Esprit University [99]. In particular, a robust RWKV-based RAG deployment system facilitates the efficient management of local knowledge bases, while also providing a question-answering bot [100].
Within interactive settings, RWKV is used for role-playing scenarios [103, 104], and also for generating WeChat responses via projects like Chatrwkv [101, 102]. Furthermore, RWKV is implemented in a Telegram LLM bot written in Rust, which can operate using either RWKV or Llama2 models [105]. Chatbots with RWKV capability also facilitate multimedia interactions, such as sending voice messages and stickers [106, 107]. Additionally, chatbots using the Rasa NLU framework alongside RWKV-4-Raven can address frequently asked questions in French, English, and Tunisian Arabic [108]. Across these diverse applications, RWKV models manifest substantial potential in advancing natural language generation and refining human-computer interaction.
### 4.2 Natural Language Understanding
Understanding human language is a crucial capability of RWKV models, highlighting their proficiency in interpreting and analyzing it. By skillfully handling the intricate details of syntax, semantics, and context, RWKV models are particularly effective in performing tasks like machine translation, categorizing text, and providing answers to interactive questions.
RWKV models have shown encouraging outcomes in the field of machine translation. For instance, a translation algorithm based on RWKV for Mongolian-Chinese utilizes contrastive learning to address issues of semantic distortion and training instability that are often encountered in conventional approaches [42]. Furthermore, a compact language model related to sports, developed using RWKV-v6, has realized notable gains in accuracy, reaching levels of performance similar to those of larger models such as SomlLM and Qwen [111].
RWKV has been utilized in traditional NLU tasks, such as text classification [112], the creation of interactive narratives [113], and functioning as a virtual assistant on platforms including Discord [109]. In addition, RWKV’s application in gaming contexts has contributed to enhancing character dialogues, thus improving player interaction and immersion [114]. Moreover, it has been employed in interactive PDF query systems, where the RWKV model, alongside tools like Langchain and Streamlit, is used for tasks like summarization and information extraction [110].
RWKV models have been successfully integrated into real-time applications, such as systems that listen for queries and interactively provide answers from the model [115], as well as question-and-answer systems that demonstrate its effectiveness in processing and interpreting complex language tasks [116]. Furthermore, educational mobile applications aimed at engaging children through customizable avatars [117] leverage RWKV technology. Another notable application involves using pure RWKV in knowledge graph implementations [118], which enhances information retrieval and contextual understanding.
### 4.3 Other NLP tasks
The RWKV model has shown remarkable efficiency in a wide range of NLP tasks, such as forecasting, code generation, and content regulation. Its ability to adapt and deliver robust results across these varied applications underscores its transformative potential within the field of natural language processing.
Within the domain of predictive tasks, the RWKV model demonstrates considerable effectiveness. The CLOPPS (Clinical Outcomes Prediction with Partially Observed Data) model enhances classification capabilities by addressing the complexities linked to actual clinical datasets [119]. Similarly, a project drawing on data from the American Community Survey has effectively generated population forecasts for numerous towns across the United States [134]. In forecasting photovoltaic power generation, the MSRWKV-2DTCN model employs a multi-scale temporal convolutional network to adeptly capture both periodic and historical dependencies, thereby improving short-term prediction accuracy [43]. In addition, utilizing the RWKV model for predicting stock prices highlights its potential in time series forecasting tasks [120, 121, 122].
The intelligent completion of code represents a significant application of RWKV, surpassing conventional RNN models in both understanding and completing code. This superiority is attributed to enhanced accuracy derived from faster training processes, an increased number of network layers, and the implementation of specific syntax tree mappings [123, 124]. Recent advancements have incorporated multiple temporal perspectives into the model, further enhancing its capacity to interpret context more effectively while preserving linear computational complexity [135]. Additionally, RWKV has shown impressive proficiency in moderating content, benefiting from the use of a uniquely crafted dataset and purpose-driven experiments. This effectiveness is further bolstered by leveraging a comprehensive dataset for knowledge distillation when handling large-scale tasks [136, 125].
In terms of efficiency and resource management, the GoldFinch framework proposes a linear attention mechanism that enhances the RWKV’s cache production, leading to substantial advancements in reasoning abilities across extended context lengths [126]. To overcome the drawbacks of conventional RNNs, the RWKV-TS model delivers a robust structure specifically designed for handling time series applications [127]. The TIM model integrates the RWKV architecture by incorporating causal interactions and amplifying local patterns, resulting in superior generation outcomes [128].
The Retrospected Receptance Weighted Key Value (RRWKV) framework incorporates retrospective features that improve its capacity to understand long-term dependencies, thereby overcoming the challenges faced by RWKV in processing distant information [129]. In addition, the application of transfer learning techniques significantly boosts the efficiency of domain generation algorithm classifiers, which facilitates accurate distinction between non-malicious and harmful exemplars [137]. In the realm of personalized news recommendation systems, the fusion of large language models with knowledge graphs significantly boosts the precision of recommendations [130]. The Decision-RWKV model, which blends decision transformers with experience replay frameworks, exhibits strong performance in reinforcement learning and decision-making tasks [131]. To combat the forgetting phenomenon prevalent in language models, a strategy utilizing synthetic gradients is introduced. This method encodes prompts directly, thereby enhancing memory retention during text generation [132]. Leveraging the RWKV architecture, the SpikeGPT model employs event-driven sparse activations to generate natural language effectively [133].
### 4.4 Computer Vision
Within computer vision, RWKV models are capturing growing interest due to their ability to integrate visual information with sophisticated analytical skills. Taking advantage of the distinct characteristics of RWKV frameworks, these models are capable of considerably improving a range of tasks. These tasks include segmenting medical images, perceiving 3D point clouds, and detecting actions in real-time.
RWKV models have made significant contributions to the field of medical image processing. BSBP-RWKV, for example, excels in the segmentation of medical images, merging traditional denoising techniques with a robust network framework for high precision and efficiency [138]. Likewise, Restore-RWKV stands out in medical image restoration by implementing an innovative omnidirectional token translation layer and reconstructive attention mechanism, which successfully capture spatial relationships in 2D images [139]. Additionally, RWKV-based Multi-Instance Learning (MIL) has achieved substantial success in classifying high-resolution Whole Slide Images (WSI), proficiently learning global feature representations [140].
In the domain of 3D point cloud perception, RWKV models demonstrate considerable proficiency. PointRWKV employs linear complexity and enhanced multi-head matrix states to process point cloud data, greatly boosting the ability to capture local geometric features [142]. Moreover, the LION framework employs a linear group RNN to adeptly model long-range relationships, adapting well to the nature of sparse point clouds, thereby ensuring efficient 3D object detection [141]. OccRWKV serves as an efficient semantic occupancy network, utilizing RWKV modules for precise predictions within sparse environments [143].
In the domain of real-time action detection, the TLS-RWKV model is noteworthy for its ability to grasp temporal patterns without sacrificing computational efficiency. This capability allows for the prompt prediction of action categories in streaming video, as detailed in [144]. RuleVM sets itself apart with a dual-branch framework tailored for the weakly supervised monitoring of violent content. It fuses visual features with language-image alignment to enhance both interpretability and performance as noted in [145]. Additionally, the RWKV models have significant applications in image generation and denoising. For instance, the Diffusion-RWKV is pertinent to high-resolution image generation tasks, adeptly handling extensive inputs by mitigating spatial aggregation complexities [154]. LineRWKV is efficient in hyperspectral image compression, employing line-based inference mechanisms to curtail memory use [146]. In the realm of image segmentation, RWKV-SAM delivers efficient results with a hybrid backbone architecture, markedly outperforming traditional transformer methods [147].
Regarding high-resolution image processing, Vision-RWKV (VRWKV) demonstrates the potential to handle high-resolution images by reducing spatial aggregation complexity, surpassing the performance of Vision Transformers (ViT) [148]. VisualRWKV-HD and VisualRWKV-UHD focus on processing high-resolution visual inputs, significantly enhancing performance in text-rich tasks [149]. The exploration of RWKV-based facial expression recognition technology continues to unfold its applications in affective computing [157]. Additionally, research on RWKV for image noise reduction provides new insights into image processing [158]. An enhanced transformer model, RWKV-IR, is introduced to improve image restoration with advanced linear attention mechanisms [155].
The LSTM CrossRWKV (LCR) framework employs a Cross RWKV gate along with an improved LSTM mechanism to adeptly capture both spatial and temporal aspects pertinent to video analysis [150]. For image-text interactions, RWKV-CLIP stands out as the pioneer RWKV-integrated visual-language representation model, merging the parallel training efficiency of transformers with the inference efficacy of RNNs [151]. Studies involving Spiking Neural Networks (SNNs) underscore their capabilities in generating images [152]. Moreover, approaches for predicting multimodal pedestrian trajectories utilize conditional variational autoencoders to adeptly account for the diversity and uncertainty in human motion [153]. StyleRWKV [156] leverages RWKV models for efficient style transfer, introducing Re-WKV attention, a Deform-Shifting layer, and Skip Scanning to enhance global and local context while maintaining low memory usage and linear time complexity.
### 4.5 Audio and Music Tasks
In music generation, the Music-Diff architecture enhances sample diversity by using a joint probability diffusion model, overcoming the limitations of traditional language models in generative tasks [162]. RWKV has extended its capabilities to MIDI music generation [159, 160] and has been applied in music genre classification [168]. Its creative versatility is further demonstrated in a mini-game that integrates Stable Diffusion with the RWKV music model [161].
In the field of speech recognition, RWKV has made notable advancements. RWKV-based semantic voice activity detection (VAD) shows enhanced resilience to noise interference [163]. At the same time, the RWKV-Transducer model has achieved competitive accuracy in streaming Automatic Speech Recognition (ASR), optimizing both latency and memory consumption [164, 165]. Additionally, RWKV-SpeechChat uses the frozen 3B RWKV model to support real-time conversational tasks, including ASR, speech translation, and speech question answering [169]. RWKV has also been successfully applied in audio pattern recognition [166].
### 4.6 Web Application
The exceptional flexibility and easy accessibility of RWKV models have significantly driven the development of a wide variety of web-based applications. This development has, in turn, considerably enhanced user engagement with advanced AI technologies. By merging user-friendly interfaces with robust back-end frameworks, these implementations have effectively connected modern models with end-users. As a result, they have simplified seamless interactions and facilitated efficient deployment across numerous platforms.
Implementations based on web technologies have notably increased the usability and interactivity of RWKV models [170, 171, 175]. As an illustration, the RWKV v4 version accessible via web browsers features a user-friendly interface, facilitating easy model access and use [173]. The role-playing web UI, created with Gradio, exemplifies the adaptability of RWKV for interactive applications [174]. In addition, projects focusing on both local and server-side web interfaces offer efficient means to incorporate RWKV [172, 176].
For improving user engagement, the ChatRWKV web chat interface facilitates real-time conversations between users and the model [177, 178, 179, 180]. Furthermore, derived from the rwkv.cpp initiative and combined with GPT-SoVITS for novel reading, this highlights the broad applicability of RWKV in multimedia contexts [184, 182]. The RWKV Flask service provides a minimalistic solution for deploying applications, and the Vue.js-based user interface for the RWKV Role Playing API ensures a refined user interaction experience [181, 183].
### 4.7 Evaluation of RWKV Models
Thorough evaluation of RWKV models is essential to fully grasp their performance and efficacy across a broad range of natural language processing benchmarks. The assessment process employs an array of metrics and methodologies crafted to evaluate both qualitative and quantitative dimensions of the model outcomes. As RWKV models have advanced, a number of evaluation methods have been developed to assess their performance. In this research, we have assembled 17 benchmark tests, detailed in Table I, each highlighting distinct facets and criteria for evaluation.
TABLE I: Summary of RWKV model evaluation
| BIPIA [194] | Indirect prompt injection attacks |
| --- | --- |
| CoDI-Eval [195] | The capability of LLMs to respond to the constraints in instructions. |
| CMATH [197] | Mathematical problem solving |
| VasE [196] | Evaluation through voting theory framework |
| Head-to-Tail [198] | Assess the ability of LLMs to internalize facts |
| REGBENCH [185] | In-context Language Learning(ICLL) |
| Freshbench [200] | A dynamic assessment framework for temporal generalisation |
| LongctxBench [186] | Efficiency in handling long contexts |
| LongICLBench [201] | Benchmark for extreme-label classification in long contexts |
| LooGLE [187] | Benchmark for long-dependency understanding. |
| MAGNIFICO [190] | Learning new interpretations in-context. |
| MANGO [191] | Benchmark for mapping and navigation capabilities. |
| PRE [199] | Evaluation framework inspired by academic peer review. |
| RULER [188] | Enhanced benchmark for long-context evaluations extending the NIAH test. |
| S3EVAL [189] | Flexible evaluation method using complex synthetic tasks. |
| SuperCLUE [192] | Comprehensive benchmark for Chinese user preferences. |
| Zhujiu [193] | Multi-dimensional evaluation benchmark for Chinese LLMs. |
Several studies have focused on evaluating the performance of RWKV in long-sequence modeling and in-context learning [185, 205, 186]. The LooGLE benchmark evaluated the RWKV model’s performance on tasks that involve long dependencies, noting that it did not excel in handling complex tasks with extended dependencies [187]. Meanwhile, the RULER benchmark identified a noticeable drop in the model’s effectiveness as the length of in-context inputs increased, a challenge particularly evident in understanding within context [188]. Furthermore, the comprehensive S3EVAL evaluation suite was utilized to examine RWKV’s capabilities in long in-context tasks, exposing performance limitations when managing extremely lengthy in-context scenarios with the current versions of RWKV models [189].
The MAGNIFICO evaluation framework and the MANGO benchmark were utilized to assess RWKV’s performance on tasks requiring comprehension of new vocabulary, multiple reinterpretations, and spatial reasoning capabilities. While RWKV showed promise in managing complex tasks, it revealed notable deficiencies in accurately executing spatial reasoning and swiftly adapting to rapidly changing contexts [190, 191]. To measure the model’s efficiency in real-world scenarios, the SuperCLUE benchmark was introduced, highlighting the inadequacy of traditional accuracy measures in truly capturing user requirements [192]. Furthermore, the ZhuJiu benchmark, employing a multidimensional evaluation approach, carried out an extensive analysis of RWKV’s capabilities in knowledge graph and in-context learning tasks, with a particular emphasis on its performance in both Chinese and English language tasks [193].
In the evaluation of model robustness against malicious injection attacks, the BIPIA benchmark was proposed to assess the ability of LLMs to resist external malicious instructions [194]. Similarly, the CoDI-Eval benchmark was used to evaluate LLM performance in executing constrained instructions, revealing gaps between open-source and commercial LLMs in following instructions [195]. A voting-theory framework was applied to LLM evaluation, enhancing robustness in complex tasks by transforming tasks into voting problems [196].
The CMATH dataset evaluated the capability of LLMs in tackling mathematical problems, demonstrating that GPT-4 achieved higher accuracy than RWKV, especially when faced with distracting data [197]. The Head-to-Tail benchmark served to assess LLMs on knowledge graph Q&A tasks, identifying RWKV’s weaknesses in handling infrequent information, notably in knowledge retention and cognitive reasoning [198]. Research comparing recursive models and transformer models indicated that RWKV surpassed transformers in specific tasks, particularly those necessitating recursive processing [206].
Research has illustrated RWKV’s impressive ability to perform in-context learning tasks effectively, revealing its capacity to understand intricate data relationships within the context itself, eliminating the dependence on additional optimization tools [207]. In assessments concerning brain-machine interfaces and edge device applications, analyses comparing RWKV with GRU and Transformer models demonstrated that RWKV excels in reasoning speed and scalability. This makes it a more fitting choice for handling extensive data and models on a large scale [208].
Efforts to adapt transformer interpretability techniques to RNNs (notably RWKV) have been investigated, with suggestions to improve model performance by optimizing RNN state compression characteristics [209]. Assessments of large models in regression tasks revealed that RWKV demonstrates a superior in-context learning capacity in multi-task contexts compared to conventional supervised models [210]. Analyses of RWKV’s capability in long sequence modeling exposed constraints when managing very lengthy sequences [211]. The peer-reviewed LLM framework addresses biases in traditional evaluations by employing multiple LLM ”reviewers” to provide a more balanced assessment, emphasizing the necessity for diverse perspectives [199]. In examining temporal generalization, the dynamic generation benchmark proposes a methodology to highlight performance decline with time-related changes, indicating a need for enhancements in current update mechanisms to better accommodate evolving information [200]. The LongICLBench benchmark concentrated on RWKV’s outcomes in challenging label classification tasks, identifying shortcomings when facing extensive label sets and long inputs, especially in complex classifications [201]. Additionally, various evaluations have been performed on RWKV’s abilities in projects like the Long Range Arena [203], LongBench_RWKV [202], MT_BENCH_RWKV [204], and Compare_LLMs [212].
### 4.8 Others
The implementation of RWKV extends into several pioneering fields beyond the conventional applications in Natural Language Processing and Computer Vision, warranting significant interest. As RWKV models persistently evolve, their range of applicability continues to expand comprehensively. This part will delve into a variety of supplementary use cases, emphasizing the potential and adaptable nature of RWKV across a multitude of domains.
RWKV, when incorporated with Spiking Neural Networks (SNNs), has shown improved adversarial robustness in tasks combining vision and language [213]. The Blip2RWKV+QFormer model is tailored for dialogues involving both images and text [214]. Recent progress has seen the introduction of retrieval-augmented generation (RAG) and multimodal extensions to RWKV, broadening its functionality [100]. Furthermore, VisualRWKV exemplifies its use in Visual Language Models (VLMs), leveraging a linear RNN model to deliver performance that rivals transformer-based systems [215].
A method for image-text alignment employing a frozen RWKV-4-World-0.4B model highlights its potential in multimodal tasks [216]. The breadth of RWKV’s capabilities encompasses a wide array of tasks, including sequence classification, embedding generation, and cross-encoder applications, demonstrating its versatility in tackling various challenges in machine learning [217]. Moreover, RWKV has been successfully applied in the training of reward models enhanced by human feedback within the context of reinforcement learning [218]. Its applications further extend to the identification of Twitter users who are bots [219].
Numerous studies focus on enhancing the performance and efficiency of RWKV models. For instance, one study provides a unified perspective on attention-free layers, highlighting their sub-quadratic complexity and favourable scaling properties [220]. Additionally, techniques for parameter-efficient fine-tuning of RWKV [221], along with a quick fine-tuning package for the RWKV-PEFT project [222], and the implementation of infinite-length training for RWKV5 [223], as well as improvements in decision-making tasks for lifelong robotics [224], all demonstrate the model’s capabilities. Separately, RWKV has found applications in gaming, where it enhances the user experience through dynamic decision-making processes [225]. In the field of industrial time series forecasting, the RWKV model has demonstrated its potential in handling high-dimensional data [226]. Other projects demonstrate the integration of RWKV with various frameworks and tools, highlighting its broad application potential [227, 228, 229].
The integration of RWKV with an array of frameworks and tools greatly expands its range of applications. A comprehensive tool for automated RWKV management and deployment provides an interface that is compatible with the OpenAI API [230]. Moreover, RWKV has been successfully integrated into Node.js [231], while another initiative showcases its modular integration capabilities with the Godot engine [232]. In addition, models from the RWKV Raven/Pile/PilePlus series can be converted into the HF format, facilitating full fine-tuning with Alpaca [233].
In the field of reinforcement learning, the utilization of RWKV is on the rise. For instance, studies examine RWKV’s role in training RLHF models [234] and its potential applications in decision-making tasks with limitless context lengths [235]. The RWKV reinforcement learning toolkit boosts capabilities in this area [236] and has demonstrated impressive results in testing RWKV v6 state tuning with 4-bit quantization [237]. Additionally, MARWKV, which is based on RWKV, achieves performance on par with MAT in multi-agent reinforcement learning, while providing superior computational efficiency as the number of agents grows [238].
Diverse resources are accessible for those engaged in the fine-tuning and optimization of RWKV models. These encompass detailed tutorials on fine-tuning [239, 240, 241], along with a streamlined inference API [242]. Additionally, there are training tutorials focused specifically on the state tuning of RWKV models [243]. For optimizing prefill and training speed, one can refer to a comprehensive tutorial employing chunked parallelism [244].
The significance of enhancing the interpretability and evaluation capabilities of RWKV models is comparable. Examining the interpretability of outliers within RWKV models showcases the progress made in this research sector [245]. Visualization tools that display RWKV’s internal states during inference [246] and methods to document state transitions [247] facilitate a more profound comprehension of how these models operate. Moreover, converting RWKV models into TensorRT and ONNX formats increases the versatility of their deployment [248].
The notable performance of RWKV across various applications is worth highlighting. Notable implementations involve the utilization of the GGML library for offline operations on iOS and macOS, as detailed in [249], as well as prompt projects that integrate RWKV with Langchain, referenced in [250]. Additionally, the visualization service created for the RWKV-Ouroboros project, cited in [251], exemplifies the vast applicability of RWKV. Moreover, deployment scripts for the automatic deployment of GGML/GGUF models on Android have been developed, as seen in [252]. Compression techniques specifically designed for RWKV, such as low-rank approximation, sparsity predictors, and clustering heads, have achieved significant results, enabling up to 4.95–3.8x compression with minimal accuracy loss [261]. The integration of RWKV with the Llama architecture [253] and the implementation of GPTQ quantization techniques [254] further underscore the innovative progress within this area.
Transferring RWKV-LM models to the ROCm platform, as outlined in [255], employing inference through OpenGL technology [256], and merging RWKV with server-based applications designed for question-answering tasks [257] create promising new opportunities for discovery. The PC version of ChatRWKV [258] is equipped with a user-friendly interface, enhancing accessibility. Moreover, the exploration into generative calibration methods for RWKV, as discussed in [259], focuses on strategies for enhancing model performance during context learning. In V2X environments, an RWKV-DPA [260] estimator was proposed, combining RWKV network and data-pilot aided estimation. It outperformed other DL-based estimators under IEEE 802.11p standard with lower complexity. The availability of open datasets for RWKV [264] provides crucial resources supporting researchers in their work. The PCF-RWKV [262] model leverages RWKV architecture, task-specialized LoRAs, and Multi-Agents technology to automate lifecycle inventory construction and carbon footprint calculation. The L3TC [263] method integrates RWKV models, an outlier-aware tokenizer, and high-rank reparameterization to enable efficient, low-complexity text compression.
## 5 Future Directions and Challenges
The RWKV model holds significant promise as an alternative to conventional architectures like Transformers and RNNs. It has shown notable effectiveness in multiple tasks. Nevertheless, the widespread implementation and use of RWKV are confronted with numerous challenges that necessitate deeper investigation and research.
### 5.1 Expanding RWKV’s Capabilities in Future Applications
#### 5.1.1 Long sequence processing capability
Turning towards the future prospects of long-sequence processing using RWKV reveals great promise for theoretical innovations and practical uses. Notably, a key avenue for exploration lies in refining the model’s capacity to effectively capture and maintain information over exceedingly long contexts. Although RWKV’s mechanism mimicking recursion already provides a reduction in computational complexity when compared to Transformers, overcoming the issue of information decay across extended sequences is still a challenge. Advancements in memory mechanisms or enhancements to the architecture through explicit tracking of long-range dependencies might further boost its capabilities. Furthermore, coupling RWKV with methodologies like adaptive tokenization or hierarchical attention frameworks could lead to more efficient handling of ultra-long inputs.
#### 5.1.2 Multi-modal and Cross-modal Learning
RWKV’s distinctive structure offers promising avenues for advancements in multi-modal and cross-modal learning, although its full capabilities have not yet been thoroughly examined. Future studies might aim at developing specialized multi-modal model architectures, allowing RWKV to adeptly handle various forms of data, including text, audio, and visual content, while enhancing inter-modal linkages. In tasks involving cross-modal generation, RWKV could be refined to facilitate smoother transitions between modalities, such as the conversion of text to images or audio to subtitles. An expansion of this research might entail employing RWKV in more intricate modality mixtures, like audio-visual or tactile-language tasks, especially in contexts demanding multi-modal reasoning and decision-making. By integrating data preprocessing strategies specifically designed for different modalities or incorporating recent advances in multi-modal modeling techniques, RWKV could emerge as a pivotal asset in the field of multi-modal learning.
#### 5.1.3 Parameter-efficient Finetuning
Parameter-efficient finetuning (PEFT) techniques have become popular as a means to adapt large models to specific tasks efficiently, minimizing the need for extensive computational power. Due to RWKV’s modular structure and its computation that mimics recurrent methods, it is particularly well-suited for these approaches. Strategies like LoRA [275], adapters, and prompt tuning can be integrated smoothly into RWKV, facilitating task-specific adjustments while leaving the majority of the model’s parameters unchanged. This approach not only reduces memory utilization but also speeds up finetuning on datasets specific to various domains. Future investigations might focus on crafting new PEFT strategies designed specifically for RWKV’s distinctive structure, potentially exploiting its sequential processing capabilities to enable effective transfer learning in time-series tasks or other sequential contexts.
### 5.2 Security and Stability
#### 5.2.1 Safety and Robustness
Like other foundational models, RWKV may be susceptible to adversarial attacks, which can threaten its trustworthiness in safety-sensitive contexts. Although its recurrent-like structure provides benefits for handling sequential data, the model’s behavior in adversarial settings or when faced with distributional shifts is not yet well understood. It is vital to guarantee the stability of RWKV’s outputs against small input changes, especially as its use expands in critical areas. Future investigations could explore methods such as adversarial training to bolster RWKV’s resistance to harmful manipulations.
#### 5.2.2 Bias and fairness
Models built on the RWKV framework are trained using vast datasets, which frequently mirror societal biases and stereotypes present in their training data. These biases can subtly affect the model’s behavior and might result in outcomes that are both unintended and unfavorable. Although notable strides have been made in dealing with fairness issues within other foundational model frameworks, focused studies on strategies to reduce biases specific to RWKV-based models are still relatively scarce. It is crucial to examine methodologies for pinpointing and mitigating these biases. Potential approaches encompass ensuring a balanced representation within training datasets, integrating fairness constraints during the fine-tuning phase, and implementing corrective adjustments post hoc.
#### 5.2.3 Privacy
Concerns regarding privacy pose a considerable challenge for large models, especially in interactive applications, where they frequently process user data to generate outputs or predictions. In these situations, sensitive user information—such as personal identifiers, preferences, or behavior patterns—may inadvertently be stored by the model, elevating the risk of data misuse or unintentional exposure. To tackle these issues, it is necessary to develop strong privacy-preserving methods specifically crafted for the RWKV architecture. For example, incorporating differential privacy during training can prevent the model from memorizing or retrieving individual data points. Additionally, federated learning offers a promising solution by allowing local training on decentralized data without gathering raw user data centrally.
### 5.3 Optimizing RWKV for Hardware Acceleration
Future research endeavors should aim to increase the efficiency of RWKV-based models across various hardware platforms by tailoring them for low-power applications, such as edge devices and mobile gadgets. To achieve this, potential strategies could involve model quantization, utilizing sparse matrix algorithms, and implementing dynamic load balancing to maintain optimal performance in environments with limited resources. Additionally, creating optimization algorithms specifically for hardware like GPUs, TPUs, and custom ASICs offers a valuable opportunity. Customizing computation graphs and memory allocation methods to leverage the unique capabilities of these devices could lead to notable enhancements in both computational efficiency and processing speed.
## 6 Conclusion
The RWKV model, an innovative deep learning framework, has demonstrated significant potential across numerous fields, such as natural language processing, time series analysis, and computer vision. Its distinct structure achieves an optimal blend of high-level performance and computational efficiency, making it apt for tackling intricate tasks. However, even with its increasing influence, there is a noticeable absence of an exhaustive review of the RWKV model and its applications. This paper aims to address this deficiency by delivering an in-depth analysis of the RWKV model, emphasizing its architectural advancements, primary applications, and the fields where it has shown notable success. Furthermore, we examine the existing challenges in RWKV-related research and suggest promising avenues for future investigation.
## References
- [1] F. Rosenblatt, “The perceptron: a probabilistic model for information storage and organization in the brain.” Psychological review, vol. 65, no. 6, p. 386, 1958.
- [2] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
- [3] Y. Zhu, F. Zhuang, J. Wang, G. Ke, J. Chen, J. Bian, H. Xiong, and Q. He, “Deep subdomain adaptation network for image classification,” IEEE transactions on neural networks and learning systems, vol. 32, no. 4, pp. 1713–1722, 2020.
- [4] H. Touvron, P. Bojanowski, M. Caron, M. Cord, A. El-Nouby, E. Grave, G. Izacard, A. Joulin, G. Synnaeve, J. Verbeek et al., “Resmlp: Feedforward networks for image classification with data-efficient training,” IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 4, pp. 5314–5321, 2022.
- [5] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal processing magazine, vol. 29, no. 6, pp. 82–97, 2012.
- [6] G. E. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,” IEEE Transactions on audio, speech, and language processing, vol. 20, no. 1, pp. 30–42, 2011.
- [7] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7829–7833.
- [8] E. Cambria and B. White, “Jumping nlp curves: A review of natural language processing research,” IEEE Computational intelligence magazine, vol. 9, no. 2, pp. 48–57, 2014.
- [9] F. Incitti, F. Urli, and L. Snidaro, “Beyond word embeddings: A survey,” Information Fusion, vol. 89, pp. 418–436, 2023.
- [10] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017.
- [11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
- [12] Y. Wu, D. Inkpen, and A. El-Roby, “Dual mixup regularized learning for adversarial domain adaptation,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16. Springer, 2020, pp. 540–555.
- [13] C. Badue, R. Guidolini, R. V. Carneiro, P. Azevedo, V. B. Cardoso, A. Forechi, L. Jesus, R. Berriel, T. M. Paixao, F. Mutz et al., “Self-driving cars: A survey,” Expert systems with applications, vol. 165, p. 113816, 2021.
- [14] I. D. Raji and G. Fried, “About face: A survey of facial recognition evaluation,” arXiv preprint arXiv:2102.00813, 2021.
- [15] E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” IEEE access, vol. 8, pp. 58 443–58 469, 2020.
- [16] S. Hochreiter, “Long short-term memory,” Neural Computation MIT-Press, 1997.
- [17] D. Bahdanau, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2014.
- [18] R. Dabre, C. Chu, and A. Kunchukuttan, “A survey of multilingual neural machine translation,” ACM Computing Surveys (CSUR), vol. 53, no. 5, pp. 1–38, 2020.
- [19] M. Malik, M. K. Malik, K. Mehmood, and I. Makhdoom, “Automatic speech recognition: a survey,” Multimedia Tools and Applications, vol. 80, pp. 9411–9457, 2021.
- [20] A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017.
- [21] A. Roy, M. Saffar, A. Vaswani, and D. Grangier, “Efficient content-based sparse attention with routing transformers,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 53–68, 2021.
- [22] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” 2022. [Online]. Available: https://arxiv.org/abs/2203.02155
- [23] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” 2022. [Online]. Available: https://arxiv.org/abs/2101.03961
- [24] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong, “ibot: Image bert pre-training with online tokenizer,” 2022. [Online]. Available: https://arxiv.org/abs/2111.07832
- [25] Z. Ren, N. Cheng, R. Sun, X. Wang, N. Lu, and W. Xu, “Sigt: An efficient end-to-end mimo-ofdm receiver framework based on transformer,” in 2022 5th International Conference on Communications, Signal Processing, and their Applications (ICCSPA). IEEE, 2022, pp. 1–6.
- [26] Y. Zhang, D. S. Park, W. Han, J. Qin, A. Gulati, J. Shor, A. Jansen, Y. Xu, Y. Huang, S. Wang et al., “Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1519–1532, 2022.
- [27] J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
- [28] A. Radford, “Improving language understanding by generative pre-training,” 2018.
- [29] M. Tezgider, B. Yildiz, and G. Aydin, “Text classification using improved bidirectional transformer,” Concurrency and Computation: Practice and Experience, vol. 34, no. 9, p. e6486, 2022.
- [30] H. Zhang, H. Song, S. Li, M. Zhou, and D. Song, “A survey of controllable text generation using transformer-based pre-trained language models,” ACM Computing Surveys, vol. 56, no. 3, pp. 1–37, 2023.
- [31] T. Shao, Y. Guo, H. Chen, and Z. Hao, “Transformer-based neural network for answer selection in question answering,” IEEE Access, vol. 7, pp. 26 146–26 156, 2019.
- [32] Z. Dai, “Transformer-xl: Attentive language models beyond a fixed-length context,” arXiv preprint arXiv:1901.02860, 2019.
- [33] Y. Dong, J.-B. Cordonnier, and A. Loukas, “Attention is not all you need: Pure attention loses rank doubly exponentially with depth,” in International Conference on Machine Learning. PMLR, 2021, pp. 2793–2803.
- [34] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, L. Kaiser et al., “Rethinking attention with performers,” arXiv preprint arXiv:2009.14794, 2020.
- [35] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are rnns: Fast autoregressive transformers with linear attention,” in International conference on machine learning. PMLR, 2020, pp. 5156–5165.
- [36] S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, “Linformer: Self-attention with linear complexity,” arXiv preprint arXiv:2006.04768, 2020.
- [37] A. Gu, K. Goel, and C. Ré, “Efficiently modeling long sequences with structured state spaces,” arXiv preprint arXiv:2111.00396, 2021.
- [38] A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023.
- [39] Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei, “Retentive network: A successor to transformer for large language models,” arXiv preprint arXiv:2307.08621, 2023.
- [40] B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella et al., “Rwkv: Reinventing rnns for the transformer era,” arXiv preprint arXiv:2305.13048, 2023.
- [41] P. Bo, “Ai-writer,” https://github.com/BlinkDL/AI-Writer, 2022.
- [42] X. Liu, Y. Su, W. Nier, Y. Ji, R. Qing Dao Er Ji, and M. Lu, “An approach to mongolian neural machine translation based on rwkv language model and contrastive learning,” in International Conference on Neural Information Processing. Springer, 2023, pp. 327–340.
- [43] J. Hao, F. Liu, and W. Zhang, “Multi-scale rwkv with 2-dimensional temporal convolutional network for short-term photovoltaic power forecasting,” Energy, vol. 309, p. 133068, 2024.
- [44] B. PENG, “Rwkv-lm,” https://github.com/BlinkDL/RWKV-LM, 2021.
- [45] ——, “Chatrwkv,” https://github.com/BlinkDL/ChatRWKV, 2023.
- [46] RWKV, “rwkv.cpp,” https://github.com/RWKV/rwkv.cpp, 2023.
- [47] F. Kong, “rwkv.c,” https://github.com/kong13661/rwkv.c, 2023.
- [48] H. Vanderbyl, “rwkv-cpp-accelerated,” https://github.com/harrisonvanderbyl/rwkv-cpp-accelerated, 2023.
- [49] Z. Huang, “rwkv-cpp-server,” https://github.com/ZeldaHuang/rwkv-cpp-server, 2023.
- [50] szxysdt, “rwkv_tokenizer_cpp,” https://github.com/szxysdt/rwkv_tokenizer_cpp, 2024.
- [51] cryscan, “web-rwkv,” https://github.com/cryscan/web-rwkv, 2023.
- [52] mrsteyk, “rwkvk-rs,” https://github.com/mrsteyk/rwkvk-rs, 2023.
- [53] C. Wirawan, “rwkv-tokenizer,” https://github.com/cahya-wirawan/rwkv-tokenizer, 2024.
- [54] Kerfuffle, “smolrsrwkv,” https://github.com/KerfuffleV2/smolrsrwkv, 2023.
- [55] Y. Zero, “rwkv-rust,” https://github.com/yorkzero831/rwkv-rust, 2023.
- [56] RWKV, “rwkv-tensorflow,” https://github.com/RWKV/rwkv-tensorflow, 2023.
- [57] 123mbcz123, “Rwkv5-tensorflow2.0,” https://github.com/123mbcz123/RWKV5-Tensorflow2.0, 2023.
- [58] ——, “Rwkv-tensorflow2.0,” https://github.com/123mbcz123/RWKV-Tensorflow2.0, 2023.
- [59] A. Videla, “Tensor-rwkv,” https://github.com/andvid72/Tensor-RWKV, 2024.
- [60] rj42, “rwkv-cuda,” https://github.com/rj42/rwkv-cuda, 2023.
- [61] P. Rouast, “tensorflow-rwkv,” https://github.com/prouast/tensorflow-rwkv, 2023.
- [62] breadbread1984, “Rwkv-tf2,” https://github.com/breadbread1984/RWKV-tf2, 2024.
- [63] nlpodyssey, “rwkv.f90,” https://github.com/nlpodyssey/rwkv.f90, 2023.
- [64] Y. Guan, “rwkv.jl,” https://github.com/guanyilun/rwkv.jl, 2023.
- [65] M. Whale, “rwkv.zig,” https://github.com/metalwhale/rwkv.zig, 2023.
- [66] nlpodyssey, “rwkv,” https://github.com/nlpodyssey/rwkv, 2023.
- [67] seasonjs, “rwkv,” https://github.com/seasonjs/rwkv, 2023.
- [68] J. Price, “go-rwkv.cpp,” https://github.com/donomii/go-rwkv.cpp, 2023.
- [69] nlpodyssey, “verbaflow,” https://github.com/nlpodyssey/verbaflow, 2023.
- [70] ZaraK0, “rwkvalgorithm,” https://github.com/ZaraK0/rwkvalgorithm, 2024.
- [71] Ronsor, “rwkv-tokenizer-go,” https://github.com/Ronsor/rwkv-tokenizer-go, 2024.
- [72] pass lin, “Rwkv6-keras,” https://github.com/pass-lin/RWKV6-Keras, 2024.
- [73] D. Jack, “keras-rwkv,” https://github.com/jackd/keras-rwkv, 2023.
- [74] HighCWu, “rwkv-paddle,” https://github.com/HighCWu/rwkv-paddle, 2023.
- [75] JunnYu, “Rwkv-v2-rnn-paddle,” https://github.com/JunnYu/RWKV-v2-RNN-paddle, 2022.
- [76] tensorpro, “tpu_rwkv,” https://github.com/tensorpro/tpu_rwkv, 2023.
- [77] mrsteyk, “Rwkv-lm-jax,” https://github.com/mrsteyk/RWKV-LM-jax, 2023.
- [78] saran, “rwkv-jax,” https://github.com/saran-gangster/rwkv-jax, 2024.
- [79] RWKV, “Rwkv-cpp-node,” https://github.com/RWKV/RWKV-cpp-node, 2023.
- [80] ——, “Rwkv-tokenizer-node,” https://github.com/RWKV/RWKV-tokenizer-node, 2023.
- [81] dc-dc dc, “mlx-rwkv,” https://github.com/dc-dc-dc/mlx-rwkv, 2024.
- [82] OpenMOSE, “Rwkv5-lm-lora,” https://github.com/OpenMOSE/RWKV5-LM-LoRA, 2024.
- [83] ——, “Rwkv-infctx-trainer-lora,” https://github.com/OpenMOSE/RWKV-infctx-trainer-LoRA, 2024.
- [84] RWKV, “Rwkv-infctx-trainer,” https://github.com/RWKV/RWKV-infctx-trainer, 2023.
- [85] OpenMOSE, “Rwkv-infer,” https://github.com/OpenMOSE/RWKV-Infer, 2024.
- [86] T. Zhang, “Rwkv-android,” https://github.com/ZTMIDGO/RWKV-Android, 2023.
- [87] M. Sophia, “rwkv-ncnn,” https://github.com/MollySophia/rwkv-ncnn, 2023.
- [88] ——, “rwkv-qualcomm,” https://github.com/MollySophia/rwkv-qualcomm, 2024.
- [89] V. Crana, “rwkv.jni,” https://github.com/vaccovecrana/rwkv.jni, 2023.
- [90] RWKV6, “Rwkv6_keras_operator,” https://github.com/infiy-quine/RWKV6_Keras_Operator, 2024.
- [91] shengxia, “Rwkv_role_playing_api,” https://github.com/shengxia/RWKV_Role_Playing_API, 2023.
- [92] Hannibal046, “nanorwkv,” https://github.com/Hannibal046/nanoRWKV, 2023.
- [93] H. Vanderbyl, “rwkv_chatbot,” https://github.com/harrisonvanderbyl/rwkv_chatbot, 2023.
- [94] averyyan, “Rwkv-wechat-bot,” https://github.com/averyyan/RWKV-wechat-bot, 2024.
- [95] az13js, “rwkv_chat_command_line,” https://github.com/az13js/rwkv_chat_command_line, 2024.
- [96] A. Lommatzsch, B. Llanque, V. S. Rosenberg, S. A. M. Tahir, H. D. Boyadzhiev, and M. Walny, “Combining information retrieval and large language models for a chatbot that generates reliable, natural-style answers.” in LWDA, 2023, pp. 298–310.
- [97] Cryscan, “Eloise,” https://github.com/cryscan/eloise, 2023.
- [98] Bilibini, “Meow-ai,” https://github.com/bilibini/Meow-AI, 2024.
- [99] M. K. Akkari, “Espritchatbot-rasa-rwkv,” https://github.com/kimou6055/Espritchatbot-RASA-RWKV, 2023.
- [100] AIIRWKV, “Rwkv-rag,” https://github.com/AIIRWKV/RWKV-RAG, 2024.
- [101] MrTom34, “Chatrwkv-in-wechat-version-1,” https://github.com/MrTom34/ChatRWKV-in-wechat-Version-1, 2023.
- [102] L. Lin, “Infofusion,” https://github.com/LeoLin4258/Infofusion, 2024.
- [103] ms KuroNeko, “Rwkv-drama,” https://github.com/ms-KuroNeko/RWKV-Drama, 2024.
- [104] Y. Liu, “Rwkv_role_playing_with_gpt-sovits,” https://github.com/v3ucn/RWKV_Role_Playing_with_GPT-SoVITS, 2024.
- [105] G. Kosev, “notgpt,” https://github.com/spion/notgpt, 2023.
- [106] 123summertime, “ykkz,” https://github.com/123summertime/ykkz, 2023.
- [107] No-22-Github, “Easy_rwkv_webui,” https://github.com/No-22-Github/Easy_RWKV_webui, 2024.
- [108] K. ALOULOU, “Espitchatbot-rasa-raven,” https://github.com/karim-aloulou/Espitchatbot-RASA-RAVEN, 2023.
- [109] J. Zhang, “lala_rwkv_chatbot_2.0,” https://github.com/samttoo22-MewCat/lala_rwkv_chatbot_2.0, 2024.
- [110] ck unifr, “pdf_parsing,” https://github.com/ck-unifr/pdf_parsing, 2023.
- [111] Z. Chen, C. Li, X. Xie, and P. Dube, “Onlysportslm: Optimizing sports-domain language models with sota performance under billion parameters,” arXiv preprint arXiv:2409.00286, 2024.
- [112] Y. Lin, “Rwkv-classification,” https://github.com/yynil/RWKV-Classification, 2023.
- [113] jiawanfan yyds, “novel-rwkv_demo,” https://github.com/jiawanfan-yyds/novel-rwkv_demo, 2023.
- [114] R. AI, “ai-town-rwkv-proxy,” https://github.com/recursal/ai-town-rwkv-proxy, 2023.
- [115] P. Seitz, “Ravenwhisperer,” https://github.com/seitzquest/RavenWhisperer, 2023.
- [116] xiaol, “wenda-rwkv,” https://github.com/xiaol/wenda-RWKV, 2023.
- [117] M. K. Azzabi, “Avatario,” https://github.com/khhaliil/AVATARIO, 2023.
- [118] Ojiyumm, “rwkv_kg,” https://github.com/Ojiyumm/rwkv_kg, 2024.
- [119] M. Xia, J. Wilson, B. Goldstein, and R. Henao, “Contrastive learning for clinical outcome prediction with partial data sources,” Proceedings of machine learning research, vol. 235, p. 54156, 2024.
- [120] tomer9080, “Stock-prediction-using-rwkv,” https://github.com/tomer9080/Stock-Prediction-Using-RWKV, 2023.
- [121] C. Dong, Z. Cao, S. K. Zhou, and J. Liu, “Dft: A dual-branch framework of fluctuation and trend for stock price prediction,” arXiv preprint arXiv:2411.06065, 2024.
- [122] Z. Cao, J. Xu, C. Dong, P. Yu, and T. Bai, “Matcc: A novel approach for robust stock price prediction incorporating market trends and cross-time correlations,” in Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024, pp. 187–196.
- [123] L. Zhou and S. Li, “A code completion model based rwkv with bimodal pretraining,” 2023.
- [124] L. Zhou, Z. Xiao, and Z. Ning, “Rwkv-based encoder-decoder model for code completion,” in 2023 3rd International Conference on Electronic Information Engineering and Computer (EIECT). IEEE, 2023, pp. 425–428.
- [125] U. Yildirim, R. Dutta, B. Yildirim, and A. Vaidya, “Experimentation in content moderation using rwkv,” arXiv preprint arXiv:2409.03939, 2024.
- [126] D. Goldstein, F. Obeid, E. Alcaide, G. Song, and E. Cheah, “Goldfinch: High performance rwkv/transformer hybrid with linear pre-fill and extreme kv-cache compression,” arXiv preprint arXiv:2407.12077, 2024.
- [127] H. Hou and F. R. Yu, “Rwkv-ts: Beyond traditional recurrent neural network for time series tasks,” arXiv preprint arXiv:2401.09093, 2024.
- [128] Y. Wang, S. Wang, J. Zhang, K. Fan, J. Wu, Z. Jiang, and Y. Liu, “Temporal and interactive modeling for efficient human-human motion generation,” arXiv preprint arXiv:2408.17135, 2024.
- [129] L. Wang, “Rrwkv: capturing long-range dependencies in rwkv,” arXiv preprint arXiv:2306.05176, 2023.
- [130] H. Chen, R. Xie, X. Cui, Z. Yan, X. Wang, Z. Xuan, and K. Zhang, “Lkpnr: Large language models and knowledge graph for personalized news recommendation framework.” Computers, Materials & Continua, vol. 79, no. 3, 2024.
- [131] Y. Dong, T. Wu, and C. Song, “Optimizing robotic manipulation with decision-rwkv: A recurrent sequence modeling approach for lifelong learning,” arXiv preprint arXiv:2407.16306, 2024.
- [132] H. Luo, K. Wu, C. Dai, S. Ding, and X. Chen, “Prosg: Using prompt synthetic gradients to alleviate prompt forgetting of rnn-like language models,” arXiv preprint arXiv:2311.01981, 2023.
- [133] R.-J. Zhu, Q. Zhao, G. Li, and J. K. Eshraghian, “Spikegpt: Generative pre-trained language model with spiking neural networks,” arXiv preprint arXiv:2302.13939, 2023.
- [134] T. Tsuruga, “General population projection model with census population data,” Electronic Theses, Projects, and Dissertations., 2023.
- [135] R.-G. Dumitru, D. Peteleaza, and M. Surdeanu, “Enhancing transformer rnns with multiple temporal perspectives,” arXiv preprint arXiv:2402.02625, 2024.
- [136] K. Suzuki and T. Ogata, “Sensorimotor attention and language-based regressions in shared latent variables for integrating robot motion learning and llm,” arXiv preprint arXiv:2407.09044, 2024.
- [137] A. Drichel, B. von Querfurth, and U. Meyer, “A transfer learning-based training approach for dga classification,” in International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 2024, pp. 381–391.
- [138] X. Zhou and T. Chen, “Bsbp-rwkv: Background suppression with boundary preservation for efficient medical image segmentation,” in ACM Multimedia 2024, 2024.
- [139] Z. Yang, H. Zhang, D. Zhao, B. Wei, and Y. Xu, “Restore-rwkv: Efficient and effective medical image restoration with rwkv,” arXiv preprint arXiv:2407.11087, 2024.
- [140] G. Ji and P. Liu, “Rnn-based multiple instance learning for the classification of histopathology whole slide images,” in International Conference on Medical Imaging and Computer-Aided Diagnosis. Springer, 2023, pp. 329–339.
- [141] Z. Liu, J. Hou, X. Wang, X. Ye, J. Wang, H. Zhao, and X. Bai, “Lion: Linear group rnn for 3d object detection in point clouds,” arXiv preprint arXiv:2407.18232, 2024.
- [142] Q. He, J. Zhang, J. Peng, H. He, Y. Wang, and C. Wang, “Pointrwkv: Efficient rwkv-like model for hierarchical point cloud learning,” arXiv preprint arXiv:2405.15214, 2024.
- [143] J. Wang, W. Yin, X. Long, X. Zhang, Z. Xing, X. Guo, and Q. Zhang, “Occrwkv: Rethinking efficient 3d semantic occupancy prediction with linear complexity,” arXiv preprint arXiv:2409.19987, 2024.
- [144] Z. Zhu, W. Shao, and D. Jiao, “Tls-rwkv: Real-time online action detection with temporal label smoothing,” Neural Processing Letters, vol. 56, no. 2, p. 57, 2024.
- [145] W.-D. Jiang, C.-Y. Chang, H.-C. Chang, and D. S. Roy, “From explicit rules to implicit reasoning in an interpretable violence monitoring system,” arXiv preprint arXiv:2410.21991, 2024.
- [146] D. Valsesia, T. Bianchi, and E. Magli, “Hybrid recurrent-attentive neural network for onboard predictive hyperspectral image compression,” in IGARSS 2024-2024 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 2024, pp. 7898–7902.
- [147] H. Yuan, X. Li, L. Qi, T. Zhang, M.-H. Yang, S. Yan, and C. C. Loy, “Mamba or rwkv: Exploring high-quality and high-efficiency segment anything model,” arXiv preprint arXiv:2406.19369, 2024.
- [148] Y. Duan, W. Wang, Z. Chen, X. Zhu, L. Lu, T. Lu, Y. Qiao, H. Li, J. Dai, and W. Wang, “Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures,” arXiv preprint arXiv:2403.02308, 2024.
- [149] Z. Li and H. Hou, “Visualrwkv-hd and uhd: Advancing high-resolution processing for visual language models,” arXiv preprint arXiv:2410.11665, 2024.
- [150] Z. Yin, C. Li, and X. Dong, “Video rwkv: Video action recognition based rwkv,” arXiv preprint arXiv:2411.05636, 2024.
- [151] T. Gu, K. Yang, X. An, Z. Feng, D. Liu, W. Cai, and J. Deng, “Rwkv-clip: A robust vision-language representation learner,” arXiv preprint arXiv:2406.06973, 2024.
- [152] S. Yang, H. Ma, C. Yu, A. Wang, and E.-P. Li, “Sdit: Spiking diffusion model with transformer,” arXiv preprint arXiv:2402.11588, 2024.
- [153] B. Xu, X. Wang, S. Li, J. Li, and C. Liu, “Social-cvae: Pedestrian trajectory prediction using conditional variational auto-encoder,” in International Conference on Neural Information Processing. Springer, 2023, pp. 476–489.
- [154] Z. Fei, M. Fan, C. Yu, D. Li, and J. Huang, “Diffusion-rwkv: Scaling rwkv-like architectures for diffusion models,” arXiv preprint arXiv:2404.04478, 2024.
- [155] Y. Du, T. Hu, J. Zhang, R. Y. C. Xu, X. Hu, K. Wu, D. Luo, Y. Wang, and L. Ma, “Exploring real&synthetic dataset and linear attention in image restoration,” 2024. [Online]. Available: https://arxiv.org/abs/2412.03814
- [156] M. Dai, Q. Zhou, and L. Ma, “Stylerwkv: High-quality and high-efficiency style transfer with rwkv-like architecture,” arXiv preprint arXiv:2412.19535, 2024.
- [157] L. Vierling, “Facerwkv,” https://github.com/lukasVierling/FaceRWKV, 2023.
- [158] lll143653, “rwkv-denoise,” https://github.com/lll143653/rwkv-denoise, 2024.
- [159] T. Zhang, “Android-rwkv-midi,” https://github.com/ZTMIDGO/Android-RWKV-MIDI, 2023.
- [160] Patchbanks, “Rwkv-v4-midi,” https://github.com/patchbanks/RWKV-v4-MIDI, 2024.
- [161] agreene5, “Procedural-purgatory,” https://github.com/agreene5/Procedural-Purgatory, 2024.
- [162] S. Liu, X. Fan, and G. Wu, “Why perturbing symbolic music is necessary: Fitting the distribution of never-used notes through a joint probabilistic diffusion model,” arXiv preprint arXiv:2408.01950, 2024.
- [163] L. Zuo, K. An, S. Zhang, and Z. Yan, “Advancing vad systems based on multi-task learning with improved model structures,” arXiv preprint arXiv:2312.14860, 2023.
- [164] K. An and S. Zhang, “Exploring rwkv for memory efficient and low latency streaming asr,” arXiv preprint arXiv:2309.14758, 2023.
- [165] J. Jiang, “Rwkv-asr,” https://github.com/AGENDD/RWKV-ASR, 2024.
- [166] X. Du, “Audiorwkv,” https://github.com/diggerdu/AudioRWKV, 2024.
- [167] A. Suller, “Rwkv-musicgenerator,” https://github.com/asuller/RWKV-MusicGenerator, 2023.
- [168] AverageJoe9, “Music-genre-classification-rwkv,” https://github.com/AverageJoe9/Music-Genre-Classification-RWKV, 2024.
- [169] J. Jiang, “Rwkv-speechchat,” https://github.com/AGENDD/RWKV-SpeechChat, 2024.
- [170] YuChuXi, “Mozi-rwkv,” https://github.com/YuChuXi/MoZi-RWKV, 2024.
- [171] Y. Liu, “Rwkv_3b_7b_webui_gpt-sovits,” https://github.com/v3ucn/RWKV_3B_7B_Webui_GPT-SoVITS, 2024.
- [172] Ai00-X, “ai00_server,” https://github.com/Ai00-X/ai00_server, 2024.
- [173] josephrocca, “rwkv-v4-web,” https://github.com/josephrocca/rwkv-v4-web, 2023.
- [174] shengxia, “Rwkv_role_playing,” https://github.com/shengxia/RWKV_Role_Playing, 2023.
- [175] yuunnn w, “Rwkv_qqbot_backend,” https://github.com/yuunnn-w/RWKV_QQBot_BackEnd, 2024.
- [176] J. LI, “web-rwkv-axum,” https://github.com/Prunoideae/web-rwkv-axum, 2024.
- [177] H. Felix, “Webchatrwkvstic,” https://github.com/hizkifw/WebChatRWKVstic, 2023.
- [178] riversun, “chatux-server-rwkv,” https://github.com/riversun/chatux-server-rwkv, 2023.
- [179] t4wefan, “Chatrwkv-flask-api,” https://github.com/t4wefan/ChatRWKV-flask-api, 2023.
- [180] StarDreamAndFeng, “Chatrwkv-webui,” https://github.com/StarDreamAndFeng/ChatRWKV-webui, 2023.
- [181] RafaRed, “Rwkv-api,” https://github.com/RafaRed/RWKV-api, 2023.
- [182] Y. Liu, “rwkv.cpp_webui_macos,” https://github.com/v3ucn/rwkv.cpp_webui_Macos, 2024.
- [183] shengxia, “Rwkv_role_playing_ui,” https://github.com/shengxia/RWKV_Role_Playing_UI, 2023.
- [184] Y. Liu, “rwkv.cpp_webui_gpt-sovits,” https://github.com/v3ucn/rwkv.cpp_webui_GPT-SoVITS, 2024.
- [185] E. Akyürek, B. Wang, Y. Kim, and J. Andreas, “In-context language learning: Arhitectures and algorithms,” arXiv preprint arXiv:2401.12973, 2024.
- [186] J. Yuan, H. Liu, Y.-N. Chuang, S. Li, G. Wang, D. Le, H. Jin, V. Chaudhary, Z. Xu, Z. Liu et al., “Kv cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches,” arXiv preprint arXiv:2407.01527, 2024.
- [187] J. Li, M. Wang, Z. Zheng, and M. Zhang, “Loogle: Can long-context language models understand long contexts?” arXiv preprint arXiv:2311.04939, 2023.
- [188] C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, and B. Ginsburg, “Ruler: What’s the real context size of your long-context language models?” arXiv preprint arXiv:2404.06654, 2024.
- [189] F. Lei, Q. Liu, Y. Huang, S. He, J. Zhao, and K. Liu, “S3eval: A synthetic, scalable, systematic evaluation suite for large language models,” arXiv preprint arXiv:2310.15147, 2023.
- [190] A. Patel, S. Bhattamishra, S. Reddy, and D. Bahdanau, “Magnifico: Evaluating the in-context learning ability of large language models to generalize to novel interpretations,” arXiv preprint arXiv:2310.11634, 2023.
- [191] P. Ding, J. Fang, P. Li, K. Wang, X. Zhou, M. Yu, J. Li, M. R. Walter, and H. Mei, “Mango: A benchmark for evaluating mapping and navigation abilities of large language models,” arXiv preprint arXiv:2403.19913, 2024.
- [192] L. Xu, A. Li, L. Zhu, H. Xue, C. Zhu, K. Zhao, H. He, X. Zhang, Q. Kang, and Z. Lan, “Superclue: A comprehensive chinese large language model benchmark,” arXiv preprint arXiv:2307.15020, 2023.
- [193] B. Zhang, H. Xie, P. Du, J. Chen, P. Cao, Y. Chen, S. Liu, K. Liu, and J. Zhao, “Zhujiu: A multi-dimensional, multi-faceted chinese benchmark for large language models,” arXiv preprint arXiv:2308.14353, 2023.
- [194] J. Yi, Y. Xie, B. Zhu, K. Hines, E. Kiciman, G. Sun, X. Xie, and F. Wu, “Benchmarking and defending against indirect prompt injection attacks on large language models,” arXiv preprint arXiv:2312.14197, 2023.
- [195] Y. Chen, B. Xu, Q. Wang, Y. Liu, and Z. Mao, “Benchmarking large language models on controllable generation under diversified instructions,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, pp. 17 808–17 816.
- [196] M. Lanctot, K. Larson, Y. Bachrach, L. Marris, Z. Li, A. Bhoopchand, T. Anthony, B. Tanner, and A. Koop, “Evaluating agents using social choice theory,” arXiv preprint arXiv:2312.03121, 2023.
- [197] T. Wei, J. Luan, W. Liu, S. Dong, and B. Wang, “Cmath: Can your language model pass chinese elementary school math test?” arXiv preprint arXiv:2306.16636, 2023.
- [198] K. Sun, Y. E. Xu, H. Zha, Y. Liu, and X. L. Dong, “Head-to-tail: How knowledgeable are large language models (llm),” AKA will llms replace knowledge graphs, 2023.
- [199] Z. Chu, Q. Ai, Y. Tu, H. Li, and Y. Liu, “Pre: A peer review based large language model evaluator,” arXiv preprint arXiv:2401.15641, 2024.
- [200] C. Zhu, N. Chen, Y. Gao, and B. Wang, “Evaluating llms at evaluating temporal generalization,” arXiv preprint arXiv:2405.08460, 2024.
- [201] T. Li, G. Zhang, Q. D. Do, X. Yue, and W. Chen, “Long-context llms struggle with long in-context learning,” arXiv preprint arXiv:2404.02060, 2024.
- [202] Ojiyumm, “Longbench_rwkv,” https://github.com/Ojiyumm/LongBench_RWKV, 2024.
- [203] X. Du, “rwkv-long-range-arena,” https://github.com/diggerdu/rwkv-long-range-arena, 2023.
- [204] Ojiyumm, “Mt_bench_rwkv,” https://github.com/Ojiyumm/MT_BENCH_RWKV, 2024.
- [205] I. Lee, N. Jiang, and T. Berg-Kirkpatrick, “Is attention required for icl? exploring the relationship between model architecture and in-context learning ability,” in The Twelfth International Conference on Learning Representations, 2024.
- [206] J. A. Michaelov, C. Arnett, and B. K. Bergen, “Revenge of the fallen? recurrent models match transformers at predicting human language comprehension metrics,” arXiv preprint arXiv:2404.19178, 2024.
- [207] K. Grm, “Veliki jezikovni modeli so strojni ucenci v casu sklepanja,” 2023.
- [208] Z. Zhou, G. He, Z. Zhang, L. Leng, Q. Guo, J. Liao, X. Song, and R. Cheng, “Benchmarking neural decoding backbones towards enhanced on-edge ibci applications,” arXiv preprint arXiv:2406.06626, 2024.
- [209] G. Paulo, T. Marshall, and N. Belrose, “Does transformer interpretability transfer to rnns?” arXiv preprint arXiv:2404.05971, 2024.
- [210] R. Vacareanu, V.-A. Negru, V. Suciu, and M. Surdeanu, “From words to numbers: Your large language model is secretly a capable regressor when given in-context examples,” arXiv preprint arXiv:2404.07544, 2024.
- [211] J. Huang, “How well can a long sequence model model long sequences? comparing architechtural inductive biases on long-context abilities,” arXiv preprint arXiv:2407.08112, 2024.
- [212] Ádám Márk Biró, “compare_llms,” https://github.com/BiroAd/compare_llms, 2024.
- [213] M. Chen and E. Li, “Spiking mixers for robust and energy-efficient vision-and-language learning,” 2024. [Online]. Available: https://openreview.net/forum?id=FyZaVdQLdJ
- [214] StarRing, “Minirwkv-4,” https://github.com/StarRing2022/MiniRWKV-4, 2023.
- [215] H. Hou, P. Zeng, F. Ma, and F. R. Yu, “Visualrwkv: Exploring recurrent neural networks for visual language models,” arXiv preprint arXiv:2406.13362, 2024.
- [216] StarRing, “Dlip-rwkv,” https://github.com/StarRing2022/Dlip-RWKV, 2023.
- [217] Y. Lin, “Rwkv_lm_ext,” https://github.com/yynil/RWKV_LM_EXT, 2024.
- [218] I. Neon, “rwkv-reward-enhanced,” https://github.com/Mazidad/rwkv-reward-enhanced, 2023.
- [219] Max-SF1, “Bot-ani-rwkv-twitter-bot-detection,” https://github.com/Max-SF1/Bot-Ani-RWKV-twitter-bot-detection, 2024.
- [220] I. Zimerman, A. Ali, and L. Wolf, “A unified implicit attention formulation for gated-linear recurrent sequence models,” arXiv preprint arXiv:2405.16504, 2024.
- [221] JL-er, “Rwkv-peft,” https://github.com/JL-er/RWKV-PEFT, 2024.
- [222] Seikaiju, “Rwkv-peft-simple,” https://github.com/Seikaijyu/RWKV-PEFT-Simple, 2024.
- [223] JL-er, “Rwkv5-infctxlm,” https://github.com/JL-er/RWKV5-infctxLM, 2023.
- [224] S. Design and L. Lab, “Decisionrwkv,” https://github.com/ancorasir/DecisionRWKV, 2024.
- [225] LeC-Z, “Rwkv-nonogram,” https://github.com/LeC-Z/RWKV-nonogram, 2024.
- [226] S. Li, “Rwkv_rul,” https://github.com/ShixiangLi/RWKV_RUL, 2024.
- [227] SynthiaDL, “Trainchatgalrwkv,” https://github.com/SynthiaDL/TrainChatGalRWKV, 2023.
- [228] wozeparrot, “tinyrwkv,” https://github.com/wozeparrot/tinyrwkv, 2023.
- [229] Nintorac, “simple_rwkv,” https://github.com/Nintorac/simple_rwkv, 2023.
- [230] josStorer, “Rwkv-runner,” https://github.com/josStorer/RWKV-Runner, 2024.
- [231] Atome-FE, “llama-node,” https://github.com/Atome-FE/llama-node, 2023.
- [232] H. Vanderbyl, “godot-rwkv,” https://github.com/harrisonvanderbyl/godot-rwkv, 2024.
- [233] StarRing, “Hf-for-rwkvraven-alpaca,” https://github.com/StarRing2022/HF-For-RWKVRaven-Alpaca, 2023.
- [234] J. Kong, “rwkv_reward,” https://github.com/jiamingkong/rwkv_reward, 2023.
- [235] typoverflow, “Decision-rwkv,” https://github.com/typoverflow/Decision-RWKV, 2023.
- [236] OpenMOSE, “Rwkv-lm-rlhf,” https://github.com/OpenMOSE/RWKV-LM-RLHF, 2024.
- [237] ——, “Rwkv-lm-state-4bit-orpo,” https://github.com/OpenMOSE/RWKV-LM-State-4bit-Orpo, 2024.
- [238] J. Thapa, “Modern sequence models in context of multi-agent reinforcement learning,” 2024.
- [239] D. Tarasov, “Rwkv-finetune-script,” https://github.com/Durham/RWKV-finetune-script, 2023.
- [240] StarRing, “Hf-for-rwkvworld-loraalpaca,” https://github.com/StarRing2022/HF-For-RWKVWorld-LoraAlpaca, 2023.
- [241] if001, “Rwkv-lm-lora-ja,” https://github.com/if001/RWKV-LM-LoRA-ja, 2023.
- [242] ssg qwq, “Rwkv-light-api,” https://github.com/ssg-qwq/RWKV-Light-API, 2023.
- [243] Jellyfish042, “Rwkv-statetuning,” https://github.com/Jellyfish042/RWKV-StateTuning, 2024.
- [244] 00ffcc, “chunkrwkv6,” https://github.com/00ffcc/chunkRWKV6, 2024.
- [245] UnstoppableCurry, “Rwkv-lm-interpretability-research,” https://github.com/UnstoppableCurry/RWKV-LM-Interpretability-Research, 2023.
- [246] cryscan, “web-rwkv-inspector,” https://github.com/cryscan/web-rwkv-inspector, 2024.
- [247] J. LI, “rwkv_umap,” https://github.com/Prunoideae/rwkv_umap, 2023.
- [248] D. wanna M, “rwkv-tensorrt,” https://github.com/Dan-wanna-M/rwkv-tensorrt, 2023.
- [249] guinmoon, “Llmfarm,” https://github.com/guinmoon/LLMFarm, 2023.
- [250] J. Kong, “Rwkv_chains,” https://github.com/jiamingkong/RWKV_chains, 2023.
- [251] neromous, “Rwkv-ouroboros-app,” https://github.com/neromous/RWKV-Ouroboros-app, 2023.
- [252] K. Wegrzyniak, “Altaeraai,” https://github.com/latestissue/AltaeraAI, 2024.
- [253] Y. Lin, “Rwkvinllama,” https://github.com/yynil/RWKVinLLAMA, 2024.
- [254] F. Mom, “Gptq-for-rwkv,” https://github.com/3outeille/GPTQ-for-RWKV, 2023.
- [255] Alic-Li, “Rwkv-lm-amd-radeon-rocm-hip,” https://github.com/Alic-Li/RWKV-LM-AMD-Radeon-ROCm-hip, 2024.
- [256] SingingRivulet, “Innerdnn,” https://github.com/SingingRivulet/InnerDNN, 2023.
- [257] Ow1onp, “Easychat-server,” https://github.com/Ow1onp/EasyChat-Server, 2024.
- [258] mosterwei13, “Chatrwkv_pc,” https://github.com/mosterwei13/ChatRWKV_PC, 2023.
- [259] Z. Jiang, Y. Zhang, C. Liu, J. Zhao, and K. Liu, “Generative calibration for in-context learning,” arXiv preprint arXiv:2310.10266, 2023.
- [260] Q. Fu, H. Yuan, Y. Hou, and X. Shen, “Linear attention based channel estimation scheme for v2x communications,” in 2024 8th International Conference on Communication and Information Systems (ICCIS). IEEE, 2024, pp. 96–101.
- [261] W. Choe, Y. Ji, and F. Lin, “Rwkv-edge: Deeply compressed rwkv for resource-constrained devices,” arXiv preprint arXiv:2412.10856, 2024.
- [262] Z. Li, P. Tang, X. Wang, X. Liu, and P. Mou, “Pcf-rwkv: Product carbon footprint estimation system based on large language model,” 2024.
- [263] J. Zhang, Z. Cheng, Y. Zhao, S. Wang, D. Zhou, G. Lu, and L. Song, “L3tc: Leveraging rwkv for learned lossless low-complexity text compression,” arXiv preprint arXiv:2412.16642, 2024.
- [264] lovebull, “Plantflowerdatasets,” https://github.com/lovebull/PlantFlowerDatasets, 2023.
- [265] A. Graves and A. Graves, “Long short-term memory,” Supervised sequence labelling with recurrent neural networks, pp. 37–45, 2012.
- [266] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv preprint arXiv:1412.3555, 2014.
- [267] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” IEEE transactions on Signal Processing, vol. 45, no. 11, pp. 2673–2681, 1997.
- [268] L. Feng, F. Tung, M. O. Ahmed, Y. Bengio, and H. Hajimirsadeghi, “Were rnns all we needed?” 2024. [Online]. Available: https://arxiv.org/abs/2410.01201
- [269] M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter, “xlstm: Extended long short-term memory,” 2024. [Online]. Available: https://arxiv.org/abs/2405.04517
- [270] S. Zhai, W. Talbott, N. Srivastava, C. Huang, H. Goh, R. Zhang, and J. Susskind, “An attention free transformer,” arXiv preprint arXiv:2105.14103, 2021.
- [271] B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou et al., “Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence,” arXiv preprint arXiv:2404.05892, 2024.
- [272] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9
- [273] A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023.
- [274] M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré, “Hyena hierarchy: Towards larger convolutional language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 28 043–28 078.
- [275] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021.