2604.07628

# Trilinear Compute-in-Memory Architecture for Energy-Efficient Transformer Acceleration **Authors**: Md Zesun Ahmed Mia, Jiahui Duan, Kai Ni, Abhronil Sengupta > The Pennsylvania State University Department of Electrical Engineering University Park PA USA > University of Notre Dame Department of Electrical Engineering Notre Dame IN USA (2026) ## Abstract Self-attention in Transformers generates dynamic operands that force conventional Compute-in-Memory (CIM) accelerators into costly non-volatile memory (NVM) reprogramming cycles, degrading throughput and stressing device endurance. Existing solutions either reduce but retain NVM writes through matrix decomposition or sparsity, or move attention computation to digital CMOS at the expense of NVM density. We present TrilinearCIM, a Double-Gate FeFET (DG-FeFET)-based architecture that uses back-gate modulation to realize a three-operand multiply-accumulate primitive for in-memory attention computation without dynamic ferroelectric reprogramming. Evaluated on BERT-base (GLUE) and ViT-base (ImageNet and CIFAR), TrilinearCIM outperforms conventional CIM on seven of nine GLUE tasks while achieving up to 46.6% energy reduction and 20.4% latency improvement over conventional FeFET CIM at 37.3% area overhead. To our knowledge, this is the first architecture to perform complete Transformer attention computation exclusively in NVM cores without runtime reprogramming. Compute-in-memory, transformer acceleration, ferroelectric FET, attention mechanism, energy-efficient computing, neuromorphic computing copyright: none journalyear: 2026 doi: XXXXXXX.XXXXXXX ## 1. Introduction The Transformer architecture (Vaswani et al., 2017) has transformed the landscape of deep learning, shifting the paradigm from recurrent and convolutional inductive biases to mechanisms based purely on self-attention. This shift has enabled unprecedented scaling in Natural Language Processing (NLP), with models such as BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020) achieving state-of-the-art results by scaling to massive parameter counts (e.g., 175 billion parameters). Following this success in NLP, the Transformer architecture has successfully expanded into Computer Vision with the Vision Transformer (ViT) (Dosovitskiy et al., 2021) and hierarchical variants like the Swin Transformer (Liu et al., 2021), proving that attention-based models can outperform Convolutional Neural Networks (CNNs) at scale. However, this performance comes at a massive computational cost. The quadratic complexity of the self-attention mechanism ( $O(N^2)$ with respect to sequence length) combined with massive parameter counts has led to skyrocketing energy consumption for both training and inference, raising significant concerns regarding carbon footprint and sustainability (Strubell et al., 2019). In response to these challenges, the research community has actively pursued algorithmic optimizations to tame the computational overhead. Recent surveys on efficient transformers (Tay et al., 2022) detail software-level innovations—ranging from sparse attention patterns and low-rank linear approximations to token pruning strategies—that actively reduce the quadratic complexity and FLOPs count. On the hardware side, dedicated accelerators such as A 3 (Ham et al., 2020) exploit token and head sparsity to accelerate attention, while hardware-aware approaches such as HAT search (Wang et al., 2020) and VAQF co-design (Sun et al., 2022) automate the search for efficient architectures tailored to specific constraints. Yet these approaches remain bounded by the separation of memory and compute in von Neumann architectures. At a deeper level, all these optimizations operate within an inherently two-operand computational model—one that maps only two operands (input and weight) per operation. The self-attention mechanism, by contrast, requires computing products of three dynamically-generated matrices ( $Q$ , $K$ , $V$ ), none of which are static weights. Standard hardware lacks a physical “third operand” pathway to support this three-way interaction natively, creating a structural mismatch between the two-operand hardware primitive and the dynamic multi-operand dataflow required by self-attention. To overcome the von Neumann bottleneck, Compute-in-Memory (CIM) has emerged as a hardware paradigm that eliminates costly weight data movement. By leveraging the physical properties of resistive switching devices—such as resistive random-access memory (ReRAM), phase-change memory (PCM), and ferroelectric field-effect transistors (FeFETs)—CIM arrays perform matrix-vector multiplication (MVM) in-situ using analog current summation (output current $I=G· V$ ), where conductance ( $G$ ) encodes the weight and voltage ( $V$ ) encodes the input (Sebastian et al., 2020; Ielmini and Wong, 2018). Pioneering architectures such as PRIME (Chi et al., 2016), ISAAC (Shafiee et al., 2016), and PUMA (Ankit et al., 2019) demonstrated orders-of-magnitude energy-delay product improvements for CNN workloads. However, CIM relies on a critical assumption: weight stationarity. In CNNs and standard multi-layer perceptrons (MLPs), weight matrices are static after training, allowing CIM accelerators to program the non-volatile memory once and reuse it for millions of inference cycles. Transformers violate this assumption: the Self-Attention mechanism generates key operands ( $Q,K,V$ ) dynamically from every input, forcing a “Compute-Write-Compute” cycle that repeatedly reprograms NVM arrays. Because NVM writes are orders of magnitude slower, more energy-intensive, and endurance-limited compared to reads (Peng et al., 2019; Zhang et al., 2024), this reprogramming dominates execution time and degrades device lifetime—a bottleneck we quantify in Section 3. Architectural solutions such as ReTransformer (Yang et al., 2020) and TransPIM (Zhou et al., 2022b) have attempted to mitigate the bottleneck through matrix decomposition and dataflow optimization, but cannot eliminate the writes entirely. Recognizing the insurmountable endurance and latency penalties of NVM writes, recent state-of-the-art accelerators have pivoted towards hybrid architectures. A prime example is X-Former (Sridharan et al., 2023), which partitions the workload: it uses NVM tiles for the static projection weights (where CIM excels) but retreats to digital CMOS engines for the attention mechanism ( $Q· K^T$ ) to avoid writing to NVM. While this hybrid approach effectively solves the endurance problem, it represents a significant retreat from the promise of “All-in-Memory” computing. By reverting to CMOS for the attention score computation ( $Q· K^T$ , which scales quadratically with sequence length), these designs sacrifice the superior area density and leakage characteristics of NVM—accepting a trade-off: solve Endurance by sacrificing Area/Efficiency. To resolve this dichotomy, we propose TrilinearCIM, an architecture that extends the conventional FeFET-based CIM model. A ferroelectric field-effect transistor (FeFET) stores a non-volatile conductance state in its ferroelectric gate stack, making it well suited for weight-stationary CIM. However, a standard single-gate FeFET still exposes only the usual two operands: stored conductance and applied input voltage. We therefore adopt a double-gate FeFET (DG-FeFET) structure (Mulaosmanovic et al., 2021; Jiang et al., 2022, 2025), in which a ferroelectric top gate stores the non-volatile weight while a non-ferroelectric back gate provides an independent volatile modulation path. This additional control terminal creates the third operand pathway needed by TrilinearCIM to execute the complete attention dataflow in-memory without runtime NVM reprogramming. Our specific contributions are as follows. (1) Novel Trilinear CIM Primitive. We propose a three-operand Compute-in-Memory operation ( $Y=A· B· C$ ) enabled by the electrical properties of the DG-FeFET back-gate. This extends the standard two-operand ( $Input·Weight$ ) CIM paradigm to natively support three-operand operations. (2) Elimination of Dynamic NVM Writes. By mapping static weights ( $W_Q,W_K,W_V$ ) to the non-volatile top gate and dynamic activation vectors ( $X,X^T$ ) to the volatile back-gate voltage ( $V_BG$ ), TrilinearCIM computes the attention mechanism entirely within the memory array without ever reprogramming the ferroelectric state. (3) Simplified Dataflow and Reduced Buffer Pressure. Fusing the projection and attention steps into trilinear stages reshapes the dataflow. The intermediate dynamic matrices are never stored in the global buffer, reducing the number of matrices that must reside simultaneously in the buffer from three ( $X$ , $Q$ , $K$ ) to one ( $X$ ), thereby lowering buffer capacity requirements by approximately $3×$ . (4) Recovery of Compute Efficiency. By removing dynamic write overhead from the critical path, our evaluation shows that TrilinearCIM significantly reduces both energy consumption and latency compared to conventional CIM baselines. This restores the competitive advantage of “All-in-Memory” computing for Transformer workloads. ## 2. Background ### 2.1. Transformer Architecture and Self-Attention The Standard Transformer architecture is composed of stacked encoder blocks, where each block processes an input sequence $X∈ℝ^N× d$ (with $N$ tokens and embedding dimension $d$ ) through two sub-layers: Multi-Head Self-Attention (MHSA) and a Feed-Forward Network (FFN), as illustrated in Fig. 1. Both sub-layers employ residual connections (denoted by circled “+” symbols) and Layer Normalization (LN), resulting in the following dataflow: $$ \displaystyle Z \displaystyle=LN(X+MHSA(X)) \displaystyle Y \displaystyle=LN(Z+FFN(Z)) \tag{1} $$ While the FFN (Eq. 2) relies purely on static weight matrices that are inherently compatible with standard CIM, the MHSA (Eq. 1) introduces dynamic data dependencies. <details> <summary>2604.07628v1/x1.png Details</summary>  ### Visual Description ## Diagram: Transformer Model Architecture ### Overview The diagram illustrates the architecture of a transformer model, highlighting two core components: **Multi-Head Self-Attention** (purple section) and **Feed-Forward** (pink section). Input `X` flows through the architecture, with residual connections and layer normalization applied at each stage. ### Components/Axes - **Input**: Labeled as `Input X` at the top. - **Multi-Head Self-Attention**: - **Weight Matrices**: `W_Q`, `W_K`, `W_V` (purple blocks) generate query (`Q`), key (`K`), and value (`V`) vectors. - **Scaled Dot-Product Attention**: Combines `Q`, `K`, and `V` (purple block). - **Concat + Linear**: Merges attention outputs with a linear layer (purple block). - **LayerNorm**: Normalizes the output (black block). - **Residual Connection**: Adds the original input to the normalized output (black `+` symbol). - **Feed-Forward**: - **Linear → GELU → Linear**: Two linear layers with a GELU activation (pink blocks). - **LayerNorm**: Normalizes the output (black block). - **Residual Connection**: Adds the original input to the normalized output (black `+` symbol). - **Output**: Final output at the bottom. - **Legend**: Located on the right, with: - **Purple**: Multi-Head Self-Attention components. - **Pink**: Feed-Forward components. - **Black**: Residual connections and layer normalization. ### Detailed Analysis - **Multi-Head Self-Attention**: - Input `X` is transformed into `Q`, `K`, and `V` via weight matrices `W_Q`, `W_K`, and `W_V`. - The **Scaled Dot-Product Attention** computes attention scores between `Q` and `K`, scales them, and applies softmax to derive weights for `V`. - The output is concatenated with a linear layer to produce a combined representation. - **Feed-Forward**: - The input passes through two linear layers with a GELU activation, introducing non-linearity. - The output is normalized via **LayerNorm** and combined with the original input via a residual connection. ### Key Observations - **Residual Connections**: Both the Multi-Head and Feed-Forward sections use residual connections (black `+` symbols), enabling gradient flow and mitigating vanishing gradients. - **Layer Normalization**: Applied after each sub-layer (Multi-Head and Feed-Forward) to stabilize training. - **Color Coding**: Purple and pink blocks visually separate the two core components, aiding in distinguishing their roles. ### Interpretation The diagram represents a standard transformer block, emphasizing its modular design: 1. **Self-Attention**: Captures contextual relationships between input elements via `Q`, `K`, and `V`. 2. **Feed-Forward**: Processes the attended representation with non-linear transformations. 3. **Residual Connections**: Allow the model to learn incremental improvements while preserving input information. 4. **Layer Normalization**: Ensures stable training by normalizing activations. This architecture underpins modern NLP models, enabling parallel processing and scalability. The absence of numerical data suggests this is a conceptual diagram, not a performance metric visualization. </details> Figure 1. Transformer Encoder Block architecture. The input sequence $X$ is projected through learned weight matrices $W_Q$ , $W_K$ , $W_V$ to produce Query ( $Q$ ), Key ( $K$ ), and Value ( $V$ ) representations. These are processed by Scaled Dot-Product Attention (Eq. 4), followed by concatenation and linear projection to form the Multi-Head Self-Attention output. The Feed-Forward Network applies two linear transformations with GELU activation. Residual connections (circled “+”) add the sub-layer input to its output before Layer Normalization (LayerNorm). Mathematically, the attention mechanism first projects the input into Query ( $Q$ ), Key ( $K$ ), and Value ( $V$ ) subspaces as shown in Eq. 3: $$ Q=XW_Q^T, K=XW_K^T, V=XW_V^T \tag{3} $$ where $W_Q,W_K,W_V∈ℝ^d_k× d$ are learnable parameters, and $d_k$ is the dimensionality of the key subspace. The core attention scores are then computed via a scaled dot-product (Eq. 4): $$ Attention(Q,K,V)=softmax≤ft(\frac{QK^T}{√{d_k}}\right)V \tag{4} $$ Here, the matrix multiplication $QK^T$ in Eq. 4 represents the token-to-token correlation. Unlike the projection steps where weights are static, both operands in this equation differ for every input sample. Finally, Multi-Head Attention aggregates outputs from $h$ parallel heads, each operating on a $d_k$ -dimensional subspace: $$ MultiHead(Q,K,V)=Concat(head_1,\dots,head_h)W_O \tag{5} $$ where $W_O∈ℝ^d× d$ is the output projection matrix. As defined in Eq. 5, this formulation underscores the distinct requirements of the two sub-layers: static weights for projections/FFNs, and dynamic variable-variable multiplication for attention scores. ### 2.2. CIM and the DG-FeFET Device To execute these matrix operations efficiently, Compute-in-Memory (CIM) arrays exploit analog physics. In a resistive memory array, the current through each device, $I$ , follows Ohm’s Law ( $I=G· V$ ), where $G$ is the programmable conductance (representing a weight) and $V$ is the applied voltage (representing an input). By summing currents along the columns via Kirchhoff’s Current Law, the array computes the dot product in a single step: $$ I_j=∑_iG_ij· V_i \tag{6} $$ where $G_ij$ is the cell conductance at row $i$ and column $j$ , and $V_i$ is the input voltage applied to row $i$ . The resulting current $I_j$ collected along column $j$ represents the dot-product output. This eliminates the repeated movement of stored weights from memory to compute units, which dominates energy consumption in conventional von Neumann systems. However, standard CIM devices still support only two operands per operation. To support the trilinear computations used in our attention dataflow ( $A· B· C$ ), we leverage the DG-FeFET device introduced in Section 1, whose back-gate provides a volatile third-operand pathway. As illustrated in Fig. 2 (b), this device integrates two electrically-coupled gates: a Top-Gate (TG), whose ferroelectric layer stores the non-volatile weight via polarization state, and a Back-Gate (BG), a volatile control terminal separated by a standard dielectric (buried oxide) that provides dynamic modulation of the channel conductance. The thin, fully-depleted silicon channel is sandwiched between these gates, enabling strong electrostatic coupling. <details> <summary>2604.07628v1/x2.png Details</summary>  ### Visual Description ## Diagram: Transistor Structure and Capacitance Model ### Overview The image presents two technical diagrams illustrating a transistor structure and its associated capacitance model. Diagram (a) shows a vertical stack of labeled components with a capacitance formula, while diagram (b) depicts the physical layer structure of the transistor with n+ regions and voltage annotations. ### Components/Axes **Diagram (a):** - **Vertical Stack Components (Left to Right):** - Top: Dark green block labeled **TG** (Top Gate) - Red block labeled **FE** (Fixed Electrode) - Yellow block labeled **IL** (Interlayer) - Light blue block labeled **CH** (Channel) - Orange block labeled **BOX** (Buried Oxide) - Bottom: Dark green block labeled **BG** (Bulk Gate) - **Voltage Labels:** - Top: **V_TG** (Top Gate Voltage) - Bottom: **V_BG** (Bulk Gate Voltage) - **Capacitance Labels:** - **C_FE**, **C_IL**, **C_CH**, **C_BGOX** (capacitances of respective components) - **C_TGOX** (total gate oxide capacitance) with formula: **C_TGOX = (C_FE * C_IL) / (C_FE + C_IL)** **Diagram (b):** - **Vertical Stack Components (Top to Bottom):** - **TG** (Top Gate) - **FE** (Fixed Electrode) - **Interlayer** - **Channel** (with **n+** regions on both sides) - **BOX** (Buried Oxide) - **BG** (Bulk Gate) - **Voltage Labels:** - Top: **V_TG** - Bottom: **V_BG** ### Detailed Analysis **Diagram (a):** - The capacitance model breaks down the total gate oxide capacitance (**C_TGOX**) into contributions from the fixed electrode (**C_FE**) and interlayer (**C_IL**). The formula represents a parallel capacitance combination. - Colors are used to differentiate components: red (FE), yellow (IL), blue (CH), orange (BOX), and dark green (TG/BG). **Diagram (b):** - Physical transistor structure with distinct layers: - **TG** (Top Gate) and **FE** (Fixed Electrode) form the gate stack. - **Interlayer** separates the gate stack from the **Channel**, which includes **n+** regions (source/drain). - **BOX** (Buried Oxide) and **BG** (Bulk Gate) complete the structure. ### Key Observations 1. **Capacitance Model (a):** - **C_TGOX** is derived from the parallel combination of **C_FE** and **C_IL**, indicating these components contribute additively to the total gate oxide capacitance. - The **CH** (Channel) and **BOX** (Buried Oxide) are distinct from the gate oxide components. 2. **Physical Structure (b):** - The **n+** regions flank the **Channel**, typical of MOSFET source/drain regions. - The **BOX** layer underlies the **Channel**, separating it from the **BG** (Bulk Gate). ### Interpretation - **Relationship Between Diagrams:** Diagram (a) provides an electrical model of capacitance distribution, while diagram (b) shows the physical layout. The **FE** and **IL** components in (a) correspond to the **FE** (Fixed Electrode) and **Interlayer** in (b), respectively. - **Functional Insights:** - The **n+** regions in (b) suggest a p-channel or n-channel configuration depending on doping. - The **BOX** layer in (b) aligns with the **BOX** capacitance in (a), acting as a substrate for the channel. - The formula for **C_TGOX** implies that increasing either **C_FE** or **C_IL** would proportionally increase the total gate oxide capacitance, critical for device performance. - **Design Implications:** The separation of **FE** and **IL** capacitances allows optimization of gate oxide properties. The **n+** regions in (b) ensure low-resistance contacts for current flow. No numerical values or trends are provided beyond the formula. The diagrams focus on structural and electrical relationships rather than quantitative data. </details> Figure 2. DG-FeFET device concept. (a) Capacitor stack model illustrating the series combination of ferroelectric ( $C_FE$ ) and interlayer ( $C_IL$ ) capacitances forming the effective top-gate oxide capacitance $C_TGOX$ . (b) Device cross-section showing the Top-Gate (TG), Back-Gate (BG), and localized n + source/drain implant regions within the channel. Physically, the back-gate voltage ( $V_BG$ ) linearly modulates the effective threshold voltage ( $V_th$ ) of the top-gate (Mulaosmanovic et al., 2021; Lim and Fossum, 1983). This coupling is characterized by the coefficient $γ_TG$ , defined by the equivalent capacitance network shown in Fig. 2 (a): $$ γ_TG=\frac{C_CH· C_BGOX}{C_TGOX(C_CH+C_BGOX)} \tag{7} $$ where $C_CH$ and $C_BGOX$ are the channel and back-gate oxide capacitances. The effective top-gate capacitance $C_TGOX$ is the series combination of the ferroelectric capacitance $C_FE$ and the interfacial layer capacitance $C_IL$ : $$ C_TGOX=\frac{C_FE· C_IL}{C_FE+C_IL} \tag{8} $$ The resulting threshold voltage shift is: $$ Δ V_th=-γ_TG· V_BG \tag{9} $$ This threshold shift, combined with mobility enhancement at higher $V_BG$ (the carrier centroid shifts away from the top-gate interface, reducing Coulomb and surface roughness scattering (Nier et al., 2013; Al Mamun et al., 2022; Han et al., 2022)), translates into a multiplicative modulation of the channel conductance $G_DS$ . In the deep triode regime, where $μ_n(V_BG)$ denotes the field-dependent electron mobility, the full relationship is (Jiang et al., 2025): $$ G_DS(V_BG)=\frac{μ_n(V_BG)}{μ_n(0)}· G_DS(0)+γ_TG·μ_n(V_BG)· C_TGOX· V_BG \tag{10} $$ where $G_DS(0)$ is the channel conductance at zero back-gate bias (hereafter denoted $G_0$ ). Consistent with the linear $G_DS(V_BG)$ behavior reported for DG-FeFETs (Jiang et al., 2025), the mobility can be approximated as $μ_n(V_BG)≈μ_n(0)(1+α V_BG)$ , where $α$ is the mobility-sensitivity coefficient (Nier et al., 2013). Under this first-order approximation, the conductance response can be written as $$ G_DS(V_BG)≈ G_0·(1+η_BG· V_BG) \tag{11} $$ This linearization drops the second-order term $γ_TGμ_n(0)α C_TGOXV_BG^2$ . Accordingly, Eq. 11 serves as a first-order approximation for CIM analysis within this linear operating regime. In the CIM paradigm, $G_0$ encodes the stored weight: it is programmed once during model initialization and remains stationary throughout inference, serving as the stationary weight operand of the trilinear primitive developed in Section 4. Comparing Eqs. 10 and 11, and defining the electrostatic coupling coefficient $M=γ_TG· C_TGOX·μ_n(0)$ , we identify the modulation sensitivity: $$ η_BG=α+\frac{M}{G_0} \tag{12} $$ The first term ( $α$ ) arises from the linear mobility dependence on $V_BG$ and captures the mobility-enhancement contribution, while the second term $M/G_0$ captures electrostatic threshold modulation; $η_BG$ therefore represents the combined first-order back-gate sensitivity. We extract $α=0.137$ V -1 and $M=1.54$ $μ$ S/V by numerically fitting our physics-inspired polynomial constraints to the experimentally reported $G_DS$ vs. $V_BG$ data from Jiang et al. (Jiang et al., 2025). Eq. 11 captures the key device behavior for this work: the back-gate voltage modulates the stored conductance multiplicatively through the effective sensitivity $η_BG$ . This provides the physical basis for the trilinear CIM primitive developed in Section 4.2. ## 3. Motivation: Why Trilinear CIM? Having established the device physics of the DG-FeFET, we now quantify the cost of the conventional write-based approach to attention. ### 3.1. Write Bottleneck and Endurance The “Compute-Write-Compute” dataflow introduced in Section 1 is severely penalized by the read/write asymmetry intrinsic to NVM devices. Table 1 quantifies this asymmetry for FeFET devices. Table 1. FeFET read vs. write asymmetry (Jerry et al., 2017; Peng et al., 2019). | Latency Energy/cell | $∼$ 10 ns $∼$ fJ | $∼$ 50 ns $∼$ sub-pJ | | --- | --- | --- | For a BERT-Base configuration ( $N=512$ tokens, $d_k=64$ , $h=12$ heads, $L=12$ layers), the aggregate runtime programming volume becomes: $$ N_prog=2· N· d_k· h· L·≤ft\lceil\frac{8}{2}\right\rceil· 2≈ 75.5M \tag{13} $$ where the first factor of 2 accounts for the two dynamic operands ( $K^T$ and $V$ ), $\lceil 8/2\rceil=4$ maps each 8-bit value onto four 2-bit FeFET cells, and the final factor of 2 reflects separate positive and negative arrays for signed representation. This write volume has severe implications for device lifetime. FeFET devices exhibit $10^6$ – $10^12$ write cycles depending on oxide quality (Jerry et al., 2017). Because the temporary attention arrays must be reprogrammed during every inference, these runtime writes repeatedly stress the cells assigned to $K^T$ and $V$ storage. Moreover, this estimate is for BERT-Base—a relatively small model. Scaling to BERT-Large ( $h{=}16$ , $L{=}24$ ) would increase the aggregate programming volume by approximately 2.7 $×$ . Even technologies with substantially higher nominal endurance, such as STT-MRAM ( $>$ $10^12$ (Chen, 2016)) or SOT-MRAM ( $>$ $10^15$ (Van Beek et al., 2023)), would still face the same fundamental bottleneck: runtime rewriting of attention operands grows with model size and sequence length. Together, the latency, energy, and endurance penalties of this repeated reprogramming critically limit the viability of write-based CIM for dynamic attention. ### 3.2. Case for Trilinear Operations The DG-FeFET resolves the two-operand limitation by providing a third volatile operand pathway through the back-gate (Section 2.2). This additional control terminal enables a different attention dataflow: instead of repeatedly reprogramming dynamic operands into non-volatile memory, the trilinear approach keeps projection weights stationary in the array and applies input dependent modulation through the back-gate. As a result, attention computations can be carried out in-situ without runtime ferroelectric rewriting. This directly addresses the two bottlenecks identified in Section 3.1: (i) write latency is eliminated because dynamic operands are conveyed through back-gate voltage modulation rather than polarization switching; and (ii) inference-time endurance stress is eliminated because attention execution does not require ferroelectric reprogramming. ## 4. Proposed Trilinear CIM Architecture ### 4.1. System Overview <details> <summary>2604.07628v1/x3.png Details</summary>  ### Visual Description ## Diagram: Hardware Architecture for Parallel Processing System ### Overview The diagram illustrates a multi-stage hardware architecture for parallel data processing, divided into four functional quadrants. It depicts data flow from input buffers through processing elements (PEs/ARRAYs) to output buffers, with specialized components for signal routing and analog/digital conversion. ### Components/Axes 1. **Top-Left Quadrant (Global Buffer)** - **Global Buffer**: Central data reservoir - **TILEs (4 units)**: Orange blocks labeled "TILE" - **Processing Pipeline**: - Accumulation (green) - Activation (green) - LayerNorm (green) - Softmax (green) 2. **Top-Right Quadrant (Tile Input Buffer)** - **Tile Input Buffer**: Gray vertical bar - **PEs (4 units)**: Blue blocks labeled "PE" - **Accumulation and Output Buffer**: Gray horizontal bar 3. **Bottom-Left Quadrant (Tile Input Buffer)** - **Tile Input Buffer**: Gray vertical bar - **ARRAYs (4 units)**: Green blocks labeled "ARRAY" - **Accumulation and Output Buffer**: Gray horizontal bar 4. **Bottom-Right Quadrant (Signal Routing Matrix)** - **BGL/SL Switch Matrix/DAC**: - Red vertical lines (BGL) - Blue vertical lines (SL) - **WL/CL Switch Matrix**: - Green horizontal lines - **MUX Decoder**: - 3-stage decoder with "MUX" labels - **ADC/Adder/Shift Register Array**: - 3 columns of ADC/Adder/Shift Register units ### Detailed Analysis - **Data Flow Path**: 1. Global Buffer → TILEs/ARRAYs (parallel processing) 2. Tile Input Buffers → PEs/ARRAYs (local processing) 3. Accumulation Buffers → Output Buffer (final aggregation) 4. Switch Matrix → MUX Decoder → ADC/Adder/Shift Registers (signal conditioning) - **Component Relationships**: - TILEs and ARRAYs represent different processing paradigms (matrix vs. vector) - PEs handle parallel computations with local accumulation - Switch Matrix enables dynamic routing between processing units - MUX Decoder manages multiplexing for efficient data distribution - ADC/Adder/Shift Registers suggest analog signal processing capabilities ### Key Observations 1. **Dual Processing Paths**: - TILEs (orange) and ARRAYs (green) suggest complementary processing capabilities - PEs (blue) appear specialized for matrix operations 2. **Signal Conditioning Complexity**: - 3-stage MUX Decoder indicates hierarchical multiplexing - 9 ADC/Adder/Shift Register units suggest high-precision signal processing 3. **Buffer Hierarchy**: - Global Buffer → Tile Input Buffers → Accumulation Buffers → Output Buffer - Implies multi-level caching for performance optimization ### Interpretation This architecture appears designed for high-performance neural network inference or scientific computing. The combination of TILEs (likely for matrix multiplications) and ARRAYs (for vector operations) enables efficient handling of different computational primitives. The PEs with local accumulation buffers suggest a systolic array architecture for parallel computation. The bottom-right quadrant's complex routing matrix indicates support for dynamic computation graphs or adaptive processing paths. The presence of ADC/Adder/Shift Registers implies the system can handle mixed-signal processing, potentially for edge AI applications requiring sensor data fusion. The buffer hierarchy demonstrates a thoughtful approach to data locality, minimizing latency by keeping intermediate results close to processing units. The switch matrix architecture enables flexible reconfiguration of data paths, allowing the system to adapt to different computational workloads without hardware changes. Notably, the absence of explicit control logic in the diagram suggests these components interface with a separate control plane, likely managed by the MUX Decoder and associated logic. This separation of compute and control planes is common in modern accelerator architectures for improved scalability. </details> Figure 3. Hierarchical organization of the trilinear CIM accelerator. (Top-left) Chip level: 2 $×$ 2 tile grid with global buffer, accumulation unit, and Special Function Unit (SFU) for Softmax, LayerNorm, and Activation. (Top-right) Tile level: 2 $×$ 2 Processing Element (PE) grid with tile input buffer and accumulation/output buffer. (Bottom-left) PE level: 2 $×$ 2 array grid with tile input buffer. (Bottom-right) SubArray level: DG-FeFET cells driven by wordline/control-line (WL/CL) switch matrix (rows) and back-gate-line/source-line (BGL/SL) switch matrix/DAC (columns), with multiplexer (MUX) $→$ ADC $→$ Adder $→$ Shift Register readout pipeline. Fig. 3 illustrates our hierarchical DG-FeFET accelerator architecture, which performs attention through static trilinear computation. Chip Level. The top-level architecture organizes tiles in a scalable 2D mesh connected via H-tree interconnect for balanced latency (Chen et al., 2018). A global buffer interfaced with the tile array stores input sequences $X∈ℝ^N× d$ and broadcasts them to the tiles. Peripheral to the compute fabric, dedicated functional units (Softmax, LayerNorm, Activation) handle operations incompatible with analog computation. This split compute model—analog multiplication for attention scores, digital for non-linearities—maximizes energy efficiency while maintaining accuracy. Grid dimensions (tile array size, processing elements (PEs) per tile, subarrays per PE) are automatically determined by our TransCIM framework (Section 5.1) via its floorplanning algorithm (Peng et al., 2019) based on model weight capacity and target chip area. Tile Level. Each tile comprises a grid of PEs with shared local buffering. The tile input buffer retains frequently reused operands and intermediate vectors, reducing global memory traffic. Partial sums from PE outputs converge through an accumulation network before being written to the tile output buffer, supporting parallel execution across attention heads and partitioned embedding dimensions. PE and SubArray Microarchitecture. Within each PE, DG-FeFET arrays support the trilinear attention operations described in Section 4.2 using DG-FeFET cells in a selector-less configuration. Row drivers control two signals: wordlines (WL) carry input activations $X$ to device drains, while control lines (CL) bias the top-gate—enabling row selection during inference or applying programming voltage ( $V_TG$ ) during weight updates. Column-wise drivers handle back-gate lines (BGL) that apply the dynamic modulator voltage ( $V_BG$ ) via integrated DACs, and source lines (SL) that collect output currents. The resulting analog currents flow to a multi-stage readout pipeline: column multiplexer for output selection, time-multiplexed shared ADCs for digitization, digital adders for partial sum accumulation, and shift registers for multi-bit weight alignment. This mixed-signal readout organization balances throughput with precision by amortizing ADC resources across columns via time-multiplexing. Static linear layers outside attention—including the FFN projections and output projection layers—are mapped to separate single-gate FeFET CIM arrays, whereas the DG-FeFET back-gate is exercised only for attention stages that require dynamic modulation. ### 4.2. Trilinear Operation Concept The DG-FeFET enables a trilinear multiply-accumulate (MAC) primitive. From the conductance modulation (Eq. 11), the full output current is: $$ I_DS=V_DS· G_0·(1+η_BG· V_BG) \tag{14} $$ where $V_DS$ encodes input activations, $G_0$ stores weight values, and $V_BG$ provides dynamic conductance modulation. The trilinear product of interest resides in the second term ( $V_DS· G_0·η_BG· V_BG$ ), while the first term ( $V_DS· G_0$ ) is a constant DC bias that is removed through baseline subtraction (Section 5). Table 2 maps these signals to operand roles across attention stages. <details> <summary>2604.07628v1/x4.png Details</summary>  ### Visual Description ## Scatter Plot: Relationship Between Stored Conductance (G₀) and Background Conductance (η_BG) ### Overview The image is a scatter plot showing the relationship between stored conductance (G₀, μS) and background conductance (η_BG, V⁻¹). It includes measured data points, fitted lines, and shaded regions representing operational parameters. The plot is divided into two regions: a pink shaded area (0–30 μS) and a green shaded area (29–69 μS), with the green region labeled as the "Operating band." --- ### Components/Axes - **X-axis**: "Stored Conductance G₀ (μS)" with values ranging from 0 to 70 μS. - **Y-axis**: "η_BG (V⁻¹)" with values ranging from 0.10 to 0.50 V⁻¹. - **Legend**: Located in the top-right corner, with the following entries: - **Blue circles**: "Measured" data points. - **Dashed gray line**: "η_BG = α + M/G₀" (theoretical model). - **Green solid line**: "Linear fit (band)". - **Light green shaded area**: "Operating band [29–69] μS". - **Red dotted line**: "η_BG = 0.157" (constant value). --- ### Detailed Analysis #### Measured Data Points (Blue Circles) - **Trend**: The measured data points show a decreasing trend as G₀ increases. - **Key Values**: - At G₀ = 5 μS: η_BG ≈ 0.43 V⁻¹. - At G₀ = 20 μS: η_BG ≈ 0.22 V⁻¹. - At G₀ = 30 μS: η_BG ≈ 0.19 V⁻¹. - At G₀ = 50 μS: η_BG ≈ 0.16 V⁻¹. - At G₀ = 70 μS: η_BG ≈ 0.14 V⁻¹. #### Linear Fit (Green Solid Line) - **Trend**: A linear decrease with a negative slope. - **Key Values**: - At G₀ = 30 μS: η_BG ≈ 0.20 V⁻¹. - At G₀ = 70 μS: η_BG ≈ 0.14 V⁻¹. #### Theoretical Model (Dashed Gray Line) - **Equation**: η_BG = α + M/G₀. - **Trend**: A decreasing curve with a steeper decline at lower G₀ values. - **Key Values**: - At G₀ = 0 μS: η_BG ≈ 0.45 V⁻¹. - At G₀ = 70 μS: η_BG ≈ 0.15 V⁻¹. #### Operating Band (Light Green Shaded Area) - **Range**: G₀ = 29–69 μS. - **Observation**: The measured data and fitted lines fall within this range for G₀ > 30 μS. #### Red Dotted Line (η_BG = 0.157) - **Trend**: A horizontal line at η_BG = 0.157 V⁻¹. - **Observation**: The measured data and fitted lines approach this value as G₀ increases. --- ### Key Observations 1. **Decreasing Trend**: Both measured data and theoretical models show η_BG decreasing as G₀ increases. 2. **Operating Band**: The green shaded region (29–69 μS) aligns with the majority of the measured data points. 3. **Theoretical vs. Measured**: The dashed gray line (theoretical model) closely follows the measured data but with a slightly higher η_BG at lower G₀ values. 4. **Red Line Significance**: The red dotted line (η_BG = 0.157) acts as a reference threshold, with measured values approaching it at higher G₀. --- ### Interpretation - **Data Relationship**: The plot demonstrates that increasing stored conductance (G₀) reduces background conductance (η_BG), consistent with the theoretical model η_BG = α + M/G₀. - **Operational Relevance**: The green shaded "Operating band" (29–69 μS) suggests optimal performance or stability within this range. - **Anomalies**: The measured data at G₀ = 5 μS (η_BG ≈ 0.43 V⁻¹) deviates significantly from the theoretical model (η_BG ≈ 0.45 V⁻¹), indicating potential experimental variability or unaccounted factors. - **Threshold Behavior**: The red dotted line (η_BG = 0.157) may represent a critical threshold for device functionality, as measured values approach it at high G₀. This analysis highlights the inverse relationship between stored conductance and background conductance, with the operating band and theoretical model providing context for device performance. </details> Figure 4. $η_BG$ vs. $G_0$ for the DG-FeFET, following Eq. 12 with parameters $α$ and $M$ extracted from experimental data (Jiang et al., 2025) (see Section 2.2). The green-shaded region indicates the selected operating band ( $G_0∈[29,69]$ $μ$ S); the dashed horizontal line marks the band-averaged $\bar{η}_BG=0.157$ V -1 adopted as a uniform constant. As derived in Section 2.2, $η_BG$ depends on $G_0$ (Eq. 12). Fig. 4 validates this relationship using the simulated numerical parameters calibrated to the empirical device data (Jiang et al., 2025). To ensure uniform trilinear behavior, we constrain $G_0∈[29,69]$ $μ$ S—an operating band where the residual $η_BG$ variation remains strictly bounded. Below this range, $η_BG$ uniformity degrades rapidly, justifying the choice of the lower bound. Within the selected band, we approximate the cell-specific modulation sensitivity $η_BG$ with a single band-averaged constant, $\bar{η}_BG=0.157$ V -1. ### 4.3. Attention Dataflow Fig. 5 contrasts conventional two-operand attention with our trilinear approach. In the figure, Static (Once) denotes operands associated with one-time programming of non-volatile weights into the array, whereas Dynamic denotes inference-time operands that vary with the input sequence. In the conventional dataflow shown in Fig. 5 (a), query, key, and value projections $Q=X· W_Q^T$ , $K=X· W_K^T$ , $V=X· W_V^T$ are computed independently, with intermediate tensors stored in and fetched from off-chip DRAM. Once $K$ is available, score computation $Q· K^T$ proceeds, followed by a separate digital scaling step ( $÷√{d_k}$ ), softmax normalization, and finally $Score· V$ . In a conventional dataflow, softmax and value aggregation can be token-pipelined to hide some latency (i.e., computing the softmax for token $i$ while accumulating the weighted sum over value vectors for token $i-1$ ). However, this optimization provides diminishing returns because the preceding step—writing the massive intermediate tensors to off-chip DRAM—imposes an overwhelming latency and energy wall that dominates total inference time. The trilinear dataflow in Fig. 5 (b) eliminates both the DRAM bottleneck and the separate scaling step by fusing projections with subsequent operations. Like conventional attention, softmax and value aggregation can be token-pipelined; however, trilinear implementation further reduces latency by bypassing explicit $Q$ , $K$ , and $V$ projection steps entirely—once the static weights are programmed, tokens can stream directly into the attention pipeline: earlier partial results feed downstream stages while later tokens are still being processed, rather than waiting for full tensors to be formed explicitly: <details> <summary>2604.07628v1/x5.png Details</summary>  ### Visual Description ## Diagram: Hybrid Neural Network Inference Process with Crossbar Arrays and Off-Chip DRAM ### Overview The diagram illustrates a two-phase hybrid approach for neural network inference, combining static and dynamic operations. It emphasizes time savings by optimizing data movement between on-chip crossbar arrays and off-chip DRAM. The process involves matrix multiplications, softmax scoring, and result computation, with a focus on reducing latency through on-chip computation. ### Components/Axes - **Static Phase (a)**: - **Components**: Off-Chip DRAM, Crossbar Arrays (W_Q, W_K, W_V), Store/Fetch operations. - **Flow**: Weights (W_Q, W_K, W_V) are statically stored in DRAM once. Input X is multiplied by weight transposes (W_Q^T, W_K^T, W_V^T) to compute Q, K, V. K^T is fetched from DRAM for dynamic computation of Out = Q·K^T. Softmax scoring and result computation (Score·V) follow. - **Dynamic Phase (b)**: - **Components**: Crossbar Arrays (W_Q), on-chip operations (R1, R2, Scaling, Softmax). - **Flow**: Input X is multiplied by W_Q^T with scaling (1/√d_k). R1 and R2 are computed via crossbar reads (FeFET/Trilinear). Softmax (LUT-based) and result computation (Score·W_V^T) occur on-chip, avoiding DRAM access for K^T. - **Timeline**: - **Color-coded blocks** represent operations: - **Blue**: Crossbar Write (static phase). - **Green**: Crossbar Read (Regular/FeFET). - **Orange**: Crossbar Read (Trilinear/FeFET). - **Yellow**: Scaling (Digital). - **Purple**: Softmax (LUT-Digital). - **Gray**: Off-Chip DRAM. ### Detailed Analysis 1. **Static Phase (a)**: - **Weights (W_Q, W_K, W_V)**: Stored once in Off-Chip DRAM (gray). - **Operations**: - Q = X·W_Q^T, K = X·W_K^T, V = X·W_V^T (green crossbar reads). - Out = Q·K^T (requires fetching K^T from DRAM via red arrows). - Score = Softmax(Out/√d_k) (purple). - Result = Score·V (green). 2. **Dynamic Phase (b)**: - **On-Chip Computations**: - R1 = X·W_Q^T·(1/√d_k) (orange crossbar read with scaling). - R2 = R1·W_K·X^T (orange crossbar read). - Score = Softmax(R2) (purple LUT). - Result = Score·W_V^T (orange crossbar read). - **Time Saved**: Red dashed arrow highlights reduced latency by avoiding DRAM access for K^T. ### Key Observations - **Time Efficiency**: Dynamic phase eliminates repeated DRAM fetches for K^T, saving time (red arrow). - **Hybrid Architecture**: Combines static DRAM storage (for weights) with dynamic on-chip computation (crossbar arrays). - **FeFET Integration**: Trilinear/FeFET crossbar reads (orange) enable analog computations, reducing digital overhead. - **Scaling**: Digital scaling (1/√d_k) is applied during crossbar reads to maintain numerical stability. ### Interpretation The diagram demonstrates a **memory-efficient neural network inference pipeline** optimized for low latency. By offloading initial weight storage to DRAM (static phase) and leveraging on-chip crossbar arrays for repeated computations (dynamic phase), the system minimizes data movement bottlenecks. The use of FeFET-based crossbar reads and LUT-based softmax further enhances energy efficiency. This hybrid approach is critical for real-time applications like edge AI, where balancing computational speed and memory access costs is paramount. The "Time Saved" annotation underscores the architectural advantage of avoiding redundant DRAM accesses in the dynamic phase. </details> Figure 5. Attention dataflow comparison. (a) Conventional: $Q/K/V$ projections require off-chip DRAM transfers before score computation. (b) Trilinear: weights remain static while $X$ modulates device conductance via back-gate, eliminating DRAM traffic and saving compute time. Stage 1: Scaled Query Generation. Query vectors with built-in scaling are computed as $R_1=X· W_Q^T·(1/√{d_k})$ , where $W_Q^T$ is stored in the crossbar, $X$ is applied through the row input path, and the static scaling factor is applied via the back gate. This single trilinear MAC replaces the conventional projection and separate scaling steps. Stage 2: Score Synthesis. Attention scores are computed as $R_2=R_1· W_K· X^T$ , without forming the intermediate key matrix. The query vector $R_1$ is applied as the row-side input, $W_K$ is stored in the crossbar, and $X^T$ modulates conductance dynamically via the back gate. This fused operation directly yields pre-softmax scores without computing or storing key tensors. Stage 3: Value Aggregation. After digital softmax produces $Score=softmax(R_2)$ , the final output is $Result=Score· X· W_V^T$ . The input sequence $X$ provides the row-side input, Score modulates conductance via back-gate broadcast, and $W_V^T$ is stored. Key Insight: Static vs. Dynamic Modulation. Table 2 summarizes the operand-to-terminal mapping for each stage, distinguishing fixed from time-varying back-gate modulation during inference. Stage 1 uses static modulation: the scaling factor $1/√{d_k}$ is constant for all tokens and is applied as a fixed back-gate voltage. As a result, this stage does not require dynamic back-gate updates and could equivalently be realized using a standard single-gate FeFET. Stages 2 and 3 use dynamic modulation: the back-gate voltage changes with the token-dependent operand ( $X^T$ in Stage 2 and Score in Stage 3). This distinction is architecturally significant because dynamic modulation requires per-token back-gate updates and therefore incurs DAC switching overhead. Memory Traffic Reduction. The trilinear dataflow completely eliminates the need to store intermediate projection tensors. In conventional attention, a sequence of length $N$ and embedding dimension $d$ requires $3× N× d$ elements of intermediate storage ( $Q$ , $K$ , $V$ projections), which often spills to off-chip DRAM for long sequences. Our approach eliminates the need to store the intermediate projection tensors, while retaining only the input sequence $X$ for reuse and residual connections. This translates to substantial energy savings, as a DRAM access consumes roughly two orders of magnitude more energy than a small on-chip SRAM/cache access (Horowitz, 2014). Table 2. Trilinear attention stages and operand mapping. | Scaled $Q$ Score $V$ -Agg | $X· W_Q^T/√{d_k}$ $R_1· W_K· X^T$ $Score· X· W_V^T$ | $X$ $R_1$ $X$ | $W_Q^T$ $W_K$ $W_V^T$ | $1/√{d_k}$ $X^T$ Score | | --- | --- | --- | --- | --- | ### 4.4. DG-FeFET Crossbar Array Design The DG-FeFET crossbar employs a selector-less architecture where each device connects directly between its row and column lines without an access transistor. The high on/off current ratio ( $>10^4$ ) inherent to FeFETs (Ni et al., 2019; Yin et al., 2018; Mulaosmanovic et al., 2019) mitigates sneak-path currents. As illustrated in Fig. 6, the crossbar organization maps directly to the trilinear primitive (Eq. 14): static weights reside in device conductance ( $G_0$ ), sequence inputs stream along rows ( $V_DS$ ), and dynamic modulators are applied along columns ( $V_BG$ ). The per-column routing enables either independent per-column biasing (for element-wise operand application in Configuration (a)) or uniform broadcast across all columns (for common-scalar input in Configuration (b)). <details> <summary>2604.07628v1/x6.png Details</summary>  ### Visual Description ## Diagram: Trilinear Operation (O = A · Bᵀ · C) ### Overview The diagram illustrates a trilinear operation (O = A · Bᵀ · C) implemented through a sequence of matrix operations and element-wise additions. It visualizes the flow of inputs through a computational graph, combining sequence and backgate inputs via weighted sums and matrix multiplications. ### Components/Axes 1. **Inputs**: - **Sequence Input**: Labeled "Sequence input (c₁₁ ... c₁ᵣ) via V_seq" (leftmost column). - **Backgate Input**: Labeled "Backgate input (c₁₁ ... c₁ᵣ) via V_Bd" (top row). 2. **Matrix Operations**: - A grid of blue (B) and red (C) components, each connected to inputs via vertical/horizontal lines. - Intermediate additions (green boxes) combine B and C components. 3. **Output**: - Final output labeled "Trilinear output" (bottom right), derived from summing intermediate results. ### Detailed Analysis - **Matrix Dimensions**: - A: [D₁ × D₂], B: [D₂ × N₂], C: [N₂ × N₃] (explicitly labeled). - Bᵀ (transpose of B) is used in the operation. - **Equations**: - Bᵀ = [[b₁₁, b₁₂, ..., b₁ᵣ], ..., [bᵣ₁, bᵣ₂, ..., bᵣᵣ]]. - C = [[c₁₁, c₁₂, ..., c₁ᵣ], ..., [cᵣ₁, cᵣ₂, ..., cᵣᵣ]]. - O' = ΣΣΣ (aᵢⱼ · bⱼₖ · cₖₗ) for all i, j, k, l. - **Color Coding**: - Blue (B) and red (C) components are consistently color-coded across the matrix. - No explicit legend, but colors align with matrix definitions. ### Key Observations 1. **Flow Direction**: Inputs flow left-to-right (sequence) and top-to-bottom (backgate), converging in the matrix. 2. **Intermediate Steps**: Each matrix cell combines B and C components via addition before contributing to the final output. 3. **Mathematical Consistency**: The equations match the diagram’s structure, confirming the trilinear operation’s decomposition. ### Interpretation The diagram demonstrates how a trilinear operation is implemented using matrix multiplications and additions. The sequence and backgate inputs are transformed through matrices A, B, and C, with their interactions encoded in the summation of products (aᵢⱼ · bⱼₖ · cₖₗ). This structure suggests applications in tasks requiring multi-dimensional input integration, such as neural network layers or tensor operations. The use of backgate inputs implies a mechanism to modulate sequence processing, potentially for attention or gating mechanisms in models like transformers or RNNs. </details> <details> <summary>2604.07628v1/x7.png Details</summary>  ### Visual Description ## Diagram: Trilinear Operation (O = A · B · C^T) ### Overview The diagram illustrates a trilinear operation involving three matrices (A, B, C) and their interaction to produce an output O. It depicts a three-stage processing pipeline with labeled components, mathematical relationships, and data flow. The operation is defined as a tensor contraction across three dimensions, with explicit voltage-controlled inputs and output summation. ### Components/Axes 1. **Title**: "Trilinear operation: O = A · B · C^T" (top-left) 2. **Stages**: - **Stage 1 (Left)**: - **Backgate input**: Labeled as "(a₁₁…a₁₂) via V_AC" (green arrows) - **Sequence input**: Labeled as "(a₁₁…a₁₂) via V_AC" (blue arrows) - **Weighted Non-volatility C**: Labeled as "via V_TG" (orange arrows) - **Stage 2 (Center)**: - **Addition nodes**: Labeled "Addition" (gray dashed lines) - **Stage 3 (Right)**: - **Output nodes**: Labeled "Trilinear output" (purple arrows) 3. **Mathematical Expressions**: - Trilinear output formula: $ o_{r^{\prime}i^{\prime}} = \sum_{j=1}^{N_2} \sum_{k=1}^{D_3} a_{r^{\prime}j} b_{j k} c_{i' k} $ - Matrix definitions: - $ A = \begin{bmatrix} a_{11} & \cdots & a_{1N_2} \\ \vdots & \ddots & \vdots \\ a_{N_11} & \cdots & a_{N_1N_2} \end{bmatrix} $ - $ B = \begin{bmatrix} b_{11} & \cdots & b_{1D_2} \\ \vdots & \ddots & \vdots \\ b_{D_1} & \cdots & b_{D_1D_2} \end{bmatrix} $ - $ C^T = \begin{bmatrix} c_{11} & \cdots & c_{1D_3} \\ \vdots & \ddots & \vdots \\ c_{D_2} & \cdots & c_{D_2D_3} \end{bmatrix} $ - Output matrix: $ O = \begin{bmatrix} o_{11} & \cdots & o_{1D_3} \\ \vdots & \ddots & \vdots \\ o_{N_1} & \cdots & o_{N_1D_3} \end{bmatrix} $ ### Detailed Analysis - **Data Flow**: - Backgate and sequence inputs (Stage 1) feed into weighted non-volatility C (Stage 1), which propagates through addition nodes (Stage 2) to produce the trilinear output (Stage 3). - Voltage-controlled inputs (V_AC, V_TG) modulate the interaction between matrices. - **Matrix Dimensions**: - $ A \in \mathbb{R}^{N_1 \times N_2} $, $ B \in \mathbb{R}^{D_1 \times D_2} $, $ C^T \in \mathbb{R}^{D_2 \times D_3} $, with $ N_1 = N_2 = D_1 = D_2 = D_3 $ (explicitly stated in the diagram). - **Output Structure**: - The trilinear output $ O $ is a $ N_1 \times D_3 $ matrix, computed via nested summations over $ j $ and $ k $. ### Key Observations - **Symmetry**: The diagram emphasizes symmetry in matrix dimensions ($ N_1 = N_2 = D_1 = D_2 = D_3 $), simplifying the trilinear contraction. - **Non-Volatility**: The term "Weighted Non-volatility C" suggests a hardware-aware design, possibly for energy-efficient computation. - **Addition Nodes**: The central stage aggregates intermediate results, indicating a parallel processing architecture. ### Interpretation This diagram represents a **tensor-based computational model** for trilinear operations, likely in the context of neuromorphic or analog computing. The use of voltage-controlled inputs (V_AC, V_TG) implies a physical implementation where electrical signals modulate matrix interactions. The operation combines three matrices (A, B, C) through element-wise multiplication and summation, producing a compact output matrix $ O $. The explicit dimension constraints ($ N_1 = N_2 = D_1 = D_2 = D_3 $) suggest a balanced design for efficient computation, avoiding redundant dimensions. The "Weighted Non-volatility C" component hints at a focus on energy efficiency, as non-volatile memory elements retain state without power, reducing dynamic energy consumption. This model could be applied to tasks like signal processing, machine learning, or quantum computing, where trilinear operations are fundamental. </details> Figure 6. DG-FeFET crossbar array for trilinear MAC. (a) $O=A· B^T· C$ : weights $B^T$ stored in device conductance, input $A$ applied to rows, modulator $C$ applied to columns. (b) $O=A· B· C^T$ : weights $C^T$ stored in device conductance, input $B$ applied to rows, modulator $A$ broadcast across all columns. The two crossbar configurations differ in their accumulation strategy: Configuration (a) (Fig. 6 a): $O=A· B^T· C$ . Each crossbar stores weight matrix $B^T$ in device conductance. Parallel crossbars compute independent rows of the output matrix over time: crossbar $i$ receives input row $A_i,:$ applied to its rows, and back-gate vector $C_:,j$ applied to its columns. Within each crossbar, analog column currents sum via Kirchhoff’s current law (over the inner-product dimension), and a digital adder aggregates all column outputs. This intra-crossbar addition produces one output element $o_ij$ per crossbar, per cycle. The back-gate loops through the columns of $C$ : in the first cycle, all crossbars receive column $C_:,1$ across their back-gates; in the second cycle, $C_:,2$ ; and so on. Configuration (b) (Fig. 6 b): $O=A· B· C^T$ . Each crossbar stores weight matrix $C^T$ in device conductance—all parallel crossbars share identical weights. Crossbar $i$ takes row $B_i,:$ as its row input. Column currents from corresponding positions across all crossbars are summed via inter-crossbar addition to produce the output elements. The back-gate loops through the rows of $A$ : in the first cycle, crossbar $i$ receives element $a_1i$ (broadcast to all its columns); in the second cycle, it receives $a_2i$ ; and so on through $a_r^\primei$ . Stage 2 (Score Synthesis) uses the configuration in Fig. 6 (a), while Stage 3 (Value Aggregation) uses Fig. 6 (b). Stage 1 (Scaled Query Generation) can use either configuration because its back-gate operand is fixed. ### 4.5. Digital Operations Integration Transformer inference requires several digital operations incompatible with analog memory arrays. These include distributed cross-tile accumulation, as well as dedicated non-linear functions (Softmax, LayerNorm, Activation) handled by a Special Function Unit (SFU) peripheral to the tile array (Fig. 3). Softmax. Attention score normalization is implemented via a four-stage pipeline: (1) a comparator tree finds the maximum value across the sequence for numerical stability, (2) an exponential lookup table (LUT) computes $e^x_i-x_\max$ for each element (where $x_\max=\max_i(x_i)$ ), (3) an adder tree sums the exponentiated values, and (4) a reciprocal LUT followed by multipliers divides each exponential by the sum. The overall pipeline has fixed, deterministic latency, with the LUT stages completing in a single cycle using 256-entry tables for 8-bit precision. LayerNorm. Layer normalization is implemented as a two-pass pipeline over the $d$ -dimensional embedding vector. The first pass computes the mean $μ$ using an adder tree followed by fixed-point division. The second pass subtracts $μ$ , squares the residuals, accumulates the variance $σ^2$ , and applies an inverse-square-root LUT to normalize the vector. A final affine stage then applies the learned per-dimension scale and bias to the normalized vector. Activation (GELU). The FFN sub-layer uses the sigmoid-based approximation $GELU(x)≈ x·σ(1.702 x)$ (Hendrycks and Gimpel, 2016), where $σ(·)$ denotes the logistic sigmoid function. Our hardware implements this in a three-stage pipeline: (1) a shift-and-add scaler approximates the constant multiplication $1.702 x$ without a dedicated multiplier, (2) a 256-entry sigmoid LUT maps the scaled value to $σ(1.702 x)$ , and (3) a fixed-point multiplier produces the final product $x·σ(1.702 x)$ . This decomposition avoids the expensive error-function evaluation of the exact GELU (Hendrycks and Gimpel, 2016) while reusing the same LUT and multiplier primitives employed by the softmax unit. Accumulation. Partial sums are reduced via a local hierarchical adder network within each PE and tile. These local accumulation buffers aggregate sub-array and PE outputs, minimizing inter-tile communication bandwidth by only forwarding condensed tile-level outputs to the chip-level accumulation unit for final cross-tile accumulation. ## 5. Implementation Details ### 5.1. TransCIM Framework We introduce TransCIM (Transformer CIM), a simulation framework built on the backbone of the open-source NeuroSim (Peng et al., 2019) platform, to enable end-to-end evaluation of transformer workloads on CIM architectures. TransCIM extends NeuroSim’s baseline two-operand CIM models to accommodate all transformer layer types: attention stages (including the proposed trilinear operations), linear projections (MLP layers, embedding matrices), and digital operations (Softmax, LayerNorm, GELU). The framework operates in two execution modes. The digital baseline mode quantizes inputs and weights to INT8 but accumulates in FP32 with no ADC or output quantization, serving as a quantization-aware accuracy ceiling. The CIM emulation mode adds hardware-aware effects including ADC quantization (output clipped to ADC bit-width), back-gate modulation non-uniformity ( $η_BG$ operating-band constraints, Section 4.2), and the hierarchical adder-based accumulation path described in Sections 4.4 and 4.5. Both modes apply identical INT8 input/weight quantization, isolating the accuracy impact of analog non-idealities from the quantization baseline. INT8 quantization parameters are obtained via post-training quantization (PTQ): activation scales are calibrated on a small representative dataset, after which weights and inputs are quantized using a symmetric uniform scheme. Multi-bit weights are mapped to multiple cells when device precision is limited—for example, an 8-bit weight with 2-bit cells uses 4 adjacent cells per synapse, with a shift-add stage recombining partial sums ( $output=∑_ipartial_i× 2^i· b_cell$ ), where $b_cell$ is the number of bits stored per cell. Input voltages are applied bit-serially via the switch matrix, cycling from LSB to MSB over multiple time steps. ### 5.2. TransCIM PPA Modeling TransCIM derives its performance, power, and area (PPA) estimates from the validated circuit models of the underlying NeuroSim backbone (Peng et al., 2019). We adopt a heterogeneous integration approach: CMOS peripheral circuits (ADCs, multiplexers, sense amplifiers, shift-add logic, buffers, and drivers) are modeled at a 7nm FinFET technology node using TSMC/IRDS transistor parameters (Peng et al., 2019), while FeFET memory cells use 22nm device characteristics (write voltage = 4.0V, write pulse = 50ns, $R_on$ = 240k $Ω$ , $R_off$ = 24M $Ω$ ), consistent with the shared 22nm FDSOI ferroelectric top-gate stack used in both single-gate (Ni et al., 2019) and DG-FeFET (Jiang et al., 2025) implementations. This reflects a back-end-of-line (BEOL) integration where NVM devices are fabricated at a relaxed feature size above the dense CMOS logic layer—a physically realistic configuration since FeFET devices do not scale as aggressively as CMOS transistors (Ni et al., 2019). Conductance values are constrained within the linear operating band established in Section 4.2, which is calibrated to experimental device data (Jiang et al., 2025). Table 3. Default system configuration. | Parameter | Value | | --- | --- | | Technology node | 7nm CMOS / 22nm FeFET (BEOL) | | SubArray size | 64 $×$ 64 (scalable) | | Input precision | 8-bit | | Weight precision | 8-bit (2-bit/cell) | | ADC precision | 8-bit | | Column muxing | 8:1 (ADC sharing) | | Write voltage | 4.0V (Ni et al., 2019) | | Write pulse | 50ns (Ni et al., 2019) | | Global buffer | 4MB SRAM * | | * Scales linearly with sequence length (4MB valid for seq. length 64). | | For trilinear operations, the back-gate modulation energy model accounts for: (1) DAC switching energy to generate the analog back-gate voltage, (2) driver circuits that buffer and distribute the signal, (3) wire capacitance along the per-column back-gate lines (estimated at 0.2 fF/ $μ$ m for local interconnect), and (4) device gate capacitance at each FeFET. These components contribute to the core read energy of Stages 2 and 3, where the back-gate operand varies with the token. The DC offset term (the “1” in Eq. 14) is removed through a reference read with $V_BG=0$ on the same crossbar under the same row-input condition, which measures the $V_DS· G_0$ component for subtraction from the trilinear readout. This reference read reuses the existing readout path and adds only minor execution overhead. Multi-head attention parallelism is modeled with latency taking the maximum across parallel heads and energy summing across all heads, reflecting the hardware reality of concurrent but independent head computation. Table 3 lists the default hardware parameters used unless otherwise noted in Section 6. ## 6. Experiments ### 6.1. Experimental Setup We evaluate the trilinear CIM accelerator on two representative transformer architectures: BERT-base-uncased (Devlin et al., 2019) (12 layers, 12 heads, $d$ =768) for NLP and ViT-base (Dosovitskiy et al., 2021) (12 layers, 12 heads, $d$ =768) for computer vision. For NLP evaluation, we use nine tasks from the GLUE benchmark (Wang et al., 2018): CoLA, SST-2, MRPC, RTE, STS-B, WNLI, QNLI, QQP, and MNLI, covering sentiment analysis, paraphrase detection, textual entailment, and semantic similarity. For vision, we evaluate on CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and ImageNet-1K (Deng et al., 2009). All models use INT8 post-training quantization for both weights and activations. We compare three evaluation modes. Quantized-Digital performs inference on ideal digital hardware with no analog non-idealities, serving as the accuracy ceiling. CIM-Bilinear uses conventional single-gate FeFET CIM where $K$ / $V$ matrices are dynamically reprogrammed onto the crossbar each sequence, subject to write latency overhead. CIM-Trilinear is the proposed DG-FeFET architecture where projection weights remain static on the top gate and the input sequence drives sequence-dependent modulation of conductance via the back-gate. ### 6.2. Accuracy Results Tables 4 and 5 summarize accuracy across NLP and vision benchmarks, respectively. All reported values are mean $±$ standard deviation over three independent runs per evaluation mode. GLUE Results (Table 4). The trilinear mode outperforms bilinear on seven of nine tasks, with the largest improvements on QNLI (+3.74%), MNLI (+3.14%), STS-B (+1.74%), and SST-2 (+1.60%). This consistent advantage arises because bilinear repeatedly forms intermediate tensors across mixed-signal interfaces: analog CIM outputs must be digitized, requantized/remapped, and written back to the array before subsequent operations. These extra conversion and remapping steps accumulate numerical error, whereas trilinear keeps all projection weights static and applies the dynamic operand through the volatile back-gate. The lower standard deviation of trilinear results (typically $<$ 1%) compared to bilinear (up to ${∼}8.5$ %) further suggests that eliminating these repeated mixed-signal conversion and remapping steps yields more stable inference. The only task where trilinear underperforms bilinear is RTE ( $-$ 1.33%). CoLA shows a modest trilinear advantage ( $+$ 1.53%), but both CIM modes remain well below digital ( $-$ 1.76 and $-$ 3.29 for trilinear and bilinear, respectively). This gap likely reflects the Matthews Correlation Coefficient’s heightened sensitivity to false-positive/negative balance under analog noise. WNLI yields identical results across all three modes (56.34%) because its tiny dataset converges to the majority-class baseline regardless of hardware non-idealities. Vision Results (Table 5). In contrast to GLUE, the gap between trilinear and digital widens from CIFAR-10 to CIFAR-100 to ImageNet, while bilinear remains consistently closer to digital across all three ViT benchmarks. This reversal suggests that, for ViT, the error introduced by the back-gate quantization path outweighs the accuracy benefit obtained by avoiding bilinear’s repeated mixed-signal conversion and remapping steps. The trilinear computation introduces an additional quantization layer via the back-gate DAC that bilinear does not require: the dynamic modulating operand is discretized through a uniform DAC before it modulates device conductance. For NLP tasks, this extra quantization is well-tolerated because discrete token semantics provide natural noise resilience—small perturbations to attention weights rarely change which token is attended. For ViT, however, attention maps exhibit extreme non-uniform distributions with sparse, high-magnitude outlier scores critical for patch discrimination (Lin et al., 2022; Yuan et al., 2022). The uniform back-gate DAC systematically distorts these outlier values, collapsing the sharp attention peaks that ViT relies on. Additionally, ViT post-LayerNorm activations exhibit wider inter-channel variance than NLP models (Li et al., 2023), further amplifying the back-gate quantization error. Table 4. GLUE benchmark scores. Bold indicates the best CIM mode per task. Seq: sequence length in tokens. Metrics—MCC: Matthews Correlation Coefficient; Acc: Accuracy; F1: F1-score; Pears.: Pearson correlation. | CoLA (MCC) SST-2 (Acc) MRPC (F1) | 64 64 128 | 45.46 $±$ 1.46 91.59 $±$ 0.40 88.04 $±$ 0.87 | 42.17 $±$ 8.49 89.72 $±$ 1.07 84.24 $±$ 3.62 | 43.70 $±$ 1.03 91.32 $±$ 0.43 85.54 $±$ 0.62 | | --- | --- | --- | --- | --- | | RTE (Acc) | 128 | 70.40 $±$ 0.72 | 68.11 $±$ 2.18 | 66.78 $±$ 1.53 | | STS-B (Pears.) | 128 | 85.79 $±$ 0.23 | 82.02 $±$ 2.63 | 83.76 $±$ 0.77 | | WNLI (Acc) | 128 | 56.34 $±$ 0.00 | 56.34 $±$ 0.00 | 56.34 $±$ 0.00 | | QNLI (Acc) | 128 | 88.21 $±$ 0.08 | 82.04 $±$ 4.79 | 85.78 $±$ 0.93 | | QQP (F1) | 128 | 86.53 $±$ 0.24 | 82.61 $±$ 2.93 | 83.45 $±$ 0.30 | | MNLI (Acc) | 128 | 75.67 $±$ 0.35 | 72.44 $±$ 3.63 | 75.58 $±$ 0.55 | Table 5. Vision benchmark accuracy (%) for ViT-base (default configuration per Table 3). | Dataset CIFAR-10 CIFAR-100 | Digital 97.01 $±$ 0.28 86.05 $±$ 0.28 | Bilinear 96.74 $±$ 0.65 83.90 $±$ 0.84 | Trilinear 95.53 $±$ 0.09 81.44 $±$ 1.40 | | --- | --- | --- | --- | | ImageNet-1K | 79.39 $±$ 1.04 | 79.11 $±$ 1.06 | 74.98 $±$ 2.05 | ### 6.3. PPA Analysis Table 6 presents the per-inference performance, power, and area (PPA) comparison between bilinear and trilinear for BERT-base inference under the default configuration with 2-bit cells and an 8-bit ADC (2b/8b). Because per-inference PPA is dominated by model architecture and sequence length rather than dataset content, tasks sharing the same sequence length exhibit only minor differences in these metrics. Per-Inference Efficiency (Table 6). The trilinear mode achieves consistent improvements at both evaluated sequence lengths (64 and 128 tokens): 18.6–20.4% latency reduction and 39.7–46.6% energy savings, with the largest energy gains on 64-token tasks. In TransCIM, the energy saved by eliminating dynamic writes becomes less significant at longer sequence lengths. This trend reflects the different scaling of the underlying operations. As sequence length increases, attention computation must evaluate interactions between many more token pairs, so the associated CIM reads and accumulations grow approximately quadratically. In contrast, bilinear write energy is tied to forming dynamic operands once per token and therefore grows only linearly. Consequently, at sequence length 128, the energy saved by eliminating dynamic writes becomes a smaller fraction of total inference energy. Trilinear energy efficiency reaches 12.2–13.5 TOPS/W under default GLUE sequence-length caps (64 tokens for CoLA/SST-2 and 128 tokens for the remaining tasks), an improvement of 23–39% over bilinear at the same caps, demonstrating that eliminating write operations yields substantial efficiency gains beyond the raw energy savings. Area and Compute Density. The trilinear mode incurs a constant 37.3% area overhead, independent of sequence length, reflecting the additional back-gate driver circuitry and per-column DAC infrastructure. Compute density is mixed: at 128-token context, TOPS/mm 2 increases by 9.5% versus bilinear, whereas at 64-token context the smaller bilinear footprint yields ${∼}$ 10% lower trilinear TOPS/mm 2 despite higher trilinear TOPS/W. Memory utilization remains stable at ${∼}$ 87% for trilinear (vs. ${∼}$ 84% for bilinear), reflecting slightly better tile-level packing under the trilinear attention mapping. Table 6. Per-inference PPA comparison by sequence length (BERT-base, default configuration per Table 3). | Metric Area (mm 2) | Seq 64 Bil. 326 | Seq 128 Tri. 447 | $Δ$ % +37.3 | Bil. 651 | Tri. 894 | $Δ$ % +37.3 | | --- | --- | --- | --- | --- | --- | --- | | Latency (ms) | 7.63 | 6.08 | $-$ 20.4 | 8.19 | 6.67 | $-$ 18.6 | | Energy ( $μ$ J) | 1,522 | 813 | $-$ 46.6 | 3,132 | 1,889 | $-$ 39.7 | | Throughput (inf/s) | 131 | 164 | +25.5 | 122 | 150 | +22.7 | | TOPS/W | 9.97 | 12.24 | +22.8 | 9.68 | 13.47 | +39.2 | | TOPS/mm 2 | 0.0056 | 0.0050 | $-$ 9.9 | 0.0053 | 0.0058 | +9.5 | | Mem. Util. (%) | 84.5 | 87.4 | +3.4 | 84.5 | 87.4 | +3.4 | ### 6.4. Ablation Studies We ablate three design-space axes—sub-array size, bitcell/ADC precision, and sequence length—to characterize the trilinear architecture’s sensitivity to key parameters. Fig. 7 visualizes the sub-array size sweep, while Table 7 and Fig. 8 detail the precision sweep’s PPA and per-task performance metric, respectively. A. Sub-Array Size (2b/8b). Fig. 7 compares 32 $×$ 32 and 64 $×$ 64 sub-arrays with bitcell (2b) and ADC precision (8b) fixed; PPA values are per-inference metrics representative of all seq = 128 GLUE tasks. As the figure shows, 32 2 yields the largest latency reduction ( $-$ 40.9%) and near-perfect memory utilization, while 64 2 delivers the larger relative energy reduction on this seq = 128 bucket ( $-$ 39.7% vs. $-$ 30.9%) and the higher absolute TOPS/W (13.47 vs. 9.38). Although the trilinear area overhead is smaller at 32 2 (+17.8% vs. +37.3%), its absolute chip area remains larger because smaller sub-arrays replicate more peripheral circuitry. TOPS/W improves at both sizes (6.70 $→$ 9.38 at 32 2; 9.68 $→$ 13.47 at 64 2), and the higher absolute TOPS/W at 64 2 reflects its greater per-read analog parallelism (64 rows vs. 32). Accuracy remains robust at 32 2: trilinear SST-2 reaches 91.70 $±$ 0.05% (vs. bilinear 89.64 $±$ 0.52%) and MNLI 73.48 $±$ 0.70% (vs. 67.77 $±$ 6.46%), closely tracking or matching the 64 2 default (Table 4). The substantially lower trilinear standard deviation at 32 2 ( $<$ 1% on most tasks) versus bilinear (up to ${∼}$ 6.5%) reinforces the stability advantage of write-free inference. Across both configurations, trilinear avoids dynamic writes entirely (0 vs. 18.9M cells per inference for bilinear at seq = 128). B. Bitcell/ADC Precision (SA=64 $×$ 64). Table 7 summarizes PPA for four bitcell/ADC configurations, with per-task performance metric detailed in Fig. 8. The 1b/6b configuration emerges as the strongest accuracy point: trilinear achieves the largest per-task improvements over bilinear (MNLI $+$ 7.13%, SST-2 $+$ 1.30%, CoLA $+$ 9.66% as shown in Fig. 8), while delivering 37.5% energy reduction and 26.0% latency improvement at only 32.4% area overhead—consistently lower than the ${∼}$ 37% incurred by 2-bit configurations (Table 7), making it particularly compelling for accuracy-critical deployments. The 2b/7b configuration (omitted from the table) demonstrates that 2-bit cells require at least 8-bit ADC precision—below this threshold, both CIM modes collapse to chance-level accuracy, establishing ADC precision as the binding constraint for multi-bit cells. Increasing ADC precision beyond the default brings diminishing returns: 2b/9b gains only marginal accuracy while increasing energy and area (37.4% area overhead). Similarly, 1b/7b and 1b/8b (not shown) offer progressively lower TOPS/W, confirming that 1b/6b is the optimal low-precision operating point. <details> <summary>2604.07628v1/x8.png Details</summary>  ### Visual Description ## Bar Chart: Performance Comparison of Bilinear and Trilinear Methods ### Overview The image contains three side-by-side bar charts comparing the performance of Bilinear (gray bars) and Trilinear (blue bars) methods across three metrics: Energy (mJ), Latency (ms), and Area (mm²) for two input sizes: 32² and 64². Each chart includes percentage-based annotations indicating relative improvements or increases. ### Components/Axes - **X-Axes**: - Energy: 32², 64² - Latency: 32², 64² - Area: 32², 64² - **Y-Axes**: - Energy: 0–5 mJ - Latency: 0–10 ms - Area: 0–1500 mm² - **Legend**: - Position: Bottom center - Labels: - Gray: Bilinear - Blue: Trilinear ### Detailed Analysis #### Energy (mJ) - **32²**: - Bilinear: 4.2 mJ (↓31%) - Trilinear: 2.8 mJ (↓40%) - **64²**: - Bilinear: 3.2 mJ (↓40%) - Trilinear: 1.8 mJ (↓44%) #### Latency (ms) - **32²**: - Bilinear: 9.0 ms (↓41%) - Trilinear: 5.4 ms (↓19%) - **64²**: - Bilinear: 8.0 ms (↓19%) - Trilinear: 6.0 ms (↓25%) #### Area (mm²) - **32²**: - Bilinear: 1000 mm² (↑18%) - Trilinear: 1180 mm² (↑37%) - **64²**: - Bilinear: 600 mm² (↑37%) - Trilinear: 820 mm² (↑18%) ### Key Observations 1. **Energy Efficiency**: - Trilinear consistently reduces energy consumption by 18–44% compared to Bilinear across both sizes. 2. **Latency**: - Trilinear reduces latency by 19–41% for 32² but shows smaller gains (19–25%) for 64². 3. **Area**: - Trilinear increases area by 18–37% for 32² but reduces the increase to 18% for 64², while Bilinear shows a larger area increase (37%) for 64². ### Interpretation - **Trade-offs**: Trilinear optimizes energy and latency at the cost of higher area usage, particularly for smaller input sizes (32²). Bilinear uses less area but sacrifices energy efficiency and latency improvements. - **Scalability**: The performance gap between methods narrows for larger input sizes (64²), suggesting diminishing returns for Trilinear in terms of energy and latency savings. - **Design Implications**: The choice between methods depends on application priorities. Trilinear is preferable for energy-constrained systems, while Bilinear may suit area-constrained designs. </details> Figure 7. Sub-array size ablation (2b/8b, seq = 128, per inference). Energy, latency, and chip area for 32 2 and 64 2 sub-arrays. Green annotations denote trilinear improvement over bilinear; red annotations denote area overhead. The 64 2 default (shaded) is used throughout the paper. Table 7. Bitcell/ADC precision ablation (SA=64 $×$ 64, seq 128). PPA deltas are trilinear vs. bilinear. Per-task GLUE score (task-specific metric; see Table 4) is shown in Fig. 8. | | $Δ$ % | $Δ$ % | $Δ$ % | Bil. | Tri. | | --- | --- | --- | --- | --- | --- | | 1b/6b | +32.4 | $-$ 26.0 | $-$ 37.5 | 10.45 | 13.77 | | 1b/7b | +32.4 | $-$ 35.4 | $-$ 32.5 | 7.52 | 10.22 | | 2b/8b | +37.3 | $-$ 18.6 | $-$ 39.7 | 9.68 | 13.47 | | 2b/9b | +37.4 | $-$ 27.5 | $-$ 31.5 | 5.31 | 7.61 | <details> <summary>2604.07628v1/x9.png Details</summary>  ### Visual Description ## Bar Chart: Model Accuracy Comparison Across Configurations ### Overview The image contains four grouped bar charts comparing the accuracy of bilinear (gray) and trilinear (blue) models across eight natural language processing tasks (SST-2, MRPC, RTE, STS-B, WNLI, QNLI, QQP, MNLI) under different computational configurations (1b/6b, 1b/7b, 2b/8b, 2b/9b). Each chart shows mean accuracy percentages with error bars representing standard deviation. ### Components/Axes - **X-axis**: NLP tasks (SST-2, MRPC, RTE, STS-B, WNLI, QNLI, QQP, MNLI) - **Y-axis**: Accuracy (%) from 50% to 90% - **Legend**: - Gray = Bilinear - Blue = Trilinear - **Configuration Labels**: Positioned at top-right of each chart (1b/6b, 1b/7b, 2b/8b, 2b/9b) ### Detailed Analysis #### 1b/6b Configuration - **SST-2**: Bilinear 88% (±2%), Trilinear 87% (±1%) - **MRPC**: Bilinear 85% (±3%), Trilinear 84% (±2%) - **RTE**: Bilinear 65% (±4%), Trilinear 62% (±3%) - **STS-B**: Bilinear 82% (±2%), Trilinear 81% (±1%) - **WNLI**: Bilinear 55% (±3%), Trilinear 55% (±2%) - **QNLI**: Bilinear 88% (±1%), Trilinear 87% (±2%) - **QQP**: Bilinear 83% (±2%), Trilinear 82% (±1%) - **MNLI**: Bilinear 70% (±4%), Trilinear 75% (±3%) #### 1b/7b Configuration - **SST-2**: Bilinear 89% (±1%), Trilinear 88% (±2%) - **MRPC**: Bilinear 86% (±2%), Trilinear 85% (±3%) - **RTE**: Bilinear 63% (±3%), Trilinear 60% (±2%) - **STS-B**: Bilinear 84% (±1%), Trilinear 83% (±2%) - **WNLI**: Bilinear 55% (±2%), Trilinear 55% (±1%) - **QNLI**: Bilinear 87% (±2%), Trilinear 86% (±3%) - **QQP**: Bilinear 85% (±1%), Trilinear 84% (±2%) - **MNLI**: Bilinear 72% (±3%), Trilinear 76% (±2%) #### 2b/8b (Default) Configuration - **SST-2**: Bilinear 89% (±1%), Trilinear 88% (±2%) - **MRPC**: Bilinear 86% (±2%), Trilinear 85% (±3%) - **RTE**: Bilinear 68% (±3%), Trilinear 65% (±2%) - **STS-B**: Bilinear 83% (±1%), Trilinear 82% (±2%) - **WNLI**: Bilinear 55% (±2%), Trilinear 55% (±1%) - **QNLI**: Bilinear 86% (±2%), Trilinear 85% (±3%) - **QQP**: Bilinear 84% (±1%), Trilinear 83% (±2%) - **MNLI**: Bilinear 73% (±3%), Trilinear 77% (±2%) #### 2b/9b Configuration - **SST-2**: Bilinear 89% (±1%), Trilinear 88% (±2%) - **MRPC**: Bilinear 87% (±2%), Trilinear 86% (±3%) - **RTE**: Bilinear 65% (±3%), Trilinear 62% (±2%) - **STS-B**: Bilinear 84% (±1%), Trilinear 83% (±2%) - **WNLI**: Bilinear 55% (±2%), Trilinear 55% (±1%) - **QNLI**: Bilinear 87% (±2%), Trilinear 86% (±3%) - **QQP**: Bilinear 85% (±1%), Trilinear 84% (±2%) - **MNLI**: Bilinear 74% (±3%), Trilinear 78% (±2%) ### Key Observations 1. **Consistent Performance**: Bilinear models consistently outperform trilinear models across all tasks and configurations, with the largest gap in RTE (65% vs 62% in 1b/6b). 2. **Task-Specific Variance**: WNLI shows minimal difference between architectures (55% for both), while RTE exhibits the largest performance gap. 3. **Configuration Impact**: Larger configurations (2b/9b) show slightly improved accuracy for both architectures compared to smaller ones (1b/6b), with MNLI showing the most improvement (70%→78% for trilinear). 4. **Error Bar Patterns**: Trilinear models generally show smaller error bars, suggesting more consistent performance across runs. ### Interpretation The data demonstrates that bilinear architectures maintain higher accuracy across diverse NLP tasks and computational constraints. The trilinear models show comparable performance in most cases but exhibit notable drops in RTE and WNLI tasks. The performance gap between architectures narrows in larger configurations (2b/9b), suggesting that increased computational resources may mitigate some of the trilinear model's limitations. The consistent performance of bilinear models across configurations implies architectural advantages that persist regardless of scale, while trilinear models show more variability in task-specific performance. The WNLI task's parity between architectures might indicate task-specific architectural suitability rather than general superiority. </details> Figure 8. Per-task GLUE benchmark scores for four bitcell/ADC configurations (SA=64 $×$ 64, seq = 128). Grey: bilinear; blue: trilinear. Error bars show $±$ 1 std over 2–3 seeds. CoLA (Matthews Corr.) omitted due to its different metric scale. C. Sequence Length Scaling (2b/8b, SA=64 $×$ 64). The trilinear advantage in energy and latency is maintained when each GLUE task’s default context is doubled: CoLA and SST-2 scale from 64 to 128 tokens, while MRPC, RTE, STS-B, WNLI, QNLI, QQP, and MNLI scale from 128 to 256 tokens. For CoLA and SST-2, doubling context from 64 to 128 tokens changes the energy reduction from 46.6% to 39.5% and the latency reduction from 20.4% to 18.6%, while the TOPS/W gain rises from +22.8% to +38.5%. For the remaining GLUE tasks, doubling context from 128 to 256 tokens changes the energy reduction from 39.7% to 27.4% and the latency reduction from 18.6% to 16.1%, while the TOPS/W gain rises from +39.2% to +68.6%. Accuracy remains task-dependent under doubled context. From an endurance perspective, the critical advantage at longer sequences appears in write volume: in the doubled-context sweep, bilinear incurs 9.4M cell writes per inference for the 128-token CoLA/SST-2 bucket and 18.9M for the 256-token bucket used by the remaining GLUE tasks, while trilinear remains at exactly zero. For deployment scenarios with long contexts (document understanding, multi-turn dialogue), eliminating these runtime writes becomes increasingly important as sequence length grows. ### 6.5. Discussion Design trade-offs. The ablation studies reveal two promising operating points beyond the default: (1) the 1b/6b configuration, which delivers the strongest accuracy advantage while still reducing energy by 37.5% at only 32.4% area overhead (Table 7, Fig. 8); (2) the 32 $×$ 32 sub-array, which delivers the largest latency reduction, near-perfect memory utilization, and a smaller trilinear-vs.-bilinear area overhead than 64 2 (+17.8% vs. +37.3%) (Fig. 7). The default 64 $×$ 64 with 2b/8b balances these trade-offs by combining reliable accuracy across all GLUE tasks with higher absolute TOPS/W than the 32 $×$ 32 alternative and a smaller absolute chip footprint on the seq = 128 bucket. Area overhead justification. The area overhead (Table 6) is partially offset at 128-token context by improved TOPS/mm 2; at 64-token context, energy efficiency gains dominate despite slightly lower trilinear TOPS/mm 2. These results suggest that the extra area can be justified when energy savings are prioritized. Scalability. The sequence length sweep (Section 6.4) confirms that the trilinear advantage is maintained as context length grows. For decoder-style causal attention, future tokens can be masked by zeroing the corresponding back-gate voltages, though KV-cache management under this dataflow requires further investigation. Limitations. Our evaluation relies on the TransCIM framework (Section 5), which builds on validated NeuroSim circuit models (Peng et al., 2019), with DG-FeFET parameters from reported experimental characterizations (Jiang et al., 2025); however, the chosen back-gate operating range and its band-averaged linear approximation should still be validated in fabricated hardware. We also leave algorithmic mitigation of the ViT accuracy gap to future work, including hardware-aware fine-tuning or noise-aware training (Li et al., 2023). Endurance. Under our operating assumption, trilinear attention computation bypasses ferroelectric switching: back-gate modulation acts through a non-ferroelectric dielectric and therefore does not intentionally perturb the stored polarization state. Read-disturb and other device-level reliability effects should still be validated experimentally. ## 7. Related Work Table 8. Comparison of attention-computation strategies across CIM, PIM, and digital transformer accelerators. | ISAAC/PRIME (Shafiee et al., 2016; Chi et al., 2016) | ReRAM | Crossbar architectures for static-weight CNNs/MLPs; no attention dataflow | — | Cannot handle dynamic operands in $QK^T$ or Score $· V$ | | --- | --- | --- | --- | --- | | PipeLayer (Song et al., 2017) | ReRAM | Weight-gradient pipelined training on ReRAM crossbars; no inference attention path | — | Training-only; no mechanism for dynamic attention during inference | | ReTransformer (Yang et al., 2020) | ReRAM | Decomposes $QK^T{=}(QW_K)X^T$ to keep static $W_K$ on crossbar, avoiding $K^T$ write | Reduced | Score $· V$ still writes $V$ to ReRAM; write endurance consumed | | Sanger (Lu et al., 2021) | Digital ASIC | Reconfigurable score-stationary architecture with structured attention pruning | — | Digital-only; no NVM integration for weight storage | | X-Former (Sridharan et al., 2023) | ReRAM + CMOS | Dual-engine: ReRAM for static projections, CMOS attention engine for dynamic attention | Zero | Forfeits NVM density for the attention engine; additional CMOS area and leakage overhead | | iMTransformer (Laguna et al., 2022) | FeFET + CMOS | FeFET crossbars for static projections; CMOS crossbars/CAMs cache attention scores | Zero | Heterogeneous FeFET+CMOS fabric; attention engine sacrifices NVM density | | TransPIM (Zhou et al., 2022b) | DRAM-PIM | HBM-based PIM with token-parallel digital dataflow | — | Volatile DRAM: static leakage, refresh overhead, limited compute density vs. NVM | | Yang et al. (Yang et al., 2022) | Memristor | Full-analog memristor circuit with capacitor-based analog memory for attention intermediates | Zero | Capacitor charge decay limits retention; demonstrated only at $3{×}3$ scale | | Trilinear CIM (Ours) | DG-FeFET | Back-gate modulation enables in-memory attention without runtime ferroelectric rewriting | Zero | Requires per-column BGL drivers and operation within the selected conductance range | CIM Accelerators for Static Workloads. ISAAC (Shafiee et al., 2016) and PRIME (Chi et al., 2016) pioneered ReRAM-based crossbar architectures with pipelined inter-layer execution and dual-mode memory-compute cells, respectively. PipeLayer (Song et al., 2017) extended this paradigm to on-chip training through weight-gradient parallelism, and PUMA (Ankit et al., 2019) introduced a programmable ISA for general DNN graph mapping. These designs target static weight matrices and do not address the dynamic operand challenge analyzed in Section 3.1. Transformer-Specific CIM and Processing-in-Memory (PIM) Accelerators. Adapting CIM to Transformers requires addressing the dynamic nature of attention, where operands ( $Q$ , $K$ , $V$ ) change with every input sequence. Prior works span four main strategies. (i) Algorithmic write reduction: ReTransformer (Yang et al., 2020) reformulates the score computation as $Q· K^T=(Q· W_K)· X^T$ via matrix decomposition, keeping the static projection weight $W_K$ on the ReRAM crossbar and avoiding the explicit write of the intermediate $K^T$ matrix. However, the value-aggregation stage ( $Score· V$ ) still requires writing $V$ to the crossbar, and the fundamental endurance and write-latency constraints of ReRAM remain. (ii) Hybrid NVM+CMOS: X-Former (Sridharan et al., 2023) partitions the workload across a ReRAM-based Projection Engine for static weights and a separate CMOS attention engine for the dynamic $Q· K^T$ and $Score· V$ operations, avoiding NVM writes entirely for attention at the cost of NVM density. Similarly, iMTransformer (Laguna et al., 2022) combines FeFET crossbars for static projection weights with CMOS-based crossbars and CAMs that cache attention scores for reuse. The same group also explored FeFET-based attention-in-memory for few-shot learning (Reis et al., 2021). (iii) DRAM-PIM: TransPIM (Zhou et al., 2022b) operates entirely in the DRAM domain using HBM-based processing-in-memory with a token-based dataflow, sidestepping NVM altogether but inheriting the limited compute density and leakage of volatile DRAM. (iv) Full-analog transient storage: Yang et al. (Yang et al., 2022) propose a full-analog memristor circuit that stores intermediate attention results in capacitor-based analog memory rather than NVM, avoiding write endurance concerns but demonstrating only at a $3× 3$ scale. Table 8 summarizes these approaches and includes representative digital transformer accelerators for context. Digital Transformer Accelerators. In the purely digital domain, SpAtten (Wang et al., 2021) introduces cascade token and head pruning with progressive quantization to reduce attention computation and DRAM access. A 3 (Ham et al., 2020) accelerates attention through approximate search, while Energon (Zhou et al., 2022a) employs mix-precision multi-round filtering to dynamically identify critical query-key pairs. Sanger (Lu et al., 2021) co-designs structured attention pruning with a reconfigurable score-stationary architecture to reduce dense attention computation, operating entirely in the digital domain without NVM integration. These designs achieve significant speedups over GPUs but remain fundamentally data-movement-bound, as the attention matrices must still traverse the memory hierarchy. ## 8. Conclusion This paper evaluates a DG-FeFET-based trilinear CIM dataflow that addresses the dynamic-operand bottleneck of Transformer attention without runtime ferroelectric rewriting. By using the back-gate as a third operand pathway, the design keeps projection weights stationary in the top gate while modulating the effective conductance during inference through volatile back-gate control. In our evaluation, this dataflow removes the dynamic write burden that dominates conventional bilinear CIM attention execution. Across BERT-base and ViT-base workloads, eliminating dynamic writes yields up to 20.4% faster inference and 46.6% lower energy per inference, with trilinear matching or exceeding the bilinear baseline on seven of nine GLUE tasks; the ablation sweep further highlights 1b/6b as the accuracy-optimal and 32 $×$ 32 as a latency-optimized, high-utilization alternative to the 64 $×$ 64 default. These benefits trade against a 37.3% silicon footprint increase from the per-column back-gate drivers and lower vision accuracy than the bilinear baseline. Because the study is simulation-based and grounded in the TransCIM framework with experimentally characterized DG-FeFET parameters, array-level validation of the chosen back-gate operating range and device-level reliability remains future work. Overall, the results indicate that performing the full attention dataflow inside NVM arrays is a viable architectural direction, especially for workloads where avoiding dynamic writes outweighs the added back-gate circuitry. Acknowledgements. Research was sponsored primarily by the the National Science Foundation under award No. EFRI BRAID #2318101. This work was also supported partially by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR) program as part of the NSF/NIH/DOE/ANR/BMBF/ BSF/NICT/AEI/ISCIII Collaborative Research in Computational Neuroscience (CRCNS) Program, project PARADIGM. The authors acknowledge the use of LLMs in the preparation of this manuscript. Specifically, GPT-5.4 (OpenAI) and Claude 4.6 (Anthropic) were used for wording refinement, limited code and figure-script edits, and identifying potentially relevant prior work. All technical claims, numerical values, cited references, and final text were independently verified by the authors, who take full responsibility for the manuscript. ## References - (1) - Al Mamun et al. (2022) Fahad Al Mamun, Dragica Vasileska, and Ivan Sanchez Esqueda. 2022. Impact of back-gate biasing on the transport properties of 22 nm FD-SOI MOSFETs at cryogenic temperatures. IEEE Transactions on Electron Devices 69, 10 (2022), 5417–5423. - Ankit et al. (2019) Aayush Ankit, Izzat El Hajj, Sai Rahul Chalamalasetti, Geoffrey Ndu, Martin Foltin, R. Stanley Williams, Paolo Faraboschi, Wen-mei W. Hwu, John Paul Strachan, Kaushik Roy, and Dejan Milojicic. 2019. PUMA: A programmable ultra-efficient memristor-based accelerator for machine learning inference. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems. 715–731. - Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33. 1877–1901. - Chen (2016) An Chen. 2016. A review of emerging non-volatile memory (NVM) technologies and applications. Solid-State Electronics 125 (2016), 25–38. https://doi.org/10.1016/j.sse.2016.07.006 - Chen et al. (2018) Pai-Yu Chen, Xiaochen Peng, and Shimeng Yu. 2018. NeuroSim: A circuit-level macro model for benchmarking neuro-inspired architectures in online learning. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 37, 12 (2018), 3067–3080. - Chi et al. (2016) Ping Chi, Shuangchen Li, Cong Xu, Tao Zhang, Jishen Zhao, Yongpan Liu, Yu Wang, and Yuan Xie. 2016. Prime: A novel processing-in-memory architecture for neural network computation in reram-based main memory. ACM SIGARCH Computer Architecture News 44, 3 (2016), 27–39. - Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248–255. - Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 4171–4186. - Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR). - Ham et al. (2020) Tae Jun Ham, Sung Jun Jung, Seonghak Kim, Young H. Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W. Lee, and Deog-Kyoon Jeong. 2020. A3: Accelerating Attention Mechanisms in Neural Networks with Approximation. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 328–341. - Han et al. (2022) Hung-Chi Han, Farzan Jazaeri, Antonio D’Amico, Zhixing Zhao, Steffen Lehmann, Claudia Kretzschmar, Edoardo Charbon, and Christian Enz. 2022. Back-gate effects on DC performance and carrier transport in 22 nm FDSOI technology down to cryogenic temperatures. Solid-State Electronics 193 (2022), 108296. - Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415 (2016). - Horowitz (2014) Mark Horowitz. 2014. 1.1 computing’s energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC). IEEE, 10–14. - Ielmini and Wong (2018) Daniele Ielmini and H-S Philip Wong. 2018. In-memory computing with resistive switching devices. Nature electronics 1, 6 (2018), 333–343. - Jerry et al. (2017) Matthew Jerry, Pai-Yu Chen, Jianchi Zhang, Pankaj Sharma, Kai Ni, Shimeng Yu, and Suman Datta. 2017. Ferroelectric FET analog synapse for acceleration of deep neural network training. In 2017 IEEE international electron devices meeting (IEDM). IEEE, 6–2. - Jiang et al. (2025) Zhouhang Jiang, A. N. M. Nafiul Islam, Zhuangyu Han, Zijian Zhao, Franz Müller, Jiahui Duan, Halid Mulaosmanovic, Stefan Dünkel, Sven Beyer, Sourav Dutta, Vijaykrishnan Narayanan, Thomas Kämpfe, Suma George Cardwell, Frances Chance, Abhronil Sengupta, and Kai Ni. 2025. A Bio-inspired Asymmetric Double-Gate Ferroelectric FET for Emulating Astrocyte and Dendrite Dynamics in Neuromorphic Systems. arXiv preprint arXiv:2504.14466 (2025). - Jiang et al. (2022) Zhouhang Jiang, Yi Xiao, Swetaki Chatterjee, Halid Mulaosmanovic, Stefan Duenkel, Steven Soss, Sven Beyer, Rajiv Joshi, Yogesh Singh Chauhan, Hussam Amrouch, Vijaykrishnan Narayanan, and Kai Ni. 2022. Asymmetric double-gate ferroelectric FET to decouple the tradeoff between thickness scaling and memory window. In 2022 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits). IEEE, 395–396. - Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009). - Laguna et al. (2022) Ann Franchesca Laguna, Mohammed Mehdi Sharifi, Arman Kazemi, Xunzhao Yin, Michael Niemier, and X Sharon Hu. 2022. Hardware-software co-design of an in-memory transformer network accelerator. Frontiers in Electronics 3 (2022), 847069. - Li et al. (2023) Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. 2023. RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 17227–17236. - Lim and Fossum (1983) Hyung-Kyu Lim and Jerry G Fossum. 1983. Threshold voltage of thin-film silicon-on-insulator (SOI) MOSFET’s. IEEE Transactions on Electron Devices 30, 10 (1983), 1244–1251. - Lin et al. (2022) Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. 2022. FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI). 1173–1179. - Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 10012–10022. - Lu et al. (2021) Lizi Lu, Jasyuant Xie, Zhenhua Huang, and Huazhong Wang. 2021. Sanger: A Co-Design Framework for Enabling Sparse Attention using Reconfigurable Architecture. In 2021 54th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 977–991. - Mulaosmanovic et al. (2019) Halid Mulaosmanovic, Evelyn T Breyer, Thomas Mikolajick, and Stefan Slesazeck. 2019. Ferroelectric FETs with 20-nm-thick HfO 2 layer for large memory window and high performance. IEEE Transactions on Electron Devices 66, 9 (2019), 3828–3833. - Mulaosmanovic et al. (2021) Halid Mulaosmanovic, Dominik Kleimaier, Stefan Dünkel, Sven Beyer, Thomas Mikolajick, and Stefan Slesazeck. 2021. Ferroelectric transistors with asymmetric double gate for memory window exceeding 12 V and disturb-free read. Nanoscale 13, 38 (2021), 16258–16266. - Ni et al. (2019) Kai Ni, Xunzhao Yin, Ann Franchesca Laguna, Siddharth Joshi, Stefan Dünkel, Martin Trentzsch, Johannes Müller, Sven Beyer, Michael Niemier, Xiaobo Sharon Hu, and Suman Datta. 2019. Ferroelectric ternary content-addressable memory for one-shot learning. Nature Electronics 2, 11 (2019), 521–529. - Nier et al. (2013) Olivier Nier, Denis Rideau, Yann-Michel Niquet, Frédéric Monsieur, Viet Hung Nguyen, François Triozon, Antoine Cros, Raphael Clerc, Jean-Christophe Barbé, Pierpaolo Palestri, David Esseni, Ivan Duchemin, Lee Smith, Luca Silvestri, Frédéric Nallet, Christine Tavernier, Hugues Jaouen, and Luca Selmi. 2013. Multi-scale strategy for high-k/metal-gate UTBB-FDSOI devices modeling with emphasis on back bias impact on mobility. Journal of Computational Electronics 12 (2013), 675–684. - Peng et al. (2019) Xiaochen Peng, Shanshi Huang, Yandong Luo, Xiaoyu Sun, and Shimeng Yu. 2019. DNN+ NeuroSim: An end-to-end benchmarking framework for compute-in-memory accelerators with versatile device technologies. In 2019 IEEE international electron devices meeting (IEDM). IEEE, 32–5. - Reis et al. (2021) Dayane Reis, Ann Franchesca Laguna, Michael Niemier, and Xiaobo Sharon Hu. 2021. Attention-in-memory for few-shot learning with configurable ferroelectric FET arrays. In Proceedings of the 26th Asia and South Pacific Design Automation Conference. 49–54. - Sebastian et al. (2020) Abu Sebastian, Manuel Le Gallo, Riduan Khaddam-Aljameh, and Evangelos Eleftheriou. 2020. Memory devices and applications for in-memory computing. Nature nanotechnology 15, 7 (2020), 529–544. - Shafiee et al. (2016) Ali Shafiee, Anirban Nag, Naveen Muralimanohar, Rajeev Balasubramonian, John Paul Strachan, Miao Hu, R Stanley Williams, and Vivek Srikumar. 2016. ISAAC: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars. ACM SIGARCH Computer Architecture News 44, 3 (2016), 14–26. - Song et al. (2017) Linghao Song, Xuehai Qian, Hai Li, and Yiran Chen. 2017. Pipelayer: A pipelined reram-based accelerator for deep learning. In 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 541–552. - Sridharan et al. (2023) Shrihari Sridharan, Jacob R Stevens, Kaushik Roy, and Anand Raghunathan. 2023. X-Former: In-memory acceleration of transformers. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 31, 8 (2023), 1223–1233. - Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 3645–3650. - Sun et al. (2022) Mengshu Sun, Haoyu Ma, Guoliang Liu, Yancheng Jiang, Tianhao Shen, and Bei Yu. 2022. VAQF: Fully automatic software-hardware co-design framework for low-bit vision transformer. arXiv preprint arXiv:2201.06618 (2022). - Tay et al. (2022) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022. Efficient Transformers: A Survey. ACM Comput. Surv. 55, 6, Article 109 (Dec. 2022), 28 pages. - Van Beek et al. (2023) S. Van Beek, K. Cai, F. Yasin, H. Hody, G. Talmelli, V. D. Nguyen, N. Franchina Vergel, A. Palomino, A. Trovato, K. Wostyn, S. Rao, G. S. Kar, and S. Couet. 2023. Scaling the SOT track – A path towards maximizing efficiency in SOT-MRAM. In 2023 International Electron Devices Meeting (IEDM). IEEE, 1–4. https://doi.org/10.1109/IEDM45741.2023.10413749 - Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30. - Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. 353–355. - Wang et al. (2020) Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. 2020. HAT: Hardware-Aware Transformers for Efficient Natural Language Processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 7675–7688. - Wang et al. (2021) Hanrui Wang, Zhekai Zhang, and Song Han. 2021. SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 97–110. - Yang et al. (2022) Chao Yang, Xiaoping Wang, and Zhigang Zeng. 2022. Full-circuit implementation of transformer network based on memristor. IEEE Transactions on Circuits and Systems I: Regular Papers 69, 4 (2022), 1395–1407. https://doi.org/10.1109/TCSI.2021.3136355 - Yang et al. (2020) Xiaoxuan Yang, Bonan Yan, Hai Li, and Yiran Chen. 2020. ReTransformer: ReRAM-based processing-in-memory architecture for transformer acceleration. In Proceedings of the 39th International Conference on Computer-Aided Design. 1–9. - Yin et al. (2018) Xunzhao Yin, Xiaoming Chen, Michael Niemier, and Xiaobo Sharon Hu. 2018. Ferroelectric FETs-based nonvolatile logic-in-memory circuits. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 27, 1 (2018), 159–172. - Yuan et al. (2022) Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun. 2022. PTQ4ViT: Post-Training Quantization for Vision Transformers with Twin Uniform Quantization. In European Conference on Computer Vision (ECCV). 191–207. - Zhang et al. (2024) Xuan Zhang, Zhuoran Song, Xing Li, Zhezhi He, Naifeng Jing, Li Jiang, and Xiaoyao Liang. 2024. Watt: A Write-Optimized RRAM-Based Accelerator for Attention. In European Conference on Parallel Processing (Euro-Par). Springer, 111–125. - Zhou et al. (2022b) Minxuan Zhou, Weihong Xu, Jaeyoung Kang, and Tajana Rosing. 2022b. Transpim: A memory-based acceleration via software-hardware co-design for transformer. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 1071–1085. - Zhou et al. (2022a) Zhe Zhou, Junlin Liu, Zhenyu Gu, and Guangyu Sun. 2022a. Energon: Toward efficient acceleration of transformers using dynamic sparse attention. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 42, 1 (2022), 136–149.