## FPGA-Based CNN Inference Accelerator Synthesized from Multi-Threaded C Software
†
Samsung Semiconductor Inc., San Jose, CA, USA
Jin Hee Kim, Brett Grady, Ruolong Lian, John Brothers , Jason H. Anderson Dept. of Electrical and Computer Engineering, University of Toronto, Toronto, ON, Canada †
Email:
{ kimjin14, bgrady, janders } @ece.utoronto.ca
Abstract -A deep-learning inference accelerator is synthesized from a C -language software program parallelized with Pthreads. The software implementation uses the well-known producer/consumer model with parallel threads interconnected by FIFO queues. The LegUp high-level synthesis (HLS) [1] tool synthesizes threads into parallel FPGA hardware, translating software parallelism into spatial parallelism. A complete system is generated where convolution, pooling and padding are realized in the synthesized accelerator, with remaining tasks executing on an embedded ARM processor. The accelerator incorporates reduced precision, and a novel approach for zero-weight-skipping in convolution. On a mid-sized Intel Arria 10 SoC FPGA, peak performance on VGG-16 is 138 effective GOPS.
## I. INTRODUCTION
State-of-the-art accuracy results in image recognition, language translation, image-caption generation, and many other tasks are being achieved with deep convolutional neural networks (CNNs) (e.g. [2], [3], [4]). CNN training is very compute intensive, requiring hours, days or weeks of time using state-of-the-art graphics processing units (GPUs). Applying a trained CNN to a recognition task, inference , can involve billions of operations. Hardware acceleration is particularly desirable for inference, as training is typically done once offline, whereas inference with a trained network is applied repeatedly. Moreover, there is increased emphasis on performing CNN inference in an embedded-computing context (e.g. mobile handsets, self-driving cars), where low-power and low latency are important metrics. In this paper, we focus on acceleration of CNN inference.
CNN inference has been accelerated with GPUs, custom ASICs, and recently, field-programmable gate arrays (FPGAs). At present, it is unclear which IC media will ultimately prevail as best for CNN inference acceleration. However, the speed at which CNN research is evolving, as well as recent research on low-precision CNNs [5] bodes well for FPGA technology. With FPGAs, the accelerator architecture and its datapath widths can be precisely tailored to the target CNN, whereas an ASIC or GPU are necessarily over-engineered with fixed-sized datapaths to handle a broad set of precisions. Moreover, the reconfigurability of FPGAs permits an accelerator design to be adapted to incorporate new research findings as they arise, for example, the ability to achieve high recognition accuracy with 2-bit precision [6]. Lastly, high-level synthesis (HLS) is a relatively mature design methodology for FPGAs [7], permitting a software specification of the accelerator to be synthesized into hardware. HLS lowers NRE costs by allowing design and debugging to proceed at a higher level of abstraction vs. manual RTL design.
We apply HLS and use an FPGA to realize a CNN inference accelerator. The accelerator is described in C and synthesized with the LegUp HLS framework [1]. A unique aspect of LegUp is its ability to synthesize software parallelized with the Pthreads standard into parallel hardware [8]. We leverage the Pthreads synthesis to exploit spatial parallelism on the FPGA. Specifically, we specify the accelerator in software using the producer/consumer parallelization idiom, well known to software engineers. 20 parallel software threads are synthesized by LegUp HLS into streaming hardware comprising compute kernels interconnected by FIFO queues.
The inference accelerator performs key compute-intensive operations: convolution, subsampling (pooling), and padding. Software executing on an embedded on-die ARM processor performs remaining operations to provide a complete end-toend embedded solution. The accelerator architecture incorporates novel features for tiling, data-reuse, and zero-weightskipping, as many CNNs can be pruned without significant loss of accuracy [9]. The accelerator's computations are realized in reduced precision, specifically 8-bit magnitude + sign format. We demonstrate our accelerator on the VGG-16 CNN for image recognition (ImageNet database). Our contributions are as follows:
- An FPGA-based CNN accelerator synthesized from multi-threaded (Pthreads) C software. The software behavior closely resembles the synthesized hardware, easing design and debugging by allowing it to proceed in software.
- Generation and exploration of accelerator architectural variants via software/constraint changes alone.
- Analysis of the efficiency of the HLS implementation, in terms of cycles spent, compared to the theoretical minimum for the architecture.
- A novel architecture for zero-skipping; use of reducedprecision arithmetic.
- A complete end-to-end solution for CNN inference, integrated with Caffe for network training.
- 138 GOPS peak effective performance implementing the VGG-16 CNN for image recognition on a midsized Arria 10 SoC SX660 20 n m FPGA.
## II. BACKGROUND
## A. LegUp High-Level Synthesis
The Pthreads synthesis flow of LegUp HLS is used to synthesize parallel software threads into parallel hardware. The multi-threaded software is written using the producer/consumer paradigm, where threads represent computational kernels and communicate with one another through FIFO queues [8]. Producer threads deposit computed partial results into output queues, which are then retrieved by concurrently running consumer threads, which perform further processing. Note that a given thread can be both a producer and a consumer, receiving inputs from FIFO queues, computing on those inputs, and depositing results to output FIFO queues. The support in LegUp HLS for hardware synthesis of the producer/consumer parallel model is well-aligned with the computational and communication requirements of deep CNN inference.
FIFO queues that interconnect kernels are realized with a LegUp HLS-provided LEGUP\_PTHREAD\_FIFO structure and API, and can be created with user-provided lengths and bitwidths. To illustrate the coding style (used heavily throughout our implementation), the example below shows a function with one input queue, inQ , and one output queue, outQ . The function body contains an infinite while loop that reads an input, inputData from inQ , performs computation to produce output outData , and deposits into outQ . pthread\_fifo\_read and pthread\_fifo\_write are API functions to read-from and write-to queues, respectively. The while loop is pipelined in hardware by LegUp to realize a streaming kernel that accepts new input each clock cycle.
```
void prodCons(LEGUP_PTHREAD_FIFO *inQ, LEGUP_PTHREAD_FIFO *outQ) {
...
while (1) {
inputData = pthread_fifo_read(inQ);
outputData = compute(inputData);
pthread_fifo_write(outQ, outputData);
}
}
```
## B. VGG-16 CNN
The VGG-16 CNN [3] is used as the test vehicle for our accelerator. The input to the CNN is a 224 × 224 RGB image drawn from the 1000-category ImageNet database. The image is first passed through 13 convolution layers interspersed with occasional max-pooling layers, ending with 3 fully connected layers. All convolutional filters are 3 × 3 pixels in dimension. Before each convolutional layer, the input feature maps are padded with 0 s around the perimeter. Max-pooling is done for 2 × 2 regions with a stride of 2. ReLU activation is applied in all cases ( y = max (0 , x ) , where x is the neuron output). The VGG-16 network has over 130M parameters and the reader is referred to [3] for complete details.
## III. ARCHITECTURE
Fig. 1 depicts the system-on-chip (SoC) architecture, consisting of a Cortex A9 hard processor system (HPS), accelerator, DMA controller, and SRAM banks within the Arria 10 FPGA fabric. The on-FPGA banks are backed by offchip DDR4 RAM. The components are connected to one another using Avalon, Intel's on-chip memory mapped bus interface (discussed below). Bus masters are designated with M in the figure; slaves are designated with S. The processor issues instructions to the DMA and accelerator by writing to the memory mapped address (connected through the L3 interconnect). DMA transfers between the off-chip DRAM and FPGA are realized by a direct connection from the DMA unit to the SDRAM controller.
## A. Accelerator Architecture
We first introduce the data representation, as it is necessary to understand the zero-skipping approach. Feature maps are organized into tiles of 4 × 4 values, as shown on the left side of Fig. 2. The center of the figure shows a 16 × 16 feature map, comprising 4 × 4 tiles. These tiles are stored
X0
X4
X8
X1
X5
X9
X2
X6
XA
X3
X7
XB
XC
XD
XE
XF
!"#$%&'%()(%*+#,$-
Fig. 1. System architecture.
<details>
<summary>Image 1 Details</summary>
### Visual Description
\n
## Diagram: FPGA System Architecture with CNN Inference Accelerator
### Overview
The image depicts a block diagram illustrating the architecture of an FPGA (Field-Programmable Gate Array) system, specifically an Arria 10 SoC 20nm FPGA, integrated with a CNN (Convolutional Neural Network) Inference Accelerator. The diagram showcases the data flow and interconnections between various components, including the FPGA HPS (Hard Processor System), ARM Cortex-A9 MPCore, memory components, and the CNN accelerator.
### Components/Axes
The diagram is composed of several key components:
* **FPGA HPS:** Located on the left side of the diagram, labeled "FPGA HPS".
* **Arria 10 SoC 20nm FPGA:** Located below the "FPGA HPS" label.
* **L3 Interconnect:** Connects the FPGA HPS to other components.
* **ARM Cortex-A9 MPCore:** A dual-core processor with CPU 0 and CPU 1, each having L1 Caches (ACP and SCU).
* **L2 Cache:** Connected to the ARM Cortex-A9 MPCore.
* **SDRAM Controller:** Connected to the L2 Cache.
* **Off-chip DDR4 SDRAM:** Memory connected to the SDRAM Controller.
* **DMA:** Direct Memory Access controller.
* **CNN Inference Accelerator:** Located at the top of the diagram.
* **On-FPGA SRAM Banks:** Memory connected to the CNN Inference Accelerator.
* **Interfaces:** Marked with "M" (Master) and "S" (Slave) labels, indicating the interface type.
* **Data Flow Arrows:** Indicate the direction of data transfer between components.
### Detailed Analysis or Content Details
The diagram illustrates the following data flow:
1. Data flows from the FPGA HPS through the L3 Interconnect to the ARM Cortex-A9 MPCore.
2. Within the ARM Cortex-A9 MPCore, data is processed by CPU 0 and CPU 1, utilizing their respective L1 Caches (ACP and SCU).
3. Data is then passed to the L2 Cache and subsequently to the SDRAM Controller.
4. The SDRAM Controller manages access to the Off-chip DDR4 SDRAM.
5. Data also flows from the FPGA HPS to the DMA controller.
6. The DMA controller transfers data to the CNN Inference Accelerator.
7. The CNN Inference Accelerator utilizes On-FPGA SRAM Banks for data storage and processing.
8. The CNN Inference Accelerator has a slave interface to the ARM Cortex-A9 MPCore.
9. The FPGA HPS has a master interface to the ARM Cortex-A9 MPCore.
The interfaces are labeled as follows:
* The connection between the FPGA HPS and the ARM Cortex-A9 MPCore is labeled with both "M" (Master) and "S" (Slave) interfaces.
* The connection between the CNN Inference Accelerator and the ARM Cortex-A9 MPCore is labeled with a "S" (Slave) interface.
* The connection between the DMA and the FPGA HPS is labeled with "M" (Master) and "S" (Slave) interfaces.
### Key Observations
The diagram highlights the integration of a dedicated CNN Inference Accelerator within an FPGA system. This suggests an architecture optimized for accelerating CNN-based applications. The use of both on-chip (SRAM Banks) and off-chip (DDR4 SDRAM) memory indicates a hierarchical memory system designed to balance speed and capacity. The presence of DMA suggests efficient data transfer mechanisms. The dual-core ARM processor provides general-purpose processing capabilities alongside the specialized CNN accelerator.
### Interpretation
This diagram represents a heterogeneous computing architecture. The FPGA provides a flexible platform for implementing custom hardware accelerators like the CNN Inference Accelerator. The ARM Cortex-A9 MPCore handles control and general-purpose tasks, while the CNN accelerator offloads computationally intensive CNN operations. The DMA controller facilitates efficient data transfer between the FPGA HPS, the CNN accelerator, and the memory system. The architecture is designed to achieve high performance and energy efficiency for CNN-based applications by leveraging the strengths of both hardware and software components. The use of both on-FPGA SRAM and off-chip DDR4 SDRAM suggests a strategy to optimize memory access patterns for the CNN accelerator, utilizing fast on-chip memory for frequently accessed data and larger off-chip memory for storing the entire model and datasets. The master/slave interface designations indicate the control flow and data access permissions between the different components. The diagram suggests a system where the FPGA HPS initiates data transfers and controls the overall operation, while the CNN accelerator operates as a co-processor.
</details>
-/0"2$
X0
X4
X8
XC
X0
X4
X8
XC
X0
X4
X8
XC
X0
X4
X8
XC
X3
X7
XB
XF
X3
X7
XB
XF
X3
X7
XB
XF
X3
X7
XB
XF
X0
X4
X8
XC
X0
X4
X8
XC
X0
X4
X8
XC
X0
X4
X8
XC
X1
X5
X9
XD
X1
X5
X9
XD
X1
X5
X9
XD
X1
X5
X9
XD
X2
X6
XA
XE
X2
X6
XA
XE
X2
X6
XA
XE
X2
X6
XA
XE
X3
X7
XB
XF
X3
X7
XB
XF
X3
X7
XB
XF
X3
X7
XB
XF
X2
X6
XA
XE
X2
X6
XA
XE
X2
X6
XA
XE
X2
X6
XA
X3
X7
XB
XF
X3
X7
XB
XF
X3
X7
XB
XF
X3
X7
XB
X0
X4
X8
XC
X0
X4
X8
XC
X0
X4
X8
XC
X0
X4
X8
X1
X5
X9
XD
X1
X5
X9
XD
X1
X5
X9
XD
X1
X5
X9
X2
X6
XA
XE
X2
X6
XA
XE
X2
X6
XA
XE
X2
X6
XA
X3
X0
X1
X7
XB
XF
X3
X7
XB
XF
X3
X7
XB
XF
X3
X7
XB
XF
X4
X8
XC
X0
X4
X8
XC
X0
X4
X8
XC
X0
X4
X8
X5
X9
XD
X1
X5
X9
XD
X1
X5
X9
XD
X1
X5
X9
XC
XD
XE
XF
XC
XD
XE
X1
X5
X9
XD
X1
X5
X9
XD
X1
X5
X9
XD
X1
X5
X9
XD
X2
X6
XA
XE
X2
X6
XA
XE
X2
X6
XA
XE
X2
X6
XA
XE
.$+/,0$%1+2
Fig. 2. Tile concept, feature map, stripe, data layout.
in memory in row-major order, depicted on the right, where colors of tiles correspond to those in the center image. Fig. 2 also introduces the notion of a stripe , which is a region of tiles spanning the entire width of a feature map. Striping is used to subdivide large convolutional layers into smaller ones that can be accommodated in on-chip memory.
A block diagram of the accelerator is shown in Fig. 3. Four banks of on-FPGA SRAM are shown in orange. An entire tile of data (16 values) can be read from an SRAM bank in a single cycle. The on-FPGA SRAM banks are dual-port: reads are from port A; writes are to port B. The primary computing
Fig. 3. Accelerator block diagram (each blue module is synthesized from a software thread to hardware).
<details>
<summary>Image 2 Details</summary>
### Visual Description
\n
## Diagram: Convolutional Neural Network Architecture on FPGA
### Overview
The image depicts a diagram of a convolutional neural network (CNN) architecture implemented on a Field-Programmable Gate Array (FPGA). The diagram illustrates the data flow between various processing units (Convolution, PoolPad, Accum) and memory banks (FPGA SRAM). The architecture appears to be designed for parallel processing, with multiple instances of convolution and accumulation units operating concurrently.
### Components/Axes
The diagram consists of the following key components:
* **Main Controller:** Located at the bottom-left, providing instructions.
* **Data Staging / Control:** Located on the left and right sides, managing data input and output.
* **FPGA SRAM (Bank 0, Bank 1, Bank 2, Bank 3):** Four memory banks, positioned in the corners, storing data.
* **PoolPad:** Units for pooling and padding operations.
* **Convolution:** Units for convolutional operations.
* **Accum0, Accum1, Accum2, Accum3:** Accumulation units.
* **Write2Mem:** Units for writing data to memory.
* **Instruction Block:** A small block connected to the Main Controller, containing the following labels:
* Instruction
* InstructionType
* IFM Address
* IFM Size
* OFM Depth
* OFM Address
### Detailed Analysis or Content Details
The diagram shows a data flow starting from the "Data Staging / Control" blocks on the left. The data is then split into multiple parallel paths.
1. **Top Path:**
* Data flows from "Data Staging / Control" to "PoolPad".
* "PoolPad" outputs to "Write2Mem".
* "Write2Mem" writes to "FPGA SRAM (Bank 2)".
* Data flows from "FPGA SRAM (Bank 2)" to "Convolution".
* "Convolution" outputs to "Accum2".
* "Accum2" outputs to "Convolution".
* "Convolution" outputs to "PoolPad".
* "PoolPad" outputs to "Data Staging / Control".
2. **Middle Path:**
* Data flows from "Data Staging / Control" to "Convolution".
* "Convolution" outputs to "Accum0".
* "Accum0" outputs to "Convolution".
* "Convolution" outputs to "Accum1".
* "Accum1" outputs to "Convolution".
* "Convolution" outputs to "PoolPad".
* "PoolPad" outputs to "Data Staging / Control".
3. **Bottom Path:**
* Data flows from "Data Staging / Control" to "PoolPad".
* "PoolPad" outputs to "Write2Mem".
* "Write2Mem" writes to "FPGA SRAM (Bank 0)".
* Data flows from "FPGA SRAM (Bank 0)" to "Convolution".
* "Convolution" outputs to "Accum2".
* "Accum2" outputs to "Convolution".
* "Convolution" outputs to "PoolPad".
* "PoolPad" outputs to "Data Staging / Control".
4. **Right Path:**
* Data flows from "Data Staging / Control" to "PoolPad".
* "PoolPad" outputs to "Write2Mem".
* "Write2Mem" writes to "FPGA SRAM (Bank 3)".
* Data flows from "FPGA SRAM (Bank 3)" to "Convolution".
* "Convolution" outputs to "Accum3".
* "Accum3" outputs to "Convolution".
* "Convolution" outputs to "PoolPad".
* "PoolPad" outputs to "Data Staging / Control".
The "Main Controller" is connected to the "Instruction" block, which provides control signals to the entire system. The connections between the components are represented by arrows indicating the direction of data flow.
### Key Observations
* The architecture is highly parallel, with multiple convolution and accumulation units operating simultaneously.
* The use of FPGA SRAM banks suggests a distributed memory architecture.
* The "PoolPad" units are strategically placed to perform pooling and padding operations.
* The "Write2Mem" units are used to store intermediate results in the FPGA SRAM banks.
* The "Data Staging / Control" blocks manage the flow of data into and out of the system.
### Interpretation
This diagram illustrates a hardware implementation of a CNN on an FPGA. The parallel architecture is designed to accelerate the computationally intensive convolution and accumulation operations. The use of FPGA SRAM banks allows for efficient storage and retrieval of data. The "Main Controller" provides the necessary control signals to orchestrate the entire process. The diagram suggests a pipelined architecture, where data flows through the various processing units in a sequential manner. The design aims to maximize throughput and minimize latency, making it suitable for real-time applications such as image recognition and object detection. The instruction block suggests the system is programmable, allowing for flexibility in configuring the CNN architecture. The diagram does not provide specific numerical values or performance metrics, but it clearly demonstrates the overall structure and data flow of the system. The arrangement of the components suggests a focus on maximizing data reuse and minimizing memory access, which are critical for achieving high performance on an FPGA.
</details>
!"!"!
3&451+6&0%/"#$%1$1&07%#+7&,/
units are shown in blue - each is a software thread in the software implementation. Observe that there are 4 instances of 5 different compute units: 20 units (threads) in total. Edges in the figure represent multiple FIFO queues for communication of data/control between the units; queues are not shown for clarity.
The high-level behavior is as follows: The datastaging/control units receive an instruction from the ARM processor to perform convolution, padding, or max-pooling. We do not focus on fully connected layers, since it is essentially matrix multiplication and most CNN computational work comprises convolution. Although the padding and subsampling operations can be performed by a processor fairly efficiently, in most CNNs, they are tightly interleaved with convolution operations. Supporting them in hardware minimizes memory traffic between the FPGA and HPS.
For a convolution instruction, four tiles from different output feature maps (OFMs) are computed simultaneously. The four concurrently computed OFM tiles are at the same x/y location. Each data-staging/control unit loads a subset of input feature maps (IFMs) from an on-FPGA SRAM bank, as well as corresponding filter weight data from four filters. On each clock cycle weights and IFM data are injected into the convolution units. Each convolution unit performs 64 multiply operations each clock cycle, thus the entire accelerator performs 256 multiplication operations per cycle. Products from the convolution units are sent to the accumulator units (center of the figure). Each accumulator unit is responsible for maintaining the values of one tile (16 values) in an OFM. In the figure, for clarity, some edges between convolution units and accumulator units are omitted. When an OFM tile is completed, it is sent to the write-to-memory unit and written to an on-FPGA SRAM bank.
For a padding or max-pooling instruction, the datastaging/control units send IFM data and instructions to the pool/pad units, capable of performing any style of padding/max-pooling, described below. Pooled/padded tiles are then forwarded to the write-to-memory units and written to SRAM banks. Padding/pooling of four OFM tiles is done concurrently.
## B. Convolution and Zero-Weight Skipping
OFMs are computed on a tile-by-tile basis to completion without any intermediate swap-out to off-chip memory in an output-stationary manner. This style allows us to keep a fixed datapath width and not compromise accuracy by rounding partial sums. The convolution unit contains a computational sub-module that multiplies one weight per clock cycle to 16 IFM values and accumulates the resulting 16 products to the corresponding 16 OFM values in an OFM tile being computed.
Fig. 4(a) shows an OFM tile (lower right), a weight tile (upper right), and four contiguous IFM tiles. For this example, assume that the upper-left IFM tile is x/y aligned with the OFM tile being computed and that the filter dimensions are smaller than 4 × 4 (the tile size). In a given clock cycle, a weight in the weight tile is selected and multiplied by 16 IFM values. The example illustrates, with a dotted rectangle, the region of 16 IFM values with which the weight W 5 is multiplied. Observe that the intra-tile x/y offset of W 5 defines the region of IFM values which with it is multiplied. This produces 16 products: W 5 · A 5 , W 5 · A 6 . . . W 5 · D 0 . The products are accumulated to the corresponding OFM values: O 0 , O 1 , . . . , O F , respectively.
| A 0 A 4 A 8 | A 1 3 5 6 7 B | A 2 A 0 B 1 B 2 B 3 4 |
|---------------|-----------------------|-------------------------|
| | A A A B B 5 B 6 | B 7 |
| | A 9 A A A B B B A | 8 B 9 B B |
| A C | A D A E A F B B E | C B D B F |
| C 0 | C 1 C 2 C 3 D D 2 | 0 D 1 D 3 |
| C 4 | C 5 C 6 C 7 D D 6 | 4 D 5 D 7 |
| C 8 | C 9 C A C B D 8 D 9 D | A D B |
| C C | C D C E C F D D D D E | C D F |
)(&*+%$%&'(
| W 0 | W 1 | W 2 | W 3 |
|-------|-------|-------|-------|
| W 4 | W 5 | W 6 | W 7 |
| W 8 | W 9 | W A | W B |
| W C | W D | W E | W F |
,"#$./'0(-$/--12&/%(3$4&%+ /55'&2/%&16$17$)8
| O 0 | O 1 | O 2 | O 3 |
|-------|-------|-------|-------|
| O 4 | O 5 | O 6 | O 7 |
| O 8 | O 9 | O A | O B |
| O C | O D | O E | O F |
!"#$%&'(
/@$,"#A$4(&*+%$/63$!"#$%&'(-$71B$216.1'0%&16$
<details>
<summary>Image 3 Details</summary>
### Visual Description
\n
## Diagram: Data-steering and Multiply-Accumulate Hardware
### Overview
The image depicts a block diagram illustrating a data-steering and multiply-accumulate hardware architecture. It shows the flow of input data (A0 to Af) through a data steering mechanism, a multiplier, and an accumulator to produce an output (O0). The diagram is labeled "b) Data-steering and multiply-accumulate HW" at the bottom.
### Components/Axes
The diagram consists of the following components:
* **Input Data:** Represented by A0, A1, ..., Af. These are vertically stacked inputs.
* **Data Steering Block:** A rectangular block receiving the input data (A0 to Af) and an offset signal (Wi's offset in tile).
* **Multiplier:** A circle labeled "x", receiving input from the data steering block and a weight signal (Wi).
* **Accumulator:** A circle labeled "+", receiving input from the multiplier and a feedback loop from its output.
* **Output:** Represented by O0, the output of the accumulator.
* **Offset Input:** Labeled "Wi's offset in tile", connected to the data steering block.
* **Weight Input:** Labeled "Wi", connected to the multiplier.
### Detailed Analysis or Content Details
The diagram illustrates a computational flow:
1. Input data A0 through Af are fed into the data steering block.
2. The data steering block receives an offset signal "Wi's offset in tile".
3. The output of the data steering block is multiplied by a weight "Wi" in the multiplier.
4. The result of the multiplication is added to the previous output of the accumulator in the accumulator.
5. The output of the accumulator (O0) is fed back into the accumulator for the next accumulation step.
There are no numerical values or specific scales present in the diagram. The diagram is conceptual and does not provide quantitative data.
### Key Observations
The diagram highlights a common hardware architecture used in digital signal processing and machine learning for performing multiply-accumulate operations. The data steering block suggests a mechanism for selecting or routing specific input data based on the offset signal. The feedback loop in the accumulator indicates an iterative accumulation process.
### Interpretation
This diagram represents a fundamental building block in many hardware accelerators, particularly those used for neural network computations. The data steering mechanism allows for efficient data reuse and reduces memory access, while the multiply-accumulate unit performs the core computation. The feedback loop in the accumulator is essential for accumulating the weighted inputs, which is a key operation in neural network layers. The "Wi's offset in tile" suggests that the data is processed in tiles, which is a common technique for handling large datasets in hardware. The diagram demonstrates a pipelined architecture where data flows sequentially through the different stages (data steering, multiplication, and accumulation). This architecture is designed to maximize throughput and minimize latency.
</details>
Fig. 4. Data and architecture for convolution and zero-weight skipping.
Fig. 4(b) shows steering and multiply-accumulate hardware for value O 0 -the top-left value in an OFM tile being computed. Registers are omitted for clarity. When a weight W i is selected from the weight tile, the specific IFM value with which the weight W i is multiplied depends on W i 's offset within the weight tile. For the OFM value O 0 , this may be any of the IFM values A 0 , A 1 , . . . , A F , shown on the data inputs of the multiplexer, while the select inputs receive the weight's offset within the weight tile.
With the proposed tiling and convolution approach, zeroweight skipping is straightforward. For a given neural network model, the non-zero weights and their intra-tile offsets are 'packed' offline in advance in software. The packing procedure only needs to be done once for a given CNN model such as VGG-16. During inference, the accelerator receives the weight values and their intra-tile offsets in a packed format that is read directly into scratchpad memory. One non-zero weight is applied per clock cycle; no cycles are spent on weights having a value of 0.
1) Scaling Out Convolution: Each of the four datastaging/control units in Fig. 3 manages one quarter of the IFMs and corresponding weights. Every clock cycle, a datastaging/control unit injects IFM data and four weights (from four different filters) and their respective weight offsets into a convolution unit. Each of the four weights is multiplied by 16 IFM values, as described above. Thus, each convolution unit performs 4 × 16 = 64 multiplications/cycle. While the weights from a tile are being applied to (i.e. multiplied with) IFM data, the next required tiles of IFM data are simultaneously preloaded from the on-FPGA SRAM banks. Fig. 4 shows that four IFM tiles are needed to apply a weight tile and hence, since one tile/cycle can be loaded from an SRAM bank, at least four clock cycles must be spent processing a weight tile. This restriction implies that the upper-bound cycle-count reduction from zero-skipping is (16 -4) / 16 = 75% in our implementation. Moreover, note that OFMs being computed simultaneously may have different numbers of nonzero weights in their filters, causing pipeline bubbles and reduced efficiency. The completion of all four OFM tiles at a
Fig. 5. Hardware architecture of padding/pooling unit.
<details>
<summary>Image 4 Details</summary>
### Visual Description
\n
## Diagram: Max Pooling Architecture
### Overview
The image depicts a diagram of a max pooling architecture, likely within a convolutional neural network. It illustrates the flow of data from an Input Feature Map (IFM) tile through max pooling units to an Output Feature Map (OFM) tile. The diagram highlights the selection process within the max pooling units and the subsequent multiplexing to generate the output.
### Components/Axes
The diagram consists of the following key components:
* **IFM Tile:** Labeled "IFM tile", this represents the input feature map. It contains elements labeled A₀ through A₉, arranged in a 3x4 grid.
* **MAX Units:** Two "MAX" blocks are shown, representing the max pooling operation. Each MAX unit receives multiple inputs (A₀-A₉).
* **Max₀Sel & Max₁Sel:** Selection signals for the first and second MAX units, respectively.
* **Max₀, Max₁, Max₂, Max₃:** Outputs from the first MAX unit.
* **Max₀, Max₁, Max₂, Max₃:** Outputs from the second MAX unit.
* **O₀Sel & O₁Sel:** Selection signals for the output multiplexers.
* **O₀ & Oғ:** Outputs from the output multiplexers.
* **OFM Tile:** Labeled "OFM tile", this represents the output feature map. It contains elements labeled O₀ through O₉, arranged in a 3x4 grid.
* **Annotation:** "4 MAX units" and "16 4-to-1 MUXes for select" are present, providing information about the number of units used.
* **Arrows:** Curved arrows indicate the data flow between the IFM tile, MAX units, multiplexers, and OFM tile.
### Detailed Analysis or Content Details
The diagram shows a 2x2 max pooling operation repeated across the IFM tile.
* **IFM Tile:** The input tile contains 12 elements (A₀ to A₉).
* **MAX Units:** Each MAX unit receives four inputs (A₀-A₃ for the first, A₄-A₉ for the second). The MAX unit selects the maximum value from these four inputs.
* **Multiplexers:** The outputs of the MAX units (Max₀-Max₃) are fed into multiplexers. The multiplexers select one of the four maximum values based on the selection signal (O₀Sel or O₁Sel).
* **OFM Tile:** The output tile contains 12 elements (O₀ to O₉). Each output element is the result of a max pooling operation on a 2x2 region of the input tile.
* **Data Flow:** The curved arrows indicate that the output of the multiplexers is written to the corresponding location in the OFM tile.
### Key Observations
* The diagram illustrates a simplified max pooling architecture.
* The use of multiplexers allows for flexible selection of the maximum value.
* The diagram suggests a parallel processing scheme, with multiple MAX units operating simultaneously.
* The annotation indicates that 16 4-to-1 multiplexers are used for selection.
### Interpretation
The diagram demonstrates the core principle of max pooling: reducing the spatial dimensions of the input feature map while retaining the most important features (represented by the maximum values). This process helps to reduce computational complexity and improve the robustness of the model to variations in the input. The multiplexers provide a mechanism for selecting the appropriate maximum value based on the desired pooling strategy. The diagram highlights the parallel nature of the max pooling operation, which is crucial for efficient processing of large feature maps. The annotation about the number of MAX units and multiplexers provides insight into the hardware implementation of the max pooling layer. The diagram is a conceptual representation and does not specify the exact details of the implementation, such as the size of the input and output feature maps or the specific selection criteria for the multiplexers.
</details>
given x/y tile position is synchronized using a Pthreads barrier.
## C. Padding and Pooling
Fig. 5 shows the padding/max-pooling unit. The controller/data-staging unit injects an IFM tile (shown on the left), as well as an instruction that specifies the desired padding/max-pooling behavior. There are four MAX units that, based on the instruction, select the maximum of any of the 16 IFM values in the input tile. The MAX units feed 16 multiplexers: one for each value of the OFM tile being computed. Based on the instruction, each value in the OFM tile may be updated with one of the MAX unit outputs, or alternately, may retain its old value.
To implement padding, which does not involve taking the maximum of multiple IFM values, the MAX units return a single value from the IFM (i.e. find the maximum among a single value). The specific number of MAX units (four in this case), is inspired by the needs of VGG-16, which requires 2 × 2 max-pooling regions with a stride of 2. However, with just a few instructions, the padding/max-pooling unit is capable of realizing any padding/max-pooling layer (e.g. a variety of max-pooling region sizes or strides). Moreover, since all units are described in software, it is straightforward to increase the number of MAX functional units within the padding/maxpooling unit.
## IV. IMPLEMENTATION
## A. High-Level Synthesis
Use of the LegUp HLS Pthreads flow permitted accelerator design, development and debugging to proceed in software. Debug and test were simplified, as the parallel software execution aligns closely with the target hardware architecture.
The primary HLS constraints applied were loop pipelining, if-conversion, automated bitwidth minimization [10], and clock-period constraints. To achieve optimized loop pipelines (initiation interval [II] = 1), it was necessary to remove control flow from the C code to the extent possible by making use of the ternary operator ( <cond> ? <val1> : <val2> ) to implement MUX'ing instead of using conditionals. The C specification of the accelerator is ∼ 5,600 LOC. The Verilog RTL automatically produced by LegUp was modified (using a script) in three ways: 1) pragmas were added to direct the Intel synthesis tools to implement the FIFO queues using LUT RAM instead of block RAM (saving precious block-RAM resources); 2) the on-FPGA SRAM banks were brought to the top-level of the hierarchy, making them accessible to the DMA
unit for writing/reading; and 3) the port assignment for the onFPGA RAM banks was altered so that reads and writes have exclusive ports, reducing contention/arbitration. Our design goal was skewed towards maximizing performance; however, it is possible to apply directives in LegUp that lower resource utilization at the expense of some performance.
A challenge with HLS is the disconnect between the software source code and the generated hardware -it can be difficult for one to know how to change the source code to effect desired changes in the hardware. In this work, this issue arose in the context of the controller/data-staging unit, which synthesized to a module with a large number of FSM states (hundreds), and consequent high-fanout signals (e.g. the FSM stall logic). To mitigate this, we split the unit into two C functions, one for convolution instructions and one for padding/max-pooling instructions, each having a simpler FSM than the original monolithic controller.
With reference to Fig. 3, the pool/pad, convolution, writeto-memory, and accumulator units are all streaming kernels generated by LegUp HLS that can accept new inputs every clock cycle (II = 1). The data-staging/control unit is capable of injecting data into the compute units every cycle. We reinforce that the entire accelerator, including the compute pipelines and the relatively complex control as described in Section III-A, is synthesized to Verilog RTL from parallel C software. Manual RTL design was used solely for the DMA unit. While streaming audio/video applications are typically emblematic as being 'ideal' for the use of HLS, here, we show that HLS can effectively be used to generate sophisticated control and data-staging for streaming hardware.
## B. VGG-16 Reduced Precision and Pruning
Beginning with the pre-trained VGG-16 model [3], we increased the sparsity by pruning and reduced the precision to 8-bit magnitude-plus-sign representation by scaling. Pruning and precision reduction were done using Caffe, in a manner similar to [9]. We consider two VGG-16 models: 1) with reduced precision and 2) with reduced precision and pruning. With variant #2, inference accuracy in validation was within 2% of the original unpruned floating point, which can be improved further through training.
## C. Software
Software executing on the on-chip ARM processor handles the loading and pre-processing of network weights, biases and test images. Pre-processing includes the reordering of data into tiled format for our accelerator. The framework sends the instruction and calls the hardware driver for inference.
## D. System Integration, Accelerator Scale-Out
The accelerator core, DMA controller, and host processor communicate via an interconnect network synthesized using Intel's Qsys System Integration tool. Two separate systems are instantiated: System I is a high bandwidth 256-bit bus that performs DMA to and from system DRAM to the accelerator banks. System II is a set of Avalon Memory-Mapped (AMM) interfaces between the host ARM processor and control and status registers on the accelerator core and DMA unit. The accelerator and DMA unit are controlled using System II by the host ARM processor.
We target a mid-range-sized Intel Arria 10 SX660 FPGA, whose size permits us to instantiate two instances of the
Fig. 6. ALM usage by each unit in the accelerator.
<details>
<summary>Image 5 Details</summary>
### Visual Description
\n
## Pie Chart: Hardware Unit Distribution
### Overview
This image presents a pie chart illustrating the distribution of hardware units within a system. The chart is divided into nine segments, each representing a different unit type and its corresponding percentage of the total. A legend is positioned to the right of the chart, providing color-coded labels for each unit.
### Components/Axes
The chart lacks traditional axes, as it's a pie chart representing proportions of a whole. The key components are:
* **Pie Segments:** Representing different hardware units.
* **Percentages:** Displayed next to each segment, indicating the proportion of the whole.
* **Legend:** Located on the right side, mapping colors to unit names.
The legend contains the following labels:
* Accumulator Unit (Blue)
* Convolution Unit (Orange)
* Pad/Pool Unit (Gray)
* Write-To-Memory Unit (Yellow)
* Data-Staging/Control Unit (Dark Blue)
* Banks (Green)
* FIFO (Light Blue)
* DMA (Red)
### Detailed Analysis
The pie chart segments and their approximate percentages are as follows:
* **Accumulator Unit:** 32% (Dark Blue) - Largest segment.
* **Convolution Unit:** 16% (Orange) - Second largest segment.
* **Pad/Pool Unit:** 11% (Gray)
* **Data-Staging/Control Unit:** 20% (Dark Blue)
* **Banks:** 5% (Green)
* **FIFO:** 1% (Light Blue)
* **DMA:** 1% (Red)
* **Write-To-Memory Unit:** 14% (Yellow)
The percentages sum to 100%.
### Key Observations
* The Accumulator Unit constitutes the largest portion (32%) of the hardware units.
* The Convolution Unit is the second most significant component, accounting for 16%.
* FIFO and DMA each represent a very small fraction (1%) of the total.
* The Data-Staging/Control Unit and Pad/Pool Unit are significant components, representing 20% and 11% respectively.
### Interpretation
The data suggests a system heavily reliant on Accumulator Units and Convolution Units. This could indicate a system designed for intensive computation, potentially related to machine learning or signal processing, where accumulation and convolution operations are fundamental. The relatively small percentages of FIFO and DMA suggest that data transfer and buffering are not the primary bottlenecks in this system. The presence of Pad/Pool Units suggests image processing or similar operations where downsampling or feature extraction are important. The distribution highlights the architectural priorities of the system, emphasizing processing capabilities over data handling. The two dark blue segments, Accumulator Unit and Data-Staging/Control Unit, may indicate a tightly coupled control and processing architecture.
</details>
accelerator shown in Fig. 3, where each instance operates concurrently on separate stripes of FMs (see Fig. 2). The overall multi-accelerator system is capable of 512 MACs/cycle.
## V. EXPERIMENTAL STUDY
A unique advantage of HLS is that one can synthesize multiple architecture variants from software and constraint changes alone. We analyze area, power and performance for four architecture variants running the two VGG-16 CNN models mentioned above: reduced precision without and with pruning (higher fraction of zero weights). The four architecture variants considered are as follows (labels in brackets):
- 1) A non-optimized simplified accelerator variant with a single convolution sub-module capable of performing at most 16 MACs/cycle (16-unopt).
- 2) A non-performance-optimized variant with one instance of the accelerator in Fig. 3 capable of performing at most 256 MACs/cycle (256-unopt).
- 3) A performance-optimized variant of Fig. 3 (256-opt).
- 4) A variant with two instances of the accelerator in Fig. 3 capable of performing at most 512 MACs/cycle (512-opt).
The 16-unopt architecture computes a single OFM tile at a time, and consequently requires no synchronization among multiple control/data-staging units. Analysis of the 16-unopt architectures gives insight into the HLS hardware quality in the absence of synchronization overhead. Both the 16unopt and the 256-unopt architectures were not performanceoptimized and as such, they consume minimal area and, to verify functional correctness, were clocked at 55MHz. To produce higher-performance variants, we tightened the clockperiod constraint supplied to the LegUp HLS tool, and also invoked performance optimizations in the Intel RTL-synthesis tool: retiming, physical synthesis, higher place/route effort. The 256-opt and 512-opt architectures were clocked at 150 MHz and 120 MHz, respectively. Routing of the 512-opt architecture failed at higher performance targets due to high congestion.
Intel FPGAs have 3 main types of resources: Adaptive Logic Modules (ALMs - lookup-table-based logic), DSP and RAM blocks. Our 256-opt accelerator uses 44% of the ALM logic, 25% of the DSP and 49% of the RAM blocks. Fig. 6 shows the breakdown of ALM usage for each module. The convolution, accumulator and data-staging/control modules take up most of the area, due to the heavy MUX'ing required in these units. Most of the DSP blocks are used in the convolution and accumulator modules. We adjust the RAM block usage to maximize our bank size given the number of available RAMs.
We consider the efficiency of the HLS-generated hardware by comparing the experimentally observed throughput
Fig. 7. Efficiency of each accelerator variant for VGG-16 inference.
<details>
<summary>Image 6 Details</summary>
### Visual Description
## Bar Chart: Effective Efficiency Comparison
### Overview
This bar chart compares the "Effective Efficiency (%)" of different configurations, labeled as "16-unopt", "16-opt-pr", "256-unopt", "256-unopt-pr", "256-opt", "256-opt-pr", "512-opt", and "512-opt-pr". For each configuration, three bars represent the "Best", "Mean", and "Worst" efficiency values. A horizontal dashed line indicates the "Ideal" efficiency level of 100%.
### Components/Axes
* **X-axis:** Configuration names: "16-unopt", "16-opt-pr", "256-unopt", "256-unopt-pr", "256-opt", "256-opt-pr", "512-opt", "512-opt-pr".
* **Y-axis:** "Effective Efficiency (%)", ranging from 0% to 180%.
* **Legend:**
* Dark Blue: "Best"
* Medium Blue: "Mean"
* Light Red: "Worst"
* **Ideal Line:** A horizontal dashed black line at 100% representing the ideal efficiency.
### Detailed Analysis
The chart consists of eight groups of three bars, one for each configuration. I will analyze each configuration individually, noting the approximate values for each bar.
* **16-unopt:**
* Best: ~155%
* Mean: ~95%
* Worst: ~75%
* **16-opt-pr:**
* Best: ~160%
* Mean: ~130%
* Worst: ~90%
* **256-unopt:**
* Best: ~130%
* Mean: ~80%
* Worst: ~70%
* **256-unopt-pr:**
* Best: ~135%
* Mean: ~100%
* Worst: ~80%
* **256-opt:**
* Best: ~140%
* Mean: ~105%
* Worst: ~85%
* **256-opt-pr:**
* Best: ~130%
* Mean: ~115%
* Worst: ~90%
* **512-opt:**
* Best: ~145%
* Mean: ~120%
* Worst: ~90%
* **512-opt-pr:**
* Best: ~145%
* Mean: ~110%
* Worst: ~80%
### Key Observations
* The "Best" efficiency consistently exceeds 100% for all configurations, indicating potential for optimization beyond the ideal.
* The "Worst" efficiency is generally lower than the "Mean" efficiency, showing variability in performance.
* Configurations with "opt-pr" generally have higher "Mean" and "Best" efficiencies than their "unopt" counterparts.
* The "16-opt-pr" configuration exhibits the highest "Best" efficiency, while "256-unopt" has the lowest.
* The "Mean" efficiency values tend to cluster around the 100% mark, with some configurations exceeding it.
### Interpretation
The data suggests that optimization techniques ("opt" and "opt-pr") generally improve the effective efficiency of the configurations. The "opt-pr" configurations consistently outperform their "unopt" counterparts, indicating the benefit of the "pr" optimization. The significant difference between "Best" and "Worst" efficiency highlights the importance of considering performance variability. The fact that "Best" efficiency exceeds 100% suggests that the configurations can be optimized to perform better than the defined "Ideal" level. The configurations with larger sizes (512) show a trend of higher "Best" efficiency, but also a wider range between "Best" and "Worst" performance. This could indicate that larger configurations offer more potential for optimization but also require more careful tuning to avoid performance degradation. The chart provides a comparative analysis of different configurations, allowing for informed decisions about which configurations to prioritize based on desired efficiency levels and performance consistency.
</details>
Fig. 8. Absolute GOPS/s across accelerator variants for VGG-16.
<details>
<summary>Image 7 Details</summary>
### Visual Description
\n
## Bar Chart: Effective GOPS vs. Configuration
### Overview
This bar chart compares the Average and Peak GOPS (Giga Operations Per Second) for different configurations. The x-axis represents the configuration name, and the y-axis represents the Effective GOPS. Each configuration has two bars: one for Average GOPS (light blue) and one for Peak GOPS (dark blue). Values are displayed above each bar.
### Components/Axes
* **X-axis:** Configuration Name. Categories are: 16-unopt, 16-opt-pr, 256-unopt, 256-unopt-pr, 256-opt, 256-opt-pr, 512-opt, 512-opt-pr.
* **Y-axis:** Effective GOPS. Scale ranges from 0 to 140, with increments of 20.
* **Legend:** Located in the top-left corner.
* Light Blue: Average GOPS
* Dark Blue: Peak GOPS
### Detailed Analysis
The chart presents paired bars for each configuration, showing both average and peak GOPS.
* **16-unopt:** Average GOPS = 1, Peak GOPS = 1
* **16-opt-pr:** Average GOPS = 1, Peak GOPS = 2
* **256-unopt:** Average GOPS = 10, Peak GOPS = 14
* **256-unopt-pr:** Average GOPS = 14, Peak GOPS = 14
* **256-opt:** Average GOPS = 25, Peak GOPS = 32
* **256-opt-pr:** Average GOPS = 38, Peak GOPS = 33
* **512-opt:** Average GOPS = 40, Peak GOPS = 61
* **512-opt-pr:** Average GOPS = 53, Peak GOPS = 138
**Trends:**
* Generally, Peak GOPS is higher than Average GOPS for each configuration.
* As the configuration complexity increases (moving from 16 to 512), both Average and Peak GOPS tend to increase.
* The difference between Average and Peak GOPS is most pronounced for the 512-opt-pr configuration.
* For 16-unopt and 16-opt-pr, the Average and Peak GOPS are nearly identical.
### Key Observations
* The 512-opt-pr configuration exhibits the highest Peak GOPS (138), significantly exceeding all other configurations.
* The 16-unopt and 16-opt-pr configurations have the lowest GOPS values.
* The "opt-pr" configurations generally show higher average GOPS than their "unopt" counterparts, but the peak GOPS can vary.
* There is a slight dip in Peak GOPS from 256-opt (32) to 256-opt-pr (33).
### Interpretation
The data suggests that increasing the configuration size (from 16 to 512) and applying optimizations ("opt-pr") generally leads to higher performance, as measured by GOPS. The large difference between Average and Peak GOPS for the 512-opt-pr configuration indicates that this configuration is capable of bursts of high performance, but its sustained performance (average) is lower. The relatively small difference between average and peak for the 16 configurations suggests a more consistent performance profile. The slight dip in peak GOPS from 256-opt to 256-opt-pr could indicate a trade-off introduced by the optimization process, or simply be a statistical fluctuation. The chart demonstrates the impact of configuration choices on computational performance, highlighting the benefits of larger configurations and optimizations.
</details>
( ops/elapsed time ) with the theoretically minimum ideal throughput numbers. Ideal throughput is defined as peak throughput * total number of computations . We add an overhead ( ∼ 15% but varies by layer) for the increased number of MAC operation due to limited on-FPGA SRAM bank size 'striping'. Results in Fig. 7 illustrate the efficiency of various architectures with the pruned and unpruned VGG-16 model. Results obtained using a pruned network are labeled as '-pr'. 'Best' and 'worst' refer to the highest and lowest throughput for any single convolutional layer of VGG-16, respectively. Mean refers to the average throughput across all VGG-16 layers. The ideal throughput value is indicated as a dotted line on Fig. 7.
The underlying reason for differences between best, worst, and mean is that, for deeper layers of VGG-16, the ratio of weight data to FM data increases, imposing a higher overhead for unpacking weights and offsets in our accelerator, reducing effective throughput. Using the pruned network we see greater than 100% efficiency, due to the zero-skipping avoiding some multiply-accumulates altogether. For the non-pruned VGG-16, we are not far from the ideal throughput -usually within ∼ 10%. This analysis shows the HLS-generated hardware is quite efficient; overhead from HLS is minimal from the cycle latency perspective.
Secondly, we look at our results in terms of absolute performance (GOPS). Fig. 8 shows that our 512-opt accelerator achieved the highest average (peak) throughput of 39.5 GOPS (61 GOPS), and with the pruned network, the effective average (peak) performance increases to 53.3 GOPS (138 GOPS). We can clearly see the effects of zero-skipping in these results. By pruning the network, we were able to increase our performance by ∼ 1.3 × , on average, and ∼ 2.2 × in the
TABLE I. POWER CONSUMPTION
| Accelerator Variant | Peak Power (mW) | GOPS/W | GOPS/W (peak) |
|-----------------------|-------------------|----------|-----------------|
| 256-opt (FPGA) | 2300 (500) | 13.4 | 37.4 |
| 512-opt (FPGA) | 3300 (800) | 13.9 | 41.8 |
| 256-opt (Board) | 9500 | 3.5 | 9.05 |
| 512-opt (Board) | 10800 | 5.6 | 12.7 |
*dynamic power is parenthesized peak case. The peak performance numbers are considerably higher than the average in the pruned case, as peak throughput requires uniformly sparse filters applied concurrently for even workload balancing. Future work could include grouping filters in advance according to similarity in non-zero-entry counts to maximize available zero skipping and balance the work.
While the absolute performance numbers are relatively modest and in line with prior work (cf. Section VI), the results in Fig. 8 underscore a key advantage of HLS, namely, a wide range of architectures with distinct performance/area trade-offs can be produced by software and HLS constraint changes alone. For example, in the opt vs. unopt variants, the clock-period constraint applied in HLS impacts the degree of pipelining in the compute units and control. It would be expensive and time-consuming to produce hand-written RTL for all architecture variants considered. Finally, we note that on a larger Arria 10 FPGA family member (e.g. GT1150), with nearly double the capacity, software changes alone would allow us to scale out the design further.
Power consumption measurements are given in Table I. All measurements are peak power measured while running the accelerator on the worst-case VGG-16 layer. A board -level power measurement is provided, alongside the power consumption of the FPGA by itself. Power numbers in parentheses are dynamic power; numbers outside parentheses are static + dynamic power.
## VI. RELATED WORK
Recent years have seen considerable research on custom hardware accelerators for deep CNN inference designed in manual RTL, realized in ASICs (e.g. [11], [12]) and FPGAs (e.g. [13]). The works most comparable to ours use HLS and target FPGAs. [14] applied Xilinx's Vivado HLS to synthesize an accelerator for the convolution operation, achieving 61 GFLOPS peak performance. Suda et al. [15] synthesized a CNN accelerator from OpenCL, achieving peak performance of 136 GOPS for convolution. In [16], Intel synthesized a CNN accelerator from OpenCL that incorporates the Winograd transform, half-precision floating point, and achieves over 1 TFLOPS performance. In [17], Xilinx synthesized significant parts of a binarized CNN accelerator with Vivado HLS. In binarized CNNs, both weights and activations are represented by 1-bit values. OpenCL along with Xilinx's SDAccel OpenCL HLS tool was also used in [18] to synthesize a single-precision floating-point CNN accelerator incorporating Winograd and achieving 50 GFLOPS performance. The latter work also provides integration with the Caffe framework.
Generally, the OpenCL implementations above use PCIe connectivity with a host processor and are more suited for data center applications; whereas, our system is intended for embedded. To the best of the authors' knowledge, our work is the first to synthesize a CNN accelerator from parallelized C -language software that incorporates a novel zero-skipping approach and reduced precision, and illustrates how, beginning with a software-parallelized C specification of the architecture, constraints to the HLS and RTL synthesis tools can be applied to generate a variety of accelerator architectures with different performance/area trade-offs.
## VII. CONCLUSIONS AND FUTURE WORK
An FPGA-based CNN inference accelerator was synthesized from parallel C software using the LegUp HLS tool, incorporating a zero-weight skipping architecture and reducedprecision arithmetic. The end-to-end system contains a dualcore ARM processor, the accelerator, on-FPGA memories (backed by DDR4) and DMA. Software running on the ARM issues instructions to the accelerator for convolution, padding and pooling. Complex datapath and control logic was synthesized entirely from C , and the use of HLS permitted a range of architectures to be evaluated, through software and constraint changes alone. Future work involves the use of HLS to synthesize accelerators for other neural network styles, including binarized, ternary and recurrent networks.
## REFERENCES
- [1] A. Canis, J. Choi, M. Aldham, V. Zhang, A. Kammoona, T. Czajkowski, S. D. Brown, , and J. H. anderson, 'LegUp: An open source high-level synthesis tool for FPGA-based processor/accelerator systems,' ACM Tranasactions on Embedded Computing Systems , 2013.
- [2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, 'ImageNet classification with deep convolutional neural networks,' in Neural Information Processing Systems , 2012, pp. 1106-1114.
- [3] K. Simonyan and A. Zisserman, 'Very deep convolutional networks for large-scale image recognition,' CoRR , vol. abs/1409.1556, 2014.
- [4] C. Szegedy and et al., 'Going deeper with convolutions,' in IEEE CVPR , 2015, pp. 1-9.
- [5] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, 'Binarized neural networks,' in NIPS , 2016, pp. 4107-4115.
- [6] G. Venkatesh, E. Nurvitadhi, and D. Marr, 'Accelerating deep convolutional networks using low-precision and sparsity,' CoRR , vol. abs/1610.00324, 2016. [Online]. Available: http://arxiv.org/abs/1610. 00324
- [7] J. Cong, B. Liu, S. Neuendorffer, J. Noguera, K. A. Vissers, and Z. Zhang, 'High-level synthesis for FPGAs: From prototyping to deployment,' IEEE TCAD , vol. 30, no. 4, pp. 473-491, 2011.
- [8] J. Choi, R. Lian, S. D. Brown, and J. H. Anderson, 'A unified software approach to specify pipeline and spatial parallelism in FPGA hardware,' in IEEE ASAP , 2016, pp. 75-82.
- [9] S. Han, H. Mao, and W. J. Dally, 'Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding,' CoRR , vol. abs/1510.00149, 2015.
- [10] M. Gort and J. H. Anderson, 'Range and bitmask analysis for hardware optimization in high-level synthesis,' in IEEE/ACM ASP-DAC , 2013, pp. 773-779.
- [11] Y. Chen, T. Krishna, J. S. Emer, and V. Sze, 'Eyeriss: An energyefficient reconfigurable accelerator for deep convolutional neural networks,' J. Solid-State Circuits , vol. 52, no. 1, pp. 127-138, 2017.
- [12] T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, and O. Temam, 'DianNao: A small-footprint high-throughput accelerator for ubiquitous machine-learning,' in ACM ASPLOS , 2014, pp. 269-284.
- [13] E. Nurvitadhi and et al., 'Can FPGAs beat GPUs in accelerating nextgeneration deep neural networks?' in ACM FPGA , 2017, pp. 5-14.
- [14] C. Zhang, P. Li, G. Sun, Y. Guan, B. Xiao, and J. Cong, 'Optimizing FPGA-based accelerator design for deep convolutional neural networks,' in ACM FPGA , 2015, pp. 161-170.
- [15] N. Suda, V. Chandra, G. Dasika, A. Mohanty, Y. Ma, S. Vrudhula, J.-s. Seo, and Y. Cao, 'Throughput-optimized OpenCL-based FPGA accelerator for large-scale convolutional neural networks,' in ACM FPGA , 2016.
- [16] U. Aydonat, S. O'Connell, D. Capalija, A. C. Ling, and G. R. Chiu, 'An OpenCL Deep Learning Accelerator on Arria 10,' ArXiv e-prints , Jan. 2017.
- [17] Y. Umuroglu, N. J. Fraser, G. Gambardella, M. Blott, P. Leong, M. Jahre, and K. Vissers, 'FINN: A Framework for Fast, Scalable Binarized Neural Network Inference,' ArXiv e-prints , 2016.
- [18] R. DiCecco, G. Lacey, J. Vasiljevic, P. Chow, G. W. Taylor, and S. Areibi, 'Caffeinated FPGAs: FPGA framework for convolutional neural networks,' CoRR , vol. abs/1609.09671, 2016.