1708.02002v1

# ArXiv Paper 1708.02002v1 ## Focal Loss for Dense Object Detection Tsung-Yi Lin Priya Goyal Ross Girshick Kaiming He Piotr Doll´ ar Facebook AI Research (FAIR) Figure 1. We propose a novel loss we term the Focal Loss that adds a factor (1 -p t ) γ to the standard cross entropy criterion. Setting γ > 0 reduces the relative loss for well-classified examples ( p t > . 5 ), putting more focus on hard, misclassified examples. As our experiments will demonstrate, the proposed focal loss enables training highly accurate dense object detectors in the presence of vast numbers of easy background examples. <details> <summary>Image 1 Details</summary>  ### Visual Description ## Line Graph: Cross-Entropy vs. Focal Loss with Varying Gamma Values ### Overview The graph compares two loss functions—Cross-Entropy (CE) and Focal Loss (FL)—across different gamma (γ) values. The x-axis represents the "probability of ground truth class" (p_t), ranging from 0 to 1, while the y-axis shows the loss magnitude (0 to 5). Five curves are plotted, each corresponding to a γ value (0, 0.5, 1, 2, 5), with annotations for well-classified examples (p_t > 0.8). ### Components/Axes - **X-axis**: "probability of ground truth class" (0 to 1, linear scale). - **Y-axis**: "loss" (0 to 5, linear scale). - **Legend**: Located in the top-right corner, mapping γ values to colors: - γ = 0 (blue) - γ = 0.5 (orange) - γ = 1 (yellow) - γ = 2 (purple) - γ = 5 (green) - **Equations**: - CE(p_t) = -log(p_t) - FL(p_t) = -(1 - p_t)^γ log(p_t) - **Annotation**: A shaded region labeled "well-classified examples" spans p_t = 0.8 to 1.0. ### Detailed Analysis 1. **Cross-Entropy (CE) Loss**: - Represented by the blue line (γ = 0). - Equation: CE(p_t) = -log(p_t). - Behavior: Steeply decreasing as p_t increases, with a sharp drop near p_t = 0. 2. **Focal Loss (FL) Curves**: - **γ = 0.5 (orange)**: Slightly flatter than CE near p_t = 1, but steeper near p_t = 0. - **γ = 1 (yellow)**: Balanced trade-off; less steep than γ = 0.5 near p_t = 1. - **γ = 2 (purple)**: Flatter near p_t = 1, emphasizing well-classified examples. - **γ = 5 (green)**: Nearly flat near p_t = 1, minimizing loss for high-confidence predictions. 3. **Key Trends**: - All FL curves converge to CE at p_t = 1 (loss = 0). - Higher γ values reduce the loss gradient for well-classified examples (p_t > 0.8). - At p_t = 0.5, FL loss decreases with increasing γ (e.g., γ = 5 has ~0.3 loss vs. γ = 0’s ~0.7). ### Key Observations - **Well-Classified Examples**: The shaded region (p_t > 0.8) shows minimal loss across all γ values, with γ = 5 having the lowest loss. - **Misclassified Examples**: At p_t = 0.2, γ = 5 produces the highest loss (~4.5), while γ = 0 matches CE (~1.6). - **Gamma Sensitivity**: Higher γ values amplify the penalty for low-p_t predictions, prioritizing hard examples during training. ### Interpretation The graph demonstrates how Focal Loss modulates the impact of misclassified vs. well-classified examples via γ. Lower γ values (e.g., 0.5) retain CE-like behavior, while higher γ (e.g., 5) suppresses loss for confident predictions, focusing training on uncertain samples. This trade-off is critical for imbalanced datasets, where overemphasizing easy examples can degrade model performance. The convergence of FL curves at p_t = 1 underscores its design to address class imbalance by down-weighting easy examples. </details> ## Abstract The highest accuracy object detectors to date are based on a two-stage approach popularized by R-CNN, where a classifier is applied to a sparse set of candidate object locations. In contrast, one-stage detectors that are applied over a regular, dense sampling of possible object locations have the potential to be faster and simpler, but have trailed the accuracy of two-stage detectors thus far. In this paper, we investigate why this is the case. We discover that the extreme foreground-background class imbalance encountered during training of dense detectors is the central cause. We propose to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples. Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training. To evaluate the effectiveness of our loss, we design and train a simple dense detector we call RetinaNet. Our results show that when trained with the focal loss, RetinaNet is able to match the speed of previous one-stage detectors while surpassing the accuracy of all existing state-of-the-art two-stage detectors. Figure 2. Speed (ms) versus accuracy (AP) on COCO test-dev . Enabled by the focal loss, our simple one-stage RetinaNet detector outperforms all previous one-stage and two-stage detectors, including the best reported Faster R-CNN [27] system from [19]. We show variants of RetinaNet with ResNet-50-FPN (blue circles) and ResNet-101-FPN (orange diamonds) at five scales (400-800 pixels). Ignoring the low-accuracy regime (AP < 25), RetinaNet forms an upper envelope of all current detectors, and a variant trained for longer (not shown) achieves 39.1 AP. Details are given in § 5. <details> <summary>Image 2 Details</summary>  ### Visual Description ## Line Chart: COCO AP vs Inference Time for Object Detection Models ### Overview The chart compares the performance (COCO Average Precision, AP) of various object detection models against their inference time (in milliseconds). Two primary data series are plotted: RetinaNet-50 (blue circles) and RetinaNet-101 (orange diamonds). A table on the right lists additional models with their AP and time values. ### Components/Axes - **X-axis**: Inference time (ms), ranging from 50 to 250 ms in 50 ms increments. - **Y-axis**: COCO AP, ranging from 28 to 38 in 2-point increments. - **Legend**: Located in the top-right corner, with: - Blue circles: RetinaNet-50 - Orange diamonds: RetinaNet-101 - **Table**: Right-aligned, listing models with AP and time values (see Content Details). ### Detailed Analysis 1. **RetinaNet-50 (Blue Line)**: - Starts at ~32.5 AP at 73 ms (point D). - Increases to ~34.4 AP at 90 ms (point F). - Shows a steady upward trend with minimal time increase. 2. **RetinaNet-101 (Orange Line)**: - Begins at ~34.4 AP at 90 ms (point F). - Rises to ~37.8 AP at 198 ms (point G). - Demonstrates a steeper slope, indicating faster AP gains with increased time. 3. **Table Data**: The following table lists the models (A to G) with their AP and inference time: | Model | AP | Time (ms) | Note | |-------|----|-----------|------| | YOLOv2 | 21.6 | 25 | † | | SSD321 | 28.0 | 61 | | | DSSD321 | 28.0 | 85 | | | R-FCN | 29.9 | 85 | ‡ | | SSD513 | 31.2 | 125 | | | DSSD513 | 33.2 | 156 | | | FPN FRCN | 36.2 | 172 | | **RetinaNet Variants**: - RetinaNet-50-500: 32.5 AP, 73 ms - RetinaNet-101-500: 34.4 AP, 90 ms - RetinaNet-101-800: 37.8 AP, 198 ms **Footnotes**: - †: Not plotted - ‡: Extrapolated time ### Key Observations - **Performance-Time Tradeoff**: RetinaNet-101 achieves higher AP but requires significantly more inference time than RetinaNet-50. - **Extrapolation**: R-FCN's time (85 ms) is marked as extrapolated, suggesting uncertainty in its placement. - **Model Efficiency**: RetinaNet-50-500 matches RetinaNet-101-500 in AP (34.4) but with 17 ms faster inference. - **Outlier**: YOLOv2 (A) has the lowest AP (21.6) but fastest time (25 ms), indicating poor accuracy for speed. ### Interpretation The chart illustrates a clear tradeoff between accuracy and computational efficiency in object detection models. RetinaNet-101 variants dominate in AP but demand higher inference times, making them suitable for applications prioritizing precision (e.g., medical imaging). RetinaNet-50 offers a balance, while models like YOLOv2 prioritize speed at the cost of accuracy. The extrapolated time for R-FCN raises questions about its real-world applicability. The table's ordering suggests a correlation between AP and inference time, though exceptions exist (e.g., RetinaNet-50-500 vs. RetinaNet-101-500). This data is critical for optimizing model selection in resource-constrained environments. </details> ## 1. Introduction Current state-of-the-art object detectors are based on a two-stage, proposal-driven mechanism. As popularized in the R-CNN framework [11], the first stage generates a sparse set of candidate object locations and the second stage classifies each candidate location as one of the foreground classes or as background using a convolutional neural network. Through a sequence of advances [10, 27, 19, 13], this two-stage framework consistently achieves top accuracy on the challenging COCO benchmark [20]. Despite the success of two-stage detectors, a natural question to ask is: could a simple one-stage detector achieve similar accuracy? One stage detectors are applied over a regular, dense sampling of object locations, scales, and aspect ratios. Recent work on one-stage detectors, such as YOLO [25, 26] and SSD [21, 9], demonstrates promising results, yielding faster detectors with accuracy within 1040% relative to state-of-the-art two-stage methods. This paper pushes the envelop further: we present a onestage object detector that, for the first time, matches the state-of-the-art COCO AP of more complex two-stage detectors, such as the Feature Pyramid Network (FPN) [19] or Mask R-CNN [13] variants of Faster R-CNN [27]. To achieve this result, we identify class imbalance during training as the main obstacle impeding one-stage detector from achieving state-of-the-art accuracy and propose a new loss function that eliminates this barrier. Class imbalance is addressed in R-CNN-like detectors by a two-stage cascade and sampling heuristics. The proposal stage ( e.g ., Selective Search [34], EdgeBoxes [37], DeepMask [23, 24], RPN [27]) rapidly narrows down the number of candidate object locations to a small number ( e.g ., 1-2k), filtering out most background samples. In the second classification stage, sampling heuristics, such as a fixed foreground-to-background ratio (1:3), or online hard example mining (OHEM) [30], are performed to maintain a manageable balance between foreground and background. In contrast, a one-stage detector must process a much larger set of candidate object locations regularly sampled across an image. In practice this often amounts to enumerating ∼ 100k locations that densely cover spatial positions, scales, and aspect ratios. While similar sampling heuristics may also be applied, they are inefficient as the training procedure is still dominated by easily classified background examples. This inefficiency is a classic problem in object detection that is typically addressed via techniques such as bootstrapping [32, 28] or hard example mining [36, 8, 30]. In this paper, we propose a new loss function that acts as a more effective alternative to previous approaches for dealing with class imbalance. The loss function is a dynamically scaled cross entropy loss, where the scaling factor decays to zero as confidence in the correct class increases, see Figure 1. Intuitively, this scaling factor can automatically down-weight the contribution of easy examples during training and rapidly focus the model on hard examples. Experiments show that our proposed Focal Loss enables us to train a high-accuracy, one-stage detector that significantly outperforms the alternatives of training with the sampling heuristics or hard example mining, the previous state-ofthe-art techniques for training one-stage detectors. Finally, we note that the exact form of the focal loss is not crucial, and we show other instantiations can achieve similar results. To demonstrate the effectiveness of the proposed focal loss, we design a simple one-stage object detector called RetinaNet , named for its dense sampling of object locations in an input image. Its design features an efficient in-network feature pyramid and use of anchor boxes. It draws on a variety of recent ideas from [21, 6, 27, 19]. RetinaNet is efficient and accurate; our best model, based on a ResNet-101FPN backbone, achieves a COCO test-dev AP of 39.1 while running at 5 fps, surpassing the previously best published single-model results from both one and two-stage detectors, see Figure 2. ## 2. Related Work Classic Object Detectors: The sliding-window paradigm, in which a classifier is applied on a dense image grid, has a long and rich history. One of the earliest successes is the classic work of LeCun et al . who applied convolutional neural networks to handwritten digit recognition [18, 35]. Viola and Jones [36] used boosted object detectors for face detection, leading to widespread adoption of such models. The introduction of HOG [4] and integral channel features [5] gave rise to effective methods for pedestrian detection. DPMs [8] helped extend dense detectors to more general object categories and had top results on PASCAL [7] for many years. While the sliding-window approach was the leading detection paradigm in classic computer vision, with the resurgence of deep learning [17], two-stage detectors, described next, quickly came to dominate object detection. Two-stage Detectors: The dominant paradigm in modern object detection is based on a two-stage approach. As pioneered in the Selective Search work [34], the first stage generates a sparse set of candidate proposals that should contain all objects while filtering out the majority of negative locations, and the second stage classifies the proposals into foreground classes / background. R-CNN [11] upgraded the second-stage classifier to a convolutional network yielding large gains in accuracy and ushering in the modern era of object detection. R-CNN was improved over the years, both in terms of speed [14, 10] and by using learned object proposals [6, 23, 27]. Region Proposal Networks (RPN) integrated proposal generation with the second-stage classifier into a single convolution network, forming the Faster RCNN framework [27]. Numerous extensions to this framework have been proposed, e.g . [19, 30, 31, 15, 13]. One-stage Detectors: OverFeat [29] was one of the first modern one-stage object detector based on deep networks. More recently SSD [21, 9] and YOLO [25, 26] have renewed interest in one-stage methods. These detectors have been tuned for speed but their accuracy trails that of twostage methods. SSD has a 10-20% lower AP, while YOLO focuses on an even more extreme speed/accuracy trade-off. See Figure 2. Recent work showed that two-stage detectors can be made fast simply by reducing input image resolution and the number of proposals, but one-stage methods trailed in accuracy even with a larger compute budget [16]. In contrast, the aim of this work is to understand if one-stage detectors can match or surpass the accuracy of two-stage detectors while running at similar or faster speeds. The design of our RetinaNet detector shares many similarities with previous dense detectors, in particular the concept of 'anchors' introduced by RPN [27] and use of features pyramids as in SSD [21] and FPN [19]. We emphasize that our simple detector achieves top results not based on innovations in network design but due to our novel loss. Class Imbalance: Both classic one-stage object detection methods, like boosted detectors [36, 5] and DPMs [8], and more recent methods, like SSD [21], face a large class imbalance during training. These detectors evaluate 10 4 -10 5 candidate locations per image but only a few locations contain objects. This imbalance causes two problems: (1) training is inefficient as most locations are easy negatives that contribute no useful learning signal; (2) en masse, the easy negatives can overwhelm training and lead to degenerate models. A common solution is to perform some form of hard negative mining [32, 36, 8, 30, 21] that samples hard examples during training or more complex sampling/reweighing schemes [2]. In contrast, we show that our proposed focal loss naturally handles the class imbalance faced by a one-stage detector and allows us to efficiently train on all examples without sampling and without easy negatives overwhelming the loss and computed gradients. Robust Estimation: There has been much interest in designing robust loss functions ( e.g ., Huber loss [12]) that reduce the contribution of outliers by down-weighting the loss of examples with large errors (hard examples). In contrast, rather than addressing outliers, our focal loss is designed to address class imbalance by down-weighting inliers (easy examples) such that their contribution to the total loss is small even if their number is large. In other words, the focal loss performs the opposite role of a robust loss: it focuses training on a sparse set of hard examples. ## 3. Focal Loss The Focal Loss is designed to address the one-stage object detection scenario in which there is an extreme imbalance between foreground and background classes during training ( e.g ., 1:1000). We introduce the focal loss starting from the cross entropy (CE) loss for binary classification 1 : $$CE ( p , y ) = \begin { cases } - \log ( p ) & \text { if } y = 1 \\ ( 1 - \log ( 1 -$$ In the above y ∈ {± 1 } specifies the ground-truth class and p ∈ [0 , 1] is the model's estimated probability for the class with label y = 1 . For notational convenience, we define p t : $$p _ { t } = \{ p \mid y = 1 - p \text{ otherwise} \}$$ and rewrite CE ( p, y ) = CE ( p t ) = -log( p t ) . The CE loss can be seen as the blue (top) curve in Figure 1. One notable property of this loss, which can be easily seen in its plot, is that even examples that are easily classified ( p t . 5 ) incur a loss with non-trivial magnitude. When summed over a large number of easy examples, these small loss values can overwhelm the rare class. 1 Extending the focal loss to the multi-class case is straightforward and works well; for simplicity we focus on the binary loss in this work. ## 3.1. Balanced Cross Entropy A common method for addressing class imbalance is to introduce a weighting factor α ∈ [0 , 1] for class 1 and 1 -α for class -1 . In practice α may be set by inverse class frequency or treated as a hyperparameter to set by cross validation. For notational convenience, we define α t analogously to how we defined p t. We write the α -balanced CE loss as: $$( 1 ) = - ( 1 6 )$$ This loss is a simple extension to CE that we consider as an experimental baseline for our proposed focal loss. ## 3.2. Focal Loss Definition As our experiments will show, the large class imbalance encountered during training of dense detectors overwhelms the cross entropy loss. Easily classified negatives comprise the majority of the loss and dominate the gradient. While α balances the importance of positive/negative examples, it does not differentiate between easy/hard examples. Instead, we propose to reshape the loss function to down-weight easy examples and thus focus training on hard negatives. More formally, we propose to add a modulating factor (1 -p t ) γ to the cross entropy loss, with tunable focusing parameter γ ≥ 0 . We define the focal loss as: $$\log ( p _ { 1 } ) = - ( 1 - p _ { 1 } ) ^ { \prime }$$ The focal loss is visualized for several values of γ ∈ [0 , 5] in Figure 1. We note two properties of the focal loss. (1) When an example is misclassified and p t is small, the modulating factor is near 1 and the loss is unaffected. As p t → 1 , the factor goes to 0 and the loss for well-classified examples is down-weighted. (2) The focusing parameter γ smoothly adjusts the rate at which easy examples are downweighted. When γ = 0 , FL is equivalent to CE, and as γ is increased the effect of the modulating factor is likewise increased (we found γ = 2 to work best in our experiments). Intuitively, the modulating factor reduces the loss contribution from easy examples and extends the range in which an example receives low loss. For instance, with γ = 2 , an example classified with p t = 0 . 9 would have 100 × lower loss compared with CE and with p t ≈ 0 . 968 it would have 1000 × lower loss. This in turn increases the importance of correcting misclassified examples (whose loss is scaled down by at most 4 × for p t ≤ . 5 and γ = 2 ). In practice we use an α -balanced variant of the focal loss: $$\log ( p _ { 1 } ) \cdot \ldots \cdot \log ( p _ { n } )$$ We adopt this form in our experiments as it yields slightly improved accuracy over the nonα -balanced form. Finally, we note that the implementation of the loss layer combines the sigmoid operation for computing p with the loss computation, resulting in greater numerical stability. While in our main experimental results we use the focal loss definition above, its precise form is not crucial. In the appendix we consider other instantiations of the focal loss and demonstrate that these can be equally effective. ## 3.3. Class Imbalance and Model Initialization Binary classification models are by default initialized to have equal probability of outputting either y = -1 or 1 . Under such an initialization, in the presence of class imbalance, the loss due to the frequent class can dominate total loss and cause instability in early training. To counter this, we introduce the concept of a 'prior' for the value of p estimated by the model for the rare class (foreground) at the start of training . We denote the prior by π and set it so that the model's estimated p for examples of the rare class is low, e.g . 0 . 01 . We note that this is a change in model initialization (see § 4.1) and not of the loss function. We found this to improve training stability for both the cross entropy and focal loss in the case of heavy class imbalance. ## 3.4. Class Imbalance and Two-stage Detectors Two-stage detectors are often trained with the cross entropy loss without use of α -balancing or our proposed loss. Instead, they address class imbalance through two mechanisms: (1) a two-stage cascade and (2) biased minibatch sampling. The first cascade stage is an object proposal mechanism [34, 23, 27] that reduces the nearly infinite set of possible object locations down to one or two thousand. Importantly, the selected proposals are not random, but are likely to correspond to true object locations, which removes the vast majority of easy negatives. When training the second stage, biased sampling is typically used to construct minibatches that contain, for instance, a 1:3 ratio of positive to negative examples. This ratio is like an implicit α -balancing factor that is implemented via sampling. Our proposed focal loss is designed to address these mechanisms in a one-stage detection system directly via the loss function. ## 4. RetinaNet Detector RetinaNet is a single, unified network composed of a backbone network and two task-specific subnetworks . The backbone is responsible for computing a convolutional feature map over an entire input image and is an off-the-self convolutional network. The first subnet performs convolutional object classification on the backbone's output; the second subnet performs convolutional bounding box regression. The two subnetworks feature a simple design that we propose specifically for one-stage, dense detection, see Figure 3. While there are many possible choices for the details of these components, most design parameters are not particularly sensitive to exact values as shown in the experiments. We describe each component of RetinaNet next. Feature Pyramid Network Backbone: We adopt the Feature Pyramid Network (FPN) from [19] as the backbone network for RetinaNet. In brief, FPN augments a standard convolutional network with a top-down pathway and lateral connections so the network efficiently constructs a rich, multi-scale feature pyramid from a single resolution input image, see Figure 3(a)-(b). Each level of the pyramid can be used for detecting objects at a different scale. FPN improves multi-scale predictions from fully convolutional networks (FCN) [22], as shown by its gains for RPN [27] and DeepMask-style proposals [23], as well at two-stage detectors such as Fast R-CNN [10] or Mask R-CNN [13]. Following [19], we build FPN on top of the ResNet architecture [15]. We construct a pyramid with levels P 3 through P 7 , where l indicates pyramid level ( P l has resolution 2 l lower than the input). As in [19] all pyramid levels have C = 256 channels. Details of the pyramid generally follow [19] with a few modest differences. 2 While many design choices are not crucial, we emphasize the use of the FPN backbone is; preliminary experiments using features from only the final ResNet layer yielded low AP. Anchors: We use translation-invariant anchor boxes similar to those in the RPN variant in [19]. The anchors have areas of 32 2 to 512 2 on pyramid levels P 3 to P 7 , respectively. As in [19], at each pyramid level we use anchors at three aspect ratios { 1 : 2 , 1 : 1 , 2 : 1 } . For denser scale coverage than in [19], at each level we add anchors of sizes { 2 0 , 2 1 / 3 , 2 2 / 3 } of the original set of 3 aspect ratio anchors. This improve AP in our setting. In total there are A = 9 anchors per level and across levels they cover the scale range 32 813 pixels with respect to the network's input image. Each anchor is assigned a length K one-hot vector of classification targets, where K is the number of object classes, and a 4-vector of box regression targets. We use the assignment rule from RPN [27] but modified for multiclass detection and with adjusted thresholds. Specifically, anchors are assigned to ground-truth object boxes using an intersection-over-union (IoU) threshold of 0.5; and to background if their IoU is in [0, 0.4). As each anchor is assigned to at most one object box, we set the corresponding entry in its length K label vector to 1 and all other entries to 0 . If an anchor is unassigned, which may happen with overlap in [0.4, 0.5), it is ignored during training. Box regression targets are computed as the offset between each anchor and its assigned object box, or omitted if there is no assignment. 2 RetinaNet uses feature pyramid levels P 3 to P 7 , where P 3 to P 5 are computed from the output of the corresponding ResNet residual stage ( C 3 through C 5 ) using top-down and lateral connections just as in [19], P 6 is obtained via a 3 × 3 stride-2 conv on C 5 , and P 7 is computed by applying ReLU followed by a 3 × 3 stride-2 conv on P 6 . This differs slightly from [19]: (1) we don't use the high-resolution pyramid level P 2 for computational reasons, (2) P 6 is computed by strided convolution instead of downsampling, and (3) we include P 7 to improve large object detection. These minor modifications improve speed while maintaining accuracy. Figure 3. The one-stage RetinaNet network architecture uses a Feature Pyramid Network (FPN) [19] backbone on top of a feedforward ResNet architecture [15] (a) to generate a rich, multi-scale convolutional feature pyramid (b). To this backbone RetinaNet attaches two subnetworks, one for classifying anchor boxes (c) and one for regressing from anchor boxes to ground-truth object boxes (d). The network design is intentionally simple, which enables this work to focus on a novel focal loss function that eliminates the accuracy gap between our one-stage detector and state-of-the-art two-stage detectors like Faster R-CNN with FPN [19] while running at faster speeds. <details> <summary>Image 3 Details</summary>  ### Visual Description ## Diagram: Neural Network Architecture for Object Detection ### Overview The diagram illustrates a multi-stage neural network architecture for object detection, combining a ResNet backbone, feature pyramid network (FPN), and specialized subnetworks for classification and bounding box prediction. The flow progresses from input image processing through hierarchical feature extraction to specialized prediction tasks. ### Components/Axes 1. **(a) ResNet**: - Stacked residual blocks with upward arrows indicating residual connections. - Input: Image (e.g., snowy landscape with ski equipment). 2. **(b) Feature Pyramid Net**: - Hierarchical feature maps with downward arrows showing multi-scale feature extraction. - Labels: "class+box subnets" at each pyramid level. 3. **(c) Class Subnetwork (Top)**: - Green blocks labeled with dimensions: `WxH x 256 → WxH x 256 → WxH x KA`. - Arrows indicate transformations (e.g., `x4` scaling). 4. **(d) Box Subnetwork (Bottom)**: - Blue blocks labeled with dimensions: `WxH x 256 → WxH x 256 → WxH x 4A`. - Arrows show scaling factors (e.g., `x4A`). ### Detailed Analysis - **ResNet (a)**: - Input image processed through residual blocks (gray arrows) to generate feature maps. - **Feature Pyramid Net (b)**: - Feature maps from ResNet are downsampled/upsampled (green arrows) to create a pyramid of features at different scales. - Each pyramid level feeds into "class+box subnets" for joint prediction tasks. - **Class Subnetwork (c)**: - Processes features for object classification (`WxH x KA` likely represents class-specific feature maps, where `K` = number of classes). - Scaling factor `x4` suggests upsampling to refine predictions. - **Box Subnetwork (d)**: - Handles bounding box regression (`WxH x 4A` likely encodes box coordinates, where `A` = 4 for [x, y, width, height]). - Scaling factor `x4A` indicates spatial refinement for box localization. ### Key Observations 1. **Multi-Scale Processing**: The FPN enables detection of objects at varying scales by combining features from different ResNet layers. 2. **Specialized Subnetworks**: Separation of class and box prediction tasks allows focused optimization (e.g., class subnetwork prioritizes semantic features, box subnetwork focuses on spatial precision). 3. **Dimensional Consistency**: - `WxH x 256` likely denotes feature map dimensions (width × height × channels). - `KA` and `4A` suggest task-specific output dimensions (class logits and box coordinates, respectively). ### Interpretation This architecture demonstrates a **two-stage object detection pipeline**: 1. **Backbone + FPN**: Extracts rich, multi-scale features from the input image. 2. **Subnetworks**: Specialized networks process these features for classification and regression tasks, improving efficiency and accuracy. The use of residual connections (ResNet) mitigates vanishing gradients, enabling deeper networks. The FPN's top-down pathway with lateral connections (implied by pyramid structure) enhances feature reuse across scales. The separation of class and box subnetworks may reduce computational redundancy, as each subnetwork focuses on distinct prediction tasks. The scaling factors (`x4`, `x4A`) suggest adaptive feature map adjustments to balance resolution and computational cost. </details> Classification Subnet: The classification subnet predicts the probability of object presence at each spatial position for each of the A anchors and K object classes. This subnet is a small FCN attached to each FPN level; parameters of this subnet are shared across all pyramid levels. Its design is simple. Taking an input feature map with C channels from a given pyramid level, the subnet applies four 3 × 3 conv layers, each with C filters and each followed by ReLU activations, followed by a 3 × 3 conv layer with KA filters. Finally sigmoid activations are attached to output the KA binary predictions per spatial location, see Figure 3 (c). We use C = 256 and A = 9 in most experiments. In contrast to RPN [27], our object classification subnet is deeper, uses only 3 × 3 convs, and does not share parameters with the box regression subnet (described next). We found these higher-level design decisions to be more important than specific values of hyperparameters. Box Regression Subnet: In parallel with the object classification subnet, we attach another small FCN to each pyramid level for the purpose of regressing the offset from each anchor box to a nearby ground-truth object, if one exists. The design of the box regression subnet is identical to the classification subnet except that it terminates in 4 A linear outputs per spatial location, see Figure 3 (d). For each of the A anchors per spatial location, these 4 outputs predict the relative offset between the anchor and the groundtruth box (we use the standard box parameterization from RCNN [11]). We note that unlike most recent work, we use a class-agnostic bounding box regressor which uses fewer parameters and we found to be equally effective. The object classification subnet and the box regression subnet, though sharing a common structure, use separate parameters. ## 4.1. Inference and Training Inference: RetinaNet forms a single FCN comprised of a ResNet-FPN backbone, a classification subnet, and a box regression subnet, see Figure 3. As such, inference involves simply forwarding an image through the network. To improve speed, we only decode box predictions from at most 1k top-scoring predictions per FPN level, after thresholding detector confidence at 0.05. The top predictions from all levels are merged and non-maximum suppression with a threshold of 0.5 is applied to yield the final detections. Focal Loss: We use the focal loss introduced in this work as the loss on the output of the classification subnet. As we will show in § 5, we find that γ = 2 works well in practice and the RetinaNet is relatively robust to γ ∈ [0 . 5 , 5] . We emphasize that when training RetinaNet, the focal loss is applied to all ∼ 100k anchors in each sampled image. This stands in contrast to common practice of using heuristic sampling (RPN) or hard example mining (OHEM, SSD) to select a small set of anchors ( e.g ., 256) for each minibatch. The total focal loss of an image is computed as the sum of the focal loss over all ∼ 100k anchors, normalized by the number of anchors assigned to a ground-truth box . We perform the normalization by the number of assigned anchors, not total anchors, since the vast majority of anchors are easy negatives and receive negligible loss values under the focal loss. Finally we note that α , the weight assigned to the rare class, also has a stable range, but it interacts with γ making it necessary to select the two together (see Tables 1a and 1b). In general α should be decreased slightly as γ is increased (for γ = 2 , α = 0 . 25 works best). Initialization: We experiment with ResNet-50-FPN and ResNet-101-FPN backbones [19]. The base ResNet-50 and ResNet-101 models are pre-trained on ImageNet1k; we use the models released by [15]. New layers added for FPN are initialized as in [19]. All new conv layers except the final one in the RetinaNet subnets are initialized with bias b = 0 and a Gaussian weight fill with σ = 0 . 01 . For the final conv layer of the classification subnet, we set the bias initialization to b = -log((1 -π ) /π ) , where π specifies that at Table 1. Ablation experiments for RetinaNet and Focal Loss (FL). All models are trained on trainval35k and tested on minival unless noted. If not specified, default values are: γ = 2 ; anchors for 3 scales and 3 aspect ratios; ResNet-50-FPN backbone; and a 600 pixel train and test image scale. (a) RetinaNet with α -balanced CE achieves at most 31.1 AP. (b) In contrast, using FL with the same exact network gives a 2.9 AP gain and is fairly robust to exact γ / α settings. (c) Using 2-3 scale and 3 aspect ratio anchors yields good results after which point performance saturates. (d) FL outperforms the best variants of online hard example mining (OHEM) [30, 21] by over 3 points AP. (e) Accuracy/Speed trade-off of RetinaNet on test-dev for various network depths and image scales (see also Figure 2). | α | AP | AP | AP 50 AP 75 | AP 50 AP 75 | γ | γ | α | α | AP AP | AP AP | 50 AP | 50 AP | 75 | 75 | #sc #ar | #sc #ar | | | AP AP 50 | AP AP 50 | AP 75 | |-------------------------------------------------|-------------------------------------------------|-------------------------------------------------|-------------------------------------------------|-------------------------------------------------|-------------------------------------------------|-------------------------------------------------|-------------------------------------------------|-------------------------------------------------|-------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------| | .10 | 0.0 | 0.0 | 0.0 0.0 | 0.0 0.0 | 0 | 0 | .75 | .75 | 31.1 | 31.1 | 49.4 | 49.4 | 33.0 | 33.0 | 1 1 | 1 1 | 30.3 | 30.3 | 49.0 | 49.0 | 31.8 | | .25 | 10.8 | 10.8 | 16.0 11.7 | 16.0 11.7 | 0.1 | 0.1 | .75 | .75 | 31.4 | 31.4 | 49.9 33.1 | 49.9 33.1 | | | 2 1 | 2 1 | 31.9 50.0 | 31.9 50.0 | | | 34.0 | | .50 | 30.2 | 30.2 | 46.7 32.8 | 46.7 32.8 | 0.2 | 0.2 | .75 | .75 | 31.9 50.7 | 31.9 50.7 | 33.4 | 33.4 | | | 3 1 | 3 1 | 31.8 49.4 | 31.8 49.4 | | | 33.7 | | .75 | 31.1 | 31.1 | 49.4 33.0 | 49.4 33.0 | 0.5 | 0.5 | .50 | .50 | 32.9 51.7 | 32.9 51.7 | 35.2 | 35.2 | | | 1 3 | 1 3 | 32.4 52.3 | 32.4 52.3 | | | 33.9 | | .90 | 30.8 49.7 | 30.8 49.7 | 32.3 | 32.3 | 1.0 | 1.0 | .25 33.7 | .25 33.7 | 52.0 | 52.0 | 36.2 | 36.2 | | | 2 3 | 2 3 | 34.2 53.1 | 34.2 53.1 | | | 36.5 | | .99 | 28.7 47.4 | 28.7 47.4 | 29.9 | 29.9 | 2.0 | 2.0 | .25 34.0 | .25 34.0 | 52.5 | 52.5 | 36.5 | 36.5 | | | 3 3 | 3 3 | 34.0 52.5 | 34.0 52.5 | | | 36.5 | | .999 | 25.1 41.7 | 25.1 41.7 | 26.1 | 26.1 | 5.0 | 5.0 | .25 32.2 | .25 32.2 | 49.6 | 49.6 | 34.8 | 34.8 | 4 | 4 | 3 | 3 | 33.8 52.1 | 33.8 52.1 | | | 36.2 | | (a) Varying α for CE loss ( γ = 0 ) | (a) Varying α for CE loss ( γ = 0 ) | (a) Varying α for CE loss ( γ = 0 ) | (a) Varying α for CE loss ( γ = 0 ) | (a) Varying α for CE loss ( γ = 0 ) | (b) Varying γ for FL (w. optimal α | (b) Varying γ for FL (w. optimal α | (b) Varying γ for FL (w. optimal α | (b) Varying γ for FL (w. optimal α | (b) Varying γ for FL (w. optimal α | (b) Varying γ for FL (w. optimal α | (b) Varying γ for FL (w. optimal α | (c) Varying anchor scales and aspects | (c) Varying anchor scales and aspects | (c) Varying anchor scales and aspects | (c) Varying anchor scales and aspects | (c) Varying anchor scales and aspects | (c) Varying anchor scales and aspects | (c) Varying anchor scales and aspects | (c) Varying anchor scales and aspects | (c) Varying anchor scales and aspects | (c) Varying anchor scales and aspects | | method | method | batch size | nms thr | AP | AP 50 | AP 75 | | depth | scale | AP | | AP 50 | | AP 75 | | AP S | | AP M | AP | L | time | | OHEM | OHEM | 128 | .7 | 31.1 | 47.2 | 33.2 | 33.2 | 50 400 | 50 400 | | 30.5 | 47.8 | | 32.7 | | 11.2 | 33.8 | | 46.1 | | 64 | | OHEM | OHEM | 256 | .7 | 31.8 | 48.8 | 33.9 | 33.9 | 50 | 50 | 500 | 32.5 | 50.9 | | 34.8 | | 13.9 | 35.8 | | 46.7 | | 72 | | OHEM | OHEM | 512 | .7 | 30.6 | 47.0 | 32.6 | 32.6 | 50 | 50 | 600 | 34.3 | 53.2 | | 36.9 | 16.2 | | 37.4 | | 47.4 | | 98 | | OHEM | OHEM | 128 | .5 | 32.8 | 50.3 | 35.1 | 35.1 | 50 | 50 | 700 | 35.1 | | 54.2 | 37.7 | | 18.0 | 39.3 | | 46.4 | | 121 | | OHEM | OHEM | 256 | .5 | 31.0 | 47.4 | 33.0 | 33.0 | 50 | 50 | 800 | 35.7 | 55.0 | 38.5 | | 18.9 | | 38.9 | | 46.3 | | 153 | | OHEM | OHEM | 512 | .5 | 27.6 | 42.0 | 29.2 | 29.2 | 101 | 101 | 400 | 31.9 | 49.5 | 34.1 | | 11.6 | | 35.8 | | 48.5 | | 81 | | OHEM 1:3 | OHEM 1:3 | 128 | .5 | 31.1 | 47.2 | 33.2 | 33.2 | 101 | 101 | 500 | 34.4 | 53.1 | 36.8 | | | 14.7 | 38.5 | | 49.1 | | 90 | | OHEM 1:3 | OHEM 1:3 | 256 | .5 | 28.3 | 42.4 | 30.3 | 30.3 | 101 | 101 | 600 | 36.0 | 55.2 | | 38.7 | | 17.4 | 39.6 | | 49.7 | | 122 | | OHEM 1:3 | OHEM 1:3 | 512 | .5 | 24.0 | 35.5 | 25.8 | 25.8 | 101 | 101 | 700 | 37.1 | 56.6 | | 39.8 | | 19.1 | 40.6 | | 49.4 | | 154 | | FL | FL | n/a | n/a | 36.0 | 54.9 | 38.7 | 38.7 | 101 | 101 | 800 | 37.8 | 57.5 | | 40.8 | | 20.2 | 41.1 | | 49.2 | | 198 | | (d) FL vs .OHEM baselines (with ResNet-101-FPN) | (d) FL vs .OHEM baselines (with ResNet-101-FPN) | (d) FL vs .OHEM baselines (with ResNet-101-FPN) | (d) FL vs .OHEM baselines (with ResNet-101-FPN) | (d) FL vs .OHEM baselines (with ResNet-101-FPN) | (d) FL vs .OHEM baselines (with ResNet-101-FPN) | (d) FL vs .OHEM baselines (with ResNet-101-FPN) | (d) FL vs .OHEM baselines (with ResNet-101-FPN) | (d) FL vs .OHEM baselines (with ResNet-101-FPN) | (d) FL vs .OHEM baselines (with ResNet-101-FPN) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | (e) Accuracy/speed trade-off RetinaNet (on test-dev ) | the start of training every anchor should be labeled as foreground with confidence of ∼ π . We use π = . 01 in all experiments, although results are robust to the exact value. As explained in § 3.4, this initialization prevents the large number of background anchors from generating a large, destabilizing loss value in the first iteration of training. Optimization: RetinaNet is trained with stochastic gradient descent (SGD). We use synchronized SGD over 8 GPUs with a total of 16 images per minibatch (2 images per GPU). Unless otherwise specified, all models are trained for 90k iterations with an initial learning rate of 0.01, which is then divided by 10 at 60k and again at 80k iterations. We use horizontal image flipping as the only form of data augmentation unless otherwise noted. Weight decay of 0.0001 and momentum of 0.9 are used. The training loss is the sum the focal loss and the standard smooth L 1 loss used for box regression [10]. Training time ranges between 10 and 35 hours for the models in Table 1e. ## 5. Experiments We present experimental results on the bounding box detection track of the challenging COCO benchmark [20]. For training, we follow common practice [1, 19] and use the COCO trainval35k split (union of 80k images from train and a random 35k subset of images from the 40k image val split). We report lesion and sensitivity studies by evaluating on the minival split (the remaining 5k images from val ). For our main results, we report COCO AP on the test-dev split, which has no public labels and requires use of the evaluation server. ## 5.1. Training Dense Detection We run numerous experiments to analyze the behavior of the loss function for dense detection along with various optimization strategies. For all experiments we use depth 50 or 101 ResNets [15] with a Feature Pyramid Network (FPN) [19] constructed on top. For all ablation studies we use an image scale of 600 pixels for training and testing. Network Initialization: Our first attempt to train RetinaNet uses standard cross entropy (CE) loss without any modifications to the initialization or learning strategy. This fails quickly, with the network diverging during training. However, simply initializing the last layer of our model such that the prior probability of detecting an object is π = . 01 (see § 4.1) enables effective learning. Training RetinaNet with ResNet-50 and this initialization already yields a respectable AP of 30.2 on COCO. Results are insensitive to the exact value of π so we use π = . 01 for all experiments. Figure 4. Cumulative distribution functions of the normalized loss for positive and negative samples for different values of γ for a converged model. The effect of changing γ on the distribution of the loss for positive examples is minor. For negatives, however, increasing γ heavily concentrates the loss on hard examples, focusing nearly all attention away from easy negatives. <details> <summary>Image 4 Details</summary>  ### Visual Description ## Line Graphs: Cumulative Normalized Loss vs. Fraction of Examples ### Overview The image contains two side-by-side line graphs comparing cumulative normalized loss against the fraction of examples (foreground and background). Each graph displays four curves corresponding to different γ (gamma) values: 0, 0.5, 1, and 2. The curves illustrate how loss accumulates as the proportion of examples increases. ### Components/Axes - **Left Graph (Foreground Examples)**: - **X-axis**: "fraction of foreground examples" (0 to 1 in increments of 0.2). - **Y-axis**: "cumulative normalized loss" (0 to 1 in increments of 0.2). - **Legend**: Top-left, mapping γ values to colors: - γ = 0: Blue - γ = 0.5: Orange - γ = 1: Yellow - γ = 2: Purple - **Right Graph (Background Examples)**: - **X-axis**: "fraction of background examples" (0 to 1 in increments of 0.2). - **Y-axis**: "cumulative normalized loss" (0 to 1 in increments of 0.2). - **Legend**: Top-left, identical color coding as the left graph. ### Detailed Analysis #### Left Graph (Foreground Examples) - **γ = 0 (Blue)**: The curve starts at (0,0) and rises gradually, reaching ~0.8 at x=1. - **γ = 0.5 (Orange)**: Steeper than γ=0, reaching ~0.85 at x=1. - **γ = 1 (Yellow)**: Even steeper, reaching ~0.9 at x=1. - **γ = 2 (Purple)**: The steepest curve, reaching ~0.95 at x=1. - **Trend**: All curves start at the origin and end at (1,1), with higher γ values producing sharper increases in loss. #### Right Graph (Background Examples) - **γ = 0 (Blue)**: Starts at (0,0) and rises slowly, reaching ~0.6 at x=1. - **γ = 0.5 (Orange)**: Steeper than γ=0, reaching ~0.7 at x=1. - **γ = 1 (Yellow)**: Reaches ~0.8 at x=1. - **γ = 2 (Purple)**: The steepest curve, reaching ~0.9 at x=1. - **Trend**: Similar to the left graph, but all curves plateau below 1.0, with γ=2 approaching ~0.9. ### Key Observations 1. **Consistent γ Behavior**: Higher γ values consistently produce steeper curves in both graphs, indicating a direct relationship between γ and loss sensitivity. 2. **Asymmetry Between Foreground/Background**: For the same γ, the loss increase is more pronounced for foreground examples (left graph) than background examples (right graph). 3. **Convergence at x=1**: All curves reach 1.0 for foreground examples but plateau below 1.0 for background examples, suggesting a ceiling effect for background loss. ### Interpretation The graphs demonstrate that: - **γ Controls Sensitivity**: Higher γ values amplify the impact of example inclusion on cumulative loss, particularly for foreground data. - **Imbalanced Impact**: Foreground examples drive loss more aggressively than background examples, even at the same γ. This could reflect class imbalance or differential weighting in the model. - **Practical Implications**: Tuning γ may help balance loss contributions from foreground/background examples, critical for tasks like anomaly detection or imbalanced classification. No outliers or anomalies are observed. The data suggests a systematic relationship between γ, example type, and loss accumulation. </details> Balanced Cross Entropy: Our next attempt to improve learning involved using the α -balanced CE loss described in § 3.1. Results for various α are shown in Table 1a. Setting α = . 75 gives a gain of 0.9 points AP. Focal Loss: Results using our proposed focal loss are shown in Table 1b. The focal loss introduces one new hyperparameter, the focusing parameter γ , that controls the strength of the modulating term. When γ = 0 , our loss is equivalent to the CE loss. As γ increases, the shape of the loss changes so that 'easy' examples with low loss get further discounted, see Figure 1. FL shows large gains over CE as γ is increased. With γ = 2 , FL yields a 2.9 AP improvement over the α -balanced CE loss . For the experiments in Table 1b, for a fair comparison we find the best α for each γ . We observe that lower α 's are selected for higher γ 's (as easy negatives are downweighted, less emphasis needs to be placed on the positives). Overall, however, the benefit of changing γ is much larger, and indeed the best α 's ranged in just [.25,.75] (we tested α ∈ [ . 01 , . 999] ). We use γ = 2 . 0 with α = . 25 for all experiments but α = . 5 works nearly as well (.4 AP lower). Analysis of the Focal Loss: To understand the focal loss better, we analyze the empirical distribution of the loss of a converged model. For this, we take take our default ResNet101 600-pixel model trained with γ = 2 (which has 36.0 AP). We apply this model to a large number of random images and sample the predicted probability for ∼ 10 7 negative windows and ∼ 10 5 positive windows. Next, separately for positives and negatives, we compute FL for these samples, and normalize the loss such that it sums to one. Given the normalized loss, we can sort the loss from lowest to highest and plot its cumulative distribution function (CDF) for both positive and negative samples and for different settings for γ (even though model was trained with γ = 2 ). Cumulative distribution functions for positive and negative samples are shown in Figure 4. If we observe the positive samples, we see that the CDF looks fairly similar for different values of γ . For example, approximately 20% of the hardest positive samples account for roughly half of the positive loss, as γ increases more of the loss gets concentrated in the top 20% of examples, but the effect is minor. The effect of γ on negative samples is dramatically different. For γ = 0 , the positive and negative CDFs are quite similar. However, as γ increases, substantially more weight becomes concentrated on the hard negative examples. In fact, with γ = 2 (our default setting), the vast majority of the loss comes from a small fraction of samples. As can be seen, FL can effectively discount the effect of easy negatives, focusing all attention on the hard negative examples. Online Hard Example Mining (OHEM): [30] proposed to improve training of two-stage detectors by constructing minibatches using high-loss examples. Specifically, in OHEM each example is scored by its loss, non-maximum suppression (nms) is then applied, and a minibatch is constructed with the highest-loss examples. The nms threshold and batch size are tunable parameters. Like the focal loss, OHEM puts more emphasis on misclassified examples, but unlike FL, OHEM completely discards easy examples. We also implement a variant of OHEM used in SSD [21]: after applying nms to all examples, the minibatch is constructed to enforce a 1:3 ratio between positives and negatives to help ensure each minibatch has enough positives. We test both OHEM variants in our setting of one-stage detection which has large class imbalance. Results for the original OHEM strategy and the 'OHEM 1:3' strategy for selected batch sizes and nms thresholds are shown in Table 1d. These results use ResNet-101, our baseline trained with FL achieves 36.0 AP for this setting. In contrast, the best setting for OHEM (no 1:3 ratio, batch size 128, nms of .5) achieves 32.8 AP. This is a gap of 3.2 AP, showing FL is more effective than OHEM for training dense detectors. We note that we tried other parameter setting and variants for OHEM but did not achieve better results. Hinge Loss: Finally, in early experiments, we attempted to train with the hinge loss [12] on p t, which sets loss to 0 above a certain value of p t. However, this was unstable and we did not manage to obtain meaningful results. Results exploring alternate loss functions are in the appendix. Table 2. Object detection single-model results (bounding box AP), vs . state-of-the-art on COCO test-dev . We show results for our RetinaNet-101-800 model, trained with scale jitter and for 1.5 × longer than the same model from Table 1e. Our model achieves top results, outperforming both one-stage and two-stage models. For a detailed breakdown of speed versus accuracy see Table 1e and Figure 2. | | backbone | AP | AP 50 | AP 75 | AP S | AP M | AP L | |----------------------------|--------------------------|------|---------|---------|--------|--------|--------| | Two-stage methods | | | | | | | | | Faster R-CNN+++ [15] | ResNet-101-C4 | 34.9 | 55.7 | 37.4 | 15.6 | 38.7 | 50.9 | | Faster R-CNN w FPN [19] | ResNet-101-FPN | 36.2 | 59.1 | 39.0 | 18.2 | 39.0 | 48.2 | | Faster R-CNN by G-RMI [16] | Inception-ResNet-v2 [33] | 34.7 | 55.5 | 36.7 | 13.5 | 38.1 | 52.0 | | Faster R-CNN w TDM [31] | Inception-ResNet-v2-TDM | 36.8 | 57.7 | 39.2 | 16.2 | 39.8 | 52.1 | | One-stage methods | | | | | | | | | YOLOv2 [26] | DarkNet-19 [26] | 21.6 | 44.0 | 19.2 | 5.0 | 22.4 | 35.5 | | SSD513 [21, 9] | ResNet-101-SSD | 31.2 | 50.4 | 33.3 | 10.2 | 34.5 | 49.8 | | DSSD513 [9] | ResNet-101-DSSD | 33.2 | 53.3 | 35.2 | 13.0 | 35.4 | 51.1 | | RetinaNet (ours) | ResNet-101-FPN | 39.1 | 59.1 | 42.3 | 21.8 | 42.7 | 50.2 | ## 5.2. Model Architecture Design Anchor Density: One of the most important design factors in a one-stage detection system is how densely it covers the space of possible image boxes. Two-stage detectors can classify boxes at any position, scale, and aspect ratio using a region pooling operation [10]. In contrast, as one-stage detectors use a fixed sampling grid, a popular approach for achieving high coverage of boxes in these approaches is to use multiple 'anchors' [27] at each spatial position to cover boxes of various scales and aspect ratios. We sweep over the number of scale and aspect ratio anchors used at each spatial position and each pyramid level in FPN. We consider cases from a single square anchor at each location to 12 anchors per location spanning 4 sub-octave scales ( 2 k/ 4 , for k ≤ 3 ) and 3 aspect ratios [0.5, 1, 2]. Results using ResNet-50 are shown in Table 1c. A surprisingly good AP (30.3) is achieved using just one square anchor. However, the AP can be improved by nearly 4 points (to 34.0) when using 3 scales and 3 aspect ratios per location. We used this setting for all other experiments in this work. Finally, we note that increasing beyond 6-9 anchors did not shown further gains. Thus while two-stage systems can classify arbitrary boxes in an image, the saturation of performance w.r.t. density implies the higher potential density of two-stage systems may not offer an advantage. Speed versus Accuracy: Larger backbone networks yield higher accuracy, but also slower inference speeds. Likewise for input image scale (defined by the shorter image side). We show the impact of these two factors in Table 1e. In Figure 2 we plot the speed/accuracy trade-off curve for RetinaNet and compare it to recent methods using public numbers on COCO test-dev . The plot reveals that RetinaNet, enabled by our focal loss, forms an upper envelope over all existing methods, discounting the low-accuracy regime. Remarkably, RetinaNet with ResNet-101-FPN and a 600 pixel image scale (which we denote by RetinaNet-101-600 for simplicity) matches the accuracy of the recently published ResNet-101-FPN Faster R-CNN [19], while running in 122 ms per image compared to 172 ms (both measured on an Nvidia M40 GPU). Using larger image sizes allows RetinaNet to surpass the accuracy of all two-stage approaches, while still being faster. For faster runtimes, there is only one operating point (500 pixel input) at which using ResNet-50FPN improves over ResNet-101-FPN. Addressing the high frame rate regime will likely require special network design, as in [26], rather than use of an off-the-shelf model and is beyond the scope of this work. ## 5.3. Comparison to State of the Art We evaluate RetinaNet on the bounding box detection task of the challenging COCO dataset and compare test-dev results to recent state-of-the-art methods including both one-stage and two-stage models. Results are presented in Table 2 for our RetinaNet-101-800 model trained using scale jitter and for 1.5 × longer than the models in Table 1e (giving a 1.3 AP gain). Compared to existing onestage methods, our approach achieves a healthy 5.9 point AP gap (39.1 vs . 33.2) with the closest competitor, DSSD [9], while also being faster, see Figure 2. Compared to recent two-stage methods, RetinaNet achieves a 2.3 point gap above the top-performing Faster R-CNN model based on Inception-ResNet-v2-TDM [31]. ## 6. Conclusion In this work, we identify class imbalance as the primary obstacle preventing one-stage object detectors from surpassing top-performing, two-stage methods, such as Faster R-CNN variants. To address this, we propose the focal loss which applies a modulating term to the cross entropy loss in order to focus learning on hard examples and down-weight the numerous easy negatives. Our approach is simple and highly effective. We demonstrate its efficacy by designing a fully convolutional one-stage detector and report extensive experimental analysis showing that it achieves state-of-theart accuracy and run time on the challenging COCO dataset. Figure 5. Focal loss variants compared to the cross entropy as a function of x t = yx . Both the original FL and alternate variant FL ∗ reduce the relative loss for well-classified examples ( x t > 0 ). <details> <summary>Image 5 Details</summary>  ### Visual Description ## Line Graph: Loss Function Comparison ### Overview The image depicts a line graph comparing four loss functions (CE, FL γ=2, FL* γ=2, β=1, and FL* γ=4, β=0) across a range of input values (x_t). All lines originate from the same point (x_t = -5, loss = 5) and exhibit decreasing trends as x_t increases, with distinct divergence patterns after x_t = 0. ### Components/Axes - **X-axis**: Labeled "x_t" with tick marks at -5, 0, and 5. - **Y-axis**: Labeled "loss" with tick marks at 0, 1, 2, 3, 4, and 5. - **Legend**: Located in the top-right corner, associating colors with functions: - Blue: CE - Orange: FL γ=2 - Yellow: FL* γ=2, β=1 - Purple: FL* γ=4, β=0 ### Detailed Analysis 1. **CE (Blue Line)**: - Starts at (x_t = -5, loss = 5). - Decreases linearly until x_t = 0, where loss = 0. - For x_t > 0, the loss increases slightly, forming a shallow upward curve. 2. **FL γ=2 (Orange Line)**: - Starts at (x_t = -5, loss = 5). - Decreases more steeply than CE until x_t = 0 (loss = 0). - For x_t > 0, the loss rises sharply, surpassing the CE line. 3. **FL* γ=2, β=1 (Yellow Line)**: - Starts at (x_t = -5, loss = 5). - Decreases faster than CE but slower than FL γ=2 until x_t = 0 (loss = 0). - For x_t > 0, the loss increases moderately, staying below FL γ=2 but above CE. 4. **FL* γ=4, β=0 (Purple Line)**: - Starts at (x_t = -5, loss = 5). - Decreases steepest among all lines until x_t = 0 (loss = 0). - For x_t > 0, the loss rises sharply but remains below all other lines. ### Key Observations - All lines converge at (x_t = 0, loss = 0), suggesting a shared minimum point. - For x_t > 0, the purple line (FL* γ=4, β=0) maintains the lowest loss, while the orange line (FL γ=2) has the highest. - The CE line diverges upward for x_t > 0, unlike the others, which continue decreasing before rising. ### Interpretation The graph illustrates how loss functions behave under varying parameters (γ, β). Higher γ values (e.g., γ=4) with β=0 produce steeper initial declines but lower asymptotic loss values for x_t > 0. The inclusion of β=1 (yellow line) moderates the loss increase for x_t > 0 compared to β=0. The CE line’s unique upward trend for x_t > 0 suggests it penalizes positive x_t values differently, possibly reflecting a different regularization strategy. This analysis highlights the trade-offs between parameter choices in optimizing loss functions for specific input ranges. </details> Table 3. Results of FL and FL ∗ versus CE for select settings. | loss | γ | β | AP | AP 50 | AP 75 | |--------|-----|-----|------|---------|---------| | CE | - | - | 31.1 | 49.4 | 33 | | FL | 2.0 | - | 34 | 52.5 | 36.5 | | FL ∗ | 2.0 | 1.0 | 33.8 | 52.7 | 36.3 | | FL ∗ | 4.0 | 0.0 | 33.9 | 51.8 | 36.4 | ## Appendix A: Focal Loss* The exact form of the focal loss is not crucial. We now show an alternate instantiation of the focal loss that has similar properties and yields comparable results. The following also gives more insights into properties of the focal loss. Webegin by considering both cross entropy (CE) and the focal loss (FL) in a slightly different form than in the main text. Specifically, we define a quantity x t as follows: $$1 1$$ where y ∈ {± 1 } specifies the ground-truth class as before. We can then write p t = σ ( x t ) (this is compatible with the definition of p t in Equation 2). An example is correctly classified when x t > 0 , in which case p t > . 5 . We can now define an alternate form of the focal loss in terms of x t. We define p ∗ t and FL ∗ as follows: $$1 ^ { \prime } = n ( n + 1 )$$ $$1 ^ { 2 } = - 1 \times ( - 1 ) ^ { 2 } = 1$$ FL ∗ has two parameters, γ and β , that control the steepness and shift of the loss curve. We plot FL ∗ for two selected settings of γ and β in Figure 5 alongside CE and FL. As can be seen, like FL, FL ∗ with the selected parameters diminishes the loss assigned to well-classified examples. We trained RetinaNet-50-600 using identical settings as before but we swap out FL for FL ∗ with the selected parameters. These models achieve nearly the same AP as those trained with FL, see Table 3. In other words, FL ∗ is a reasonable alternative for the FL that works well in practice. Figure 6. Derivates of the loss functions from Figure 5 w.r.t. x . <details> <summary>Image 6 Details</summary>  ### Visual Description ## Line Graph: Gradient vs. x_t ### Overview The image depicts a line graph comparing four distinct gradient behaviors across a range of x_t values (-10 to 10). The y-axis represents gradient values (-1.2 to 0), while the x-axis represents x_t. Four data series are plotted, each with unique curvature and inflection points, suggesting different functional relationships between x_t and gradient. ### Components/Axes - **X-axis (x_t)**: Labeled "x_t", scaled from -10 to 10 in increments of 5. - **Y-axis (gradient)**: Labeled "gradient", scaled from -1.2 to 0 in increments of 0.2. - **Legend**: Positioned in the top-right corner, with four entries: - **CE**: Blue line - **FL γ=2**: Orange line - **FL* γ=2, β=1**: Yellow line - **FL* γ=4, β=0**: Purple line - **Gridlines**: Faint horizontal and vertical lines at major axis intervals. ### Detailed Analysis 1. **CE (Blue Line)**: - Starts at approximately -1.0 at x_t = -10. - Gradually increases, reaching ~-0.2 by x_t = 0. - Plateaus near 0 for x_t > 0. 2. **FL γ=2 (Orange Line)**: - Begins at ~-1.0 at x_t = -10. - Dips sharply to ~-1.1 at x_t ≈ -5. - Rises steeply, crossing 0 at x_t ≈ 2. - Plateaus near 0 for x_t > 2. 3. **FL* γ=2, β=1 (Yellow Line)**: - Starts at ~-1.0 at x_t = -10. - Rises gradually, crossing 0 at x_t ≈ 1. - Plateaus near 0 for x_t > 1. 4. **FL* γ=4, β=0 (Purple Line)**: - Starts at ~-1.0 at x_t = -10. - Sharp ascent begins at x_t ≈ -3, crossing 0 at x_t ≈ 0. - Remains flat at 0 for x_t > 0. ### Key Observations - All lines converge at the lower bound (y ≈ -1.0) for x_t ≤ -10. - The **FL* γ=4, β=0** (purple) line exhibits the steepest gradient increase, reaching 0 earliest (x_t ≈ 0). - The **FL γ=2** (orange) line shows a unique dip below -1.0 before rising, unlike other series. - **CE** (blue) and **FL* γ=2, β=1** (yellow) lines share similar shapes but differ in timing: CE reaches 0 at x_t ≈ 0, while the yellow line does so at x_t ≈ 1. ### Interpretation The graph illustrates how gradient behavior varies with parameters γ (scaling factor) and β (offset). Higher γ values (e.g., γ=4) correlate with faster gradient increases, while β=0 eliminates delays in gradient ascent. The **FL γ=2** line’s anomalous dip suggests a transient negative gradient effect, potentially indicating a local minimum or damping effect. The **CE** line may represent a control or baseline scenario, showing gradual improvement without parameter-driven acceleration. These trends imply that parameter tuning (γ, β) critically influences gradient dynamics in the modeled system. </details> Figure 7. Effectiveness of FL ∗ with various settings γ and β . The plots are color coded such that effective settings are shown in blue. <details> <summary>Image 7 Details</summary>  ### Visual Description ## Line Graph: Loss vs. x_t with AP Thresholds ### Overview The image depicts a line graph comparing loss values against a variable `x_t` for two groups defined by an AP (Average Precision) threshold of 33.5. The graph shows multiple lines representing different data series, with distinct slopes and starting points based on AP values. ### Components/Axes - **X-axis**: Labeled `x_t`, ranging from -5 to 5 in increments of 5. - **Y-axis**: Labeled `loss`, ranging from 0 to 5 in increments of 1. - **Legend**: Located in the top-right corner, with two entries: - **Blue lines**: AP > 33.5 - **Gray lines**: AP < 33.5 ### Detailed Analysis - **Blue Lines (AP > 33.5)**: - Start at approximately `x_t = -5` with loss values between 4 and 5. - Slope steeply downward, reaching near-zero loss at `x_t = 0`. - All blue lines converge at the origin (`x_t = 0`, loss ≈ 0). - **Gray Lines (AP < 33.5)**: - Start at approximately `x_t = -5` with loss values between 1 and 2. - Slope gradually downward, reaching near-zero loss at `x_t = 0`. - Lines are less steep compared to blue lines and show minor variability in starting points. ### Key Observations 1. **AP Threshold Impact**: Lines for AP > 33.5 (blue) exhibit a significantly steeper decline in loss compared to AP < 33.5 (gray), suggesting a stronger sensitivity to `x_t` changes in higher-AP scenarios. 2. **Convergence at Origin**: All lines intersect near `x_t = 0`, indicating that this point minimizes loss regardless of AP value. 3. **Line Variability**: Blue lines show tighter clustering, while gray lines display slight dispersion in starting loss values. ### Interpretation The graph demonstrates that higher AP values (AP > 33.5) correlate with a more rapid reduction in loss as `x_t` increases toward 0. This implies that models with higher AP may achieve optimal performance (minimal loss) more efficiently when `x_t` is adjusted. The convergence at `x_t = 0` suggests this is the universal optimal point for loss minimization. However, the steeper slopes for higher AP could indicate that these models require more precise tuning of `x_t` to reach the minimum loss, potentially reflecting differences in model complexity or sensitivity to input variables. </details> Wefound that various γ and β settings gave good results. In Figure 7 we show results for RetinaNet-50-600 with FL ∗ for a wide set of parameters. The loss plots are color coded such that effective settings (models converged and with AP over 33.5) are shown in blue. We used α = . 25 in all experiments for simplicity. As can be seen, losses that reduce weights of well-classified examples ( x t > 0 ) are effective. More generally, we expect any loss function with similar properties as FL or FL ∗ to be equally effective. ## Appendix B: Derivatives For reference, derivates for CE, FL, and FL ∗ w.r.t. x are: $$\frac { d C E } { d x } = u ( n - 1 ) \cdot ( 9 )$$ $$\frac { d F L } { d x } = y ( 1 - p ) ^ { \prime } ( r p _ { t } l _ { t } ) + n - 1$$ $$\frac { d F L ^ { \prime } } { d x } = g ( x ^ { 2 } - 1 )$$ Plots for selected settings are shown in Figure 6. For all loss functions, the derivative tends to -1 or 0 for high-confidence predictions. However, unlike CE, for effective settings of both FL and FL ∗ , the derivative is small as soon as x t > 0 . ## References - [1] S. Bell, C. L. Zitnick, K. Bala, and R. Girshick. Insideoutside net: Detecting objects in context with skip pooling and recurrent neural networks. In CVPR , 2016. 6 - [2] S. R. Bulo, G. Neuhold, and P. Kontschieder. Loss maxpooling for semantic image segmentation. In CVPR , 2017. 3 - [3] J. Dai, Y . Li, K. He, and J. Sun. R-FCN: Object detection via region-based fully convolutional networks. In NIPS , 2016. 1 - [4] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR , 2005. 2 - [5] P. Doll´ ar, Z. Tu, P. Perona, and S. Belongie. Integral channel features. 2009. 2, 3 - [6] D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov. Scalable object detection using deep neural networks. In CVPR , 2014. 2 - [7] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes (VOC) Challenge. IJCV , 2010. 2 - [8] P. F. Felzenszwalb, R. B. Girshick, and D. McAllester. Cascade object detection with deformable part models. In CVPR , 2010. 2, 3 - [9] C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg. DSSD: Deconvolutional single shot detector. arXiv:1701.06659 , 2016. 1, 2, 8 - [10] R. Girshick. Fast R-CNN. In ICCV , 2015. 1, 2, 4, 6, 8 - [11] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , 2014. 1, 2, 5 - [12] T. Hastie, R. Tibshirani, and J. Friedman. The elements of statistical learning . Springer series in statistics Springer, Berlin, 2008. 3, 7 - [13] K. He, G. Gkioxari, P. Doll´ ar, and R. Girshick. Mask RCNN. In ICCV , 2017. 1, 2, 4 - [14] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In ECCV . 2014. 2 - [15] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , 2016. 2, 4, 5, 6, 8 - [16] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, and K. Murphy. Speed/accuracy trade-offs for modern convolutional object detectors. 2017. 2, 8 - [17] A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS , 2012. 2 - [18] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation , 1989. 2 - [19] T.-Y. Lin, P. Doll´ ar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In CVPR , 2017. 1, 2, 4, 5, 6, 8 - [20] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV , 2014. 1, 6 - [21] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and S. Reed. SSD: Single shot multibox detector. In ECCV , 2016. 1, 2, 3, 6, 7, 8 - [22] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR , 2015. 4 - [23] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In NIPS , 2015. 2, 4 - [24] P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Doll´ ar. Learning to refine object segments. In ECCV , 2016. 2 - [25] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In CVPR , 2016. 1, 2 - [26] J. Redmon and A. Farhadi. YOLO9000: Better, faster, stronger. In CVPR , 2017. 1, 2, 8 - [27] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS , 2015. 1, 2, 4, 5, 8 - [28] H. Rowley, S. Baluja, and T. Kanade. Human face detection in visual scenes. Technical Report CMU-CS-95-158R, Carnegie Mellon University, 1995. 2 - [29] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. In ICLR , 2014. 2 - [30] A. Shrivastava, A. Gupta, and R. Girshick. Training regionbased object detectors with online hard example mining. In CVPR , 2016. 2, 3, 6, 7 - [31] A. Shrivastava, R. Sukthankar, J. Malik, and A. Gupta. Beyond skip connections: Top-down modulation for object detection. arXiv:1612.06851 , 2016. 2, 8 - [32] K.-K. Sung and T. Poggio. Learning and Example Selection for Object and Pattern Detection. In MIT A.I. Memo No. 1521 , 1994. 2, 3 - [33] C. Szegedy, S. Ioffe, and V. Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv:1602.07261 , 2016. 8 - [34] J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W. Smeulders. Selective search for object recognition. IJCV , 2013. 2, 4 - [35] R. Vaillant, C. Monrocq, and Y. LeCun. Original approach for the localisation of objects in images. IEE Proc. on Vision, Image, and Signal Processing , 1994. 2 - [36] P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In CVPR , 2001. 2, 3 - [37] C. L. Zitnick and P. Doll´ ar. Edge boxes: Locating object proposals from edges. In ECCV , 2014. 2