2312.12878v1

# Rule-Extraction Methods From Feedforward Neural Networks: A Systematic Literature Review **Authors**: Sara El Mekkaoui, Équipe AMIPS, École Mohammadia d’Ingénieurs, Rabat, Morocco, Loubna Benabbou, Qc, Canada, Abdelaziz Berrado, Équipe AMIPS, École Mohammadia d’Ingénieurs, Rabat, Morocco ## Abstract Motivated by the interpretability question in ML models as a crucial element for the successful deployment of AI systems, this paper focuses on rule extraction as a means for neural networks interpretability. Through a systematic literature review, different approaches for extracting rules from feedforward neural networks, an important block in deep learning models, are identified and explored. The findings reveal a range of methods developed for over two decades, mostly suitable for shallow neural networks, with recent developments to meet deep learning models’ challenges. Rules offer a transparent and intuitive means of explaining neural networks, making this study a comprehensive introduction for researchers interested in the field. While the study specifically addresses feedforward networks with supervised learning and crisp rules, future work can extend to other network types, machine learning methods, and fuzzy rule extraction. Keywords XAI $·$ Neural Networks $·$ Interpretability $·$ Deep Learning $·$ Rule-based systems $·$ Black-box models ## 1 Introduction The growing deployment of Artificial Intelligence (AI) systems in sensitive domains and the emerging regulatory and ethical considerations surrounding AI have fostered the development of Machine Learning (ML) models’ interpretability. This area of research has gained significant attention in recent years [104], especially with the emergence of complex models in real-world applications. Feedforward neural networks play an essential role in Deep Learning (DL), serving as the fundamental building blocks for numerous advanced models. These networks are important because they can process and transform input data through multiple layers of interconnected neurons, enabling the extraction of useful representations. Their hierarchical structure allows for the construction of deep neural networks, which have demonstrated remarkable success in various domains, including computer vision, natural language processing, and speech recognition. While DL models have succeeded in numerous fields, the drive for improved accuracy has led to the development of increasingly complex architectures and techniques. The opacity of their internal mechanisms poses significant challenges in understanding their decision-making process, particularly in critical domains such as healthcare, finance, and safety-critical applications. Addressing the interpretability challenge in DL is a pressing concern, as it not only improves the understanding of the model but also enables fairness and ethical evaluation, enhancing control and robustness and discovering insightful knowledge. While interpretability in ML, especially in the context of neural networks, is often considered a relatively recent area of research, its origins can be traced back to the late 1980s with the advent of neural-symbolic computing [35]. Neural symbolic systems represent an interdisciplinary approach combining principles from neural networks and symbolic artificial intelligence. These systems aim to integrate the powerful learning capabilities of neural networks with the interpretability and reasoning abilities offered by symbolic reasoning. A fundamental aspect of neural-symbolic systems lies in extracting knowledge from neural networks in the form of rules, leading to the development of various rule extraction techniques from neural networks. Notably, these approaches have similarities with contemporary methods employed for DL explainability [101]. The significance of these techniques has even been discussed in major conferences, such as the NIPS and ICML, with a dedicated NIPS 96 Workshop, reflecting their relevance within the scientific community. Motivated by the significant contributions in rule extraction from neural networks, this paper contributes a systematic review of rule extraction methods from feedforward neural networks employed in supervised learning tasks. The study offers a comprehensive and extended taxonomy incorporating contemporary classification criteria for explainability and a novel classification according to different approaches. Furthermore, it addresses the transferability of existing rule extraction techniques to deep feedforward networks. The comprehensive nature of this survey renders it invaluable to researchers interested in this field, as it consolidates many studies within a single research. The contribution of this study can be summarised as follows: - A systematic review of papers on rule extraction methods from feedforward neural networks. - An extended taxonomy including some modern explainability techniques classification criteria and a categorization of the rule extraction approaches offering a comprehensive overview of available techniques. - A visual literature browser to improve information accessibility using SurVis [6] This paper is organized as follows: Section 2 presents the main concepts and definitions. Section 3 provides details of the review methodology and descriptive findings. Section 4 gives the main classification framework and details the dimension related to the rule extraction approaches categories. Section 5 discusses the challenges of using these methods for DL models, and Section 6 concludes this paper. ## 2 Background and main concepts ### 2.1 Feedforward Neural Networks DL models are nonlinear parametric ML models inspired by neuroscience [37]. Previously known as connectionism or artificial neural networks, the modern term DL represents today a general principle of learning multiple levels of composition. Deep feedforward networks are typical DL models called Multilayer Perceptron (MLP). In a supervised learning setting, the objective of a deep feedforward network is to map an input $x$ to an output $o$ . The output can be a real number in the case of a regression problem or a category in the case of a classification problem. Deep feedforward networks, hereafter referred to as networks, are composed of processing elements called neurons, nodes, or units organized as a network as depicted in Figure 1. The units are spread between multiple layers. The input layer $L_0$ receives the input features, followed by the first hidden layer $L_1$ . Each hidden layer, including the first, consists of a specific number of units. Each unit $u_ij$ from layer $L_j$ receives inputs from the previous layer $L_j-1$ , performs a linear combination of these inputs then computes an activation value. The last layer is the output layer that provides predictions. The most popular activation functions are sigmoid, rectifier linear unit, and hyperbolic tangent functions [70]. In a supervised learning setting, fitting a network refers to learning the parameter values, i.e., weights and biases, that result in a low error between the network predictions and the true outputs. Backpropagation gradient descent is the most used learning algorithm consisting of propagating the error backward through the network and updating its parameters weights to reduce the gradient [ Ref]. The error is measured using a loss function. The most popular loss functions are the cross entropy loss in classification problems and the mean squared error in regression problems. Effective development of DL models is a highly empirical process that requires searching for the best hyper-parameters such as the number of hidden layers, number of hidden units, learning rate, number of iterations, and activation function. Different architectures could be validated using a validation dataset to determine the best configuration [71]. <details> <summary>extracted/5307088/neural_network_represenation.jpg Details</summary>  ### Visual Description ## Diagram: Artificial Neural Network Architecture ### Overview The image is a technical diagram illustrating the architecture of a feedforward artificial neural network (ANN), specifically a multi-layer perceptron (MLP). It visually represents the flow of data from an input layer, through one or more hidden layers, to an output layer. The diagram uses a standard node-and-edge notation where circles represent neurons (nodes) and lines represent weighted connections between them. ### Components/Axes The diagram is organized into three primary vertical sections, labeled from left to right: 1. **Input Layer** (Leftmost section, light blue nodes): * Label: "Input Layer" * Nodes: Labeled `x₁`, `x₂`, `x₃`, followed by an ellipsis (`...`), and ending with `xₚ`. This indicates an input vector of `p` dimensions. 2. **Hidden Layers** (Central section, light green nodes): * Main Label: "Hidden Layers" (centered above the section). * The section is subdivided into multiple layers, each with its own label: * Layer 1: Labeled `L₁`. Contains nodes labeled `h₁₁`, `h₁₂`, `h₁₃`, `h₁₄`, followed by an ellipsis (`...`), and ending with `h₁ₙ₁`. This indicates the first hidden layer has `n₁` neurons. * Layer 2: Labeled `L₂`. Contains nodes labeled `h₂₁`, `h₂₂`, `h₂₃`, `h₂₄`, followed by an ellipsis (`...`), and ending with `h₂ₙ₂`. This indicates the second hidden layer has `n₂` neurons. * Intermediate Layers: An ellipsis (`...`) is placed between Layer 2 and the final hidden layer, indicating the potential for additional hidden layers. * Final Hidden Layer (m-th layer): Labeled `Lₘ`. Contains nodes labeled `hₘ₁`, `hₘ₂`, `hₘ₃`, `hₘ₄`, followed by an ellipsis (`...`), and ending with `hₘₙₘ`. This indicates the m-th hidden layer has `nₘ` neurons. 3. **Output Layer** (Rightmost section, light yellow nodes): * Label: "Output Layer" * Nodes: Labeled `o₁` and `o₂`. This indicates the network produces a 2-dimensional output vector. **Connections (Edges):** * **Input to Hidden (L₁):** Blue lines connect every input node (`x₁` through `xₚ`) to every node in the first hidden layer (`h₁₁` through `h₁ₙ₁`). This represents a fully connected layer. * **Hidden to Hidden:** Green lines connect every node in one hidden layer to every node in the subsequent hidden layer (e.g., all nodes in `L₁` connect to all nodes in `L₂`). This pattern continues through all hidden layers. * **Hidden (Lₘ) to Output:** Green lines connect every node in the final hidden layer (`hₘ₁` through `hₘₙₘ`) to every output node (`o₁` and `o₂`). ### Detailed Analysis * **Network Topology:** The diagram depicts a fully connected, feedforward neural network. The flow of information is strictly from left (input) to right (output), with no cycles or backward connections. * **Scalability Notation:** The use of ellipses (`...`) in the input layer, between hidden layers, and within each hidden layer is a critical notational element. It explicitly shows that the network is generalized: * The input dimension `p` can be any positive integer. * The number of hidden layers `m` can be any positive integer (including `m=1` for a single hidden layer). * The number of neurons `n₁`, `n₂`, ..., `nₘ` in each hidden layer can vary independently. * **Node Labeling Convention:** The labeling follows a systematic pattern: * Input nodes: `xᵢ` where `i` is the feature index (1 to `p`). * Hidden nodes: `hₗₖ` where `l` is the layer index (1 to `m`) and `k` is the neuron index within that layer (1 to `nₗ`). * Output nodes: `oⱼ` where `j` is the output index (1 to 2 in this case). ### Key Observations 1. **Fully Connected Layers:** Every possible connection between adjacent layers is drawn, emphasizing that this is a dense network where each neuron receives input from all neurons in the previous layer. 2. **Generalized Architecture:** The diagram is not a specific instance but a template for a class of neural networks. The variables (`p`, `m`, `n₁`, `n₂`, ..., `nₘ`) allow it to represent networks of vastly different sizes and complexities. 3. **Color Coding:** A consistent color scheme is used to differentiate layer types: light blue for input, light green for hidden, and light yellow for output. The connection lines also follow a scheme: blue for input-to-first-hidden, and green for all subsequent hidden-to-hidden and hidden-to-output connections. 4. **Output Dimension:** The output layer has exactly two nodes (`o₁`, `o₂`), suggesting this specific template is for a network designed for a task with two outputs (e.g., binary classification with two output neurons, or regression with two target variables). ### Interpretation This diagram serves as a foundational schematic for understanding deep learning models. It visually communicates the core concepts of a multi-layer perceptron: * **Hierarchical Feature Learning:** The multiple hidden layers (`L₁` to `Lₘ`) suggest the network's ability to learn increasingly abstract representations of the input data. Early layers (`L₁`) might learn simple features, while deeper layers (`Lₘ`) combine them into complex patterns. * **Universal Approximation:** The structure, with at least one hidden layer and non-linear activation functions (implied but not shown), represents a model that, in theory, can approximate any continuous function given sufficient neurons (`nₗ`). The ellipses highlight that increasing `m` and `nₗ` increases the model's capacity. * **Parameterization:** The diagram implicitly represents a large set of learnable parameters: the weights associated with every drawn connection line and biases for every hidden and output neuron (biases are not visualized). The total number of parameters is determined by the values of `p`, `m`, and the `nₗ`'s. * **Task Specification:** The fixed output layer with two nodes indicates the network's final purpose is to produce a two-value output, framing the type of problem it is designed to solve. The input layer's variable size `p` shows it can accept feature vectors of different lengths. In essence, this image is a canonical representation of a deep neural network's "wiring diagram," abstracting away the specific mathematical operations (like weighted sums and activation functions) to focus on the architecture's topology and scalability. </details> Figure 1: A representation of a deep feedforward neural network ### 2.2 Rule extraction from feedforward neural networks #### 2.2.1 Rule extraction problem The problem of rule extraction from feedforward neural networks consists of generating comprehensible hypotheses in the form of symbolic logical representations from a trained network. For instance, in the case of Boolean inputs $x_i$ used for a classification task with two classes $o_1$ and $o_2$ , a rule extraction method providing propositional rules would generate rules in the format of: IF ( $x_1\wedge¬ x_2\wedge x_5$ ) THEN $o_1$ Most existing rule extraction methods focus on networks with discrete inputs and outputs. Inputs are considered Boolean or nominal valued, and outputs as classes. In cases where inputs or outputs are continuous, a discretization step is typically required. Various approaches have been proposed for rule extraction from feedforward neural networks, and previous literature reviews in this field have provided classification schemes to categorize these methods. The diversity of these approaches illustrates the significant research effort invested in this area. #### 2.2.2 Overview of previous surveys Rule extraction from neural networks has been an area of research since the late 80s. Table 1 defines the main concepts discussed in this section, and Table 2 summarises the existing surveys on rule extraction methods. The ADT taxonomy, referring to the authors’ initials, for rule extraction methods was proposed in 1995 [2] and refined in 1998 [99]. It follows five dimensions: - The expressive power of the extracted rules: we refer to this dimension as rules form to indicate the type of the rules extracted from the network, such as Boolean, fuzzy, and first-order logic rules. - Translucency into the network: this dimension includes three classes of extraction methods as pedagogical, decompositional, and eclectic. They characterize the infiltration of the rule extraction method into the internals of the network. This dimension is similar to the modern model-related dimension of agnostic or specific methods. - Portability of the technique across neural network architectures and training regimes: this dimension is about the requirements of the extraction methods for a particular training regime to simplify the extraction task. It is similar to the modern interpretability techniques characteristic of being intrinsic or post-hoc. - The quality of the extracted rules: including accuracy, fidelity, consistency, and comprehensibility metrics to assess the performance of the extracted rules. - The complexity of the rule extraction algorithm: characterizing the efficiency of the rule extraction process. Table 1: Definitions of related concepts to rule extraction methods from feedforward neural networks | Dimension | Concept | Definitions | | --- | --- | --- | | Rules form | Propositional rules | The conditional part contains conjunctions, disjunctions, and negations. It is expressed as a combination of Boolean conditions. Examples of propositional rules in the case of discrete features and a classification task could be "if $X=1$ and $Y=0$ then class 1". For continuous variables, values are restricted in forms like "if $X∈[a,b],X≤ a$ with $a,b∈ℝ$ ". | | MofN rules | They are Boolean expressions specified by the integer threshold $m$ and a set of $n$ Boolean conditions. They can take the form "if at least/exactly/at most $m$ of the $n$ conditions are satisfied then class 1". They can be converted into propositional rules. | | | Obliques rules | They express nonparallel decision boundaries with input axes as linear inequalities "if $aX_1+bX_2<c$ and $dX_4+eX_3>f$ then class 0, with $a,b,c,d,e,f∈ℝ$ ". | | | Fuzzy rules | They are expressed in linguistic concepts, e.g., "if X is high and Y is low then class 1". | | | Translucency into the network | Pedagogical | The methods target the function learned by the network and treat the network as a black box. The extracted rules describe the behavior of the network output in terms of inputs without using the network internals. | | Decompositional | The methods target the network internals by extracting rules at each hidden and output unit and then aggregating them into global rules linking inputs to outputs. | | | Eclectic | The methods are hybrid, examining the network units but extracting rules describing the relationship between the inputs and the outputs directly. | | | Portability | Constrained | The extraction methods require a modification of the network or a special training regime. | | Unconstrained | The extraction methods do not impose any changes to the network structure or training regime. | | | Rules quality | Accuracy / Correctness | Measuring the extracted rules predictions accuracy with regards to the true targets. | | Fidelity | Measuring the prediction accuracy with regards to the network prediction. This measures how well the extracted rules are in mimicking the network behavior. | | | Consistency | Measuring how consistent the extracted rules predictions are under different network training sessions. | | | Comprehensibility | Measuring the ability to understand the extracted rules using their size, i.e., number of rules and antecedents per rule or average rule length. | | | Completeness | Characterizing the extracted rules power in terms of the ratio of input examples covered by the extracted rule over total input examples. | | | Complexity | The complexity of the extraction method, which for instance, could be exponential or polynomial. | | | Application | Model-agnostic | The extraction method can be applied to any ML technique. | | Model-specific | The extraction method is specific to feedforward neural networks. | | | Design | Intrinsic | The extraction method requires a simplification of the network complexity using techniques such as regularization. | | Post-hoc | The extraction method intervenes after the network training. | | | Scope | Global | The extraction method offers an explanation of the entire network behavior. | | Local | The extraction methods give an explanation of the decision related to a particular input example. | | The proposed taxonomy has various dimensions indicating the diversity of the proposed extraction methods during the 90s. Table 2 shows the existing surveys on rule extraction methods from neural networks. Surveys following [2, 99] mainly classify papers using the translucency dimension as pedagogical, decompositional, or eclectic. However, the existing classification scheme does not offer a clear overview of the nature of methods used to extract rules from networks. Unlike recent reviews, this study performs an extensive review and classifies the existing literature following a comprehensive dimension describing the main approaches to rule extraction. The following section presents the review methodology and main descriptive findings. Table 2: Surveys of rule extraction methods from feedforward neural networks | [2] | Survey and critique of techniques for extracting rules from trained artificial neural networks | ADT classification of rule extraction methods. | | --- | --- | --- | | [22] | Extracting comprehensible models from trained neural networks | Classification into Global (pedagogical) and local (decompositional and eclectic). | | [99] | The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks | Extension of the ADT classification. | | [46] | Using Rule Extraction to Improve the Comprehensibility of Predictive Models | Classification into dependent and independent methods from the network. | | [3] | Rule extraction from neural networks — A comparative study | Classification into pedagogical, decompositional, and eclectic. | | [40] | Rule Extraction Algorithm for Deep Neural Networks: A Review | Classification into pedagogical, decompositional, and eclectic. | | [42] | Extract interpretability-accuracy balanced rules from artificial neural networks: A review | Classification into pedagogical, decompositional, and eclectic. | ## 3 Literature Review ### 3.1 Review methodology This study conducted a systematic literature review based on the guidelines suggested by [51]. The methodology includes three steps of planning, conducting, and reporting the review. The first step is planning the review by identifying the research area and developing a research protocol to guide the review process. The review protocol involves the following steps: definition of the research area, research questions, search procedure, inclusion and exclusion criteria, quality assessment, data extraction, and data synthesis. This review’s research scope is "rule extraction methods from feedforward neural networks for supervised learning", with a focus on crisp rules. The review covers techniques for extracting rules from both shallow and deep feedforward neural networks. This study has four goals: (i) identify and introduce a comprehensive and exhaustive classification framework, (ii) identify the trends, (iii) describe the different approaches to rule extraction methods, and (iv) identify which approaches are suitable for DL models. To achieve the objective of the review, the following research questions were considered: - RQ1 What classification dimensions could be considered for rule extraction methods? - RQ2 What are the research trends in this field? - RQ3 What are the main approaches to rule extraction from feedforward neural networks? - RQ4 What approaches are applicable in the case of DL models? The inclusion/exclusion criteria are applied to ensure that the selected studies are within the review scope. The main selection criteria are (i) presenting a method for rule extraction from feedforward networks and (ii) being written in the English language. The exclusion criteria include (i) applying rule extraction methods and (ii) studies on rule extraction methods from other ML techniques. The search process was conducted by examining papers from major academic databases and the previous surveys listed in Table 2. Search terms related to “rule extraction” and “neural networks” and their common variations were used to select papers. As we are interested in the whole evolution of this field, no restrictions were imposed on the publication year. The extracted papers were scanned based on their titles, abstracts, and keywords to determine if they met the selection criteria. Some papers also required scanning the introduction and/or the conclusion sections. The obtained list was completed with papers from the surveys. The selected papers’ “review” or “related work” sections were scanned to complete the list of relevant papers. A detailed scan of the obtained papers was conducted in order to assess their quality. Papers with clear methodology descriptions and experiments to showcase the results of the proposed method were chosen. In order to identify classification dimensions and main approaches, a detailed analysis of each study was conducted, and a synthesis describing the proposed method was generated. An initial extraction form was developed to capture relevant information, including publication year, authors, publication venue, translucency type (pedagogical, decompositional, or eclectic), and method portability. The extraction form was refined during the data synthesis phase to ensure comprehensive data collection and analysis. The final list includes 89 papers. ### 3.2 Taxonomy Figure 2 illustrates the taxonomy for rule-extraction methods from feedforward neural networks. The taxonomy dimensions are defined in Table 1. The classification dimensions can be related to the extraction method or the extracted rules. The translucency dimension is specific to rule extraction methods from feedforward neural networks, describing how the extraction is performed. Pedagogical methods extract rules that describe the behavior of output units in terms of input units. Decompositional methods extract rules at each hidden and output unit, with the concept associated with each unit being the rule consequent and certain subsets of units feeding into this unit representing the antecedents of the rule. The process of rule extraction involves finding sufficient conditions for each consequent. Decompositional rule extraction methods assume that hidden and output unit activation values are discrete after training. The extracted rules are then aggregated to form rules that describe the relationship between inputs and outputs. Eclectic methods lie between the pedagogical and decompositional types, as they can analyze the internals of the network but extract global rules between inputs and outputs. The portability dimension, proposed by Andrews et al. [2], characterizes how dependent an extraction method is on the network. Portable methods do not make any assumptions about the network and do not use any internal information, and can be used with any network. The portability dimension can be seen as a special case of the modern interpretability dimension of application, separating interpretability tools into model-agnostic or model-specific tools. The application, complexity, design, and scope dimensions can be seen as general classification criteria for any interpretability tool. The complexity of the rule extraction method can vary depending on the algorithm used, the number of rules being extracted, and their complexity. The complexity of the rule extraction methods may also depend on the complexity of the network architecture or the high dimensionality of the data being processed. The design of the extraction method can be intrinsic or post-hoc. Intrinsic methods aim to simplify the network complexity to facilitate rule extraction, and regularization and pruning techniques can be used to make the network more interpretable and help in efficiently extracting rules. Post-hoc techniques are applied after the network is completely trained and do not impose any training requirements. The scope can be global when the extracted rules offer a general explanation of the network or local when the explanations are specific to certain samples. This review introduces a dimension related to the employed approach to categorize rule extraction methods from a technical perspective. In Section 4, we will describe in detail the various approaches to building rules, including explore & test, induced models, attribution, optimization, and hybrid methods. The extracted rules can take different forms, and their quality can be assessed using different metrics, as described in detail in Table 1. <details> <summary>extracted/5307088/rule_ext_taxonomy.jpg Details</summary>  ### Visual Description ## Diagram: Hierarchical Flowchart of Rule Extraction Methods from Feedforward Neural Networks ### Overview The image is a hierarchical flowchart diagram illustrating the taxonomy and classification of "Rule Extraction Methods from Feedforward Neural Networks." It organizes the field into two primary branches: the characteristics of the extracted rules themselves and the methods used for extraction. The diagram uses a top-down tree structure with rectangular nodes connected by directional arrows. ### Components/Axes * **Main Title (Top-Center):** "Rule Extraction Methods from Feedforward Neural Networks" in a dark grey box. * **Primary Branches (Second Level):** Two dark grey boxes stemming from the main title: * Left: "Extracted Rules" * Right: "Extraction Methods" * **Secondary Categories (Third Level):** Lighter grey boxes connected to the primary branches. * **Leaf Nodes (Final Level):** White boxes with black text, representing specific types, criteria, or sub-methods. * **Flow Direction:** The flow is strictly top-down and hierarchical. Arrows point from parent categories to their constituent sub-categories. ### Detailed Analysis The diagram is segmented into two main regions for analysis: **Region 1: Left Branch - "Extracted Rules"** This branch classifies the output of the extraction process. * **"Extracted Rules"** splits into two sub-categories: 1. **"Rules form"** (Light grey box, center-left). This node branches into four specific rule types: * "Propositional rules" * "M-of-N rules" * "Fuzzy rules" * "Oblique rules" 2. **"Rules quality"** (Light grey box, below "Rules form"). This node branches into five evaluation criteria: * "Accuracy" * "Fidelity" * "Consistency" * "Comprehensibility" * "Completeness" **Region 2: Right Branch - "Extraction Methods"** This branch classifies the techniques used to derive rules from neural networks. * **"Extraction Methods"** (Dark grey box, top-right) is the parent node for seven classification dimensions, each represented by a light grey box: 1. **"Translucency"**: Branches into "Pedagogical", "Decompositional", "Eclectic". 2. **"Portability"**: Branches into "Constrained", "Unconstrained". 3. **"Complexity"**: A terminal node with no further sub-branches shown. 4. **"Application"**: Branches into "Agnostic", "Specific". 5. **"Design"**: Branches into "Intrinsic", "Post-hoc". 6. **"Scope"**: Branches into "Global", "Local". 7. **"Approach"**: Branches into five methodological approaches: "Explore & test", "Induced models", "Attribution", "Optimization", "Hybrid". ### Key Observations * **Structural Symmetry:** The diagram presents a balanced taxonomy, with the "Extracted Rules" branch focusing on the *what* (outputs and their qualities) and the "Extraction Methods" branch focusing on the *how* (techniques and their attributes). * **Categorization Depth:** The "Extraction Methods" branch is more extensively subdivided, indicating a richer and more varied landscape of techniques compared to the forms and quality metrics of the rules themselves. * **Terminal Nodes:** Some categories, like "Complexity" under Extraction Methods, are presented as standalone concepts without further subdivision in this diagram. * **Visual Hierarchy:** Color coding (dark grey for primary categories, light grey for secondary, white for tertiary) and spatial positioning (left vs. right main branch) are used effectively to denote hierarchy and separate conceptual domains. ### Interpretation This diagram serves as a conceptual map for the field of rule extraction from neural networks. It demonstrates that the discipline is not monolithic but is organized along multiple, orthogonal axes of classification. * **Relationship Between Elements:** The structure implies that any specific rule extraction technique can be described by selecting one option from each of the seven dimensions under "Extraction Methods" (e.g., a method could be Pedagogical, Unconstrained, Agnostic, Post-hoc, Global, and based on an Optimization approach). The quality of the rules it produces ("Extracted Rules") can then be evaluated against the five criteria listed. * **Underlying Message:** The taxonomy highlights the trade-offs and design choices inherent in the field. For instance, the "Translucency" dimension (Pedagogical vs. Decompositional) represents a fundamental choice between treating the neural network as a black box or opening it up. The "Design" dimension (Intrinsic vs. Post-hoc) distinguishes between methods built into the network's training and those applied afterward. * **Utility:** This framework is essential for researchers to position their work, for practitioners to select appropriate methods for their needs (prioritizing, e.g., "Comprehensibility" or "Fidelity"), and for students to gain a structured overview of a complex topic. It systematizes what could otherwise be a disjointed collection of algorithms and metrics. </details> Figure 2: Taxonomy of rule extraction methods from feedforward neural networks ### 3.3 Descriptive findings Figure 3 illustrates the distribution of the selected studies over time. The first rule extraction method was introduced in 1988, followed by a significant number of papers during the 1990s. However, the research field declined thereafter, possibly due to reduced interest in neural networks. However, there has been renewed interest in the field in recent years, as the interpretability of neural networks has resurfaced recently as an important area of research. Out of the 89 papers, some have proposed multiple methods within the same paper, while others have proposed the same method in different papers. There are 81 distinct methods, of which 78 are global and 3 are local. <details> <summary>extracted/5307088/studies_chrono.jpg Details</summary>  ### Visual Description ## Vertical Bar Chart: Annual Frequency (1988-2022) ### Overview This is a vertical bar chart displaying a numerical count or frequency for each year from 1988 to 2022. The chart shows significant variation over time, with notable peaks in the early 1990s and late 2010s, and several years with zero or near-zero values. ### Components/Axes * **Chart Type:** Vertical Bar Chart. * **X-Axis (Horizontal):** Represents years. The axis is labeled with individual years from **1988** to **2022**, inclusive. The labels are rotated approximately 45 degrees for readability. * **Y-Axis (Vertical):** Represents a numerical scale, likely a count or frequency. The axis is labeled with integers from **0** to **8**, with major gridlines at each integer value. * **Legend:** There is no legend present. The chart contains a single data series represented by blue bars. * **Data Series:** A single series of blue vertical bars, one for each year on the x-axis. The height of each bar corresponds to the value on the y-axis. ### Detailed Analysis **Trend Verification:** The data series shows a highly variable pattern. It begins at a low value, rises to a major peak in 1993, declines with fluctuations, hits a prolonged low period from 2011-2015, and then rises again to a second major peak in 2018 before stabilizing at a moderate level. **Year-by-Year Data Extraction (Approximate Values):** * **1988:** 2 * **1989:** 0 (No bar visible) * **1990:** 3 * **1991:** 4 * **1992:** 0 (No bar visible) * **1993:** 7 (Highest peak in the first half of the chart) * **1994:** 6 * **1995:** 3 * **1996:** 5 * **1997:** 3 * **1998:** 3 * **1999:** 4 * **2000:** 5 * **2001:** 4 * **2002:** 2 * **2003:** 0 (No bar visible) * **2004:** 3 * **2005:** 0 (No bar visible) * **2006:** 4 * **2007:** 0 (No bar visible) * **2008:** 2 * **2009:** 3 * **2010:** 1 * **2011:** 0 (No bar visible) * **2012:** 2 * **2013:** 0 (No bar visible) * **2014:** 0 (No bar visible) * **2015:** 0 (No bar visible) * **2016:** 5 * **2017:** 3 * **2018:** 6 (Second highest peak, tied with 1994) * **2019:** 3 * **2020:** 3 * **2021:** 3 * **2022:** 3 ### Key Observations 1. **Peaks:** The highest recorded value is **7** in **1993**. The second-highest value is **6**, occurring in both **1994** and **2018**. 2. **Troughs/Zero Values:** There are multiple years with a value of **0** (no visible bar): **1989, 1992, 2003, 2005, 2007, 2011, 2013, 2014, 2015**. 3. **Periods of Activity:** * **High Activity (1993-2001):** A cluster of relatively high values, mostly between 3 and 7. * **Low Activity (2002-2015):** Characterized by sporadic, lower values (1-4) interspersed with many zero-value years. The period from **2011 to 2015** is notably quiet, with four out of five years at zero. * **Resurgence (2016-2022):** A clear return to consistent, moderate activity, with values between 3 and 6, and no zero-value years. 4. **Recent Stability:** The last four years shown (**2019-2022**) all have an identical value of **3**, indicating a recent plateau. ### Interpretation The chart demonstrates a cyclical or episodic pattern in the measured phenomenon. The data suggests two major periods of heightened activity: one in the early-to-mid 1990s and another in the late 2010s. The long lull between approximately 2002 and 2015, especially the near-complete absence from 2011-2015, is a significant anomaly that would require contextual explanation (e.g., a change in methodology, a real-world event suppressing the phenomenon, or a data collection gap). The relationship between the elements is purely temporal; the chart shows how a single variable changes over a 35-year span. The recent stability (2019-2022) at a value of 3 could indicate a new baseline or the beginning of another cycle. Without a title or y-axis label, the specific subject (e.g., number of patents, frequency of a natural event, count of product recalls) is unknown, but the visual pattern itself is clear and provides a strong foundation for further investigation into the causes of the peaks, troughs, and dormant periods. </details> Figure 3: Evolution of studies on rule extraction methods from feedforward neural networks expressed by the number of studies per year Figure 4 depicts the distribution of 81 methods based on the dimensions of "Translucency", "Application", and "Design". It is worth noting that Pedagogical approaches are inherently agnostic and post-hoc, as they do not rely on any assumptions about the network or training regime and do not utilize any information about the network internals. However, when a method requires access to the internal workings of the network, it may fall into the categories of eclectic or decompositional, and in both cases, these methods are specifically designed for neural networks. Eclectic and decompositional methods can be considered intrinsic if they directly affect the network internals or post-hoc if they operate after the training. <details> <summary>extracted/5307088/descriptive_results.jpg Details</summary>  ### Visual Description ## Bar Chart: Counts by Methodological Category and Approach ### Overview This is a vertical bar chart displaying numerical counts for five distinct categories. The categories are grouped into three broader methodological frameworks, as indicated by the labels on the x-axis. Each bar is annotated with its exact numerical value at the top. ### Components/Axes * **X-Axis (Categorical):** The axis is divided into three main sections, each containing one or two sub-categories. The labels are arranged in a hierarchical, multi-line format. * **Section 1 (Left):** "Specific Decompositional" * Sub-category 1: "Intrinsic" * Sub-category 2: "Post-hoc" * **Section 2 (Center):** "Specific Eclectic" * Sub-category 1: "Intrinsic" * Sub-category 2: "Post-hoc" * **Section 3 (Right):** "Agnostic Pedagogical" * Sub-category 1: "Post-hoc" * **Y-Axis (Numerical):** The vertical axis represents a count or frequency. While the axis line and scale markers are not visible, the numerical values are explicitly labeled on top of each bar, providing precise data points. The scale appears to range from 0 to at least 22. * **Legend:** There is no separate legend. The categories are defined solely by the x-axis labels. * **Data Series:** There is a single data series represented by blue bars. All bars are the same color. ### Detailed Analysis The chart presents the following data points, listed from left to right: 1. **Bar 1 (Far Left):** * **Category:** Intrinsic (under Specific Decompositional) * **Value:** 19 * **Visual Trend:** This is the second-tallest bar in the chart. 2. **Bar 2:** * **Category:** Post-hoc (under Specific Decompositional) * **Value:** 15 * **Visual Trend:** This bar is shorter than the "Intrinsic" bar within the same "Specific Decompositional" group. 3. **Bar 3 (Center):** * **Category:** Intrinsic (under Specific Eclectic) * **Value:** 10 * **Visual Trend:** This is the shortest bar in the entire chart. 4. **Bar 4:** * **Category:** Post-hoc (under Specific Eclectic) * **Value:** 15 * **Visual Trend:** This bar is taller than the "Intrinsic" bar within the same "Specific Eclectic" group and is equal in height to the "Post-hoc" bar in the "Specific Decompositional" group. 5. **Bar 5 (Far Right):** * **Category:** Post-hoc (under Agnostic Pedagogical) * **Value:** 22 * **Visual Trend:** This is the tallest bar in the chart. ### Key Observations * **Highest Value:** The "Post-hoc" approach within the "Agnostic Pedagogical" framework has the highest count (22). * **Lowest Value:** The "Intrinsic" approach within the "Specific Eclectic" framework has the lowest count (10). * **Intra-Group Comparison:** * In the **"Specific Decompositional"** group, the "Intrinsic" approach (19) has a higher count than the "Post-hoc" approach (15). * In the **"Specific Eclectic"** group, the pattern reverses: the "Post-hoc" approach (15) has a higher count than the "Intrinsic" approach (10). * **Inter-Group Comparison:** The "Post-hoc" approach appears in three different frameworks with varying counts: 15 (Specific Decompositional), 15 (Specific Eclectic), and 22 (Agnostic Pedagogical). ### Interpretation The data suggests a comparison of the frequency or application of different methodological approaches ("Intrinsic" vs. "Post-hoc") across three broader frameworks ("Specific Decompositional," "Specific Eclectic," and "Agnostic Pedagogical"). * **Framework Dominance:** The "Agnostic Pedagogical" framework, represented by a single "Post-hoc" category, shows the highest overall count, indicating it may be the most prevalent or frequently cited method in the context this data represents. * **Approach Preference:** There is no universal preference for "Intrinsic" or "Post-hoc" approaches. The preference is contingent on the broader methodological framework. "Intrinsic" is favored within "Specific Decompositional" methods, while "Post-hoc" is favored within "Specific Eclectic" methods and is the sole approach shown for "Agnostic Pedagogical" methods. * **Notable Anomaly:** The "Specific Eclectic" framework shows the largest disparity between its two approaches (10 vs. 15), suggesting a stronger leaning towards post-hoc methods within that eclectic context compared to the decompositional context. * **Potential Implication:** If this chart represents, for example, the number of research studies or models using these approaches, it implies that post-hoc methods are more commonly applied in pedagogical and eclectic settings, while intrinsic methods are more common in decompositional settings. The high count for "Post-hoc Agnostic Pedagogical" could signal a trend towards flexible, outcome-oriented methods in educational or general-purpose applications. </details> Figure 4: Distribution of the rule extraction methods from feedforward neural networks following the dimensions "Translucency", "Application", and "Design" ## 4 Rule extraction approaches The great majority of rule extraction techniques for feedforward neural networks can be regarded as proxy methods, which are interpretability strategies employed to clarify the behaviour of black-box models. A surrogate transparent model is developed through these approaches by training a simpler model, such as linear regression or decision trees, using the same input features as the black-box model and its predictions as targets. In the context of rule extraction, the mentioned proxy approach is pedagogical. However, it can also be decompositional if the proxy is fitted at the hidden and output units level. The surrogate model acts as a substitute or approximation of the unit activations, facilitating comprehension of its predictions. In the context of proxies, rules may be extracted by performing a search procedure through the space of rules and testing them against the network outputs or by utilizing induced models like decision trees. Additionally, we discuss other methods of extracting rules, including attribution methods based on feature importance and optimization methods involving defining the input space that maximizes a specific output class. Finally, some of the proposed methods combine different techniques and are considered as hybrid. ### 4.1 Explore & test Early work on neural-symbolic computing has motivated the extraction of rules from neural networks using the explore & test approach. It follows a typical rule learning process by exploring the space of possible rules and testing them against the network outputs. Table 3 summarises the explore & test methods. They all provide global explanations. Typically, learning rules can be formulated as a search problem [33] defined by (i) an appropriate search space, (ii) a search strategy for searching for rules through this space, and (iii) a quality function to evaluate the rules: - Search space: enumerating the whole space of possible rules is often infeasible. As exploring the rules’ space might be intractable, organizing and structuring the search space is necessary. Hence, rules extraction from neural networks is constructed by searching for the conjunction of features that is most predictive for a network response. Exhaustive, heuristic, or stochastic algorithms perform the search. - Search strategy: can be organized following the general-to-specific, specific–to –general, or bi-directional approaches. The general-to-specific rule search strategy is a top-down approach starting from the most general rule and further specializing it. The specific-to-general rule search or the growing strategy is a bottom-up approach that looks for specific rules, for instance, an input example, before finding more general rules. The bi-directional approach combines both of the previous approaches. - Rules evaluation: the quality of rules can be evaluated using different measures such as precision and information gain. The basic principle underlying many quality measures are consistency and coverage. A widely adopted approach in rule extraction entails the utilization of subset-type algorithms [100] to extract rules in a decompositional manner. This methodology aims to derive rules at both the hidden and output unit levels, employing discrete activation values to simplify the extraction process. Subset-type algorithms typically involve three main steps: - Exploration of unit input combinations: The algorithm searches for different combinations of inputs whose weighted sum activates the unit for each network unit. - Rule generation: Upon identifying each combination of inputs that activates a unit, the algorithm generates a rule wherein the input to the specific combination of links defines the premises. All premises of a rule are conjuncted. - Rule merging: The extracted rules are subsequently merged to establish direct linkages between inputs and outputs within the framework of decompositional methods. This merging process yields rules that directly associate input conditions with the corresponding output decisions. A breadth-first subset algorithm is a simple approach to searching for subsets of incoming weights that activate a unit. It starts with sets containing one link, verifies if they activate the unit, and rewrites valid sets as rules. Then, it increases the size of the subset until all subsets have been explored. Finally, it removes redundant rules. However, the major problem with these methods lies in the search complexity. For instance, if a network unit has $n$ inputs with binary values, there could be as many as $2^n$ distinct input sets. Another challenge is how to choose an appropriate manner for activation values discretization. Therefore, some methods make assumptions to narrow the search of the rules space. This includes considering the hidden and output units as maximally active or inactive by choosing hard-limit activation, approximating continuous activation functions by a threshold function, or changing the activation function steepness to act on the slope. This assumption is used to reduce the search by considering the network links to carry signals equal to the weights or no signal at all. Various heuristics have been employed to simplify the search process. This includes limiting the search depth by setting a maximum number of rule antecedents or considering specific combinations of rule antecedents that occur in the dataset. Some studies have also proposed heuristic search methods to find the best rules, such as CN2 [20], or reinforcement learning [73]. Stochastic search techniques, guided by Genetic Algorithms, have been used to discover effective rules [47, 59, 107, 106]. Alternative approaches involve searching for rules using association rule-based algorithms, with the Apriori algorithm being prominently used [57, 88, 86, 58, 84, 43, 44, 49]. Table 3: Summary of the explore & test methods for rule extraction from feedforward neural networks | Translucency | Design | Application | Reference | Tool name | Technique | | --- | --- | --- | --- | --- | --- | | Pedagogical | Post–hoc | A | [74] | – | Heuristic search | | A | [75] | RN | Sochastic search | | | | A | [25] | – | Stochastic search | | | | A | [109] | STARE | Heurostic search | | | | A | [63] | – | heuristic search | | | | A | [47] | G-REX | Stochastic search | | | | A | [56] | CaST | heuristic search | | | | A | [107] | NNGR | Stochastic search | | | | A | [73] | Anchors | Heuristic search | | | | A | [106] | MNNGR | Stochastic search | | | | Decompositional | Post–hoc | S | [34] | MACIE | Subset–type | | S | [31, 32] | KT–algorithm | Subset–type | | | | S | [100] | Subset | Subset–type | | | | S | [82] | – | Subset–type | | | | S | [41] | LAP | Subset–type | | | | S | [94] | Partal–RE | Subset–type | | | | S | [53] | COMBO | Subset–type | | | | S | [50] | OAS | Subset–type | | | | S | [26] | – | Subset–type | | | | Intrinsic | S | [60, 61] | – | Subset–type | | | S | [100, 23, 24] | MofN | Subset–type | | | | S | [10] | GDS | Subset–type | | | | S | [1] | – | Subset–type | | | | S | [57, 88, 86] | NeuroRule | Association rules | | | | S | [58, 84] | RX | Association rules | | | | S | [27] | RULE–OUT | Subset–type | | | | S | [29] | – | Subset–type | | | | S | [43] | Modified RX | Association rules | | | | S | [44] | – | Association rules | | | | S | [54] | OAS | Subset–type | | | | Eclectic | Post–hoc | S | [96, 97, 68] | VIA | Heuristic and stochastic search | | S | [26] | – | Exhaustive search | | | | S | [25] | – | Stochastic search | | | | Intrinsic | S | [28] | C–MLP2LN | Heuristic search | | | S | [59] | – | Stochastic search | | | | S | [49] | – | Association rules | | | The process of validating the extracted rules includes two main approaches within the context of explore & test methods. The first approach involves assessing whether the extracted rules effectively activate the network units. The second approach, proposed by Thrun [96], is known as the Validity-Interval-Analysis (VIA) algorithm. This method ensures the consistency of the generated rules by propagating the rule constraints throughout the network. It employs linear programming techniques to determine the extent to which the input constraints propagate consistently through the activations of the network. The explore & test methods, which involve extracting rules from the network after training, can pose challenges, particularly for large networks. Intrinsic methods are thus employed to mitigate these challenges by simplifying the network structure, making rule extraction more feasible. One approach to achieving this is by imposing regularization constraints on the network parameters or activation values during training. These regularization constraints help restrict the network’s complexity, resulting in a sparser dependency structure. As a result, mapping the network to a set of rules becomes more straightforward, facilitating the extraction of rules from the trained network. ### 4.2 Induced models based methods Proxy models for neural networks can also be constructed using alternative techniques, such as decision trees, from which rules can be extracted. This section reviews studies on proxy models that are capable of providing explanations in the form of rules. These methods can be categorized as either local or global, depending on whether they provide explanations for a specific example or the entire behavior of the network. These methods are summarised in Table 4. #### 4.2.1 Local methods Using the following procedure, local methods create a proxy model to generate explanations of the network decision for a particular input example. Given an input $x$ and its network prediction: - Generate a neighborhood of similar samples of $x$ . - Calculate the predictions of the network of the generated neighborhood samples. - Train an interpretable proxy model on the generated samples as inputs and the network predictions as targets to mimic its behavior at that specific input area. - Extract rules from the proxy model. Local proxies include program expression trees [93] and Local Rule Based Explanations (LORE) [38]. The program expression trees method uses programs as interpretable representations to explain local decisions. These programs are similar to the pseudo codes used to represent decision rules. The neighborhood is generated using perturbation techniques, and program induction is performed using a Simulated Annealing algorithm. The LORE method derives local decision rules and counterfactuals to characterize the network behavior locally. They use Genetic algorithms to generate the neighborhood and a decision tree local surrogate to approximate the network. #### 4.2.2 Global methods Global methods build proxy models to mimic the behavior of the network and explain it globally. Rules can be extracted following pedagogical, decompositional, or eclectic approaches according to how the proxy model is fitted. Pedagogical methods include decision trees-based proxy models considering the network as a black box and using input data and the network’s predictions as targets to learn the proxies. The standard approach considers as targets the hard labels given by the network, i.e., discrete classes [21, 78, 63]. Distillation methods, however, consider soft labels as targets, which are the probabilities predicted by the network [19, 30, 55]. Another approach consists of using decision tables to approximate the network [83] and extract rules. Boolean functions have also been used to approximate the network behavior by minimizing truth tables Boolean expressions [94, 62]. Decompositional methods use surrogates to approximate each unit in the network. In this case, the training data are the unit inputs, and the targets are the unit activation values. The decompositional approach can be post-hoc or intrinsic. In the post-hoc case, this approach does not impose any particular training procedure and is applied after the network training. The intrinsic case, however, applies particular training procedures to simplify the rule extraction task. [102] proposed an approximation of the network’s units using Boolean functions. The Continuous/discrete Rule Extractor via Decision tree induction (CRED) [77] consists of approximating the hidden and output units using decision trees. [110] extended it to the case of networks with multiple layers using the Deep neural network Rule Extraction via Decision tree induction (DeepRED) technique and pruning unimportant inputs. The Rule Extraction Methodology from DNN (REM-D) [92] improves the DeepRED by using decision trees C5 instead of C4.5 as a proxy model and merging the extracted rules incrementally as soon as they are generated. In DeepRED, rules are extracted for all layers and merged, improving time and memory consumption. [108] proposed a method to overcome the exponential runtime with respect to the training set and the network depth encountered in DeepRED and REM-D. These methods are post-hoc, except for the DeepRED, which considers a pruning stage. Other intrinsic decompositional methods were proposed and follow mainly this procedure: - Train a network using regularization techniques. - Prune the network to remove insignificant units. - Fit a surrogate model at each unit level and extract rules. - Derive and simplify rules expressing the relationship between the inputs and the outputs. This category includes [90, 89] studies that simplify the network using regularization and pruning techniques and approximates the hidden and output units using piecewise linear functions. These functions are used to express oblique rules. Eclectic methods consider fitting proxy models while integrating some knowledge from the network internals. [28] used regularization techniques to push the network to behave like a Boolean function. [91] considered regularizing and pruning the network, including the inputs, and extracting rules using a decision tree proxy. [95] proposed an approach where the network decision is explained in terms of the latent variables instead of the input features. Each input example is mapped to feature factors from the representation space for which explanations are generated from a decision tree-based proxy method. [14] incorporated knowledge about the network’s last hidden layer into the decision tree extraction process. [105] applied a particular regularization technique incorporating constraint into the loss function to make the network decision boundaries behave like a decision tree. The fitted decision tree proxy gives better results than using standard regularization techniques. Table 4: Summary of the induced models based methods for rule extraction from feedforward neural networks | Translucency | Design | Application | Scope | Reference | Tool name | Technique | | --- | --- | --- | --- | --- | --- | --- | | Pedagogical | Post–hoc | A | G | [21] | TREPAN | Decision trees | | A | G | [78] | ANN-DT | Decision trees | | | | A | G | [94] | BIO-RE | Boolean functions | | | | A | G | [63] | - | Decision trees | | | | A | G | [83] | KDRuleEx | Decision tables | | | | A | L | [93] | - | Program learner | | | | A | G | [19] | - | Decision trees | | | | A | G | [30] | Distilling | Decision trees | | | | A | L | [38] | LORE | Decision trees | | | | A | G | [55] | - | Decision trees | | | | A | G | [55] | - | Boolean functions | | | | Decompositional | Post–hoc | S | G | [102] | - | Boolean functions | | S | G | [77] | CRED | Decision trees | | | | S | G | [92] | REM-D | Decision trees | | | | S | G | [108] | ECLAIRE | Decision trees | | | | Intrinsic | S | G | [90] | REFANN | Piecewise linear functions | | | S | G | [89] | REFANN | Piecewise linear functions | | | | S | G | [110] | DeepRED | Decision trees | | | | Eclectic | Post–hoc | S | G | [95] | Tree-View | Decision Trees | | Intrinsic | S | G | [28] | MLP2LN | Boolean functions | | | S | G | [91] | RE-RX | Decision trees | | | | S | G | [14] | NNKX | Decision Trees | | | | S | G | [105] | Tree regularization | Decision trees | | | ### 4.3 Attribution methods Attribution-based methods provide explanations by defining the input features’ effect on the outputs. They have recently gained importance as a class of interpretability methods for DL models. To simplify the challenge of explaining a deep network, one can identify the important features, i.e., inputs or hidden units, for a particular output. The techniques for performing such a task can have different appellations, such as saliency, relevance, importance, or attribution. Feature importance can be visualized using heat maps or threshold masks. Many approaches are based on back-propagation of the signal output to input or local approximation of the decision boundaries using liner classifiers or decision trees. The existing methods can be classified into [101]: - Backpropagation of feature importance: these methods show positive contributions to a class. They are based on the backpropagation of signals from a deeper layer to a shallower one. Usually performed from the output layer to the input layer. The signal could be the activation, gradient, or some other metric derived from the output (e.g. deconvolution, gradient-based methods, and guided-backpropagation). - Backpropagation of bipolar feature importance: they can indicate evidence for and against a class (e.g. layerwise relevance propagation and DeepLIFT). - Perturbation methods: are based on evaluating the effect of removing or modifying a feature on the prediction (e.g. Prediction Difference Analysis, LIME, and SHAP). In recent literature reviews, there has been a growing interest in attribution-based methods for explaining DL models. These methods primarily focus on providing local explanations, which involve explaining a specific decision made by the model in relation to a single input by highlighting the important features that contributed to that decision. [72] proposed one approach towards achieving global interpretation from local attribution-based explanations. Their method involves explaining a set of representative individual instances and formulating the problem of defining the optimal set of instances as a submodular optimization problem. This approach aims to provide a holistic understanding of the model’s decision-making process by considering multiple instances together rather than focusing solely on individual inputs. It has also been observed that most of attribution-based methods do not establish the underlying relationships between the extracted important features, for instance, in the form of logical rules [101]. The idea of defining the most important features and extracting important relationships between them in the context of feedforward neural networks is not new. As early as the 1990s, this challenge was addressed from different perspectives. For instance, [5] proposed a causal index reflecting the influence of an individual input $x_i$ on a specific output $o_j$ . In the network, different paths could link an input $x_i$ to an output $o_j$ , with each path defined as a succession of links. The causal index $CI_ij$ is calculated as the sum of the products of path weights along all the paths between $x_i$ and $o_j$ in the network. The sign of the index reflects a positive or a negative correlation between the input and the output signals. The absolute value of the index gives the degree of correlation. [36] proposed a sensitivity-based approach to calculating features’ importance globally. The Point Sensitivity Index (PSI) is calculated for the entire range of the input showing decisive inputs with peaks in their PSI at critical values. The Sensitivity Causal Index (SCI) is an indicator of the global impact of a feature on an output. Unlike [5] method using network weights, sensitivity analysis is applied without using the network’s internals. [98] proposed the DEcision DEteCtion by rule extraction from neural networks (DEDEC) method to rank the input features that should be considered for rule extraction. DEDEC analyses the weight vectors to determine causal factors and functional dependencies in a trained neural network. Causal Attributions [18] consider the neural network as a Structural Causal Model to calculate the global causal effect of each feature on the output using the Average Causal Effect (ACE) [69]. Based on the magnitude and the sign of the feature importance indicator, rules between the inputs and the outputs can be extracted in the form “IF $x_2$ is large THEN $o_1$ is large”. For simple data cases, the decision boundaries can be clearly separated [18]. The Building Representations for AI using Neural NEtworks (BRAINNE) approach [81, 79, 80, 11] analyzes the network weights to define the important inputs to use for rule extraction. First, the method was proposed for single-layered networks composed of input and output layers trained with Hebb’s rule [17]. The method was then extended to the network with one hidden layer trained using Hebb’s rule and backpropagation, with a particular architecture considering the network outputs as inputs. [11] extended the method to extract disjunctive rules using a hybrid network composed of an unsupervised Kohonen model [52] and a second supervised part of a standard output layer. [39] proposed the Generalized Analytic Rule Extraction (GLARE), using an analytical approach to rank the inputs’ importance for each output class based on the network weights magnitude. [3] and [8, 9] used a reverse engineering technique to identify the important inputs and find mandatory input data ranges to classify a particular output class. Table 5 summarizes the studies based on attribution methods. [11] method is considered decompositional as they extract rules at an intermediate level between inputs and outputs layers and proceed with substitution to get rules linking inputs and output directly. It is also intrinsic as it considers a hybrid network using a self-organizing map to cluster the inputs and a standard feedforward network to classify them. [5], [81, 79, 80], [98], and [39] used weights to define features’ importance, hence their methods can be classified as eclectic. [36], [3], [8, 9], and [18] are pedagogical as they do not use the model’s internals. However, [18] considers the network as a Structural Causal Model to use causality principles and calculate Causal Attributions. Hence it is considered a model-specific method. All attribution methods are global. Table 5: Summary of the attribution methods for rule extraction from feedforward neural networks | Translucency | Design | Application | Reference | Tool name | Technique | | --- | --- | --- | --- | --- | --- | | Pedagogical | Post–hoc | A | [36] | Point Sensitivity Index (PSI) and Sensitivity Causal Index (SCI) | Causality and sensitivity | | A | [3, 8] | RxREN | Reverse engineering | | | | A | [9] | RxNCM | Reverse engineering | | | | Decompositional | Intrinsic | S | [11] | BRAINNE | Weights analysis | | Eclectic | Post–hoc | S | [5] | Causal Index | Gradient | | S | [81, 79, 80] | BRAINNE | Network weights analysis | | | | S | [98] | DEDEC | Network weights analysis | | | | S | [39] | GLARE | Network weights analysis | | | | S | [18] | Causal Attributions | Causality (Average Causal Effect) | | | ### 4.4 Optimization methods The rule extraction task from neural networks can be formulated as an optimization problem. The objective is to find the inputs’ optimal values that maximize a particular class’s probability. Considering a network with one hidden layer and sigmoid activations, the $j^th$ output activation can be expressed by Equation (1). It is a non-linear function of the inputs parameterized by the network weights $W_1$ and $W_2$ , where $W_1$ are the weights linking the inputs to the hidden units and $W_2$ are the weights linking the hidden units to the output. After network training, the weights are fixed, and the activation function of the $j^th$ output can be expressed as a function of the inputs. Finding the input vector which maximizes $Y_j(x_i)$ is an optimization problem. If the decision variables or inputs $x_i$ are binary and the objective function $Y_j$ is nonlinear, the problem can be expressed as a nonlinear integer programming problem. Solving this problem enables expressing rules linking the input features to the output class. Table 6 summarizes studies on rule extraction from neural networks based on optimization techniques. All methods are specific to neural networks and provide global explanations. [66, 67] solved the optimization problem using the Touring Ant Colony Optimization (TACO) algorithm, [48] using Artificial Immune Systems algorithm, and [103] using Variable Neighbourhood Search algorithm. [67] used the TACO method with an intrinsic approach where the rules are extracted during network training, and their quality metrics are used to regularize the network towards generating good rules. $$ Y_j=\frac{1}{1+e^-[{∑_j=1^{J(W_2)_j,k(\frac{1}{1+e^-[∑_i=1 ^Ix_i(W_1)i,j]})}]}} \tag{1} $$ $x_i=0\>or\>1$ Optimization-based techniques for extracting rules from neural networks are not suitable for deep networks. This approach faces challenges in scaling up to deep networks due to the increased complexity that arises with additional hidden layers. Furthermore, applying it to high-dimensional data can be challenging. Additionally, accessing the weights of the network is required. Table 6: Summary of the optimization methods for rule extraction from feedforward neural networks | Eclectic | Pos–thoc | [65, 66] | TACO | Ant Colony Algorithm | | --- | --- | --- | --- | --- | | [48] | - | Artificial Immune Systems algorithm | | | | [103] | - | Variable Neighbourhood Search | | | | Intrinsic | [67] | DIFACONN-miner | Ant Colony Algorithm | | ### 4.5 Hybrid methods Hybrid methods combine different techniques from the previously presented for rules extraction from neural networks. Table 7 presents hybrid methods, which are all global and specific. Considering networks with one hidden layer, one approach expresses the relationship between inputs and hidden units as linear inequalities. Rules between hidden and output units can be extracted using an explore & test method [87, 64], Boolean functions [13], or decision trees [12, 85]. The rule extraction problem can also be expressed as a linear programming standard problem to extract rules between binary or discretized input features and hidden units [94]. [76] proposed the Rule extraction from Neural networks version 2 (RN2) to produce regression rules on a specific network architecture to allow extraction of rules with logical formula on the premise part over the nominal attributes and a polynomial equation over the continuous attributes in the conclusion part. They used decision trees to generate rules between nominal attributes and discretized hidden activations. The rules conclusions are obtained by injecting the activation values in the polynomial term, including the continuous values. [15] proposed the Eclectic Rule Extraction from Neural Network Recursively (ERENNR) method using a rule learner to extract rules between hidden and output units. The conditions are then replaced by input ranges obtained from analyzing each hidden unit from its classified patterns. It was extended to Eclectic Rule Extraction from Neural Network with Multi Hidden Layer (ERENN MHL) by [16] method for multi-hidden layer networks. Table 7: Summary of hybrid methods for rule extraction from feedforward neural networks | Translucency | Design | Reference | Tool name | Technique | | --- | --- | --- | --- | --- | | Decompositional | Post–hoc | [94] | Full-RE | Optimization and explore & test | | [76] | RN2 | Decision trees and analytical expressions | | | | Intrinsic | [87] | Neurolinear | linear inequalities and explore & test | | | [85] | FERNN | linear inequalities and decision trees | | | | [64] | GRG | Linear inequalities and explore & test | | | | [45] | – | Decision trees and linear inequalities | | | | [16] | ERENN MHL | Attribution methods and Boolean functions | | | | Eclectic | Post–hoc | [13] | IMLP | Linear inequalities and Boolean functions | | [12] | DIMLP | Linear inequalities and decision trees | | | | Intrinsic | [15] | – | Attribution methods and Boolean functions | | ## 5 Discussion Most of the rule extraction methods from neural networks reviewed in this study were proposed for shallow networks with a depth not exceeding three hidden layers [42]. Rule extraction from neural networks can be computationally expensive, especially for larger, more complex networks or datasets. The complexity of a rule extraction method depends on various factors, such as the type of extraction algorithm, the architecture and the size of the network, and the complexity of the data being processed. The explore & test, optimization, and decompositional-based extraction methods particularly involve complex operations even with shallow networks. Moreover, deep neural networks with a large number of layers and nodes may require more computational resources and time to extract rules. Furthermore, when dealing with high-dimensional data or intricate patterns, the level of complexity increases. Decompositional-based techniques represent an important part of rule extraction methods from neural networks. The decompositional approach introduces intermediate terms corresponding to each of the hidden and output units in the network. The intuition behind this is that rules extracted at individual units can be merged to explain the network behavior. This approach, however, assumes that each hidden and output unit can behave like a Boolean variable, and the individual hidden units of the network correspond to meaningful concepts in the context of the domain. Relying on intuition-based methods for interpreting deep neural networks can be misleading, and it is important to determine if single-unit behavior can help understanding deep neural networks. Decompositional methods focus on the properties of individual units by finding rules that activate a specific unit. Neural networks, however, learn distributed representations in which the activations of many hidden units encode each concept, and each hidden unit plays a part in representing many different concepts. Given such representations, the right intermediate terms might represent patterns of activity across groups of hidden units instead of individual hidden units themselves. For a rule extraction method to be adaptable even for large networks, the following properties defined by [96] are highly desirable: (i) it should have no architectural requirements and is able to work with all networks; (ii) it has no training requirements and makes no assumptions about how the network was built and trained, (iii) it provides correct rules describing the underlying network, (iv) it has high expressive power in terms of the language and compactness of the rules. Given these properties, post-hoc model agnostic proxy methods are the most appropriate to use with deep neural networks. Proxy methods are currently a popular approach for interpreting DL models. One of their key advantages is their agnostic nature, as they treat the black-box model as an oracle and only require a dataset along with the corresponding model’s predictions. They offer a simple way to interpret. However, it’s important to note that proxy methods have their limitations and may not always provide a complete or accurate explanation of the network behavior, as they are approximations and not a direct representation of the original model. Attribution methods also offer a cheap way to local interpretation of DL models. This helps in understanding why a particular prediction was made in a specific context, which is important in object detection, image segmentation, or natural language processing tasks. However, attribution methods might struggle to capture the behavior of the entire model to provide concise global explanations. Finally, the extraction of interpretable rules from neural networks includes an essential phase that often remains underemphasized in the existing literature: the pruning of extracted rules. This phase becomes particularly significant when employing decompositional approaches or dealing with large networks, as the extraction process can yield rule sets with an excessive number of redundant or irrelevant rules. Such redundancy and irrelevance not only affect the interpretability but also limit the generalization performance. Consequently, the development of efficient techniques for rule pruning becomes imperative, enabling the selective removal of unnecessary rules while retaining the crucial information encoded in the network. Effective rule pruning not only enhances the interpretability of neural networks but also enhances their computational efficiency and scalability in rule extraction methodologies. By facilitating the extraction of simpler and more concise rule sets, these pruning techniques enable a comprehensive understanding of the underlying behavior of the neural network. Consequently, investing in the advancement of efficient rule pruning techniques [7, 4] is important in interpreting complex neural networks through simple decision rules. ## 6 Conclusion Improving the interpretability of ML models is crucial for the successful deployment of AI enabled systems. This paper aimed to contribute to this objective by focusing on a specific tool for interpretability, namely rule extraction for neural networks. The main objective of this research was to conduct a systematic literature review to identify and understand the various approaches used to generate rules from feedforward neural networks. The findings of this study indicate that the field of rule extraction from neural networks has been actively explored for over two decades, resulting in the development of diverse methods. While most of the proposed techniques are well-suited for shallow networks, it was observed that some of them can also be adapted for DL models. Rules, due to their simplicity and transparency, offer an intuitive means of explaining neural network behavior to users. Therefore, this study provides a comprehensive introduction for researchers interested in this field, laying the groundwork for further investigations and advancements. It is important to note that this study specifically focused on feedforward networks with supervised learning tasks and crisp extracted rules. However, there is ample opportunity for future research to extend these findings to rule extraction from other types of neural networks, such as RNN and CNN models. Additionally, it would be valuable to explore rule extraction methods in the context of other ML techniques and to include approaches related to fuzzy rules extraction. In conclusion, this research contributes to the understanding of rule extraction methods for feedforward neural networks and serves as a valuable resource for researchers aiming to enhance interpretability in AI systems. Expanding this work to encompass other neural network types and ML methods, as well as incorporating fuzzy rule extraction, presents exciting avenues for future exploration in this field. ## References - Alexander and Mozer [1994] J. Alexander and M. C. Mozer. Template-based algorithms for connectionist rule extraction. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7, pages 609–616. MIT Press, 1994. URL https://proceedings.neurips.cc/paper/1994/file/24896ee4c6526356cc127852413ea3b4-Paper.pdf. - Andrews et al. [1995] R. Andrews, J. Diederich, and A. B. Tickle. Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge-based systems, 8(6):373–389, 1995. - Augasta and Kathirvalavakumar [2012] M. G. Augasta and T. Kathirvalavakumar. Reverse engineering the neural networks for rule extraction in classification problems. Neural processing letters, 35:131–150, 2012. doi: 10.1007/s11063-011-9207-8. - Azmi et al. [2019] M. Azmi, G. C. Runger, and A. Berrado. Interpretable regularized class association rules algorithm for classification in a categorical data space. Information Sciences, 483:313–331, 2019. - Baba et al. [1990] K. Baba, I. Enbutu, and M. Yoda. Explicit representation of knowledge acquired from plant historical data using neural network. In 1990 IJCNN International Joint Conference on Neural Networks, volume 3, pages 155–160, 1990. doi: 10.1109/IJCNN.1990.137838. - Beck et al. [2016] F. Beck, S. Koch, and D. Weiskopf. Visual analysis and dissemination of scientific literature collections with survis. IEEE Transactions on Visualization and Computer Graphics, 22(1):180–189, 2016. doi: 10.1109/TVCG.2015.2467757. - Berrado and Runger [2007] A. Berrado and G. C. Runger. Using metarules to organize and group discovered association rules. Data mining and knowledge discovery, 14:409–431, 2007. - Biswas et al. [2017] S. K. Biswas, M. Chakraborty, B. Purkayastha, P. Roy, and D. M. Thounaojam. Rule extraction from training data using neural network. International Journal on Artificial Intelligence Tools, 26(03):1750006, 2017. doi: 10.1142/S0218213017500063. URL https://doi.org/10.1142/S0218213017500063. - Biswas et al. [2018] S. K. Biswas, M. Chakraborty, and B. Purkayastha. A rule generation algorithm from neural network using classified and misclassified data. International Journal of Bio-Inspired Computation, 11(1):60–70, 2018. doi: 10.1504/IJBIC.2018.090070. URL https://www.inderscienceonline.com/doi/abs/10.1504/IJBIC.2018.090070. - Blasig [1993] R. Blasig. Gds: Gradient descent generation of symbolic classification rules. In J. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems, volume 6, pages 1993–1100. Morgan-Kaufmann, 1993. URL https://proceedings.neurips.cc/paper/1993/file/4e0928de075538c593fbdabb0c5ef2c3-Paper.pdf. - Bloomer et al. [1996] W. Bloomer, T. Dillon, and M. Witten. Hybrid brainne: a method for developing symbolic disjunctive rules from a hybrid neural network. In 1996 IEEE International Conference on Systems, Man and Cybernetics. Information Intelligence and Systems (Cat. No.96CH35929), volume 4, pages 2745–2750, 1996. doi: 10.1109/ICSMC.1996.561374. - Bologna [2000] G. Bologna. Symbolic rule extraction from the dimlp neural network. In S. Wermter and R. Sun, editors, Hybrid Neural Systems, pages 240–254, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg. ISBN 978-3-540-46417-4. - Bologna and Pellegrini [1998] G. Bologna and C. Pellegrini. Constraining the mlp power of expression to facilitate symbolic rule extraction. In 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227), volume 1, pages 146–151, 1998. doi: 10.1109/IJCNN.1998.682252. - Bondarenko et al. [2017] A. Bondarenko, L. Aleksejeva, V. Jumutc, and A. Borisov. Classification tree extraction from trained artificial neural networks. Procedia Computer Science, 104:556–563, 2017. ISSN 1877-0509. doi: 10.1016/j.procs.2017.01.172. URL https://www.sciencedirect.com/science/article/pii/S1877050917301734. ICTE 2016, Riga Technical University, Latvia. - Chakraborty et al. [2019] M. Chakraborty, S. K. Biswas, and B. Purkayastha. Rule extraction from neural network using input data ranges recursively. New Generation Computing, 37:67–96, 2019. doi: 10.1007/s00354-018-0048-0. - Chakraborty et al. [2020] M. Chakraborty, S. K. Biswas, and B. Purkayastha. Rule extraction from neural network trained using deep belief network and back propagation. Knowledge and Information Systems, 62:3753–3781, 2020. doi: 10.1007/s10115-020-01473-0. - Chakraverty et al. [2019] S. Chakraverty, D. M. Sahoo, N. R. Mahato, S. Chakraverty, D. M. Sahoo, and N. R. Mahato. Hebbian learning rule. Concepts of Soft Computing: Fuzzy and ANN with Programming, pages 175–182, 2019. - Chattopadhyay et al. [2019] A. Chattopadhyay, P. Manupriya, A. Sarkar, and V. N. Balasubramanian. Neural network attributions: A causal perspective. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 981–990. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/chattopadhyay19a.html. - Che et al. [2016] Z. Che, S. Purushotham, R. Khemani, and Y. Liu. Interpretable deep models for icu outcome prediction. In AMIA annual symposium proceedings, volume 2016, pages 371–380, 2016. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5333206/. - Clark and Niblett [1989] P. Clark and T. Niblett. The cn2 induction algorithm. Machine learning, 3(4):261–283, 1989. - Craven and Shavlik [1995] M. Craven and J. Shavlik. Extracting tree-structured representations of trained networks. In D. Touretzky, M. Mozer, and M. Hasselmo, editors, Advances in Neural Information Processing Systems, volume 8, pages 24–30. MIT Press, 1995. URL https://proceedings.neurips.cc/paper/1995/file/45f31d16b1058d586fc3be7207b58053-Paper.pdf. - Craven [1996] M. W. Craven. Extracting comprehensible models from trained neural networks. The University of Wisconsin-Madison, 1996. - Craven and Shavlik [1993a] M. W. Craven and J. W. Shavlik. Learning symbolic rules using artificial neural networks. In Machine Learning Proceedings 1993, pages 73–80. Morgan Kaufmann, San Francisco (CA), 1993a. ISBN 978-1-55860-307-3. doi: 10.1016/B978-1-55860-307-3.50016-2. URL https://www.sciencedirect.com/science/article/pii/B9781558603073500162. - Craven and Shavlik [1993b] M. W. Craven and J. W. Shavlik. Understanding neural networks via rule extraction and pruning. In Proceedings of the 1993 Connectionist Models Summer School, page 184. Psychology Press, 1993b. URL https://books.google.co.ma/books?hl=en&lr=&id=0SEBAwAAQBAJ&oi=fnd&pg=PA184&dq=Understanding+neural+networks+via+rule+extraction+and+pruning&ots=U5vxpNw3CH&sig=1eNzxACVFHmoEBk4M3fldMCKNZY&redir_esc=y#v=onepage&q=Understanding%20neural%20networks%20via%20rule%20extraction%20and%20pruning&f=false. - Craven and Shavlik [1994] M. W. Craven and J. W. Shavlik. Using sampling and queries to extract rules from trained neural networks. In W. W. Cohen and H. Hirsh, editors, Machine Learning Proceedings 1994, pages 37–45. Morgan Kaufmann, San Francisco (CA), 1994. ISBN 978-1-55860-335-6. doi: https://doi.org/10.1016/B978-1-55860-335-6.50013-1. URL https://www.sciencedirect.com/science/article/pii/B9781558603356500131. - d’Avila Garcez et al. [2001] A. S. d’Avila Garcez, K. Broda, and D. M. Gabbay. Symbolic knowledge extraction from trained neural networks: A sound approach. Artificial Intelligence, 125(1):155–207, 2001. ISSN 0004-3702. doi: 10.1016/S0004-3702(00)00077-1. URL https://www.sciencedirect.com/science/article/pii/S0004370200000771. - Decloedt et al. [1996] L. Decloedt, F. Osório, and B. Amy. Rule_out method: A new approach for knowledge explicitation from trained ann. In Andrews and Diederich, editors, Proceedings of the AISB’96 - Workshop on Rule Extraction from Trained Neural Nets, pages 34–42, 1996. URL https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.46.8091&rep=rep1&type=pdf. - Duch et al. [1998] W. Duch, R. Adamczak, and K. Grąbczewski. Extraction of logical rules from neural networks. Neural Processing Letters, 7:211–219, 1998. doi: 10.1023/A:1009670302979. - Duygu Arbatli and Levent Akin [1997] A. Duygu Arbatli and H. Levent Akin. Rule extraction from trained neural networks using genetic algorithms. Nonlinear Analysis: Theory, Methods & Applications, 30(3):1639–1648, 1997. ISSN 0362-546X. doi: https://doi.org/10.1016/S0362-546X(96)00267-2. URL https://www.sciencedirect.com/science/article/pii/S0362546X96002672. - Frosst and Hinton [2017] N. Frosst and G. Hinton. Distilling a neural network into a soft decision tree. In CEX workshop at AI*IA 2017 conference, 2017. URL https://arxiv.org/pdf/1711.09784.pdf. - Fu [1991] L. Fu. Rule learning by searching on adapted nets. In 9th National Conference on Artificial Intelligence, July 14-19, 1991, Anaheim, California, pages 590–595, Menlo Park, California, 1991. AAAI Press. - Fu [1994] L. Fu. Rule generation from neural networks. IEEE Transactions on Systems, Man, and Cybernetics, 24(8):1114–1124, 1994. doi: 10.1109/21.299696. - Fürnkranz et al. [2012] J. Fürnkranz, D. Gamberger, and N. Lavrač. Foundations of rule learning. Springer Science & Business Media, 2012. - Gallant [1988] S. I. Gallant. Connectionist expert systems. Commun. ACM, 31(2):152–169, feb 1988. ISSN 0001-0782. doi: 10.1145/42372.42377. URL https://doi.org/10.1145/42372.42377. - Garcez et al. [2019] A. d. Garcez, M. Gori, L. C. Lamb, L. Serafini, M. Spranger, and S. N. Tran. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. arXiv preprint arXiv:1905.06088, 2019. - Goh [1993] T.-H. Goh. Semantic extraction using neural network modelling and sensitivity analysis. In Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan), volume 1, pages 1031–1034, 1993. doi: 10.1109/IJCNN.1993.714088. - Goodfellow et al. [2016] I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org. - Guidotti et al. [2018] R. Guidotti, A. Monreale, S. Ruggieri, D. Pedreschi, F. Turini, and F. Giannotti. Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820, 2018. doi: 10.48550/ARXIV.1805.10820. - Gupta et al. [1999] A. Gupta, S. Park, and S. M. Lam. Generalized analytic rule extraction for feedforward neural networks. IEEE Transactions on Knowledge and Data Engineering, 11(6):985–991, 1999. doi: 10.1109/69.824621. - Hailesilassie [2016] T. Hailesilassie. Rule extraction algorithm for deep neural networks: A review. arXiv preprint arXiv:1610.05267, 2016. - Hayward et al. [1996] R. Hayward, A. Tickle, and J. Diederich. Extracting rules for grammar recognition from cascade-2 networks. In S. Wermter, E. Riloff, and G. Scheler, editors, Connectionist, Statistical and Symbolic Approaches to Learning for Natural Language Processing, pages 48–60, Berlin, Heidelberg, 1996. Springer Berlin Heidelberg. ISBN 978-3-540-49738-7. - He et al. [2020] C. He, M. Ma, and P. Wang. Extract interpretability-accuracy balanced rules from artificial neural networks: A review. Neurocomputing, 387:346–358, 2020. - Hruschka [1998] E. R. Hruschka. Rule extraction from neural networks in data mining applications. WIT Transactions on Information and Communication Technologies, 22, 1998. URL https://www.witpress.com/elibrary/wit-transactions-on-information-and-communication-technologies/22/6942. - Hruschka and Ebecken [2006] E. R. Hruschka and N. F. Ebecken. Extracting rules from multilayer perceptrons in classification problems: A clustering-based approach. Neurocomputing, 70(1):384–397, 2006. ISSN 0925-2312. doi: 10.1016/j.neucom.2005.12.127. URL https://www.sciencedirect.com/science/article/pii/S0925231206000403. - Huynh and Reggia [2009] T. Q. Huynh and J. A. Reggia. Improving rule extraction from neural networks by modifying hidden layer representations. In 2009 International Joint Conference on Neural Networks, pages 1316–1321, 2009. doi: 10.1109/IJCNN.2009.5178685. - Huysmans et al. [2006] J. Huysmans, B. Baesens, and J. Vanthienen. Using rule extraction to improve the comprehensibility of predictive models. K.U. Leuven KBI Working Paper No. 0612, 2006. doi: http://dx.doi.org/10.2139/ssrn.961358. - Johansson et al. [2004] U. Johansson, R. König, and L. Niklasson. The truth is in there-rule extraction from opaque models using genetic programming. In Proceedings of the FLAIRS Conference, pages 658–663, 2004. URL https://www.aaai.org/Library/FLAIRS/2004/flairs04-113.php. - Kahramanli and Allahverdi [2009] H. Kahramanli and N. Allahverdi. Rule extraction from trained adaptive neural networks using artificial immune systems. Expert Systems with Applications, 36(2, Part 1):1513–1522, 2009. ISSN 0957-4174. doi: 10.1016/j.eswa.2007.11.024. URL https://www.sciencedirect.com/science/article/pii/S0957417407005878. - Kaikhah and Doddameti [2006] K. Kaikhah and S. Doddameti. Discovering trends in large datasets using neural networks. Applied Intelligence, 24:51–60, 2006. doi: 10.1007/s10489-006-6929-9. - Kim [2000] H. Kim. Computationally efficient heuristics for if-then rule extraction from feed-forward neural networks. In S. Arikawa and S. Morishita, editors, Discovery Science, pages 170–182, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg. ISBN 978-3-540-44418-3. doi: 10.1007/3-540-44418-1_14. - Kitchenham [2004] B. Kitchenham. Procedures for performing systematic reviews. Keele, UK, Keele University, 33(2004):1–26, 2004. - Kohonen [1990] T. Kohonen. The self-organizing map. Proceedings of the IEEE, 78(9):1464–1480, 1990. - Krishnan et al. [1999] R. Krishnan, G. Sivakumar, and P. Bhattacharya. A search technique for rule extraction from trained neural networks. Pattern Recognition Letters, 20(3):273–280, 1999. ISSN 0167-8655. doi: https://doi.org/10.1016/S0167-8655(98)00145-7. URL https://www.sciencedirect.com/science/article/pii/S0167865598001457. - Lee and Kim [2021] H. Lee and H. Kim. Uncertainty of rules extracted from artificial neural networks. Applied Artificial Intelligence, 35(8):589–604, 2021. doi: 10.1080/08839514.2021.1922845. URL https://doi.org/10.1080/08839514.2021.1922845. - Liu et al. [2018] X. Liu, X. Wang, and S. Matwin. Improving the interpretability of deep neural networks with knowledge distillation. In 2018 IEEE International Conference on Data Mining Workshops (ICDMW), pages 905–912, 2018. doi: 10.1109/ICDMW.2018.00132. - Löfström et al. [2004] T. Löfström, U. Johansson, and L. Niklasson. Rule extraction by seeing through the model. In N. R. Pal, N. Kasabov, R. K. Mudi, S. Pal, and S. K. Parui, editors, Neural Information Processing, pages 555–560, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg. ISBN 978-3-540-30499-9. doi: 10.1007/978-3-540-30499-9_85. - Lu et al. [1995] H. Lu, R. Setiono, and H. Liu. Neurorule: A connectionist approach to data mining. In Proceedings of the 21st VLDB Conference Zurich, Swizerland, 1995, 1995. URL https://arxiv.org/abs/1701.01358. - Lu et al. [1996] H. Lu, R. Setiono, and H. Liu. Effective data mining using neural networks. IEEE Transactions on Knowledge and Data Engineering, 8(6):957–961, 1996. doi: 10.1109/69.553163. - Lu et al. [2006] J. Lu, S. Tokinaga, and Y. Ikeda. Explanatory rule extraction based on the trained neural network and the genetic programming. Journal of the Operations Research Society of Japan, 49(1):66–82, 2006. doi: 10.15807/jorsj.49.66. - McMillan et al. [1991a] C. McMillan, M. C. Mozer, and P. Smolensky. The connectionist scientist game: rule extraction and refinement in a neural network. In Proceedings of the 13th Annual Conference of the Cognitive Science Society, pages 424–430, 1991a. - McMillan et al. [1991b] C. McMillan, M. C. Mozer, and P. Smolensky. Rule induction through integrated symbolic and subsymbolic processing. In J. Moody, S. Hanson, and R. Lippmann, editors, Advances in Neural Information Processing Systems, volume 4, pages 969–976. Morgan-Kaufmann, 1991b. URL https://proceedings.neurips.cc/paper/1991/file/cf67355a3333e6e143439161adc2d82e-Paper.pdf. - Mereani and Howe [2019] F. A. Mereani and J. M. Howe. Exact and approximate rule extraction from neural networks with boolean features. In Proceedings of the 11th International Joint Conference on Computational Intelligence, pages 424–433. SCITEPRESS, 2019. doi: 10.5220/0008362904240433. - Milaré et al. [2001] C. R. Milaré, A. C. de Carvalho, and M. C. Monard. Extracting rules from neural networks using symbolic algorithms: preliminary results. In Proceedings Fourth International Conference on Computational Intelligence and Multimedia Applications. ICCIMA 2001, pages 384–388, 2001. doi: 10.1109/ICCIMA.2001.970500. - Odajima et al. [2006] K. Odajima, Y. Hayashi, and R. Setiono. Greedy rule generation from discrete data and its use in neural network rule extraction. In The 2006 IEEE International Joint Conference on Neural Network Proceedings, pages 1833–1839, 2006. doi: 10.1109/IJCNN.2006.246902. - Özbakır et al. [2008] L. Özbakır, A. Baykasoğlu, and S. Kulluk. Rule extraction from neural networks via ant colony algorithm for data mining applications. In V. Maniezzo, R. Battiti, and J.-P. Watson, editors, Learning and Intelligent Optimization, pages 177–191, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540-92695-5. doi: 10.1007/978-3-540-92695-5_14. - Özbakır et al. [2009] L. Özbakır, A. Baykasoğlu, S. Kulluk, and H. Yapıcı. Taco-miner: An ant colony based algorithm for rule extraction from trained neural networks. Expert Systems with Applications, 36(10):12295–12305, 2009. ISSN 0957-4174. doi: 10.1016/j.eswa.2009.04.058. URL https://www.sciencedirect.com/science/article/pii/S0957417409004084. - Özbakır et al. [2010] L. Özbakır, A. Baykasoğlu, and S. Kulluk. A soft computing-based approach for integrated training and rule extraction from artificial neural networks: Difaconn-miner. Applied Soft Computing, 10(1):304–317, 2010. ISSN 1568-4946. doi: 10.1016/j.asoc.2009.08.008. URL https://www.sciencedirect.com/science/article/pii/S1568494609001318. - Palade et al. [2001] V. Palade, D.-C. Neagu, and R. J. Patton. Interpretation of trained neural networks by rule extraction. In B. Reusch, editor, Computational Intelligence. Theory and Applications, pages 152–161, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg. ISBN 978-3-540-45493-9. doi: 10.1007/3-540-45493-4_20. - Pearl [2009] J. Pearl. Causality. Cambridge university press, 2009. - Rasamoelina et al. [2020] A. D. Rasamoelina, F. Adjailia, and P. Sinčák. A review of activation function for artificial neural network. In 2020 IEEE 18th World Symposium on Applied Machine Intelligence and Informatics (SAMI), pages 281–286. IEEE, 2020. - Raschka [2018] S. Raschka. Model evaluation, model selection, and algorithm selection in machine learning. arXiv preprint arXiv:1811.12808, 2018. - Ribeiro et al. [2016] M. T. Ribeiro, S. Singh, and C. Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016. - Ribeiro et al. [2018] M. T. Ribeiro, S. Singh, and C. Guestrin. Anchors: High-precision model-agnostic explanations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaai.v32i1.11491. URL https://ojs.aaai.org/index.php/AAAI/article/view/11491. - Saito and Nakano [1988] K. Saito and R. Nakano. Medical diagnostic expert system based on pdp model. In IEEE 1988 International Conference on Neural Networks, volume 1, pages 255–262, 1988. doi: 10.1109/ICNN.1988.23855. - Saito and Nakano [1990] K. Saito and R. Nakano. Automatic extraction of classification rules. In International Neural Network Conference, July 9-13, 1990, Palais des Congres, Paris, France, pages 379–382, 1990. - Saito and Nakano [2002] K. Saito and R. Nakano. Extracting regression rules from neural networks. Neural Networks, 15(10):1279–1288, 2002. ISSN 0893-6080. doi: 10.1016/S0893-6080(02)00089-8. URL https://www.sciencedirect.com/science/article/pii/S0893608002000898. - Sato and Tsukimoto [2001] M. Sato and H. Tsukimoto. Rule extraction from neural networks via decision tree induction. In IJCNN’01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222), volume 3, pages 1870–1875, 2001. doi: 10.1109/IJCNN.2001.938448. - Schmitz et al. [1999] G. P. Schmitz, C. Aldrich, and F. S. Gouws. Ann-dt: an algorithm for extraction of decision trees from artificial neural networks. IEEE Transactions on Neural Networks, 10(6):1392–1401, 1999. doi: 10.1109/72.809084. - Sestito and Dillon [1991] S. Sestito and T. Dillon. Using single-layered neural networks for the extraction of conjunctive rules and hierarchical classifications. Applied Intelligence, 1:157–173, 1991. doi: 10.1007/BF00058881. - Sestito and Dillon [1993] S. Sestito and T. Dillon. Knowledge acquisition of conjunctive rules using multilayered neural networks. International Journal of Intelligent Systems, 8(7):779–805, 1993. doi: https://doi.org/10.1002/int.4550080704. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/int.4550080704. - Sestito and Dillon [1990] S. Sestito and T. S. Dillon. Machine learning using single-layered and multi-layered neural networks. In Proceedings of the 2nd International IEEE Conference on Tools for Artificial Intelligence, pages 269–275, 1990. doi: 10.1109/TAI.1990.130346. - Sethi and Yoo [1994] I. K. Sethi and J. H. Yoo. Symbolic approximation of feedforward neural networks. In E. S. GELSEMA and L. S. KANAL, editors, Pattern Recognition in Practice IV, volume 16 of Machine Intelligence and Pattern Recognition, pages 313–324. North-Holland, 1994. doi: https://doi.org/10.1016/B978-0-444-81892-8.50032-8. URL https://www.sciencedirect.com/science/article/pii/B9780444818928500328. - Sethi et al. [2012] K. K. Sethi, D. K. Mishra, and B. Mishra. Kdruleex: A novel approach for enhancing user comprehensibility using rule extraction. In 2012 Third International Conference on Intelligent Systems Modelling and Simulation, pages 55–60, 2012. doi: 10.1109/ISMS.2012.116. - Setiono [1997] R. Setiono. Extracting Rules from Neural Networks by Pruning and Hidden-Unit Splitting. Neural Computation, 9(1):205–225, 01 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.1.205. URL https://doi.org/10.1162/neco.1997.9.1.205. - Setiono and Leow [2000] R. Setiono and W. K. Leow. Fernn: An algorithm for fast extraction of rules from neural networks. Applied Intelligence, 12:15–25, 2000. doi: 10.1023/A:1008307919726. - Setiono and Liu [1996] R. Setiono and H. Liu. Symbolic representation of neural networks. Computer, 29(3):71–77, 1996. doi: 10.1109/2.485895. - Setiono and Liu [1997] R. Setiono and H. Liu. Neurolinear: From neural networks to oblique decision rules. Neurocomputing, 17(1):1–24, 1997. ISSN 0925-2312. doi: https://doi.org/10.1016/S0925-2312(97)00038-6. URL https://www.sciencedirect.com/science/article/pii/S0925231297000386. - Setiono and Liu [August 20–25,1995] R. Setiono and H. Liu. Understanding neural networks via rule extraction. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence (IJCAI), Montreal, Quebec, Canada, pages 480–485, August 20–25,1995. URL https://www.ijcai.org/Proceedings/95-1/Papers/063.pdf. - Setiono and Thong [2004] R. Setiono and J. Y. Thong. An approach to generate rules from neural networks for regression problems. European Journal of Operational Research, 155(1):239–250, 2004. URL https://ssrn.com/abstract=3766019. - Setiono et al. [2002] R. Setiono, W. K. Leow, and J. M. Zurada. Extraction of rules from artificial neural networks for nonlinear regression. IEEE Transactions on Neural Networks, 13(3):564–577, 2002. doi: 10.1109/TNN.2002.1000125. - Setiono et al. [2008] R. Setiono, B. Baesens, and C. Mues. Recursive neural network rule extraction for data with mixed attributes. IEEE Transactions on Neural Networks, 19(2):299–307, 2008. doi: 10.1109/TNN.2007.908641. - Shams et al. [2021] Z. Shams, B. Dimanov, S. Kola, N. Simidjievski, H. A. Terre, P. Scherer, U. Matjašec, J. Abraham, M. Jamnik, and P. Liò. Rem: An integrative rule extraction methodology for explainable data analysis in healthcare. medRxiv, 2021. doi: 10.1101/2021.01.25.21250459. URL https://www.medrxiv.org/content/early/2021/04/28/2021.01.25.21250459. - Singh et al. [2016] S. Singh, M. T. Ribeiro, and C. Guestrin. Programs as black-box explanations, 2016. URL https://arxiv.org/abs/1611.07579. - Taha and Ghosh [1999] I. Taha and J. Ghosh. Symbolic interpretation of artificial neural networks. IEEE Transactions on Knowledge and Data Engineering, 11(3):448–463, 1999. doi: 10.1109/69.774103. - Thiagarajan et al. [2016] J. J. Thiagarajan, B. Kailkhura, P. Sattigeri, and K. N. Ramamurthy. Treeview: Peeking into deep neural networks via feature-space partitioning. arXiv preprint arXiv:1611.07429, 2016. URL https://arxiv.org/abs/1611.07429. - Thrun [1993] S. Thrun. Extracting provably correct rules from artificial neural networks. Technical report, University of Bonn, Institut fur Informatik III, 1993. URL https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.2.2110&rep=rep1&type=pdf. - Thrun [1994] S. Thrun. Extracting rules from artificial neural networks with distributed representations. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7. MIT Press, 1994. URL https://proceedings.neurips.cc/paper/1994/file/bea5955b308361a1b07bc55042e25e54-Paper.pdf. - Tickle et al. [1994] A. Tickle, M. Orlowski, and J. Diederich. Dedec: Decision detection by rule extraction from neural networks. queensland university of technology. Neurocomputing Research Center QUT NRC, 1994. - Tickle et al. [1998] A. B. Tickle, R. Andrews, M. Golea, and J. Diederich. The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks. IEEE Transactions on Neural Networks, 9(6):1057–1068, 1998. - Towell and Shavlik [1993] G. G. Towell and J. W. Shavlik. Extracting refined rules from knowledge-based neural networks. Machine Learning, 13:71–101, 1993. doi: 10.1007/BF00993103. - Townsend et al. [2019] J. Townsend, T. Chaton, and J. M. Monteiro. Extracting relational explanations from deep neural networks: A survey from a neural-symbolic perspective. IEEE transactions on neural networks and learning systems, 31(9):3456–3470, 2019. - Tsukimoto [2000] H. Tsukimoto. Extracting rules from trained neural networks. IEEE Transactions on Neural Networks, 11(2):377–389, 2000. doi: 10.1109/72.839008. - UZUN et al. [2016] Y. UZUN, H. ARIKAN, and G. TEZEL. Rule extraction from training artificial neural network using variable neighbourhood search for wisconsin breast cancer. Journal of Multidisciplinary Engineering Science and Technology, 3(8), 2016. URL http://www.jmest.org/wp-content/uploads/JMESTN42351743.pdf. - Vilone and Longo [2020] G. Vilone and L. Longo. Explainable artificial intelligence: a systematic review, 2020. - Wu et al. [2018] M. Wu, M. Hughes, S. Parbhoo, M. Zazzi, V. Roth, and F. Doshi-Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. doi: 10.1609/aaai.v32i1.11501. URL https://ojs.aaai.org/index.php/AAAI/article/view/11501. - Yedjour [2020] D. Yedjour. Extracting classification rules from artificial neural network trained with discretized inputs. Neural Processing Letters, 52:2469–2491, 2020. doi: 10.1007/s11063-020-10357-x. - Yedjour and Benyettou [2018] D. Yedjour and A. Benyettou. Symbolic interpretation of artificial neural networks based on multiobjective genetic algorithms and association rules mining. Applied Soft Computing, 72:177–188, 2018. ISSN 1568-4946. doi: 10.1016/j.asoc.2018.08.007. URL https://www.sciencedirect.com/science/article/pii/S1568494618304551. - Zarlenga et al. [2021] M. E. Zarlenga, Z. Shams, and M. Jamnik. Efficient decompositional rule extraction for deep neural networks, 2021. URL https://arxiv.org/abs/2111.12628. - Zhou et al. [2000] Z.-H. Zhou, S.-F. Chen, and Z.-Q. Chen. A statistics based approach for extracting priority rules from trained neural networks. In Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, volume 3, pages 401–406, 2000. doi: 10.1109/IJCNN.2000.861337. - Zilke et al. [2016] J. R. Zilke, E. Loza Mencía, and F. Janssen. Deepred – rule extraction from deep neural networks. In T. Calders, M. Ceci, and D. Malerba, editors, Discovery Science, pages 457–473, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46307-0. doi: 10.1007/978-3-319-46307-0_29.