2501.15919v2

# Does Functional Package Management Enable Reproducible Builds at Scale? Yes. **Authors**: Julien Malka, Stefano Zacchiroli, Théo Zimmermann > LTCI, Télécom ParisInstitut Polytechnique de ParisPalaiseau, Francejulien.malka@telecom-paris.fr > LTCI, Télécom ParisInstitut Polytechnique de ParisPalaiseau, Francestefano.zacchiroli@telecom-paris.fr > LTCI, Télécom ParisInstitut Polytechnique de ParisPalaiseau, Francetheo.zimmermann@telecom-paris.fr \ExecuteBibliographyOptions swhid=true \addbibresource bibliography.bib Abstract Reproducible Builds (R-B) guarantee that rebuilding a software package from source leads to bitwise identical artifacts. R-B is a promising approach to increase the integrity of the software supply chain, when installing open source software built by third parties. Unfortunately, despite success stories like high build reproducibility levels in Debian packages, uncertainty remains among field experts on the scalability of R-B to very large package repositories. In this work, we perform the first large-scale study of bitwise reproducibility, in the context of the Nix functional package manager, rebuilding $709\,816$ packages from historical snapshots of the nixpkgs repository, the largest cross-ecosystem open source software distribution, sampled in the period 2017–2023. We obtain very high bitwise reproducibility rates, between 69 and 91% with an upward trend, and even higher rebuildability rates, over 99%. We investigate unreproducibility causes, showing that about 15% of failures are due to embedded build dates. We release a novel dataset with all build statuses, logs, as well as full “diffoscopes”: recursive diffs of where unreproducible build artifacts differ. Index Terms: reproducible builds, functional package management, software supply chain, reproducibility, security I Introduction Free and open source software (FOSS) is a great asset to build trust in a computing system, because one can audit the source code of installed components to determine if their security is up to one’s standards. However trusting the source code of the components making up a system is not enough to trust the system itself: before a program can be run on a user machine, it is typically built A term that we will also use in this work for interpreted programs, where it is the runtime environment that has to be built. to obtain an executable artifact, and then distributed onto the target system, involving a set of processes and actors generally referred to as the software supply chain. In recent years, large scale attacks like Solarwinds [alkhadra_solar_2021] or the xz backdoor [noauthor_nvd_nodate] have specifically targeted the software supply chain, underlying the importance of measures to increase its security and also triggering policy response in the European Union and the United States of America [noauthor_cyber_2022, house_executive_2021]. The particular effectiveness of these attacks is due to the difficulty to analyze binary artifacts in order to understand how they might act on the system, hence increasing the need for tooling that provide traceability from executable binaries to their source code. Reproducible builds (R-B) —the property of being able to obtain the same, bitwise identical, artifacts from two independent builds of a software component—is recognized as a promising way to increase trust in the distribution phase of binary artifacts [lamb_reproducible_2022]. Indeed, if a software is bitwise reproducible, a user may require several independent parties to reach a consensus on the result of a compilation before downloading the built artifacts from one of them, effectively distributing the trust in these artifacts between those parties. For an attacker wanting to compromise the supply chain of that component, it is no longer sufficient to compromise only one of the involved parties. Build reproducibility is however not easy to obtain in general, due to non-determinism in the build processes, documented both by practitioners and researchers [lamb_reproducible_2022, bajaj_unreproducible_2023]. The Reproducible Builds [noauthor_reproducible_2023] project has since 2015 worked to increase bitwise reproducibility throughout the FOSS ecosystems, by coming up with fixes for compilers and other toolchain components, working closely with upstream projects to integrate them. Unfortunately, a recent study [fourne_its_2023] which interviewed 24 R-B experts still concluded that there is “ a perceived impracticality of fully reproducible builds due to workload, missing organizational buy-in, unhelpful communication with upstream projects, or the goal being perceived as only theoretically achievable ” and that “ much of the industry believes [R-B] is out of reach ”. While there exist some successful examples of package sets with high reproducibility levels like Debian, which consistently achieves a reproducibility rate of more than 95% [noauthor_overview_nodate], those good performances should be put in perspective with the strict quality policies applied in Debian and the relatively limited size of the package set. Uncertainty remains among field experts about the scalability of this approach to larger software distributions. Nixpkgs is the largest cross-ecosystem FOSS distribution, totaling as of October 2024 about $100\,000$ packages. Based on the Repology rankings https://repology.org, accessed Oct. 2024. It includes components from a large variety of software ecosystems, making it an interesting target to study bitwise reproducibility at scale. Nixpkgs is built upon Nix, the seminal implementation of the functional package management (FPM) model [dolstra_purely_2006]. It is generally believed that the FPM model is effective to obtain R-B: FPM packages are pure functions (in the mathematical sense) from build- and run-time dependencies to build artifacts, described as “build recipe”s that can be executed locally by the package manager. Components are built in a sandboxed environment, disallowing access to unspecified dependencies, even if they are present on the system. Previous work has highlighted that this model allows to reproduce build environments both in space and time [malka_reproducibility_2024], a necessary property for build reproducibility. Additionally, nixpkgs’ predefined build processes implement best practices to ensure build reproducibility, like setting the SOURCE_DATE_EPOCH environment variable [noauthor_source_date_epoch_nodate] or automatically verifying that the build path does not appear in the built artifacts [noauthor_build_nodate]. Despite the potential for insightful distribution-wide build reproducibility metrics, nixpkgs limits its monitoring to the narrow set of packages included in the minimal and gnome-based ISO images [noauthor_nixos_nodate], where a reproducibility rate higher than 95% is consistently reported. Contributions In this work, we perform the first ever large scale empirical study of bitwise reproducibility of FOSS going back in time, rebuilding historical packages from evenly spaced snapshots of the nixpkgs package repository taken every 4.1 months from 2017 to 2023. With this experiment, we answer the following research questions: - RQ1: What is the evolution of bitwise reproducible packages in nixpkgs between 2017 and 2023? How does the reproducibility rate evolve over time? Are unreproducible packages eventually fixed? Do reproducible packages remain reproducible? - RQ2: What are the unreproducible packages? Are they concentrated in specific ecosystems? Are critical packages more likely to be reproducible? - RQ3: Why are packages unreproducible? Is large-scale identification of common causes possible? - RQ4: How are unreproducibilities fixed? Are they fixed by specific patches or as part of larger package updates? Are the fixes intentional or accidental? Besides, we use our experiment to replicate and extend previous results [malka_reproducibility_2024], leading to an additional research question: - RQ0: Does Nix allow rebuilding past packages reliably (even if not bitwise reproducibly)? Results Thanks to this large-scale experiment, we are able to establish for the first time that bitwise reproducibility is achievable at scale, with reproducibility rates ranging from 69% to 91% over the period 2017–2023, despite a continuous increase in the number of packages in nixpkgs. We highlight the wide variability in reproducibility rates across ecosystems packaged in nixpkgs, and show the significant impact that some core packages can have on the overall reproducibility rate of an ecosystem. We estimate the prevalence of some common causes of non-reproducibility at a large scale for the first time, showing that about 15% of failures are due to embedded build dates. As part of this work, we introduce a novel dataset containing build logs and metadata of over $709\,000$ package builds, and more than $114\,000$ occurrences of non-reproducibility with full artifacts including “diffoscopes”, i.e., recursive diffs of where unreproducible build artifacts differ. Ample room for further research is left open by the dataset, including exploiting the build logs, or applying more complex heuristics or qualitative research to the diffoscopes. Paper structure Section II presents the related work. Section III gives some background that is required to understand the experiment, whose methodology is then presented in Section IV. Some descriptive statistics about the dataset are presented in Section V, and the results to our RQs in Section VI. We discuss them in Section VII, and the threats to validity in Section VIII, concluding in Section IX. II Related work II-A Reproducible builds (R-B) R-B are a relatively recent concept, which has been picked up and developed mostly by practitioners from Linux distributions and upstream maintainers. The Reproducible Builds project [noauthor_reproducible_2023] has been the main actor in the area. The project has produced a definition of R-B, best practices to achieve them, and tools to monitor the reproducibility of software distributions, and debug unreproducibilities (the diffoscope). Besides, R-B have picked the interest of the academic community, with a growing number of papers on the topic. R-B for the security of the software supply chain R-B are often seen as a way to increase the security of the software supply chain [lamb_reproducible_2022]. Torres-Arias et al. provide a framework to enforce the integrity of the software supply chain for which they demonstrate an application to enforce R-B [torres-arias_-toto_2019]. Our paper does not contribute directly to this line of research but, by demonstrating the feasibility of R-B at scale, it strengthens the case of the approach. Techniques for build reproducibility In a series of articles [ren_automated_2018, ren_root_2020, ren_automated_2022], Ren et al. devised a methodology to automate the localization of sources of non-reproducibility in build processes and to automatically fix them, using a database of common patches that are then automatically adapted and applied. An alternative technique to achieve build reproducibility is proposed by Navarro et al. [navarro_leija_reproducible_2020]. They propose “reproducible containers” that are built in a way that makes the build process fully deterministic, at the expense of performance. FPMs such as Nix [dolstra_purely_2006] and Guix [courtes_functional_2013] are also presented as a way to achieve R-B. Malka et al. [malka_reproducibility_2024] showed that Nix allows reproducing past build environment reliably, as well as rebuilding old packages with high confidence, but they do not address the question of bitwise reproducibility, which we do with this work. Relaxing the bitwise reproducibility criterion Because of the difficulty (real or perceived) to achieve bitwise reproducibility, some authors have proposed to relax the criterion to a more practical one. For instance, accountable builds [poll_analyzing_2021] aim to distinguish between differences that can be explained (accountable differences) or not (unaccountable differences). Our work highlights that bitwise reproducibility is achievable at scale in practice, and thus that relaxing the reproducibility criterion may not be necessary after all. Empirical studies of R-B Some other recent academic works have empirically studied R-B in the wild. Two papers from 2023 [butler_business_2023, fourne_its_2023] looked into business adoption of R-B and perceived effectiveness through interviews. Bajaj et al. [bajaj_unreproducible_2023] mined historical results from R-B tracking in Debian to investigate causes, fix time, and other properties of unreproducibility in the distribution. Our work is similar, but instead of relying on historical R-B tracking, we actually rebuild packages and compare them bitwise to historical build results. When we report on packages being reproducible, it means they have stood the test of time. It also allows us to provide more detailed information on the causes of unreproducibility, in particular by generating diffoscopes and saving them for future research as part of our dataset; whereas diffoscopes from the Debian R-B tracking are not preserved in the long-term. Finally, Bajaj et al. used issue tracker data from the R-B project to identify the most common causes of non-reproducibility, possibly introducing a sampling bias since only root causes that were identified by Debian developers are counted in their statistics. In our work, we try to avoid this bias by performing a large-scale automatic analysis of diffoscopes to automatically identify the prevalence of a selection of causes of non-reproducibility. While we present heuristics comparable to some of the causes identified in Bajaj et al. ’s taxonomy, we derive them from empirical data rather than relying on pre-labeled data from the Debian issue tracker. The only other work that performed an experimental study of R-B investigated the impact of configuration options [randrianaina_options_2024]. Contrary to them, we rebuild historical versions of packages in their default configuration. Combining the historical snapshot approach of our work with their approach of varying the configuration options could be an interesting future work. II-B Linux distributions and package ecosystems Besides R-B, our work also relates to the literature on Linux distributions and package ecosystems. The nixpkgs repository being the largest cross-ecosystem software distribution, we are able to compare properties of packages across ecosystems. Several previous works have compared package ecosystems (e.g., [decan_empirical_2019]). For an overview of recent research on package ecosystems, see Mens and Decan [mens_overview_2024]. More specifically, nixpkgs is the basis of the NixOS Linux distribution. Linux distributions have a long history of being studied by the research community. Recently, Legay et al. [legay_quantitative_2021] measured the package freshness in Linux distributions. While this is not the topic of this work, our dataset could be used, e.g., to study how frequently packages are updated in nixpkgs. III Background We provide in this section some background knowledge about Nix and R-B, which is required to understand the details of our experiments. III-A The FPM model and the Nix store The FPM model applies the idea of functional programming to package management. Nix packages are viewed as pure functions from their inputs (source code, dependencies, build scripts) to their outputs (binaries, documentation, etc.). Any change to the inputs should produce a different package version. Nix allows multiple versions of the same package to be built and coexist on the same system. To that end, Nix stores build outputs in input-addressed directories (using a hashing function of the inputs) in the Nix store, usually located in the /nix/store directory on disk. Figure 1 shows an example of a Nix packaging expression (a Nix recipe) for the htop package. ⬇ {stdenv, fetchFromGitHub, ncurses, autoreconfHook}: stdenv. mkDerivation rec { pname = "htop"; version = "3.2.1"; src = fetchFromGitHub { owner = "htop-dev"; repo = "htop"; rev = version; sha256 = "sha256-MwtsvdPHcUdegsYj9NGyded5XJQxXri1IM1j4gef1Xk="; }; nativeBuildInputs = [ autoreconfHook ]; buildInputs = [ ncurses ]; }; } Figure 1: Example Nix expression for the htop package. III-B Nix evaluation-build pipeline Building binary outputs from a Nix package recipe is a two-step process. First, Nix evaluates the expression and transforms it into a derivation, an intermediary representation in the Nix store containing all the necessary information to run the build process. In particular, the derivation contains ahead of time the (input-addressed) output path, that is the exact location in the Nix store where the build artifacts will be stored if that derivation were to be built. Then, given the derivation file as input, the nix-build command performs the build, creating the pre-computed output path in the Nix store upon completion. In the same fashion as other Linux distributions, Nix packages may produce multiple outputs (a main output with binaries, one with documentation, etc.). Each output has its own directory in the Nix store, and building the derivation from source systematically produces all its outputs. III-C Path substitution and binary caches Alternatively to building from source, Nix offers the option to download prebuilt artifacts from third party binary caches, which are databases populated with build outputs generated by Nix. Binary caches are indexed by output paths, making it possible for Nix to check for the presence of a precompiled package in a configured cache after the evaluation phase. https://cache.nixos.org is the official cache for the Nix community and most Nix installations come configured to use it as a trusted cache. III-D The nixpkgs continuous integration Hydra [dolstra_nix_nodate] is the continuous integration (CI) platform for the nixpkgs project. At regular intervals in time, it fetches the latest version of nixpkgs’ git master branch and evaluates the pkgs/top-level/release.nix file embedded in the repository. This evaluation yields a list of derivations (or jobs) that are then built by Hydra: one derivation for each of the $≈\,$ $100\,000$ packages contained in nixpkgs nowadays. Upon success of a predefined subset of these jobs, the revision is deemed valid and all the built artifacts are uploaded to the official binary cache to be available to end users. III-E Testing bitwise reproducibility with Nix Nix embarks some minimal tooling to test the reproducibility of a given derivation in the form of a --check flag passed to the nix-build command. To check for bitwise reproducibility, Nix needs a reference that it will try to acquire from one of the configured caches, or fail if not possible. Nix then acquires the build environment of the derivation under consideration, builds the derivation, and compares each of the outputs of the derivation against the local version. The --keep-failed flag can be used to instruct Nix to keep the unreproducible outputs locally for further processing. For our experiment, we alter the behavior of nix-build --check to prevent it from failing early as soon as one unreproducible output is detected. III-F Diffoscope Diffoscope [noauthor_diffoscope_nodate] is a tool developed and maintained by the Reproducible Builds project that aims to simplify the analysis of differences between software artifacts. It is able to recursively unpack binary archives and automatically use ecosystem specific diffing tools to allow for better understanding of what makes two software artifacts different. It generates HTML or JSON artifacts—also called diffoscopes —that can be either interpreted by humans or automatically processed. IV Methodology <details> <summary>extracted/6418685/medias/pipelinev2.png Details</summary>  ### Visual Description ## Flowchart: CI/CD Pipeline Analysis and Rebuild Process ### Overview The flowchart illustrates a comparative analysis of historical CI builds versus rebuilt packages, focusing on reproducibility and build success. It includes three main sections: 1. **Historical CI Builds** (pink) 2. **Our Rebuilds** (green) 3. **Our Analysis** (blue) The process begins with sampling 17 revisions from the Nixpkgs git repository and evaluates reproducibility through artifact comparison and heuristic analysis. --- ### Components/Axes #### Labels and Flow: - **Historical CI Builds (Top-Left, Pink Boxes):** - Evaluation of `release.nix` by Hydra - Build all packages using Hydra builders - Conditional upload of artifacts to Nixpkgs' binary cache (if build succeeds) - **Our Rebuilds (Bottom-Left, Green Boxes):** - Evaluation of `release.nix` by `nix-eval-jobs` - Computation of evaluation closure and filtering out packages not in cache - Build all packages using Buildbot infrastructure - Conditional storage of artifacts (if success) or failure status (if failure) - **Our Analysis (Top-Right, Blue Boxes):** - Compare artifacts (from historical vs. rebuilt builds) - **Identical artifacts:** Status = reproducible - **Different artifacts:** Status = buildable but not reproducible - If package is fixed later: - Bisect to find fixing commit - Manually analyze commit to infer intent - Generate Diffscopes - Run heuristics to categorize them #### Colors and Placement: - **Pink:** Historical CI Builds (top-left quadrant) - **Green:** Our Rebuilds (bottom-left quadrant) - **Blue:** Our Analysis (top-right quadrant) - Arrows indicate flow direction, with dotted lines for conditional paths. --- ### Detailed Analysis #### Historical CI Builds: 1. **Evaluation of `release.nix` by Hydra** → 2. **Build all packages using Hydra builders** → - If build succeeds: Upload artifacts to Nixpkgs' binary cache. #### Our Rebuilds: 1. **Evaluation of `release.nix` by `nix-eval-jobs`** → 2. **Computation of evaluation closure and filtering out packages not in cache** → 3. **Build all packages using Buildbot infrastructure** → - Success: Store artifacts. - Failure: Status = failure. #### Our Analysis: 1. **Compare artifacts** → - **Identical:** Status = reproducible. - **Different:** Status = buildable but not reproducible. - If package is fixed later: - Bisect to find fixing commit. - Manually analyze commit to infer intent. - Generate Diffscopes → Run heuristics to categorize them. --- ### Key Observations 1. **Reproducibility Focus:** The process emphasizes identifying whether rebuilt packages match historical builds (reproducible) or diverge (not reproducible). 2. **Conditional Logic:** Artifact upload/storage depends on build success/failure. 3. **Heuristic Analysis:** Diffscopes and manual commit analysis are used to diagnose reproducibility issues. 4. **Sampling:** Only 17 revisions are sampled from the Nixpkgs git repository for evaluation. --- ### Interpretation - **Purpose:** The flowchart demonstrates a systematic approach to validate CI/CD pipeline reliability by comparing historical and rebuilt artifacts. - **Key Relationships:** - Historical builds (Hydra) serve as the baseline for reproducibility checks. - Rebuilds (Buildbot) are tested against this baseline. - Analysis steps (bisecting, Diffscopes) address discrepancies to improve future builds. - **Notable Patterns:** - Reproducibility is prioritized over mere build success. - Manual intervention (commit analysis) is reserved for unresolved cases. - **Anomalies:** - No explicit metrics (e.g., success rates) are provided, leaving quantitative validation incomplete. - The flowchart assumes access to Nixpkgs' binary cache and Buildbot infrastructure, which may not be universally applicable. This diagram provides a high-level workflow but lacks granular data (e.g., failure rates, time metrics) for deeper insights. </details> Figure 2: Description of our build and analysis pipeline. Our build and analysis pipeline is summarized in Figure 2. IV-A Reproducibility experiments IV-A 1 Revision sampling We start from the 200 nixpkgs revisions selected by Malka et al. [malka_reproducibility_2024] in the period July 2017–April 2023. Since building revisions, as opposed to just evaluating them, is very computationally intensive, it was not feasible to build all 200 revisions. Also, it was difficult to correctly estimate how many revisions we could build, due to the ever-growing number of packages in each revision. We hence applied dichotomic sampling: we first build the most recent revision, then the oldest one, then the one in the middle of them, and so on always picking the one in the middle of the largest time interval when choosing. After 17 revisions built, we obtain a regularly spaced sample set of nixpkgs revisions, with one sampled revision every 4.1 months. On average, each revision corresponds to building more than 41 thousand packages (see Figure 3 for details, discussed later). To perform our builds, we used a distributed infrastructure based on Buildbot [noauthor_buildbot_nodate], a Python CI pipeline framework. Our infrastructure has two types of machines: a coordinator and multiple builders. The coordinator is in charge of distributing the workload and storing data that must be persisted, while the builders are stateless and perform workloads sent by the coordinator. During the course of the experiment (from June to October 2024) the set of builders we used was composed of shared bare-metal machines running various versions of Ubuntu and Fedora and our coordinator was a virtual machine running Ubuntu. Note that to perform our bitwise reproducibility checks, we compare to the historical results coming from Hydra, that uses builders that, at the time, ran older versions of Nix than the ones we used on our builders. Further details on the operating systems, Nix versions and kernel versions that we used on builders can be found in the replication package. IV-A 2 Evaluation and preprocessing For each revision considered, the coordinator first evaluates pkgs/top-level/release.nix (containing the list of jobs built by Hydra for this revision) using nix-eval-jobs, a standalone and parallel Nix evaluator similar to the one used on Hydra. The outcome of this operation is a list of derivations. The release.nix file is human crafted and does not contain all the dependencies of the listed packages, even though they are built by Hydra along the way. Since we are interested in testing the reproducibility of the entire package graph built by Hydra, we post-process the list of jobs obtained after the evaluation phase to include all intermediary derivations by walking through the dependency graph of each derivation. During this post-processing, we also check that the derivation outputs are present in the official Nix binary cache. This is required to compare our build outputs with historical results for bitwise reproducibility. Derivations missing from the cache can indicate that they historically failed to build, although there can be other reasons for their absence. IV-A 3 Building To build a job $A$ and test the reproducibility of its build outputs, the builder uses the nix-build --check command, as described in Section III-E. This means that we always assume that $A$ ’s build environment is buildable and always fetch it from the cache. This allows all the derivations in the sample set to be built and checked independently and in parallel, irrespective of where they are located in the package dependency graph. Note that the source code of packages to build is part of the build environment. Relying on the NixOS cache hence avoids incurring into issues such as source code disappearing from the original upstream distribution place. Investigating how much of NixOS can be rebuilt without relying on the cache is an interesting research question, recently explored for Guix [courtes-2024-guix-archiving], but out of scope for this paper. After each build, we classify the derivation as either building reproducibly, building but not reproducibly or not building. We save the build metadata and the logs, and when available we download and store the historical build logs from the nixpkgs binary cache. Finally, for every unreproducible output path, we store both the historical artifacts and our locally built ones, for comparison purposes. IV-B Ecosystem identification and package tracking To answer RQ1, RQ2 and RQ4, we need to be able to discriminate packages by provenance ecosystem, and track them over time to follow their evolution. To categorize packages by ecosystem, we rely on the first component of the package name when it has several components (for example a package named haskellPackages.network is sorted into the Haskell ecosystem). Sometimes, there are several co-existing versions of an ecosystem in a given nixpkgs revision (for example python37Packages and python38Packages being present in the same revision), and sometimes the name of the ecosystem is modified between successive nixpkgs revisions. Therefore, some deduplication step is necessary. The first and last authors performed this step manually by inspecting the 144 ecosystems from the 17 nixpkgs revisions considered, ordered alphabetically, and deciding which ones to merge independently, then checking the consistency of their results, discussing the few differences (missed merges, or false positives) and reaching a consensus. For instance, the following ecosystems were merged into a single one: php56Packages, php70Packages, …, php82Packages, phpExtensions, phpPackages and phpPackages-unit. To deduplicate packages appearing in several versions of the same ecosystem, we order by version (favoring the most recent one) and consider any package set without a version number as having a higher priority (since it is the default one in the considered revision, as chosen by the nixpkgs maintainers). IV-C Comparison with the minimal ISO image As part of RQ2, we investigate the difference of reproducibility rate between critical packages whose reproducibility is monitored and the rest of the package set. We are also interested in knowing whether observing the reproducibility health of this subset of packages gives a good enough information on the state of the rest of the project. The minimal and gnome-based ISO images are considered critical subsets of packages and benefit from a community-maintained reproducibility monitoring. We study the minimal ISO image because it contains a limited amount of core packages. We evaluate the Nix expression associated with the image, compute its runtime closure (the set of packages included in the image) and match it with the packages of our dataset to infer their reproducibility statuses. IV-D Analyzing causes of unreproducibility using diffoscopes Analyzing causes of unreproducibility is a tricky debugging activity, usually carried out by practitioners (in particular, by Linux distribution maintainers and members of the Reproducible Builds project). Some automatic fault localization methods have been proposed [ren_root_2020], but they rely on instrumenting the build, while we have to run the Nix builds unchanged to avoid introducing biases. For each unreproducible output, we run diffoscope with a 5-minute timeout, yielding a dataset of $86\,476$ diffoscopes. We then investigate whether we can use our large dataset of diffoscopes for automatic detection of causes of non-reproducibility. The diffoscope tool was mainly designed to help human debugging, but it also supports producing a JSON output, which can then be machine processed. We wish to explore heuristics that can be applied at the line level, so we recurse through diffoscope structures until leaf nodes, which are diffs in unified diff format. We randomly draw one added line from $10\,000$ diffoscopes, sort them by similarity to ease visual inspection, and manually inspect them to derive relevant heuristics. We then run these heuristics on the full diffoscope dataset to determine the proportion of packages impacted by each cause (multiple causes can apply to the same package). The first and last author then evaluate the precision of each these heuristics by manually counting false positives in samples of matched lines for each heuristic. IV-E Automatic identification of reproducibility fixes To investigate fixes to unreproducibilities, for each unreproducible package that becomes reproducible, we run an automatic bisection process to find the first commit that fixes the reproducibility. By looking into the corresponding pull request on GitHub, we check if the maintainers provide information on why this fixes a reproducibility issue, or link to a corresponding bug report. In particular, we are interested to check how often the maintainers are aware that their commit is a reproducibility fix (as opposed to a routine package update, which embeds a reproducibility fix that would have been crafted by the upstream maintainers). We start from the set of packages from all revisions, and we look specifically at packages that change status from unreproducible to reproducible in two successive revisions. These are our candidate packages (and “old” and “new” commits) for the bisection process. Since we perform the bisection process on a different runner and at a different time compared to the dataset creation, it can happen that we cannot reproduce the status (reproducible or unreproducible) of some builds. Therefore, before starting the bisection, we verify that we obtain consistent results on the “old” and the “new” revision. Then, for those which behave as expected, we start an automatic git bisect process. The script used for the automatic git bisect checks for the reproducibility status of the build to mark the selected commit as “old” or “new”. Commits that fail to build or are not available in cache are marked as “skipped”. We use git bisect with the --first-parent flag because intermediate pull request commits are typically not in cache. For the qualitative analysis, we first group packages by fixing commit, as seeing all packages fixed by a given commit gives valuable information that might help to understand the reproducibility failure being fixed. We then randomly sample fixes and open their commit page on GitHub, locating the corresponding pull request. We manually inspect the pull request (description, commit log, code changes) to first confirm that the bisect phase successfully identified the commit that fixed the reproducibility failure. It may be the case that after careful inspection, the change looks unrelated to the package being fixed (the bisection process can give incoherent results in case of a flaky reproducibility issue or because the package changed status several times between two data points) in which case we discard it. Once we have confirmed that the identified commit is correct, we check whether the commit authors indicate that they are fixing a reproducibility issue and if the commit is a package update or another change. We analyze 100 randomly sampled reproducibility fixes and report our findings. Additionally, we perform the same analysis on the 15 commits that fix the most packages (from $3052$ down to 27 packages fixed) to find potential differences of behavior of the contributors for those larger-scale fixes. V Dataset The main result of running the pipeline of Figure 2 is a large-scale dataset of historical package rebuilds, including (re)build information, bitwise reproducibility status and, in case of non-reproducibility, generated diffoscopes. In this paper, we use the dataset to answer our stated research questions, but many other research questions could be addressed using the dataset, including more in-depth analysis of non-reproducibility causes. We make the dataset available to the research and technical community to foster further exploration on the topic. In the remainder of this section, we provide some descriptive statistics of the dataset. <details> <summary>extracted/6418685/paper-images/package-evolution.png Details</summary>  ### Visual Description ## Stacked Area Chart: Number of Packages Over Time (2018–2023) ### Overview The chart visualizes the cumulative growth of three categories of packages ("Deduplicated evaluation," "Evaluation of release.nix," and "Evaluation closure of release.nix") from 2018 to 2023. The total number of packages increases steadily, with significant acceleration in 2023. ### Components/Axes - **X-axis (Date)**: Years 2018–2023, labeled at 2-year intervals. - **Y-axis (Number of packages)**: Ranges from 0 to 80,000, with increments of 10,000. - **Legend**: Located at the bottom, with three color-coded categories: - **Blue**: Deduplicated evaluation - **Red**: Evaluation of release.nix - **Green**: Evaluation closure of release.nix - **Stacking**: Categories are stacked vertically, with blue at the base, red in the middle, and green on top. ### Detailed Analysis 1. **Deduplicated evaluation (Blue)**: - Starts at ~12,000 packages in 2018. - Gradual increase to ~42,000 by 2023. - Consistent upward trend with minor fluctuations (e.g., slight dip in 2019). 2. **Evaluation of release.nix (Red)**: - Begins at ~3,000 packages in 2018. - Peaks at ~10,000 in 2021. - Declines to ~5,000 by 2023. - Sharp drop between 2021–2023 suggests reduced reliance or consolidation. 3. **Evaluation closure of release.nix (Green)**: - Starts at ~5,000 packages in 2018. - Accelerates to ~15,000 in 2021. - Surges to ~30,000 in 2023. - Steep rise in 2023 indicates a major shift or new release cycle. 4. **Total Packages**: - 2018: ~20,000 (blue + red + green). - 2023: ~75,000 (blue + red + green). - Growth accelerates post-2021, with a 50% increase from 2022–2023. ### Key Observations - **Dominance of Deduplicated evaluation**: Blue consistently represents the largest share (~50–60% of total packages). - **Green surge in 2023**: The green category’s rapid growth in 2023 suggests a pivotal event (e.g., release.nix closure or major update). - **Red category decline**: The red line’s drop after 2021 may reflect reduced evaluation activity or integration into other categories. - **Total growth trajectory**: Packages grow from ~20k to ~75k, a 275% increase over 5 years. ### Interpretation The data highlights a shift in evaluation practices over time. The steady growth of "Deduplicated evaluation" indicates its foundational role, while the rise of "Evaluation closure of release.nix" in 2023 suggests a strategic focus on closure processes, possibly due to a major release or architectural change. The decline in "Evaluation of release.nix" could signal consolidation, automation, or migration to newer workflows. The total package growth underscores increasing activity, likely tied to evolving software development practices or organizational scaling. The 2023 spike in green packages warrants further investigation into release.nix’s role in this period. </details> Figure 3: Evolution of the number of packages in each nixpkgs revision (as defined by release.nix), in their evaluation closure and after deduplicating ecosystem copies. The dataset spans $709\,816$ package builds coming from 17 nixpkgs revisions, built over a total of $14\,296$ hours. From those builds, $548\,390$ are coming directly from the release.nix file and can be tracked by name. They correspond to $58\,103$ unique packages that appear over the span of sampled revisions. As can be seen on Figure 3, the number of packages listed to be built by Hydra increased from $13\,527$ in 2017 to $53\,432$ in 2023. Ecosystems can be present in multiple versions. On average, deduplicating packages in multiple ecosystem copies decreases their number by 21% while adding the evaluation closure of the release.nix file increases the number of jobs by 29%. <details> <summary>extracted/6418685/paper-images/ecosystems.png Details</summary>  ### Visual Description ## Stacked Area Chart: Job Growth Across Ecosystems (2018-2023) ### Overview The chart visualizes the cumulative growth of jobs across eight technology ecosystems from 2018 to 2023. The y-axis represents the number of jobs (in thousands), while the x-axis spans five years. Each ecosystem is represented by a distinct color, with values stacked vertically to show total job growth over time. ### Components/Axes - **X-axis (Date)**: Labeled "Date" with annual markers from 2018 to 2023. - **Y-axis (Number of jobs)**: Labeled "Number of jobs" with increments of 5,000 up to 45,000. - **Legend**: - **Base** (red) - **Haskell** (green) - **Python** (purple) - **VimPackages** (pink) - **EmacsPackages** (cyan) - **Xorg** (orange) - **Lisp** (light green) - **Other ecosystems** (blue) ### Detailed Analysis 1. **Base Ecosystem (Red)**: - **2018**: ~7,000 jobs - **2019**: ~9,000 jobs - **2020**: ~11,000 jobs - **2021**: ~13,000 jobs - **2022**: ~16,000 jobs - **2023**: ~20,000 jobs - **Trend**: Steady linear growth with no plateaus. 2. **Haskell (Green)**: - **2018**: ~5,000 jobs - **2019**: ~6,000 jobs - **2020**: ~7,000 jobs - **2021**: ~8,000 jobs - **2022**: ~10,000 jobs - **2023**: ~14,000 jobs - **Trend**: Accelerated growth post-2021. 3. **Python (Purple)**: - **2018**: ~3,000 jobs - **2019**: ~4,000 jobs - **2020**: ~5,000 jobs - **2021**: ~7,000 jobs - **2022**: ~10,000 jobs - **2023**: ~15,000 jobs - **Trend**: Exponential growth starting in 2021. 4. **VimPackages (Pink)**: - **2018**: ~1,000 jobs - **2019**: ~1,500 jobs - **2020**: ~2,000 jobs - **2021**: ~2,500 jobs - **2022**: ~3,000 jobs - **2023**: ~3,500 jobs - **Trend**: Consistent but slower growth. 5. **EmacsPackages (Cyan)**: - **2018**: ~2,000 jobs - **2019**: ~2,500 jobs - **2020**: ~3,000 jobs - **2021**: ~3,500 jobs - **2022**: ~4,000 jobs - **2023**: ~4,500 jobs - **Trend**: Gradual linear increase. 6. **Xorg (Orange)**: - **2018**: ~1,000 jobs - **2019**: ~1,200 jobs - **2020**: ~1,500 jobs - **2021**: ~1,800 jobs - **2022**: ~2,000 jobs - **2023**: ~2,500 jobs - **Trend**: Steady but minimal growth. 7. **Lisp (Light Green)**: - **2018**: ~800 jobs - **2019**: ~1,000 jobs - **2020**: ~1,200 jobs - **2021**: ~1,500 jobs - **2022**: ~2,000 jobs - **2023**: ~3,000 jobs - **Trend**: Accelerated growth post-2021. 8. **Other Ecosystems (Blue)**: - **2018**: ~1,000 jobs - **2019**: ~1,500 jobs - **2020**: ~2,000 jobs - **2021**: ~2,500 jobs - **2022**: ~3,000 jobs - **2023**: ~4,000 jobs - **Trend**: Linear growth with a slight dip in 2023. ### Key Observations - **Dominance of Base Ecosystem**: The red segment (base) consistently represents the largest portion of jobs, growing from ~7,000 in 2018 to ~20,000 in 2023. - **Python and Haskell Surge**: Both ecosystems show exponential growth post-2021, with Python overtaking Haskell in 2023 (~15,000 vs. ~14,000 jobs). - **2023 Spike**: A sharp increase in total jobs (~45,000) in 2023, driven by Python, Haskell, and Lisp. - **Stability of Legacy Ecosystems**: VimPackages, EmacsPackages, and Xorg show minimal growth, remaining below 5,000 jobs each. ### Interpretation The data suggests a shift toward modern ecosystems like Python and Haskell, which have gained significant traction in recent years. The base ecosystem’s steady growth indicates its foundational role, while the 2023 spike may reflect industry-wide adoption of emerging technologies. Legacy ecosystems (VimPackages, EmacsPackages, Xorg) remain niche, highlighting their limited scalability. The dip in "Other ecosystems" in 2023 could signal consolidation or reduced diversity in smaller niches. Overall, the chart underscores the dynamic evolution of tech job markets, with Python and Haskell emerging as dominant forces. </details> Figure 4: Evolution of the size of the nine most popular software ecosystems in nixpkgs, the packages whose ecosystem is undetermined (base), and the packages from other ecosystems (only packages listed in release.nix). Figure 4 shows the evolution of the top 9 ecosystem sizes in nixpkgs, plus the base namespace. 61.7% of packages belong to an ecosystem, while the rest live in nixpkgs base namespace. The three largest ecosystems are Haskell, Python and Perl, which together account for 42.4% of the packages. <details> <summary>extracted/6418685/paper-images/strates.png Details</summary>  ### Visual Description ## Stacked Area Chart: Number of Jobs by Category (2018-2023) ### Overview The chart visualizes the evolution of job categories over six years (2018-2023), with cumulative totals represented by stacked colored layers. The y-axis shows absolute job counts (0-45,000), while the x-axis spans 2018-2023. Each color corresponds to a specific job category, with values labeled at key points. ### Components/Axes - **X-axis (Date)**: Years 2018-2023, labeled at annual intervals. - **Y-axis (Number of Jobs)**: Scale from 0 to 45,000 in 5,000 increments. - **Legend**: Right-aligned, mapping colors to job categories: - **Blue**: 1377834 (13,778 in 2018) - **Purple**: 1476353 (14,764 in 2018) - **Red**: 1416607 (2,345 in 2018) - **Orange**: 1498441 (1,234 in 2018) - **Green**: 1724521 (1,098 in 2018) - **Light Blue**: 1787891 (1,567 in 2018) - **Pink**: 1544614 (1,234 in 2018) - **Yellow**: 1622813 (890 in 2018) - **Light Green**: 1569427 (765 in 2018) - **Dark Blue**: 1653283 (678 in 2018) - **Dark Purple**: 1755200 (543 in 2018) - **Dark Red**: 1794205 (432 in 2018) ### Detailed Analysis #### 2018 Baseline - Total jobs: ~14,000 - Dominant category: Blue (13,778) - Smallest category: Dark Red (432) #### 2019 Growth - Total jobs: ~18,000 - Blue category declines to 11,234 - Red category rises to 3,456 - Light Blue category emerges as new top contributor (1,567) #### 2020 Expansion - Total jobs: ~22,000 - Blue category stabilizes at 10,500 - Red category surges to 5,678 - Green category grows to 1,234 #### 2021 Acceleration - Total jobs: ~27,000 - Red category dominates with 8,901 jobs - Blue category drops to 9,876 - Light Green category emerges (765) #### 2022 Surge - Total jobs: ~33,000 - Red category peaks at 12,345 - Blue category declines to 9,000 - Dark Purple category grows to 1,000 #### 2023 Peak - Total jobs: ~45,000 - Red category reaches 15,678 - Blue category stabilizes at 9,500 - Light Blue category surges to 3,000 ### Key Observations 1. **Total Job Growth**: 225% increase from 14k (2018) to 45k (2023). 2. **Category Shifts**: - Blue category (originally dominant) declines by 30%. - Red category grows 650% (2,345 → 15,678). - Light Blue category emerges as a new top contributor in 2023. 3. **2023 Anomaly**: Sharp spike in Light Blue (1787891) and Red (1794205) categories, suggesting a structural shift. ### Interpretation The data demonstrates a dynamic labor market transformation: - **Decline of Traditional Roles**: Blue category (likely legacy industries) shrinks as automation or outsourcing reduces demand. - **Rise of Tech/Innovation Sectors**: Red and Light Blue categories (possibly tech, AI, or green energy) drive growth, with Light Blue's 2023 surge indicating a new economic pillar. - **Policy or Market Catalysts**: The 2023 spike may reflect stimulus packages, regulatory changes, or technological breakthroughs favoring specific sectors. The chart underscores the importance of tracking granular job data to identify emerging industries and declining fields, critical for workforce planning and economic policy. </details> Figure 5: Number of packages introduced by every revision of the dataset and their survival in the package set over time. Figure 5 outlines the number of packages introduced in the dataset by each revision, and their survival over time. In particular, as of July 2017 the package set contained $13\,114$ elements, $8929$ of which were still present in April 2023. VI Results We present our experimental results below, organized by research question. Their discussion is provided later, in Section VII. VI-A RQ0: Does Nix allow rebuilding past packages reliably (even if not bitwise reproducibly)? This research question aims to reproduce the results from Malka et al. [malka_reproducibility_2024], as a starting baseline. That earlier work only built one nixpkgs revision, the most ancient in their dataset; in our case, we rebuilt that revision alongside with 16 others, evenly spaced over time to study trends. Figure 6 shows the proportion of packages between 2017 and 2023 that we successfully rebuilt (not necessarily in a bitwise reproducible manner, merely “successfully built” for this RQ). This proportion varies between 99.68% and 99.95%, confirming previously reported findings: Nix reproducibility of build environments allows for very high rebuildability rate over time. Note that this is not an exact replication of the revision in [malka_reproducibility_2024], because we also included packages not explicitly listed in release.nix, but present in the dependency graph, whereas they did not. <details> <summary>extracted/6418685/paper-images/rebuildability.png Details</summary>  ### Visual Description ## Line Graph: Proportion of Successful Rebuilds Over Time ### Overview The image depicts a line graph illustrating the proportion of successful rebuilds as a percentage over a six-year period (2018–2023). The y-axis represents the proportion of successful rebuilds (ranging from 99.50% to 100.00%), while the x-axis represents the years. The blue line fluctuates between 99.70% and 99.90%, showing a general upward trend with periodic dips. --- ### Components/Axes - **Y-Axis**: - Label: "Proportion of successful rebuilds" - Scale: 99.50% to 100.00% (in 0.01% increments) - Units: Percentages - **X-Axis**: - Label: "Date" - Scale: 2018 to 2023 (annual intervals) - **Legend**: - A single blue line representing the proportion of successful rebuilds. --- ### Detailed Analysis - **Data Points**: - **2018**: ~99.85% - **2019**: ~99.75% - **2020**: ~99.70% - **2021**: ~99.85% - **2022**: ~99.90% - **2023**: ~99.75% - **2024**: ~99.95% (note: x-axis ends at 2023, but data extends to 2024) - **Trends**: - The line exhibits a **general upward trend** from 2018 to 2023, with a peak of ~99.90% in 2022. - Notable dips occur in **2019** (~99.75%), **2020** (~99.70%), and **2023** (~99.75%). - The final data point (2024) suggests a rebound to ~99.95%, though this falls outside the x-axis range. --- ### Key Observations 1. **Fluctuations**: The proportion of successful rebuilds varies year-to-year, with no consistent pattern of growth or decline. 2. **Peaks and Troughs**: - Highest value: ~99.90% in 2022. - Lowest value: ~99.70% in 2020. 3. **Final Data Point**: The 2024 value (~99.95%) exceeds the x-axis range, indicating potential data inconsistency or a typo. --- ### Interpretation The graph suggests that the proportion of successful rebuilds has **improved overall** from 2018 to 2023, with a notable increase in 2022. However, the dips in 2019, 2020, and 2023 may reflect external factors (e.g., operational challenges, resource constraints) or measurement variability. The 2024 data point, while higher than previous years, conflicts with the x-axis range, raising questions about data accuracy or scope. The upward trend implies that rebuild processes may have become more efficient or reliable over time, but the fluctuations highlight the need for further investigation into the causes of periodic declines. </details> Figure 6: Proportion of rebuildable packages over time. VI-B RQ1: What is the evolution of bitwise reproducible packages in nixpkgs between 2017 and 2023? Apart from a significant regression in 2020, we obtain bitwise reproducibility levels between 69% and 91% (see Figure 7). The trends in Figure 8 show that the absolute number of bitwise reproducible packages has consistently gone up and followed the fast growth of the package set. The only exception is the data point for June 2020, where the number of reproducible packages dropped even though the total number of packages grew. We study and explain this reproducibility regression in Section VI-C below. <details> <summary>extracted/6418685/paper-images/reproducibility-overall-proportion.png Details</summary>  ### Visual Description ## Line Chart: Proportion of Successful Rebuilds Over Time (2018–2023) ### Overview The chart visualizes the proportion of successful rebuilds categorized by three statuses: **reproducible** (blue), **buildable** (red), and **failed** (green). Data spans from 2018 to 2023, with the y-axis representing the proportion of successful rebuilds as a percentage (30%–90%). The failed status consistently dominates, while reproducible rebuilds show the most variability. --- ### Components/Axes - **X-axis (Date)**: Years 2018–2023, labeled at annual intervals. - **Y-axis (Proportion of successful rebuilds)**: Percentage scale from 30% to 90%, with gridlines at 10% intervals. - **Legend**: Located at the bottom, mapping: - **Blue**: Reproducible rebuilds - **Red**: Buildable rebuilds - **Green**: Failed rebuilds - **Data Representation**: Lines with markers (dots) for each status. --- ### Detailed Analysis 1. **Failed Rebuilds (Green)**: - **Trend**: Nearly flat line at **~99%** across all years. - **Values**: - 2018: 99% - 2019: 99% - 2020: 99% - 2021: 99% - 2022: 99% - 2023: 99% 2. **Buildable Rebuilds (Red)**: - **Trend**: Stable with minor fluctuations, starting at ~90% and ending at ~91%. - **Values**: - 2018: 90% - 2019: 90% - 2020: 88% - 2021: 90% - 2022: 90% - 2023: 91% 3. **Reproducible Rebuilds (Blue)**: - **Trend**: Significant variability, starting at 70%, peaking at 90% in 2023, with a sharp dip in 2020. - **Values**: - 2018: 70% - 2019: 85% - 2020: 62% - 2021: 84% - 2022: 85% - 2023: 90% --- ### Key Observations 1. **Failed Rebuilds Dominance**: The green line remains at **~99%** throughout, indicating that nearly all rebuilds fail to meet the highest standard. 2. **Reproducible Rebuilds Volatility**: - Sharp decline in 2020 (**62%**) suggests a temporary systemic issue. - Recovery to **90% in 2023** indicates improved reproducibility post-2020. 3. **Buildable Stability**: The red line shows minimal change, hovering around **90%**, suggesting consistent mid-tier rebuild success. 4. **Color Consistency**: All data points match the legend (blue = reproducible, red = buildable, green = failed). --- ### Interpretation - **Systemic Challenges**: The persistent **99% failure rate** implies unresolved issues in rebuild processes, possibly due to technical debt or resource constraints. - **Reproducibility Recovery**: The rebound in reproducible rebuilds post-2020 suggests targeted improvements (e.g., process optimization, tooling upgrades). - **Buildable Plateau**: The stable buildable rate may reflect a baseline threshold for acceptable rebuilds, with limited room for growth. - **2020 Anomaly**: The dip in reproducible rebuilds could correlate with external factors (e.g., global disruptions, resource reallocation) or internal bottlenecks. --- ### Spatial Grounding - **Legend**: Bottom-center, clearly aligned with line colors. - **Data Points**: Markers (dots) placed directly on lines, ensuring visual clarity. - **Axis Labels**: Positioned at the ends of axes, with gridlines aiding readability. --- ### Content Details - **Y-axis Scale**: Starts at 30% (unlabeled lower bound) to 90% (upper bound), with 10% increments. - **X-axis Labels**: Annual ticks (2018–2023), no intermediate years shown. - **Line Smoothness**: Lines are continuous, with no breaks, indicating no missing data. --- ### Final Notes The chart highlights a critical imbalance: while most rebuilds fail, reproducible rebuilds have shown resilience and improvement, particularly after 2020. The buildable category remains stagnant, suggesting a need for innovation to elevate rebuild quality beyond basic functionality. </details> Figure 7: Proportion of reproducible, rebuildable (but unreproducible) and non-rebuildable packages over time. <details> <summary>extracted/6418685/paper-images/reproducibility-overall-absolute.png Details</summary>  ### Visual Description ## Line Chart: Number of Successful Rebuilds by Status (2018–2023) ### Overview The chart visualizes the number of successful rebuilds categorized by three statuses: **reproducible** (blue), **buildable** (red), and **failed** (green) over six years. The y-axis represents the count of rebuilds, while the x-axis tracks years from 2018 to 2023. The green line (failed) consistently exceeds the red line (buildable), and the blue line (reproducible) shows the steepest growth. ### Components/Axes - **X-axis (Date)**: Years 2018–2023, labeled at 2-year intervals. - **Y-axis (Number of successful rebuilds)**: Scale from 0 to 70k, with increments of 10k. - **Legend**: Located at the bottom-left, mapping colors to statuses: - **Blue**: Reproducible - **Red**: Buildable - **Green**: Failed - **Data Points**: Marked with dots matching the legend colors. ### Detailed Analysis 1. **Reproducible (Blue)**: - **2018**: ~14k - **2019**: ~24k - **2020**: ~30k - **2021**: ~35k - **2022**: ~40k - **2023**: ~65k - **Trend**: Steep upward trajectory, especially post-2020. 2. **Buildable (Red)**: - **2018**: ~6k - **2019**: ~5k - **2020**: ~8k - **2021**: ~10k - **2022**: ~12k - **2023**: ~7k - **Trend**: Relatively stable until 2022, then a sharp decline in 2023. 3. **Failed (Green)**: - **2018**: ~3k - **2019**: ~5k - **2020**: ~10k - **2021**: ~15k - **2022**: ~20k - **2023**: ~7k - **Trend**: Gradual increase until 2022, followed by a drop in 2023. ### Key Observations - **Reproducible rebuilds** dominate growth, surging from ~14k to ~65k (364% increase). - **Buildable rebuilds** peak at ~12k in 2022 but collapse to ~7k in 2023, suggesting a potential anomaly or shift in criteria. - **Failed rebuilds** rise steadily until 2022 (~20k) but drop sharply in 2023 (~7k), mirroring the buildable trend. - The green line (failed) always exceeds the red line (buildable), indicating more rebuilds fail than are buildable. ### Interpretation The data highlights a critical imbalance: while reproducible rebuilds grow exponentially, buildable and failed categories show conflicting trends. The 2023 drop in both buildable and failed rebuilds may reflect improved processes (fewer failures) or stricter definitions of "buildable." The dominance of reproducible rebuilds suggests a focus on reliability, but the high failure rate (green line) implies systemic issues in build processes that require investigation. The chart’s y-axis label ("successful rebuilds") conflicts with the inclusion of "failed" as a category, raising questions about data categorization or labeling accuracy. </details> Figure 8: Absolute numbers of reproducible, rebuildable (but unreproducible) and non-rebuildable packages over time. Figure 9 shows for each revision the cumulative amount of unreproducibilities introduced by that revision getting fixed over time. The large slope between the two first points of each plot indicates that most of the unreproducibilities introduced in a revision are fixed in the next revision, on average 62% of them (even raising to 85% if we account for packages fixed after the 2020 reproducibility regression). <details> <summary>extracted/6418685/paper-images/fixes-per-revision.png Details</summary>  ### Visual Description ## Line Chart: Unreproducibility Introduced in Various IDs Over Time ### Overview The chart visualizes the number of fixed packages (y-axis) introduced in 12 distinct software IDs (x-axis) from 2018 to 2023. Each ID is represented by a unique colored line, with values plotted annually. The y-axis ranges from 0 to 3,000, while the x-axis spans 2018–2023. The legend at the bottom maps colors to IDs, and the title explicitly states the focus on "Unreproducibility introduced in:". --- ### Components/Axes - **X-Axis (Years)**: 2018, 2019, 2020, 2021, 2022, 2023 (discrete intervals). - **Y-Axis (Number of Fixed Packages)**: Linear scale from 0 to 3,000. - **Legend**: 12 entries, each with a color and ID (e.g., red = 1416607, orange = 1593430). Positioned at the bottom-center. - **Lines**: 12 colored lines, each corresponding to an ID. Lines are plotted with markers at yearly intervals. --- ### Detailed Analysis #### Key Trends: 1. **ID 1593430 (Orange Line)**: - **Trend**: Sharp increase from 0 in 2018 to 2,900 in 2021, then gradual rise to 3,000 by 2023. - **Notable**: Largest absolute increase (3,000 packages) and most volatile trend. 2. **ID 1544614 (Cyan Line)**: - **Trend**: Steady rise from 450 in 2020 to 750 in 2023. - **Notable**: Consistent upward trajectory without plateaus. 3. **ID 1448353 (Green Line)**: - **Trend**: Gradual increase from 150 in 2019 to 210 in 2023. - **Notable**: Slow but steady growth. 4. **ID 1476353 (Blue Line)**: - **Trend**: Increase from 100 in 2019 to 150 in 2023. - **Notable**: Moderate growth with minor fluctuations. 5. **ID 1724521 (Purple Line)**: - **Trend**: Rise from 50 in 2020 to 120 in 2023. - **Notable**: Accelerated growth post-2021. 6. **ID 1622813 (Dark Green Line)**: - **Trend**: Increase from 180 in 2021 to 220 in 2023. - **Notable**: Steady but modest growth. 7. **ID 1416607 (Red Line)**: - **Trend**: Increase from 20 in 2018 to 100 in 2023. - **Notable**: Slow but consistent growth. 8. **ID 1520586 (Pink Line)**: - **Trend**: Rise from 50 in 2020 to 120 in 2023. - **Notable**: Similar trajectory to ID 1724521. 9. **ID 1687868 (Brown Line)**: - **Trend**: Increase from 20 in 2022 to 90 in 2023. - **Notable**: Rapid growth in final years. 10. **ID 1755200 (Gray Line)**: - **Trend**: Increase from 10 in 2022 to 80 in 2023. - **Notable**: Sharp rise in final years. 11. **ID 1653283 (Light Blue Line)**: - **Trend**: Increase from 20 in 2022 to 100 in 2023. - **Notable**: Rapid growth in final years. 12. **ID 1787891 (Dark Orange Line)**: - **Trend**: Increase from 50 in 2022 to 120 in 2023. - **Notable**: Accelerated growth post-2022. --- ### Key Observations - **Dominant Trends**: - ID 1593430 (orange) and ID 1544614 (cyan) show the most significant increases, suggesting systemic issues in these IDs. - IDs introduced after 2020 (e.g., 1687868, 1755200) exhibit rapid growth, indicating potential late-stage vulnerabilities. - **Anomalies**: - ID 1593430’s spike in 2021 (2,900 packages) is an outlier compared to other IDs. - IDs with later introduction dates (e.g., 1687868, 1755200) show steeper slopes, possibly reflecting newer codebases with unresolved issues. --- ### Interpretation The chart highlights **systemic unreproducibility** in software development, with certain IDs (e.g., 1593430, 1544614) contributing disproportionately to fixed packages over time. The sharp rise in ID 1593430 in 2021 suggests a critical event (e.g., major refactoring, bug discovery) that required extensive fixes. IDs introduced later (e.g., 1687868, 1755200) show rapid growth, implying that newer codebases may inherit or introduce vulnerabilities more aggressively. The gradual increases in older IDs (e.g., 1416607, 1448353) reflect ongoing maintenance challenges. This data underscores the need for proactive code quality measures and targeted debugging efforts for high-risk IDs. </details> Figure 9: Evolution of the cumulative number of fixes to non-reproducibilities, separated by the revision in which the non-reproducibilities were introduced. Excluding the June 2020 revision (corresponding to the observed bitwise reproducibility regression), Figure 10 depicts the average evolution of the package set between two consecutive revisions. In particular, only 0.60% of the packages transition from reproducible to buildable but not reproducible status between two revisions, while 2.07% of the buildable but not reproducible packages become reproducible. The average growth rate of the package set is 1.07 and, on average, 80.02% of the new packages are reproducible. <details> <summary>extracted/6418685/paper-images/sankey-average.png Details</summary>  ### Visual Description ## Sankey Diagram: Process Flow Analysis ### Overview The image depicts a Sankey diagram illustrating a multi-stage process flow with feedback loops. Nodes represent process states ("Reproducible," "Non-Existant," "Failed," "Buildable"), while arrows indicate transitions between states. The diagram emphasizes flow magnitude through arrow thickness, with no explicit numerical values provided. ### Components/Axes - **Nodes (States)**: - **Left Side**: - Reproducible (blue) - Non-Existant (red) - Failed (black) - Buildable (purple) - **Right Side**: - Reproducible (blue) - Non-Existant (red) - Failed (black) - Buildable (purple) - **Arrows (Transitions)**: - Gray arrows of varying thickness connect nodes. - No explicit legend, but arrow thickness correlates with flow magnitude. - **Spatial Layout**: - Nodes are positioned symmetrically on left/right sides. - Arrows flow predominantly left-to-right, with some feedback loops. ### Detailed Analysis 1. **Primary Flow**: - Thickest arrow connects **left Reproducible** to **right Reproducible**, suggesting the dominant path. - Secondary flows: - **left Reproducible → right Non-Existant** (moderate thickness) - **left Reproducible → right Failed** (thinner) - **left Buildable → right Buildable** (moderate thickness) 2. **Feedback Loops**: - **right Non-Existant → left Non-Existant** (thin) - **right Failed → left Failed** (thin) - **right Buildable → left Buildable** (thin) 3. **Color Coding**: - Blue = Reproducible - Red = Non-Existant - Black = Failed - Purple = Buildable ### Key Observations - **Dominant Path**: 70-80% of flow (estimated by thickness) moves directly from left to right Reproducible. - **Losses**: 15-20% of flow transitions to Non-Existant or Failed states. - **Buildable Path**: 5-10% of flow remains in the Buildable state. - **Feedback**: Minimal recycling (2-5% of flow) occurs in Non-Existant, Failed, and Buildable states. ### Interpretation The diagram likely represents a project lifecycle or quality control process: 1. **Success Metrics**: Most efforts (Reproducible) succeed end-to-end, but a significant portion fails or becomes non-existent. 2. **Failure Points**: The "Failed" and "Non-Existant" states act as bottlenecks, with limited feedback for correction. 3. **Buildable State**: A distinct pathway suggests preparatory work that may not integrate into the main process. 4. **Feedback Loops**: Sparse recycling implies limited iterative improvement mechanisms. The absence of numerical values prevents precise quantification, but the relative thickness of arrows strongly suggests a hierarchical success rate: Reproducible > Buildable > Non-Existant/Failed. The feedback loops hint at potential for process refinement but are underutilized in this flow. </details> Figure 10: Sankey graph of the average flow of packages between two revisions, excluding the revision from June 2020, considered as an outlier. VI-C RQ2: What are the unreproducible packages? We observe large disparities both in trends and in reproducibility by package ecosystem (see Figure 11). In the top three most popular ecosystems, Perl has consistently maintained a proportion of reproducible packages above 98%, while Haskell reproducibility rate stagnated around the 60% mark, even decreasing by more than 7 percentage points during the time of our study. A fix to a long-standing reproducibility issue has been introduced in GHC 9.12.1 (new -fobject-determinism flag). As of January 2025, nixpkgs does not make use of this flag to improve Haskell build reproducibility. The Python ecosystem on the contrary sees a positive evolution over time, with reproducibility rates as low as 27.64% in December 2017, reaching 98.28% in April 2023. As can be seen on Figure 11, the Python ecosystem has however known a major dip in reproducibility in June 2020, with a drop to 6.01% (for almost $-90$ percentage points!). <details> <summary>extracted/6418685/paper-images/reproducibility-ecosystems.png Details</summary>  ### Visual Description ## Line Chart: Proportion of Bitwise Reproducible Packages Over Time ### Overview The chart visualizes the temporal evolution of bitwise reproducibility across four programming ecosystems (base, Haskell, Python, Perl) from 2018 to 2023. It highlights dramatic shifts in reproducibility metrics, particularly for Python and Perl, while showing relative stability in base and Haskell ecosystems. ### Components/Axes - **X-axis (Date)**: Years 2018–2023, with annual intervals. - **Y-axis (Proportion of bitwise reproducible packages)**: Percentage scale from 0% to 100%. - **Legend**: Located at the bottom, mapping colors to ecosystems: - **Blue**: base - **Red**: Haskell - **Green**: Python - **Purple**: Perl ### Detailed Analysis 1. **Perl (Purple Line)**: - Maintains **100% reproducibility** consistently from 2019 onward. - Slight dip to ~98% in 2020, recovering to 100% by 2021. 2. **Python (Green Line)**: - **2018–2019**: Starts at ~28% in 2018, surges to ~95% by 2019. - **2020**: Sharp anomaly: drops to **~5%** (potential data artifact or methodological change). - **2021–2023**: Recovers to ~98–100%, stabilizing near Perl's level. 3. **Haskell (Red Line)**: - Gradual decline from **~60% in 2018** to **~50% in 2023**. - Minor fluctuations (e.g., ~55% in 2020, ~52% in 2022). 4. **Base (Blue Line)**: - Stable baseline: ~80% in 2018, rising to **~88–90%** by 2020. - Peaks at **~92%** in 2022, then stabilizes at ~90% through 2023. ### Key Observations - **Python's Volatility**: The 2020 collapse and subsequent recovery suggest a potential methodological shift (e.g., versioning changes, measurement criteria). - **Perl's Consistency**: Unchanged 100% reproducibility implies a highly controlled or standardized ecosystem. - **Haskell's Decline**: Steady reduction may reflect evolving tooling or reduced emphasis on bitwise reproducibility. - **Base's Stability**: Acts as a reliable reference point, showing minimal deviation. ### Interpretation The data underscores divergent priorities in ecosystem maintenance: - **Perl's rigidity** (100% reproducibility) suggests a focus on backward compatibility. - **Python's volatility** hints at potential fragmentation or evolving standards post-2020. - **Haskell's decline** could indicate shifting developer priorities or toolchain maturation. - **Base's stability** positions it as a pragmatic middle ground, balancing reproducibility with flexibility. The 2020 Python anomaly warrants investigation—it may reflect a temporary measurement error, a versioning overhaul (e.g., Python 3.x adoption), or a change in reproducibility criteria. The convergence of Python and Perl near 100% by 2023 suggests improved tooling across ecosystems, though Haskell's trajectory raises questions about long-term sustainability. </details> Figure 11: Proportion of reproducible packages belonging to the three most popular ecosystems and the base namespace of nixpkgs. As can be seen in Figure 12, this dip in the proportion of reproducible Python packages can be explained by a large number of packages transitioning from the reproducible to the buildable but not reproducible state, indicating a regression in bitwise reproducibility at that time. To identify the root cause of that regression, we used a git bisection on the nixpkgs repository between June and October 2020 in order to identify the commit fixing the unreproducibility issue, and derived that the root cause of the regression was an update in the pip executable changing the behavior of byte-compilation. Details can be found in https://github.com/pypa/pip/issues/7808. Figure 13 shows the difference in reproducibility rates between the NixOS minimal ISO image—a package set for which there exists community-based reproducibility monitoring, and made of packages that are considered critical—and the whole package set. The minimal ISO image has a very high proportion of reproducible packages in the considered period, with rates consistently higher than 95% starting from May 2019. Its reproducibility rates however do not follow the evolution of the overall package set, such that observing the reproducibility of the minimal ISO image does not give any clue about the reproducibility of the package set as a whole. <details> <summary>extracted/6418685/paper-images/reproducibility-absolute-python.png Details</summary>  ### Visual Description ## Line Chart: Number of Packages by Status (2018-2023) ### Overview The chart displays the number of software packages categorized by three statuses—reproducible, buildable, and failed—over six years (2018–2023). All three categories show an upward trend, with failed packages initially dominating but being overtaken by buildable packages in later years. ### Components/Axes - **X-axis (Date)**: Years from 2018 to 2023, labeled at 2018, 2019, 2020, 2021, 2022, and 2023. - **Y-axis (Number of packages)**: Ranges from 0 to 6000 in increments of 1000. - **Legend**: Located at the bottom, with three colors: - **Blue**: Reproducible packages - **Red**: Buildable packages - **Green**: Failed packages ### Detailed Analysis 1. **Reproducible Packages (Blue)**: - 2018: 500 - 2019: 2000 - 2020: 2700 - 2021: 3300 - 2022: 4500 - 2023: 5500 - **Trend**: Steady linear growth, starting from 0 in 2018. 2. **Buildable Packages (Red)**: - 2018: 1700 - 2019: 2000 - 2020: 2700 - 2021: 3400 - 2022: 4600 - 2023: 5600 - **Trend**: Consistent upward trajectory, surpassing failed packages in 2021. 3. **Failed Packages (Green)**: - 2018: 1700 - 2019: 2000 - 2020: 2800 (noted as a "dip" in the image, though this is an increase from 2019) - 2021: 3100 - 2022: 4700 - 2023: 5700 - **Trend**: Initial growth, followed by a slower rise after 2020. ### Key Observations - **Crossover in 2021**: Buildable packages (3400) exceed failed packages (3100) for the first time. - **Failed Packages Anomaly**: The image describes a "dip" in 2020, but the data shows an increase from 2000 (2019) to 2800 (2020). This may indicate a misinterpretation or error in the original description. - **Growth Rates**: All categories grow at similar rates, with reproducible packages maintaining the lowest absolute values but following the same upward trajectory. ### Interpretation The data suggests a systemic increase in the total number of packages across all statuses, reflecting expanding software development activity. The crossover between buildable and failed packages in 2021 implies improvements in build processes or stricter quality controls reducing failures. The persistent growth in reproducible packages highlights their increasing importance in ensuring reliable software deployment. The discrepancy in the "dip" description for failed packages warrants further investigation to confirm whether the trend aligns with external factors (e.g., toolchain updates, policy changes). </details> Figure 12: Evolution of the absolute number of reproducible, rebuildable (but unreproducible) and non-rebuildable packages from the Python ecosystem. <details> <summary>extracted/6418685/paper-images/minimal-iso-reproducibility.png Details</summary>  ### Visual Description ## Line Chart: Proportion of Bitwise Reproducible Packages Over Time ### Overview The chart displays two time-series data lines representing the proportion of bitwise reproducible packages in two categories: "Whole distribution" (blue) and "Minimal ISO image" (red). The x-axis spans 2018–2023, while the y-axis shows percentages from 50% to 90%. ### Components/Axes - **X-axis (Date)**: Labeled "Date," with yearly intervals from 2018 to 2023. - **Y-axis (Proportion)**: Labeled "Proportion of bitwise reproducible packages," scaled from 50% to 90% in 10% increments. - **Legend**: Located at the bottom, with: - **Blue line**: "Whole distribution" - **Red line**: "Minimal ISO image" ### Detailed Analysis #### Whole Distribution (Blue Line) - **2018**: Starts at ~70%. - **2019**: Rises to ~85%, then dips to ~81% by mid-2019. - **2020**: Peaks at ~84%, then sharply drops to ~62% in 2021. - **2021–2023**: Recovers steadily, reaching ~90% by 2023. #### Minimal ISO Image (Red Line) - **2018**: Begins at ~92%. - **2019–2020**: Gradually increases to ~96%. - **2021–2022**: Peaks at ~98% in 2022. - **2023**: Slightly declines to ~96%. ### Key Observations 1. **Volatility in Whole Distribution**: The blue line shows significant fluctuations, including a sharp 2021 drop to 62% (lowest point) and a recovery to 90% by 2023. 2. **Stability in Minimal ISO Image**: The red line remains consistently high (>92%), peaking at 98% in 2022 before a minor decline. 3. **Divergence in 2021**: The blue line’s 2021 dip contrasts sharply with the red line’s stability, suggesting external factors affected the "Whole distribution" category that year. ### Interpretation - **Trend Implications**: The "Whole distribution" category demonstrates long-term improvement but is prone to short-term disruptions (e.g., 2021 anomaly). The "Minimal ISO image" category maintains high reproducibility, indicating robust quality control. - **Anomaly Analysis**: The 2021 dip in the blue line may reflect temporary issues in package distribution processes, while the red line’s resilience suggests stricter adherence to reproducibility standards in the Minimal ISO subset. - **Practical Insight**: The data highlights the importance of monitoring reproducibility across different package categories, as systemic improvements (e.g., post-2021 recovery) can offset temporary setbacks. </details> Figure 13: Evolution of the proportion of reproducible packages belonging to the minimal ISO image and in the entire package set. VI-D RQ3: Why are packages unreproducible? We identified four heuristics that are both effective (they give a large number of matches in our diffoscope dataset) and relevant (they correspond to a software engineering practice that can be changed) to automatically determine why packages are not bitwise reproducible: - Embedded dates: presence of build dates in various places in the built artifacts (version file, log file, file names, etc.). To detect dates embedded in build outputs, we look specifically for the year 2024 in added lines, given that we ran all our builds during 2024, but none of the historical builds ran during this year. - uname output: some unreproducible builds embed build information such as the machine that the build was run on. While the real host name is hidden by the Nix sandbox, other pieces of information (such as the Linux kernel version or OS version, reported by uname) are still available. - Environment variables: some builds embed some or all available environment variables in their build outputs. This typically causes unreproducibility because an environment variable containing the number of cores available to build on the machine is set by Nix. - Build ID: some ecosystems (Go for example) embed a unique (but not deterministic) build ID into the artifacts. Despite a well-known recommendation by the Reproducible Builds project to avoid embedding build dates in build artifacts to achieve bitwise reproducibility, we find that $12\,831$ of our $86\,317$ packages with diffoscopes contain a date, accounting for 14.8% of them. Still in the most recent nixpkgs revision, we find $470$ instances of this non-reproducibility cause, showing that embedding dates is still an existing practice. Additionally, we find that embedded environment variables and build IDs each account for 2.2% of our unreproducible packages. Finally, we find uname outputs in $1097$ of our unreproducible builds (1.3%), $721$ of which also include a date as part of the uname output. Figure 14 shows the evolution of the number of packages matched by each heuristic over time. Altogether, a total of 19.7% of the packages for which we have generated a diffoscope have an unreproducibility that can be explained by at least one of our heuristics. We evaluate the precision of each heuristic by manually verifying $500$ matched lines for every heuristic and counting false positives. For the uname, build ID, and environment variables heuristics, we report a precision of 100%. For the date heuristic, we obtain a precision of 97.8% (11 false positives). <details> <summary>extracted/6418685/paper-images/evolution-heuristics.png Details</summary>  ### Visual Description ## Line Chart: Number of Packages Matched by Heuristic Over Time ### Overview The chart displays four time-series data lines representing different heuristics used to match packages over a six-year period (2018–2023). The y-axis measures the number of packages matched, while the x-axis tracks chronological progression. Four distinct heuristics are visualized with unique colors and trends. ### Components/Axes - **X-axis (Date)**: Labeled "Date" with annual markers from 2018 to 2023. - **Y-axis (Number of packages matched by the heuristic)**: Labeled with increments of 200, ranging from 0 to 1000. - **Legend**: Located at the bottom, with four entries: - **Blue**: Embedded dates - **Red**: Embedded uname output - **Green**: Embedded environment variable - **Purple**: Embedded build ID ### Detailed Analysis 1. **Embedded dates (Blue line)**: - Starts at ~700 packages in 2018. - Peaks at ~900 packages in 2020. - Declines sharply to ~500 packages by 2023. - Shows volatility between 2019–2021, with a notable drop in 2022. 2. **Embedded uname output (Red line)**: - Remains consistently low (~50–100 packages) throughout the period. - Peaks at ~80 packages in 2021. - Fluctuates minimally, with no significant upward or downward trend. 3. **Embedded environment variable (Green line)**: - Starts near 0 in 2018. - Spikes dramatically to ~500 packages in 2022. - Drops abruptly to ~20 packages in 2023. - Shows a secondary peak at ~150 packages in 2021. 4. **Embedded build ID (Purple line)**: - Begins at 0 in 2018. - Gradually increases to ~150 packages by 2022. - Rises to ~180 packages in 2023. - Exhibits steady growth with minor fluctuations. ### Key Observations - **Dominance of Embedded dates**: The blue line consistently represents the highest values until 2020, after which it declines. - **Environment variable anomaly**: The green line’s 2022 spike (500 packages) and subsequent collapse (20 packages in 2023) suggest a sudden, short-lived impact. - **Build ID growth**: The purple line’s gradual rise indicates increasing effectiveness over time. - **Uname output stagnation**: The red line remains the least impactful heuristic across all years. ### Interpretation The data suggests that **Embedded dates** were the most effective heuristic initially but lost prominence after 2020, possibly due to changes in package metadata or system updates. The **Embedded environment variable**’s 2022 spike implies a temporary, high-impact event (e.g., a system migration or data enrichment phase), followed by a sharp decline that may indicate reduced relevance or data quality issues. The **Embedded build ID**’s steady growth highlights its increasing utility, potentially reflecting improved tracking mechanisms or expanded build systems. Meanwhile, **Embedded uname output**’s stagnation suggests it is the least effective heuristic, possibly due to limited variability in user data or outdated implementation. The chart underscores the dynamic nature of heuristic effectiveness over time, emphasizing the need for periodic reevaluation of matching strategies as systems evolve. </details> Figure 14: Evolution of the number of packages for which we generated diffoscopes that are matched by each of our heuristics, over time. VI-E RQ4: How are unreproducibilities fixed? Our analysis of 100 randomly sampled reproducibility fixes showed that, in most cases, authors are not aware that they are fixing a reproducibility issue, or they do not mention it: in 93 instances, we did not find any trace of reproducibility being mentioned. In fact, in 76 cases we found out that the reproducibility fix was just a routine package update, and it is very likely that the package becoming reproducible had more to do with changes in the upstream software than with a conscious action from the package maintainer. Other fixes included internal nixpkgs changes that had reproducibility impact as a side effect of the main reason behind the change. Our study of the 15 most impactful fixes suggests that those are more often done with the intent of fixing reproducibility: in more than half of them (8 out of 15) the author mentioned and documented the reproducibility issue being fixed and 9 of them were changes internal to nixpkgs. Some large-scale reproducibility fixes were still package updates like toolchain updates. VII Discussion This work brings valuable insights to the ongoing discussions about software supply chain security, reproducible builds, and functional package management. False myth: reproducible builds (R-B) do not scale One strongly held belief about R-B is that they work well in limited contexts—either selected “critical” applications (e.g., cryptocurrencies or anonymity technology) or strictly curated distributions—but either do not scale or are not worth the effort in larger settings. Our results can be interpreted as counter proof of this belief: in nixpkgs, the largest general-purpose repository with more than 70k packages (in April 2023), more than 90.5% packages are nowadays bitwise reproducible from source. R-B do scale to general purpose, regularly used software. Recommendation: invest in infrastructure for binary caches and build attestations To reap the security benefits of the R-B vision [lamb_reproducible_2022] in practice, users need multiple components: (1) signed attestations by each independent builder, publishing the obtained checksums at the end of a build; (2) caches of built artifacts, so that users can avoid rebuilding from source; (3) verification technology in package managers to verify that built artifacts in (2) match the consensus checksums in (1). With few exceptions, the infrastructure to support all this does not exist yet. Now that the “R-B do not scale” excuse is out of the picture, we call for investments to develop this infrastructure, preferably in a distributed architectural style to make it more secure, robust, and sustainable. False myth: Nix implies bitwise reproducibility Contradictory to the previous myth, there is also a belief that Nix (or functional package management more generally) implies bitwise reproducibility. Our results disprove this belief: about 9.4% of nixpkgs packages are not bitwise reproducible. This is not surprising, because R-B violations happen for multiple reasons, only some of which are mitigated by FPM per se. Recommendation: use FPM for bitwise reproducibility and rebuildability Still, Nix appears to perform really well at bitwise reproducibility “out of the box”, and even more so at rebuildability (above 99%). Based on this, we recommend to use Nix, or FPM more generally, for all use cases that require either mere rebuildability or full bitwise reproducibility. At worse, they provide a very good starting point. This work has investigated the causes of the lingering non reproducibility in nixpkgs, but not those of reproducibility; it would be interesting to know why Nix performs so well, possibly for adoption in different contexts. It is possible that bitwise reproducibility is an emerging property of FPM, or that it comes from technical measures during build like sandboxing, or that Nix is simply benefiting (more than other distributions?) from the decade-long efforts of the R-B project in upstream toolchains. Exploring all this is left as future work. QA monitoring for bitwise reproducibility The significantly higher (and stably so) bitwise reproducibility performances of the NixOS minimal ISO image (see Figure 13) suggests that quality assurance (QA) monitoring for build reproducibility is an effective way to increase it. The fact that Debian uses a similar approach with good results [lamb_reproducible_2022] is compatible with this interpretation. A deeper analysis of the relationship between QA monitoring and reproducibility rate is out of scope for this work, but it is quite likely that extending reproducibility monitoring to all packages will result in an easy win for nixpkgs (assuming bitwise reproducibility is seen as a project goal). VIII Threats to validity VIII-A Construct validity We rebuilt Nix packages from revisions in the period July 2017–April 2023, for about 6 years. That is enough to observe arguably long-term trends, which also appear to be stable in our analysis; except for one temporary regression, which we analyzed and explained. Still, we cannot exclude significant differences in reproducibility trends before/after the studied period. Due to the high computation cost, build time, and environmental impact of rebuilding general purpose software from source, we sampled nixpkgs revisions to rebuild uniformly (every 4.1 months) within the studied period, totaling $14\,296$ hours of build time. We cannot exclude trend anomalies between sampled revisions either, but that seems unlikely due to the stability of the observed long-term trends. More importantly, this means that we are less likely to catch short-spanned reproducibility regressions, which would be introduced and fixed within the same period between two sampled revisions. This can be improved upon by sampling and rebuilding more nixpkgs revisions, complementing this work. When rebuilding a given package, we relied on the Nix binary cache for all its transitive dependencies. As we have rebuilt all packages from any sampled nixpkgs revisions, our package coverage is complete. But this way we have not measured the impact of unreproducible packages on transitive reverse dependencies, i.e., how many additional packages become unreproducible if systematically built from scratch, including all their dependencies? Our experiment matches real-world use cases and is hence adequate to answer our RQs, but it would still be interesting to know. Also, during rebuilds we have not attempted to re-download online assets, but relied on the Nix cache. Hence, we have not measured the impact of lacking digital preservation practices on reproducibility and rebuildability. It would be interesting and possible to measure such impact by, e.g., first trying to download assets from their original hosting places and, for source code assets, fallback to archives such as Software Heritage [dicosmo_2017_swh] in case of failure. VIII-B External validity We have rebuilt packages from nixpkgs, the largest cross-ecosystem FOSS distribution, using the Nix functional package manager (FPM). Other FPMs with significant user bases exist, such as Guix [courtes_functional_2013]. Given the similarity in design choices, we do not have reasons to believe that Guix would perform any differently in terms of build reproducibility than Nix, but we have not empirically verified it; doing so would be useful complementary work. Neither did we verify historical build reproducibility in classic FOSS distributions (e.g., Debian, Fedora, etc.), as out of the FPM scope. Doing so would still be interesting, for comparison purposes. It would be more challenging, though, due to the fact that outside the FPM context, it is significantly harder to recreate exact build environments from years ago. IX Conclusion In this work we conducted the first large-scale experiment of bitwise reproducibility in the context of the Nix functional package manager, by rebuilding $709\,816$ packages coming from 17 revisions of the nixpkgs software repository, sampled every 4.1 months from 2017 to 2023. Our findings show that bitwise reproducibility in nixpkgs is very high and has known an upward trend, from 69% in 2017 to 91% in 2023. The mere ability to rebuild packages (whether bitwise reproducibly or not) is even higher, stably around 99.8%. We have highlighted disparities in reproducibility across ecosystems that coexist in nixpkgs, as well as between packages for which bitwise reproducibility is actively monitored and the others. We have developed heuristics to understand common (un)reproducibility causes, finding that 15% of unreproducible packages were embedding the date during the build process. Finally, we studied reproducibility fixes and found out that only a minority of changes inducing a reproducibility fix were done intentionally; the rest appear to be incidental. Data availability statement The dataset produced as part of this work is archived on and available from Zenodo [replication_metadata, replication_diffoscopes]. A full replication package containing the code used to run the experiment describe in this paper is archived on and available from Software Heritage [replication-code:1.0]. Acknowledgments This work would have not been possible without the availability of historical data from the official Nix package cache at https://cache.nixos.org with its long-standing “no expiry” policy. In the context of ongoing discussions to prune the cache, we strongly emphasize its usefulness for performing empirical research on package reproducibility, like this work. We also thank the NixOS Foundation for guaranteeing us that the revisions that we needed for this research would not be garbage-collected. \printbibliography