2502.09858

## Automated Hypothesis Validation with Agentic Sequential Falsifications Kexin Huang * 1 Ying Jin * 2 Ryan Li * 1 Michael Y. Li 1 Emmanuel Cand` es 3 4 Jure Leskovec 1 ## Abstract Hypotheses are central to information acquisition, decision-making, and discovery. However, many real-world hypotheses are abstract, highlevel statements that are difficult to validate directly. This challenge is further intensified by the rise of hypothesis generation from Large Language Models (LLMs), which are prone to hallucination and produce hypotheses in volumes that make manual validation impractical. Here we propose POPPER, an agentic framework for rigorous automated validation of free-form hypotheses. Guided by Karl Popper's principle of falsification, POPPER validates a hypothesis using LLM agents that design and execute falsification experiments targeting its measurable implications. A novel sequential testing framework ensures strict Type-I error control while actively gathering evidence from diverse observations, whether drawn from existing data or newly conducted procedures. We demonstrate POPPER on six domains including biology, economics, and sociology. POPPER delivers robust error control, high power, and scalability. Furthermore, compared to human scientists, POPPER achieved comparable performance in validating complex biological hypotheses while reducing time by 10 folds, providing a scalable, rigorous solution for hypothesis validation. POPPER is freely available at https://github. com/snap-stanford/POPPER . ## 1. Introduction A hypothesis is a theory or an explanation based on limited evidence. It forms the backbone of decision-making, information acquisition, and discovery across domains (Thompson & Skau, 2023). For example, a robot evaluates different * Equal contribution, alphabetic order 1 Department of Computer Science, Stanford University 2 Data Science Initiative & Department of Health Care Policy, Harvard University 3 Department of Statistics, Stanford University 4 Department of Mathematics, Stanford University. Correspondence to: Kexin Huang < kexinh@cs.stanford.edu > . hypotheses to decide what action to take next. A scientist decides which experiments to run to evaluate a hypothesis/theory. The marketing strategy decisions are guided by the hypothesized effect on increasing customer retention. Similarly, policymakers may rely on hypotheses about the outcomes of proposed interventions. Given their profound implications, it is important to validate hypotheses with supporting evidence. This need has grown increasingly urgent with the recent surge in hypotheses generated by Large Language Models (LLMs) (Wang et al., 2024b; Zhou et al., 2024). While these systems exhibit remarkable creativity and diversity, the plausibility of their generated hypotheses can vary significantly due to potential hallucinations (Huang et al., 2023). Moreover, the sheer volume of LLM-generated hypotheses makes it impractical to invest in each one immediately. Therefore, obtaining a reliable, scalable understanding of the quality of these hypotheses is essential to fully unlock their potential. Having said this, many real-world hypotheses are abstract natural language statements that are difficult to directly evaluate (Thompson & Skau, 2023; Godfrey-Smith, 2009). For example, while we might hypothesize that 'a gene causes a disease,' it is infeasible to test this statement directly as it stands. Instead, it must be translated into specific, measurable implications that can be experimented rigorously (Jun et al., 2022). Yet, even for a single hypothesis, the space of potential supportive implications is vast, highlighting the need for frameworks that can automate this evaluation process. Notably, such frameworks must also be statistically rigorous, avoiding false verifications of hypotheses that are not true (Neyman & Pearson, 1928; 1933; Fisher, 1936). Without such control, research efforts risk being misdirected, resources wasted, and harmful conclusions drawn, ultimately undermining progress and trust. Overall, this raises a critical question: How can we rigorously validate free-form hypotheses at scale? Present work. We introduce POPPER, a novel framework for rigorous and automated validation of free-form natural language hypotheses using LLM agents. Inspired by Karl Popper's principle of falsification (Popper, 2005), POPPER systematically challenges hypotheses by sequentially testing their measurable implications through diverse experiments, ranging from data analysis and simulations to real-world experiments and interventions. To automate this process, POPPER employs two specialized LLM agents with complementary roles. The Experiment Design Agent leverages reasoning capabilities and domain knowledge to identify a measurable implication (sub-hypothesis) of the main hypothesis and design a falsification experiment. Notably, this sub-hypothesis needs to be falsifiable with clear null and alternative definitions. Once designed, the Experiment Execution Agent implements the experiments, which may involve data collection, simulations, statistical analyses, or real-world procedures. This agent ultimately produces a p-value that summarizes the outcome of the falsification experiment. To maintain statistical rigor, POPPER introduces a novel sequential testing framework that aggregates evidence from multiple, potentially dependent LLM-generated tests while strictly controlling the Type-I error rate (i.e., the probability of incorrectly rejecting a true null hypothesis). Individual p-values are converted into e-values (Vovk & Wang, 2021), enabling the aggergation of cumulative evidence. By adaptively combining these e-values, POPPER determines whether to reject the hypothesis, conduct further experiments, or terminate the validation process. The framework's ability to make dynamic, statistically sound decisions is ensured by the any-time validity property of the combined e-values (Gr¨ unwald et al., 2020). By iteratively testing adaptively solicited implications of a hypothesis, POPPER systematically explores its flexibility while adhering to rigorous statistical principles. This provides a scalable and automated approach to hypothesis validation. We instantiated POPPER across six diverse domains, including biology, sociology, and economics. In our implementation, POPPER designs falsification experiments by leveraging large-scale, hypothesis-free datasets and executes them with a Python code environment. The process involves systematic data identification, preprocessing, analysis, and statistical evaluation, ultimately generating sequentially valid p-values. Our results demonstrate that POPPER effectively controls the Type-I error rate while achieving significant power improvements over existing methods. Additionally, an expert user study involving nine PhD-level biostatisticians and computational biologists found that POPPER matched human performance in hypothesis validation tasks while reducing validation time by an order of magnitude. ## 2. POPPER: a general framework for automated hypothesis validation ## 2.1. Background and Problem Formulation Following Majumder et al. (2024); Thompson & Skau (2023), we broadly define hypothesis H as a statement that defines relationships ( r ) between a set of variables ( V ) under contexts ( c ). For example, in the hypothesis H 'Gene VAV1 regulates IL2 production in immune tissue', V = { 'VAV1', 'IL2 production' } , r = 'regulate', and c = 'in the immune tissue'. To formalize the discussion, the hypothesis H is associated with a null hypothesis H 0 . H 0 describes a family P 0 of distributions that generate the data under the null, i.e., in uninteresting situations (such as 'Gene V A V1 does not regulate IL2 production'). In this way, H 0 being incorrect is of interest (the alternative hypothesis). Hypothesis validation aims to test the null hypothesis H 0 and suggest evidence for the alternative. The hypothesis validation task is defined as f : H →{ 0 , 1 } , where 0 stands for unvalidated and 1 stands for validated (claiming the alternative). Given a hypothesis H , a system or a program f designs and performs experiments and generates an answer in { 0 , 1 } . We denote ˆ y as the predicted validation status. An experiment is typically associated with the collection (or retrieval) and processing of datasets denoted as D . An LLM agent A is broadly defined as a program that takes in instructions in natural language and performs actions T with reasoning capabilities to solve the task given the instruction and outputs a natural language answer (Yao et al., 2023). For rigorous hypothesis validation, we adopt the classical Type-I error control as our primary criterion. The Type-I error is the probability of the system incorrectly claiming an 'interesting' finding (e.g., enriched gene expression) when the null hypothesis is true. Formally, the Type-I error rate is sup P ∈P 0 P (ˆ y = 1) , where the probability is over the data and the validation system. To ensure rigor, our goal is to control the Type-I error at a pre-defined level α ∈ (0 , 1) . Another important concept is the power of the validation system, which we define as P (ˆ y = 1) where P is the data distribution. While power-the ability to detect true effects-is important, its improvement is meaningful only when Type-I error control is ensured. Without this foundation, increased power risks invalid conclusions. ## 2.2. Overview of POPPER POPPER is an agentic framework to systematically validate a hypothesis by actively designing and executing a sequence of falsification experiments . This perspective is inspired by Karl Popper's philosophy of falsification (Popper, 2005): rather than trying to directly prove a hypothesis of interest, one can attempt to refute its logical implications through experiments. Suppose we want to investigate whether gene X is related to disease Y . Directly establishing such a relationship may be difficult; however, we can test one of its implications: if X truly has no relationship to Y , we might expect no significant difference in X 's expression levels when comparing cell types implicated in Y versus unrelated cell types. <latexi sh 1\_b 64="Yg 9M HO G CwL3 v QE >A B n c VD T J zF +I p r kPo5 d / W S u f U Z N K R j y q m X <latexi sh 1\_b 64="9 QTmpPAZGrd z3HY W 2c > B7n VDL gJ EO F +I k o5 U CK w N8 v S q f u / M XR y j <latexi sh 1\_b 64="qXk z U cNYo R5G3EP nH+ >A B7 VDLTgJ O F I p r CK dZm w v S W Q f u / M y j <latexi sh 1\_b 64="/k V F pM3 fR jL XuAS rI > B7H c N EJ U q C DgWv YG z wO K P Q T Z d y n o + m <latexi sh 1\_b 64="d pHQgIYwzPB vLR2 DZW nXo >A c V NS M EJ3Ur q/ F u j0 Cm5 T f K G k O y + i +1 <latexi sh 1\_b 64="q /G 5Q + Lm9 A BTcur VP U > D S N FJ 2v O zW Cg Kf Y 0I w oZ M R X Hj p d n k E y <latexi sh 1\_b 64="o9 mqL P Z7/ RDYcQ UwW d k >A C X V S N FJ3 2v E g pI fGy u B K H zr M j O T n + <latexi sh 1\_b 64=" K+Erm GR 8npILwMS cjzu3y >A C H VD N FJ PU d W BT ZQX7g v 2/o q O Y k f <latexi sh 1\_b 64="BCT/j KnV5Wr9dmv M p J D8 >A c LS N F U qO Gz u w f R E X Q k + Z g y I oH P Y <latexi sh 1\_b 64="cv V 3d9P8C FWU5D fE R >A n LS N J q M gKrkp u y O T mQ B + X j z wI o H Z Y G / Figure 1: Illustration of POPPER. Given a hypothesis and a pre-defined significance level α ∈ (0 , 1) , POPPER constructs sequential experiments to falsify the hypothesis. Each iteration proceeds as follows. First, an experiment design agent proposes a falsification experiment, which is refined through a self-critique process considering factors such as causality, data availability, and redundancy. The experiment is then evaluated by an LLM-as-a-judge relevance checker to ensure its alignment with the main hypothesis. If deemed relevant, the test is implemented by a ReAct-based experiment execution agent which obtains a p-value. P-values from multiple falsification experiments are aggregated into sequential e-values using a sequential testing framework. If the aggregated e-value exceeds 1 /α , we declare sufficient evidence to reject the null hypothesis. Otherwise, the process continues with the next falsification test. <details> <summary>Image 1 Details</summary>  ### Visual Description ## Diagram Type: Flowchart ### Overview The image is a flowchart that illustrates the process of designing and executing an experiment to test the hypothesis that GRAP2 regulates IL-2. The flowchart is divided into several stages, each represented by a box with a label and arrows indicating the flow of the process. ### Components/Axes - **Main Hypothesis (H)**: The central hypothesis that GRAP2 regulates IL-2. - **Type I Error Rate (α)**: The significance level of the experiment. - **Experiment Design Agent**: The entity responsible for designing the experiment. - **Experiment Execution Agent**: The entity responsible for executing the experiment. - **Falsification Experiment Proposal**: The initial proposal for the experiment. - **Null Sub-hypothesis (h₀)**: The hypothesis that GRAP2 levels are the same in immune tissues. - **Alternative Sub-hypothesis (h₁)**: The hypothesis that GRAP2 levels are different in immune tissues. - **Sequential Error Control**: A process for controlling the error rate in the experiment. - **Historical p-values**: The p-values from previous experiments. - **P-value (Pᵢ)**: The p-value for the current experiment. - **E-value (eᵢ)**: The E-value for the current experiment. - **Aggregation (E)**: The aggregated E-value from all experiments. - **Relevance Checker**: A tool for checking the relevance of the experiment. - **Self-refine**: A process for refining the experiment based on feedback. ### Detailed Analysis or ### Content Details - **Initial Proposal**: The initial proposal for the experiment is to test if GRAP2 is preferentially expressed in immune tissues. - **Null Sub-hypothesis**: The null sub-hypothesis is that GRAP2 levels are the same in immune tissues. - **Alternative Sub-hypothesis**: The alternative sub-hypothesis is that GRAP2 levels are different in immune tissues. - **Sequential Error Control**: The sequential error control process involves calculating the p-value for the current experiment and comparing it to the historical p-values. If the p-value is less than the significance level (α), the experiment is considered significant, and the null hypothesis is rejected. If the p-value is greater than or equal to α, the experiment is considered insignificant, and the null hypothesis is not rejected. - **Historical p-values**: The historical p-values are used to set the significance level (α) for the current experiment. - **P-value (Pᵢ)**: The p-value for the current experiment is calculated using a statistical test, such as the Mann-Whitney U test. - **E-value (eᵢ)**: The E-value for the current experiment is calculated as the inverse of the p-value. - **Aggregation (E)**: The aggregated E-value from all experiments is used to determine the overall significance of the experiment. - **Relevance Checker**: The relevance checker is used to determine whether the experiment is relevant to the main hypothesis. - **Self-refine**: The self-refine process involves refining the experiment based on feedback from the relevance checker. ### Key Observations - The experiment is designed to test the hypothesis that GRAP2 regulates IL-2. - The null sub-hypothesis is that GRAP2 levels are the same in immune tissues. - The alternative sub-hypothesis is that GRAP2 levels are different in immune tissues. - The sequential error control process is used to control the error rate in the experiment. - The historical p-values are used to set the significance level (α) for the current experiment. - The p-value (Pᵢ) and E-value (eᵢ) are calculated using a statistical test, such as the Mann-Whitney U test. - The aggregated E-value from all experiments is used to determine the overall significance of the experiment. - The relevance checker is used to determine whether the experiment is relevant to the main hypothesis. - The self-refine process involves refining the experiment based on feedback from the relevance checker. ### Interpretation The data suggests that the experiment is designed to test the hypothesis that GRAP2 regulates IL-2. The null sub-hypothesis is that GRAP2 levels are the same in immune tissues, and the alternative sub-hypothesis is that GRAP2 levels are different in immune tissues. The sequential error control process is used to control the error rate in the experiment, and the historical p-values are used to set the significance level (α) for the current experiment. The p-value (Pᵢ) and E-value (eᵢ) are calculated using a statistical test, such as the Mann-Whitney U test, and the aggregated E-value from all experiments is used to determine the overall significance of the experiment. The relevance checker is used to determine whether the experiment is relevant to the main hypothesis, and the self-refine process involves refining the experiment based on feedback from the relevance checker. The data suggests that the experiment is designed to test the hypothesis that GRAP2 regulates IL-2, and the results of the experiment will be used to determine whether the null hypothesis is rejected or not. </details> Hence, a potential falsification experiment is to measure expression for X , collect the relevant samples, and apply a statistical test ( e.g. a t -test) for the null hypothesis that there is no difference in mean expression. In this sense, each experimental design leverages a logical implication of the main hypothesis to gather evidence. One can design multiple experiments like this to refute the primary hypothesis. POPPER implements an iterative, LLM-driven framework for systematic falsification. At each round i , an experiment design agent proposes a falsification test for a subhypothesis h 0 i (e.g., 'no difference in expression'), based on the main hypothesis and available resources. An experiment execution agent then carries out the test - either by analyzing existing data, conducting lab measurements, or running simulations - and reports a p-value p i . A sequential error control step converts p i into an e-value e i (detailed in Section 2.3), ensuring statistically valid accumulation of evidence. This process repeats over multiple iterations, collecting e-values until either (i) the aggregated evidence surpasses a predefined threshold, leading to a rejection of the null hypothesis H 0 , or (ii) a maximum number of iterations is reached. Each experiment may involve real-world data collection or simulations. The only restriction is that it produces a valid p-value suitable for e-value computation under the specified null sub-hypothesis. Next, we formalize the theoretical underpinnings of this sequential approach and provide descriptions of the POPPER framework. ## 2.3. Validity of Type-I Error Control in POPPER This part lays out the general conditions needed for valid Type-I error control in POPPER. Assumption 1 (Implication) . If H 0 is true, then the null sub-hypothesis h 0 i is true for all i ≥ 1 . Assumption 1 requires that the null sub-hypothesis h 0 i describes a range of data generating processes that are contained in those described by H 0 . As we are to detail in Section 3, we leverage the reasoning capabilities of LLMs, as well as additional checks to overcome the intrinsic randomness in LLM agents to approximately fulfill this condition. Recall that an e-value e i ∈ R is computed based on the collected data in each iteration. Following Vovk & Wang (2021), an e-value is a non-negative random variable whose expectation is below 1 under the null hypothesis and such that if it were to take a large value, it would indicate strong evidence for refuting the null. E-values are our key instruments for Type-I error control. Compared with the classical concept of p-values, their advantages include (i) flexible combination of evidence 1 and (ii) adaptive stopping of the validation process (Gr¨ unwald et al., 2020). Let D be the data, POPPER could potentially interact with (including yetto-collect ones). To achieve these benefits, in POPPER we require the e-values to be sequentially valid. Assumption 2 (Sequential information) . The training process of the agents is independent of D . Let D i -1 := { D s } s ≤ i -1 be the datasets used by the agents before iteration i . The e-values obey E [ e i | D i -1 ] ≤ 1 under h 0 i . Assumption 2 requires that the e-value at each iteration is valid conditional on prior information. As we shall see in Section 3, POPPER achieves this by carefully controlling the information used at each iteration. In specific, suppose at iteration i , the agents determine a sub-hypothesis h 0 i and a test function f i ( · ) , and then compute e i = f i ( D i ) based on a collected dataset D i ( e.g. through transforming a p-value). Then, Assumption 2 holds if (1) the selection of h 0 i , and f i ( · ) only relies on D i -1 and metadata without involving the samples in the unused, yet-to-be-chosen datasets, and (2) E [ f ( D )] ≤ 1 for any fixed value of h (resp. f ) that f i (resp. h 0 i ) may take and any dataset D whose distribution obeys h . If D i is a dataset from a static database, then condition (1) means the decision of using D i does not involve the data in it; if D i is actively collected, then (1) is natural as the data must be collected after the design stage. The last assumption concerns the stopping rule of the validation process. It ensures that the aggregated evidence at the terminal iteration supports rigorous validation outputs. Assumption 3 (Optional stopping) . The random variable τ ∈ N + denoting the termination iteration is a stopping time with respect to the filtration F i = σ ( D i ) . That is, for every i , the event { τ = i } is measurable with respect to F i . Assumption 3 holds if the decision to stop or continue at iteration i only depends on D i . In POPPER, we determine termination through the aggregated evidence E i := ∏ i s =1 e s . These assumptions ensure the aggregated evidence { E i } is a super-martingale (also called e-process (Shafer, 2019; Gr¨ unwald et al., 2020)), and thus the E i at the terminal step can be used to produce the validation output with error control. Theorem 4 is a standard result following (Gr¨ unwald et al., 2020), proved in Appendix A.2 for completeness. Theorem 4. Define the aggregated evidence at the termination iteration as E := ∏ τ s =1 e s . Under Assumptions 1, 2 and 3, E is a valid e-value, i.e., E [ E ] ≤ 1 under H 0 . In addition, define the validation status as ˆ y = 1 { E ≥ 1 /α } . Then, P (ˆ y = 1) ≤ α under H 0 , where the probability P is over the randomness in the agents and the collected data. 1 Traditional methods like Fisher's combined test (Fisher, 1970) or Brown's method (Brown, 1975) rely on strong assumptions such as independent p-values or accurate modeling. They also cannot ensure Type-I error control with optional stopping (Assumption 3). ## 2.4. Agentic hypothesis validation framework We now introduce each component of POPPER in a general form. Although the particular implementation we showcase later uses a static database, POPPER can be deployed in any environment capable of producing valid p-values whether that involves laboratory experiments, real-time data collection, or computational simulations. The essence is to iteratively design and execute falsification experiments on sub-hypotheses derived from a main hypothesis H . Below, we describe how our agents accomplish this while maintaining the assumptions needed for Type-I error control. Experiment design agent. Given the main hypothesis H and history of previously tested sub-hypotheses (and their outcomes), the design agent proposes a new falsification experiment intended to refute H 0 . Concretely, it specifies: - A sub-hypothesis capturing a concrete implication of the main hypothesis. - The null h 0 i and alternative h 1 i to be tested. - Details of how to conduct the experiment in a given domain. This may involve recommending the collection of new laboratory samples, setting up a targeted simulation, or identifying a suitable dataset (if available). The design agent is assumed to have domain expertise or access to domain knowledge, allowing it to propose experiments that are both relevant for falsifying H 0 and feasible to implement. For instance, it might propose measuring gene-expression levels, or running a randomized simulation study, or analyzing an existing database - whatever is best to challenge the null sub-hypothesis. Critically, the design agent must ensure that the proposed experiment can, in principle, yield a valid p-value under h 0 i . We will later show how this agent's operations are automated in practice in Section 3. Experiment execution agent. Once an experiment is designed, it is handed off to the execution agent , which is responsible for carrying it out. In a laboratory setting, this agent might interface with robotic lab equipment or prompt human technicians to conduct the specified protocol. In a simulation, it would set up and run the relevant computational model. In a data analytics context, it would query and analyze the dataset. Regardless of the experimental modality, the only restriction is that it outputs a valid pvalue under h 0 i (Assumption 2). If an experiment fails because the protocol cannot be completed or the data are insufficient - it is simply recorded as a failed attempt, and the procedure moves on. In Section 3, we show how this agent is instantiated using a code-generation framework that automatically executes data queries and statistical analyses. Sequential aggregation of statistics for error control. After obtaining the new p-value p i , we aggregate existing falsification tests to collectively measure evidence for the main hypothesis while maintaining Type-I error control. As de- Table 1: Experiment design example. Designs for the hypothesis 'Gene ZAP70 regulates the production of Interleukin-2'. | Round | Falsification experiment description generated from POPPER experiment design agent | P-value | Cum. e-value | |---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|----------------| | 1 | 'Test if ZAP70 has significant physical protein-protein inter- actions with IL-2 pathway components using affinity capture Mass Spectrometry data' | 1.0 | 0.5 ✗ | | 2 | 'Test if ZAP70 expression levels correlate with IL-2 pathway genes across tissues using GTEx tissue expression data' | 8.8e-3 | 2.67 ✗ | | 3 | 'Test if genetic variants affecting ZAP70 expression (eQTLs) are also associated with changes in IL-2 pathway activity in immune cells using UKBB eQTL data' | - | - | | 4 | 'Test if rare missense variants in ZAP70 are significantly associated with immune phenotypes related to IL-2 function using GeneBASS missense variant data' | 4.7e-04 | 30.78 " | Table 2: Experiment execution example. Execution steps for the experiment 'Test if variants in the MAK16 locus region show overrepresentation of immune-trait GWAS associations.' We provide a summarized pseudo-code here for illustration purposes. | Step | Execution steps description from POPPER experiment execution agent | |--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 | Define a helper function to check if a trait is immune-related | | 2 | Find the MAK16 gene in df gene info | | 3 | Determine gene region bounds on chromosome (100 kb) | | 4 | Subset df variant table for variants in this region | | 5 | Merge with GWAS catalog | | 6 | Filter merged results for (a) p-value 5e-8 (b) immune-related traits using helper function in 1 | | 7 | Perform 500 permutations by randomly selecting a chromosome and a matching-length region, gathering variants, merging with the GWAS cata- log, filtering for immune-related traits with p-value 5e-8, and recording the immune-hit count for each permutation. | | 8 | Compute the empirical p-value | scribed in the proposed sequential testing framework in Section 2.3, the main technical tools we use are e-values (V ovk & Wang, 2021), which are amenable to combination of evidence and adaptive decisions to continue or not (safe testing) (Gr¨ unwald et al., 2020). Many e-value constructions ( e.g. likelihood ratios) require modeling assumptions, which are unsuitable given the flexibility given to our agent. Thus, we use the general 'p-to-e calibrator' (Vovk & Wang, 2021) to construct $$e _ { i } = \kappa \times p _ { i } ^ { \kappa - 1 } , \quad \kappa \in ( 0 , 1 ) . \quad ( 1 ) \quad t h e r a t i$$ It is straightforward to check that E [ e i | D i -1 ] ≤ 1 if each p i is a conditionally valid p-value, i.e., P ( p i ≤ t | D i -1 ) ≤ t for any t ∈ [0 , 1] . We then compute the aggregated evidence E i = ∏ i s =1 e s . If E i ≥ 1 /α , then H 0 is rejected and H is verified (obeying Assumption 3). If not, we proceed to the next iteration until a budget is reached. Theorem 4 ensures the Type-I error control of this procedure. ## 3. Instantiation of POPPER Thus far, we have described POPPER as a general, agentic framework capable of executing any type of experiment laboratory procedures, simulations, or data analyses - to test sub-hypotheses under a unifying Popperian falsification paradigm. In this section, we focus on our current instantiation , where experiments are drawn from a static corpus of massive hypothesis-free datasets ( D ) rather than real-world or real-time data acquisition. We emphasize that this is only one possible deployment of POPPER, chosen here for ease of implementation and reproducibility. Domains and hypotheses. Our demonstration uses two collections. The first, Target Validation (TargetVal) , addresses genotype-phenotype hypotheses in biology; it aggregates 22 tables (totaling ∼ 85 million records) from sources such as GTEx (Consortium, 2020), GWAS Catalog (MacArthur et al., 2017), and BioGrid (Oughtred et al., 2019). Hypotheses in TargetVal follow the template 'Gene A regulates Phenotype B,' and we assess them using two subtasks: Interleukin-2 (TargetVal-IL2) and Interferon-gamma (TargetVal-IFNG). Ground-truth hypotheses (treated as 'positive' references) were approximated based on genome-wide CRISPR screen data (Schmidt et al., 2022). The second, DiscoveryBench (Majumder et al., 2024), spans six domains (sociology, biology, humanities, economics, engineering, and meta-science), yielding 86 non-null hypotheses (after deduplication) that are grounded in peer-reviewed research. Each hypothesis is paired with a set of relevant dataset. In all cases, POPPER is provided only with the high-level schema (row and column names, any available short text descriptions) of each dataset and the main hypothesis H . It must then propose and implement sub-hypothesis falsification experiments by querying and analyzing the raw data. Instantiation of the experiment design agent. At iteration i , the Design Agent A design receives the main hypothesis H , previously proposed falsification subhypotheses { h 1 , . . . , h i -1 } , their corresponding p-values { p 1 , . . . , p i -1 } , and the metadata from the database D , and then intelligently designs a new falsification experiment with sub-hypothesis h i . To ensure robustness, A design operates under metadata-only access, meaning it sees only the schema of each table but has no access to raw data or summary statistics, thereby satisfying Assumption 2. In the experiment proposal step, the agent generates a concise rationale, along with a null hypothesis h 0 i and an alternative hypothesis h 1 i . To enhance quality, we incorporate Self-Refinement (Madaan et al., 2024), employing a chainof-thought approach that prompts the LLM to iteratively improve its proposal based on three key criteria: novelty (avoiding redundant sub-hypotheses), implementability (ensuring feasibility given metadata), and logical relevance (confirming that H implies h i ). A real-world example is illustrated in Table 1. This demonstrates the agent's ability to systematically design rigorous and biologically meaningful experiments, highlighting its effectiveness in guiding the falsification process. A detailed analysis of the proposed experiments is available at Section 4.2. Relevance checker. Even with self-refinement, the Design Agent may produce experiments that are tangential to the main hypothesis H . To enforce Assumption 1, we Table 3: Type-I error/power across baselines, variations, ablations, and POPPER. A method is considered to achieve Type I-error control if the pre-defined threshold falls within 1 standard deviation of the method's result. For methods that fail to meet this criterion, the power metric is grayed out, as it becomes invalid. Mean and standard deviation for all metrics are calculated from 5 independent runs. | Method | Type I Error ( α = 0.1) | Type I Error ( α = 0.1) | Type I Error ( α = 0.1) | Power | Power | Power | |-------------------------------------------|-----------------------------------------------------------------|-----------------------------------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------| | Method | DiscoveryBench | TargetVal-IL2 | TargetVal-IFNG | DiscoveryBench | TargetVal-IL2 | TargetVal-IFNG | | CodeGen CodeGen (o1) ReAct Self-Refine | 0.145 ± 0.031 ✗ 0.248 ± 0.015 ✗ 0.078 ± 0.061 " 0.117 ± 0.028 ✗ | 0.020 ± 0.014 " 0.013 ± 0.012 " 0.000 ± 0.000 " 0.100 ± 0.069 " | 0.004 ± 0.009 " 0.000 ± 0.000 " 0.000 ± 0.000 " 0.067 ± 0.064 " | 0.378 ± 0.066 0.419 ± 0.028 0.383 ± 0.017 0.476 ± 0.066 | 0.140 ± 0.022 0.250 ± 0.100 0.010 ± 0.022 0.183 ± 0.029 | 0.040 ± 0.042 0.183 ± 0.076 0.020 ± 0.045 0.067 ± 0.064 | | Fisher Combined Test LLM-Likelihood ratio | 0.311 ± 0.040 ✗ 0.152 ± 0.031 ✗ | 0.264 ± 0.083 ✗ 0.016 ± 0.014 " | 0.173 ± 0.023 ✗ 0.180 ± 0.028 ✗ | 0.741 ± 0.058 0.428 ± 0.034 | 0.800 ± 0.071 0.185 ± 0.074 | 0.650 ± 0.050 0.357 ± 0.132 | | POPPER-NoReleCheck POPPER-CodeGen | 0.134 ± 0.021 ✗ 0.140 ± 0.022 ✗ | 0.340 ± 0.139 ✗ 0.105 ± 0.017 " | 0.300 ± 0.113 ✗ 0.090 ± 0.045 " | 0.610 ± 0.042 0.544 ± 0.032 | 0.897 ± 0.004 0.526 ± 0.133 | 0.717 ± 0.126 0.450 ± 0.079 | | POPPER (Ours) | 0.103 ± 0.020 " | 0.082 ± 0.046 " | 0.085 ± 0.028 " | 0.638* ± 0.066 | 0.580* ± 0.125 | 0.591* ± 0.069 | introduce a relevance checker , an LLM-based function R ( h ) ∈ [0 , 1] that estimates how strongly the proposed null sub-hypothesis h is implied by H 0 . If R ( h ) < r 0 (a predefined threshold), we discard that experiment and prompt A design to propose a new one. This pruning mitigates the risk that an irrelevant null might be 'falsified,' incorrectly supporting the hypothesis (thus inflating the Type-I error). Instantiation of the experiment execution agent. Once a proposed experiment passes the relevance check, the Execution Agent A exec carries it out by querying and analyzing the raw data in D to output a p-value. To give the agent flexibility, we provide a coding environment where it can write and run Python scripts using essential libraries including pandas , statsmodels , and scipy . Concretely, we employ ReAct (Yao et al., 2023) where the agent incrementally executes the experiment via a cycle of actions (executing code), observations (inspecting code output), and reasoning based on the observed output. In practice, A exec typically inspects and retrieves the dataset, performs preprocessing, fixes errors, runs appropriate statistical tests, fits models, and finally summarizes or visualizes the findings. Without explicit prompting, it selects suitable tests (e.g., t -test, chi-squared, Mann-Whitney U -test) based on the data distribution. Table 2 shows an example, and Section 4.2 analyzes the execution steps in detail. ## 4. Experiments We evaluate POPPER in terms of Type-I error control, power improvements, expert user studies, ablations, human annotations, and failure analysis. Evaluation setup. We assess Type-I error by creating negative examples through random column-wise permutations in each dataset, ensuring the null hypothesis holds. For DiscoveryBench, we generate as many negative examples as positive ones. For the target validation benchmark (with only 20 positives), we create 50 negatives. We measure Type-I error by the proportion of 'reject' decisions ( ˆ y = 1 ) on negative examples and Power by the proportion of 'reject' decisions on positive examples. We set a nominal Type-I error level α = 0 . 1 . Unless noted otherwise, we use Claude-Sonnet-3.5 as our LLM, with a maximum of 3 tests on DiscoveryBench and 5 on target validation (due to more complex hypotheses in the latter scenario). Baselines & variations. We group comparing methods into two categories. (1) Baselines. Since this is a novel application with no direct references, we compare against three general-purpose task resolvers: CodeGen (Ridnik et al., 2024), which generates code; ReAct (Yao et al., 2023), which iteratively combines reasoning and coding; and Selfrefine (Madaan et al., 2024), which refines CodeGen outputs via a critic. None include specialized mechanisms for statistical rigor. We also evaluated an enhanced CodeGen-o1 with improved reasoning. (2) Variations of POPPER . These include Fisher , which uses p-values and Fisher's combined test (Fisher, 1970) instead of e-values; LLM-Likelihood Ratio , which relies on an LLM to estimate the (optimal) likelihood ratio (Zheng et al., 2023) rather than a p-to-e calibrator; POPPER -NoReleCheck , omitting the relevance checker; and POPPER -CodeGen , which substitutes ReAct with direct code generation for statistical tests. ## 4.1. Results POPPER achieves Type-I error control. Table 3 reports the Type-I error rates and several key observations are in order. First, most baselines fail to consistently control the Type-I error, while POPPER remains below the nominal level across all datasets. This underscores the necessity of principled statistical design in LLM-driven hypothesis validation; without such rigor, the flexibility of LLM agents can inflate Type-I errors. Second, the comparison against Fisher's combined test highlights the benefits of e-values in aggregating evidence. Third, the LLM-Likelihood Ratio method lacks calibration, overly conservative for TargetVal- IL2 and too liberal for DiscoveryBench and TargetVal-IFNG, illustrating the need for strict statistical control rather than relying solely on LLM-based estimations. Finally, removing the relevance checker (POPPER-NoReleCheck) significantly raises the Type-I error due to irrelevant and misleading tests. Together, these results establish POPPER as a robust framework for agentic hypothesis validation. POPPER has significant power improvement. Table 3 shows the power across three benchmarks. First, we exclude any method with an uncontrolled Type-I error (gray-shaded in the table), as their power estimates are invalid. Among methods that do control the Type-I error, POPPER consistently achieves the highest power: on DiscoveryBench, it delivers 66.5% greater power than ReAct, and on TargetValIL2, it outperforms Self-Refine by a factor of 3.17. This highlights the strength of POPPER's iterative testing mechanism, which continually accumulates evidence to improve validation. Second, POPPER with the ReAct coding agent outperforms POPPER-CodeGen in power - even with a lower Type-I error. The likely cause is that its reasoning module enables more effective falsification tests. Overall, these results confirm the ability of POPPER to balance high power with error control, making it a reliable and efficient approach to hypothesis validation. POPPER compares with human experts. We recruited nine computational biologists and bioinformaticians (either PhD holders or candidates) to perform hypothesis validation on TargetVal-IL2 (details in Appendix F). Figure 2 shows that the Type-I error and power of POPPER closely match those of the human participants, with no statistically significant differences given the small sample size. Notably, POPPER completed tasks 9.7 times faster, generated 3.6 times more lines of code, and performed 2.5 times more statistical tests, underscoring its efficiency gains. Qualitative analysis (the right half of Figure 2, where the numbers represent the amount of distinct statistical tests in each category) revealed substantial overlap between human experts and POPPER in both biological falsification experiments (e.g., correlation in gene expression levels, network interactions, eQTL tests) and statistical methods (e.g., permutation, t -test, chi-squared), reinforcing the soundness of POPPER in automating validation tasks. Performance varies across a wide range of LLMs. Since POPPER must propose meaningful falsification tests and compute valid p-values (per Assumptions 1 and 2), it requires strong reasoning and coding capabilities. We evaluated several LLMs on DiscoveryBench and TargetVal-IL2, including closed-source models (Claude Haiku 3.5, Sonnet 3.5, GPT-4o, o1) and the open-source Llama 3.3 70B. Table 4 shows that higher-capability models are critical: Claude Haiku 3.5 has a high Type-I error, whereas Llama, GPT-4o, Sonnet, and o1 maintained reasonable error control. <latexi sh 1\_b 64=" f/JYK9QjF5O y IgG P XU >A B73 c VDLT E z C qZ R 0dM H Sm p w v o k N + r nW u Figure 2: Expert human study. POPPER achieved similar power and Type-I error rates to human experts while significantly reducing task completion time. It also generated more lines of code and conducted more statistical tests. Qualitatively, POPPER and human experts exhibited substantial overlap in both the designed falsification experiments and the statistical methods employed. <details> <summary>Image 2 Details</summary>  ### Visual Description ## Heatmap: Type I Error Estimation ### Overview The heatmap illustrates the Type I error estimation for two groups: Popper and Human. The rows represent different types of errors, while the columns indicate the percentage of errors for each group. ### Components/Axes - **Rows**: Type I error estimation - **Columns**: Popper and Human - **Values**: Percentage of errors ### Detailed Analysis or ### Content Details - **Popper**: 11.1% (Type I error) and 22.2% (Type II error) - **Human**: 11.1% (Type I error) and 22.2% (Type II error) ### Key Observations - Both groups have the same percentage of Type I errors (11.1%). - Both groups have the same percentage of Type II errors (22.2%). ### Interpretation The heatmap suggests that both Popper and Human have similar error rates in Type I error estimation. This indicates that the error rates are comparable between the two groups, which could imply that the methods used by both groups are equally effective in detecting Type I errors. </details> Table 4: Evaluation of various LLM backbones with POPPER. | Method | Type I Error ( α = 0.1) | Type I Error ( α = 0.1) | Power | Power | |------------------------------------------------------------|------------------------------------------------------------------------|------------------------------------------------------------------------|------------------------------------------------------------------------|------------------------------------------------------------------------| | | DiscoveryBench | TargetVal-IL2 | DiscoveryBench | TargetVal-IL2 | | Claude-Haiku-3.5 Llama 3.3 70B GPT-4o Claude-Sonnet-3.5 o1 | 0.230 ± 0.079 0.147 ± 0.036 0.143 ± 0.039 0.103 ± 0.020 0.091* ± 0.015 | 0.780 ± 0.120 0.116 ± 0.020 0.096 ± 0.043 0.082 ± 0.046 0.031* ± 0.015 | 0.844 ± 0.017 0.690 ± 0.027 0.730 ± 0.054 0.638 ± 0.066 0.654* ± 0.019 | 0.835 ± 0.113 0.515 ± 0.078 0.385 ± 0.102 0.580* ± 0.125 0.336 ± 0.121 | Among them, o1 performed best on DiscoveryBench, and GPT-4o excelled in power for DiscoveryBench, whereas Sonnet led on TargetVal-IL2. These results emphasize the importance of robust reasoning and coding skills for effective hypothesis validation and highlight nuanced performance trade-offs. ## 4.2. Analysis and Discussion Qualitative characterization. We characterize the trajectories of POPPER in Figure 3 (procedure described in Appendix E). In TargetVal, we observe that POPPER designed experiments that span a broad set of biological tests, including protein-protein interaction networks, expression correlation analyses, eQTL regulatory tests, loss-of-function studies, and genetic perturbation tests. During each iteration, the execution agent typically performs up to 14 distinct steps: dataset inspection, preprocessing, model fitting, error handling, statistical testing, visualization, and summarization. Notably, POPPER carefully selects statistical methods based on modeling assumptions (e.g., chi-squared, hypergeometric, Fisher's, and permutation tests) and often includes well-chosen negative controls. Interestingly, non-parametric tests are most frequent, making them robust to various data distributions. Visualizing the e-value trajectories reveals that evidence against the null accumulates quickly under the alternative while remaining below the nominal threshold un- Figure 3: Characterization of POPPER. (1) POPPER designs biologically relevant falsification experiments. (2) It performs multiple logical steps to execute the experiment. (3) It employs a wide range of statistical tests. (4) Progression of cumulative e-values across multiple iterations of falsification tests. More details are available in Appendix E. <details> <summary>Image 3 Details</summary>  ### Visual Description ## [Chart/Diagram Type]: [Brief Title] ### Overview The image displays a collection of four technical diagrams, each representing different aspects of a research or analysis process. The diagrams are labeled as 1, 2, 3, and 4, and they are arranged in a grid format with two rows and two columns. ### Components/Axes - **Diagram 1**: This bar chart shows the proportion of domain falsification tests proposed for different types of tests. The categories are listed on the x-axis, and the proportion is represented by the height of the bars on the y-axis. - **Diagram 2**: This bar chart illustrates the frequency of actions taken within a round. The actions are listed on the x-axis, and the frequency is represented by the height of the bars on the y-axis. - **Diagram 3**: This bar chart displays the proportion of tests completed in each round. The rounds are listed on the x-axis, and the proportion is represented by the height of the bars on the y-axis. - **Diagram 4**: This line chart shows the cumulative value of a variable over time. The x-axis represents time, and the y-axis represents the cumulative value. ### Detailed Analysis or ### Content Details - **Diagram 1**: The chart indicates that the majority of domain falsification tests are proposed for genetic variant impact tests, followed by genetic interaction network tests, and then genetic variant association tests. The proportion of tests proposed decreases as we move from genetic variant impact tests to cross-phenotype association tests. - **Diagram 2**: The chart shows that the frequency of actions taken within a round is relatively high, with the majority of actions occurring in the first few rounds. The frequency of actions decreases as we move to later rounds. - **Diagram 3**: The chart indicates that the proportion of tests completed is relatively high, with the majority of tests completed in the first few rounds. The proportion of tests completed decreases as we move to later rounds. - **Diagram 4**: The chart shows that the cumulative value of the variable increases over time, with the rate of increase slowing down as time progresses. ### Key Observations - The majority of domain falsification tests are proposed for genetic variant impact tests. - The frequency of actions taken within a round is relatively high. - The proportion of tests completed is relatively high. - The cumulative value of the variable increases over time, with the rate of increase slowing down as time progresses. ### Interpretation The data suggests that the majority of domain falsification tests are proposed for genetic variant impact tests, and that the frequency of actions taken within a round is relatively high. The proportion of tests completed is relatively high, and the cumulative value of the variable increases over time, with the rate of increase slowing down as time progresses. This data could be used to inform future research or analysis processes, and to identify areas for improvement. </details> Figure 4: Sensitivity analysis. (1) Empirical Type-I error at various nominal levels α . (2) Power and Type-I error at various budgets as a function of the number of maximum tests. <details> <summary>Image 4 Details</summary>  ### Visual Description ## Heatmap: Relationship between Pre-defined Alpha and Type I Error Rate ### Overview The heatmap illustrates the relationship between the pre-defined alpha level and the Type I error rate in statistical hypothesis testing. The x-axis represents the pre-defined alpha levels, ranging from 0.10 to 0.20, while the y-axis shows the Type I error rate, ranging from 0.00 to 0.60. The color intensity indicates the magnitude of the error rate. ### Components/Axes - **Pre-defined Alpha (x-axis)**: Ranges from 0.10 to 0.20. - **Type I Error Rate (y-axis)**: Ranges from 0.00 to 0.60. - **Color Intensity**: Indicates the magnitude of the Type I error rate, with darker shades representing higher error rates. ### Detailed Analysis or ### Content Details The heatmap shows a clear trend where the Type I error rate increases as the pre-defined alpha level increases. This is evident from the diagonal line that runs from the bottom-left to the top-right of the heatmap. The line is represented by a dashed blue line, which serves as a reference to the expected Type I error rate for a given pre-defined alpha level. ### Key Observations - **Trend**: The Type I error rate increases with the pre-defined alpha level. - **Reference Line**: The dashed blue line represents the expected Type I error rate for a given pre-defined alpha level. - **Color Gradient**: The color gradient indicates the magnitude of the Type I error rate, with darker shades representing higher error rates. ### Interpretation The heatmap suggests that as the pre-defined alpha level increases, the likelihood of making a Type I error also increases. This is because a higher alpha level means a higher threshold for rejecting the null hypothesis, which increases the probability of incorrectly rejecting a true null hypothesis. The reference line provides a benchmark to compare the actual Type I error rate against the expected rate for a given pre-defined alpha level. The color gradient helps in visualizing the magnitude of the error rate, with darker shades indicating higher error rates. </details> der the null, underscoring the rigor and power of sequential testing. Sensitivity analysis. Figure 4 presents the robustness of POPPER under different settings. First, we varied the significance level α and found that POPPER consistently maintained Type-I error control. Second, we examined the effect of increasing the budget (maximum number of tests). While Type-I error remained well-controlled, the power rose with additional tests, indicating that POPPER can accumulate more diverse evidence when given more computational resources. These results demonstrate the scalability of evalues to both small and large numbers of sequential tests, allowing POPPER to achieve higher discovery rates as resources increase. Human annotations of falsification test quality. To assess the implication strength of LLM-generated falsification tests, three authors independently rated 90 randomly selected proposals using the same rubric provided to the ReleCheck agent (Appendix 4). After calibration, the annotators achieved a high inter-rater agreement (Kendall's W = 0 . 91 ). The agent's ratings correlated strongly with human judgments (Spearman's ρ = 0 . 55 , p = 5 × 10 -6 ), though it slightly overestimated the relevance of the implications: it labeled 85% of proposals as 'strongly implied,' compared to a 77% pass rate among human evaluators. These findings indicate that while the ReleCheck agent aligns reasonably well with human perspectives, further calibration and domain-specific expertise are needed to enhance the reliability of falsification test selection. Error analysis. We analyzed potential failure modes in POPPER's hypothesis validation workflow. Using an LLM to categorize errors followed by human inspections, we identified the top reasons for failure: misinterpreted p-values (35.9%), ineffective falsification experiment design (28.1%), falsification test breaks implication (17.2%), and incorrect test implementation (8.6%). Hallucination was minimal (0.8%). More details are provided in Appendix D. Overall, while agentic automation holds promise, our findings highlight areas needing further improvement, guiding future work on more robust hypothesis validation pipelines. ## 5. Related Work We discuss here related works that are closest to POPPER and provide extended discussion on other related works in Appendix B. LLMs have been widely explored for hypothesis generation, with works focusing on domain-specific ideas (Wang et al., 2024a; Baek et al., 2024; Yang et al., 2024b) and comparisons between AI-generated and expert proposals (Si et al., 2024). Beyond idea generation, some studies refine hypotheses (Honovich et al., 2023; Wang et al., 2024c) or ground them in datasets (Majumder et al., 2024), yet few systematically test free-form hypotheses under rigorous statistical controls. While certain works evaluate LLM-driven experimental protocols (Tian et al., 2024; Gu et al., 2024) or integrate hypothesis and code generation (Li et al., 2024b; Lu et al., 2024; Ifargan et al., 2024; Majumder et al., 2024), they often lack strong error control. Unlike these, POPPER conducts robust statistical validation of both LLM- and human-generated hypotheses through a sequential falsification framework, ensuring reliability. Although Li et al. (2024a) also uses hypothesis testing as a way to challenge language models, POPPER uniquely targets freeform natural language hypotheses and offers rigorous error control. ## 6. Conclusion We proposed POPPER, an LLM-based framework for validating free-form hypotheses. By integrating a sequential testing paradigm with automated experiment design and execution, POPPER delivers scalable, statistically rigorous hypothesis validation. This work represents an early exploration, and several aspects can be further improved. Refining test relevance and ensuring robust LLM implementations remain challenges. Future work can also extend POPPER to control other error metrics (e.g., false discovery rate), further broadening its utility in scientific discovery and beyond. ## Impact Statement This work introduces POPPER, a statistically rigorous agentic framework for hypothesis validation using Large Language Model (LLM) agents. By combining advanced natural language processing capabilities with robust statistical methodologies, POPPER addresses the critical challenge of evaluating and validating hypotheses generated by LLMs, ensuring that only evidence-backed hypotheses guide future research. The broader implications of this work span multiple domains, including biology, economics, and social sciences, where hypothesis generation and validation play a pivotal role in advancing knowledge. From an ethical perspective, POPPER's emphasis on rigorous statistical validation and Type-I error control mitigates the risks associated with hallucinated or unsupported hypotheses. This ensures that research resources are directed toward meaningful and plausible hypotheses, reducing the potential for wasted efforts and false conclusions that could mislead scientific progress or policy decisions. Additionally, by automating and accelerating the hypothesis validation process, POPPER democratizes access to high-quality scientific methodologies, enabling smaller research teams and resource-limited institutions to conduct advanced analyses. ## Acknowledgement We thank Tatsunori Hashimoto and members of the Jure Leskovec lab for discussions and for providing feedback on our manuscript. We thank the expert user study participants: Michael Bereket, Minta Lu, Peter Pao-Huang, Weixu Wang, Boyang Fu, Hanchen Wang, Hao Xue, Serena Zhang, Yanay Rosen, and Zoe Piran. We also gratefully acknowledge the support of NSF under Nos. OAC-1835598 (CINES), CCF-1918940 (Expeditions), DMS-2327709 (IHBEM), IIS2403318 (III); Stanford Data Applications Initiative, Wu Tsai Neurosciences Institute, Stanford Institute for HumanCentered AI, Chan Zuckerberg Initiative, Amazon, Genentech, GSK, Hitachi, SAP, and UCB. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities. ## References Agassi, J. Popper and his popular critics: Thomas kuhn, paul feyerabend and imre lakatos. In SpringerBriefs in Philosophy . Springer, 2014. doi: 10.1007/978-3-319-06587-8. Ajith, A., Xia, M., Chevalier, A., Goyal, T., Chen, D., and Gao, T. Litsearch: A retrieval benchmark for scientific literature search, 2024. URL https://arxiv.org/ abs/2407.18940 . Alet, F., Lopez-Contreras, J., Koppel, J., Nye, M., SolarLezama, A., Lozano-Perez, T., Kaelbling, L., and Tenenbaum, J. A large-scale benchmark for few-shot program induction and synthesis. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pp. 175-186. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/ v139/alet21a.html . Baek, J., Jauhar, S. K., Cucerzan, S., and Hwang, S. J. Researchagent: Iterative research idea generation over scientific literature with large language models, 2024. URL https://arxiv.org/abs/2404.07738 . Benjamini, Y. Selective inference: The silent killer of replicability. 2020. Benjamini, Y. and Hochberg, Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological) , 57(1):289-300, 1995. Brown, M. B. 400: A method for combining nonindependent, one-sided tests of significance. Biometrics , pp. 987-992, 1975. Collaboration, O. S. Estimating the reproducibility of psychological science. Science , 349(6251):aac4716, 2015. Consortium, G. The gtex consortium atlas of genetic regulatory effects across human tissues. Science , 369(6509): 1318-1330, 2020. - D'Arcy, M., Hope, T., Birnbaum, L., and Downey, D. Marg: Multi-agent review generation for scientific papers, 2024. URL https://arxiv.org/abs/2401.04259 . - Fisher, R. A. Design of experiments. British Medical Journal , 1(3923):554, 1936. - Fisher, R. A. Statistical methods for research workers. In Breakthroughs in statistics: Methodology and distribution , pp. 66-70. Springer, 1970. - Gendron, G., Bao, Q., Witbrock, M., and Dobbie, G. Large language models are not strong abstract reasoners, 2024. URL https://arxiv.org/abs/2305.19555 . - Godfrey-Smith, P. Theory and reality: An introduction to the philosophy of science . University of Chicago Press, 2009. - Goodman, N. Fact, Fiction, and Forecast . Harvard University Press, Cambridge, MA, 1983. - Gr¨ unwald, P., de Heide, R., and Koolen, W. M. Safe testing. In 2020 Information Theory and Applications Workshop (ITA) , pp. 1-54. IEEE, 2020. - Gu, K., Shang, R., Jiang, R., Kuang, K., Lin, R.-J., Lyu, D., Mao, Y., Pan, Y., Wu, T., Yu, J., Zhang, Y., Zhang, T. M., Zhu, L., Merrill, M. A., Heer, J., and Althoff, T. Blade: Benchmarking language model agents for datadriven science, 2024. URL https://arxiv.org/ abs/2408.09667 . - Han, S. J., Ransom, K., Perfors, A., and Kemp, C. Inductive reasoning in humans and large language models, 2023. URL https://arxiv.org/abs/2306.06548 . - Honovich, O., Shaham, U., Bowman, S. R., and Levy, O. Instruction induction: From few examples to natural language task descriptions. In Rogers, A., BoydGraber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 19351952, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long. 108. URL https://aclanthology.org/2023. acl-long.108 . - Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems , 2023. - Ifargan, T., Hafner, L., Kern, M., Alcalay, O., and Kishony, R. Autonomous llm-driven research from data to humanverifiable research papers, 2024. URL https:// arxiv.org/abs/2404.17605 . - Ioannidis, J. P. Why most published research findings are false. PLoS medicine , 2(8):e124, 2005. - Jun, E., Birchfield, M., De Moura, N., Heer, J., and Just, R. Hypothesis formalization: Empirical findings, software limitations, and design implications. ACM Transactions on Computer-Human Interaction (TOCHI) , 29(1):1-28, 2022. - Kuhn, T. S. The Structure of Scientific Revolutions . University of Chicago Press, Chicago, 1st edition, 1962. - Lakatos, I. The Methodology of Scientific Research Programmes . Cambridge University Press, Cambridge, 1978. - Lehr, S. A., Caliskan, A., Liyanage, S., and Banaji, M. R. Chatgpt as research scientist: Probing gpt's capabilities as a research librarian, research ethicist, data generator and data predictor, 2024. URL https://arxiv.org/ abs/2406.14765 . - Li, M. Y., Vajipey, V., Goodman, N. D., and Fox, E. B. Critical: Critic automation with language models. arXiv preprint arXiv:2411.06590 , 2024a. - Li, R., Patel, T., Wang, Q., and Du, X. Mlr-copilot: Autonomous machine learning research based on large language models agents, 2024b. URL https://arxiv. org/abs/2408.14033 . - Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y ., McFarland, D., and Zou, J. Can large language models provide useful feedback on research papers? a large-scale empirical analysis, 2023. URL https://arxiv.org/abs/ 2310.01783 . - Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery, 2024. URL https: //arxiv.org/abs/2408.06292 . - MacArthur, J., Bowler, E., Cerezo, M., Gil, L., Hall, P., Hastings, E., Junkins, H., McMahon, A., Milano, A., Morales, J., et al. The new nhgri-ebi catalog of published genome-wide association studies (gwas catalog). Nucleic acids research , 45(D1):D896-D901, 2017. - Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. NeurIPS , 36, 2024. - Majumder, B. P., Surana, H., Agarwal, D., Mishra, B. D., Meena, A., Prakhar, A., Vora, T., Khot, T., Sabharwal, A., and Clark, P. Discoverybench: Towards data-driven discovery with large language models. arXiv preprint arXiv:2407.01725 , 2024. - Manning, B. S., Zhu, K., and Horton, J. J. Automated social science: Language models as scientist and subjects, 2024. URL https://arxiv.org/abs/2404.11794 . - Maxwell, N. Popper, kuhn, lakatos and aim-oriented empiricism. arXiv preprint , 2012. - Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern machines, 2023. URL https://arxiv.org/abs/2307.04721 . - Moskvichev, A., Odouard, V. V., and Mitchell, M. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain, 2023. URL https: //arxiv.org/abs/2305.07141 . - Neyman, J. and Pearson, E. S. On the use and interpretation of certain test criteria for purposes of statistical inference part i. Biometrika , 20(1-2):175-240, 1928. - Neyman, J. and Pearson, E. S. The testing of statistical hypotheses in relation to probabilities a priori. In Mathematical proceedings of the Cambridge philosophical society , volume 29, pp. 492-510. Cambridge University Press, 1933. - Oughtred, R., Stark, C., Breitkreutz, B.-J., Rust, J., Boucher, L., Chang, C., Kolas, N., ODonnell, L., Leung, G., McAdam, R., et al. The biogrid interaction database: 2019 update. Nucleic acids research , 47(D1):D529D541, 2019. - Philosophy Institute. Imre lakatos approach: Bridging popper and kuhn in philosophy of science, 2023. URL https://philosophy.institute/ philosophy-of-science-and-cosmology/ imre-lakatos-philosophy-science-bridge/ Accessed: 2025-01-29. - Popper, K. The Logic of Scientific Discovery . Hutchinson, London, 1959. - Popper, K. The logic of scientific discovery . Routledge, 2005. - Press, C. U. Normal science and dogmatism, paradigms and progress: Kuhn versus popper and lakatos. 2009. - Press, O., Hochlehnert, A., Prabhu, A., Udandarao, V., Press, O., and Bethge, M. Citeme: Can language models accurately cite scientific claims?, 2024. URL https://arxiv.org/abs/2407.12861 . - Qiu, L., Jiang, L., Lu, X., Sclar, M., Pyatkin, V., Bhagavatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., and Ren, X. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis . - refinement, 2024. URL https://arxiv.org/abs/ 2310.08559 . - Ridnik, T., Kredo, D., and Friedman, I. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500 , 2024. - Rubin, M. The replication crisis is less of a 'crisis' in lakatos' philosophy of science. European Journal for Philosophy of Science , 15(5), 2025. doi: 10.1007/ s13194-024-00629-x. - Schmidt, R., Steinhart, Z., Layeghi, M., Freimer, J. W., Bueno, R., Nguyen, V. Q., Blaeschke, F., Ye, C. J., and Marson, A. Crispr activation and interference screens decode stimulation responses in primary human t cells. Science , 375(6580):eabj4008, 2022. - Shafer, G. The language of betting as a strategy for statistical and scientific communication. arXiv preprint arXiv:1903.06991 , 2019. - Si, C., Yang, D., and Hashimoto, T. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers, 2024. URL https://arxiv. org/abs/2409.04109 . - Tang, X., Zheng, Z., Li, J., Meng, F., Zhu, S.-C., Liang, Y., and Zhang, M. Large language models are in-context semantic reasoners rather than symbolic reasoners, 2023. URL https://arxiv.org/abs/2305.14825 . - Thompson, W. H. and Skau, S. On the scope of scientific hypotheses. Royal Society Open Science , 10(8):230607, 2023. - Tian, M., Gao, L., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., Liu, S., Luo, D., Ma, Y., Tong, H., Trinh, K., Tian, C., Wang, Z., Wu, B., Xiong, Y., Yin, S., Zhu, M., Lieret, K., Lu, Y., Liu, G., Du, Y., Tao, T., Press, O., Callan, J., Huerta, E., and Peng, H. Scicode: A research coding benchmark curated by scientists, 2024. URL https://arxiv.org/abs/ 2407.13168 . - van Fraassen, B. C. The Scientific Image . Clarendon Press, Oxford, 1980. - Vovk, V. and Wang, R. E-values: Calibration, combination and applications. The Annals of Statistics , 49(3):17361754, 2021. - Wang, Q., Downey, D., Ji, H., and Hope, T. Scimon: Scientific inspiration machines optimized for novelty, 2024a. URL https://arxiv.org/abs/2305.14259 . - Wang, R. and Ramdas, A. False discovery rate control with e-values. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(3):822-852, 2022. - Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. Hypothesis search: Inductive reasoning with language models. ICLR , 2024b. - Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. Hypothesis search: Inductive reasoning with language models, 2024c. URL https://arxiv. org/abs/2309.05660 . - Webb, T., Holyoak, K. J., and Lu, H. Emergent analogical reasoning in large language models, 2023. URL https: //arxiv.org/abs/2212.09196 . - Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., and Cambria, E. Are large language models really good logical reasoners? a comprehensive evaluation and beyond, 2024a. URL https://arxiv.org/abs/2306.09841 . - Xu, Y., Li, W., Vaezipoor, P., Sanner, S., and Khalil, E. B. Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations, 2024b. URL https://arxiv.org/ abs/2305.18354 . - Yang, Z., Dong, L., Du, X., Cheng, H., Cambria, E., Liu, X., Gao, J., and Wei, F. Language models as inductive reasoners, 2024a. URL https://arxiv.org/abs/ 2212.10923 . - Yang, Z., Du, X., Li, J., Zheng, J., Poria, S., and Cambria, E. Large language models for automated open-domain scientific hypotheses discovery, 2024b. URL https: //arxiv.org/abs/2309.02726 . - Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. ICLR , 2023. - Zhang, X., Xie, Y., Huang, J., Ma, J., Pan, Z., Liu, Q., Xiong, Z., Ergen, T., Shim, D., Lee, H., and Mei, Q. Massw: A new dataset and benchmark tasks for ai-assisted scientific workflows, 2024. URL https://arxiv.org/abs/ 2406.06357 . - Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36: 46595-46623, 2023. - Zhou, Y., Liu, H., Srivastava, T., Mei, H., and Tan, C. Hypothesis generation with large language models. arXiv preprint arXiv:2404.04326 , 2024. ## A. Algorithm and theory ## A.1. Detailed algorithm for POPPER ## Algorithm 1 Sequential Falsification with HypothesisAgent Initialize Experiment Design Agent A design, Relevance Checker A rel, Experiment Execution Agent A exec, Summarizer S , Coding Agent Implementation I , Implication Strength Threshold τ , Alpha Threshold α , Max Number of Tests N tests max , Max Retries N retries max , Aggregation Method A Input: main hypothesis H , dataset D ``` Input: main hypothesis H, dataset D Initialize Experiment Design Agent A_design, Relel Coding Agent Implementation I, Implication S Max Retries N_retries , Aggregation Method A F_success <- 0, F_failed <- 0, O <- 0 done <- false for i = 1 to N_tests max do T <- A_design(H, D, F_success, F_failed) if A_rel(T)< - t then F_failed <- F_failed \{T} else success <- false, obs_i <- None for j = 1 to N_retries do (success,obs_) <- A_exec(T, D, I) if success then exit inner loop end if end for end for if not success then F_failed <- F_failed \{T} else F_success <- F_success \{T} O <- O \{obs_i} if A(O)> 1 then done <- true end if end if end if end if if done then exit outer loop end if end return S(H, e1,...,i, ,a, F_success,F_failed) A.2. Proof of Theorem 4 ``` ## A.2. Proof of Theorem 4 Proof of Theorem 4. Throughout, we condition on the training process of the LLM agents. Under Assumptions 1 and 2, each e-value also obeys E [ e i | D i -1 ] ≤ 1 under H 0 since H 0 implies h 0 i for each i ≥ 1 . Define E i = ∏ i s =1 e s as the aggregated evidence at each iteration i ≥ 1 , and E 0 = 1 . Also, recall that F i = σ ( D i ) is the filtration in Assumption 3. Then, we have $$\mathbb { E } [ E _ { i } \, | \, \mathcal { F } _ { i - 1 } ] = E _ { i - 1 } \cdot \mathbb { E } [ e _ { i } \, | \, \mathcal { F } _ { i - 1 } ] \leq E _ { i - 1 } ,$$ where we use the takeout property and the fact that E i -1 is measurable with respect to F i -1 . In addition, it is clear that E i is measurable with respect to F i . Therefore, { E i } i ≥ 1 is a non-negative super-martingale adapted to the filteration {F i } i ≥ 1 . Applying Doob's optional stopping theorem, we know that for any stopping time τ adapted to the filteration {F i } i ≥ 1 , E := E τ obeys E [ E ] ≤ E 0 = 1 under H 0 . Finally, by Markov's inequality, we know that P (ˆ y = 1) = P ( E ≥ 1 /α ) ≤ α · E [ E ] ≤ α under H 0 , thus completing the proof of Theorem 4. ## B. Full related works Philosophy of science The philosophical foundations of hypothesis validation are rooted in debates about the nature of scientific inquiry. Central to our framework is Karl Popper's falsificationism (Popper, 1959), which argues that scientific hypotheses cannot be definitively proven but can only be refuted through empirical tests. While Popper emphasized iterative falsification, critiques such as those synthesized in Agassi (Agassi, 2014) highlight tensions between his ideas and those of contemporaries like Thomas Kuhn. Kuhn's paradigm shifts (Kuhn, 1962) challenged falsificationism by emphasizing the sociotechnical embeddedness of scientific progress, a perspective further refined by Lakatos' methodology of scientific research programmes(Lakatos, 1978). Lakatos' framework, which evaluates hypotheses within evolving theoretical systems, aligns with our treatment of auxiliary assumptions (e.g., dataset relevance) as prerequisites for testing, as discussed in (Philosophy Institute, 2023). Modern critiques, such as Rubin (Rubin, 2025), argue that Lakatos' approach mitigates challenges like the replication crisis by emphasizing progressive problem shifts over strict falsification. Similarly, van Fraassen's constructive empiricism (van Fraassen, 1980), which prioritizes empirical adequacy over ontological truth, mirrors our focus on observable implications rather than abstract claims. Goodman's 'grue' paradox (Goodman, 1983), which interrogates inductive reasoning, underscores the epistemic risks inherent in generalizing from data-risks our framework pragmatically addresses through statistical safeguards like e-values. Maxwell (Maxwell, 2012) positions aim-oriented empiricism as a synthesis of Popperian, Kuhnian, and Lakatosian ideas, advocating for explicit epistemic aims in scientific practice. This resonates with our adaptive sequential testing paradigm, which balances empirical rigor with iterative refinement. While our framework abstracts sociotechnical dimensions noted in Kuhn and Lakatos, the need for transparency in automated systems echoes their emphasis on communal validation (Press, 2009). By integrating these perspectives, POPPER bridges classical philosophy of science and modern data-driven inquiry, offering a scalable yet philosophically grounded approach to hypothesis validation. LLMfor hypothesis generation. Many methods have used LLM to generate novel research ideas. For example, Wang et al. (2024a),Baek et al. (2024), and Yang et al. (2024b) propose methods for generating creative, domain-specific research ideas. Si et al. (2024) conducted large-scale human studies comparing AI-generated research ideas with those from experts. Moving beyond ideas, many also explore hypothesis generation with LLMs with a focus in the commonsense domains (Gendron et al., 2024; Yang et al., 2024a; Moskvichev et al., 2023; Mirchandani et al., 2023; Tang et al., 2023; Xu et al., 2024a; Han et al., 2023; Xu et al., 2024b; Alet et al., 2021; Webb et al., 2023). Notably, Honovich et al. (2023) explores LLMs' capabilities in inducing rules from example demonstrations. Qiu et al. (2024) and Wang et al. (2024c) further extends this idea to generating and iteratively refining candidate hypotheses from a set of examples or observations. (Majumder et al., 2024) grounds hypothesis generation with a given dataset and a question. However, these works focus on hypothesis generation rather than rigorous validation. POPPER is complementary to this line of research as it takes in a hypothesis (generated from either LLM or human) and develops a systematic, data-driven process for evaluating whether a hypothesis withstands statistical scrutiny. LLM for hypothesis testing and experiments. To the best of our knowledge, there is no work that investigates rigorous validation of a free-form hypothesis grounded with data using AI agent. Some studies have tested LLMs' abilities to implement experiments as a form of validation. For example, Tian et al. (2024) and Gu et al. (2024) evaluate LLMs' coding capabilities in executing experimental protocols. While these works focus narrowly on code generation, POPPER presents a framework for validating natural language-based free-form hypothesis. Additionally, prior research into automated scientific discovery has explored combining hypothesis and code generation for end-to-end workflows (Li et al., 2024b; Lu et al., 2024; Ifargan et al., 2024; Majumder et al., 2024). While these studies focus on automation, they often lack rigorous statistical grounding. In contrast, POPPER focuses on the hypothesis testing component and incorporates robust Type-I error control, ensuring the reliability and scientific rigor of its results. (Li et al., 2024a) (CriticAL) used LLMs to identify and evaluate discrepancies between model predictions and data through hypothesis testing. While CriticAL focuses on validating statistical predefined models, POPPER tackles the challenge of validating free-form natural language hypotheses with a sequential falsification framework. LLM for automating research. LLMs have also been used for several other research-related tasks, including automated review generation (D'Arcy et al., 2024; Liang et al., 2023), related work curation (Ajith et al., 2024; Press et al., 2024), experiment outcome predictions (Manning et al., 2024; Zhang et al., 2024; Lehr et al., 2024), and future work recommendations (Zhang et al., 2024). While these are interesting applications, our work focuses on hypothesis testing. ## C. Limitations Type-I error v.s. false discoveries. We view hypothesis validation with POPPER as an initial step towards rigorous automatic scientific discovery. One limitation of our current framework arises from the limitation of Type-I error as an error criterion for scientific discovery. Let us denote rejecting the null as a 'discovery', and it is a true discovery if the alternative holds. The shortcoming of Type-I error control is that it does not necessarily imply the discoveries are true (which is more pronounced when POPPER is used to validate many hypotheses). Awareness of this issue emerged much later than the appearance of concept of Type-I error, but has been quite important nowadays in the fields of hypothesis testing, selective inference, and replicability (Ioannidis, 2005; Collaboration, 2015; Benjamini, 2020). To see this point, consider an extreme case where all hypotheses being passed on to POPPER are null ones. Then, the Type-I error control only implies that we reject each hypothesis with no greater than a chance of α , but every discovery, once made, must be false. Therefore, we stress that one should be cautious in interprating the validated hypotheses by POPPER as true discoveries to act upon. In the following, we discuss possible extended uses of POPPER for more advanced error critria on false discoveries. Consider using POPPER to validate M ≥ 1 (abstract) hypotheses. The family-wise-error-rate (FWER) is the probability of making any false discovery; FWER control at level α ∈ (0 , 1) can be achieved by Bonferroni's correction, i.e., running POPPER for each hypothesis at level α/M and gather all rejected null hypothesis. A more liberal criterion is the false discovery rate (FDR) (Benjamini & Hochberg, 1995), which is the average fraction of false discoveries among all discoveries. The FDR is suitable for measuring the wastage of follow-up resources on validated (rejected) hypotheses. Since our framework produces a valid e-value E τ for each hypothesis, these e-values can be readily used to derive a set of validated hypotheses with FDR control by employing the eBH procedure (Wang & Ramdas, 2022). However, these use cases are beyond the scope of this work, and we leave the evaluation and further developments of such capabilities of POPPER for future investigation. ## D. Error analysis In this section, we provide insights into the common failure modes of POPPER. We first manually inspected 20 randomly sampled failed experiment logs produced by POPPER, and created a list of 10 possible failure categories based on the model's behaviors. Table 5 provides detailed definitions of the 10 failure categories. Then, we collected a total of 128 failed experiment logs from benchmark runs across TargetVal-IFNG , TargetVal-IL2 , and DiscoveryBench . We then query a reasoning LLM (OpenAI O1) with the failed trajectory logs, the agent's incorrect conclusion, and the ground truth conclusion to automatically categorize each failed experiment into one or more failure modes described in Table 5. We manually checked 30 labeled experiment logs for quality assurance. 93.3% of O1's labels aligned with human judgment. According to Figure 5, 35.9% of the failures accompany the agent misinterpreting the context for p-values. 28.1% and 17.2% of the errors occur when the agent fails to find effective falsification tests or uses tests that breaks implication. 8.6% and 7.0% of the errors are caused by incorrect test implementation and failure to locate relevant data. It is worth noting that we only observed 1 instance of hallucination across 128 failure cases, and no signs of p-hacking were observed. Figure 5: Failure mode distribution for POPPER, labaled automatically by O1 and manually checked by humans. <details> <summary>Image 5 Details</summary>  ### Visual Description ## Pie Chart: Failure Modes ### Overview The pie chart illustrates the distribution of failure modes in a testing process. It is divided into six segments, each representing a different type of failure mode. ### Components/Axes - **Title**: Failure Modes - **Legend**: - Falsification Test Breaks Implication (17.2%) - Ineffective Test Selection (28.1%) - Misinterpreted P-Value (35.9%) - Failed to Locate Relevant Data (8.6%) - Hallucination (7.0%) - Failed to Recover from Test Errors (1.0%) - **Axes**: - No visible axes in the image. ### Detailed Analysis or ### Content Details The chart shows that the majority of failures (35.9%) are due to misinterpreting the P-value. This is followed by ineffective test selection (28.1%), and falsification test breaks implication (17.2%). The remaining failure modes are relatively small in proportion, with failed to locate relevant data (8.6%), hallucination (7.0%), and failed to recover from test errors (1.0%). ### Key Observations The chart highlights that misinterpreting the P-value is the most common failure mode, suggesting that there may be a need for better training or clearer guidelines for interpreting statistical results in testing processes. The ineffective test selection and falsification test breaks implication are also significant, indicating that there may be issues with the testing methodology or the selection of tests. ### Interpretation The data suggests that there are significant issues with the testing process, particularly in terms of misinterpreting statistical results and ineffective test selection. These issues may be contributing to the high rate of failures in the testing process. The need for better training and clearer guidelines for interpreting statistical results in testing processes is emphasized. Additionally, the need for better testing methodology and the selection of tests is highlighted. </details> Table 5: Definitions of failure mode categories | Failure Type | Definition | |---------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Falsification Test Breaks Implication | The agent selects falsification tests that are not logically implied by the main hypothesis. This occurs when the falsification sub-hypothesis could be true even if the main hypothesis is false, leading to irrelevant p-values and misleading results. | | Ineffective Test Selection | The agent fails to identify or design falsification tests that are capable of effectively addressing the main hypothesis, resulting in weak or inconclusive evidence. | | Malformed Falsification Test | The design of the falsification test is flawed. For example, the test assesses an alternative sub-hypothesis that contradict the main hypothesis, or lack a clear framework for accepting or rejecting the null sub-hypothesis. | | Incorrect Test Implementation | The agent incorrectly implements the falsification test. While the test appears to execute successfully, it contains undetected bugs or methodological errors that result in invalid or misleading p-values and conclusions. | | P-Hacking | The agent manipulate data analysis, experimental procedures, or selectively report results to artificially achieve statistically significant p-values, leading to misleading conclusion. | | Misinterpreted P-Value | The agent misinterprets or overlooks important context when analyzing p-values. This includes failing to recognize invalid p-values, ignoring assumptions of the statistical test, or drawing incorrect conclusions from the results. | | Hallucination | The agent generates data entries, data interpretations, assumptions, observations, p-values, or conclusions that are fabricated or not grounded in the provided data or context. | | Failed to Recover from Test Errors | The agent encounters errors during test execution and fails to recover or adapt. This may result in the agent repeating the same errors or becoming stuck in an unproductive loop of failed tests. | | Failed to Locate Relevant Data | The agent is unable to identify, retrieve, or preprocess the necessary data required for conducting critical falsification tests, preventing effective hypothesis evaluation. | | Other | There was some other problem that prevented the agent from arriving at the correct conclusion. | ## E. Tests and trajectory analysis In this section, we detail how we categorized the statistical and domain-specific tests performed by POPPER during falsification experiments, as well as how we summarized the agent's trajectories for executing each falsification test, as visualized in Figure 3. We parsed and sampled 1500 falsification test designs and their execution logs, and then asked GPT-4o to identify and group the statistical tests performed in the falsification experiments. We limit our analysis of domain-specific tests to biological hypotheses only, as we have an abundance of biological hypotheses from TargetVal benchmark. The other five domains provided by DiscoveryBench contains limited number of unique hypotheses per domain, and the analysis does not converge. We sampled 462 falsification tests proposed by the experiment design agent and used GPT-4o to extract and group them into standardized biological tests. For agent trajectories, we first manually inspected the behaviors of the experiment execution agent over 20 experiments and summarized a list of 11 possible high-level actions taken by the agent. Detailed definitions of these actions are listed in Table 6. We then randomly sampled 80 trajectories of the experiment execution agent, and prompted GPT-4o to convert each trajectory into a list of high-level actions as detailed in Table 6. We observe that the agent's workflow closely mirrors that of a human data analyst. It begins by inspecting the dataset and assembling relevant information, then proceeds with a cycle of test implementation, execution, and iterative error resolution. Upon observing the test results, the agent may optionally check validity criteria (e.g., model assumptions and sample sizes) and refine its approach if necessary. Finally, the agent compiles all findings into a summary to draw a final conclusion. Table 6: Names and definitions of actions taken by the experiment execution agent. | Action Name | Definition | |----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Inspect Dataset | Actions where the agent checks or explores the structure/content of the dataset (e.g., looking at dimensions, columns, and sample rows). | | Visualize Data | Actions where the agent creates visualizations to explore the distribution and relationships within the data. | | Retrieve Data | Actions where the agent extracts specific portions of the dataset relevant to the current hypothesis or analysis. | | Prepare Data | Actions where the agent cleans, transforms, and structures data (e.g., grouping, calculating summary statistics, handling missing values) before applying tests or models. | | Fit Model | Actions where the agent employs a statistical or machine-learning model to test or explore relationships in the data. | | Implement Test | Actions where the agent applies a formal statistical test (e.g., correlation test, t-test, ANOVA) or other relevant procedure to evaluate a hypothesis. | | Fix Errors | Actions where the agent identifies and corrects issues or bugs in the testing procedure (e.g., coding errors, incorrect data handling, syntax problems). | | Inspect Test | Actions where the agent verifies the results of a test-checking the shape of data arrays, the number of observations, and ensuring that the calculations (e.g., p-values, effect sizes) are valid. | | Analyze Results | TActions where the agent interprets the output of a test or model (e.g., evaluating coefficients, p-values, confidence intervals) to determine whether the data supports or refutes the hypothesis. | | Summarize Conclusion | Actions where the agent provides a final statement or verdict about the hypothesis. | | Other | Any agent actions that are not covered by the ones above. | ## F. Human study details We recruited 11 computational biologists and bioinformaticians (PhD holders or candidates) for our human study, and 9 adhered. Each participant was asked to complete a short questionnaire on their educational background and relevant experience (Listing 1). We present the background distributions of recruited participants in Figure 6. Of the 9 participants, 6 hold (or are pursuing) a PhD, 1 holds a Master's degree, and 2 are postdoctoral researchers. In terms of experience with data analysis and coding for genetic and genomic data, 2 participants identified as beginners, 1 as intermediate, and 6 as experts. Regarding familiarity with statistical hypothesis testing, 2 participants identified as beginners, 2 as intermediate, and 5 as experts. Finally, 6 participants reported that they have never performed wet-lab experiments, while 3 indicated having done so. We sampled a total of 18 tasks from the TargetVal-IL2 benchmark to evaluate the Type-I error (9 tasks) and statistical power (9 tasks) of our method. Each participant was randomly assigned two tasks to complete. To prevent inference of one hypothesis from the other, a participant might receive two positive, two null, or one positive and one null hypothesis. Participants were free to use the internet or large language models for general coding questions (e.g., library usage, syntax) and statistical tests, but not to query the specific biological hypothesis directly. All conclusions were to be derived solely from the data provided in the TargetVal-IL2 benchmark, with each hypothesis tested at significance level α = 0 . 1 . All work was documented in Jupyter Notebooks. ``` ``` Listing 1: Background questionnaire used for human study recruitment ## G. Human Annotation Details Werandomly sampled 90 falsification test proposals from the three benchmarks. Each of the three annotators first individually annotated a common set of 20 proposals using the same 0.1-1.0 rubric as the Relevance Checker 4. The annotators then discuss and calibrate their decisions and independently annotate 10 more proposals after the calibration. The annotators achieved a Kendall's W of 0.62 before the calibration, and 0.91 post calibration. Finally, each annotator individually annotate a separate set of 20 falsification proposals. The human annotators and the relevance checker agent achieved a Kendall's Tau of 0.43 ( p = 1 e -06 ) and Spearman's correlation of 0.55 ( p = 5 e -6 ). The relevance checker agent ranked 84% of the proposed falsification tests as 'Strongly Relevant' (score > = 0.8), whereas human annotators ranked 77% of the test proposals as 'Strongly Relevant'. ## H. Qualitative Analysis This section provides qualitative analysis on one successful falsification trajectory and one failure case trajectory on the TargetVal-IL2 benchmark. Figure 10 presents an example trajectory of POPPER running on a TargetVal-IL2 hypothesis. We can see the agent attempted multiple rounds of diverse falsification experiments, including expression correlation analysis, LCP2 regulatory network analysis, LCP2 variant-immune phenotype association test, and LCP2 eQTL-IL2 regulatory region test. POPPER performs sequential error control to rigorously aggregate the evidence from all four experiments, and then rejects the main null hypothesis as the summarized sequential statistics (i.e., cumulated e-values) passes our alpha-threshold of 0.1. We observe that the experiment design agent autonomously refines its proposal to enhance the implication strength and feasibility of the proposed falsification experiment. The experiment execution agent iteratively inspects and interacts with multiple data sources to evaluate the feasibility of the experiment, before implementing and conducting the statistical tests. The experiment execution agent also shows attempts to account for model assumptions and inspect the validity of test statistics before arriving at a final conclusion (e.g., Round 3). We note that with rigorous Type I error control, POPPER also provides more tolerance and leniency for test execution failures. Notice that in Round 1, the experiment execution agent incorrectly concluded that LCP2 and IL2 are not present in the datasets. However, benefiting from the sequential <details> <summary>Image 6 Details</summary>  ### Visual Description ## Bar Chart: Experience with Data Analysis/Code on Genetic & Genomic Data ### Overview The bar chart displays the experience levels of individuals in data analysis and writing code on genetic and genomic data. The chart is divided into four categories: Beginner, Intermediate, and Expert. Each category is represented by a different color. ### Components/Axes - **X-axis**: Represents the experience level, with categories ranging from Beginner to Expert. - **Y-axis**: Represents the count of individuals in each experience level category. - **Legend**: Located at the top right corner, it indicates the color coding for each experience level category. ### Detailed Analysis or ### Content Details - **Genomics Exp**: The majority of individuals (7) have an Expert level of experience in data analysis and writing code on genetic and genomic data. - **Hypothesis Testing**: The majority of individuals (7) have an Intermediate level of experience in statistical hypothesis testing. - **Education**: The majority of individuals (7) have a Master level of education. - **Wet Lab Exp**: The majority of individuals (7) have an Intermediate level of experience in performing wet-lab experiments in a biology lab. ### Key Observations - The majority of individuals have an Intermediate or Expert level of experience in data analysis and writing code on genetic and genomic data. - The majority of individuals have a Master level of education. - The majority of individuals have an Intermediate level of experience in performing wet-lab experiments in a biology lab. ### Interpretation The data suggests that individuals with an Intermediate or Expert level of experience in data analysis and writing code on genetic and genomic data are the most common. Additionally, individuals with a Master level of education are the most common. The majority of individuals have an Intermediate level of experience in performing wet-lab experiments in a biology lab. This data could be useful for identifying areas where additional training or resources may be needed. </details> Count Figure 6: Backgrounds of human study participants. falsification frameowkr, POPPER is eventually able to reach the correct conclusion after multiple experiment trials. Figure 11 shows an example false positive trajectory on TargetVal-IL2 . The critical error lies in Round 3, where the experiment design agent proposes to test whether 'genetic variants near RAB39A are significant QTLs for IL-2 related immune phenotypes,' but the experiment execution agent only looked at eQTLs for RAB39A expression in neutrophils, a cell type that may or may not produce IL-2. The agent then converts the eQTL score to a highly significant p-value for 'RAB39A expression in neutrophils', but it does not imply that RAB39A regulates IL-2. Hence, while the proposed falsification experiment is valid, the implementation of the test violates the implication assumption. We categorize this failure case as 'Incorrect Test Implementation', and 'Misinterpreted P-Value'. Overall, we found that understanding and reasoning about the context and validity of effect sizes and p-values remains to be a main challenge for POPPER. ## I. Prompting Details Listings 2, 3, 4, 5, and 6 detail the prompts used for different modules of POPPER. Once you open the notebook, run the following cell to start the time clock ``` In [1]: import time start_time = time.time() ``` ## Instructions Given a biology hypothesis "Gene MAK16 regulates the production of Interleukin-2 (IL-2)." , your task is to validate it using the given raw databases by performing relevant data analysis, formulating statistical tests, and implementing them. The validation should be purely datadriven, not literature-driven. For statistical test, use significance level of alpha=0.1. Output (1) If the hypothesis is valid or not given the data (2) relevant statistics (e.g. p-value, etc) ## IMPORTANT - You must only use the database folder in the current task folder to perform the analysis. DO NOT use the data from the other task or any external data. - You must NOT use LLMs or internet about the direct answer to the biological hypothesis. - You can use internet/LLMs if you are not sure about the code syntax or library usage or statistical tests or have biological questions in general. - You can use any python library to perform the analysis. - The tasks are randomly sampled and may be one true & one false / all true / all false Here are the list of available data sources with columns and example rows: ``` Here are the list of available data sources with columns and example rows: df_gtex_tissue_gene_tpm: {'Description': 'ENSG00000186092', 'Tissue': 'Adipose - Subcutaneous', 'Expression': 0.0453961, 'Gene': 'OR4F5'} df_gene_info: {'gene_id': 'ENSG00000228037', 'transcript_id': 'ENST00000424215', 'chr': '1', 'gene_start': 2581560, 'gene_end': 2584533, 'strand': 1, 'transcript_start': 2581560, 'transcript_end': 2584533, 'tss': 2581560, 'transcript_is_canonical': 1.0, 'gene_name': nan, 'percentage_gene_gc_content': 51.11, 'gene_type': 'lncRNA'} df_genetic_interaction: {'interaction_id': 206363, 'gene_a_id': 'YCR011C', 'gene_b_id': 'YCL025C', 'experimental_system_type': 'genetic', 'pubmed_id': 'PUBMED:16269340', 'organism_id_a': 559292, 'organism_id_b': 559292, 'throughput_type': 'High Throughput', 'experimental_score': '-5.6431'} # some dataframes are omitted for presentation purposes In [3]: ## loading the datasets import pandas as pd ``` ``` In [3]: ## loading the datasets import pandas as pd import glob database = {} ``` Figure 7: Example human study interface (1/2). ``` time ``` ``` ``` ``` for path in glob.glob('./database/*.pkl'): database['df_' + path.split('/')[-1].split('.pkl')[0]] = pd.read_pickle(path) ``` ## Record all your analysis here Once you finished the analysis, and reached the conclusion, please write the conclusion in the following cell and end the time clock by running the cell below. Is the hypothesis valid? (Yes/No) ## Your answer What is the statistics that support your conclusion? ## Your answer ``` In [ ]: end_time = time.time() print("Execution time: ", end_time - start_time) ``` Figure 8: Example human study interface (2/2). ``` You are an expert statistician specialized in the field of {domain}. You are tasked to validate rigorously if a {domain} hypothesis H is true by implementing an falsification test proposed by the user. You should write code to implement the falsification test. The test should be relevant to the main hypothesis and aims to falsify it. The test should use the available data described below, and use data processing, extraction, and perform statistical analysis to produce a p-value measuring the falsification of the main hypothesis. The test should be extremely rigorous. The p-value should be theoretically grounded. The code should be clear, concise, and efficient. Do progress bar when necessary. It will have a time limit, so please be efficient. For example, if possible, you can set the number of permutations to be small (e.g. <1000). The code should be self-contained, and do not need additional modifications from user. You have access to the following pandas dataframe tables, where each table, it shows the precise column names and a preview of column values: {{context}} Each of these dataframes have already been loaded into the global namespace. You may access each dataframe **directly as variables**. Make sure to use the **EXACT** dataframe names as shown above. Create a code from the user request. Ensure any code you provide can be executed with all required imports and variables defined. Structure your answer: 1) a prefix describing the code solution, 2) the imports, 3) the functioning code block. Invoke the code tool to structure the output correctly. NEVER PRODUCE ANY PLACEHOLDER IN ANY FUNCTION. PLACEHOLDER IS WORSE THAN FAILURE TO PRODUCE CODE. PLACEHOLDER including coming up with placeholder genes, names, ids, functions, p-value, or any other placeholder. The output should be a single p-value. If there are multiple p-values produced by the test , you should aggregate them in a meaningful and rigorous way. When printing p-values, please use scientific notations (e.g. 3.50e-03) instead of the raw number. ----------------------------------------------------------------------------- Here is the user requested falsification test specification: ``` Here is the user requested falsification test specification: Listing 2: System Prompt For Coding Agent - Given a {domain} hypothesis "{main\_hypothesis}", your goal is to propose a novel falsification test given the available {domain} data sources. A falsification test is a test that can potentially falsify the main hypothesis. The outcome of the falsification test is to return a p-value that measures the evidence to falsify the main hypothesis. Notably, the falsification test should satisfy the following property: if the main hypotheiss is null, then the falsification sub-hypothesis should also be null. Here are the list of available data sources, and you can directly call the dataframe as it has already been loaded; no need to load from file path. Each is a pandas dataframe with columns and example rows: {data} For the final test, return (1) Name: name of the test (2) Test description: be clear and concise. Describe the falsification outcomes. (3) Null sub-hypothesis h\_0: what is the statistical null sub-hypothesis does this falsification test aim to test? (4) Alternate sub-hypothesis h\_1: what is the statistical alternative sub-hypothesis does this falsification test aim to test? Here are the falsification tests that you've created in the previous rounds and their corresponding test results: """ {existing\_falsification\_test} """ You may use these information to formulate your next subhypothesis and falsification test, but make sure the proposed falsification test is non-redundant with any of the existing tests. The proposed test should also avoid these failed falsification tests in the previous rounds: """ {failed\_falsification\_test} """ A good falsification test should serve as a strong evidence for the main hypothesis. However, make sure it is answerable with the given available data sources. You should aim to maximize the implication strength of the proposed falsification test using the relevant parts of the provided data. ----First produce an initial falsification test proposal. Then, in each round i, you will do the following: (1) critic: ask if the main hypothesis is null, is this test also null? be rigorous. this is super important, otherwise, the test is invalid. Is it redundant on capabilities with existing tests? Is it overlapping with failed tests? Can this be answered and implemented based on the given data? (2) reflect: how to improve this test definition. If you think the test definition is good enough, return the final test definition to the user. propose - If not, either refine the test definition that is better than the previous one or a new test definition, then go to the next round. Listing 3: System Prompt For Statistical Agent - Given a main hypothesis and a proposed sub-hypothesis test, assess the relevance of this sub-hypothesis test to the main hypothesis. Use the following rubric to guide your response, providing a score from 0.1 to 1.0 and a brief justification for the score. Each score level represents a different degree of relevance based on evidence strength , mechanistic connection, and predictive value of the test results. ## Rubric: - 1.0 -Highly Relevant: The sub-hypothesis provides direct evidence or a clear mechanistic insight that strongly supports or refutes the main hypothesis. The test is specific to variables or mechanisms involved in the main hypothesis, with significant predictive value. - 0.8 -Strongly Relevant: The test addresses a major component of the main hypothesis, providing substantial supporting or refuting evidence, and shows strong mechanistic alignment. The results would significantly impact the confidence in the main hypothesis. - 0.6 -Moderately Relevant: The test examines elements supporting the main hypothesis without direct mechanistic insight. Some aspects align with the main hypothesis, offering moderate predictive value. - 0.4 -Slightly Relevant: The test is related to the main hypothesis but provides limited direct evidence. It explores loosely associated variables and has minimal predictive value. - 0.2 -Barely Relevant: The test is tangentially related, providing minimal information that could impact the main hypothesis, with no clear mechanistic link and negligible predictive value. - 0.1 -Irrelevant: The sub-hypothesis does not provide relevant evidence or mechanistic connection to the main hypothesis, with no predictive value. ## Instructions: 1. Read the main hypothesis and the sub-hypothesis test carefully. 2. Choose the relevance score from the rubric that best matches the relationship. 3. Explain your reasoning for selecting this score, referring to evidence strength, mechanistic connection, and predictive value of the sub-hypothesis test results. Listing 4: Relevance Checker System Prompt You are a helpful assistant trained to help scientists summarize their experiment observations. - You have observed a sequential falsification test procedure of a scientific hypothesis and your goal is to accurately summarize and extract insights to present to a human scientist. - For the observed list of falsification tests, each test includes the test description and its test results. The final output should state the following: - (1) The main scientific hypothesis under study - (2) The result of the sequential falsification test - (3) Reasoning, summarizing, and analyzing these results - (4) Your conclusion on whether or not this hypothesis is true or false; just return True/ False - (5) Rationale of the conclusion Remember, your MUST STRICTLY ADHERE to the experiment observations WITHOUT your personal bias or interpretations. For example, if the experiments fail to reject the null hypothesis, you MUST output the conclusion as False EVEN IF YOU BELIEVE THE STATEMENT IS TRUE. Listing 5: Summarizer System Prompt ``` <loc_65><loc_64><loc_430><loc_83>Given a scientific hypothesis H, you have designed a sub-hypothesis test h to falsify the main hypothesis. You have also collected evidence from data for the null hypothesis ( h _0) and the alternative hypothesis (h _1). ``` Listing 6: E-Value Estimation Agent System Prompt Figure 9: Example sequential falsification trajectory on the TargetVal-IL2 Benchmark with o1 as the LLM Backbone. Note that for o1, the experimental execution agent can only do code generation instead of ReAct due to the rate limit per minute. This is a correct hypothesis. ``` This is a correct hypothesis. Problem Statement Main Hypothesis: Gene GRAP2 regulates the production of Interleukin-2 (IL-2). Data sources: df_gtex_tissue_gene_tpm: {'Description': 'ENSG00000186092', 'Tissue': 'Adipose - Subcutaneous', 'Expression': 0.0453961, 'Gene': 'OR4F5'} df_gene_info: {'gene_id': 'ENSG00000228037', 'transcript_id': 'ENST00000424215', 'chr': '1', 'gene_start': 2581560, 'gene_end': 2584533, 'strand': 1, 'transcript_start': 2581560, 'transcript_end': 2584533, 'tss': 2581560, 'transcript_is_canonical': 1.0, 'gene_name': nan, 'percentage_gene_gc_content': 51.11, 'gene_type': 'lncRNA'} df_genetic_interaction: {'interaction_id': 206363, 'gene_a_id': 'YCR011C', 'gene_b_id': 'YCL025C', 'experimental_system_type': 'genetic', 'pubmed_id': 'PUBMED:16269340', 'organism_id_a': 559292, 'organism_id_b': 559292, 'throughput_type': 'High Throughput', 'experimental_score': '-5.6431'} # 19 more datasets omitted for presentation ``` - Problem Statement # 19 more datasets omitted for presentation Experiment Design Agent Round 1 ``` Experiment Design Agent Round 1 INITIAL PROPOSAL ----------------------------------------------------------------------------- (1) Name of the Test: "Tissue-Wide Expression Correlation Test between GRAP2 and IL-2" (2) Test Description: Using the GTEx expression data (df_gtex_tissue_gene_tpm), compute the correlation between GRAP2 and IL-2 transcript levels across all tissues for which both genes have measured TPM. If GRAP2 truly regulates IL-2, we would expect a consistent positive or negative correlation across multiple tissues, reflecting regulatory coupling. A standard correlation test (e.g., Pearson's or Spearman's) can be performed for each tissue, and these can then be combined or meta-analyzed to return an overall p-value testing whether the correlation significantly differs from zero. (3) Null Sub-Hypothesis h_0: "There is no systematic correlation between GRAP2 and IL-2 expression across tissues." If the main hypothesis (that GRAP2 regulates IL-2) is null (i.e., GRAP2 does not regulate IL-2), then we would not expect a meaningful correlation of their transcript levels across tissues. (4) Alternative Sub-Hypothesis h_1: "There is a significant correlation between GRAP2 and IL-2 expression across tissues." If GRAP2 truly regulates IL-2, we would expect a consistent correlation (positive or negative) in expression across multiple tissues. ``` ``` Automated Hypothesis Validation with Agentic Sequential Falsifications ----------------------------------------------------------------------------- ROUND 1 ----------------------------------------------------------------------------- (1) CRITIC: If the main hypothesis "Gene GRAP2 regulates the production of IL-2" is null (i. e., false), then indeed we would expect no biology-driven correlation of expression between GRAP2 and IL-2. Hence, under the main null, the sub- ``` ## Relevance Checker ``` Relevance Checker: Proposed falsification test passes relevance check: Proposal: Main hypothesis: Gene GRAP2 regulates the production of Interleukin-2 (IL-2). -1. Falsification Test name: Tissue-Wide Expression Correlation Test between GRAP2 and IL-2 IL-2 Falsification Test description: Using df_tigex_tissue_gene_tpm, extract the TPM measurements of GRAP2 and IL-2 across all available tissues. Compute the correlation in each tissue and/or perform an overall meta-correlation analysis across tissues. A significant nonzero correlation would suggest regulatory coupling consistent with GRAP2 modulating IL-2. A lack of correlation would serve as evidence to falsify the main hypothesis. Falsification Test Null sub-hypothesis: No correlation exists between GRAP2 and IL-2 across tissues. -1. Falsification Test Alternate sub-hypothesis: A significant correlation exists between GRAP2 and IL-2 across tissues. Relevance score 0.8 Reasoning: The tissue-wide correlation test directly examines whether GRAP2 and IL-2 expressions align across multiple tissues. While correlation is not definitive ``` proof of causation, a strong and consistent correlation would be substantial evidence supporting the hypothesis that GRAP2 influences IL-2 production. Thus, the test strongly addresses a key element of the main hypothesis, though it does not alone provide a complete mechanistic explanation. ## Experiment Execution Agent ``` Experiment Execution Agent import numpy as np import pandas as pd import scipy.stats as stats from tqdm import tqdm # -- Step 1: Filter for GRAP2 and IL2 in the GTEx TPM data subdf_grap2 = df_gtex_tissue_gene_tpm[df_gtex_tissue_gene_tpm['Gene'] == 'GRAP2'] subdf_il2 = df_gtex_tissue_gene_tpm[df_gtex_tissue_gene_tpm['Gene'] == 'IL2'] # -- Step 2: Merge on Tissue merged = pd.merge(subdf_grap2[['Tissue','Expression']], subdf_il2[['Tissue','Expression']], on='Tissue', how='inner', suffixes=('_grap2','_il2')) # Check if we have sufficient data if len(merged) < 3: print("Not enough tissues with both GRAP2 and IL2 expression data. Unable to perform test.") else: # -- Step 3: Compute the observed correlation observed_r, _ = stats.pearsonr(merged['Expression_grap2'], merged['Expression_il2']) observed_r_abs = abs(observed_r) # -- Step 4: Permutation Test n_perm = 999 # user suggested limit count_extreme = 0 exp_il2_original = merged['Expression_il2'].values for _ in tqdm(range(n_perm), desc='Permutation test'): perm_il2 = np.random.permutation(exp_il2_original) r_perm, _ = stats.pearsonr(merged['Expression_grap2'], perm_il2) if abs(r_perm) >= observed_r_abs: count_extreme += 1 # -- Calculate two-sided p-value p_value = (count_extreme + 1) / (n_perm + 1) # -- Print in scientific notation print(f"Permutation-based correlation p-value: {p_value:.3e}") ``` ## Observation ``` Process completed within the time limit. Permutation test: 100% Permutation-based correlation p-value: 3.100e-02 ``` ## Sequential Error Control ``` ================================= Serial Testing ============================ List of p-values: [0.031] Summarized sequential statistics: 2.8398091712353244 Sequential test result: insufficient evidence - CONTINUE ============= Testing decision is insufficient evidence - CONTINUE =========== ``` ## Experiment Design Agent ``` ``` ``` Automated Hypothesis Validation with Agentic Sequential Falsifications FINAL TEST DEFINITION (1) Name: "Tissue-Specific Expression Contrast Test for GRAP2 in High- vs. Low IL-2 Expressing Tissues" (2) Test Description: Using df_gtex_tissue_gene_tpm, identify tissues exhibiting relatively high IL-2 expression and those with low or negligible IL-2 expression. Split the tissues into two groups accordingly. Then perform a two-sample test (e.g., t-test) comparing GRAP2's mean TPM in the high IL-2 group versus the low IL-2 group. If GRAP2 truly regulates IL-2, we would expect a statistically significant expression difference in tissues that produce more IL-2. A non-significant result (or inverted effect) would serve as evidence to falsify the main hypothesis. ``` ## Relevance Checker ``` Proposed falsification test is not relevant enough to the main hypothesis! Proposal: Main hypothesis: Gene GRAP2 regulates the production of Interleukin-2 (IL-2). Falsification Test name: Tissue-Specific Expression Contrast Test for GRAP2 in High - vs. Low IL-2 Expressing Tissues Falsification Test description: Using df_gtex_tissue_gene_tpm, identify tissues exhibiting relatively high IL-2 expression and those with low or negligible IL-2 expression. Split the tissues into two groups accordingly. Then perform a two-sample test (e.g., t-test) comparing GRAP2's mean TPM in the high IL-2 group versus the low IL-2 group. If GRAP2 truly regulates IL-2, a statistically significant expression difference in tissues that produce more IL-2 would be expected; a non-significant result (or inverted effect) would serve as evidence to falsify the main hypothesis. ``` Falsification Test Null sub-hypothesis: GRAP2 expression does not differ between tissues with high IL-2 expression and tissues with low IL-2 expression. Falsification Test Alternate sub-hypothesis: GRAP2 expression is significantly higher (or lower) in tissues that exhibit high IL-2 expression than in tissues that exhibit low IL-2 expression. ``` that exhibit low IL-2 expression. Relevance score: 0.6 Reasoning: This expression-contrast analysis is a relevant check if up/down regulation of GRAP2 is linked to high or low IL-2 production across tissues, that would be consistent with GRAP2's proposed regulatory role. However, it largely provides an association rather than direct mechanistic evidence, so while it does inform the hypothesis, it is a moderate rather than a strong or definitive test. ``` ## Experiment Design Agent Previous experiment does not pass relevance checker, proposing another experiment. ## Round 2 Initial Falsification Test Proposal: - (1) Name of the test: "Essential Expression Overlap Test" - (2) Test description (concise): Use df\_gtex\_tissue\_gene\_tpm to tabulate, for each tissue, whether GRAP2 is expressed above a small threshold (present vs. absent) and whether IL-2 is expressed above that threshold (present vs. absent). If GRAP2 truly regulates IL-2 production (implying GRAP2 expression is necessary for IL-2 expression), then no tissue should exhibit IL-2 expression in the absence of GRAP2 expression. Statistically, one can form a 2x2 contingency table (presence/absence of GRAP2 versus presence/absence of IL-2) across tissues and apply Fisher's exact test. A "failure" case tissues without GRAP2 expression but with IL-2 expression would tend to falsify the main hypothesis. - (3) Null sub-hypothesis h\_0: "There is no strict dependency of IL-2 expression on GRAP2 expression, i.e., tissues lacking GRAP2 can still show IL-2 expression." - (4) Alternate sub-hypothesis h\_1: "Tissues expression, that lack GRAP2 expression also consistently lack IL-2 consistent with GRAP2 being necessary for IL-2 production." ## Round 1 ## (1) Critic: If the main hypothesis (GRAP2 regulates IL-2) is truly null, then we should see at least some tissues without GRAP2 but with IL-2 expression, violating the necessary ' relationship. Hence under a null main hypothesis, the subhypothesis that n o tissue expresses IL-2 if GRAP2 is absent' would also be null. This aligns with the requirement. Is it redundant or overlapping with the prior correlation tests? This test specifically looks for a necessary-expression pattern (i.e., IL-2 cannot be present when GRAP2 is absent), rather than measuring linear correlation or differences in mean expression. It is not merely a repeat of the correlationbased or high-vs.-low expression tests. Hence it is non-redundant. Can it be answered with the given data? Yes. df\_gtex\_tissue\_gene\_tpm provides GRAP2 and IL-2 tissue expression data. One can define a threshold (e.g., median TPM or a small nonzero TPM) to classify each gene as "expressed" or "absent" in each tissue. - (2) Reflect: The proposal is fairly straightforward, can be implemented with rather df\_gtex\_tissue\_gene\_tpm, and meets the falsification requirement (if in any tissue IL-2 is inexplicably high while GRAP2 is negligible, that falsifies " GRAP2 is necessary for IL-2 production"). This test complements correlation analyses by focusing on a threshold-based "necessary condition" approach than a predictive or trend-based one. ``` Final Test Definition: (1) Name: "Essential Expression Overlap Test" (2) Test description: Classify each tissue in df_gtex_tissue_gene_tpm as having GRAP2 expressed/not expressed and IL-2 expressed/not expressed (based on a small nonzero TPM threshold). Construct a 2 x2 table with these presence/absence calls across tissues. Apply Fisher's exact test to see if IL-2 is indeed "absent" whenever GRAP2 is absent. Failure to observe this co-absence pattern (i.e., finding tissues with no GRAP2 expression but clear IL-2 expression) would be evidence falsifying the main hypothesis. (3) Null sub-hypothesis h_0: "There is no essential overlap pattern; IL-2 may be present even when GRAP2 is not expressed." (4) Alternate sub-hypothesis h_1: "Tissues lacking GRAP2 expression consistently lack IL-2 expression, indicating that GRAP2 expression is necessary for IL-2 production." ``` ## Relevance Checker ``` Relevance Checker Proposed falsification test passes relevance check: Proposal: Main hypothesis: Gene GRAP2 regulates the production of Interleukin-2 (IL -2). Falsification Test name: Essential Expression Overlap Test Falsification Test description: Classify each tissue in df_gtex_tissue_gene_tpm as having GRAP2 expressed/not expressed and IL-2 expressed/not expressed (based on a small nonzero TPM threshold). Construct a 2 x2 table with these presence/ absence calls across tissues. Apply Fisher's exact test to see if IL-2 is indeed "absent" whenever GRAP2 is absent. Failure to observe this co-absence pattern (i.e., finding tissues with no GRAP2 expression but clear IL-2 expression) would be evidence falsifying the main hypothesis. Falsification Test Null sub-hypothesis: There is no essential overlap pattern; IL-2 may be present even when GRAP2 is not expressed. Falsification Test Alternate sub-hypothesis: Tissues lacking GRAP2 expression consistently lack IL-2 expression, indicating that GRAP2 expression is necessary for IL-2 production. Relevance score 0.8 Reasoning: This presence/absence study directly addresses a major component of the main hypothesis namely, whether GRAP2 is required for IL-2 expression but it relies on tissue-level correlation rather than a more detailed mechanistic test . Positive or negative findings would strongly influence confidence in the main hypothesis by demonstrating or refuting necessity of GRAP2, yet the approach is still correlational rather than providing definitive mechanistic proof. ``` ## Experiment Execution Agent ``` import pandas as pd import numpy as np from scipy.stats import fisher_exact # --- Code Block --- # Set a small threshold to classify presence vs absence threshold = 0.1 # Pivot df_gtex_tissue_gene_tpm: index=Tissue, columns=Gene, values=Expression. pivoted = df_gtex_tissue_gene_tpm.pivot(index='Tissue', columns='Gene', values='Expression').fillna(0) # If GRAP2 or IL2 are not in columns, handle gracefully if 'GRAP2' not in pivoted.columns or 'IL2' not in pivoted.columns: ``` ``` <loc_0><loc_0><loc_500><loc_499><_Python_>Automated Hypothesis Validation with Agentic Sequential Falsifications raise ValueError("No expression data available for GRAP2 or IL2 in df_gtex_tissue_gene_tpm.") # Determine presence/absence for GRAP2 and IL2 presence_GRAP2 = pivoted['GRAP2'] > threshold presence_IL2 = pivoted['IL2'] > threshold # Construct 2x2 table # Rows: GRAP2 presence/absence # Cols: IL2 presence/absence true_true = np.sum(presence_GRAP2 & presence_IL2) true_false = np.sum(presence_GRAP2 & (~presence_IL2)) false_true = np.sum((~presence_GRAP2) & presence_IL2) false_false = np.sum((~presence_GRAP2) & (~presence_IL2)) contingency_table = [[true_true, true_false], [false_true, false_false]] # Perform Fisher's exact test _, p_value = fisher_exact(contingency_table, alternative='two-sided') # Print results in scientific notation print("p-value:", format(p_value, '.3e')) ``` ## Observation ``` Observation Traceback (most recent call last): File "/dfs/user/kexinh/hypothesis_agent/falsification_agent/agent.py", line 365, in run_code exec(full_code, exec_globals) File "<string>", line 11, in <module> File "/dfs/user/kexinh/miniconda3/envs/hypo_agent/lib/python3.9/site-packages/ pandas/core/frame.py", line 9339, in pivot return pivot(self, index=index, columns=columns, values=values) File "/dfs/user/kexinh/miniconda3/envs/hypo_agent/lib/python3.9/site-packages/ pandas/core/reshape/pivot.py", line 570, in pivot result = indexed.unstack(columns_listlike) # type: ignore[arg-type] File "/dfs/user/kexinh/miniconda3/envs/hypo_agent/lib/python3.9/site-packages/ pandas/core/series.py", line 4615, in unstack return unstack(self, level, fill_value, sort) File "/dfs/user/kexinh/miniconda3/envs/hypo_agent/lib/python3.9/site-packages/ pandas/core/reshape/reshape.py", line 517, in unstack unstacker = _Unstacker( File "/dfs/user/kexinh/miniconda3/envs/hypo_agent/lib/python3.9/site-packages/ pandas/core/reshape/reshape.py", line 154, in __init__ self._make_selectors() File "/dfs/user/kexinh/miniconda3/envs/hypo_agent/lib/python3.9/site-packages/ pandas/core/reshape/reshape.py", line 210, in _make_selectors raise ValueError("Index contains duplicate entries, cannot reshape") ValueError: Index contains duplicate entries, cannot reshape ``` ## Experiment Execution Agent ``` import pandas as pd import numpy as np from scipy.stats import fisher_exact # --- Code Block --- # Falsification Test: Essential Expression Overlap Test # Main hypothesis: GRAP2 regulates IL2 production. # We check if tissues that lack GRAP2 expression also consistently lack IL2. # Steps: # 1) Use pivot_table with mean aggregator to handle duplicate Tissue-Gene pairs. # 2) Classify GRAP2 and IL2 expression as present/absent using a small threshold. # 3) Construct a 2 x2 contingency table. ``` ``` # 4) Perform Fisher's exact test. threshold = 0.1 # Create pivot table to handle potential duplicates by Tissue,Gene pivoted = df_gtex_tissue_gene_tpm.pivot_table( index='Tissue', columns='Gene', values='Expression', aggfunc='mean' ).fillna(0) # Check for GRAP2 and IL2 in pivoted table if 'GRAP2' not in pivoted.columns or 'IL2' not in pivoted.columns: raise ValueError("No expression data available for GRAP2 or IL2 in df_gtex_tissue_gene_tpm.") # Classify presence/absence presence_GRAP2 = pivoted['GRAP2'] > threshold presence_IL2 = pivoted['IL2'] > threshold # Construct 2 x2 contingency table true_true = np.sum(presence_GRAP2 & presence_IL2) true_false = np.sum(presence_GRAP2 & (~presence_IL2)) false_true = np.sum((~presence_GRAP2) & presence_IL2) false_false = np.sum(( ~presence_GRAP2) & ( presence_IL2)) contingency_table = [[true_true, true_false], [false_true, false_false]] # Fisher's exact test odds_ratio, p_value = fisher_exact(contingency_table, alternative='two-sided') # Print p-value in scientific notation print("p-value:", f"{p_value:.3e}") ``` ## Observation Captured Output: p-value: 1.832e-01 ## Sequential Error Control ``` ================================= Serial Testing ============================ List of p-values: [0.031, 0.1832] Summarized sequential statistics: 3.317389204898911 Sequential test result: insufficient evidence - CONTINUE ============= Testing decision is insufficient evidence - CONTINUE =========== ``` ## Experiment Design Agent ## Round 3 ``` Round 3 INITIAL PROPOSAL FOR A NEW FALSIFICATION TEST: (1) Name: "IL-2 Pathway Co-expression Test for GRAP2" (2) Test Description (initial idea): Rather than directly correlating GRAP2 expression with IL-2 itself (as in Round 1) or checking co-absence/presence patterns (as in Round 2), we will examine whether GRAP2 is co-expressed with key IL-2 signaling genes (e.g., IL2RA, IL2RB, IL2RG, JAK3, STAT5, etc.) across tissues in df_gtex_tissue_gene_tpm. Collect TPM expression for GRAP2 and for each IL-2 signaling gene across tissues . Compute the correlation (or multiple regression) across all tissues between GRAP2 and these IL-2 signaling genes. ``` ``` Automated Hypothesis Validation with Agentic Sequential Falsifications ``` tissues to assess whether GRAP2 is co-expressed with these IL-2 signaling genes. Summarize correlation results (e.g., via Fisher's z-transformation) into an overall p-value. A lack of consistent correlation would be evidence falsifying the main hypothesis that GRAP2 regulates IL-2 production. (3) Null Sub-hypothesis (h\_0): "GRAP2 expression is not correlated (no systematic co-variation) with IL-2 signaling components across tissues." (4) Alternate Sub-hypothesis (h\_1): "There is significant co-expression between GRAP2 and IL-2 signaling genes across tissues, consistent with a shared regulatory or functional link to IL-2 biology ." ## Relevance Checker ``` Relevance Checker Proposed falsification test is not relevant enough to the main hypothesis! Proposal: Main hypothesis: Gene GRAP2 regulates the production of Interleukin-2 (IL-2). Falsification Test name: IL-2 Pathway Co-expression Test for GRAP2 Falsification Test description: Using df_tigex_tissue_gene_tpm, gather expression values for GRAP2 and for canonical IL-2 signaling genes (e.g., IL2RA, IL2RB, IL2RG, JAK3, STAT5, etc.) across all available tissues. Compute a correlation ( or multiple linear regression) across tissues to assess whether GRAP2 is co- expressed with these IL-2 signaling genes. Summarize correlation results (e.g., via Fisher's z-transformation) into an overall p-value. A lack of consistent correlation would be evidence falsifying the main hypothesis that GRAP2 regulates IL-2 production. Falsification Test Null sub-hypothesis: GRAP2 expression is not correlated (no systematic co-variation) with IL-2 signaling components across tissues. Falsification Test Alternate sub-hypothesis: There is significant co-expression between GRAP2 and IL-2 signaling genes across tissues, consistent with a shared regulatory or functional link to IL-2 biology. Relevance score: 0.6 Reasoning: While evaluating tissue-level co-expression of GRAP2 with IL-2 signaling genes does address one of the main hypothesis's predictions (that GRAP2 is linked to IL-2 production), it is ultimately only an indirect measure of regulation. Demonstrating correlation across tissues does not in itself provide a direct mechanistic link or functional test of regulation (e.g., knockdown or overexpression). Hence, it offers some supportive or refuting insight, but only moderate predictive value regarding the main hypothesis. ``` ## Experiment Design Agent ## Previous experiment does not pass relevance checker, proposing another experiment. Round 3 ## Initial Proposal - (1) Name of the test: "Housekeeping Normalized Partial Correlation Test" - (2) Test description (in plain language): From df\_gtex\_tissue\_gene\_tpm, retrieve the per-tissue TPM values of GRAP2, IL2, and at least one well-established housekeeping gene (e.g., GAPDH, ACTB) across all available tissues. Fit a linear model or compute partial correlations to assess whether GRAP2's expression levels predict IL2 expression after accounting for general ( housekeeping) transcriptional activity. In other words, we ask: "Does GRAP2 covary with IL2 over and above the baseline expression captured by a housekeeping gene?" If the partial-regression coefficient (or partial-correlation) for GRAP2 in predicting IL2 is non-significant, that result would tend to falsify the main hypothesis that GRAP2 is specifically required for IL2 production (i.e., GRAP2 is not uniquely associated with IL2 once overall transcriptional levels are factored out). Report a p-value for this association; a large p-value (above conventional significance thresholds) favors falsifying the hypothesis. (3) Null sub-hypothesis h\_0: "After controlling for housekeeping-gene expression, there is no relationship between GRAP2 expression and IL2 expression across human tissues." Mathematically, the partial correlation (or partial-regression coefficient) of GRAP2 for predicting IL2 is zero. (4) Alternate sub-hypothesis h\_1: "Even after controlling for housekeeping-gene expression, GRAP2 expression remains significantly associated with IL2 expression," implying that GRAP2 specifically helps drive or track IL2 levels. Round 1 (1) CRITIC: "If the main hypothesis is null, is this sub-hypothesis also null?" The main hypothesis is that "GRAP2 regulates IL2." If that is in fact false (i.e ., GRAP2 does not meaningfully affect IL2 levels), then we would not expect GRAP2 to have a unique partial correlation with IL2 once we control for baseline expression. In other words, under the null scenario (no real GRAP2 IL2 regulatory link), we should indeed find that GRAP2 and IL2 show no unique association, so the partial correlation would be zero. Hence, if the main hypothesis is null, the falsification sub-hypothesis of "GRAP2 has a nonzero partial correlation" would also be null. So that condition is satisfied. "Is it redundant with existing tests or with those that failed?" So far, we have had: 1) A simple correlation of GRAP2 and IL2 expression across tissues. 2) A presence/absence (Fisher exact) approach. 3) A contrast of GRAP2 across highvs. low-IL2 tissues. 4) A broad co-expression test of GRAP2 with IL2-signaling genes. In contrast, the newly proposed test focuses on controlling for a housekeeping gene to rule out the trivial possibility that GRAP2 and IL2 are simply covaried because of global transcription differences between tissues. This partial-correlation approach is indeed different from a raw correlation or from presence/absence/overlap tests. It is also not the same as the multi-gene "IL-2 pathway co-expression test," because that test specifically regressed GRAP2 against IL2-signaling mediators (e.g. JAK3, STAT5). Here, we are trying to see if GRAP2 specifically goes along with IL2 beyond general expression baselines. Thus it is not redundant. "Can it be answered with the given data?" Yes. df\_gtex\_tissue\_gene\_tpm has per-tissue TPM values for many genes, including (very likely) at least one standard housekeeping gene. That allows us to implement the partial-correlation or linear-model approach. (2) REFLECT: This test appears sound, distinctly focuses on controlling for baseline expression, and can be implemented with the given GTEx-style data. It is neither a pure correlation test (as in Round 1) nor simply presence/absence or a multigene correlation (Round 2 or the later failed attempts). It also maintains the crucial property that if GRAP2 truly has no effect on IL2, the partial correlation should vanish. No further refinements seem strictly necessary. The test is implementable, nonredundant, and meets the requirement that under the main null hypothesis, the sub-hypothesis would also be null. ## Final Answer - (1) Name: "Housekeeping Normalized Partial Correlation Test" (2) Test Description: Using df\_gtex\_tissue\_gene\_tpm, retrieve expression (TPM) for GRAP2, IL2, and one or more canonical housekeeping genes across all tissues. Run a partial-correlation or linear-regression model of IL2 on GRAP2 while controlling for housekeepinggene levels (to account for baseline transcriptional activity). If GRAP2 retains a statistically significant association with IL2 after that control, it supports the main hypothesis; an insignificant partial-correlation p-value would tend to falsify it. (3) Null Sub-hypothesis h\_0: "There is zero partial correlation between GRAP2 and IL2 when controlling for housekeeping-gene expression levels." (4) Alternate Sub-hypothesis h\_1: "There is a significant partial correlation between GRAP2 and IL2, over and above housekeeping-gene expression, indicative of a specific regulatory link." ## Relevance Checker ``` Relevance Checker Proposed falsification test passes relevance check: Proposal: Main hypothesis: Gene GRAP2 regulates the production of Interleukin-2 (IL - 2). Falsification Test name: Housekeeping-Normalized Partial-Correlation Test Falsification Test description: From df_tex_tissue_gene_tpm, retrieve expression ( TPM) for GRAP2, IL2, and at least one established housekeeping gene across all tissues. Then, run a partial-correlation or linear-regression model of IL2 on GRAP2 while controlling for the housekeeping-gene expression. If the partial correlation is significantly non-zero, it supports the hypothesis that GRAP2 specifically influences IL2 levels above and beyond general transcriptional activity. If the partial correlation is zero, that would falsify the hypothesis . Falsification Test Null sub-hypothesis: After controlling for housekeeping-gene expression, there is no relationship between GRAP2 and IL2 (partial correlation = 0). Falsification Test Alternate sub-hypothesis: Even after controlling for housekeeping-gene expression, GRAP2 remains significantly associated with IL2 ( partial correlation 0). Relevance score 1.0 Reasoning: This test is highly relevant because it directly measures whether GRAP2 remains associated with IL-2 after controlling for general transcription ( housekeeping gene), providing specific, mechanistic evidence for or against a GRAP2 IL-2 regulatory link. A significant partial correlation would strongly support the main hypothesis that GRAP2 regulates IL-2 production, conferring high predictive value for the relationship. ``` ## Experiment Execution Agent ``` Experiment Execution Agent import pandas as pd import numpy as np import statsmodels.api as sm from tqdm import tqdm # ---------------------------------------- # Step 1: Prepare relevant data # ---------------------------------------- # Define the genes of interest gene_grap2 = 'GRAP2' gene_il2 = 'IL2' gene_housekeeping = 'GAPDH' # A standard housekeeping gene # Filter for the relevant genes df_filtered = df_gtex_tissue_gene_tpm[df_gtex_tissue_gene_tpm['Gene'].isin([ gene_grap2, gene_il2, gene_housekeeping])] ``` ``` </doctag> ``` ## Observation ``` ---CODE IMPORT CHECK: FAILED--- ``` ## Experiment Execution Agent ``` import pandas as pd import numpy as np # Housekeeping-Normalized Partial-Correlation Test ``` ``` ``` ## Observation ``` Captured Output: Observed partial correlation: 0.3764 Permutation-based p-value: 2.80e-02 ``` ## Sequential Error Control ``` ====================================== Sequential Testing ====================================== List of p-values: [0.031, 0.1832, 0.028] Summarized sequential statistics: 9.912596214980638 Sequential test result: insufficient evidence - CONTINUE ============= Testing decision is insufficient evidence - CONTINUE ============= ``` ## Experiment Design Agent ## Round 4 INITIAL PROPOSAL FOR A NEW FALSIFICATION TEST - (1) Name (tentative): "Random-Gene Negative Control Test for GRAP2 IL2 Co-expression " (2) Test Description (tentative): From df\_gtex\_tissue\_gene\_tpm, compute the across-tissue Pearson (or Spearman) correlation between GRAP2 and IL2. Then, randomly select a large number of other genes (e.g., 100 1000) from the same dataset and compute the correlation of each of those genes with IL2 across the same tissues. Compare the GRAP2 IL2 correlation coefficient to this "null" distribution of correlation coefficients from random genes. If the GRAP2 IL2 correlation is indistinguishable from that of randomly chosen genes (i.e., not significantly higher than the random distribution), that refutes a special regulatory relationship and thus falsifies the main hypothesis . random- Conversely, if GRAP2 IL2 stands out as significantly greater than most gene correlations with IL2, it supports (but does not prove) GRAP2's special role in IL-2 regulation. - (3) Null Sub-hypothesis h\_0 (tentative): "GRAP2 is no more correlated with IL2 across tissues than a random gene is." In other words, if the main hypothesis ("GRAP2 regulates IL2") is truly null, we expect no special correlation over and above what one would see if GRAP2 were just any arbitrary gene. - (4) Alternate Sub-hypothesis h\_1 (tentative): "GRAP2 gene," is significantly more correlated with IL2 across tissues than a random indicating a potential functional/causal link consistent with regulation. ## Round 1: CRITIC - 1) Does the test become null if the main hypothesis is Yes. If GRAP2 does not in fact regulate IL2, then any correlation computed across tissues would be comparable to random. Hence, the test's subhypothesis (that GRAP2 is more correlated with IL2 than random genes) also fail. - null? would - 2) Is this test redundant with (or overly similar to) existing or failed tests? Existing tests involved direct correlation between GRAP2 and IL2 or presence/ absence overlap, as well as partial-correlation controlling for housekeeping genes. None used a random-gene control distribution. So this is not redundant with the existing correlation or partial-correlation approaches. - 3) Is the test answerable from the given data, and does it avoid the previously failed proposals? Yes. We can implement this test with df\_gtex\_tissue\_gene\_tpm by extracting GRAP2 expression, IL2 expression, and expression for all other genes. It is not one of the cited failed tests. ## Conclusion of Critic: The proposed test appears valid. It is not redundantly overlapping with previous correlation tests (because it uses a random distribution as a control), it achieves the "if main hypothesis is null, then sub-hypothesis is also null" requirement, and it can be carried out with df\_gtex\_tissue\_gene\_tpm. Round 1: REFLECT The test definition seems sound and distinct enough from the previous correlation-based tests. It cleanly states a null vs. an alternative sub-hypothesis using a random-gene background distribution, which strengthens the "falsification" criterion. It can be readily implemented using the provided GTEx expression data ( df\_gtex\_tissue\_gene\_tpm). Therefore, the test is suitably defined and meets the requirements. No further refinement is necessary. FINAL FALSIFICATION TEST DEFINITION (1) Name of the test: "Random-Gene Negative Control Test for GRAP2 IL2 Co-expression" (2) Test description: Using df\_gtex\_tissue\_gene\_tpm, first calculate the correlation (e.g., Pearson) between GRAP2 and IL2 expression across all tissues. Then repeatedly sample other genes (e.g., 100 1000 random genes) from the same dataset and calculate each of their correlations with IL2. This yields a null distribution of correlation coefficients representing "random" relationships to IL2. Finally, compare the observed GRAP2 IL2 correlation to this distribution (e.g., compute a p-value as the fraction of random-gene correlations that exceed GRAP2 IL2's correlation). If GRAP2's correlation with IL2 is not significantly different from random, that falsifies the main hypothesis. If it is significantly stronger , it supports (but does not prove) a regulatory link. (3) Null sub-hypothesis h\_0: "GRAP2 is no more correlated with IL2 than a random gene," implying no special coexpression pattern that would suggest regulation. (4) Alternate sub-hypothesis h\_1: "GRAP2 is significantly more correlated with IL2 than a random gene," indicating a special co-expression relationship supportive of the hypothesis that GRAP2 regulates IL2. ## Relevance Checker ``` Relevance Checker Proposed falsification test passes relevance check: Proposal: Main hypothesis: Gene GRAP2 regulates the production of Interleukin-2 (IL -2). Falsification Test name: Random-Gene Negative Control Test for GRAP2 IL2 Co- expression Falsification Test description: Using df_gtex_tissue_gene_tpm, first calculate the correlation (e.g., Pearson) between GRAP2 and IL2 expression across all tissues . Then repeatedly sample other genes (e.g., 100 1000 random genes) from the same dataset and calculate each of their correlations with IL2. This yields a null distribution of correlation coefficients representing 'random' relationships to IL2. Finally, compare the observed GRAP2 IL2 correlation with this distribution (e.g., compute a p-value as the fraction of random-gene correlations that exceed GRAP2 IL2's correlation). If GRAP2's correlation with IL2 is not significantly different from random, that falsifies the main hypothesis. If it is significantly stronger, it supports (but does not prove) a regulatory link. Falsification Test Null sub-hypothesis: GRAP2 is no more correlated with IL2 across tissues than a random gene is (no special co-expression pattern). Falsification Test Alternate sub-hypothesis: GRAP2 is significantly more correlated with IL2 across tissues than a random gene, indicating a potential co- expression relationship consistent with regulation. Relevance score 0.8 ``` ``` Reasoning: This correlation-based test directly compares GRAP2 IL2 co-expression against random gene relationships, providing substantial evidence that GRAP2 is linked to IL2 expression patterns. While the data do not constitute definitive mechanistic proof of regulation, demonstrating a uniquely high correlation would significantly bolster confidence in tie hypothesis that GRAP2 regulates IL -2. ``` ## Experiment Execution Agent ``` mechanistic proof of regulation, demonstrating a uniquely high correlation would -2. ``` ## Automated Hypothesis Validation with Agentic Sequential Falsifications ``` raise ValueError("No valid random genes yielded correlation data. Test cannot proceed.") # 7) Compute the one-sided p-value as fraction of random correlations >= r_observed. count_exceed = sum(1 for r in null_correlations if r >= r_observed) p_value = count_exceed / len(null_correlations) # 8) Print the p-value in scientific notation. print(f"Observed GRAP2 IL2 correlation: {r_observed:.5f}") print(f"P-value: {p_value:.2e}") ``` ## Observation ``` Computing random correlations: 100% Observed GRAP2 IL2 correlation: 0.35043 P-value: 5.60e-02 ``` ## Sequential Error Control ``` ====================================== Sequential Testing ====================================== List of p-values: [0.031, 0.1832, 0.028, 0.056] Summarized sequential statistics: 20.944182166241685 Sequential test result: sufficient evidence - PASS =================== Testing decision is sufficient evidence - PASS =================== ``` ## Summarizer - (1) Main scientific hypothesis under study: "Gene GRAP2 regulates the production of Interleukin-2 (IL-2)." - (2) Result of the sequential falsification test: "Sufficient evidence PASS with statistics E-value of 20.944182166241685." - (3) Reasoning, summary, and analysis: 4. Round 1 (Tissue-Wide Expression Correlation): The reported test statistic (0.031) was below commonly used significance thresholds, suggesting a potential correlation between GRAP2 and IL-2 expression. 5. Round 2 (Essential Expression Overlap): The test statistic (0.1832) did not appear to reject the null sub-hypothesis; hence, no strong overlap pattern was confirmed. 6. Round 3 (Housekeeping-Normalized Partial-Correlation): The result (0.028) indicated a statistically significant partial correlation between GRAP2 and IL-2 while controlling for housekeeping gene expression. 7. Round 4 (Random-Gene Negative Control Test): The test statistic (0.056) was borderline but did not strongly refute a specific correlation signal for GRAP2 compared to random genes. 8. Combining all four tests, the sequential e-value analysis overall led to a "PASS," indicating that these tests, taken together, failed to falsify the primary hypothesis. - (4) Conclusion on whether the hypothesis is true or false (just return True/False): True - (5) Rationale of the conclusion: 11. According to the sequential testing procedure, there was sufficient evidence (PASS) to conclude that the hypothesis "Gene GRAP2 regulates the production of Interleukin-2 (IL-2)" was not falsified. Figure 10: Example sequential falsification trajectory on the TargetVal-IL2 Benchmark with Claude-Sonnet-3.5 as the LLM Backbone. ## Problem Statement ``` Problem Statement Main Hypothesis: Gene LCP2 regulates the production of Interleukin-2 (IL-2). Data sources: df_gtex_tissue_gene_tpm: {'Description': 'ENSG00000186092', 'Tissue': 'Adipose - Subcutaneous', 'Expression': 0.0453961, 'Gene': 'OR4F5'} df_gene_info: {'gene_id': 'ENSG00000228037', 'transcript_id': 'ENST00000424215', 'chr': '1', 'gene_start': 2581560, 'gene_end': 2584533, 'strand': 1, 'transcript_start': 2581560, 'transcript_end': 2584533, 'tss': 2581560, 'transcript_is_canonical': 1.0, 'gene_name': nan, 'percentage_gene_gc_content': 51.11, 'gene_type': 'lncRNA'} df_genetic_interaction: {'interaction_id': 206363, 'gene_a_id': 'YCR011C', 'gene_b_id': 'YCL025C', 'experimental_system_type': 'genetic', 'pubmed_id': 'PUBMED:16269340', 'organism_id_a': 559292, 'organism_id_b': 559292, 'throughput_type': 'High Throughput', 'experimental_score': '-5.6431'} # 19 more datasets omitted for presentation ``` ## Experiment Design Agent ``` Experiment Design Agent Round 1 Let me propose an initial falsification test: Initial Test Proposal: (1) Name: Physical Interaction Test (2) Test description: Test if LCP2 physically interacts with IL-2 or IL-2 pathway proteins using protein-protein interaction data (3) Null sub-hypothesis h_0: LCP2 does not show significant physical interactions with IL-2 or IL-2 pathway proteins (4) Alternate sub-hypothesis h_1: LCP2 shows significant physical interactions with IL-2 or IL-2 pathway proteins Let me critique this proposal: Critic: 1. Validity: If the main hypothesis "LCP2 regulates IL-2 production" is null, would this test also be null? - Not necessarily. Physical interaction is not required for regulation. A gene can regulate another gene's production through indirect pathways without physical protein interaction. - Therefore, this test fails the crucial validity requirement. 2. Data availability: - We have protein interaction data (df_affinity_capture_ms, df_two_hybrid, etc.) - However, we need gene IDs for both LCP2 and IL-2 to use these datasets 3. Redundancy: - No redundancy issues as there are no existing tests yet Reflection: ``` - We need a better test that: 1. Has proper logical implication (if main hypothesis is null, test should be null) 2. Uses available data more effectively 3. Is more directly related to regulation of production Let me propose a revised test: Revised Test Proposal: (1) Name: Expression Correlation Test (2) Test description: Test if LCP2 expression levels correlate with IL-2 expression levels across tissues in GTEx data (3) Null sub-hypothesis h\_0: There is no significant correlation between LCP2 and IL -2 expression levels across tissues (4) Alternate sub-hypothesis h\_1: There is significant positive correlation between LCP2 and IL-2 expression levels across tissues Critic: 1. Validity: If LCP2 doesn't regulate IL-2 production (null main hypothesis), then there should indeed be no correlation in expression (null sub-hypothesis). This satisfies the validity requirement. 2. Data availability: -We have df\_gtex\_tissue\_gene\_tpm which contains gene expression data across tissues -We can calculate correlation between LCP2 and IL-2 expression 3. Implementation feasibility: -This test can be implemented using standard correlation analysis -P-value can be computed from the correlation test I This revised test appears to be valid, implementable, and logically sound. recommend proceeding with this final test definition. ## Relevance Checker Proposed falsification test passes relevance check: - Proposal: Main hypothesis: Gene LCP2 regulates the production of Interleukin-2 (IL -2). Falsification Test name: Expression Correlation Test - Falsification Test description: Test if LCP2 expression levels correlate with IL-2 expression levels across tissues in GTEx data - Falsification Test Null sub-hypothesis: There is no significant correlation between LCP2 and IL-2 expression levels across tissues - Falsification Test Alternate sub-hypothesis: There is significant positive correlation between LCP2 and IL-2 expression levels across tissues Relevance score 0.8 - Reasoning: This Expression Correlation Test is strongly relevant to the main hypothesis for several reasons: 1. Direct Variable Connection: The test directly examines the relationship between the two key variables in the main hypothesis -LCP2 and IL-2. 2. Mechanistic Insight: While correlation doesn't prove causation, a strong correlation would provide substantial supporting evidence for a regulatory relationship, especially across different tissues. 3. Predictive Value: If LCP2 regulates IL-2 production, we would expect to see a significant positive correlation between their expression levels. The test results would significantly impact our confidence in the main hypothesis. - The reason this doesn't receive a 1.0 score is that correlation analysis alone cannot definitively prove regulation -it's possible to have correlated expression without direct regulation. However, this test provides strong supporting evidence and is a logical first step in investigating the regulatory relationship. ## Experiment Execution Agent ``` Let me inspect the GTEx dataset first to locate LCP2 and IL-2 expression data and prepare for the correlation analysis. ---------------------------- Action: python_repl_ast Action Input: # Check unique genes in the dataset df_gtex_tissue_gene_tpm['Gene'].unique() ``` ## Observation ['OR4F5' 'OR4F29' 'OR4F16' ... 'MT-ND5' 'MT-ND6' 'MT-CYB'] ## Experiment Execution Agent - I apologize for not being able to proceed with the falsification test as intended. After examining the dataset, I notice that the required genes (LCP2 and IL-2) are not present in the GTEx dataset provided. Without the expression data for these specific genes, I cannot perform a meaningful correlation analysis between LCP2 and IL-2 expression levels. For the sake of statistical rigor, I should return: Final Answer: The falsification test cannot be performed due to missing data for LCP2 and IL-2 genes in the provided GTEx dataset. P-value = 1.00e+00 (default when test cannot be performed). Note: To properly test this hypothesis, we would need GTEx data containing expression levels for both LCP2 and IL-2 across multiple tissues. The current dataset does not contain this information. ## Sequential Error Control ``` ================================== Sequential Testing ========================= List of p-values: [1.0] Summarized sequential statistics: 0.5 Sequential test result: insufficient evidence - CONTINUE ============= Testing decision is insufficient evidence - CONTINUE ============= ``` ## Experiment Design Agent ## Round 2 Let me propose an initial falsification test: Initial Proposal: - (1) Name: Protein-Protein Interaction Test - (2) Test description: Test if LCP2 protein physically interacts with IL-2 or key IL -2 regulatory proteins using protein interaction data - (3) Null sub-hypothesis h\_0: LCP2 does not show significant physical interactions with IL-2 or its known regulatory proteins - (4) Alternate sub-hypothesis h\_1: LCP2 shows significant physical interactions with IL-2 or its known regulatory proteins Let me critique this proposal: ## Critic: 1. Causality Check: If the main hypothesis (LCP2 regulates IL-2) is null, would this sub-hypothesis also be null? 2. -Not necessarily. Physical protein interactions don't always imply regulatory relationships. Two proteins can interact without one regulating the other's ``` Automated Hypothesis Validation with Agentic Sequential Falsifications ----------------------------- production. - The test needs to be more specific about regulatory mechanisms. 2. Data Availability: - We have protein interaction data (df_affinity_capture_ms, df_two_hybrid, etc.) - However, we need to ensure we can specifically identify IL-2 related proteins. 3. Redundancy: - This test is not redundant with the previous expression correlation test. - It examines a different molecular mechanism. Reflection: Let me refine this test to better align with regulatory relationships: Refined Proposal: (1) Name: LCP2 Regulatory Network Analysis Test (2) Test description: Test if LCP2 shows physical interactions specifically with known transcription factors or signaling proteins that are established regulators of IL-2 production using high-confidence protein interaction data (3) Null sub-hypothesis h_0: LCP2 does not show significant physical interactions with established IL-2 transcriptional regulators or upstream signaling proteins (4) Alternate sub-hypothesis h_1: LCP2 shows significant physical interactions with established IL-2 transcriptional regulators or upstream signaling proteins This refined version: - Better reflects regulatory relationships - Has a clearer causal implication (if LCP2 doesn't regulate IL-2, it shouldn't interact with IL-2's regulatory machinery) - Can be implemented using the available protein interaction datasets - Is more specific about the type of interactions we're looking for I believe this refined test is ready for implementation. ``` ## Relevance Checker - Proposed falsification test passes relevance check: Proposal: Main hypothesis: Gene LCP2 regulates the production of Interleukin-2 (IL -2). Falsification Test name: LCP2 Regulatory Network Analysis Test Falsification Test description: Test if LCP2 shows physical interactions specifically with known transcription factors or signaling proteins that are established regulators of IL-2 production using high-confidence protein interaction data Falsification Test Null sub-hypothesis: LCP2 does not show significant physical interactions with established IL-2 transcriptional regulators or upstream signaling proteins Falsification Test Alternate sub-hypothesis: LCP2 shows significant physical interactions with established IL-2 transcriptional regulators or upstream signaling proteins Relevance score 0.8 Reasoning: This sub-hypothesis test is strongly relevant to the main hypothesis for several reasons: 1. Mechanistic Connection: The test directly investigates the molecular mechanisms through which LCP2 could regulate IL-2 production by examining physical protein interactions with known IL-2 regulators. This provides a clear mechanistic link to how LCP2 might influence IL-2 production. 2. Evidence Strength: Using high-confidence protein interaction data offers strong, direct evidence of potential regulatory relationships. If LCP2 physically interacts with known IL-2 regulators, this would provide substantial support for its role in IL-2 regulation. it 3. Predictive Value: The results would significantly impact our confidence in the main hypothesis. If LCP2 shows interactions with established IL-2 regulators, strongly suggests a regulatory role. Conversely, absence of such interactions would significantly weaken the main hypothesis. While this test is highly informative, it receives 0.8 rather than 1.0 because it focuses on physical interactions alone and doesn't directly measure IL-2 production changes, which would be needed for complete validation of the regulatory relationship. ## Experiment Execution Agent ``` Experiment Execution Agent Let me help analyze this falsification test. We need to examine if LCP2 shows significant physical interactions with IL-2 related proteins. We'll focus on physical interaction datasets, particularly df_affinity_capture_ms, df_two_hybrid, df_co_fractionation, df_proximity_label_ms, and df_reconstituted_complex. Let's start by identifying LCP2's interactions. ------------------------ Action: python_repl_ast Action Input: import pandas as pd import numpy as np from scipy import stats # Combine all physical interaction datasets physical_interactions = pd.concat([ df_affinity_capture_ms, df_two_hybrid, df_co_fractionation, df_proximity_label_ms, df_reconstituted_complex ]) # Filter for human interactions (organism_id 9606) human_interactions = physical_interactions[ (physical_interactions['organism_id_a'] == 9606) & (physical_interactions['organism_id_b'] == 9606) ] # LCP2's Ensembl ID is ENSG00000043462 lcp2_interactions = human_interactions[ (human_interactions['gene_a_id'] == 'ENSG00000043462') | (human_interactions['gene_b_id'] == 'ENSG00000043462') ] print(f"Number of LCP2 physical interactions: {len(lcp2_interactions)}") lcp2_interactions.head() ``` ## Observation ``` Number of LCP2 physical interactions: 3 interaction_id gene_a_id gene_b_id \ 1939208 2846948 ENSG00000141736 ENSG00000043462 1939268 2847008 ENSG00000065361 ENSG00000043462 1939331 2847071 ENSG00000178568 ENSG00000043462 experimental_system_type pubmed_id organism_id_a \ 1939208 physical PUBMED:22973453 9606 1939268 physical PUBMED:22973453 9606 1939331 physical PUBMED:22973453 9606 organism_id_b throughput_type experimental_score 1939208 9606 High Throughput 11.66 1939268 9606 High Throughput 3.24 1939331 9606 High Throughput 6.1 ``` ## Experiment Execution Agent ``` Experiment Execution Agent Let me analyze this falsification test properly. We need to examine if LCP2 shows significant physical interactions with IL-2 related proteins compared to what would be expected by chance. First, let's get the total number of interactions in our physical interaction network as a baseline. Action: python_repl_ast Action Input: # Get total number of unique genes in the physical interaction network unique_genes = set(human_interactions['gene_a_id'].tolist() + human_interactions[' gene_b_id'].tolist()) total_genes = len(unique_genes) # Get total number of interactions total_interactions = len(human_interactions) print(f"Total number of genes in network: {total_genes}") print(f"Total number of interactions: {total_interactions}") # Known IL-2 pathway genes (including key transcription factors and signaling proteins) il2_related_genes = { 'ENSG00000109471', # IL2 'ENSG00000134460', # IL2RA 'ENSG00000102739', # NFATC2 'ENSG00000179222', # NFATC1 'ENSG00000131196', # NFKB1 'ENSG00000100906', # NFKBIA 'ENSG00000162594', # ELOB 'ENSG00000166949', # SMAD3 'ENSG00000177606', # JUN 'ENSG00000232810', # NFBK2 } # Count how many of LCP2's interactions are with IL-2 related genes lcp2_il2_interactions = lcp2_interactions[ (lcp2_interactions['gene_a_id'].isin(il2_related_genes)) | (lcp2_interactions['gene_b_id'].isin(il2_related_genes)) ] observed_count = len(lcp2_il2_interactions) print(f" Number of LCP2 interactions with IL-2 pathway genes: {observed_count}") ``` ## Observation ``` Total number of genes in network: 17203 Total number of interactions: 394723 Number of LCP2 interactions with IL-2 pathway genes: 0 ``` ## Experiment Execution Agent ``` Let's continue with a proper statistical test to determine if the lack of observed interactions is statistically significant. -------------------------------------- Action: python_repl_ast Action Input: # Calculate expected probability of interaction with IL-2 pathway genes p_il2_gene = len(il2_related_genes) / total_genes # Given we observed 3 total LCP2 interactions, calculate probability of seeing 0 IL-2 related ones # Using binomial test from scipy.stats import binomtest ``` ## Automated Hypothesis Validation with Agentic Sequential Falsifications ``` result = binomtest(k=0, n=3, p=p_il2_gene, alternative='two-sided') print(f"P-value from binomial test: {result.pvalue:.2e}") ``` ## Observation P-value from binomial test: 1.00e+00 ## Experiment Execution Agent Now I know the final answer. Based on our rigorous analysis: 1. We found 3 high-confidence physical interactions for LCP2 in the human interactome 2. Out of these 3 interactions, 0 were with known IL-2 pathway genes 3. We performed a binomial test to determine if this observation was statistically significant, comparing against the null expectation based on the proportion of IL-2 related genes in the network 4. The test yielded a p-value of 1.00e+00, indicating no significant enrichment or depletion of IL-2 pathway interactions 5. Final Answer: We fail to reject the null hypothesis with a p-value of 1.00e+00. The data does not support the alternate hypothesis that LCP2 shows significant physical interactions with established IL-2 transcriptional regulators or upstream signaling proteins. ## Sequential Error Control ``` ====================================== Sequential Testing ========================== List of p-values: [1.0, 1.0] Summarized sequential statistics: 0.25 Sequential test result: insufficient evidence - CONTINUE ============= Testing decision is insufficient evidence - CONTINUE =========== ``` ## Experiment Design Agent ## Round 3 Let me propose a falsification test and then analyze it carefully. Initial Proposal: - (1) Name: Genetic Variant Impact Test - (2) Test description: Test if genetic variants affecting LCP2 function (pLoF, missense) are associated with immune-related phenotypes that are known to be IL -2 dependent using GeneBASS data - (3) Null sub-hypothesis h\_0: Genetic variants that affect LCP2 function show no significant association with IL-2-dependent immune phenotypes - (4) Alternate sub-hypothesis h\_1: Genetic variants that affect LCP2 function show significant association with IL-2-dependent immune phenotypes ## Let's analyze: ## Critic: 1. Logical Validity: If the main hypothesis (LCP2 regulates IL-2 production) is null , then genetic variants affecting LCP2 should indeed show no association with IL -2-dependent phenotypes. This satisfies the key requirement. 2. Non-redundancy: This test is distinct from previous 3. -Round 1 looked at expression 4. -Round 2 examined protein 5. This test looks at genetic evidence through variant ``` is test is distinct from previous tests: expression correlation protein interactions netic evidence through variant analysis. ``` ``` Automated Hypothesis Validation with Agentic Sequential Falsifications 3. Data Availability: Yes, we can implement this using: - df_genebass_pLoF_filtered for loss of function variants - df_genebass_missense_LC_filtered for missense variants These contain gene-phenotype associations. 4. Strength: This test provides causal evidence through genetic variation, which is stronger than correlation or interaction evidence. Reflection: The test could be improved by: 1. Being more specific about which immune phenotypes to look at 2. Clarifying the statistical threshold for significance 3. Specifying how to combine evidence across different variant types Refined Proposal: (1) Name: LCP2 Variant-Immune Phenotype Association Test (2) Test description: Test if LCP2 loss-of-function and missense variants are significantly associated with autoimmune conditions and T-cell related phenotypes in GeneBASS, which are known to be IL-2 dependent (3) Null sub-hypothesis h_0: LCP2 genetic variants (pLoF and missense) show no significant association (p > 0.05) with autoimmune and T-cell related phenotypes in GeneBASS (4) Alternate sub-hypothesis h_1: LCP2 genetic variants show significant association (p < 0.05) with autoimmune and T-cell related phenotypes in GeneBASS This refined version is more specific and implementable. I believe this test definition is now good enough to return to the user as it: - Maintains logical validity with the main hypothesis - Is non-redundant with existing tests - Is clearly implementable with available data - Provides strong causal evidence - Has clear statistical criteria ``` ## Relevance Checker - Proposed falsification test passes relevance check: Proposal: Main hypothesis: Gene LCP2 regulates the production of Interleukin-2 (IL -2). Falsification Test name: LCP2 Variant-Immune Phenotype Association Test Falsification Test description: Test if LCP2 loss-of-function and missense variants are significantly associated with autoimmune conditions and T-cell related phenotypes in GeneBASS, which are known to be IL-2 dependent Falsification Test Null sub-hypothesis: LCP2 genetic variants (pLoF and missense) show no significant association (p > 0.05) with autoimmune and T-cell related phenotypes in GeneBASS Falsification Test Alternate sub-hypothesis: LCP2 genetic variants show significant association (p < 0.05) with autoimmune and T-cell related phenotypes in GeneBASS Relevance score 0.8 Reasoning: This sub-hypothesis test is strongly relevant to the main hypothesis for several reasons: 1. Mechanistic Connection: The test examines LCP2 genetic variants and their association with immune phenotypes that are known to be IL-2 dependent, providing a direct mechanistic link to the main hypothesis about LCP2's role in IL-2 regulation. 2. Evidence Strength: By using genetic variants (loss-of-function and missense) in LCP2 and examining their association with IL-2-dependent phenotypes, the test provides strong genetic evidence for the functional relationship between LCP2 and IL-2 production. 3. Predictive Value: If LCP2 truly regulates IL-2 production, we would expect LCP2 variants to be associated with autoimmune and T-cell related phenotypes, making this test highly predictive. The use of GeneBASS database adds statistical rigor to the analysis. The score is 0.8 rather than 1.0 because while it provides strong genetic evidence, it is still an associative study rather than a direct mechanistic demonstration of LCP2's regulation of IL-2 production. ## Experiment Execution Agent ``` Looking at the provided datasets, I notice that 'df_genebass_pLoF_filtered' and ' df_genebass_missense_LC_filtered' would be the most relevant for testing associations between LCP2 genetic variants and autoimmune/T-cell phenotypes. Let me first examine these datasets to see if they contain LCP2-related entries. ---------------------------- Action: python_repl_ast Action Input: # Check for LCP2 variants in pLoF dataset lcp2_plof = df_genebass_pLoF_filtered[df_genebass_pLoF_filtered['gene'] == 'LCP2'] print("Number of pLoF variants:", len(lcp2_plof)) # Check for LCP2 variants in missense dataset lcp2_missense = df_genebass_missense_LC_filtered[df_genebass_missense_LC_filtered['gene'] == 'LCP2'] print("Number of missense variants:", len(lcp2_missense)) ``` ## Observation ``` Number of pLoF variants: 1423 Number of missense variants: 1423 ``` ## Experiment Execution Agent ``` Experiment Execution Agent I noticed that both 'pLoF' and 'missense' datasets have exactly the same number of entries, which seems unusual. Let me examine the data more closely to verify its quality and look specifically at autoimmune and T-cell related phenotypes. ----------------------------------- Action: python_repl_ast Action Input: # Looking at unique phenotypes for LCP2 variants and filtering for immune-related ones immune_keywords = ['immune', 'autoimmune', 'T cell', 'T-cell', 'lymphocyte', 'IL-2'] # Get unique phenotypes for LCP2 lcp2_plof_phenotypes = lcp2_plof['pheno_description'].unique() lcp2_missense_phenotypes = lcp2_missense['pheno_description'].unique() # Filter immune-related phenotypes immune_plof = lcp2_plof[lcp2_plof['pheno_description'].str.contains('|'.join(immune_keywords), case=False, na=False)] print("Immune-related pLoF associations:") print(immune_plof[['pheno_description', 'Pvalue', 'Pvalue_Burden', 'Pvalue_SKAT']].head()) ``` ## Observation | Immune-related pLoF associations: | Immune-related pLoF associations: | Immune-related pLoF associations: | Immune-related pLoF associations: | Immune-related pLoF associations: | |-------------------------------------|-------------------------------------|-------------------------------------|-------------------------------------|-------------------------------------| | | pheno_description | Pvalue | Pvalue_Burden | Pvalue_SKAT | | 581 | Lymphocyte percentage | 0.000076 | 0.000067 | 0.000937 | | 581 | Lymphocyte count | 0.086099 | 0.054341 | 0.114530 | | 575 | Autoimmune thyroiditis | 0.537810 | 0.357880 | 0.818120 | ## Experiment Execution Agent ``` Let me continue analyzing the significant immune-related phenotype associations. ---------------------------- Action: python_repl_ast Action Input: # Examine the most significant association in more detail significant_result = immune_plof[immune_plof['Pvalue'] == immune_plof['Pvalue'].min()] print("Most significant immune-related association:") print(significant_result[['gene', 'pheno_description', 'Pvalue', 'Pvalue_Burden', 'Pvalue_SKAT', 'BETA_Burden', 'SE_Burden']].to_string(index=False)) ``` ## Observation ``` Most significant immune-related association: gene pheno_description Pvalue Pvalue_Burden Pvalue_SKAT BETA_Burden SE_Burden LCP2 Lymphocyte percentage 0.000076 0.000067 0.000937 0.014855 0.003725 ``` ``` 0.003725 ``` ## Experiment Execution Agent ``` ``` ## Observation ``` Original p-value: 6.66e-05 Number of tested phenotypes: 1414 Bonferroni corrected p-value: 9.42e-02 ``` ## Experiment Execution Agent ``` I notice that after applying the Bonferroni correction for multiple testing, the p-value becomes 9.42e-02, which is above the conventional significance threshold of 0.05. However, let me verify two important aspects: 1. The direction of effect (BETA) to ensure it aligns with our biological expectations 2. The sample size and test power ----------------------------- Action: python_repl_ast Action Input: # Look at the complete statistics for our top finding print("Full statistics for top association:") print(significant_result[['pheno_description', 'BETA_Burden', 'SE_Burden']].to_string(index=False)) # Calculate the effect size (Cohen's d) ``` ``` import numpy as np cohens_d = significant_result['BETA_Burden'].iloc[0] / significant_result['SE_Burden'] '.iloc[0]' print(f"\nEffect size (t-statistic): {cohens_d:.3f}") ``` ## Observation ``` Full statistics for top association: pheno_description BETA_Burden SE_Burden SE_Burden Lymphocyte percentage 0.014855 0.0037 Effect size (t-statistic): 3.988 ``` ## Experiment Execution Agent Let me summarize the findings before making the final conclusion: 1. We found associations between LCP2 variants and immune-related phenotypes, particularly with lymphocyte percentage 2. The strongest association showed: -Minimum p-value = 6.66e-05 (uncorrected) -Bonferroni corrected p-value = 9.42e-02 -Positive effect (BETA = 0.014855) -Strong effect size (t-statistic 4.0) However, after multiple testing correction, we technically cannot reject the null hypothesis at the conventional =0.05 level. Despite the strong effect size and biologically plausible direction of effect, we must maintain statistical rigor. Final Answer: Cannot reject the null hypothesis. The association between LCP2 variants and immune phenotypes, while suggestive, does not reach genome-wide significance after multiple testing correction (p-value = 9.42e-02). ## Sequential Error Control ``` ====================================== Sequential Testing ====================================== List of p-values: [1.0, 1.0, 0.0942] Summarized sequential statistics: 0.40727200778192163 Sequential test result: insufficient evidence - CONTINUE ============= Testing decision is insufficient evidence - CONTINUE ============= ``` ## Experiment Design Agent ## Round 4 ``` Round 4 Let me propose an initial falsification test: Name: LCP2 eQTL-IL2 Regulatory Region Test Test description: Test if genetic variants that affect LCP2 expression (eQTLs) are enriched in regulatory regions near the IL2 gene, which would suggest direct regulatory relationship. Null sub-hypothesis h_0: Genetic variants that are eQTLs for LCP2 are not significantly enriched in regulatory regions near IL2 gene compared to background. Alternative sub-hypothesis h_1: Genetic variants that are eQTLs for LCP2 are significantly enriched in regulatory regions near IL2 gene compared to background. Let me critique this proposal: ``` ``` SE_Burden 0.003725 ``` - Critic: 1. Causality Direction: If the main hypothesis (LCP2 regulates IL2) is null, this test might still show significance if there's a third factor affecting both genes, or if IL2 regulates LCP2. Therefore, the test doesn't fully satisfy the crucial property that null main hypothesis implies null sub-hypothesis. 2. Data Availability: Looking at the provided data sources, while we have eQTL data from UKBB (df\_eqtl\_ukbb), we don't have comprehensive regulatory region annotations or IL2 genomic location data to properly implement this test. 3. Non-redundancy: While this test examines a different aspect than previous tests ( genetic regulation vs. expression correlation or protein interactions), it may partially overlap with the variant-phenotype associations tested in Round 3. Reflection: We need a test that: 1. Has stronger directional causality aligned with the main hypothesis 2. Can be clearly implemented with available data 3. Provides unique evidence Let me propose a refined test: Name: LCP2 Immune Cell-Specific Expression Test Test description: Test if LCP2 expression is specifically elevated in immune cell types where IL2 production is known to occur (e.g., T cells) compared to other tissues using GTEx tissue expression data. Null sub-hypothesis h\_0: LCP2 expression in immune tissues is not significantly higher than in non-immune tissues. Alternative sub-hypothesis h\_1: LCP2 expression in immune tissues is significantly higher than in non-immune tissues. Critic of refined version: 1. Causality: If the main hypothesis is null, this test would also be null because if LCP2 doesn't regulate IL2, there's no requirement for it to be specifically expressed in IL2-producing cells. 2. Data Feasibility: We can implement this using df\_gtex\_tissue\_gene\_tpm data. 3. Non-redundancy: This test provides unique tissue-specific evidence different from previous tests. 4. Biological Relevance: This test aligns with the biological context of IL2 regulation. This refined test appears to meet all our criteria. ## Relevance Checker ``` Relevance Checker: Proposed falsification test passes relevance check: Proposal: Main hypothesis: Gene LCP2 regulates the production of Interleukin-2 (IL -2). Falsification Test name: LCP2 Immune Cell-Specific Expression Test Falsification Test description: Test if LCP2 expression is specifically elevated in immune cell types where IL2 production is known to occur (e.g., T cells) compared to other tissues using GTEx tissue expression data. Falsification Test Null sub-hypothesis: LCP2 expression in immune tissues is not significantly higher than in non-immune tissues. Falsification Test Alternate sub-hypothesis: LCP2 expression in immune tissues is significantly higher than in non-immune tissues. Relevance score 0.8 Reasoning: This sub-hypothesis test is strongly relevant to the main hypothesis for several reasons: 1. Mechanistic Connection: Since IL-2 is primarily produced by immune cells ( especially T cells), testing LCP2's expression pattern in immune vs non-immune tissues directly addresses a key mechanistic aspect of the hypothesized regulatory relationship. ``` 1. Mechanistic Connection: Since IL-2 is primarily produced by immune cells ( especially T cells), testing LCP2's expression pattern in immune vs tissues directly addresses a key mechanistic aspect of the hypothesized regulatory relationship. 2. Evidence Strength: The test provides strong circumstantial evidence -if LCP2 is specifically expressed in IL-2-producing immune cells, it supports its potential role as a regulator. However, it doesn't directly prove regulation, which is why it's not scored as 1.0. 3. Predictive Value: The test results would significantly impact confidence in the main hypothesis. If LCP2 is not expressed in IL-2-producing cells, it would be difficult to support its role as a direct regulator. Conversely, specific expression in these cells would strengthen the hypothesis. 3. The test doesn't get a 1.0 score because while it provides strong supporting evidence, it doesn't directly test the regulatory relationship -LCP2 could be expressed in these cells but not actually regulate IL-2 production. ## Experiment Execution Agent ``` Experiment Execution Agent I apologize, but I notice a potential issue. Looking at the datasets provided, while we have GTEx tissue expression data (df_gtex_tissue_gene_tpm), we need to ensure that: 1. The LCP2 gene is actually present in the dataset 2. We can properly identify immune vs non-immune tissues 3. We have enough samples for a meaningful statistical comparison Let me first check if LCP2 is in the dataset. Action: python_repl_ast Action Input: # Check unique genes in the dataset unique_genes = df_gtex_tissue_gene_tpm['Gene'].unique() print(f"Number of unique genes: {len(unique_genes)}") print(" First few genes:") print(list(unique_genes)[:5]) # Check if LCP2 is in the dataset lcp2_data = df_gtex_tissue_gene_tpm[df_gtex_tissue_gene_tpm['Gene'] == 'LCP2'] print(" Number of LCP2 entries:", len(lcp2_data)) ------------------------ Action: python_repl_ast Action Input: # Check unique genes in the dataset unique_genes = df_gtex_tissue_gene_tpm['Gene'].unique() print(" First few genes:") print(list(unique_genes)[:5]) # Check if LCP2 is in the dataset lcp2_data = df_gtex_tissue_gene_tpm[df_gtex_tissue_gene_tpm['Gene'] == 'LCP2'] print(" Number of LCP2 entries:", len(lcp2_data)) ``` ## Observation ``` Number of unique genes: 18639 First few genes: ['OR4F5', 'OR4F29', 'OR4F16', 'SAMD11', 'NOC2L'] Number of LCP2 entries: 54 ``` ## Experiment Execution Agent ``` I apologize, but I notice that while we have LCP2 entries in the dataset, we should first verify if we have a clear categorization of immune vs non-immune tissues and ensure we have sufficient data points for both groups before proceeding with the test. -------------------------- Action: python_repl_ast Action Input: # Let's see what tissues we have for LCP2 and their expression values lcp2_data = df_gtex_tissue_gene_tpm[df_gtex_tissue_gene_tpm['Gene'] == 'LCP2'][['Tissue', 'Expression']] print("Number of tissues with LCP2 expression:", len(lcp2_data)) print("\nTissues and their expression values:") print(lcp2_data.sort_values('Expression', ascending=False)) ``` ## Observation Number of tissues with LCP2 expression: 54 Tissues and their expression values: | | Tissue | Expression | |---------------|----------------------------------------------------|-------------------| | 994744 | Whole Blood | 91.342300 | | 882754 | Spleen | 63.694300 | | 677439 | Lung | 35.695100 | | 416129 | Cells - EBV-transformed lymphocytes | 18.756800 | | 24164 | Adipose - Visceral (Omentum) | 18.473700 | | 864089 | Small Intestine - Terminal Ileum | 18.356600 | | 5499 | Adipose - Subcutaneous | 15.769000 | | 733434 | Nerve - Tibial | 9.721680 | | 453459 | Cervix - Endocervix | 9.024360 | | 378799 | Breast - Mammary Tissue | 8.800150 | | 80159 | Artery - Coronary | 8.208850 | | 434794 | Cervix - Ectocervix | 7.951800 | | 808094 | Prostate | 7.538440 | | 117489 | Bladder | 7.335620 | | 61494 | Artery - Aorta | 6.843810 | | 565449 | Fallopian Tube | 6.762460 | | 490789 | Colon - Transverse | 6.693410 | | 938749 | Thyroid | 6.290940 | | 341469 | Brain - Spinal cord (cervical c-1) | 6.247870 | | 976079 | Vagina | 5.778990 | | 957414 | Uterus | 5.630640 | | 42829 | Adrenal Gland | 5.457470 | | 901419 | Stomach | 5.345570 | | 98824 | Artery - Tibial | 5.117030 | | 640109 | Kidney - Medulla | 4.768020 | | 696104 | Minor Salivary Gland | 4.435650 | | 546784 | Esophagus - Muscularis | 4.373440 | | 248144 | Brain - Frontal Cortex (BA9) | 4.343040 | | 509454 | Esophagus - Gastroesophageal Junction | 4.264300 | | 472124 | Colon - Sigmoid | 4.195200 | | 845424 | Skin - Sun Exposed (Lower leg) | 4.153360 | | 360134 | Brain - Substantia nigra | 3.530560 | | 229479 | Brain - Cortex | 3.333310 | | 826759 | Skin - Not Sun Exposed (Suprapubic) | 3.123410 | | 528119 | Esophagus - Mucosa | 3.122660 | | 789429 | Pituitary | 2.997920 | | 154819 | Brain - Anterior cingulate cortex (BA24) | 2.725530 | | 658774 | Liver | 2.721940 | | 752099 | Ovary | 2.544880 | | 621444 | Kidney - Cortex | 2.409530 | | 584114 | Heart - Atrial Appendage | 2.365340 | | 285474 | Brain - Hypothalamus | 2.184320 | | 266809 | Brain - Hippocampus | 2.000190 | | 920084 | Testis | 1.949650 | | 173484 | Brain - Caudate (basal ganglia) | 1.862200 | | 322804 | Brain - Putamen (basal ganglia) | 1.766780 | | 136154 | Brain - Amygdala | 1.766150 | | 602779 | Heart - Left Ventricle | 1.412480 | | 304139 | Brain - Nucleus accumbens (basal ganglia) Pancreas | 1.224700 1.145140 | | 770764 | Muscle - Skeletal | 1.140740 | | 714769 | | | | 210814 192149 | Brain - Cerebellum Brain - Cerebellar Hemisphere | 0.739690 0.654788 | | | Cells - Cultured fibroblasts | 0.022245 | | 397464 | | | ## Experiment Execution Agent ``` I can now perform the falsification test. I'll categorize the tissues into immune and non-immune groups and perform a statistical test to compare LCP2 expression between these groups. The following tissues will be considered immune-related: ``` ``` 1. Whole Blood 2. Spleen 3. Cells-EBV-transformed lymphocytes Action: python_repl_ast Action Input: # Define immune tissues immune_tissues = ['Whole Blood', 'Spleen', 'Cells-EBV-transformed lymphocytes'] # Create two groups for comparison immune_expression = lcp2_data[lcp2_data['Tissue'].isin(immune_tissues)]['Expression'] non_immune_expression = lcp2_data[~lcp2_data['Tissue'].isin(immune_tissues)]['Expression'] # Perform Mann-Whitney U test (non-parametric test since we can't assume normal distribution) from scipy.stats import mannwhitneyu stat, pvalue = mannwhitneyu(immune_expression, non_immune_expression, alternative='greater') print(f"Mann-Whitney U test statistic: {stat}") print(f"P-value: {pvalue:.2e}") print("Median expression in immune tissues:", immune_expression.median()) print("Median expression in non-immune tissues:", non_immune_expression.median()) ``` ## Observation ``` Mann-Whitney U test statistic: 152.0 P-value: 8.06e-05 Median expression in immune tissues: 63.6943 Median expression in non-immune tissues: 4.2643 ``` ## Experiment Execution Agent ``` Based on the statistical analysis, I can now provide a conclusive answer about the falsification test: Final Answer: Falsification test passes. The null hypothesis is rejected with a p-value of 8.06e-05. The analysis shows that LCP2 expression is significantly higher in immune tissues compared to non-immune tissues. The median expression in immune tissues (63.69 TPM) is substantially higher than in non-immune tissues (4.26 TPM), supporting the alternate hypothesis that LCP2 expression is specifically elevated in immune tissues. ``` ## Sequential Error Control ``` ====================================== Sequential Testing ====================================== List of p-values: [1.0, 1.0, 0.0942, 8.06e-05] Summarized sequential statistics: 22.682297641798968 Sequential test result: sufficient evidence - PASS ==================== Testing decision is sufficient evidence - PASS ==================== ``` ## Summarizer ``` Let me summarize the sequential falsification testing results: (1) Main Scientific Hypothesis: Gene LCP2 regulates the production of Interleukin-2 (IL-2) ``` - (2) Result of Sequential Falsification Test: 2. The sequential testing passed with a combined e-value of 22.68, after completing 4 tests. - (3) Reasoning and Analysis: 4. -Expression Correlation Test (Round 1) showed strong correlation between LCP2 and IL-2 expression 5. -Regulatory Network Analysis (Round 2) demonstrated significant physical interactions with IL-2 regulators 6. -Variant-Immune Phenotype Association Test (Round 3) failed to show significant association (p=0.0942 > 0.05) 7. -Immune Cell-Specific Expression Test (Round 4) showed highly significant immune tissue-specific expression (p=8.06e-05) - (4) Conclusion: True - (5) Rationale: The hypothesis is supported based on multiple lines of evidence: - -Strong expression correlation between LCP2 and IL-2 - -Confirmed physical interactions with IL-2 regulatory proteins - -Highly significant immune cell-specific expression pattern - -The combined e-value (22.68) indicates strong cumulative evidence - Although the variant association test didn't reach significance, the other three tests provided strong supporting evidence, and the overall sequential testing procedure passed the predetermined threshold. The multiple complementary approaches testing different aspects of the relationship between LCP2 and IL-2 production collectively support the regulatory relationship. Figure 11: An example false-positive trajectory from the TargetVal-IL2 benchmark. For clarity, we focus on the specific round where the agent makes the critical error. ## Problem Statement ``` Figure 11: An example false-positive trajectory from the TargetVal-IL2 benchmark. For clarity, we focus on the specific round where the agent makes the critical error. Problem Statement Main Hypothesis Gene RAB39A regulates the production of Interleukin-2 (IL-2). df_gene_info: {'gene_id': 'ENSG00000228037', 'transcript_id': 'ENST00000424215', 'chr': '1', 'gene_start': 2581560, 'gene_end': 2584533, 'strand': 1, 'transcript_start': 2581560, 'transcript_end': 2584533, 'tss': 2581560, 'transcript_is_canonical': 1.0, 'gene_name': nan, 'percentage_gene_gc_content': 51.11, 'gene_type': 'lncRNA'} df_genetic_interaction: {'interaction_id': 206363, 'gene_a_id': 'YCR011C', 'gene_b_id': 'YCL025C', 'experimental_system_type': 'genetic', 'pubmed_id': 'PUBMED:16269340', 'organism_id_a': 559292, 'organism_id_b': 559292, 'throughput_type': 'High Throughput', 'experimental_score': '-5.6431'} # 19 more datasets omitted for presentation =================================================================== Summary of previous rounds: ------- Round 1 -------- Falsification Test: Main hypothesis: Gene RAB39A regulates the production of Interleukin-2 (IL-2). Falsification Test name: RAB39A Genetic Impact on IL-2 Related Phenotypes Test Falsification Test description: Test if genetic variations in RAB39A are significantly associated with IL-2 related immune phenotypes using GeneBASS data Falsification Test Null sub-hypothesis: Genetic variations (pLoF, missense, synonymous) in RAB39A show no significant association with IL-2 related immune phenotypes Falsification Test Alternate sub-hypothesis: Genetic variations in RAB39A show significant association with IL-2 related immune phenotypes test statistics: 0.0543 ------- Round 2 -------- Falsification Test: Main hypothesis: Gene RAB39A regulates the production of Interleukin-2 (IL-2). Falsification Test name: RAB39A Regulatory Complex Formation Test Falsification Test description: Test if RAB39A forms physical interactions specifically with known transcriptional regulators and signaling proteins in the IL-2 production pathway using protein-protein interaction data. Falsification Test Null sub-hypothesis: RAB39A shows no significant physical interactions with transcriptional regulators and signaling proteins known to control IL-2 production. Falsification Test Alternate sub-hypothesis: RAB39A shows significant physical interactions with transcriptional regulators and signaling proteins known to control IL-2 production. test statistics: 1.0 ``` ## Experiment Design Agent ## Round 3 ``` Round 3 Let me propose an initial falsification test: Initial Proposal: (1) Name: RAB39A Tissue-Specific Co-Expression Pattern Test (2) Test description: Test if RAB39A shows significantly higher expression in immune system tissues where IL-2 is known to be predominantly produced (like T cells and lymphoid tissues) compared to other tissues using GTEx data (3) Null sub-hypothesis h_0: RAB39A expression levels in immune system tissues are not significantly different from other tissues ``` - (4) Alternate sub-hypothesis h\_1: RAB39A shows significantly higher expression in immune system tissues compared to other tissues Let me critique this proposal: ## Critic: 1. Main hypothesis nullity implication: If RAB39A does not regulate IL-2 production, it could still be highly expressed in immune tissues for other functions. Therefore, this test doesn't necessarily become null when the main hypothesis is null. This violates the key requirement. 2. Data availability: While we have GTEx tissue expression data ( df\_gtex\_tissue\_gene\_tpm), the tissue categories might not be granular enough to specifically identify immune tissues. 3. Redundancy: This test partially overlaps with the failed "RAB39A-IL2 Expression Correlation Test" as both use expression patterns. ## Reflection: We need a test that: 1. Has stronger logical connection to the regulatory relationship 2. Becomes null when the main hypothesis is null 3. Uses different data types than previous tests 4. Is answerable with available data ## New Proposal: - (1) Name: RAB39A Genetic Variant Impact on IL-2 QTL Test - (2) Test description: Test if genetic variants near RAB39A are significant QTLs for IL-2 related immune traits in the UK Biobank eQTL data - (3) Null sub-hypothesis h\_0: Genetic variants near RAB39A show no significant QTL associations with IL-2 related immune phenotypes - (4) Alternate sub-hypothesis h\_1: Genetic variants near RAB39A show significant QTL associations with IL-2 related immune phenotypes ## Critic Round 2: 1. Main hypothesis nullity: If RAB39A doesn't regulate IL-2, genetic variants affecting RAB39A should indeed show no QTL effects on IL-2 related traits. This satisfies the nullity requirement. 2. Data availability: We have eQTL data from UK Biobank (df\_eqtl\_ukbb) and variant information (df\_variant\_table). 3. Non-redundancy: This test uses different data types (QTL) compared to previous tests that used GeneBASS and protein interactions. 4. Implementation feasibility: The test can be implemented using the provided data sources. This refined test appears to meet all requirements. ## Relevance Checker - Proposed falsification test passes relevance check: Proposal: Main hypothesis: Gene RAB39A regulates the production of Interleukin-2 ( IL-2). Falsification Test name: RAB39A Genetic Variant Impact on IL-2 QTL Test Falsification Test description: Test if genetic variants near RAB39A are significant QTLs for IL-2 related immune traits in the UK Biobank eQTL data Falsification Test Null sub-hypothesis: Genetic variants near RAB39A show no significant QTL associations with IL-2 related immune phenotypes Falsification Test Alternate sub-hypothesis: Genetic variants near RAB39A show significant QTL associations with IL-2 related immune phenotypes Relevance score 0.8 Reasoning: This QTL test is strongly relevant to the main hypothesis for several reasons: 1. Direct genetic link: The test examines genetic variants specifically near RAB39A and their relationship to IL-2 related traits, which directly addresses the proposed regulatory relationship in the main hypothesis. population 2. Population-level evidence: Using UK Biobank eQTL data provides robust, -scale evidence that could substantiate a regulatory relationship between RAB39A and IL-2. 3. Mechanistic insight: QTL associations would suggest a genetic basis for the regulatory relationship, though it doesn't fully elucidate the exact molecular mechanism. 4. High predictive value: If significant QTL associations are found, it would strongly support (though not definitively prove) the main hypothesis. Conversely , absence of associations would significantly challenge it. 3. The score is 0.8 rather than 1.0 because while the test provides strong evidence, it 's correlative rather than demonstrating direct causation or the specific molecular mechanism of regulation. ## Experiment Execution Agent ``` Experiment Execution Agent Let me help analyze this falsification test. We need to test if genetic variants near RAB39A show significant QTL associations with IL-2 related immune phenotypes. First, I should check the eQTL data from UK Biobank for RAB39A-related variants. Action: python_repl_ast Action Input: # Filter eQTL data for RAB39A import pandas as pd # First let's check if RAB39A appears in the eQTL dataset rab39a_eqtls = df_eqtl_ukbb[df_eqtl_ukbb['gene_name'] == 'RAB39A'] print("Number of RAB39A eQTLs:", len(rab39a_eqtls)) if len(rab39a_eqtls) > 0: print("\nRAB39A eQTLs:") print(rab39a_eqtls) ``` ## Observation ``` Number of RAB39A eQTLs: 4 rs_id gene_id cell_type_name qtl_score gene_name 681397 rs11603234 ENSG00000179331 BLUEPRINT-NEUTROPHIL 12.551138 RAB39A 1260556 rs80292023 ENSG00000179331 BLUEPRINT-NEUTROPHIL 6.144924 RAB39A 1669183 rs7125285 ENSG00000179331 BLUEPRINT-NEUTROPHIL 6.943610 RAB39A 1753601 rs77885862 ENSG00000179331 BLUEPRINT-NEUTROPHIL 6.515021 RAB39A ``` ## Experiment Execution Agent ``` Experiment Execution Agent I noticed that we have some RAB39A eQTL data in neutrophils, which is relevant to immune function. Let's analyze the statistical significance of these QTL associations. ---------------------------- Action: python_repl_ast Action Input: # Convert QTL scores to p-values (QTL scores are -log10(p-value)) import numpy as np def qtl_score_to_pvalue(qtl_score): return 10**(-qtl_score) # Calculate p-values for each QTL rab39a_eqtls['p_value'] = rab39a_eqtls['qtl_score'].apply(qtl_score_to_pvalue) # Get the most significant p-value (minimum) min_pvalue = rab39a_eqtls['p_value'].min() print("Most significant p-value: {:.2e}".format(min_pvalue)) # Print all QTLs with their p-values print("All RAB39A QTLs with p-values:") ``` ``` for _, row in rab39a_eqtls.iterrows(): print(f"SNP: {row['rs_id']}, QTL score: {row['qtl_score']:.2f}, p-value: { qtl_score_to_pvalue(row['qtl_score']):.2e}") ``` ## Observation expected an indented block (<string>, line 16) ## Experiment Execution Agent ``` Experiment Execution Agent Let me fix the code and try again. ------------------------------ Action: python_repl_ast Action Input: # Convert QTL scores to p-values (QTL scores are -log10(p-value)) import numpy as np def qtl_score_to_pvalue(qtl_score): return 10**(-qtl_score) # Calculate p-values for each QTL rab39a_eqtls['p_value'] = rab39a_eqtls['qtl_score'].apply(qtl_score_to_pvalue) # Get the most significant p-value (minimum) min_pvalue = rab39a_eqtls['p_value'].min() print(f"Most significant p-value: {min_pvalue:.2e}") # Sort by p-value rab39a_eqtls_sorted = rab39a_eqtls.sort_values('p_value') print("All RAB39A QTLs sorted by significance:") print(rab39a_eqtls_sorted[['rs_id', 'qtl_score', 'p_value']].to_string()) __observation__ Observation Rab39a_eqtls.sort_values('p_value') print("All RAB39A QTLs sorted by significance:") print(rab39a_eqtls_sorted[['rs_id', 'qtl_score', 'p_value']].to_string()) ``` ## Observation ``` <string>:8: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy Most significant p-value: 2.81e-13 All RAB39A QTLs sorted by significance: rs_id qtl_score p_value 681397 rs11603234 12.551138 2.811010e-13 1669183 rs7125285 6.943610 1.138650e-07 1753601 rs7785862 6.515021 3.054770e-07 1260556 rs80292023 6.144924 7.162680e-07 ``` ## Experiment Execution Agent ``` I notice the analysis shows significant eQTL associations for RAB39A. The most significant variant (rs11603234) has a p-value of 2.81e-13, which is highly significant. This eQTL effect is observed in neutrophils, which are relevant to immune function. Since we have: 1. Multiple independent eQTL signals (4 different variants) 2. These are found in immune cells (neutrophils) 3. The associations are highly significant with the strongest p-value at 2.81e-13 4. The observation is in a relevant tissue type for IL-2 related immune traits ``` We can reject the null hypothesis that genetic variants near RAB39A show no significant QTL associations with IL-2 related immune phenotypes. Final Answer: Falsification test passes. The null hypothesis is rejected with a pvalue of 2.81e-13. Multiple independent variants near RAB39A show significant eQTL effects in immune cells (neutrophils), with the strongest association having a p-value of 2.81e-13. ## Sequential Error Control ``` ====================================== Sequential Testing ====================================== List of p-values: [0.0543, 1.0, 2.81e-13] Summarized sequential statistics: 1011945.0684107154 Sequential test result: sufficient evidence - PASS ==================== Testing decision is sufficient evidence - PASS ==================== ``` ## Summarizer Let me summarize the sequential falsification testing results: - (1) Main Scientific Hypothesis: Gene RAB39A regulates the production of Interleukin-2 - (2) Result of Sequential Falsification Test: 3. The sequential testing passed with a combined e-value of approximately 1.01 x 10ˆ6, after completing 3 tests. - (3) Reasoning and Analysis: 5. -Test 1 (Genetic Impact): Test statistic 0.0543 -showed weak association between RAB39A genetic variations and IL-2 phenotypes 6. -Test 2 (Regulatory Complex): Test statistic 1.0 -demonstrated no significant physical interactions with IL-2 regulatory proteins 7. -Test 3 (QTL Analysis): Test statistic 2.81e-13 -revealed highly significant QTL associations between RAB39A variants and IL-2 traits - (4) Conclusion: True - (5) Rationale: 10. The conclusion is True because: 11. -The sequential testing procedure officially passed with a very large combined evalue (>10ˆ6) 12. -While the first two tests showed modest or no associations, the extremely strong QTL evidence (p=2.81e-13) in Test 3 provided compelling statistical support 13. -The combined evidence, particularly driven by the strong QTL associations, meets the threshold for rejecting the null hypothesis 14. -The sequential testing framework explicitly concluded "sufficient evidence -PASS ", indicating formal rejection of the null hypothesis 15. This conclusion is based strictly on the statistical evidence presented in the sequential testing procedure, which formally rejected the null hypothesis in favor of the alternative hypothesis that RAB39A regulates IL-2 production." (IL-2)