2411.00053

# ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration **Authors**: - Andrew Estornell (ByteDance Research) - &Jean-François Ton (ByteDance Research) - &Yuanshun Yao (Meta GenAI) - &Yang Liu (University of California, Santa Cruz) > Equal contribution. Correspondence to Work done while at ByteDance Research ## Abstract Large language models (LLMs) have demonstrated a remarkable ability to serve as general-purpose tools for various language-based tasks. Recent works have demonstrated that the efficacy of such models can be improved through iterative dialog between multiple models. While these paradigms show promise in improving model efficacy, most works in this area treat collaboration as an emergent behavior, rather than a learned behavior. In doing so, current multi-agent frameworks rely on collaborative behaviors to have been sufficiently trained into off-the-shelf models. To address this limitation, we propose ACC-Collab, an A ctor- C riti c based learning framework to produce a two-agent team (an actor-agent and a critic-agent) specialized in collaboration. We demonstrate that ACC-Collab outperforms SotA multi-agent techniques on a wide array of benchmarks. ## 1 Introduction footnotetext: Code available at https://github.com/LlenRotse/ACC-Collab Recently, large language models (LLMs) have rapidly become a cornerstone in various applications, redefining how we process and generate language at scale (Thirunavukarasu et al., 2023; Hadi et al., 2023; Jiang et al., 2024). Their ability to handle diverse tasks, from translation (Zhu et al., 2024; Otter et al., 2020) to answering complex questions (Zhang et al., 2024; Hao et al., 2024; Havrilla et al., 2024), has attracted the attention of both industry as well as academia. However, despite these advancements, LLMs still exhibit notable weaknesses, particularly when it comes to answering factual questions and reasoning (Tonmoy et al., 2024; Rawte et al., 2023; Huang et al., 2023). To address these limitations, several techniques have been proposed, such as Chain-of-Thought (CoT) prompting (Wei et al., 2022), Self-Reflection (Ji et al., 2023; Shinn et al., 2023), and Multi-Agent Debate (MAD) (Du et al., 2023), to name a few. These approaches aim to improve the reasoning abilities of LLMs by guiding them toward more accurate answers through structured thinking or discourse. However, the majority of these techniques do not involve training the model specifically for these tasks but instead rely on zero-shot or few-shot capabilities. Similar to most multi-agent paradigms, MAD approaches make use of off-the-shelf general-purpose LLMs, which are not trained to collaborate. Such approaches rely on collaboration as an emergent, rather than a learned, behavior. While, in some cases, these emergent behaviors are sufficient, the question remains: Can these methods be improved by imbuing models directly with collaborative abilities? To answer this, we propose training teams of LLMs to solve tasks collaboratively. A particularly relevant work is DebateGPT (Subramaniam et al., 2024), which employs debate as a mechanism to generate higher-quality fine-tuning data. Unlike our approach, which optimizes LLMs for multi-round collaborative problem-solving, their method focuses on using debate to enhance training data for a single model that produces individual responses. In this paper, we propose a novel framework A ctor- C riti c Collaboration (ACC-Collab) which jointly trains a two-agent team to collaboratively solve problems through iterative conversation; this team consists of an actor-agent, responsible for providing answers for a given task, and a critic-agent, responsible for assisting the actor-agent with feedback on its answers. In our training pipeline, we introduce a novel off-policy learning scheme called ”Guided-Collaboration” to generate high-quality multi-turn training data to enhance the actor’s and critic’s performance on challenging tasks. To summarize, our contributions are as follows: - We are the first to propose a framework for jointly training a team of LLM agents (Actor-Critic) within the context of collaborative problem solving. - We introduce a novel data generation scheme, “Guided Collaboration Trajectories”, which enables the efficient creation of high-quality training data for both the actor and critic roles. - Our extensive experiments demonstrate that our method, ACC-Collab, significantly outperforms existing state-of-the-art approaches. ## 2 Related Work Our research is closely related to the emerging field of multi-agent deliberation, sometimes called Multi-Agent Debate (MAD), which examines how to use groups of models to solve tasks through iterative discussion Chan et al. (2023); Liang et al. (2023); Du et al. (2023); Li et al. (2023c); Khan et al. (2024); Michael et al. (2023); Rasal (2024); Pham et al. (2023); Abdelnabi et al. (2023); Hong et al. (2023); Irving et al. (2018); Li et al. (2023b; d; 2024a); Wang et al. (2023a); Zhang et al. (2023). Many of these works find that language models have naturally collaborative abilities Singhal et al. (2023); Du et al. (2023); Chan et al. (2023), while others have noted that the collaborative ability of off-the-shelf models can be quite limited Wang et al. (2024); Smit et al. . Current approaches to multi-agent deliberation can be broadly cast into two main categories: those that modify model prompts and responses during the discussion Liang et al. (2023); Khan et al. (2024); Rasal (2024); Feng et al. (2024); Yang et al. (2024), and those that modify the structure of the deliberation process Li et al. (2023a); Hong et al. (2023); Liu et al. (2023); Li et al. (2024c); Wang et al. (2023b); Wu et al. (2023); Chen et al. (2023); Chang (2024b). Importantly, both categories use off-the-shelf language models (which have not been trained to collaborate) and work by modifying either the inputs or outputs of these models. Deviating from this line of work, we aim to specifically train a team of models to collaboratively solve tasks. Two works of particular note are that of Subramaniam et al. (2024), which proposes to use debate data to fine-tune models, and Li et al. (2024b), which trains models for adversarial debate. In the former, debate is used to generate higher-quality fine-tuning data and is not used at inference time; differing from this work, we train models directly to collaborate and use multi-agent discussion both during training and inference. In the latter, models are trained to be effective arguers rather than collaborators, i.e., models are trained to give conceiving arguments such that they can win a debate against other LLMs. Differing from this work, we train models to collaboratively solve tasks. In the context of multi-agent deliberation, the concept of divergent opinions is highly relevant to our method. Several approaches to multi-agent deliberation aim to control the level of disagreement among the agents Liang et al. (2023); Khan et al. (2024); Chang (2024a). Often, these works dynamically increase disagreement to prevent early convergence of deliberation. In our study, we leverage divergent opinions to generate high-quality training data. In particular, we have agents change their opinion during the discussion and measure whether or not that change increases or decreases the likelihood that the agents’ discussion converges to a correct answer. Using this signal we can then asses the value of a given training example for training the models. Also closely related to our work are paradigms that aim to use self-generated data to improve model performance, often in the context of reasoning or chain of thought Trung et al. (2024); Huang et al. (2024); Xiong et al. (2024); Chen et al. (2024); Pang et al. (2024b). Similar to this line of research, we make use of model generations as training data. However, we are the first work to use such data in the context of multiple models debating collaboratively to solve a given task. ## 3 Preliminaries and notation In this section, we formalize multi-agent collaboration between an Actor (an agent that provides answers) and a Critic (an agent that provides feedback to the actor) while also introducing the notation that will be used throughout the remainder of the paper. Let $(x,y)∼D$ be a task-answer pair source from a distribution of tasks and answers $D$ . For a given task $x$ , two agents – an actor agent responsible for providing answers and a critic agent responsible for providing feedback and assistance to the actor agent – engage in an iterative discussion over $T$ rounds, to correctly infer the answer $y$ . Let $θ_a$ and $θ_c$ be the parameters of actor and critic agent, respectively. The iterative discussion between these two agents is as follows: 1. At round $t=0$ a task $x$ is given to the actor $θ_a$ who provides an initial response $z_a^(0)$ . 1. Next, still at round $t=0$ , the critic $θ_c$ views task $x$ and $z_a^(0)$ , then provides feedback $z_c^(0)$ . 1. For each round $t>0$ , the actor views the task $x$ , its own previous response $z_a^(t-1)$ and the critic’s feedback $z_c^(t-1)$ , then provides an updated response $z_a^(t)$ . 1. After the actor’s new response $z_a^(t)$ , the critic provides the feedback $z_c^(t)$ based on $z_a^(t)$ . The accuracy of this procedure is measured via the correctness of the actor’s final response, i.e., $I\big{[}ζ(z_a^(T))=y\big{]}$ . Where $ζ$ is a function that extracts answers from text-based responses. For example if $z_a^(T)=$ “The sky is blue”, then $ζ(z_a^(T))=$ “blue”. With this notation and formalization of multi-agent collaboration, we introduce our framework for training actor-critic teams. <details> <summary>extracted/6248978/plots/mainMethod_v4.png Details</summary>  ### Visual Description ## Diagram: Actor-Critic Preference-Based Training Process ### Overview The image is a technical flowchart illustrating a machine learning training process involving two agents: an "Actor" and a "Critic." The process generates preference data by comparing the quality of different trajectories (sequences of states/actions) produced under three distinct guidance conditions. This data is then used to train models, with the Critic receiving feedback from the training outcomes. ### Components/Axes The diagram is structured as a left-to-right flowchart with a feedback loop. Key components are labeled with text and mathematical notation. **Left Side (Agents & Initial State):** * **Actor:** Represented by an orange robot icon and the label "Actor" in orange text. Positioned at the top-left. * **Critic:** Represented by a blue robot icon and the label "Critic" in blue text. Positioned at the bottom-left. * **Initial State:** Two stacked speech bubbles represent the state at time `t-1`. * Top bubble (orange outline): Contains the text `z_a^{(t-1)}`. * Bottom bubble (blue outline): Contains the text `z_c^{(t-1)}`. **Central Process (Three Parallel Paths):** Three arrows originate from the initial state bubbles, each labeled with a process name and leading to a pair of output bubbles at time `t`. 1. **Top Path:** Labeled "Actor Guided Towards y". Leads to: * A green-outlined bubble containing `z_a^{(t)}`. * A blue-outlined bubble containing `z_c^{(t)}`. 2. **Middle Path:** Labeled "Natural Debate". Leads to: * An orange-outlined bubble containing `z_a^{(t)}`. * A blue-outlined bubble containing `z_c^{(t)}`. 3. **Bottom Path:** Labeled "Actor Guided Away From y". Leads to: * A red-outlined bubble containing `z_a^{(t)}`. * A blue-outlined bubble containing `z_c^{(t)}`. **Comparison & Decision Nodes:** The outputs from the three paths feed into two circular comparison nodes. * **Top Node:** Contains the symbol `Δ_y` (Delta subscript y). It receives inputs from the top path's green `z_a^{(t)}` and the middle path's orange `z_a^{(t)}`. * **Bottom Node:** Contains the symbol `Δ_{!y}` (Delta subscript exclamation-mark y). It receives inputs from the middle path's orange `z_a^{(t)}` and the bottom path's red `z_a^{(t)}`. * **Label:** Below these nodes is the text "Relative Quality of Trajectory". **Data Storage & Training:** * **Decision Logic:** Arrows from the comparison nodes lead to a database icon. The arrows are labeled with conditional statements: * From `Δ_y`: "if `Δ_y ≥ ε`" (if Delta y is greater than or equal to epsilon). * From `Δ_{!y}`: "elif `Δ_{!y} ≥ ε`" (else if Delta not-y is greater than or equal to epsilon). * **Database:** A cylinder icon labeled "Preference Data". Inside, it contains two pairs of notations: * `(z_a^{(t)}, z_a^{(t)})` (The first `z_a` is in orange, the second in green). * `(z_a^{(t)}, z_a^{(t)})` (The first `z_a` is in orange, the second in red). * **Training Output:** An arrow points from the database to the right, where two robot icons (orange and blue) are shown transforming into larger versions. This section is labeled "Train Models". **Feedback Loop:** A thick black arrow originates from the "Train Models" section, travels along the bottom of the diagram, and points back to the "Critic" icon on the left, indicating a feedback mechanism. ### Detailed Analysis The diagram details a method for generating comparative preference data for training AI models. 1. **Trajectory Generation:** Starting from a shared previous state (`z_a^{(t-1)}`, `z_c^{(t-1)}`), three different trajectories are generated for the next time step (`t`): * One where the Actor is guided toward a target `y`. * One resulting from a "Natural Debate" (presumably unguided or standard interaction). * One where the Actor is guided away from target `y`. The Critic's state (`z_c^{(t)}`) is generated alongside the Actor's (`z_a^{(t)}`) in each case. 2. **Quality Comparison:** The system compares the Actor's states from different trajectories: * `Δ_y` compares the "guided towards y" Actor state (green) with the "natural debate" Actor state (orange). * `Δ_{!y}` compares the "natural debate" Actor state (orange) with the "guided away from y" Actor state (red). The label "Relative Quality of Trajectory" indicates these deltas measure a quality difference. 3. **Preference Data Creation:** If the quality difference (`Δ`) exceeds a threshold `ε` (epsilon), the pair of compared Actor states is stored as a preference datum in the database. The notation in the database shows the "natural debate" state (orange) is paired with either the "guided towards" state (green) or the "guided away" state (red), implying a preference ordering. 4. **Model Training & Feedback:** The collected preference data is used to "Train Models." The updated models (represented by larger robot icons) then provide feedback to the Critic, closing the loop and presumably improving its future evaluations. ### Key Observations * **Color Coding is Critical:** The outline color of the `z_a^{(t)}` bubbles (green, orange, red) is essential for tracking which trajectory they came from and how they are paired in the database. * **Conditional Logic:** Data is only stored if the relative quality difference is significant (≥ ε), filtering out trivial comparisons. * **Asymmetric Roles:** The Actor generates the primary trajectories (`z_a`), while the Critic's state (`z_c`) is generated in parallel but is not directly compared. The Critic's main role appears to be in the feedback loop after training. * **Feedback Loop:** The system is iterative; training outputs influence the Critic, which will affect future trajectory evaluations. ### Interpretation This diagram depicts a sophisticated **preference-based reinforcement learning or alignment training scheme**. The core idea is to create a dataset of *comparative preferences* rather than absolute rewards. * **What it demonstrates:** The process systematically generates "good" (guided toward y) and "bad" (guided away from y) examples relative to a baseline (natural debate). By comparing these, it learns what makes a trajectory preferable with respect to the target `y`. The threshold `ε` ensures only clear preferences are learned. * **How elements relate:** The Actor is the policy being trained. The Critic is likely a value function or reward model that is being improved via the feedback loop. The "Natural Debate" path serves as a crucial control or reference point. The database of preference pairs `(winner, loser)` is the key training signal. * **Notable implications:** This method could be used for **Constitutional AI** or **RLHF (Reinforcement Learning from Human Feedback)**, where `y` represents a desired behavior (e.g., helpfulness, harmlessness). The "Actor Guided Away From y" path is particularly interesting, as it actively generates negative examples to contrast against. The feedback to the Critic suggests an **adversarial or co-evolutionary dynamic** where the evaluator (Critic) improves alongside the generator (Actor). </details> Figure 1: ACC-Collab training pipeline, exemplified for the actor. 1) We generate data from both natural deliberation as well as guided deliberation towards and away from the ground truth answer $y$ using the actor and critic. 2) We compute the relative quality of each trajectory based on the expected quality difference $Δ_y,Δ_!y$ w.r.t. to the natural response. 3) We store all high-quality pairwise data in our database and train the actor agent. 4) We alternate this procedure for the actor and critic. See Figure 5 of the supplement for the corresponding procedure applied to the critic. ## 4 Methodology In this section, we outline our procedure for training a two-agent team, consisting of an actor agent $f_θ_{a}$ (responsible for providing answers to a given task $x$ ) and a critic agent $f_θ_{c}$ (responsible for providing feedback and assistance to the actor). At inference time, the two trained agents engage in iterative discussion to solve a given task $x$ , generating the final response $z_a^(T)$ . ### 4.1 An Actor-Critic Collaboration Framework Building upon our established notation from the previous section and the general actor-critic framework, we formally define our optimization objective as follows For clarity, we note that the term $\arg\max\limits_θ_{a}\max\limits_θ_{b}$ captures the solution for both parameters $θ_a,θ_c$ in the corresponding bi-level max-max optimization. Here $θ_c$ is in $z_c^(T-1)=f_θ_{c}\big{(}x,z_a^(T-1)\big{)}$ : $$ \displaystyleθ_a^*, θ_c^*=\arg\max_θ_{a}\max_θ _{c}E_(x,y)∼ D\bigg{[}ζ\bigg{(}\underbrace{f_θ_{a} \big{(}x,z_a^(T-1),z_c^(T-1)\big{)}}_actor's final response~{ z_a^(T)}\bigg{)}=y\bigg{]} \tag{1} $$ Intuitively, Eq. 1 aims to simultaneously optimize the actor’s parameters $θ_a$ and the critic’s parameters $θ_c$ , ensuring that the actor’s final output at iteration $T$ matches the correct answer $y$ . In other words, we optimize the accuracy of the actor’s response at time $T$ , namely $$ z_a^(T)=f_θ_{a}≤ft(x,z_a^(T-1),z_c^(T-1)\right), $$ where accuracy is measured as $E\big{[}ζ\big{(}z_a^(T)\big{)}=y\big{]}$ . It is important to note that the recursive nature of multi-agent deliberation introduces significant complexity to the optimization process. Each response $z_a^(t)$ depends not only on the actor’s previous output $z_a^(t-1)$ but also on the critic’s previous output $z_c^(t-1)$ . This interaction closely resembles a cooperative dynamic Stackelberg game (Li & Sethi, 2017), where two players engage in hierarchical decision-making over time, leading us to adopt an iterative best-response approach (Fiez et al., 2019). In other words, we first train the critic agent, followed by training the actor to best respond to the critic’s output. We can then update the critic to adapt to the newly trained actor, and so on. More formally, this process works by first fixing $θ_a$ , and solving, $$ \displaystyleθ_c^*=\arg\max_θ_{c}E_(x,y)∼ D\bigg {[}ζ\bigg{(}f_θ_{a}\bigg{(}x,z_a^(T-1),~{}\underbrace{f_θ _{c}\big{(}x,z_a^(T-1)\big{)}}_critic's response~{z_c^(T-1)} \bigg{)}\bigg{)}=y\bigg{]} \tag{2} $$ then fixing $θ_c^*$ from above, we solve $$ \displaystyleθ_a^*=\arg\max_θ_{a}E_(x,y)∼ D\bigg {[}ζ\bigg{(}f_θ_{a}\bigg{(}x,z_a^(T-1),~{}f_θ_{c^*} \big{(}x,z_a^(T-1)\big{)}\bigg{)}\bigg{)}=y\bigg{]} \tag{3} $$ this process then repeats until a desired stopping criteria is reached. In practice, we find that a single iteration is sufficient to produce a high-quality collaborative team. While this alternating scheme allows us to optimize the actor and critic separately, the objectives of each agent still cannot be optimized directly due to the recursive nature of agent responses in this objective; responses at round $T$ depend on those given by the agent at round $t-1$ which themselves depend on the response given at round $t-2$ and so on. To deal with this temporal dependency, we next introduce the concept of Partial Trajectory rewards, which will allow us to capture the signal of each response $z^(t)$ for each $t≤ T$ . ### 4.2 Partial Trajectory Reward To address the inter-round dependencies of the above optimization, we proposed a scheme that allows us to determine the “goodness” of a given response $z^(t)$ (from either the actor or the critic) for any $t≤ T$ . Consider a conversation between the actor and the critic that was paused at time $t$ , i.e., the most recent response is $z^(t)$ . To assess the goodness of $z^(t)$ , one might ask how likely the deliberation procedure will converge to the correct answer $y$ at round $T$ , given that the procedure is already at response $z^(t)$ . Formally, we can define this as $$ \displaystyle r(z^(t),x,y)=E\bigg{[}ζ(z_a^(T))=y|~{}x,z^(t )\bigg{]} \tag{4} $$ Intuitively, the partial reward captures the expectation of arriving at the correct answer $y$ through deliberation starting at round $t$ with generation $z^(t)$ . In practice, $r(z^(t),x,y)$ can be estimated by learning the reward $r$ or by using heuristics such as one-step roll-out, i.e., Monte Carlo estimation. In our experiments, we use one-step roll-out heuristics, i.e. simulating an additional deliberation round multiple times from response $z^(t)$ . The reward $r(z^(t),x,y)$ is set as the average accuracy of these simulations. Empirically, we find this approach effective for generating high-quality training data. We leave learning-based reward functions for future work. Our objective will then be to optimize the parameters of the actor and critic, $θ_a,θ_c$ , so that the responses produced by these agents at each timestep $t$ , namely $z^(t)$ , maximize $r(z^(t),x,y)$ . That is, we optimize the actor and the critic so that at each timestep $t$ , they give a response $z^(t)$ which has a high probability of leading the deliberation to converge to the correct answer at time $T$ . To optimize the objective in Eq. 1, we will utilize preference optimization, a standard technique in LLM training. Using the iterative maximization scheme described above, we first have to gather pairwise preference data for both the actor and the critic. In the following sections, we first detail our process for generating this preference data before delving into the optimization procedure. Data: Actor and critic: $θ_a,θ_c$ , Distribution of tasks $D$ , Reward threshold $ε$ Result: A dataset of trajectories $D$ $D←∅$ /* Set of trajectories to use */ for $(x,y)∼D$ do $z^(0)←$ OneDeliberationRound $(x)$ /* actor and critic responses, i.e. $z=⟨ z_a^(0),z_c^(0)⟩$ */ for $t$ in $[1,T]$ do $z^(t)←$ OneDeliberationRound $(x,z^(t-1))$ /* updated natural responses*/ /* guided-deliberation towards, and away from, correct answer $y$ */ $z_+^(t)←$ OneGuidedDeliberationRound $(x,z^(t-1),y)$ $z_-^(t)←$ OneGuidedDeliberationRound $(x,z^(t-1),!y)$ /* Estimate final round accuracy if deliberation continue from response $z^(t)$ , i.e. $r(z^(t),x,y),~{}r(z_+^(t),x,y),~{}r(z_-^(t),x,y)$ */ $v←$ EstimateFinalAccuracy $(z^t)$ $v_+←$ EstimateFinalAccuracy $(z_+^t)$ $v_-←$ EstimateFinalAccuracy $(z_-^t)$ /* Compute the expected improvement for each trajectory */ /* Save trajectory pairs that result in sufficient accuracy improvement */ if $v_+-v≥ε$ then $D$ .add $\big{(}pos=z_+^(t),neg=z^(t)\big{)}$ end if else if $v-v_-≥ε$ then $D$ .add $\big{(}pos=z^(t),neg=z_-^(t)\big{)}$ end if end for end for Algorithm 1 Trajectory generation and selection ### 4.3 Off-Policy Trajectory Generation In this section, we describe how to generate the preference data needed to optimize the objective in Eq. 1. The classification of a sample as positive or negative is determined by the deliberation trajectory it follows. Specifically, a positive sample for training the actor corresponds to a trajectory likely to lead to the correct answer at round $T$ , while a negative sample corresponds to one that leads to an incorrect answer at round $T$ . Intuitively, we aim to push the actor agent to generate responses that lead to correct answers while reducing responses that are unlikely to do so, thus optimizing for Eq. 3. The same principle applies to the critic when optimizing Eq. 2. With this intuition in mind, we now describe how such data is generated. A deliberation trajectory can be defined as a sequence of responses $⟨ z_a^(0),z_c^(0),z_a^(1),z_c^(1),…,z_a^(T),z_ c^(T)⟩$ for a given task $x$ . A straightforward way to generate preference data would be to generate multiple rollouts at each round and select the trajectories with the highest $r(z^(t),x,y)$ as positive samples and those with the lowest $r(z^(t),x,y)$ as negative samples. This approach could enforce the desired behavior for both the actor and the critic if enough samples are collected. However, this approach is not without its limitations. In particular, if the agent performs poorly on a given dataset, it may be difficult to collect enough positive samples, resulting in low training signals. Additionally, even if the agent performs adequately, generating sufficient responses for both the actor and critic requires significant computational resources, especially to ensure that high $r(z^(t),x,y)$ values are used for positive samples and low values for negative samples. ### 4.4 Guided Collaborative Trajectories To address these limitations and improve efficiency, we propose Guided-Collaborative Trajectories, which steer the deliberation procedure in two opposing directions: one towards, and another away from, the correct. By comparing these guided trajectories with the natural deliberation trajectory, we can assess the relative goodness of each trajectory using an estimation of the reward structure $r$ . Specifically, for task $x$ with answer $y$ , let $z^(t-1)=(z_a^(t-1),z_c^(t-1))$ be the agents’ responses at time $t-1$ . Let $(z_a^(t),z_c^(t))$ be the agents’ natural responses (i.e., without guidance), let $(z_y,a^(t),z_y,c^(t))$ and $(z_!y,a^(t),z_!y,c^(t))$ be the agents responses when guided towards, and away from, supporting answer $y$ respectively. Thus, each guided response is an off-policy generation. In practice, we want guided responses to be different enough from natural responses so that learning the guided responses results in consequential changes to the agent, but not so different that they are challenging to learn; we find that prompt modification is an effective tool for striking this balance. To guide the generations of $(z_y,a^(t),z_y,c^(t))$ and $(z_!y,a^(t),z_!y,c^(t))$ , we will simply provide a correct and wrong target answer in the prompt, respectively - see “Guided Collaborative Trajectory Prompts” in Section C for further details. For each guided response, we consider how influential this response was in altering the accuracy of the final response, i.e., in the case of the actor, we define | | $\displaystyleΔ_y=r(z_y,a^(t),x,y)-r(z_a^(t),x,y) and Δ_!y=r(z_a^(t),x,y)-r(z_!y,a^(t),x,y)$ | | | --- | --- | --- | The terms $Δ_y$ and $Δ_!y$ give the expected accuracy difference if at round $t$ the actor had given response $z_y,a^(t)$ (or $z_!y,a^(t)$ ) instead of response $z_a^(t)$ . Large $Δ_y$ indicates that a one-response difference during the deliberation was sufficient to push the procedure toward the correct answer. Such responses would be desirable for the agent to learn. On the other hand, large values of $Δ_!y$ indicate that a one-response difference easily causes the agents to converge to the incorrect answer; this indicates that the deliberation procedure is particularly fragile at timestep $t$ . With these observations in hand, we use $Δ_y$ and $Δ_!y$ to define positive an negative examples, in particular for a threshold $ε$ , $$ \displaystyle(z_+^(t),z_-^(t))=\begin{cases}(z_y^(t),z^(t))& if ε≤Δ_y=r(z_y,a^(t),x,y)-r(z_a^(t),x,y )\\ (z^(t),z_!y^(t))&if ε≤Δ_!y=r(z_a^(t), x,y)-r(z_!y,a^(t),x,y)\end{cases} \tag{5} $$ if neither value is above the threshold, then the example is thrown out. **Remark 1** *Under Eq. 5, a positive example $z_+^(t)$ can be interpreted as a guided response $z_y^(t)$ which increased the probability of deliberation converging to the correct answer by at least $ε$ , when compared with the natural response $z^(t)$ . Similarly, a negative example $z_-^(t)$ is a guided response $z_y!^(t)$ which decreased the probability of deliberation converging to the answer by at least $ε$ .* Now that we have a procedure for generating high-quality training examples consisting of positive and negative pairs, we next discuss how to use those positive and negative pairs to train both agents. ### 4.5 Learning on Guided Trajectories In order to optimize each objective (Eq. 2 and 3), we use standard preference optimization Direct Preference Optimization (DPO) Rafailov et al. (2024). We choose DPO for its efficiency, but any preference optimization scheme could be used (see section B of the supplement for details on how other preference optimization schemes may be incorporated). Hence, given a preference dataset of positive and negative examples for both the actor and critic agent, of the from $z_-^(t),z_+^(t)$ , the DPO loss is defined as, $$ \displaystyleL_DPO=∑_t=0^TE_(x,y,z_{-^(t ),z_+^(t))∼ D}\bigg{[}\logσ\bigg{(}\frac{π_θ\big{(}z_+ ^(t)|x,z^(t-1)\big{)}}{π_ref\big{(}z_+^(t)|x, z^(t-1)\big{)}}-\frac{π_θ\big{(}z_-^(t)|x,z^ (t-1)\big{)}}{π_ref\big{(}z_-^(t)|x,z^(t-1)\big{)}} \bigg{)}\bigg{]} \tag{6} $$ Where $π_θ$ is the policy induced by parameters $θ_a$ or $θ_c$ and $z^(t-1)$ are the agent’s responses at the previous round (i.e., the responses prior to giving either response $z_-^(t)$ or $z_+^(t)$ ). **Remark 2** *Recall, given our generated preference data, this loss implicitly optimizes the reward $r(z^t,x,y)$ which itself is equivalent to final accuracy (i.e., the quantity being maximized by Eq. 1). By summing across all rounds, we implicitly maximize the probability that each round $t$ yields a response $z^t$ which causes deliberation to converge to the correct answer at time $T$ .* ## 5 Experiments #### Benchmarks To evaluate the efficacy of ACC-Collab we make use of 5 standard benchmark tasks: BoolQ Clark et al. (2019) ${∼}$ 12k yes-no reading comprehension questions, MMLU Hendrycks et al. (2020) ${∼}$ 15k multiple choice questions covering a wide array of subjects and difficulty, BBH Suzgun et al. (2022) ${∼}$ 5k mixed-type questions SCIQ Welbl et al. (2017) ${∼}$ 13k multiple-choice science questions, ARC Chollet (2019) ${∼}$ 7k multiple-choice reasoning-based questions. #### Baselines We compare ACC-Collab to several multi-agent and single-agent baselines. For inference-based methods, we compare to Society of Minds SoM Du et al. (2023), Persona Chan et al. (2023). For training-based methods, we compare to supervised fine-tuning SFT Radford (2018), DebateTune Li et al. (2024b), DebateGPT Subramaniam et al. (2024). We use three different base models: Llama-3-8B-Instruct Llama-3 Dubey et al. (2024), Mistral-7B-Instruct Mistral Jiang et al. (2023), and Gemma-2-2B-Instruct Gemma-2 Team et al. (2024). ### 5.1 ACC-Collab Performance #### Final Answer Accuracy We begin by examining the performance of our method ACC-Collab (a single round of training) and ACC-Collab+ (two rounds of training) In table 1, we see the average accuracy of each method after five rounds of deliberation. Our method attains superior performance compared with baseline methods in most cases. The high efficacy of ACC-Collab relative to the baselines indicates that in most cases, only a single round of training is necessary to produce a high-quality collaborative team. It is worth noting that in some cases, further training rounds may decrease performance (i.e., ACC-Collab+ can have worse performance than ACC-Collab). Hence, we have a hold-out set of tasks to determine whether further training degrades performance. #### Process Accuracy In addition to measuring the correctness of the actor’s final answer (i.e., outcome accuracy), we also analyze the correctness of the team’s reasoning and discussion steps (i.e., process accuracy). Since ground truth is not available for reasoning and discussion steps, we use GPT-4o as an oracle to evaluate whether the agents follow the correct steps to reach the final answer. We provide an outline of the experimental setup and full results in Section A.1 of the supplement. We find that our method improves or maintains the process accuracy of the actor and critic. Llama-3 (Ours) (Ours) SoM (2x) SoM (4x) Persona DebateTune SFT DebateGPT ACC-Collab ACC-Collab+ BoolQ $.812_±.01$ $.811_±.007$ $.781_±.002$ $.775_±.033$ $.798_±.006$ $.815_±.005$ $.887_±.005$ $.894_±.003$ MMLU $.62_±.004$ $.635_±.004$ $.639_±.004$ $.63_±.004$ $.642_±.005$ $.654_±.005$ $.644_±.01$ $.683_±.012$ BBH $.508_±.003$ $.514_±.005$ $.509_±.013$ $.508_±.005$ $.552_±.006$ $.551_±.008$ $.593_±.006$ $.574_±.003$ SCIQ $.925_±.002$ $.923_±.002$ $.925_±.004$ $.924_±.004$ $.925_±.003$ $.932_±.001$ $.952_±.0$ $.948_±.003$ ARC $.874_±.001$ $.874_±.001$ $.87_±.003$ $.871_±.002$ $.879_±.004$ $.876_±.002$ $.881_±.004$ $.869_±.002$ Mistral (Ours) (Ours) SoM (2x) SoM (4x) Persona DebateTune SFT DebateGPT ACC-Collab ACC-Collab+ BoolQ $.801_±.005$ $.798_±.004$ $.831_±.003$ $.83_±.003$ $.84_±.003$ $.848_±.002$ $.877_±.002$ $.893_±.002$ MMLU $.57_±.003$ $.562_±.005$ $.574_±.002$ $.562_±.005$ $.594_±.004$ $.577_±.002$ $.61_±.005$ $.672_±.004$ BBH $.428_±.002$ $.462_±.003$ $.465_±.011$ $.456_±.005$ $.439_±.006$ $.48_±.012$ $.519_±.009$ $.601_±.004$ SCIQ $.856_±.002$ $.856_±.002$ $.86_±.002$ $.863_±.003$ $.858_±.004$ $.871_±.003$ $.902_±.005$ $.905_±.002$ ARC $.824_±.001$ $.823_±.002$ $.827_±.001$ $.834_±.0$ $.825_±.003$ $.822_±.002$ $.843_±.003$ $.856_±.003$ Gemma-2 (Ours) (Ours) SoM (2x) SoM (4x) Persona DebateTune SFT DebateGPT ACC-Collab ACC-Collab+ BoolQ $.75_±.011$ $.759_±.004$ $.716_±.015$ $.767_±.003$ $.783_±.011$ $.812_±.003$ $.84_±.005$ $.845_±.005$ MMLU $.58_±.002$ $.578_±.002$ $.577_±.002$ $.578_±.001$ $.579_±.002$ $.582_±.002$ $.51_±.016$ $.555_±.003$ BBH $.454_±.007$ $.449_±.01$ $.447_±.006$ $.447_±.007$ $.498_±.006$ $.491_±.01$ $.513_±.006$ $.475_±.008$ SCIQ $.903_±.002$ $.903_±.002$ $.908_±.003$ $.903_±.001$ $.913_±.002$ $.914_±.002$ $.918_±.003$ $.909_±.003$ ARC $.841_±.003$ $.843_±.005$ $.847_±.003$ $.847_±.003$ $.848_±.002$ $.851_±.003$ $.852_±.003$ $.849_±.002$ Table 1: Average accuracy (with 95% confidence intervals) after $5$ rounds of deliberation. For each dataset, the highest accuracy is shown in bold. ### 5.2 Performance Increase of Multi-Agent Collaboration <details> <summary>extracted/6248978/plots/new_improve.png Details</summary>  ### Visual Description \n ## Bar Chart: Comparative Performance of Various Methods Across Three Language Models ### Overview The image displays a set of three bar charts arranged horizontally, comparing the percentage improvement of eight different methods or models across three base language models: Llama-3, Mistral, and Gemma-2. The charts share a common legend and y-axis label. ### Components/Axes * **Main Title:** Not present as a single overarching title. Each subplot has its own title: "Llama-3", "Mistral", "Gemma-2". * **Y-Axis:** Labeled "% Improvement". The scale varies by subplot: * Llama-3: 0 to 5 (ticks at 2, 3, 4, 5). * Mistral: 0 to 4 (ticks at 2, 3, 4). * Gemma-2: 0 to 6 (ticks at 2, 3, 4, 5, 6). * **X-Axis:** Represents categorical methods. No individual labels are present on the axis; identification is via the legend. * **Legend:** Located at the bottom of the image, spanning its full width. It defines eight distinct series: 1. **SoM (2x):** Solid blue bar. 2. **SoM (4x):** Solid teal bar. 3. **Persona:** Solid purple bar. 4. **DebateTune:** Solid brown bar. 5. **SFT:** Solid light green bar. 6. **DebateGPT:** Solid dark green bar. 7. **ACC-Collab (Ours):** Orange bar with diagonal hatching (\\). 8. **ACC-Collab + (Ours):** Red bar with diagonal hatching (\\). * **Visual Separator:** A vertical black line appears in each subplot, separating the first six methods from the last two ("ACC-Collab" variants). ### Detailed Analysis **Llama-3 Chart (Left):** * **Trend:** Performance varies significantly. The two "ACC-Collab" methods are the highest, with "ACC-Collab + (Ours)" being the tallest bar overall. * **Approximate Values (from visual estimation):** * SoM (2x): ~2.8% * SoM (4x): ~3.9% * Persona: ~2.5% * DebateTune: ~2.3% * SFT: ~1.2% (lowest) * DebateGPT: ~1.8% * ACC-Collab (Ours): ~3.1% * ACC-Collab + (Ours): ~5.2% (highest) **Mistral Chart (Center):** * **Trend:** The "ACC-Collab" methods again show the highest improvement, with "ACC-Collab (Ours)" slightly leading. The "SoM" methods show very low improvement. * **Approximate Values:** * SoM (2x): ~1.5% * SoM (4x): ~1.3% (lowest) * Persona: ~3.3% * DebateTune: ~3.2% * SFT: ~4.0% * DebateGPT: ~3.6% * ACC-Collab (Ours): ~4.3% (highest) * ACC-Collab + (Ours): ~4.2% **Gemma-2 Chart (Right):** * **Trend:** "ACC-Collab + (Ours)" shows a dramatic and dominant lead over all other methods. Most other methods cluster between 2-3% improvement. * **Approximate Values:** * SoM (2x): ~2.9% * SoM (4x): ~3.1% * Persona: ~2.2% (lowest) * DebateTune: ~3.2% * SFT: ~2.3% * DebateGPT: ~2.8% * ACC-Collab (Ours): ~3.1% * ACC-Collab + (Ours): ~6.3% (highest, by a large margin) ### Key Observations 1. **Consistent Top Performers:** The two methods labeled "(Ours)"—ACC-Collab and ACC-Collab +—are consistently among the top performers across all three base models. 2. **Model-Specific Variability:** The relative performance of other methods (e.g., SFT, Persona) varies notably between models. For instance, SFT is the lowest on Llama-3 but one of the higher non-"Ours" methods on Mistral. 3. **Dramatic Outlier:** The performance of "ACC-Collab + (Ours)" on Gemma-2 is a significant outlier, showing an improvement percentage nearly double that of the next best method on that same model. 4. **SoM Performance:** The "SoM" methods show inconsistent results, performing relatively well on Llama-3 and Gemma-2 but very poorly on Mistral. ### Interpretation This chart presents a comparative evaluation, likely from a research paper, where the authors are benchmarking their proposed methods ("ACC-Collab" and "ACC-Collab +") against several existing techniques (SoM, Persona, DebateTune, SFT, DebateGPT). The data strongly suggests that the authors' methods provide superior or highly competitive improvements in the measured metric (likely accuracy or a related performance score) across diverse underlying language models. The vertical black line visually segregates the authors' contributions from prior work, emphasizing their comparison. The exceptional result for "ACC-Collab + (Ours)" on Gemma-2 indicates a particularly strong synergy between that specific method and the Gemma-2 model architecture or its training data. The variability in other methods' rankings underscores that no single technique is universally optimal; performance is model-dependent. The chart's primary purpose is to demonstrate the efficacy and robustness of the "ACC-Collab" family of methods. </details> Figure 2: Percent improvement in accuracy after five rounds of deliberation, compared to a single round. Percent improvement (Eq. 7) for each method is averaged across all five datasets. #### Average Improvement As noted in Du et al. (2023), the key mechanism behind the success of multi-agent deliberation (or any of its many variants) is that discussion over multiple rounds allows the models to iteratively refine their answers. Thus, a natural question for any iterative multi-agent method is: how much does accuracy improve from the initial round $t=0$ to the final round $t=T$ ? Where $T=4$ in our experiments. To measure this, we look at the percent improvement in model accuracy from round $t=0$ to round $t=4$ calculated as, $$ \displaystyle\frac{acc_4-acc_0}{acc_0} where acc${_t$ is accuracy at round $t$} \tag{7} $$ In Figure 2 we see the average percent improvement for each method, averaged across all 5 datasets. For each of the three base models, ACC-Collab+ has the highest average improvement compared to all other methods. Additionally, the improvement gained by methods such as SoM, SFT or DebateGPT is far less stable than that of ACC-Collab+. In particular, for Mistral, SoM yields nearly no improvement, similarly SFT and DebateGPT offer little improvement when applied to of Llama-3. <details> <summary>extracted/6248978/plots/new_perRoundConf.png Details</summary>  ### Visual Description \n ## Line Chart: Accuracy Comparison of AI Models Over Multiple Rounds ### Overview The image displays three side-by-side line charts comparing the performance (accuracy) of eight different methods or models across five training/evaluation rounds (0 to 4). Each chart corresponds to a different base large language model: Llama-3, Mistral, and Gemma-2. The charts track how the accuracy of each method evolves over successive rounds. ### Components/Axes * **Chart Titles (Top Center):** "Llama-3", "Mistral", "Gemma-2". * **Y-Axis (Left Side):** Labeled "Accuracy". The scale varies per chart: * Llama-3: 0.725 to 0.900 (increments of 0.025). * Mistral: 0.80 to 0.90 (increments of 0.02). * Gemma-2: 0.700 to 0.850 (increments of 0.025). * **X-Axis (Bottom):** Labeled "Round". Markers at 0, 1, 2, 3, 4. * **Legend (Bottom, spanning all charts):** Contains eight entries, each with a specific color and line style: 1. `SoM (2x)`: Blue dashed line (`--`). 2. `SoM (4x)`: Teal dashed line (`--`). 3. `Persona`: Purple dashed line (`--`). 4. `DebateTune`: Brown solid line. 5. `SFT`: Light green solid line. 6. `DebateGPT`: Dark green solid line. 7. `ACC-Collab (Ours)`: Orange solid line with circular markers. 8. `ACC-Collab + (Ours)`: Red solid line with circular markers. * **Data Series:** Each method is represented by a line with a shaded region (likely indicating confidence intervals or standard deviation). ### Detailed Analysis **1. Llama-3 Chart (Left)** * **Trend Verification:** The two "ACC-Collab" methods (red, orange) show a strong upward trend from Round 0 to 1, then plateau. Other methods show more modest gains or fluctuations. * **Data Points (Approximate):** * **ACC-Collab + (Red):** Starts ~0.845 (R0), jumps to ~0.885 (R1), then slowly rises to ~0.895 (R4). * **ACC-Collab (Orange):** Starts ~0.845 (R0), jumps to ~0.885 (R1), then plateaus around ~0.885-0.890. * **DebateGPT (Dark Green):** Starts ~0.805 (R0), rises to ~0.820 (R1), peaks ~0.825 (R2), then slightly declines to ~0.815 (R4). * **SFT (Light Green):** Starts ~0.800 (R0), rises to ~0.805 (R1), then fluctuates around 0.800-0.805. * **SoM (2x) (Blue Dashed):** Starts ~0.775 (R0), rises to ~0.810 (R1), then slowly increases to ~0.815 (R4). * **SoM (4x) (Teal Dashed):** Follows a very similar path to SoM (2x), ending slightly lower at ~0.810 (R4). * **Persona (Purple Dashed):** Starts ~0.775 (R0), rises to ~0.785 (R1), then remains flat around 0.780-0.785. * **DebateTune (Brown):** Starts ~0.765 (R0), rises to ~0.775 (R1), dips to ~0.765 (R2), then recovers to ~0.775 (R4). **2. Mistral Chart (Center)** * **Trend Verification:** Similar pattern where ACC-Collab methods lead. Notably, the SoM methods (blue, teal dashed) show a clear peak at Round 1 followed by a significant decline. * **Data Points (Approximate):** * **ACC-Collab + (Red):** Starts ~0.860 (R0), jumps to ~0.890 (R1), then slowly rises to ~0.895 (R4). * **ACC-Collab (Orange):** Starts ~0.830 (R0), jumps to ~0.870 (R1), then rises to ~0.875 (R4). * **DebateGPT (Dark Green):** Starts ~0.820 (R0), rises to ~0.845 (R1), then plateaus around ~0.850. * **SFT (Light Green):** Starts ~0.815 (R0), rises to ~0.835 (R1), then slowly increases to ~0.840 (R4). * **Persona (Purple Dashed):** Starts ~0.810 (R0), rises to ~0.830 (R1), then plateaus around ~0.830. * **DebateTune (Brown):** Starts ~0.805 (R0), rises to ~0.825 (R1), then slowly increases to ~0.830 (R4). * **SoM (2x) (Blue Dashed):** Starts ~0.805 (R0), peaks at ~0.820 (R1), then declines sharply to ~0.800 (R4). * **SoM (4x) (Teal Dashed):** Follows a nearly identical path to SoM (2x), peaking at ~0.820 (R1) and declining to ~0.800 (R4). **3. Gemma-2 Chart (Right)** * **Trend Verification:** ACC-Collab methods again show strong initial gains. The DebateGPT (dark green) line shows a distinct peak at Round 2 before declining. * **Data Points (Approximate):** * **ACC-Collab (Orange):** Starts ~0.785 (R0), jumps to ~0.845 (R1), then plateaus around ~0.840-0.845. * **ACC-Collab + (Red):** Starts ~0.770 (R0), jumps to ~0.825 (R1), then rises to ~0.845 (R4). * **DebateGPT (Dark Green):** Starts ~0.770 (R0), rises to ~0.810 (R1), peaks at ~0.820 (R2), then declines to ~0.810 (R4). * **SFT (Light Green):** Starts ~0.745 (R0), rises to ~0.780 (R1), then slowly increases to ~0.790 (R4). * **DebateTune (Brown):** Starts ~0.715 (R0), rises to ~0.750 (R1), then slowly increases to ~0.770 (R4). * **SoM (2x) (Blue Dashed):** Starts ~0.720 (R0), rises to ~0.755 (R1), then plateaus around ~0.755-0.760. * **SoM (4x) (Teal Dashed):** Follows a similar path to SoM (2x), ending slightly lower at ~0.755 (R4). * **Persona (Purple Dashed):** Starts ~0.710 (R0), rises to ~0.725 (R1), then slowly declines to ~0.720 (R4). ### Key Observations 1. **Consistent Superiority:** The methods labeled "ACC-Collab (Ours)" and "ACC-Collab + (Ours)" (orange and red lines) consistently achieve the highest accuracy across all three base models (Llama-3, Mistral, Gemma-2) by the final round. 2. **Strong Initial Gains:** All methods show their most significant accuracy improvement between Round 0 and Round 1. 3. **Model-Specific Behavior:** * On **Mistral**, the SoM methods (2x and 4x) exhibit a unique "peak and decline" pattern, performing best at Round 1 but degrading afterward. * On **Gemma-2**, the DebateGPT method shows a clear performance peak at Round 2 before declining. 4. **Performance Clustering:** Methods tend to cluster in performance tiers. The ACC-Collab variants form the top tier. DebateGPT and SFT often form a middle tier. The remaining methods (SoM variants, Persona, DebateTune) generally occupy the lower tier, though their relative ordering varies by model. 5. **Uncertainty Bands:** The shaded regions indicate variability in performance. The bands are generally wider for lower-performing methods (e.g., DebateTune on Llama-3) and narrower for the top-performing ACC-Collab methods, suggesting more consistent results. ### Interpretation This data strongly suggests that the proposed "ACC-Collab" methods (both the standard and "+" variants) are more effective at improving and sustaining accuracy over multiple rounds of debate or collaboration compared to the baseline methods (SoM, Persona, DebateTune, SFT, DebateGPT). The consistent top-tier performance across three distinct LLM architectures (Llama, Mistral, Gemma) indicates the robustness of the ACC-Collab approach. The "peak and decline" behavior of some methods (SoM on Mistral, DebateGPT on Gemma-2) is a critical finding. It suggests these methods may suffer from issues like overfitting to early rounds, catastrophic forgetting, or an inability to leverage later rounds effectively. In contrast, the ACC-Collab methods show stable or improving performance after the initial jump, indicating a more sustainable learning or adaptation process. The charts collectively demonstrate that the choice of collaboration/finetuning method has a larger impact on final accuracy than the choice of base model within this set, as the performance hierarchy of methods is largely preserved across the three different models. The research appears to validate the authors' "ACC-Collab" technique as a superior method for multi-round collaborative AI tasks. </details> <details> <summary>extracted/6248978/plots/new_perRoundSCIQ.png Details</summary>  ### Visual Description ## Line Charts: Performance of Various Methods Across Rounds for Three AI Models ### Overview The image displays three horizontally arranged line charts, each comparing the performance (accuracy) of eight different methods over five rounds (0 to 4) for a specific large language model: Llama-3, Mistral, and Gemma-2. A shared legend is positioned at the bottom of the entire figure. The charts illustrate how accuracy evolves with iterative rounds for each method, with shaded regions indicating confidence intervals or variance. ### Components/Axes * **Chart Titles (Top Center of each subplot):** "Llama-3", "Mistral", "Gemma-2". * **X-Axis (Bottom of each subplot):** Labeled "Round". Markers at integer values: 0, 1, 2, 3, 4. * **Y-Axis (Left of each subplot):** Labeled "Accuracy". The scale differs for each model: * **Llama-3:** Range approximately 0.91 to 0.955. * **Mistral:** Range approximately 0.84 to 0.905. * **Gemma-2:** Range approximately 0.89 to 0.925. * **Legend (Bottom, spanning all charts):** Contains eight entries, each with a distinct line style/color and label: 1. `-- SoM (2x)` (Blue, dashed line) 2. `-- SoM (4x)` (Teal, dashed line) 3. `-- Persona` (Purple, dashed line) 4. `— DebateTune` (Brown, solid line) 5. `— SFT` (Light green, solid line) 6. `— DebateGPT` (Dark green, solid line) 7. `—●— ACC-Collab (Ours)` (Orange, solid line with circle markers) 8. `—●— ACC-Collab + (Ours)` (Red, solid line with circle markers) ### Detailed Analysis **Llama-3 Chart:** * **Trend:** All methods show a general upward trend or plateau from Round 0 to Round 4. * **Top Performers:** `ACC-Collab (Ours)` (orange) starts highest (~0.941) and peaks at Round 3 (~0.953). `ACC-Collab + (Ours)` (red) starts lower (~0.933) but rises sharply to converge near the orange line by Round 2 (~0.948), maintaining a slight lead thereafter. * **Mid-tier:** `DebateGPT` (dark green) shows steady improvement from ~0.925 to ~0.932. * **Lower Tier:** The remaining methods (`SoM (2x)`, `SoM (4x)`, `Persona`, `DebateTune`, `SFT`) are clustered between ~0.915 and ~0.925, showing modest gains. `SFT` (light green) appears to be the lowest-performing method overall. **Mistral Chart:** * **Trend:** Similar upward trajectory, with the top two methods showing the most dramatic improvement. * **Top Performers:** `ACC-Collab + (Ours)` (red) and `ACC-Collab (Ours)` (orange) start around 0.87-0.875. Both rise steeply, with the red line slightly overtaking the orange line by Round 2. They plateau near 0.90-0.905 from Round 2 to 4. * **Mid-tier:** `DebateGPT` (dark green) improves from ~0.85 to ~0.87. `DebateTune` (brown) remains relatively flat around 0.86. * **Lower Tier:** `SFT`, `SoM (2x)`, `SoM (4x)`, and `Persona` are tightly grouped between ~0.84 and ~0.86, with `SFT` starting the lowest (~0.835). **Gemma-2 Chart:** * **Trend:** All methods improve from Round 0, with most plateauing after Round 2. * **Top Performers:** `ACC-Collab (Ours)` (orange) leads throughout, starting at ~0.895 and peaking at ~0.921 at Round 3. `ACC-Collab + (Ours)` (red) follows a similar path but remains slightly below the orange line. * **Notable Mid-tier:** `DebateGPT` (dark green) performs strongly, rising from ~0.89 to ~0.915, closely following the top two. `SFT` (light green) also shows strong improvement, reaching ~0.91. * **Lower Tier:** The dashed-line methods (`SoM (2x)`, `SoM (4x)`, `Persona`) and `DebateTune` are clustered between ~0.89 and ~0.905. `SoM (2x)` (blue dashed) is the lowest-performing method in the later rounds. ### Key Observations 1. **Consistent Superiority:** The two proposed methods, `ACC-Collab (Ours)` and `ACC-Collab + (Ours)`, consistently achieve the highest or near-highest accuracy across all three models and all rounds. 2. **Performance Hierarchy:** A clear hierarchy is visible: the "ACC-Collab" variants > `DebateGPT` > other methods (`SFT`, `DebateTune`, `Persona`, `SoM` variants). 3. **Round-Dependent Improvement:** Accuracy for the top methods improves significantly in the first 1-2 rounds before stabilizing. Lower-performing methods show less dramatic gains. 4. **Model-Specific Scales:** While the relative ranking of methods is similar, the absolute accuracy values differ by model, with Llama-3 achieving the highest overall accuracy (~0.95) and Mistral the lowest starting point (~0.84). 5. **Variance:** The shaded confidence intervals are generally wider for the top-performing methods, especially in early rounds, suggesting more variability in their performance gains. ### Interpretation The data strongly suggests that the authors' proposed collaborative methods (`ACC-Collab` and its enhanced version `ACC-Collab +`) are more effective at improving model accuracy through iterative rounds than the compared baselines (DebateGPT, SFT, Persona, SoM, DebateTune). This effectiveness is robust, holding across three distinct underlying LLM architectures (Llama-3, Mistral, Gemma-2). The trend of rapid early improvement followed by a plateau indicates that the collaborative process yields the most benefit in the initial rounds. The consistent underperformance of standard Supervised Fine-Tuning (`SFT`) and the `SoM` variants highlights the added value of the debate/collaboration framework. The fact that `ACC-Collab +` often starts lower than `ACC-Collab` but catches up or surpasses it (notably in Mistral) might suggest that the "+" variant requires a "warm-up" round but has a higher performance ceiling. The charts provide compelling visual evidence for the efficacy of the proposed approach in enhancing LLM reasoning or task performance through multi-round collaboration. </details> Figure 3: Accuracy over five rounds of deliberation on BoolQ (top) and SCIQ (bottom). #### Per-Round Accuracy Next, we look more closely at the accuracy of each method across five rounds of deliberation. Figure 3, shows per-round accuracy on the BoolQ dataset. As already illustrated by Table 1, ACC-Collab and ACC-Collab+ achieve higher final round accuracy than the other methods. Notably, our method has higher accuracy both at the final round $t=4$ and at round $t=0$ . Recall that at round $t=0$ , the actor’s response is independent of the critic. This indicates that in some cases, our training pipeline can produce actor agents with superior zero-shot accuracy (without deliberation) compared to models produced by SFT and DebateGPT. Interestingly, we observe that in some cases, SFT and DebateGPT are not much better than simple deliberation with untrained models (i.e., SoM-4x), e.g., BoolQ with Llama-3. This is primarily due to the fact that these methods are designed to improve the model’s single-short performance but do little to improve their collaborative abilities. Cases such as this outline the necessity of training to improve collaboration (ACC-Collab) rather than training for raw accuracy (SFT and DebateGPT). ### 5.3 Individual Performance Next, we examine the relative effectiveness of both the actor and the critic. To do this, we train an actor and critic via ACC-Collab. Then, during deliberation, we pair the trained actor with an untrained critic and pair the trained critic with an untrained actor. In Table 2, column “Actor” corresponds to the former, while “Critic” corresponds to the latter. On average, the trained actor attains higher accuracy compared to the trained critic, this aligns with intuition as the actor is responsible for providing answers while the critic plays a supporting role. In most cases, the trained actor (paired with an untrained critic) outperforms SFT and DebateGPT. When the trained actor is paired with a trained critic (either ACC-Collab or ACC-Collab+), its performance is further improved. Llama-3 SoM (4x) DebateGPT Actor Critic ACC-Collab ACC-Collab+ BoolQ $.811_±.007$ $.815_±.005$ $.867_±.003$ $.834_±.005$ $.887_±.005$ $.894_±.003$ MMLU $.635_±.004$ $.654_±.005$ $.651_±.012$ $.65_±.008$ $.644_±.01$ $.683_±.012$ BBH $.514_±.005$ $.551_±.008$ $.583_±.01$ $.55_±.015$ $.593_±.006$ $.574_±.003$ SCIQ $.923_±.002$ $.932_±.001$ $.947_±.002$ $.945_±.001$ $.952_±.0$ $.948_±.003$ ARC $.874_±.001$ $.876_±.002$ $.885_±.003$ $.866_±.002$ $.881_±.004$ $.869_±.002$ Mistral SoM (4x) DebateGPT Actor Critic ACC-Collab ACC-Collab+ BoolQ $.798_±.004$ $.848_±.002$ $.873_±.002$ $.843_±.002$ $.877_±.002$ $.893_±.002$ MMLU $.562_±.005$ $.577_±.002$ $.598_±.002$ $.611_±.001$ $.61_±.005$ $.672_±.004$ BBH $.462_±.003$ $.48_±.012$ $.493_±.012$ $.518_±.005$ $.519_±.009$ $.601_±.004$ SCIQ $.856_±.002$ $.871_±.003$ $.891_±.001$ $.891_±.002$ $.902_±.005$ $.905_±.002$ ARC $.823_±.002$ $.822_±.002$ $.833_±.003$ $.842_±.003$ $.843_±.003$ $.856_±.003$ Gemma-2 SoM (4x) DebateGPT Actor Critic ACC-Collab ACC-Collab+ BoolQ $.759_±.004$ $.812_±.003$ $.839_±.005$ $.774_±.014$ $.84_±.005$ $.845_±.005$ MMLU $.578_±.002$ $.582_±.002$ $.519_±.026$ $.566_±.002$ $.51_±.016$ $.555_±.003$ BBH $.449_±.01$ $.491_±.01$ $.51_±.011$ $.475_±.004$ $.513_±.006$ $.475_±.008$ SCIQ $.903_±.002$ $.914_±.002$ $.923_±.002$ $.912_±.001$ $.918_±.003$ $.909_±.003$ ARC $.843_±.005$ $.851_±.003$ $.85_±.002$ $.855_±.002$ $.852_±.003$ $.849_±.002$ Table 2: Accuracy after 5 rounds of deliberation. The Actor (Critic) column corresponds to an actor-agent (critic-agent) trained via ACC-Collab and paired with an untrained Critic (Actor) during deliberation. <details> <summary>extracted/6248978/criticResp.png Details</summary>  ### Visual Description ## Diagram: AI Critic Evaluation Process ### Overview The image is a conceptual diagram illustrating a three-stage interaction between AI agents in an evaluation or learning scenario. It depicts an initial response from an "Actor" agent, followed by feedback from two types of "Critic" agents: one "Untrained" and one "Trained." The diagram uses color-coded robot icons and text panels to show the flow and quality of critique. ### Components/Axes The diagram is structured into three horizontal panels, stacked vertically. 1. **Top Panel (Actor):** * **Icon:** A green, pixelated robot icon on the left. * **Label:** The word "Actor" in green text below the icon. * **Text Content:** A statement and a final answer. * **Visual Marker:** A large red "X" on the far right, indicating the answer is incorrect. 2. **Middle Panel (Untrained Critic):** * **Icon:** A blue, pixelated robot icon on the left. * **Label:** The words "Untrained Critic" in blue text below the icon. * **Text Content:** A feedback statement that agrees with the Actor. 3. **Bottom Panel (Trained Critic):** * **Icon:** A blue, pixelated robot icon on the left (identical to the Untrained Critic's icon). * **Label:** The words "Trained Critic" in blue text below the icon. * **Text Content:** A structured feedback statement that refutes the Actor's answer with numbered counterpoints. ### Detailed Analysis / Content Details **Panel 1: Actor** * **Text Transcription:** "With the decline of honey bee populations, plant species that . . . Final Answer: (B) change their flowers so that wind will fertilize them." * **Observation:** The text presents a multiple-choice question scenario. The Actor selects option (B). The red "X" confirms this answer is wrong. **Panel 2: Untrained Critic** * **Text Transcription:** "Your response is mostly correct . . . {continues to agree with the actor}." * **Observation:** The feedback is vague and supportive, failing to identify the error in the Actor's answer. The text in curly braces `{}` appears to be a placeholder or summary of the critic's continued agreement. **Panel 3: Trained Critic** * **Text Transcription:** "While your response is a valid explanation for why option B could be a possible answer, here are two reasons why option C is also correct and refutes your response directly: 1. . . .{Counterpoint to for why B could be wrong} 2. . . .{Evidence for why C is correct}" * **Observation:** This feedback is specific and constructive. It acknowledges the Actor's reasoning but provides a direct refutation by advocating for a different answer (option C). It uses a numbered list to present counter-evidence. The text in curly braces `{}` contains placeholders for the specific logical points. ### Key Observations 1. **Visual Hierarchy:** The diagram uses a clear top-to-bottom flow to represent a sequence: Actor proposes -> Untrained Critic evaluates -> Trained Critic evaluates. 2. **Color Coding:** Green is associated with the Actor (the source of the initial, incorrect answer). Blue is associated with both Critic roles, suggesting they belong to the same category of "evaluator" agents, despite their differing capabilities. 3. **Icon Consistency:** The Untrained and Trained Critics share the same robot icon, emphasizing that the difference is in their training/knowledge, not their fundamental identity or role. 4. **Feedback Quality Contrast:** The core contrast is between the vague, agreeable feedback of the Untrained Critic and the specific, refuting feedback of the Trained Critic. The Trained Critic's response is longer and more structured. 5. **Placeholder Text:** The use of `{...}` indicates that the diagram is a template or schematic, illustrating the *form* of the interaction rather than a specific, complete example. ### Interpretation This diagram serves as a pedagogical or conceptual model for understanding the value of trained evaluation in AI systems or learning processes. It demonstrates a failure mode where an initial agent (Actor) makes an error, and a naive evaluator (Untrained Critic) fails to catch it, potentially reinforcing the mistake. The Trained Critic represents the ideal: an evaluator capable of identifying errors, providing counter-evidence, and guiding toward a correct answer (option C in this case). The underlying message is that not all feedback is equal. Effective critique requires sufficient training or domain knowledge to move beyond surface-level agreement and engage in substantive refutation. The diagram could be used to argue for the importance of investing in robust critic models, human-in-the-loop verification, or advanced training techniques for AI evaluators to ensure reliability and correctness in automated systems. The honey bee question is merely a vehicle to illustrate this meta-concept about evaluation quality. </details> Figure 4: Comparison of responses from the critic model before and after training with ACC-Collab. ### 5.4 What do the Agents Actually Learn? Lastly, we are interested in understanding how ACC-Collab improves the Actor-Critic Team. Figure 4 demonstrates an example of the difference in responses between an untrained critic and a critic trained through ACC-Collab. Although the actor provides a wrong answer, the untrained critic is too agreeable and does not provide substantive feedback for the actor to correct their answer. In contrast, the trained critic is more willing to disagree with the actor and provides more detailed feedback. Largely, we observe that this trend is common; untrained critics are too agreeable and are thus less able to change the actor’s mind, while trained critics are more willing to disagree (see Section C.2 for more examples). When examining the trained actor’s responses, we do not find a notable qualitative change compared to the responses of an untrained actor. However, as shown previously, we do observe a qualitative change in the actor’s responses to become more accurate. ## 6 Conclusion, Limitations and Impact In this paper, we propose ACC-Collab, a novel framework for jointly training a two-agent team (one actor-agent and one critic-agent) to collaboratively solve problems through iterative discussion. To train these agents, we developed an off-policy data generation scheme dubbed “Guided-Collaboration”, which produces high-quality preference data for collaborative models. We found that ACC-Collab outperforms all baselines on a wide array of domains. In particular, even a single round of training for both the actor and critic results in a high-quality team. Of particular note is the effects that ACC-Collab has on the critic model. Without ACC-Collab, the critic model is often too agreeable and lacks verbosity in their responses. In contrast, after training with ACC-Collab, the critic is far more likely to provide detailed disagreements during discussion. However, our framework ACC-Collab also comes with limitations. Firstly, even though ACC-Collab attains superior performance compared to baselines on a wide array of domains, it is important to note that we conduct experiments mainly on question-answering tasks; thus, it remains to be seen whether such a framework would continue to be effective in other types of tasks. Moreover, in our experiments, we train and test models on the same task (partitioning each task into a training and testing set). As such, the generalizability of each actor-critic team to unseen domains is unknown. Our method makes use of the fact that for each question, correct and incorrect answers can be easily established. Secondly, while we provide results for three families of models, these experiments are performed on 2B, 7B, and 8B models. While our method is effective for these sizes (standard in open-source models), it remains to be seen whether this effectiveness will scale to larger models. ### Acknowledgments We would like to thank Li Hang for his valuable insights and guidance during the development of this work. ### Reproducibility Statement Here, we outline the details necessary to reproduce our results. We provide an algorithm for our data generation procedure (Algorithm 1), as well as a description of our training procedure in Section 4.3. Each dataset, baseline method, and base model used are specified at the beginning of Section 5. We provide additional experimental details in Section A. Prompts used for our method can be found in Section C. Lastly, we publicly release the code used for our method. ## References - Abdelnabi et al. (2023) Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Llm-deliberation: Evaluating llms with interactive multi-agent negotiation games. arXiv preprint arXiv:2309.17234, 2023. - Chan et al. (2023) Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201, 2023. - Chang (2024a) Edward Y Chang. Evince: Optimizing adversarial llm dialogues via conditional statistics and information theory. arXiv preprint arXiv:2408.14575, 2024a. - Chang (2024b) Edward Y Chang. Socrasynth: Multi-llm reasoning with conditional statistics. arXiv preprint arXiv:2402.06634, 2024b. - Chen et al. (2023) Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. arXiv preprint arXiv:2309.13007, 2023. - Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024. URL https://arxiv.org/abs/2401.01335. - Chollet (2019) François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. - Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019. - Du et al. (2023) Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. - Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. - Feng et al. (2024) Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration. arXiv preprint arXiv:2402.00367, 2024. - Fiez et al. (2019) Tanner Fiez, Benjamin Chasnov, and Lillian J Ratliff. Convergence of learning dynamics in stackelberg games. arXiv preprint arXiv:1906.01217, 2019. - Hadi et al. (2023) Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023. - Hao et al. (2024) Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, et al. Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. arXiv preprint arXiv:2404.05221, 2024. - Havrilla et al. (2024) Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Railneau. Glore: When, where, and how to improve llm reasoning via global and local refinements. arXiv preprint arXiv:2402.10963, 2024. - Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. - Hong et al. (2023) Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. - Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023. - Huang et al. (2024) Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333, 2024. - Irving et al. (2018) Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899, 2018. - Ji et al. (2023) Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating llm hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1827–1843, 2023. - Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. - Jiang et al. (2024) Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al. $\{$ MegaScale $\}$ : Scaling large language model training to more than 10,000 $\{$ GPUs $\}$ . In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pp. 745–760, 2024. - Khan et al. (2024) Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive llms leads to more truthful answers. arXiv preprint arXiv:2402.06782, 2024. - Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023. - Li et al. (2023a) Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for” mind” exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023a. - Li et al. (2023b) Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. Theory of mind for multi-agent collaboration via large language models. arXiv preprint arXiv:2310.10701, 2023b. - Li et al. (2024a) Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. arXiv preprint arXiv:2402.05120, 2024a. - Li et al. (2024b) Ming Li, Jiuhai Chen, Lichang Chen, and Tianyi Zhou. Can llms speak for diverse people? tuning llms via debate to generate controllable controversial statements. arXiv preprint arXiv:2402.10614, 2024b. - Li et al. (2023c) Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02762, 2023c. - Li & Sethi (2017) Tao Li and Suresh P Sethi. A review of dynamic stackelberg game models. Discrete & Continuous Dynamical Systems-B, 22(1):125, 2017. - Li et al. (2023d) Yuan Li, Yixuan Zhang, and Lichao Sun. Metaagents: Simulating interactions of human behaviors for llm-based task-oriented coordination via collaborative generative agents. arXiv preprint arXiv:2310.06500, 2023d. - Li et al. (2024c) Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. Improving multi-agent debate with sparse communication topology. arXiv preprint arXiv:2406.11776, 2024c. - Liang et al. (2023) Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118, 2023. - Liu et al. (2023) Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170, 2023. - Michael et al. (2023) Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, and Samuel R Bowman. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702, 2023. - Otter et al. (2020) Daniel W Otter, Julian R Medina, and Jugal K Kalita. A survey of the usages of deep learning for natural language processing. IEEE transactions on neural networks and learning systems, 32(2):604–624, 2020. - Pang et al. (2024a) Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024a. - Pang et al. (2024b) Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024b. URL https://arxiv.org/abs/2404.19733. - Pham et al. (2023) Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A Plummer, Zhaoran Wang, and Hongxia Yang. Let models speak ciphers: Multiagent debate through embeddings. arXiv preprint arXiv:2310.06272, 2023. - Radford (2018) Alec Radford. Improving language understanding by generative pre-training. 2018. - Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. - Rasal (2024) Sumedh Rasal. Llm harmony: Multi-agent communication for problem solving. arXiv preprint arXiv:2401.01312, 2024. - Rawte et al. (2023) Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922, 2023. - Shinn et al. (2023) Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366. - Singhal et al. (2023) Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617, 2023. - (47) Andries Petrus Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D Barrett, and Arnu Pretorius. Should we be going mad? a look at multi-agent debate strategies for llms. In Forty-first International Conference on Machine Learning. - Subramaniam et al. (2024) Vighnesh Subramaniam, Antonio Torralba, and Shuang Li. Debategpt: Fine-tuning large language models with multi-agent debate supervision. 2024. - Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. - Team et al. (2024) Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. - Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. - Tonmoy et al. (2024) SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313, 2024. - Trung et al. (2024) Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024. - Wang et al. (2023a) Boshi Wang, Xiang Yue, and Huan Sun. Can chatgpt defend its belief in truth? evaluating llm reasoning via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11865–11881, 2023a. - Wang et al. (2024) Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of llm reasoning: Are multi-agent discussions the key? arXiv preprint arXiv:2402.18272, 2024. - Wang et al. (2023b) Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300, 2023b. - Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. - Welbl et al. (2017) Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017. - Wu et al. (2023) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023. - Xiong et al. (2024) Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! llm agent learning via iterative step-level process refinement, 2024. URL https://arxiv.org/abs/2406.11176. - Yang et al. (2024) Ruixin Yang, Dheeraj Rajagopa, Shirley Anugrah Hayati, Bin Hu, and Dongyeop Kang. Confidence calibration and rationalization for llms via multi-agent deliberation. arXiv preprint arXiv:2404.09127, 2024. - Zhang et al. (2023) Jintian Zhang, Xin Xu, and Shumin Deng. Exploring collaboration mechanisms for llm agents: A social psychology view. arXiv preprint arXiv:2310.02124, 2023. - Zhang et al. (2024) Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. Llm as a mastermind: A survey of strategic reasoning with large language models. arXiv preprint arXiv:2404.01230, 2024. - Zhu et al. (2024) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis, 2024. URL https://arxiv.org/abs/2304.04675. ## Appendix ## Appendix A Experiments Here, we provide additional details regarding our experiments. #### Datasets Each dataset is split into a training set, a validation set, and a testing set. For datasets that come with an explicit partition of these sets we use the given partitions; this includes BoolQ, MMLU, SCIQ, and ARC. For BBH, we randomly sample roughly 25% and 10% of the questions from each category in BBH to create a test and validation set, respectively; this comes out to 1260 questions for the test set and 500 questions for the validation set. All results are reported on questions in the test set. #### Compute All training was performed on a single Nvidia-H800 GPU. Inference for Llama-3 and Mistral based models is performed on a single Nvidia-v100 GPU, for Gemma-2 based models we used a single Nvidia-H800. All inference is performed with the VLLM library Kwon et al. (2023). #### Training Training for all models was performed via the trl library, using LoRAs of size 256. When training ACC-Collab with DPO we use a negative log-likelihood (NLL) regularization term (with weight $1$ ) as outlined in Pang et al. (2024a). <details> <summary>extracted/6248978/plots/mainMethod_v4.png Details</summary>  ### Visual Description ## Diagram: Actor-Critic Preference-Based Training Process ### Overview The image is a technical flowchart illustrating a machine learning training process involving two agents: an "Actor" and a "Critic." The process generates preference data by comparing the quality of different trajectories (sequences of states/actions) produced under three distinct guidance conditions. This data is then used to train models, with the Critic receiving feedback from the training outcomes. ### Components/Axes The diagram is structured as a left-to-right flowchart with a feedback loop. Key components are labeled with text and mathematical notation. **Left Side (Agents & Initial State):** * **Actor:** Represented by an orange robot icon and the label "Actor" in orange text. Positioned at the top-left. * **Critic:** Represented by a blue robot icon and the label "Critic" in blue text. Positioned at the bottom-left. * **Initial State:** Two stacked speech bubbles represent the state at time `t-1`. * Top bubble (orange outline): Contains the text `z_a^{(t-1)}`. * Bottom bubble (blue outline): Contains the text `z_c^{(t-1)}`. **Central Process (Three Parallel Paths):** Three arrows originate from the initial state bubbles, each labeled with a process name and leading to a pair of output bubbles at time `t`. 1. **Top Path:** Labeled "Actor Guided Towards y". Leads to: * A green-outlined bubble containing `z_a^{(t)}`. * A blue-outlined bubble containing `z_c^{(t)}`. 2. **Middle Path:** Labeled "Natural Debate". Leads to: * An orange-outlined bubble containing `z_a^{(t)}`. * A blue-outlined bubble containing `z_c^{(t)}`. 3. **Bottom Path:** Labeled "Actor Guided Away From y". Leads to: * A red-outlined bubble containing `z_a^{(t)}`. * A blue-outlined bubble containing `z_c^{(t)}`. **Comparison & Decision Nodes:** The outputs from the three paths feed into two circular comparison nodes. * **Top Node:** Contains the symbol `Δ_y` (Delta subscript y). It receives inputs from the top path's green `z_a^{(t)}` and the middle path's orange `z_a^{(t)}`. * **Bottom Node:** Contains the symbol `Δ_{!y}` (Delta subscript exclamation-mark y). It receives inputs from the middle path's orange `z_a^{(t)}` and the bottom path's red `z_a^{(t)}`. * **Label:** Below these nodes is the text "Relative Quality of Trajectory". **Data Storage & Training:** * **Decision Logic:** Arrows from the comparison nodes lead to a database icon. The arrows are labeled with conditional statements: * From `Δ_y`: "if `Δ_y ≥ ε`" (if Delta y is greater than or equal to epsilon). * From `Δ_{!y}`: "elif `Δ_{!y} ≥ ε`" (else if Delta not-y is greater than or equal to epsilon). * **Database:** A cylinder icon labeled "Preference Data". Inside, it contains two pairs of notations: * `(z_a^{(t)}, z_a^{(t)})` (The first `z_a` is in orange, the second in green). * `(z_a^{(t)}, z_a^{(t)})` (The first `z_a` is in orange, the second in red). * **Training Output:** An arrow points from the database to the right, where two robot icons (orange and blue) are shown transforming into larger versions. This section is labeled "Train Models". **Feedback Loop:** A thick black arrow originates from the "Train Models" section, travels along the bottom of the diagram, and points back to the "Critic" icon on the left, indicating a feedback mechanism. ### Detailed Analysis The diagram details a method for generating comparative preference data for training AI models. 1. **Trajectory Generation:** Starting from a shared previous state (`z_a^{(t-1)}`, `z_c^{(t-1)}`), three different trajectories are generated for the next time step (`t`): * One where the Actor is guided toward a target `y`. * One resulting from a "Natural Debate" (presumably unguided or standard interaction). * One where the Actor is guided away from target `y`. The Critic's state (`z_c^{(t)}`) is generated alongside the Actor's (`z_a^{(t)}`) in each case. 2. **Quality Comparison:** The system compares the Actor's states from different trajectories: * `Δ_y` compares the "guided towards y" Actor state (green) with the "natural debate" Actor state (orange). * `Δ_{!y}` compares the "natural debate" Actor state (orange) with the "guided away from y" Actor state (red). The label "Relative Quality of Trajectory" indicates these deltas measure a quality difference. 3. **Preference Data Creation:** If the quality difference (`Δ`) exceeds a threshold `ε` (epsilon), the pair of compared Actor states is stored as a preference datum in the database. The notation in the database shows the "natural debate" state (orange) is paired with either the "guided towards" state (green) or the "guided away" state (red), implying a preference ordering. 4. **Model Training & Feedback:** The collected preference data is used to "Train Models." The updated models (represented by larger robot icons) then provide feedback to the Critic, closing the loop and presumably improving its future evaluations. ### Key Observations * **Color Coding is Critical:** The outline color of the `z_a^{(t)}` bubbles (green, orange, red) is essential for tracking which trajectory they came from and how they are paired in the database. * **Conditional Logic:** Data is only stored if the relative quality difference is significant (≥ ε), filtering out trivial comparisons. * **Asymmetric Roles:** The Actor generates the primary trajectories (`z_a`), while the Critic's state (`z_c`) is generated in parallel but is not directly compared. The Critic's main role appears to be in the feedback loop after training. * **Feedback Loop:** The system is iterative; training outputs influence the Critic, which will affect future trajectory evaluations. ### Interpretation This diagram depicts a sophisticated **preference-based reinforcement learning or alignment training scheme**. The core idea is to create a dataset of *comparative preferences* rather than absolute rewards. * **What it demonstrates:** The process systematically generates "good" (guided toward y) and "bad" (guided away from y) examples relative to a baseline (natural debate). By comparing these, it learns what makes a trajectory preferable with respect to the target `y`. The threshold `ε` ensures only clear preferences are learned. * **How elements relate:** The Actor is the policy being trained. The Critic is likely a value function or reward model that is being improved via the feedback loop. The "Natural Debate" path serves as a crucial control or reference point. The database of preference pairs `(winner, loser)` is the key training signal. * **Notable implications:** This method could be used for **Constitutional AI** or **RLHF (Reinforcement Learning from Human Feedback)**, where `y` represents a desired behavior (e.g., helpfulness, harmlessness). The "Actor Guided Away From y" path is particularly interesting, as it actively generates negative examples to contrast against. The feedback to the Critic suggests an **adversarial or co-evolutionary dynamic** where the evaluator (Critic) improves alongside the generator (Actor). </details> <details> <summary>extracted/6248978/plots/mainMethod_critic.png Details</summary>  ### Visual Description \n ## Process Diagram: Actor-Critic Trajectory Generation for Preference Data ### Overview This diagram illustrates a machine learning process where an "Actor" model and a "Critic" model interact over time steps to generate pairs of trajectories. These trajectories are compared to assess their relative quality concerning a target `y`. If the quality difference meets a threshold `ε`, the trajectory pairs are stored as preference data, which is subsequently used to train both the Actor and Critic models in a feedback loop. ### Components/Axes The diagram is a flowchart with the following key components, flowing primarily from left to right: 1. **Actors & Critics (Left Side):** * **Actor:** Represented by an orange robot icon. Its state at time `t-1` is denoted as `z_a^{(t-1)}` in an orange box. * **Critic:** Represented by a blue robot icon. Its state at time `t-1` is denoted as `z_c^{(t-1)}` in a blue box. 2. **Trajectory Generation Paths (Center-Left):** Three distinct paths originate from the initial states: * **Top Path (Critic Guided Towards y):** An arrow labeled "Critic Guided Towards y" leads to a pair of boxes: an orange `z_a^{(t)}` and a green `z_c^{(t)}`. * **Middle Path (Natural Debate):** An arrow labeled "Natural Debate" leads to a pair of boxes: an orange `z_a^{(t)}` and a blue `z_c^{(t)}`. * **Bottom Path (Critic Guided Away From y):** An arrow labeled "Critic Guided Away From y" leads to a pair of boxes: an orange `z_a^{(t)}` and a red `z_c^{(t)}`. 3. **Quality Comparison (Center):** * Two circular nodes labeled "Relative Quality of Trajectory". * The top circle contains `Δ_y` (green delta symbol). It receives inputs from the top and middle trajectory pairs. * The bottom circle contains `Δ_{!y}` (red delta symbol with an exclamation mark). It receives inputs from the middle and bottom trajectory pairs. 4. **Decision Logic & Data Storage (Center-Right):** * **Condition 1:** An arrow from `Δ_y` is labeled "if `Δ_y ≥ ε`". * **Condition 2:** An arrow from `Δ_{!y}` is labeled "elif `Δ_{!y} ≥ ε`". * Both arrows point to a cylinder icon representing a database. * **Database Contents:** The database is labeled "Preference Data" and contains two pairs of trajectories: * `(z_c^{(t)}, z_c^{(t)})` (The first `z_c` is blue, the second is green). * `(z_c^{(t)}, z_c^{(t)})` (The first `z_c` is red, the second is blue). 5. **Model Training & Feedback Loop (Right Side):** * An arrow leads from the database to a section labeled "Train Models". * This section shows an orange Actor robot evolving into a larger orange Actor robot, and a blue Critic robot evolving into a larger blue Critic robot. * A long feedback arrow runs from the "Train Models" output back to the initial Critic icon on the far left, closing the loop. ### Detailed Analysis * **Process Flow:** The system starts with the previous states of the Actor (`z_a^{(t-1)}`) and Critic (`z_c^{(t-1)}`). Three different interaction modes generate new state pairs at time `t`: 1. The Critic actively guides the Actor *toward* a target `y`, producing a green Critic state. 2. The Actor and Critic engage in a "Natural Debate," producing a standard blue Critic state. 3. The Critic actively guides the Actor *away from* the target `y`, producing a red Critic state. * **Comparison Logic:** The system then compares the quality of these generated Critic trajectories: * `Δ_y` measures the quality difference between the "Guided Towards y" (green) trajectory and the "Natural Debate" (blue) trajectory. * `Δ_{!y}` measures the quality difference between the "Guided Away From y" (red) trajectory and the "Natural Debate" (blue) trajectory. * **Data Collection Rule:** A trajectory pair is stored as preference data only if the measured quality difference (`Δ_y` or `Δ_{!y}`) is greater than or equal to a predefined threshold `ε`. This ensures only significant comparisons are saved. * **Stored Data:** The database stores ordered pairs of Critic trajectories. The notation `(z_c^{(t)}, z_c^{(t)})` with different colors implies a preference order: the first element is preferred over the second. For example, `(blue, green)` likely means the blue (debate) trajectory is preferred over the green (guided towards y) one, and `(red, blue)` means the red (guided away) is preferred over the blue. * **Training:** The collected preference data is used to train both the Actor and Critic models, improving their performance. The trained Critic is then fed back into the next iteration of the process. ### Key Observations 1. **Color-Coded Semantics:** Colors are used consistently to denote meaning: Orange (Actor), Blue (Critic baseline/debate), Green (Critic guided towards target `y`), Red (Critic guided away from target `y`). 2. **Asymmetric Guidance:** The process explicitly generates data by having the Critic provide both positive (towards `y`) and negative (away from `y`) guidance, creating a contrast for learning. 3. **Threshold-Based Filtering:** The use of `ε` acts as a quality filter, preventing the storage of trivial or noisy comparisons where the quality difference is negligible. 4. **Closed-Loop System:** The diagram depicts a complete, iterative learning cycle where generated data improves the models, which in turn generate better data. ### Interpretation This diagram outlines a sophisticated **self-improving AI training framework**, likely for reinforcement learning or alignment tasks. The core idea is to use a Critic model not just to evaluate, but to actively *create* contrasting examples (trajectories) by guiding an Actor model. By comparing the quality of these deliberately contrasted examples (guided vs. natural) and storing only the decisive comparisons, the system efficiently builds a high-quality preference dataset. The process addresses a key challenge in AI training: generating meaningful feedback data. Instead of relying solely on human feedback, the Critic automates the creation of training signals. The "Natural Debate" path serves as a crucial baseline. The final training step closes the loop, suggesting the goal is to iteratively refine both models so the Critic becomes better at identifying quality and the Actor becomes better at achieving desired outcomes (`y`). The entire system is designed for autonomous, scalable improvement. </details> Figure 5: ACC-Collab training pipeline, exemplified for the actor (top) and critic (bottom). The process 1) We generate data from both natural deliberation as well as guided deliberation towards and away from the ground truth answer $y$ using the actor and critic. 2) We compute the relative quality of each trajectory based on the expected quality difference $Δ_y,Δ_!y$ w.r.t. to the natural response. 3) We store all high-quality pairwise data in our database and train the actor model. 4) We alternate this procedure for the actor and critic. As outline in Section 4 both the guidance procedure and the computation of $Δ_y,Δ_!y$ differ between the actor and critic. ### A.1 Correctness of Reasoning Steps In addition to measuring outcome accuracy (i.e., the accuracy of the actor’s final answer), we also measure process accuracy (i.e., the accuracy of the reasoning and discussion steps taken by the actor and the critic model). These results are displayed in Table 3. We observe that our training pipeline typically maintains or increases the accuracy of the reasoning steps. Unlike outcome accuracy, process accuracy does not have ground truth answers. Thus we utilize GPT-4o to serve as an oracle. We use the following steps to evaluate the correctness of the reasoning and discussion steps. 1. For both an untrained actor-critic team and a trained actor-critic team (trained via ACC-Collab), we randomly sample 500 deliberations, which resulted in the correct answer after 5 rounds of deliberation. 1. We then prompted GPT-4o to evaluate the correctness of the reasoning and discussion steps taken by the actor-critic teams in each of the 500 deliberations. 1. We then compute the average accuracy of the reasoning and discussion steps as evaluated by GPT-4o. | Dataset Llama-3 BoolQ | ACC-Collab+ Accuracy $.890_±.035$ | ACC-Collab Accuracy $.894_±.034$ | Untrained Accuracy $.794_±.045$ | | --- | --- | --- | --- | | MMLU | $.968_±.020$ | $.976_±.017$ | $.909_±.032$ | | BBH | $.519_±.064$ | $.575_±.055$ | $.610_±.055$ | | SCIQ | $.958_±.022$ | $.981_±.015$ | $.937_±.027$ | | ARC | $.882_±.046$ | $.833_±.042$ | $.883_±.036$ | | Mistral | | | | | BoolQ | $.907_±.035$ | $.928_±.029$ | $.894_±.034$ | | MMLU | $.882_±.042$ | $.884_±.036$ | $.878_±.037$ | | BBH | $.766_±.045$ | $.818_±.043$ | $.538_±.056$ | | SCIQ | $.973_±.018$ | $.972_±.018$ | $.933_±.028$ | | ARC | $.927_±.029$ | $.952_±.024$ | $.852_±.040$ | | Gemma-2 | | | | | BoolQ | $.869_±.038$ | $.872_±.037$ | $.861_±.039$ | | MMLU | $.844_±.041$ | $.843_±.041$ | $.800_±.045$ | | BBH | $.597_±.056$ | $.637_±.054$ | $.514_±.056$ | | SCIQ | $.984_±.014$ | $.978_±.016$ | $.985_±.014$ | | ARC | $.850_±.040$ | $.854_±.039$ | $.834_±.042$ | Table 3: Accuracy of the actor’s and critic’s reasoning and discussion steps as evaluated by GPT-4o. ## Appendix B Preference Optimization Here, we remark on how other preference optimization schemes can be used in place of DPO. Broadly speaking, preference optimization schemes can broken into two categories: those which optimize reward directly (e.g., PPO) and those which optimize reward indirectly (e.g., DPO). In the case of the latter, the positive and negative pairs produced by our method (Algorithm 1) can be directly plugged into the preference optimizer. In the case of explicit reward maximization, the reward function $r(z^(t),x,y)$ can first be learned automatically by simply simulating deliberation. The most straightforward way to do this is to first simulate one full deliberation between the actor and critic, i.e., $⟨(z_a^(0),z_c^(0)),…,(z_a^(T),z_c^(T))⟩$ . Then for each $z_a^(t)$ , $z_c^(t)$ the remaining $T-t$ deliberation steps can be resampled to estimate the corresponding value of $r(z_a^(t),x,y)$ and $r(z_c^(t),x,y)$ . These pairs, namely $\big{(}z_a^(t),r(z_a^(t),x,y)\big{)}$ and $\big{(}z_c^(t),r(z_c^(t),x,y)\big{)}$ can then be used to learn the reward function. This reward function can then be plugged into the desired preference optimization scheme. <details> <summary>extracted/6248978/plots/new_perRoundConf.png Details</summary>  ### Visual Description \n ## Line Chart: Accuracy Comparison of AI Models Over Multiple Rounds ### Overview The image displays three side-by-side line charts comparing the performance (accuracy) of eight different methods or models across five training/evaluation rounds (0 to 4). Each chart corresponds to a different base large language model: Llama-3, Mistral, and Gemma-2. The charts track how the accuracy of each method evolves over successive rounds. ### Components/Axes * **Chart Titles (Top Center):** "Llama-3", "Mistral", "Gemma-2". * **Y-Axis (Left Side):** Labeled "Accuracy". The scale varies per chart: * Llama-3: 0.725 to 0.900 (increments of 0.025). * Mistral: 0.80 to 0.90 (increments of 0.02). * Gemma-2: 0.700 to 0.850 (increments of 0.025). * **X-Axis (Bottom):** Labeled "Round". Markers at 0, 1, 2, 3, 4. * **Legend (Bottom, spanning all charts):** Contains eight entries, each with a specific color and line style: 1. `SoM (2x)`: Blue dashed line (`--`). 2. `SoM (4x)`: Teal dashed line (`--`). 3. `Persona`: Purple dashed line (`--`). 4. `DebateTune`: Brown solid line. 5. `SFT`: Light green solid line. 6. `DebateGPT`: Dark green solid line. 7. `ACC-Collab (Ours)`: Orange solid line with circular markers. 8. `ACC-Collab + (Ours)`: Red solid line with circular markers. * **Data Series:** Each method is represented by a line with a shaded region (likely indicating confidence intervals or standard deviation). ### Detailed Analysis **1. Llama-3 Chart (Left)** * **Trend Verification:** The two "ACC-Collab" methods (red, orange) show a strong upward trend from Round 0 to 1, then plateau. Other methods show more modest gains or fluctuations. * **Data Points (Approximate):** * **ACC-Collab + (Red):** Starts ~0.845 (R0), jumps to ~0.885 (R1), then slowly rises to ~0.895 (R4). * **ACC-Collab (Orange):** Starts ~0.845 (R0), jumps to ~0.885 (R1), then plateaus around ~0.885-0.890. * **DebateGPT (Dark Green):** Starts ~0.805 (R0), rises to ~0.820 (R1), peaks ~0.825 (R2), then slightly declines to ~0.815 (R4). * **SFT (Light Green):** Starts ~0.800 (R0), rises to ~0.805 (R1), then fluctuates around 0.800-0.805. * **SoM (2x) (Blue Dashed):** Starts ~0.775 (R0), rises to ~0.810 (R1), then slowly increases to ~0.815 (R4). * **SoM (4x) (Teal Dashed):** Follows a very similar path to SoM (2x), ending slightly lower at ~0.810 (R4). * **Persona (Purple Dashed):** Starts ~0.775 (R0), rises to ~0.785 (R1), then remains flat around 0.780-0.785. * **DebateTune (Brown):** Starts ~0.765 (R0), rises to ~0.775 (R1), dips to ~0.765 (R2), then recovers to ~0.775 (R4). **2. Mistral Chart (Center)** * **Trend Verification:** Similar pattern where ACC-Collab methods lead. Notably, the SoM methods (blue, teal dashed) show a clear peak at Round 1 followed by a significant decline. * **Data Points (Approximate):** * **ACC-Collab + (Red):** Starts ~0.860 (R0), jumps to ~0.890 (R1), then slowly rises to ~0.895 (R4). * **ACC-Collab (Orange):** Starts ~0.830 (R0), jumps to ~0.870 (R1), then rises to ~0.875 (R4). * **DebateGPT (Dark Green):** Starts ~0.820 (R0), rises to ~0.845 (R1), then plateaus around ~0.850. * **SFT (Light Green):** Starts ~0.815 (R0), rises to ~0.835 (R1), then slowly increases to ~0.840 (R4). * **Persona (Purple Dashed):** Starts ~0.810 (R0), rises to ~0.830 (R1), then plateaus around ~0.830. * **DebateTune (Brown):** Starts ~0.805 (R0), rises to ~0.825 (R1), then slowly increases to ~0.830 (R4). * **SoM (2x) (Blue Dashed):** Starts ~0.805 (R0), peaks at ~0.820 (R1), then declines sharply to ~0.800 (R4). * **SoM (4x) (Teal Dashed):** Follows a nearly identical path to SoM (2x), peaking at ~0.820 (R1) and declining to ~0.800 (R4). **3. Gemma-2 Chart (Right)** * **Trend Verification:** ACC-Collab methods again show strong initial gains. The DebateGPT (dark green) line shows a distinct peak at Round 2 before declining. * **Data Points (Approximate):** * **ACC-Collab (Orange):** Starts ~0.785 (R0), jumps to ~0.845 (R1), then plateaus around ~0.840-0.845. * **ACC-Collab + (Red):** Starts ~0.770 (R0), jumps to ~0.825 (R1), then rises to ~0.845 (R4). * **DebateGPT (Dark Green):** Starts ~0.770 (R0), rises to ~0.810 (R1), peaks at ~0.820 (R2), then declines to ~0.810 (R4). * **SFT (Light Green):** Starts ~0.745 (R0), rises to ~0.780 (R1), then slowly increases to ~0.790 (R4). * **DebateTune (Brown):** Starts ~0.715 (R0), rises to ~0.750 (R1), then slowly increases to ~0.770 (R4). * **SoM (2x) (Blue Dashed):** Starts ~0.720 (R0), rises to ~0.755 (R1), then plateaus around ~0.755-0.760. * **SoM (4x) (Teal Dashed):** Follows a similar path to SoM (2x), ending slightly lower at ~0.755 (R4). * **Persona (Purple Dashed):** Starts ~0.710 (R0), rises to ~0.725 (R1), then slowly declines to ~0.720 (R4). ### Key Observations 1. **Consistent Superiority:** The methods labeled "ACC-Collab (Ours)" and "ACC-Collab + (Ours)" (orange and red lines) consistently achieve the highest accuracy across all three base models (Llama-3, Mistral, Gemma-2) by the final round. 2. **Strong Initial Gains:** All methods show their most significant accuracy improvement between Round 0 and Round 1. 3. **Model-Specific Behavior:** * On **Mistral**, the SoM methods (2x and 4x) exhibit a unique "peak and decline" pattern, performing best at Round 1 but degrading afterward. * On **Gemma-2**, the DebateGPT method shows a clear performance peak at Round 2 before declining. 4. **Performance Clustering:** Methods tend to cluster in performance tiers. The ACC-Collab variants form the top tier. DebateGPT and SFT often form a middle tier. The remaining methods (SoM variants, Persona, DebateTune) generally occupy the lower tier, though their relative ordering varies by model. 5. **Uncertainty Bands:** The shaded regions indicate variability in performance. The bands are generally wider for lower-performing methods (e.g., DebateTune on Llama-3) and narrower for the top-performing ACC-Collab methods, suggesting more consistent results. ### Interpretation This data strongly suggests that the proposed "ACC-Collab" methods (both the standard and "+" variants) are more effective at improving and sustaining accuracy over multiple rounds of debate or collaboration compared to the baseline methods (SoM, Persona, DebateTune, SFT, DebateGPT). The consistent top-tier performance across three distinct LLM architectures (Llama, Mistral, Gemma) indicates the robustness of the ACC-Collab approach. The "peak and decline" behavior of some methods (SoM on Mistral, DebateGPT on Gemma-2) is a critical finding. It suggests these methods may suffer from issues like overfitting to early rounds, catastrophic forgetting, or an inability to leverage later rounds effectively. In contrast, the ACC-Collab methods show stable or improving performance after the initial jump, indicating a more sustainable learning or adaptation process. The charts collectively demonstrate that the choice of collaboration/finetuning method has a larger impact on final accuracy than the choice of base model within this set, as the performance hierarchy of methods is largely preserved across the three different models. The research appears to validate the authors' "ACC-Collab" technique as a superior method for multi-round collaborative AI tasks. </details> Figure 6: Accuracy over five rounds of deliberation on BoolQ. <details> <summary>extracted/6248978/plots/new_perRoundMMLU.png Details</summary>  ### Visual Description ## Multi-Panel Line Chart: Accuracy of Various Methods Across Iterative Rounds ### Overview The image displays three side-by-side line charts comparing the performance (accuracy) of eight different methods over a series of 0 to 4 iterative rounds. Each chart corresponds to a different base language model: Llama-3, Mistral, and Gemma-2. The charts share a common legend and x-axis label but have independent y-axis scales. Shaded regions around each line indicate confidence intervals or variance. ### Components/Axes * **Chart Titles (Top Center):** "Llama-3" (left), "Mistral" (center), "Gemma-2" (right). * **X-Axis (Bottom of each chart):** Labeled "Round". Ticks are at integer values 0, 1, 2, 3, 4. * **Y-Axis (Left of each chart):** Labeled "Accuracy". The scale varies: * Llama-3: ~0.62 to ~0.69 * Mistral: ~0.56 to ~0.68 * Gemma-2: ~0.48 to ~0.58 * **Legend (Bottom, spanning all charts):** Contains 8 entries, each with a distinct color and line style: 1. **SoM (2x):** Blue dashed line (`--`). 2. **SoM (4x):** Teal dashed line (`--`). 3. **Persona:** Purple dashed line (`--`). 4. **DebateTune:** Brown solid line. 5. **SFT:** Light green solid line. 6. **DebateGPT:** Dark green solid line. 7. **ACC-Collab (Ours):** Orange solid line with circular markers. 8. **ACC-Collab + (Ours):** Red solid line with circular markers. ### Detailed Analysis **1. Llama-3 Chart (Left Panel)** * **Trend Verification:** The red line (ACC-Collab +) shows a strong, consistent upward trend. The orange line (ACC-Collab) also trends upward but with a dip at round 4. The dark green (DebateGPT) and light green (SFT) lines show moderate upward trends. The dashed lines (SoM variants, Persona) and the brown line (DebateTune) are relatively flat or show very slight improvement. * **Data Points (Approximate Accuracy):** * **ACC-Collab + (Red):** R0: ~0.640, R1: ~0.668, R2: ~0.669, R3: ~0.668, R4: ~0.683. * **ACC-Collab (Orange):** R0: ~0.635, R1: ~0.640, R2: ~0.648, R3: ~0.656, R4: ~0.644. * **DebateGPT (Dark Green):** R0: ~0.640, R1: ~0.649, R2: ~0.652, R3: ~0.653, R4: ~0.654. * **SFT (Light Green):** R0: ~0.640, R1: ~0.640, R2: ~0.640, R3: ~0.640, R4: ~0.642. * **DebateTune (Brown):** R0: ~0.620, R1: ~0.631, R2: ~0.632, R3: ~0.631, R4: ~0.630. * **Persona (Purple Dashed):** R0: ~0.622, R1: ~0.633, R2: ~0.638, R3: ~0.636, R4: ~0.640. * **SoM (4x) (Teal Dashed):** R0: ~0.615, R1: ~0.629, R2: ~0.632, R3: ~0.635, R4: ~0.635. * **SoM (2x) (Blue Dashed):** R0: ~0.617, R1: ~0.620, R2: ~0.620, R3: ~0.620, R4: ~0.620. **2. Mistral Chart (Center Panel)** * **Trend Verification:** The red line (ACC-Collab +) shows a sharp initial rise from R0 to R1, then plateaus with a slight upward drift. The orange line (ACC-Collab) shows a steady, moderate upward trend. The light green (SFT) line rises initially then plateaus. The dark green (DebateGPT), brown (DebateTune), and dashed lines are clustered at the bottom with minimal change. * **Data Points (Approximate Accuracy):** * **ACC-Collab + (Red):** R0: ~0.638, R1: ~0.670, R2: ~0.662, R3: ~0.670, R4: ~0.672. * **ACC-Collab (Orange):** R0: ~0.586, R1: ~0.597, R2: ~0.605, R3: ~0.604, R4: ~0.610. * **SFT (Light Green):** R0: ~0.572, R1: ~0.589, R2: ~0.592, R3: ~0.594, R4: ~0.594. * **DebateGPT (Dark Green):** R0: ~0.558, R1: ~0.577, R2: ~0.573, R3: ~0.576, R4: ~0.578. * **DebateTune (Brown):** R0: ~0.553, R1: ~0.560, R2: ~0.561, R3: ~0.563, R4: ~0.563. * **Persona (Purple Dashed):** R0: ~0.553, R1: ~0.570, R2: ~0.571, R3: ~0.572, R4: ~0.573. * **SoM (4x) (Teal Dashed):** R0: ~0.560, R1: ~0.570, R2: ~0.571, R3: ~0.572, R4: ~0.573. * **SoM (2x) (Blue Dashed):** R0: ~0.560, R1: ~0.570, R2: ~0.571, R3: ~0.572, R4: ~0.573. **3. Gemma-2 Chart (Right Panel)** * **Trend Verification:** The red line (ACC-Collab +) shows a clear upward trend. The orange line (ACC-Collab) is highly volatile, with a sharp drop at R1, a peak at R2, another drop at R3, and a recovery at R4. The cluster of lines at the top (DebateGPT, SFT, DebateTune, Persona, SoM variants) are tightly grouped and show a very slight, steady upward trend. * **Data Points (Approximate Accuracy):** * **ACC-Collab + (Red):** R0: ~0.511, R1: ~0.531, R2: ~0.544, R3: ~0.538, R4: ~0.555. * **ACC-Collab (Orange):** R0: ~0.541, R1: ~0.497, R2: ~0.514, R3: ~0.498, R4: ~0.510. * **Top Cluster (R0 values ~0.560-0.566, R4 values ~0.576-0.581):** This group includes DebateGPT (Dark Green), SFT (Light Green), DebateTune (Brown), Persona (Purple Dashed), SoM (4x) (Teal Dashed), and SoM (2x) (Blue Dashed). Their lines are nearly indistinguishable at this scale, all trending gently upward from ~0.56 to ~0.58. ### Key Observations 1. **Consistent Superiority:** The "ACC-Collab + (Ours)" method (red line) achieves the highest or near-highest accuracy in all three models by Round 4. 2. **Model-Dependent Performance:** The relative performance of methods varies significantly by base model. For example, the "top cluster" in Gemma-2 performs similarly to the mid-tier methods in Llama-3 and Mistral. 3. **Volatility:** The "ACC-Collab (Ours)" method (orange line) shows notable volatility, especially in the Gemma-2 chart, suggesting its performance may be less stable. 4. **Diminishing Returns:** Most methods show the most significant gains between Round 0 and Round 1, with improvements slowing or plateauing in later rounds. 5. **Baseline Clustering:** The baseline methods (SoM variants, Persona, DebateTune) often cluster together at the lower end of the performance spectrum, particularly in the Mistral and Llama-3 charts. ### Interpretation This data suggests that the proposed collaborative methods, particularly the enhanced "ACC-Collab +", are effective at improving model accuracy through iterative rounds of debate or collaboration. The consistent upward trend of the red line indicates that the method successfully leverages multiple rounds to refine answers. The variation across base models (Llama-3, Mistral, Gemma-2) implies that the effectiveness of these collaborative techniques is influenced by the underlying capabilities or characteristics of the base model. The tight clustering of baselines in Gemma-2 might indicate a performance ceiling for those methods on that specific model, which the ACC-Collab methods are able to break through. The volatility of the standard "ACC-Collab" method, contrasted with the stability of "ACC-Collab +", highlights the importance of the specific enhancements made in the "+" version. These enhancements likely add robustness to the collaborative process. Overall, the charts provide strong evidence for the value of iterative, structured collaboration (like debate) for improving LLM performance on the measured task, with the proposed "ACC-Collab +" framework being the most effective approach shown. The confidence intervals (shaded areas) are generally wider for the top-performing methods, indicating greater variance in their outcomes, which is a trade-off for their higher average performance. </details> Figure 7: Accuracy over five rounds of deliberation on MMLU. <details> <summary>extracted/6248978/plots/new_perRoundBBH.png Details</summary>  ### Visual Description ## Line Charts: Performance of Various Methods Across Three AI Models ### Overview The image displays three side-by-side line charts, each tracking the "Accuracy" (y-axis) of eight different methods over five sequential "Rounds" (x-axis, 0-4). The charts are titled by the base AI model being evaluated: "Llama-3", "Mistral", and "Gemma-2". A shared legend at the bottom identifies the eight methods, which are distinguished by line color and style (solid or dashed). Each data series includes a shaded region, likely representing confidence intervals or standard deviation. ### Components/Axes * **Titles:** Three individual chart titles: "Llama-3" (left), "Mistral" (center), "Gemma-2" (right). * **Y-Axis:** Labeled "Accuracy". The scale varies per chart: * Llama-3: 0.48 to 0.60 (increments of 0.02). * Mistral: 0.40 to 0.60 (increments of 0.05). * Gemma-2: 0.44 to 0.52 (increments of 0.02). * **X-Axis:** Labeled "Round" for all three charts, with markers at 0, 1, 2, 3, and 4. * **Legend:** Located at the bottom of the entire figure, spanning its width. It defines the following eight methods: 1. `SoM (2x)`: Blue dashed line. 2. `SoM (4x)`: Teal dashed line. 3. `Persona`: Purple dashed line. 4. `DebateTune`: Brown solid line. 5. `SFT`: Light green solid line. 6. `DebateGPT`: Dark green solid line. 7. `ACC-Collab (Ours)`: Orange solid line with circular markers. 8. `ACC-Collab + (Ours)`: Red solid line with circular markers. ### Detailed Analysis **Chart 1: Llama-3** * **Trend Verification & Data Points:** * `ACC-Collab (Ours)` (Orange): Starts highest at ~0.555 (Round 0), peaks at ~0.592 (Round 1), then stabilizes around 0.585-0.592. It is the top-performing method throughout. * `ACC-Collab + (Ours)` (Red): Starts at ~0.518, shows a steady upward trend to ~0.573 by Round 4, becoming the second-highest. * `DebateGPT` (Dark Green): Starts at ~0.532, rises to ~0.558 (Round 3), then dips slightly to ~0.551 (Round 4). * `SFT` (Light Green): Starts at ~0.525, rises gently to ~0.548 (Round 3), then dips to ~0.540. * The four dashed-line methods (`SoM (2x)`, `SoM (4x)`, `Persona`, `DebateTune`) cluster tightly between ~0.475 and ~0.515. They all show a sharp initial increase from Round 0 to 1, then plateau. `SoM (4x)` (Teal) appears marginally highest within this cluster by Round 4 (~0.515). **Chart 2: Mistral** * **Trend Verification & Data Points:** * `ACC-Collab + (Ours)` (Red): Dominates this chart. Starts at ~0.562, jumps to ~0.598 (Round 1), and remains flat at ~0.600 through Round 4. * `ACC-Collab (Ours)` (Orange): Starts at ~0.482, rises steadily to ~0.518 (Round 4). It is clearly separated below the red line but above all others. * `DebateGPT` (Dark Green): Starts at ~0.448, rises to ~0.478 (Round 4). * `SFT` (Light Green): Starts lowest at ~0.405, rises to ~0.438 (Round 4). * The dashed-line cluster (`SoM (2x)`, `SoM (4x)`, `Persona`, `DebateTune`) is again tightly grouped, starting between ~0.410-0.440 and ending between ~0.455-0.465. `DebateTune` (Brown) is at the bottom of this cluster. **Chart 3: Gemma-2** * **Trend Verification & Data Points:** * `ACC-Collab (Ours)` (Orange): Starts at ~0.476, shows a strong upward trend to peak at ~0.515 (Round 3), then dips slightly to ~0.513. It is the top performer from Round 2 onward. * `SFT` (Light Green): Starts highest at ~0.493, peaks at ~0.505 (Round 1), then declines to ~0.498 (Round 4). * `DebateGPT` (Dark Green): Starts at ~0.488, remains very flat, ending at ~0.490. * `ACC-Collab + (Ours)` (Red): Starts low at ~0.445, shows a late surge from Round 2 (~0.458) to Round 3 (~0.475), ending at ~0.476. * The dashed-line cluster (`SoM (2x)`, `SoM (4x)`, `Persona`, `DebateTune`) starts between ~0.440-0.448. They show modest gains, ending between ~0.448-0.455. `DebateTune` (Brown) shows a notable dip at Round 2 before recovering. ### Key Observations 1. **Model-Dependent Performance:** The relative effectiveness of the methods varies significantly by base model. `ACC-Collab + (Ours)` is dominant on Mistral, competitive on Llama-3, but underperforms on Gemma-2 until later rounds. `ACC-Collab (Ours)` is consistently strong, leading on Llama-3 and Gemma-2. 2. **Clustering of Baselines:** The four methods represented by dashed lines (`SoM (2x)`, `SoM (4x)`, `Persona`, `DebateTune`) consistently form a low-performing cluster with very similar trajectories across all three models. 3. **Round 0 to 1 Jump:** Nearly all methods show their most significant accuracy gain between Round 0 and Round 1, after which improvements become more gradual or plateau. 4. **Uncertainty Bands:** The shaded confidence intervals are notably wider for the `ACC-Collab` variants (orange and red lines), suggesting higher variance in their performance compared to the more tightly banded baseline methods. ### Interpretation The data suggests that the proposed methods, **ACC-Collab (Ours)** and its variant **ACC-Collab + (Ours)**, generally outperform the baseline techniques (SoM, Persona, DebateTune, SFT, DebateGPT) in multi-round accuracy evaluation across different large language models. The "Ours" label indicates these are the novel contributions of the paper or work from which this figure is taken. The performance hierarchy is not absolute but is **model-sensitive**. This implies that the optimal collaborative or debate strategy may depend on the underlying capabilities or biases of the base model (Llama-3 vs. Mistral vs. Gemma-2). The consistent underperformance and clustering of the dashed-line methods suggest they represent a similar class of simpler or less effective approaches. The significant improvement from Round 0 to Round 1 across the board indicates that even a single round of interaction or refinement provides substantial benefit over the base model's initial output. The plateauing effect suggests diminishing returns with additional rounds for most methods. The wider variance for the ACC-Collab methods might be a trade-off for their higher peak performance, indicating they are potentially more sensitive to initial conditions or input variations. </details> Figure 8: Accuracy over five rounds of deliberation on BBH. <details> <summary>extracted/6248978/plots/new_perRoundSCIQ.png Details</summary>  ### Visual Description ## Line Charts: Performance of Various Methods Across Rounds for Three AI Models ### Overview The image displays three horizontally arranged line charts, each comparing the performance (accuracy) of eight different methods over five rounds (0 to 4) for a specific large language model: Llama-3, Mistral, and Gemma-2. A shared legend is positioned at the bottom of the entire figure. The charts illustrate how accuracy evolves with iterative rounds for each method, with shaded regions indicating confidence intervals or variance. ### Components/Axes * **Chart Titles (Top Center of each subplot):** "Llama-3", "Mistral", "Gemma-2". * **X-Axis (Bottom of each subplot):** Labeled "Round". Markers at integer values: 0, 1, 2, 3, 4. * **Y-Axis (Left of each subplot):** Labeled "Accuracy". The scale differs for each model: * **Llama-3:** Range approximately 0.91 to 0.955. * **Mistral:** Range approximately 0.84 to 0.905. * **Gemma-2:** Range approximately 0.89 to 0.925. * **Legend (Bottom, spanning all charts):** Contains eight entries, each with a distinct line style/color and label: 1. `-- SoM (2x)` (Blue, dashed line) 2. `-- SoM (4x)` (Teal, dashed line) 3. `-- Persona` (Purple, dashed line) 4. `— DebateTune` (Brown, solid line) 5. `— SFT` (Light green, solid line) 6. `— DebateGPT` (Dark green, solid line) 7. `—●— ACC-Collab (Ours)` (Orange, solid line with circle markers) 8. `—●— ACC-Collab + (Ours)` (Red, solid line with circle markers) ### Detailed Analysis **Llama-3 Chart:** * **Trend:** All methods show a general upward trend or plateau from Round 0 to Round 4. * **Top Performers:** `ACC-Collab (Ours)` (orange) starts highest (~0.941) and peaks at Round 3 (~0.953). `ACC-Collab + (Ours)` (red) starts lower (~0.933) but rises sharply to converge near the orange line by Round 2 (~0.948), maintaining a slight lead thereafter. * **Mid-tier:** `DebateGPT` (dark green) shows steady improvement from ~0.925 to ~0.932. * **Lower Tier:** The remaining methods (`SoM (2x)`, `SoM (4x)`, `Persona`, `DebateTune`, `SFT`) are clustered between ~0.915 and ~0.925, showing modest gains. `SFT` (light green) appears to be the lowest-performing method overall. **Mistral Chart:** * **Trend:** Similar upward trajectory, with the top two methods showing the most dramatic improvement. * **Top Performers:** `ACC-Collab + (Ours)` (red) and `ACC-Collab (Ours)` (orange) start around 0.87-0.875. Both rise steeply, with the red line slightly overtaking the orange line by Round 2. They plateau near 0.90-0.905 from Round 2 to 4. * **Mid-tier:** `DebateGPT` (dark green) improves from ~0.85 to ~0.87. `DebateTune` (brown) remains relatively flat around 0.86. * **Lower Tier:** `SFT`, `SoM (2x)`, `SoM (4x)`, and `Persona` are tightly grouped between ~0.84 and ~0.86, with `SFT` starting the lowest (~0.835). **Gemma-2 Chart:** * **Trend:** All methods improve from Round 0, with most plateauing after Round 2. * **Top Performers:** `ACC-Collab (Ours)` (orange) leads throughout, starting at ~0.895 and peaking at ~0.921 at Round 3. `ACC-Collab + (Ours)` (red) follows a similar path but remains slightly below the orange line. * **Notable Mid-tier:** `DebateGPT` (dark green) performs strongly, rising from ~0.89 to ~0.915, closely following the top two. `SFT` (light green) also shows strong improvement, reaching ~0.91. * **Lower Tier:** The dashed-line methods (`SoM (2x)`, `SoM (4x)`, `Persona`) and `DebateTune` are clustered between ~0.89 and ~0.905. `SoM (2x)` (blue dashed) is the lowest-performing method in the later rounds. ### Key Observations 1. **Consistent Superiority:** The two proposed methods, `ACC-Collab (Ours)` and `ACC-Collab + (Ours)`, consistently achieve the highest or near-highest accuracy across all three models and all rounds. 2. **Performance Hierarchy:** A clear hierarchy is visible: the "ACC-Collab" variants > `DebateGPT` > other methods (`SFT`, `DebateTune`, `Persona`, `SoM` variants). 3. **Round-Dependent Improvement:** Accuracy for the top methods improves significantly in the first 1-2 rounds before stabilizing. Lower-performing methods show less dramatic gains. 4. **Model-Specific Scales:** While the relative ranking of methods is similar, the absolute accuracy values differ by model, with Llama-3 achieving the highest overall accuracy (~0.95) and Mistral the lowest starting point (~0.84). 5. **Variance:** The shaded confidence intervals are generally wider for the top-performing methods, especially in early rounds, suggesting more variability in their performance gains. ### Interpretation The data strongly suggests that the authors' proposed collaborative methods (`ACC-Collab` and its enhanced version `ACC-Collab +`) are more effective at improving model accuracy through iterative rounds than the compared baselines (DebateGPT, SFT, Persona, SoM, DebateTune). This effectiveness is robust, holding across three distinct underlying LLM architectures (Llama-3, Mistral, Gemma-2). The trend of rapid early improvement followed by a plateau indicates that the collaborative process yields the most benefit in the initial rounds. The consistent underperformance of standard Supervised Fine-Tuning (`SFT`) and the `SoM` variants highlights the added value of the debate/collaboration framework. The fact that `ACC-Collab +` often starts lower than `ACC-Collab` but catches up or surpasses it (notably in Mistral) might suggest that the "+" variant requires a "warm-up" round but has a higher performance ceiling. The charts provide compelling visual evidence for the efficacy of the proposed approach in enhancing LLM reasoning or task performance through multi-round collaboration. </details> Figure 9: Accuracy over five rounds of deliberation on SCIQ. <details> <summary>extracted/6248978/plots/new_perRoundARC.png Details</summary>  ### Visual Description \n ## Line Charts: Accuracy vs. Round for Three Language Models ### Overview The image displays three side-by-side line charts comparing the performance of eight different methods across five rounds (0-4) for three distinct language models: Llama-3, Mistral, and Gemma-2. The y-axis represents "Accuracy," and the x-axis represents "Round." Each chart includes shaded regions around the lines, likely indicating confidence intervals or standard deviation. A shared legend at the bottom defines the eight methods. ### Components/Axes * **Chart Titles (Top Center):** "Llama-3", "Mistral", "Gemma-2" * **Y-Axis Label (Left Side, Vertical):** "Accuracy" * **X-Axis Label (Bottom Center):** "Round" * **X-Axis Markers:** 0, 1, 2, 3, 4 * **Y-Axis Scales:** * Llama-3: 0.855 to 0.885 (increments of 0.005) * Mistral: 0.81 to 0.86 (increments of 0.01) * Gemma-2: 0.81 to 0.85 (increments of 0.01) * **Legend (Bottom, spanning all charts):** * **SoM (2x):** Blue dashed line (`--`) * **SoM (4x):** Teal dashed line (`--`) * **Persona:** Purple dashed line (`--`) * **DebateTune:** Brown solid line * **SFT:** Light green solid line * **DebateGPT:** Dark green solid line * **ACC-Collab (Ours):** Orange solid line with circular markers * **ACC-Collab + (Ours):** Red solid line with circular markers ### Detailed Analysis #### **Chart 1: Llama-3** * **Trend Verification:** Most methods show an initial increase from Round 0 to Round 1, followed by a plateau or slight fluctuations. The orange line (ACC-Collab) shows the highest peak. * **Data Points (Approximate):** * **ACC-Collab (Ours) [Orange]:** Starts ~0.873 (R0), peaks ~0.882 (R2), ends ~0.881 (R4). * **ACC-Collab + (Ours) [Red]:** Starts lowest ~0.858 (R0), rises sharply to ~0.873 (R1), fluctuates, ends ~0.869 (R4). * **DebateGPT [Dark Green]:** Steady increase from ~0.868 (R0) to ~0.877 (R4). * **SFT [Light Green]:** Starts high ~0.878 (R0), dips slightly, ends ~0.879 (R4). * **DebateTune [Brown]:** Starts ~0.864 (R0), rises to ~0.871 (R1), plateaus. * **SoM (2x) [Blue Dashed]:** Starts ~0.862 (R0), rises to ~0.874 (R1), ends ~0.874 (R4). * **SoM (4x) [Teal Dashed]:** Follows a very similar path to SoM (2x). * **Persona [Purple Dashed]:** Starts ~0.863 (R0), peaks ~0.872 (R1), declines to ~0.870 (R4). #### **Chart 2: Mistral** * **Trend Verification:** The red line (ACC-Collab+) shows a dominant, steep upward trend. The orange line (ACC-Collab) also rises steadily. Other methods show modest gains or remain relatively flat. * **Data Points (Approximate):** * **ACC-Collab + (Ours) [Red]:** Clear top performer. Starts ~0.832 (R0), rises steeply to ~0.856 (R4). * **ACC-Collab (Ours) [Orange]:** Second highest. Starts ~0.822 (R0), rises to ~0.843 (R4). * **DebateTune [Brown]:** Relatively flat, ~0.830 (R0) to ~0.834 (R4). * **SoM (2x) [Blue Dashed]:** Starts ~0.820 (R0), ends ~0.824 (R4). * **SoM (4x) [Teal Dashed]:** Very close to SoM (2x). * **Persona [Purple Dashed]:** Starts lowest ~0.813 (R0), rises to ~0.827 (R4). * **SFT [Light Green]:** Starts ~0.810 (R0), ends ~0.825 (R4). * **DebateGPT [Dark Green]:** Starts ~0.809 (R0), ends ~0.822 (R4). #### **Chart 3: Gemma-2** * **Trend Verification:** All methods show a strong, consistent upward trend from Round 0 to Round 4. The lines are more tightly clustered than in the other charts, especially at later rounds. * **Data Points (Approximate):** * **ACC-Collab (Ours) [Orange]:** Top performer. Starts ~0.821 (R0), rises to ~0.852 (R4). * **ACC-Collab + (Ours) [Red]:** Very close second. Starts ~0.816 (R0), rises to ~0.849 (R4). * **SFT [Light Green]:** Strong performance. Starts ~0.837 (R0), ends ~0.851 (R4). * **DebateGPT [Dark Green]:** Starts ~0.829 (R0), ends ~0.851 (R4). * **DebateTune [Brown]:** Starts ~0.817 (R0), ends ~0.847 (R4). * **Persona [Purple Dashed]:** Starts ~0.816 (R0), ends ~0.844 (R4). * **SoM (4x) [Teal Dashed]:** Starts ~0.815 (R0), ends ~0.843 (R4). * **SoM (2x) [Blue Dashed]:** Starts ~0.815 (R0), ends ~0.841 (R4). ### Key Observations 1. **Method Performance Hierarchy:** The proposed methods, **ACC-Collab (Ours)** and **ACC-Collab + (Ours)**, are consistently among the top performers across all three models. 2. **Model-Specific Behavior:** * **Mistral:** Shows the most dramatic separation between methods. **ACC-Collab +** is the clear, dominant leader. * **Gemma-2:** Shows the strongest overall improvement for all methods, with the highest final accuracy values and the tightest clustering of results. * **Llama-3:** Shows more variability and less dramatic gains after Round 1 for most methods. 3. **Round 0 Baseline:** The starting accuracy (Round 0) varies significantly by model and method, indicating different baseline capabilities before the debate/round process begins. 4. **Trend Consistency:** With few exceptions (e.g., Persona in Llama-3), accuracy either improves or holds steady as rounds progress; no method shows a significant, sustained decline. ### Interpretation This data demonstrates the effectiveness of the **ACC-Collab** family of methods for improving the accuracy of language models through a multi-round process, likely involving debate or collaboration. The key findings are: * **Efficacy of Proposed Method:** The "Ours" methods (orange and red lines) generally outperform or match the other techniques (SoM, Persona, DebateTune, SFT, DebateGPT), suggesting the authors' approach is successful. * **Importance of Model Architecture:** The same methods yield different relative performance gains on different base models (Llama-3, Mistral, Gemma-2). This implies that the effectiveness of collaborative/ debate-based training or inference is not universal but interacts with the underlying model's characteristics. Gemma-2 appears most receptive to these techniques. * **Value of Iteration:** The general upward trend across rounds validates the core premise that iterative refinement (rounds) can boost accuracy. The most significant gains often occur in the first one to two rounds. * **Stability vs. Peak Performance:** **ACC-Collab +** (red) achieves the highest single accuracy score on Mistral but is sometimes outperformed by its non-"+" variant (**ACC-Collab**, orange) on other models. This suggests a potential trade-off between peak performance and consistency across different tasks or models. In summary, the charts provide strong evidence that the authors' collaborative methods enhance model accuracy through iterative rounds, with the degree of improvement being model-dependent. The results position ACC-Collab as a competitive technique in the landscape of methods designed to improve LLM reasoning or output quality. </details> Figure 10: Accuracy over five rounds of deliberation on ARC. ## Appendix C Examples ### C.1 Prompts Here, we provide examples of the prompts used in our experiments. For illustration, we provide prompts for the BoolQ dataset in which agents are asked a yes-no question about a passage. #### Single-Shot Prompt (No Deliberation) prompt = (’You will be given a yes-no question which is based on a passage. ’ ’You should use the passage to help you answer the question. ’ ’You should give a brief justification for your answer, ’ ’and you must provide a final answer of either Yes or No.’ ’\nQuestion: {question}?’ ’\nPassage: {passage}’ ) #### Guided Single-Shot Prompt (No Deliberation) prompt = (’You will be given a yes-no question which is based on a passage. ’ ’You should use the passage to help you answer the question ’ ’with a {target_answer}. ’ ’You should give a brief justification for your answer of {target_answer}, and you must state that your final answer is {target_answer}.’ ’\nQuestion: {question}?’ ’\nPassage: {passage}’ ) #### Deliberation Prompt for Actor prompt = (’Several people have provided answers to a yes-no question. ’ ’Below are their responses:’ ’\nPerson {1} said: {responses[1]}’ ’\nPerson {2} said: {responses[2]}’ . . . ’\nPerson {n} said: {responses[n]}’ ’\n\nYou should take these answers into consideration when answering ’ ’the following yes-no question, which is based on a passage. ’ ’You should give a brief justification for your answer, and you must ’ ’provide a final answer of either Yes or No.’ ’\nQuestion: {question}’ ’\nPassage: {passage}’ ) #### Guided Deliberation Prompt for Actor prompt = (’Several people have provided answers to a yes-no question. ’ ’Below are their responses:’ ’\nPerson {1} said: {responses[1]}’ ’\nPerson {2} said: {responses[2]}’ . . . ’\nPerson {n} said: {responses[n]}’ ’You should take these answers and the passage into consideration when ’answering the following question with {target_answer}. ’ ’You should give a brief justification for your answer of {target_answer}, ’and you must state that your final answer is {target_answer}.’ ’\nQuestion: {question}’ ’\nPassage: {passage}’ ) #### Deliberation Prompt for Critic prompt = (’I am answering a question based on a passage. ’ ’I would like you to help me improve the correctness of my response ’ ’by briefly providing some additional details my original response ’ ’may have missed. ’ ’\nQuestion: {question}’ ’\nPassage: {passage}’ ’\nMy Response: {actor_response}’ ) #### Guided Deliberation Prompt for Critic prompt = (’I would like you to be a deliberation assistant. ’ ’You will be given a question based on a passage and my response ’ ’to the question. ’ ’You should use the passage and my response to develop additional details ’for why the correct answer is {target_answer}. ’ ’Your details must be brief and must support the fact that the ’ ’correct answer is {target_answer}.’ ’\nQuestion: {question}?’ ’\nPassage: {passage}’ ’\nMy Response: {actor_response}’ ) ### C.2 Critic Responses Here, we present several examples of the critic’s responses when the actor provides the wrong answer; critic responses are shown before and after training with ACC-Collab. We observed a qualitative change in the critic’s responses after training with our pipeline; the critic was more willing to disagree with the actor, and the critic provided more detailed feedback. ACTOR: [Wrong] Final answer, No. Non-residents cannot bring a handgun into New York for possession or carry within the state based on the passage. They can only transport it through New York to a destination outside of the state where it is legal to possess and carry. UNTRAINED CRITIC: Your response is mostly correct. I would add that New York has strict gun control laws, so even if a non-resident is allowed to transport a firearm through the state, they should be aware that certain areas may have additional restrictions, such as prohibitions on carrying the firearm in certain public places or parks. It’s always a good idea to check the specific laws of the jurisdiction they will be passing through and their final destination. TRAINED CRITIC: Details refuting your response: 1. The passage mentions that non-residents can transport any lawful firearm through New York, which implies that the firearm is temporarily in New York state territory during transit. This suggests that under certain circumstances, a handgun could be brought into New York for transit purposes, even if it cannot be carried within the state. 2. The passage does not explicitly state that non-residents are prohibited from bringing handguns into New York. Instead, it only specifies that they cannot possess or carry handguns within the state. This ambiguity leaves room for interpretation that a handgun could be brought into New York for transit purposes as long as it is not carried within the state. ACTOR: [Wrong] Answer: Yes, the passage states that in the Northern Atlantic Ocean, the hurricane season starts as early as June 1 and ends on November 30, with the peak of activity occurring from late August through September. However, the passage also mentions an exception where the hurricane season started as early as January 3 in 1938. Therefore, while the typical start of the hurricane season is June 1, it is possible for the season to begin earlier, as indicated by the passage. UNTRAINED CRITIC: Your response is correct and provides a good justification based on the passage. However, you could add that the passage mentions that May is the least active month for tropical cyclone activity worldwide, but May is still within the Atlantic hurricane season. Therefore, while the peak of the Atlantic hurricane season is from late August through September, the season technically starts earlier. TRAINED CRITIC: While your response is correct in stating that the typical hurricane season in the Northern Atlantic Ocean starts on June 1 and ends on November 30, with the peak of activity occurring from late August through September, the passage also emphasizes that each particular basin has its own seasonal patterns. The passage mentions the exception of the 1938 hurricane season starting as early as January 3, which is outside the typical seasonal window. This detail directly refutes your response by emphasizing that while the typical season starts in June, it is not unheard of for the season to begin earlier, as demonstrated by historical records. Another detail from the passage that supports the fact that the answer is No, is that the passage states that ”worldwide, tropical cyclone activity peaks in late summer,” which is contradictory to your response that the hurricane season starts in spring. The passage explicitly states that the peak of tropical cyclone activity occurs in late summer, which is later than spring. ACTOR: [Wrong] Final Answer: (A) rated the group as less interesting and enjoyable. Justification: Aronson and Mills’ study found that severe initiation processes negatively impact immediate attitudes towards the group due to discomfort and negative emotions. (Person 0) Additionally, severe initiation may create pressure to conform and social comparison, making the group less appealing. (Person 1) UNTRAINED CRITIC: Your response is correct. I’d add that Aronson and Mills found that severe initiation rites can create a sense of exclusion and pain, which may negatively impact attitudes towards the group. (A) TRAINED CRITIC: Response: While your justification supports the fact that severe initiation can negatively impact attitudes towards the group, it does not fully refute the alternative answers. Let me provide brief details to argue for the correct answer, B being ”rated the group as more interesting and enjoyable”: 1. Counter-intuitive findings: Research suggests that negative experiences can sometimes lead to stronger group identification and commitment. The discomfort and negative emotions generated during severe initiation may create a sense of camaraderie and solidarity among group members, ultimately increasing their overall enjoyment of the group (Tajfel & Turner, 1986). 2. Long-term benefits: Although severe initiation might result in negative emotions initially, the feelings of bonding and group cohesion that develop during the initiation process can endure and even enhance members’ attachment to the group over time (Sue Carter & Branscombe, 1994). This increased commitment can lead to a heightened sense of enjoyment and engagement with the group. ACTOR: [Wrong] Final Answer: (B) Catalhoyuk and Gobekli Tepe Justification: These sites have provided evidence for social complexity without clear agricultural surpluses. Catalhoyuk had centralized planning and hierarchical social organization but no conclusive evidence for agricultural surpluses. Gobekli Tepe was likely a religious or ceremonial center predating the agricultural revolution. UNTRAINNED CRITIC: Your response is correct. I’d add that Catalhoyuk had a population of up to 8,000 people and was the world’s first known urban settlement, while Gobekli Tepe’s T-shaped pillars date back to around 9600 BCE and suggest the existence of social and religious institutions that predate agriculture. TRAINED CRITIC: While your justification for option (B) is correct, I would like to provide additional details for option (D) to refute potential responses in favor of other options. Option (A): Stonehenge and Caral Justification: Although these sites have evidence of social complexity, there is no clear evidence that they lacked agricultural surpluses. Stonehenge, for instance, is believed to have served as a ceremonial and astronomical center, but its relationship to agriculture remains unclear. Caral, on the other hand, has evidence of irrigation systems, suggesting some agricultural production, even if not a surplus. Option (C): La Venta and Catalhoyuk Justification: La Venta is an example of a complex civilization with evidence of agricultural surpluses. While Catalhoyuk is an example of social complexity without clear agricultural surpluses, as you mentioned, La Venta challenges the idea that social complexity always required surpluses. Option (D): Gobekli Tepe and Poverty Point Justification: Gobekli Tepe is believed to be a religious or ceremonial center that predates the agricultural revolution. Poverty Point, on the other hand, is a complex society that existed around 3200 BCE and had advanced engineering skills but no clear evidence of agricultural surpluses. These sites’ existence undermines the hypothesis that social complexity always required agricultural surpluses, providing further evidence for option (D) being the correct answer. ACTOR: [Wrong] Final Answer: (A) psychological reactance. This is because your friend’s statement suggests that they feel frustrated or resistant when external factors prevent them from going on vacation, indicating a desire to have more freedom to make their own choices. Psychological reactance is the tendency to react negatively when our freedom is threatened. UNTRAINED CRITIC: Your response is correct. I’d add that psychological reactance often results in an exaggerated perception of the importance or severity of the obstacles, leading your friend to believe that there’s always a crisis preventing them from vacationing when in reality, the obstacles may not be as significant as they seem. TRAINED CRITIC: While your response is plausible and accurately describes psychological reactance, let me provide two brief reasons why the correct answer is still (C) illusory correlation: 1. Illusory correlation refers to the perception of a relationship between variables when none or only a weak relationship actually exists. In this case, your friend might perceive a strong correlation between planning a vacation and experiencing crises at home or work, leading them to believe that their vacations are consistently disrupted, even if the actual correlation is weak or non-existent. 2. Psychological reactance is primarily concerned with an individual’s desire to react negatively when their freedom is restricted or threatened, not necessarily a misperception of causal relationships. However, the friend’s statement does imply a belief that external factors are consistently preventing them from going on vacations, which could be a manifestation of illusory correlation. ACTOR: [Wrong] Final Answer: (D) a zygote. Justification: A zygote is the first cell formed when two gametes merge, marking the beginning of organismic development. UNTRAINED CRITIC: Your response is correct. Just to add, a zygote is formed by the fusion of an sperm (male gamete) and an egg (female gamete), resulting in a diploid cell containing genetic material from both parents. This is the initial stage of an organism’s development. TRAINED CRITIC: I agree with your response that the answer is (D) a zygote. However, to further support why (A) a gamete and (C) a nucleus are incorrect, let me provide brief details: (A) A gamete is a reproductive cell that fuses with another gamete during fertilization to form a zygote. Therefore, it is not an organism in the earliest stages of development but rather a part of the process leading to the formation of an organism. (C) A nucleus is an organelle within a cell that contains genetic material. It is not an organism but rather a component of a cell, and it is present in the zygote after fertilization has occurred. So, both (A) and (C) can be ruled out as they do not represent the organism in the earliest stages of development, and the correct answer is (D) a zygote. ACTOR: [Wrong] Final Answer: (A) metamorphic rocks and (B) any types of rocks (specifically igneous rocks when they undergo metamorphosis). UNTRAINED CRITIC: Your response is mostly correct, but it would be more precise to clarify that options B and C also include metamorphic rocks when they mention ”any types of rocks” and ”igneous rocks,” respectively. Metamorphic rocks form from the alteration of pre-existing rocks under heat, pressure, or chemical conditions. Thus, all three options (A, B, and C) are correct, but option A is the most explicit about the transformation process involved. TRAINED CRITIC: Sure! Here are two brief details supporting the correct answer (A) metamorphic rocks and (B) any types of rocks (specifically igneous rocks when they undergo metamorphosis): 1. Metamorphic rocks result from the transformation of pre-existing rocks under high pressure and temperature conditions. This process causes the minerals in the rocks to change, resulting in a new type of rock. For example, sedimentary rocks like shale or limestone can be transformed into metamorphic rocks like slate or marble, respectively. 2. Igneous rocks can also undergo metamorphosis when they are exposed to high pressure and temperature conditions. This process results in the recrystallization of minerals and the formation of new textures. For example, granite can be transformed into gneiss or schist when subjected to metamorphic conditions. Therefore, any type of rock, including igneous rocks, can change and become new types of rocks through metamorphosis. ACTOR: [Wrong] Final Answer: (C) white present Justification: Gertrude initially had a yellow present, but she swapped it with Alice, who had a white present. Therefore, Gertrude now has the white present. UNTRAINED CRITIC: Your response is correct. I’d add that after Gertrude swaps with Alice, none of the following swaps involve Gertrude giving away or receiving the white present. TRAINED CRITIC: While your justification is correct that Gertrude initially had a yellow present and swapped it with Alice’s white present, I would like to add two additional details that refute your response and support option D, brown present. First, prior to swapping with Alice, Gertrude swapped her yellow present with Dave’s brown present. Second, in the final round of swaps, Fred and Eve swap their presents, and Gertrude’s white present (which she received from Alice) is not mentioned in the swap between Fred and Eve. Therefore, based on the information provided, Gertrude could still have the brown present that she received from Dave earlier in the game. ACTOR: [Wrong] Final Answer: None of the given options are correct. The construction site was closed during the hours Thomas was engaged in the mentioned activities, and there is no information indicating that Thomas visited the construction site before 6am. UNTRAINED CRITIC: Your response is correct. I would add that Thomas’ activities from 6am to 8am (gym, yoga studio, and movie theater) all occurred before the construction site opened. Therefore, option (D) can be eliminated as well. TRAINED CRITIC: While it’s true that none of the given options are correct, I’d like to provide some brief additional details to support the fact that the correct answer is D (6am to 8am), if that was an option. Here’s why: 1. Thomas was seen at the gym from 8am to 11am, and at the yoga studio from 11am to 3pm, meaning he was engaged in those activities between those hours. There’s no information indicating that he left those places earlier or returned later. 2. He was seen at the movie theater from 3pm to 6pm, and at the Leaning Tower of Pisa from 6pm to 7pm. Again, there’s no information indicating that he left the movie theater earlier or arrived at the Leaning Tower of Pisa later. 3. The construction site was closed after 7pm. However, the options don’t include any time before 7pm. If Thomas could have gone to the construction site before 7pm, the correct answer would be one of the options A, B, or C. 4. Since none of the options include the time before 7pm, and Thomas was engaged in activities during the entire day, the only time left for him to visit the construction site, assuming he did, would be before 7am. However, the options don’t include that time frame. Therefore, if we had the option D (6am to 8am), it would be the correct answer, as it’s the only time frame that fits the given information. In conclusion, while the given options are not correct, if option D (6am to 8am) was an option, it would be the correct answer based on the information provided.