2411.00053

# ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration **Authors**: - Andrew Estornell (ByteDance Research) - &Jean-François Ton (ByteDance Research) - &Yuanshun Yao (Meta GenAI) - &Yang Liu (University of California, Santa Cruz) > Equal contribution. Correspondence to{andrew.estornell,jeanfrancois}@bytedance.comWork done while at ByteDance Research Abstract Large language models (LLMs) have demonstrated a remarkable ability to serve as general-purpose tools for various language-based tasks. Recent works have demonstrated that the efficacy of such models can be improved through iterative dialog between multiple models. While these paradigms show promise in improving model efficacy, most works in this area treat collaboration as an emergent behavior, rather than a learned behavior. In doing so, current multi-agent frameworks rely on collaborative behaviors to have been sufficiently trained into off-the-shelf models. To address this limitation, we propose ACC-Collab, an A ctor- C riti c based learning framework to produce a two-agent team (an actor-agent and a critic-agent) specialized in collaboration. We demonstrate that ACC-Collab outperforms SotA multi-agent techniques on a wide array of benchmarks. 1 Introduction footnotetext: Code available at https://github.com/LlenRotse/ACC-Collab Recently, large language models (LLMs) have rapidly become a cornerstone in various applications, redefining how we process and generate language at scale (Thirunavukarasu et al., 2023; Hadi et al., 2023; Jiang et al., 2024). Their ability to handle diverse tasks, from translation (Zhu et al., 2024; Otter et al., 2020) to answering complex questions (Zhang et al., 2024; Hao et al., 2024; Havrilla et al., 2024), has attracted the attention of both industry as well as academia. However, despite these advancements, LLMs still exhibit notable weaknesses, particularly when it comes to answering factual questions and reasoning (Tonmoy et al., 2024; Rawte et al., 2023; Huang et al., 2023). To address these limitations, several techniques have been proposed, such as Chain-of-Thought (CoT) prompting (Wei et al., 2022), Self-Reflection (Ji et al., 2023; Shinn et al., 2023), and Multi-Agent Debate (MAD) (Du et al., 2023), to name a few. These approaches aim to improve the reasoning abilities of LLMs by guiding them toward more accurate answers through structured thinking or discourse. However, the majority of these techniques do not involve training the model specifically for these tasks but instead rely on zero-shot or few-shot capabilities. Similar to most multi-agent paradigms, MAD approaches make use of off-the-shelf general-purpose LLMs, which are not trained to collaborate. Such approaches rely on collaboration as an emergent, rather than a learned, behavior. While, in some cases, these emergent behaviors are sufficient, the question remains: Can these methods be improved by imbuing models directly with collaborative abilities? To answer this, we propose training teams of LLMs to solve tasks collaboratively. A particularly relevant work is DebateGPT (Subramaniam et al., 2024), which employs debate as a mechanism to generate higher-quality fine-tuning data. Unlike our approach, which optimizes LLMs for multi-round collaborative problem-solving, their method focuses on using debate to enhance training data for a single model that produces individual responses. In this paper, we propose a novel framework A ctor- C riti c Collaboration (ACC-Collab) which jointly trains a two-agent team to collaboratively solve problems through iterative conversation; this team consists of an actor-agent, responsible for providing answers for a given task, and a critic-agent, responsible for assisting the actor-agent with feedback on its answers. In our training pipeline, we introduce a novel off-policy learning scheme called ”Guided-Collaboration” to generate high-quality multi-turn training data to enhance the actor’s and critic’s performance on challenging tasks. To summarize, our contributions are as follows: - We are the first to propose a framework for jointly training a team of LLM agents (Actor-Critic) within the context of collaborative problem solving. - We introduce a novel data generation scheme, “Guided Collaboration Trajectories”, which enables the efficient creation of high-quality training data for both the actor and critic roles. - Our extensive experiments demonstrate that our method, ACC-Collab, significantly outperforms existing state-of-the-art approaches. 2 Related Work Our research is closely related to the emerging field of multi-agent deliberation, sometimes called Multi-Agent Debate (MAD), which examines how to use groups of models to solve tasks through iterative discussion Chan et al. (2023); Liang et al. (2023); Du et al. (2023); Li et al. (2023c); Khan et al. (2024); Michael et al. (2023); Rasal (2024); Pham et al. (2023); Abdelnabi et al. (2023); Hong et al. (2023); Irving et al. (2018); Li et al. (2023b; d; 2024a); Wang et al. (2023a); Zhang et al. (2023). Many of these works find that language models have naturally collaborative abilities Singhal et al. (2023); Du et al. (2023); Chan et al. (2023), while others have noted that the collaborative ability of off-the-shelf models can be quite limited Wang et al. (2024); Smit et al. . Current approaches to multi-agent deliberation can be broadly cast into two main categories: those that modify model prompts and responses during the discussion Liang et al. (2023); Khan et al. (2024); Rasal (2024); Feng et al. (2024); Yang et al. (2024), and those that modify the structure of the deliberation process Li et al. (2023a); Hong et al. (2023); Liu et al. (2023); Li et al. (2024c); Wang et al. (2023b); Wu et al. (2023); Chen et al. (2023); Chang (2024b). Importantly, both categories use off-the-shelf language models (which have not been trained to collaborate) and work by modifying either the inputs or outputs of these models. Deviating from this line of work, we aim to specifically train a team of models to collaboratively solve tasks. Two works of particular note are that of Subramaniam et al. (2024), which proposes to use debate data to fine-tune models, and Li et al. (2024b), which trains models for adversarial debate. In the former, debate is used to generate higher-quality fine-tuning data and is not used at inference time; differing from this work, we train models directly to collaborate and use multi-agent discussion both during training and inference. In the latter, models are trained to be effective arguers rather than collaborators, i.e., models are trained to give conceiving arguments such that they can win a debate against other LLMs. Differing from this work, we train models to collaboratively solve tasks. In the context of multi-agent deliberation, the concept of divergent opinions is highly relevant to our method. Several approaches to multi-agent deliberation aim to control the level of disagreement among the agents Liang et al. (2023); Khan et al. (2024); Chang (2024a). Often, these works dynamically increase disagreement to prevent early convergence of deliberation. In our study, we leverage divergent opinions to generate high-quality training data. In particular, we have agents change their opinion during the discussion and measure whether or not that change increases or decreases the likelihood that the agents’ discussion converges to a correct answer. Using this signal we can then asses the value of a given training example for training the models. Also closely related to our work are paradigms that aim to use self-generated data to improve model performance, often in the context of reasoning or chain of thought Trung et al. (2024); Huang et al. (2024); Xiong et al. (2024); Chen et al. (2024); Pang et al. (2024b). Similar to this line of research, we make use of model generations as training data. However, we are the first work to use such data in the context of multiple models debating collaboratively to solve a given task. 3 Preliminaries and notation In this section, we formalize multi-agent collaboration between an Actor (an agent that provides answers) and a Critic (an agent that provides feedback to the actor) while also introducing the notation that will be used throughout the remainder of the paper. Let $(x,y)\sim\mathcal{D}$ be a task-answer pair source from a distribution of tasks and answers $\mathcal{D}$ . For a given task $x$ , two agents – an actor agent responsible for providing answers and a critic agent responsible for providing feedback and assistance to the actor agent – engage in an iterative discussion over $T$ rounds, to correctly infer the answer $y$ . Let $\theta_{a}$ and $\theta_{c}$ be the parameters of actor and critic agent, respectively. The iterative discussion between these two agents is as follows: 1. At round $t=0$ a task $x$ is given to the actor $\theta_{a}$ who provides an initial response $z_{a}^{(0)}$ . 1. Next, still at round $t=0$ , the critic $\theta_{c}$ views task $x$ and $z_{a}^{(0)}$ , then provides feedback $z_{c}^{(0)}$ . 1. For each round $t>0$ , the actor views the task $x$ , its own previous response $z_{a}^{(t-1)}$ and the critic’s feedback $z_{c}^{(t-1)}$ , then provides an updated response $z_{a}^{(t)}$ . 1. After the actor’s new response $z_{a}^{(t)}$ , the critic provides the feedback $z_{c}^{(t)}$ based on $z_{a}^{(t)}$ . The accuracy of this procedure is measured via the correctness of the actor’s final response, i.e., $\mathbb{I}\big{[}\zeta(z_{a}^{(T)})=y\big{]}$ . Where $\zeta$ is a function that extracts answers from text-based responses. For example if $z_{a}^{(T)}=$ “The sky is blue”, then $\zeta(z_{a}^{(T)})=$ “blue”. With this notation and formalization of multi-agent collaboration, we introduce our framework for training actor-critic teams. <details> <summary>extracted/6248978/plots/mainMethod_v4.png Details</summary>  ### Visual Description ## Diagram: Actor-Critic Model with Preference Data ### Overview The image illustrates a diagram of an actor-critic model that incorporates preference data for training. The diagram shows the flow of information between the actor and critic, and how their interactions influence the generation of preference data used to train the models. ### Components/Axes * **Actor:** Represented by an orange robot icon. * `z_a^(t-1)`: Actor's state at time t-1 (orange box with orange outline). * `z_a^(t)`: Actor's state at time t (green, orange, or red box with matching outline). * **Critic:** Represented by a blue robot icon. * `z_c^(t-1)`: Critic's state at time t-1 (blue box with blue outline). * `z_c^(t)`: Critic's state at time t (blue box with blue outline). * **Actor Guided Towards y:** Text label indicating the actor's actions are guided towards a target 'y'. * **Natural Debate:** Text label indicating a natural debate between the actor and critic. * **Actor Guided Away From y:** Text label indicating the actor's actions are guided away from a target 'y'. * **Δy:** Green circle containing the delta y symbol. * **Δ!y:** Red circle containing the delta !y symbol. * **Relative Quality of Trajectory:** Text label indicating the relative quality of the trajectory. * **Preference Data:** Represents a database or storage of preference data. Contains tuples of actor states: `(z_a^(t), z_a^(t))` * **Train Models:** Indicates the training of models using the preference data. * **ε:** Epsilon, a threshold value. * **if Δy ≥ ε:** Conditional statement. * **elif Δ!y ≥ ε:** Conditional statement. ### Detailed Analysis 1. **Actor and Critic Interaction:** * The diagram starts with the Actor and Critic at time `t-1`, represented by `z_a^(t-1)` (orange) and `z_c^(t-1)` (blue) respectively. * The Actor's actions are guided "Towards y" and "Away From y", while a "Natural Debate" occurs between the Actor and Critic. * At time `t`, the Actor's state `z_a^(t)` can be green, orange, or red, depending on the guidance. The Critic's state `z_c^(t)` remains blue. 2. **Trajectory Evaluation:** * The trajectories are evaluated based on "Relative Quality of Trajectory", resulting in `Δy` (green) and `Δ!y` (red). 3. **Preference Data Generation:** * If `Δy ≥ ε`, the tuple `(z_a^(t), z_a^(t))` (where `z_a^(t)` is green) is added to the "Preference Data". * If `Δ!y ≥ ε`, the tuple `(z_a^(t), z_a^(t))` (where `z_a^(t)` is red) is added to the "Preference Data". * The preference data consists of pairs of actor states at time t. 4. **Model Training:** * The "Preference Data" is used to "Train Models", resulting in updated Actor (orange) and Critic (blue) models. ### Key Observations * The Actor's state at time `t` (`z_a^(t)`) can be in one of three states (green, orange, or red), while the Critic's state at time `t` (`z_c^(t)`) is always blue. * The preference data is generated based on the relative quality of the trajectory, as determined by `Δy` and `Δ!y`. * The conditional statements `if Δy ≥ ε` and `elif Δ!y ≥ ε` determine which trajectories are used to generate preference data. ### Interpretation The diagram illustrates a reinforcement learning process where an actor and critic interact to generate trajectories. The relative quality of these trajectories is evaluated, and preference data is created based on this evaluation. This preference data is then used to train the actor and critic models. The use of preference data allows the models to learn from successful and unsuccessful trajectories, improving their performance over time. The thresholds `ε` determine the sensitivity of the preference data generation process. </details> Figure 1: ACC-Collab training pipeline, exemplified for the actor. 1) We generate data from both natural deliberation as well as guided deliberation towards and away from the ground truth answer $y$ using the actor and critic. 2) We compute the relative quality of each trajectory based on the expected quality difference $\Delta_{y},\Delta_{!y}$ w.r.t. to the natural response. 3) We store all high-quality pairwise data in our database and train the actor agent. 4) We alternate this procedure for the actor and critic. See Figure 5 of the supplement for the corresponding procedure applied to the critic. 4 Methodology In this section, we outline our procedure for training a two-agent team, consisting of an actor agent $f_{\theta_{a}}$ (responsible for providing answers to a given task $x$ ) and a critic agent $f_{\theta_{c}}$ (responsible for providing feedback and assistance to the actor). At inference time, the two trained agents engage in iterative discussion to solve a given task $x$ , generating the final response $z_{a}^{(T)}$ . 4.1 An Actor-Critic Collaboration Framework Building upon our established notation from the previous section and the general actor-critic framework, we formally define our optimization objective as follows For clarity, we note that the term $\arg\max\limits_{\theta_{a}}\max\limits_{\theta_{b}}$ captures the solution for both parameters $\theta_{a},\theta_{c}$ in the corresponding bi-level max-max optimization. Here $\theta_{c}$ is in $z_{c}^{(T-1)}=f_{\theta_{c}}\big{(}x,z_{a}^{(T-1)}\big{)}$ : $$ \displaystyle\theta_{a}^{*},\,\theta_{c}^{*}=\arg\max_{\theta_{a}}\max_{\theta% _{c}}\mathbb{E}_{(x,y)\sim D}\bigg{[}\zeta\bigg{(}\underbrace{f_{\theta_{a}}% \big{(}x,z_{a}^{(T-1)},z_{c}^{(T-1)}\big{)}}_{\text{actor's final response}~{}% z_{a}^{(T)}}\bigg{)}=y\bigg{]} \tag{1} $$ Intuitively, Eq. 1 aims to simultaneously optimize the actor’s parameters $\theta_{a}$ and the critic’s parameters $\theta_{c}$ , ensuring that the actor’s final output at iteration $T$ matches the correct answer $y$ . In other words, we optimize the accuracy of the actor’s response at time $T$ , namely $$ z_{a}^{(T)}=f_{\theta_{a}}\left(x,z_{a}^{(T-1)},z_{c}^{(T-1)}\right), $$ where accuracy is measured as $\mathbb{E}\big{[}\zeta\big{(}z_{a}^{(T)}\big{)}=y\big{]}$ . It is important to note that the recursive nature of multi-agent deliberation introduces significant complexity to the optimization process. Each response $z_{a}^{(t)}$ depends not only on the actor’s previous output $z_{a}^{(t-1)}$ but also on the critic’s previous output $z_{c}^{(t-1)}$ . This interaction closely resembles a cooperative dynamic Stackelberg game (Li & Sethi, 2017), where two players engage in hierarchical decision-making over time, leading us to adopt an iterative best-response approach (Fiez et al., 2019). In other words, we first train the critic agent, followed by training the actor to best respond to the critic’s output. We can then update the critic to adapt to the newly trained actor, and so on. More formally, this process works by first fixing $\theta_{a}$ , and solving, $$ \displaystyle\theta_{c}^{*}=\arg\max_{\theta_{c}}\mathbb{E}_{(x,y)\sim D}\bigg% {[}\zeta\bigg{(}f_{\theta_{a}}\bigg{(}x,z_{a}^{(T-1)},~{}\underbrace{f_{\theta% _{c}}\big{(}x,z_{a}^{(T-1)}\big{)}}_{\text{critic's response}~{}z_{c}^{(T-1)}}% \bigg{)}\bigg{)}=y\bigg{]} \tag{2} $$ then fixing $\theta_{c}^{*}$ from above, we solve $$ \displaystyle\theta_{a}^{*}=\arg\max_{\theta_{a}}\mathbb{E}_{(x,y)\sim D}\bigg% {[}\zeta\bigg{(}f_{\theta_{a}}\bigg{(}x,z_{a}^{(T-1)},~{}f_{\theta_{c}^{*}}% \big{(}x,z_{a}^{(T-1)}\big{)}\bigg{)}\bigg{)}=y\bigg{]} \tag{3} $$ this process then repeats until a desired stopping criteria is reached. In practice, we find that a single iteration is sufficient to produce a high-quality collaborative team. While this alternating scheme allows us to optimize the actor and critic separately, the objectives of each agent still cannot be optimized directly due to the recursive nature of agent responses in this objective; responses at round $T$ depend on those given by the agent at round $t-1$ which themselves depend on the response given at round $t-2$ and so on. To deal with this temporal dependency, we next introduce the concept of Partial Trajectory rewards, which will allow us to capture the signal of each response $z^{(t)}$ for each $t≤ T$ . 4.2 Partial Trajectory Reward To address the inter-round dependencies of the above optimization, we proposed a scheme that allows us to determine the “goodness” of a given response $z^{(t)}$ (from either the actor or the critic) for any $t≤ T$ . Consider a conversation between the actor and the critic that was paused at time $t$ , i.e., the most recent response is $z^{(t)}$ . To assess the goodness of $z^{(t)}$ , one might ask how likely the deliberation procedure will converge to the correct answer $y$ at round $T$ , given that the procedure is already at response $z^{(t)}$ . Formally, we can define this as $$ \displaystyle r(z^{(t)},x,y)=\mathbb{E}\bigg{[}\zeta(z_{a}^{(T)})=y|~{}x,z^{(t% )}\bigg{]} \tag{4} $$ Intuitively, the partial reward captures the expectation of arriving at the correct answer $y$ through deliberation starting at round $t$ with generation $z^{(t)}$ . In practice, $r(z^{(t)},x,y)$ can be estimated by learning the reward $r$ or by using heuristics such as one-step roll-out, i.e., Monte Carlo estimation. In our experiments, we use one-step roll-out heuristics, i.e. simulating an additional deliberation round multiple times from response $z^{(t)}$ . The reward $r(z^{(t)},x,y)$ is set as the average accuracy of these simulations. Empirically, we find this approach effective for generating high-quality training data. We leave learning-based reward functions for future work. Our objective will then be to optimize the parameters of the actor and critic, $\theta_{a},\theta_{c}$ , so that the responses produced by these agents at each timestep $t$ , namely $z^{(t)}$ , maximize $r(z^{(t)},x,y)$ . That is, we optimize the actor and the critic so that at each timestep $t$ , they give a response $z^{(t)}$ which has a high probability of leading the deliberation to converge to the correct answer at time $T$ . To optimize the objective in Eq. 1, we will utilize preference optimization, a standard technique in LLM training. Using the iterative maximization scheme described above, we first have to gather pairwise preference data for both the actor and the critic. In the following sections, we first detail our process for generating this preference data before delving into the optimization procedure. Data: Actor and critic: $\theta_{a},\theta_{c}$ , Distribution of tasks $\mathcal{D}$ , Reward threshold $\varepsilon$ Result: A dataset of trajectories $D$ $D←\emptyset$ /* Set of trajectories to use */ for $(x,y)\sim\mathcal{D}$ do $\mathbf{z}^{(0)}←$ OneDeliberationRound $(x)$ /* actor and critic responses, i.e. $\mathbf{z}=\langle z_{a}^{(0)},z_{c}^{(0)}\rangle$ */ for $t$ in $[1,T]$ do $\mathbf{z}^{(t)}←$ OneDeliberationRound $(x,\mathbf{z}^{(t-1)})$ /* updated natural responses*/ /* guided-deliberation towards, and away from, correct answer $y$ */ $\mathbf{z}_{+}^{(t)}←$ OneGuidedDeliberationRound $(x,\mathbf{z}^{(t-1)},y)$ $\mathbf{z}_{-}^{(t)}←$ OneGuidedDeliberationRound $(x,\mathbf{z}^{(t-1)},!y)$ /* Estimate final round accuracy if deliberation continue from response $z^{(t)}$ , i.e. $r(z^{(t)},x,y),~{}r(z_{+}^{(t)},x,y),~{}r(z_{-}^{(t)},x,y)$ */ $v←$ EstimateFinalAccuracy $(\mathbf{z}^{t})$ $v_{+}←$ EstimateFinalAccuracy $(\mathbf{z}_{+}^{t})$ $v_{-}←$ EstimateFinalAccuracy $(\mathbf{z}_{-}^{t})$ /* Compute the expected improvement for each trajectory */ /* Save trajectory pairs that result in sufficient accuracy improvement */ if $v_{+}-v≥\varepsilon$ then $D$ .add $\big{(}\text{pos=}\mathbf{z}_{+}^{(t)},\text{neg=}\mathbf{z}^{(t)}\big{)}$ end if else if $v-v_{-}≥\varepsilon$ then $D$ .add $\big{(}\text{pos=}\mathbf{z}^{(t)},\text{neg=}\mathbf{z}_{-}^{(t)}\big{)}$ end if end for end for Algorithm 1 Trajectory generation and selection 4.3 Off-Policy Trajectory Generation In this section, we describe how to generate the preference data needed to optimize the objective in Eq. 1. The classification of a sample as positive or negative is determined by the deliberation trajectory it follows. Specifically, a positive sample for training the actor corresponds to a trajectory likely to lead to the correct answer at round $T$ , while a negative sample corresponds to one that leads to an incorrect answer at round $T$ . Intuitively, we aim to push the actor agent to generate responses that lead to correct answers while reducing responses that are unlikely to do so, thus optimizing for Eq. 3. The same principle applies to the critic when optimizing Eq. 2. With this intuition in mind, we now describe how such data is generated. A deliberation trajectory can be defined as a sequence of responses $\langle z_{a}^{(0)},z_{c}^{(0)},z_{a}^{(1)},z_{c}^{(1)},...,z_{a}^{(T)},z_{% c}^{(T)}\rangle$ for a given task $x$ . A straightforward way to generate preference data would be to generate multiple rollouts at each round and select the trajectories with the highest $r(z^{(t)},x,y)$ as positive samples and those with the lowest $r(z^{(t)},x,y)$ as negative samples. This approach could enforce the desired behavior for both the actor and the critic if enough samples are collected. However, this approach is not without its limitations. In particular, if the agent performs poorly on a given dataset, it may be difficult to collect enough positive samples, resulting in low training signals. Additionally, even if the agent performs adequately, generating sufficient responses for both the actor and critic requires significant computational resources, especially to ensure that high $r(z^{(t)},x,y)$ values are used for positive samples and low values for negative samples. 4.4 Guided Collaborative Trajectories To address these limitations and improve efficiency, we propose Guided-Collaborative Trajectories, which steer the deliberation procedure in two opposing directions: one towards, and another away from, the correct. By comparing these guided trajectories with the natural deliberation trajectory, we can assess the relative goodness of each trajectory using an estimation of the reward structure $r$ . Specifically, for task $x$ with answer $y$ , let $\mathbf{z}^{(t-1)}=(z_{a}^{(t-1)},z_{c}^{(t-1)})$ be the agents’ responses at time $t-1$ . Let $(z_{a}^{(t)},z_{c}^{(t)})$ be the agents’ natural responses (i.e., without guidance), let $(z_{y,a}^{(t)},z_{y,c}^{(t)})$ and $(z_{!y,a}^{(t)},z_{!y,c}^{(t)})$ be the agents responses when guided towards, and away from, supporting answer $y$ respectively. Thus, each guided response is an off-policy generation. In practice, we want guided responses to be different enough from natural responses so that learning the guided responses results in consequential changes to the agent, but not so different that they are challenging to learn; we find that prompt modification is an effective tool for striking this balance. To guide the generations of $(z_{y,a}^{(t)},z_{y,c}^{(t)})$ and $(z_{!y,a}^{(t)},z_{!y,c}^{(t)})$ , we will simply provide a correct and wrong target answer in the prompt, respectively - see “Guided Collaborative Trajectory Prompts” in Section C for further details. For each guided response, we consider how influential this response was in altering the accuracy of the final response, i.e., in the case of the actor, we define | | $\displaystyle\Delta_{y}=r(z_{y,a}^{(t)},x,y)-r(z_{a}^{(t)},x,y)\qquad\text{and% }\qquad\Delta_{!y}=r(z_{a}^{(t)},x,y)-r(z_{!y,a}^{(t)},x,y)$ | | | --- | --- | --- | The terms $\Delta_{y}$ and $\Delta_{!y}$ give the expected accuracy difference if at round $t$ the actor had given response $z_{y,a}^{(t)}$ (or $z_{!y,a}^{(t)}$ ) instead of response $z_{a}^{(t)}$ . Large $\Delta_{y}$ indicates that a one-response difference during the deliberation was sufficient to push the procedure toward the correct answer. Such responses would be desirable for the agent to learn. On the other hand, large values of $\Delta_{!y}$ indicate that a one-response difference easily causes the agents to converge to the incorrect answer; this indicates that the deliberation procedure is particularly fragile at timestep $t$ . With these observations in hand, we use $\Delta_{y}$ and $\Delta_{!y}$ to define positive an negative examples, in particular for a threshold $\varepsilon$ , $$ \displaystyle(z_{+}^{(t)},z_{-}^{(t)})=\begin{cases}(z_{y}^{(t)},z^{(t)})&% \text{if}\quad\varepsilon\leq\Delta_{y}=r(z_{y,a}^{(t)},x,y)-r(z_{a}^{(t)},x,y% )\\ (z^{(t)},z_{!y}^{(t)})&\text{if}\quad\varepsilon\leq\Delta_{!y}=r(z_{a}^{(t)},% x,y)-r(z_{!y,a}^{(t)},x,y)\end{cases} \tag{5} $$ if neither value is above the threshold, then the example is thrown out. **Remark 1** *Under Eq. 5, a positive example $z_{+}^{(t)}$ can be interpreted as a guided response $z_{y}^{(t)}$ which increased the probability of deliberation converging to the correct answer by at least $\varepsilon$ , when compared with the natural response $z^{(t)}$ . Similarly, a negative example $z_{-}^{(t)}$ is a guided response $z_{y!}^{(t)}$ which decreased the probability of deliberation converging to the answer by at least $\varepsilon$ .* Now that we have a procedure for generating high-quality training examples consisting of positive and negative pairs, we next discuss how to use those positive and negative pairs to train both agents. 4.5 Learning on Guided Trajectories In order to optimize each objective (Eq. 2 and 3), we use standard preference optimization Direct Preference Optimization (DPO) Rafailov et al. (2024). We choose DPO for its efficiency, but any preference optimization scheme could be used (see section B of the supplement for details on how other preference optimization schemes may be incorporated). Hence, given a preference dataset of positive and negative examples for both the actor and critic agent, of the from $z_{-}^{(t)},z_{+}^{(t)}$ , the DPO loss is defined as, $$ \displaystyle\mathcal{L}_{\text{DPO}}=\sum_{t=0}^{T}\mathbb{E}_{(x,y,z_{-}^{(t% )},z_{+}^{(t)})\sim D}\bigg{[}\log\sigma\bigg{(}\frac{\pi_{\theta}\big{(}z_{+}% ^{(t)}|x,\mathbf{z}^{(t-1)}\big{)}}{\pi_{\text{ref}}\big{(}z_{+}^{(t)}|x,% \mathbf{z}^{(t-1)}\big{)}}-\frac{\pi_{\theta}\big{(}z_{-}^{(t)}|x,\mathbf{z}^{% (t-1)}\big{)}}{\pi_{\text{ref}}\big{(}z_{-}^{(t)}|x,\mathbf{z}^{(t-1)}\big{)}}% \bigg{)}\bigg{]} \tag{6} $$ Where $\pi_{\theta}$ is the policy induced by parameters $\theta_{a}$ or $\theta_{c}$ and $\mathbf{z}^{(t-1)}$ are the agent’s responses at the previous round (i.e., the responses prior to giving either response $z_{-}^{(t)}$ or $z_{+}^{(t)}$ ). **Remark 2** *Recall, given our generated preference data, this loss implicitly optimizes the reward $r(z^{t},x,y)$ which itself is equivalent to final accuracy (i.e., the quantity being maximized by Eq. 1). By summing across all rounds, we implicitly maximize the probability that each round $t$ yields a response $z^{t}$ which causes deliberation to converge to the correct answer at time $T$ .* 5 Experiments Benchmarks To evaluate the efficacy of ACC-Collab we make use of 5 standard benchmark tasks: BoolQ Clark et al. (2019) ${\sim}$ 12k yes-no reading comprehension questions, MMLU Hendrycks et al. (2020) ${\sim}$ 15k multiple choice questions covering a wide array of subjects and difficulty, BBH Suzgun et al. (2022) ${\sim}$ 5k mixed-type questions SCIQ Welbl et al. (2017) ${\sim}$ 13k multiple-choice science questions, ARC Chollet (2019) ${\sim}$ 7k multiple-choice reasoning-based questions. Baselines We compare ACC-Collab to several multi-agent and single-agent baselines. For inference-based methods, we compare to Society of Minds SoM Du et al. (2023), Persona Chan et al. (2023). For training-based methods, we compare to supervised fine-tuning SFT Radford (2018), DebateTune Li et al. (2024b), DebateGPT Subramaniam et al. (2024). We use three different base models: Llama-3-8B-Instruct Llama-3 Dubey et al. (2024), Mistral-7B-Instruct Mistral Jiang et al. (2023), and Gemma-2-2B-Instruct Gemma-2 Team et al. (2024). 5.1 ACC-Collab Performance Final Answer Accuracy We begin by examining the performance of our method ACC-Collab (a single round of training) and ACC-Collab+ (two rounds of training) In table 1, we see the average accuracy of each method after five rounds of deliberation. Our method attains superior performance compared with baseline methods in most cases. The high efficacy of ACC-Collab relative to the baselines indicates that in most cases, only a single round of training is necessary to produce a high-quality collaborative team. It is worth noting that in some cases, further training rounds may decrease performance (i.e., ACC-Collab+ can have worse performance than ACC-Collab). Hence, we have a hold-out set of tasks to determine whether further training degrades performance. Process Accuracy In addition to measuring the correctness of the actor’s final answer (i.e., outcome accuracy), we also analyze the correctness of the team’s reasoning and discussion steps (i.e., process accuracy). Since ground truth is not available for reasoning and discussion steps, we use GPT-4o as an oracle to evaluate whether the agents follow the correct steps to reach the final answer. We provide an outline of the experimental setup and full results in Section A.1 of the supplement. We find that our method improves or maintains the process accuracy of the actor and critic. Llama-3 (Ours) (Ours) SoM (2x) SoM (4x) Persona DebateTune SFT DebateGPT ACC-Collab ACC-Collab+ BoolQ $.812_{±.01}$ $.811_{±.007}$ $.781_{±.002}$ $.775_{±.033}$ $.798_{±.006}$ $.815_{±.005}$ $.887_{±.005}$ $\mathbf{.894}_{±.003}$ MMLU $.62_{±.004}$ $.635_{±.004}$ $.639_{±.004}$ $.63_{±.004}$ $.642_{±.005}$ $.654_{±.005}$ $.644_{±.01}$ $\mathbf{.683}_{±.012}$ BBH $.508_{±.003}$ $.514_{±.005}$ $.509_{±.013}$ $.508_{±.005}$ $.552_{±.006}$ $.551_{±.008}$ $\mathbf{.593}_{±.006}$ $.574_{±.003}$ SCIQ $.925_{±.002}$ $.923_{±.002}$ $.925_{±.004}$ $.924_{±.004}$ $.925_{±.003}$ $.932_{±.001}$ $\mathbf{.952}_{±.0}$ $.948_{±.003}$ ARC $.874_{±.001}$ $.874_{±.001}$ $.87_{±.003}$ $.871_{±.002}$ $.879_{±.004}$ $.876_{±.002}$ $\mathbf{.881}_{±.004}$ $.869_{±.002}$ Mistral (Ours) (Ours) SoM (2x) SoM (4x) Persona DebateTune SFT DebateGPT ACC-Collab ACC-Collab+ BoolQ $.801_{±.005}$ $.798_{±.004}$ $.831_{±.003}$ $.83_{±.003}$ $.84_{±.003}$ $.848_{±.002}$ $.877_{±.002}$ $\mathbf{.893}_{±.002}$ MMLU $.57_{±.003}$ $.562_{±.005}$ $.574_{±.002}$ $.562_{±.005}$ $.594_{±.004}$ $.577_{±.002}$ $.61_{±.005}$ $\mathbf{.672}_{±.004}$ BBH $.428_{±.002}$ $.462_{±.003}$ $.465_{±.011}$ $.456_{±.005}$ $.439_{±.006}$ $.48_{±.012}$ $.519_{±.009}$ $\mathbf{.601}_{±.004}$ SCIQ $.856_{±.002}$ $.856_{±.002}$ $.86_{±.002}$ $.863_{±.003}$ $.858_{±.004}$ $.871_{±.003}$ $.902_{±.005}$ $\mathbf{.905}_{±.002}$ ARC $.824_{±.001}$ $.823_{±.002}$ $.827_{±.001}$ $.834_{±.0}$ $.825_{±.003}$ $.822_{±.002}$ $.843_{±.003}$ $\mathbf{.856}_{±.003}$ Gemma-2 (Ours) (Ours) SoM (2x) SoM (4x) Persona DebateTune SFT DebateGPT ACC-Collab ACC-Collab+ BoolQ $.75_{±.011}$ $.759_{±.004}$ $.716_{±.015}$ $.767_{±.003}$ $.783_{±.011}$ $.812_{±.003}$ $.84_{±.005}$ $\mathbf{.845}_{±.005}$ MMLU $.58_{±.002}$ $.578_{±.002}$ $.577_{±.002}$ $.578_{±.001}$ $.579_{±.002}$ $\mathbf{.582}_{±.002}$ $.51_{±.016}$ $.555_{±.003}$ BBH $.454_{±.007}$ $.449_{±.01}$ $.447_{±.006}$ $.447_{±.007}$ $.498_{±.006}$ $.491_{±.01}$ $\mathbf{.513}_{±.006}$ $.475_{±.008}$ SCIQ $.903_{±.002}$ $.903_{±.002}$ $.908_{±.003}$ $.903_{±.001}$ $.913_{±.002}$ $.914_{±.002}$ $\mathbf{.918}_{±.003}$ $.909_{±.003}$ ARC $.841_{±.003}$ $.843_{±.005}$ $.847_{±.003}$ $.847_{±.003}$ $.848_{±.002}$ $.851_{±.003}$ $\mathbf{.852}_{±.003}$ $.849_{±.002}$ Table 1: Average accuracy (with 95% confidence intervals) after $5$ rounds of deliberation. For each dataset, the highest accuracy is shown in bold. 5.2 Performance Increase of Multi-Agent Collaboration <details> <summary>extracted/6248978/plots/new_improve.png Details</summary>  ### Visual Description ## Bar Chart: Model Performance Comparison ### Overview The image presents three bar charts comparing the performance of different models (Llama-3, Mistral, and Gemma-2) using various techniques. The y-axis represents the "% Improvement," and the x-axis represents different methods or models being compared. The legend at the bottom identifies the color-coded methods. ### Components/Axes * **Titles:** Llama-3 (top-left), Mistral (top-center), Gemma-2 (top-right) * **Y-axis:** "% Improvment" (vertical, left side of all charts). Scale ranges from 2 to 5 for Llama-3 and Mistral, and 2 to 6 for Gemma-2, with tick marks at each integer value. * **X-axis:** Categorical, representing different methods/models. No explicit labels on the x-axis itself, but the legend provides the categories. * **Legend:** Located at the bottom of the image. * Blue: SoM (2x) * Teal: SoM (4x) * Purple: Persona * Brown: DebateTune * Light Green: SFT * Dark Green: DebateGPT * Orange (diagonal lines): ACC-Collab (Ours) * Red (diagonal lines): ACC-Collab + (Ours) * **Vertical Black Line:** A vertical black line is present in each chart, separating the first six methods from the last two. ### Detailed Analysis **Llama-3 Chart:** * **SoM (2x) (Blue):** Approximately 2.8% improvement. * **SoM (4x) (Teal):** Approximately 3.9% improvement. * **Persona (Purple):** Approximately 2.5% improvement. * **DebateTune (Brown):** Approximately 2.3% improvement. * **SFT (Light Green):** Approximately 1.3% improvement. * **DebateGPT (Dark Green):** Approximately 1.7% improvement. * **ACC-Collab (Ours) (Orange, diagonal lines):** Approximately 3.1% improvement. * **ACC-Collab + (Ours) (Red, diagonal lines):** Approximately 5.3% improvement. **Mistral Chart:** * **SoM (2x) (Blue):** Approximately 1.5% improvement. * **SoM (4x) (Teal):** Approximately 1.4% improvement. * **Persona (Purple):** Approximately 3.3% improvement. * **DebateTune (Brown):** Approximately 3.1% improvement. * **SFT (Light Green):** Approximately 4.0% improvement. * **DebateGPT (Dark Green):** Approximately 3.8% improvement. * **ACC-Collab (Ours) (Orange, diagonal lines):** Approximately 5.3% improvement. * **ACC-Collab + (Ours) (Red, diagonal lines):** Approximately 5.2% improvement. **Gemma-2 Chart:** * **SoM (2x) (Blue):** Approximately 2.9% improvement. * **SoM (4x) (Teal):** Approximately 3.0% improvement. * **Persona (Purple):** Approximately 2.1% improvement. * **DebateTune (Brown):** Approximately 3.2% improvement. * **SFT (Light Green):** Approximately 2.4% improvement. * **DebateGPT (Dark Green):** Approximately 2.8% improvement. * **ACC-Collab (Ours) (Orange, diagonal lines):** Approximately 3.1% improvement. * **ACC-Collab + (Ours) (Red, diagonal lines):** Approximately 6.3% improvement. ### Key Observations * **ACC-Collab + (Ours)** consistently shows the highest improvement across all three models (Llama-3, Mistral, and Gemma-2). * **SFT** generally shows the lowest improvement among the first six methods. * The performance of **SoM (2x)** and **SoM (4x)** varies across the models. * The vertical black line separates the standard methods from the "Ours" methods, highlighting the comparison. ### Interpretation The charts demonstrate the performance improvement achieved by different methods on three different language models. The "ACC-Collab + (Ours)" method consistently outperforms the other methods, suggesting it is the most effective technique for improving the performance of these models. The relative performance of the other methods varies depending on the specific model being used. The separation of the "Ours" methods by a vertical line emphasizes their superior performance compared to the standard methods. </details> Figure 2: Percent improvement in accuracy after five rounds of deliberation, compared to a single round. Percent improvement (Eq. 7) for each method is averaged across all five datasets. Average Improvement As noted in Du et al. (2023), the key mechanism behind the success of multi-agent deliberation (or any of its many variants) is that discussion over multiple rounds allows the models to iteratively refine their answers. Thus, a natural question for any iterative multi-agent method is: how much does accuracy improve from the initial round $t=0$ to the final round $t=T$ ? Where $T=4$ in our experiments. To measure this, we look at the percent improvement in model accuracy from round $t=0$ to round $t=4$ calculated as, $$ \displaystyle\frac{\text{acc}_{4}-\text{acc}_{0}}{\text{acc}_{0}}\qquad\text{ % where acc${}_{t}$ is accuracy at round $t$} \tag{7} $$ In Figure 2 we see the average percent improvement for each method, averaged across all 5 datasets. For each of the three base models, ACC-Collab+ has the highest average improvement compared to all other methods. Additionally, the improvement gained by methods such as SoM, SFT or DebateGPT is far less stable than that of ACC-Collab+. In particular, for Mistral, SoM yields nearly no improvement, similarly SFT and DebateGPT offer little improvement when applied to of Llama-3. <details> <summary>extracted/6248978/plots/new_perRoundConf.png Details</summary>  ### Visual Description ## Chart: Model Accuracy Comparison Across Rounds ### Overview The image presents three line charts comparing the accuracy of different language models (Llama-3, Mistral, and Gemma-2) across multiple rounds of interaction or training. Each chart displays the performance of several models and training configurations, with accuracy on the y-axis and round number on the x-axis. The legend at the bottom identifies each line by color and label. ### Components/Axes * **Titles:** Each chart has a title indicating the language model being evaluated: "Llama-3", "Mistral", and "Gemma-2". * **X-Axis:** Labeled "Round", with integer markers at 0, 1, 2, 3, and 4. * **Y-Axis:** Labeled "Accuracy", with a scale from 0.725 to 0.900 for Llama-3 and Mistral, and from 0.700 to 0.850 for Gemma-2. * **Legend:** Located at the bottom of the image, it maps line colors to model/training configurations: * Blue dashed line: SoM (2x) * Teal dashed line: SoM (4x) * Purple dashed line: Persona * Brown solid line: DebateTune * Light Green solid line: SFT * Dark Green solid line: DebateGPT * Orange solid line: ACC-Collab (Ours) * Red solid line: ACC-Collab + (Ours) ### Detailed Analysis #### Llama-3 Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.845 at Round 0, increases to about 0.890 at Round 1, and then plateaus around 0.890 for Rounds 2-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.840 at Round 0, increases to about 0.885 at Round 1, and then plateaus around 0.885 for Rounds 2-4. * **DebateGPT (Dark Green):** Starts at approximately 0.800 at Round 0, increases to about 0.820 at Round 1, and then plateaus around 0.820 for Rounds 2-4. * **SoM (4x) (Teal):** Starts at approximately 0.780 at Round 0, increases to about 0.815 at Round 1, and then plateaus around 0.815 for Rounds 2-4. * **SoM (2x) (Blue):** Starts at approximately 0.770 at Round 0, increases to about 0.810 at Round 1, and then plateaus around 0.810 for Rounds 2-4. * **SFT (Light Green):** Starts at approximately 0.780 at Round 0, increases to about 0.800 at Round 1, and then plateaus around 0.800 for Rounds 2-4. * **Persona (Purple):** Starts at approximately 0.770 at Round 0, increases to about 0.780 at Round 1, and then plateaus around 0.780 for Rounds 2-4. * **DebateTune (Brown):** Starts at approximately 0.780 at Round 0, decreases to about 0.760 at Round 1, increases to about 0.780 at Round 2, and then plateaus around 0.780 for Rounds 3-4. #### Mistral Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.860 at Round 0, increases to about 0.890 at Round 1, and then plateaus around 0.890 for Rounds 2-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.830 at Round 0, increases to about 0.870 at Round 1, and then plateaus around 0.875 for Rounds 2-4. * **DebateGPT (Dark Green):** Starts at approximately 0.830 at Round 0, increases to about 0.850 at Round 1, and then plateaus around 0.850 for Rounds 2-4. * **SoM (4x) (Teal):** Starts at approximately 0.810 at Round 0, increases to about 0.820 at Round 1, and then decreases to about 0.800 for Rounds 2-4. * **SoM (2x) (Blue):** Starts at approximately 0.805 at Round 0, increases to about 0.815 at Round 1, and then decreases to about 0.800 for Rounds 2-4. * **SFT (Light Green):** Starts at approximately 0.815 at Round 0, increases to about 0.830 at Round 1, and then plateaus around 0.830 for Rounds 2-4. * **Persona (Purple):** Starts at approximately 0.815 at Round 0, increases to about 0.830 at Round 1, and then plateaus around 0.830 for Rounds 2-4. * **DebateTune (Brown):** Starts at approximately 0.825 at Round 0, increases to about 0.830 at Round 1, and then plateaus around 0.830 for Rounds 2-4. #### Gemma-2 Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.770 at Round 0, increases to about 0.840 at Round 1, and then plateaus around 0.840 for Rounds 2-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.780 at Round 0, increases to about 0.845 at Round 1, and then plateaus around 0.840 for Rounds 2-4. * **DebateGPT (Dark Green):** Starts at approximately 0.790 at Round 0, increases to about 0.825 at Round 1, and then plateaus around 0.820 for Rounds 2-4. * **SoM (4x) (Teal):** Starts at approximately 0.720 at Round 0, increases to about 0.760 at Round 1, and then plateaus around 0.760 for Rounds 2-4. * **SoM (2x) (Blue):** Starts at approximately 0.740 at Round 0, increases to about 0.760 at Round 1, and then plateaus around 0.760 for Rounds 2-4. * **SFT (Light Green):** Starts at approximately 0.760 at Round 0, increases to about 0.790 at Round 1, and then plateaus around 0.790 for Rounds 2-4. * **Persona (Purple):** Starts at approximately 0.715 at Round 0, increases to about 0.730 at Round 1, and then plateaus around 0.720 for Rounds 2-4. * **DebateTune (Brown):** Starts at approximately 0.740 at Round 0, increases to about 0.765 at Round 1, and then plateaus around 0.765 for Rounds 2-4. ### Key Observations * **ACC-Collab Performance:** The "ACC-Collab + (Ours)" and "ACC-Collab (Ours)" configurations consistently outperform other methods across all three language models. * **Initial Round Improvement:** Most models show a significant increase in accuracy from Round 0 to Round 1, suggesting that the initial round is crucial for learning or adaptation. * **Plateau Effect:** After Round 1, the accuracy of most models tends to plateau, indicating diminishing returns from additional rounds. * **Model-Specific Performance:** The relative performance of different training configurations varies slightly depending on the base language model. For example, the gap between "ACC-Collab" and other methods is more pronounced for Gemma-2 compared to Llama-3. * **SoM (2x) and SoM (4x) Decline in Mistral:** The accuracy of SoM (2x) and SoM (4x) declines after Round 1 in Mistral. ### Interpretation The data suggests that the "ACC-Collab" training configurations are highly effective in improving the accuracy of language models, particularly in the initial rounds of interaction or training. The plateau effect observed after Round 1 indicates that there may be a limit to the improvements achievable through the tested methods, or that different strategies are needed to further enhance performance beyond this point. The model-specific variations highlight the importance of tailoring training approaches to the specific characteristics of each language model. The decline of SoM (2x) and SoM (4x) in Mistral suggests that these configurations may not be as well-suited for this particular model. </details> <details> <summary>extracted/6248978/plots/new_perRoundSCIQ.png Details</summary>  ### Visual Description ## Line Charts: Model Accuracy vs. Round ### Overview The image presents three line charts comparing the accuracy of different models (Llama-3, Mistral, and Gemma-2) across multiple rounds of interaction or training. Each chart displays the performance of several methods, including "SoM (2x)", "SoM (4x)", "Persona", "DebateTune", "SFT", "DebateGPT", "ACC-Collab (Ours)", and "ACC-Collab + (Ours)". The x-axis represents the round number (0 to 4), and the y-axis represents the accuracy score. Each line represents a different method, and shaded areas around the lines indicate the confidence interval or variability in the results. ### Components/Axes * **Titles:** The charts are titled "Llama-3" (top-left), "Mistral" (top-center), and "Gemma-2" (top-right). * **X-axis:** Labeled "Round", with markers at 0, 1, 2, 3, and 4. * **Y-axis:** Labeled "Accuracy", with different scales for each chart: * Llama-3: 0.91 to 0.95 * Mistral: 0.84 to 0.90 * Gemma-2: 0.89 to 0.92 * **Legend:** Located at the bottom of the image, mapping line colors/styles to methods: * Blue dashed line: SoM (2x) * Teal dashed line: SoM (4x) * Purple dashed line: Persona * Brown solid line: DebateTune * Light Green solid line: SFT * Dark Green solid line: DebateGPT * Orange solid line: ACC-Collab (Ours) * Red solid line: ACC-Collab + (Ours) ### Detailed Analysis #### Llama-3 Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.933 at round 0, increases to about 0.947 by round 1, and then plateaus around 0.948-0.949 for rounds 2-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.941 at round 0, increases to about 0.952 by round 1, and then plateaus around 0.953-0.954 for rounds 2-4. * **DebateGPT (Dark Green):** Starts at approximately 0.916 at round 0, increases to about 0.930 by round 1, and then plateaus around 0.932-0.933 for rounds 2-4. * **SoM (2x) (Blue Dashed):** Starts at approximately 0.917 at round 0, increases to about 0.924 by round 1, and then plateaus around 0.925-0.926 for rounds 2-4. * **SoM (4x) (Teal Dashed):** Starts at approximately 0.918 at round 0, increases to about 0.922 by round 1, and then plateaus around 0.923-0.924 for rounds 2-4. * **Persona (Purple Dashed):** Starts at approximately 0.918 at round 0, increases to about 0.924 by round 1, and then plateaus around 0.925-0.926 for rounds 2-4. * **DebateTune (Brown):** Starts at approximately 0.916 at round 0, increases to about 0.920 by round 1, and then plateaus around 0.921-0.922 for rounds 2-4. * **SFT (Light Green):** Starts at approximately 0.914 at round 0, increases to about 0.921 by round 1, and then plateaus around 0.922-0.923 for rounds 2-4. #### Mistral Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.872 at round 0, increases to about 0.899 by round 2, and then plateaus around 0.901-0.902 for rounds 3-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.877 at round 0, increases to about 0.895 by round 2, and then plateaus around 0.896-0.897 for rounds 3-4. * **DebateGPT (Dark Green):** Starts at approximately 0.859 at round 0, increases to about 0.871 by round 2, and then plateaus around 0.872-0.873 for rounds 3-4. * **SoM (2x) (Blue Dashed):** Starts at approximately 0.842 at round 0, increases to about 0.857 by round 2, and then plateaus around 0.858-0.859 for rounds 3-4. * **SoM (4x) (Teal Dashed):** Starts at approximately 0.860 at round 0, increases to about 0.863 by round 2, and then plateaus around 0.864-0.865 for rounds 3-4. * **Persona (Purple Dashed):** Starts at approximately 0.860 at round 0, increases to about 0.863 by round 2, and then plateaus around 0.864-0.865 for rounds 3-4. * **DebateTune (Brown):** Starts at approximately 0.859 at round 0, increases to about 0.860 by round 2, and then plateaus around 0.861-0.862 for rounds 3-4. * **SFT (Light Green):** Starts at approximately 0.837 at round 0, increases to about 0.857 by round 2, and then plateaus around 0.858-0.859 for rounds 3-4. #### Gemma-2 Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.890 at round 0, increases to about 0.906 by round 1, and then plateaus around 0.907-0.908 for rounds 2-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.895 at round 0, increases to about 0.918 by round 2, and then decreases to about 0.916 for round 4. * **DebateGPT (Dark Green):** Starts at approximately 0.898 at round 0, increases to about 0.913 by round 2, and then plateaus around 0.914-0.915 for rounds 3-4. * **SoM (2x) (Blue Dashed):** Starts at approximately 0.892 at round 0, increases to about 0.903 by round 2, and then plateaus around 0.904-0.905 for rounds 3-4. * **SoM (4x) (Teal Dashed):** Starts at approximately 0.901 at round 0, increases to about 0.905 by round 2, and then plateaus around 0.906-0.907 for rounds 3-4. * **Persona (Purple Dashed):** Starts at approximately 0.890 at round 0, increases to about 0.908 by round 2, and then plateaus around 0.909-0.910 for rounds 3-4. * **DebateTune (Brown):** Starts at approximately 0.888 at round 0, increases to about 0.904 by round 2, and then plateaus around 0.905-0.906 for rounds 3-4. * **SFT (Light Green):** Starts at approximately 0.899 at round 0, increases to about 0.909 by round 2, and then plateaus around 0.910-0.911 for rounds 3-4. ### Key Observations * **Initial Improvement:** Most methods show a significant increase in accuracy from round 0 to round 1 or 2. * **Plateau Effect:** After the initial improvement, the accuracy tends to plateau, indicating diminishing returns from further rounds. * **ACC-Collab Superiority:** The "ACC-Collab (Ours)" and "ACC-Collab + (Ours)" methods generally outperform the other methods across all three models. * **Model-Specific Performance:** The absolute accuracy values differ across the models, with Llama-3 generally achieving higher accuracy scores than Mistral and Gemma-2. ### Interpretation The charts demonstrate the impact of different methods on the accuracy of three language models (Llama-3, Mistral, and Gemma-2) over multiple rounds. The "ACC-Collab" methods appear to be the most effective in improving accuracy, suggesting that collaborative approaches enhance model performance. The plateau effect observed after a few rounds indicates that the models may reach a point of diminishing returns with the given methods and data. The differences in absolute accuracy across the models highlight the inherent variations in their architectures and training data. The shaded areas around the lines provide insights into the variability of the results, which is crucial for assessing the robustness of the methods. </details> Figure 3: Accuracy over five rounds of deliberation on BoolQ (top) and SCIQ (bottom). Per-Round Accuracy Next, we look more closely at the accuracy of each method across five rounds of deliberation. Figure 3, shows per-round accuracy on the BoolQ dataset. As already illustrated by Table 1, ACC-Collab and ACC-Collab+ achieve higher final round accuracy than the other methods. Notably, our method has higher accuracy both at the final round $t=4$ and at round $t=0$ . Recall that at round $t=0$ , the actor’s response is independent of the critic. This indicates that in some cases, our training pipeline can produce actor agents with superior zero-shot accuracy (without deliberation) compared to models produced by SFT and DebateGPT. Interestingly, we observe that in some cases, SFT and DebateGPT are not much better than simple deliberation with untrained models (i.e., SoM-4x), e.g., BoolQ with Llama-3. This is primarily due to the fact that these methods are designed to improve the model’s single-short performance but do little to improve their collaborative abilities. Cases such as this outline the necessity of training to improve collaboration (ACC-Collab) rather than training for raw accuracy (SFT and DebateGPT). 5.3 Individual Performance Next, we examine the relative effectiveness of both the actor and the critic. To do this, we train an actor and critic via ACC-Collab. Then, during deliberation, we pair the trained actor with an untrained critic and pair the trained critic with an untrained actor. In Table 2, column “Actor” corresponds to the former, while “Critic” corresponds to the latter. On average, the trained actor attains higher accuracy compared to the trained critic, this aligns with intuition as the actor is responsible for providing answers while the critic plays a supporting role. In most cases, the trained actor (paired with an untrained critic) outperforms SFT and DebateGPT. When the trained actor is paired with a trained critic (either ACC-Collab or ACC-Collab+), its performance is further improved. Llama-3 SoM (4x) DebateGPT Actor Critic ACC-Collab ACC-Collab+ BoolQ $.811_{±.007}$ $.815_{±.005}$ $.867_{±.003}$ $.834_{±.005}$ $.887_{±.005}$ $\mathbf{.894}_{±.003}$ MMLU $.635_{±.004}$ $.654_{±.005}$ $.651_{±.012}$ $.65_{±.008}$ $.644_{±.01}$ $\mathbf{.683}_{±.012}$ BBH $.514_{±.005}$ $.551_{±.008}$ $.583_{±.01}$ $.55_{±.015}$ $\mathbf{.593}_{±.006}$ $.574_{±.003}$ SCIQ $.923_{±.002}$ $.932_{±.001}$ $.947_{±.002}$ $.945_{±.001}$ $\mathbf{.952}_{±.0}$ $.948_{±.003}$ ARC $.874_{±.001}$ $.876_{±.002}$ $\mathbf{.885}_{±.003}$ $.866_{±.002}$ $.881_{±.004}$ $.869_{±.002}$ Mistral SoM (4x) DebateGPT Actor Critic ACC-Collab ACC-Collab+ BoolQ $.798_{±.004}$ $.848_{±.002}$ $.873_{±.002}$ $.843_{±.002}$ $.877_{±.002}$ $\mathbf{.893}_{±.002}$ MMLU $.562_{±.005}$ $.577_{±.002}$ $.598_{±.002}$ $.611_{±.001}$ $.61_{±.005}$ $\mathbf{.672}_{±.004}$ BBH $.462_{±.003}$ $.48_{±.012}$ $.493_{±.012}$ $.518_{±.005}$ $.519_{±.009}$ $\mathbf{.601}_{±.004}$ SCIQ $.856_{±.002}$ $.871_{±.003}$ $.891_{±.001}$ $.891_{±.002}$ $.902_{±.005}$ $\mathbf{.905}_{±.002}$ ARC $.823_{±.002}$ $.822_{±.002}$ $.833_{±.003}$ $.842_{±.003}$ $.843_{±.003}$ $\mathbf{.856}_{±.003}$ Gemma-2 SoM (4x) DebateGPT Actor Critic ACC-Collab ACC-Collab+ BoolQ $.759_{±.004}$ $.812_{±.003}$ $.839_{±.005}$ $.774_{±.014}$ $.84_{±.005}$ $\mathbf{.845}_{±.005}$ MMLU $.578_{±.002}$ $\mathbf{.582}_{±.002}$ $.519_{±.026}$ $.566_{±.002}$ $.51_{±.016}$ $.555_{±.003}$ BBH $.449_{±.01}$ $.491_{±.01}$ $.51_{±.011}$ $.475_{±.004}$ $\mathbf{.513}_{±.006}$ $.475_{±.008}$ SCIQ $.903_{±.002}$ $.914_{±.002}$ $\mathbf{.923}_{±.002}$ $.912_{±.001}$ $.918_{±.003}$ $.909_{±.003}$ ARC $.843_{±.005}$ $.851_{±.003}$ $.85_{±.002}$ $\mathbf{.855}_{±.002}$ $.852_{±.003}$ $.849_{±.002}$ Table 2: Accuracy after 5 rounds of deliberation. The Actor (Critic) column corresponds to an actor-agent (critic-agent) trained via ACC-Collab and paired with an untrained Critic (Actor) during deliberation. <details> <summary>extracted/6248978/criticResp.png Details</summary>  ### Visual Description ## Dialogue Snippet: Actor-Critic Interaction ### Overview The image presents a snippet of a dialogue between an "Actor" and two "Critics" ("Untrained Critic" and "Trained Critic"). The Actor proposes a final answer to a question, which is then evaluated by the two critics. The Untrained Critic mostly agrees with the Actor, while the Trained Critic provides counterarguments and suggests an alternative answer. ### Components/Axes * **Actor:** Represented by a green robot icon. The background is a light yellow. * **Untrained Critic:** Represented by a blue robot icon. The background is a light pink. * **Trained Critic:** Represented by a blue robot icon. The background is a light pink. * **Red X:** Indicates disagreement or incorrectness, placed to the right of the Actor's statement. ### Detailed Analysis or Content Details 1. **Actor:** * Text: "With the decline of honey bee populations, plant species that..." * Text: "Final Answer: (B) change their flowers so that wind will fertilize them." * Icon: Green robot. 2. **Untrained Critic:** * Text: "Your response is mostly correct... {continues to agree with the actor}." * Icon: Blue robot. 3. **Trained Critic:** * Text: "While your response is a valid explanation for why option B could be a possible answer, here are two reasons why option C is also correct and refutes your response directly:" * Text: "1....{Counterpoint to for why B could be wrong}" * Text: "2....{Evidence for why C is correct}" * Icon: Blue robot. ### Key Observations * The Actor provides a specific answer (B) related to plant adaptation due to declining bee populations. * The Untrained Critic shows agreement with the Actor's response. * The Trained Critic acknowledges the validity of the Actor's answer but introduces a counterargument, suggesting that option C is also correct and provides reasons to support this claim. * The red X next to the Actor's statement suggests that the Actor's answer is not entirely correct or complete. ### Interpretation The dialogue illustrates a scenario where an initial answer is refined through critical evaluation. The Trained Critic's response highlights the importance of considering multiple perspectives and providing evidence-based arguments. The interaction suggests that while the Actor's answer might have some merit, it is not the most accurate or comprehensive solution, as indicated by the Trained Critic's counterpoints and the presence of the red "X". The scenario demonstrates a process of iterative improvement through constructive criticism. </details> Figure 4: Comparison of responses from the critic model before and after training with ACC-Collab. 5.4 What do the Agents Actually Learn? Lastly, we are interested in understanding how ACC-Collab improves the Actor-Critic Team. Figure 4 demonstrates an example of the difference in responses between an untrained critic and a critic trained through ACC-Collab. Although the actor provides a wrong answer, the untrained critic is too agreeable and does not provide substantive feedback for the actor to correct their answer. In contrast, the trained critic is more willing to disagree with the actor and provides more detailed feedback. Largely, we observe that this trend is common; untrained critics are too agreeable and are thus less able to change the actor’s mind, while trained critics are more willing to disagree (see Section C.2 for more examples). When examining the trained actor’s responses, we do not find a notable qualitative change compared to the responses of an untrained actor. However, as shown previously, we do observe a qualitative change in the actor’s responses to become more accurate. 6 Conclusion, Limitations and Impact In this paper, we propose ACC-Collab, a novel framework for jointly training a two-agent team (one actor-agent and one critic-agent) to collaboratively solve problems through iterative discussion. To train these agents, we developed an off-policy data generation scheme dubbed “Guided-Collaboration”, which produces high-quality preference data for collaborative models. We found that ACC-Collab outperforms all baselines on a wide array of domains. In particular, even a single round of training for both the actor and critic results in a high-quality team. Of particular note is the effects that ACC-Collab has on the critic model. Without ACC-Collab, the critic model is often too agreeable and lacks verbosity in their responses. In contrast, after training with ACC-Collab, the critic is far more likely to provide detailed disagreements during discussion. However, our framework ACC-Collab also comes with limitations. Firstly, even though ACC-Collab attains superior performance compared to baselines on a wide array of domains, it is important to note that we conduct experiments mainly on question-answering tasks; thus, it remains to be seen whether such a framework would continue to be effective in other types of tasks. Moreover, in our experiments, we train and test models on the same task (partitioning each task into a training and testing set). As such, the generalizability of each actor-critic team to unseen domains is unknown. Our method makes use of the fact that for each question, correct and incorrect answers can be easily established. Secondly, while we provide results for three families of models, these experiments are performed on 2B, 7B, and 8B models. While our method is effective for these sizes (standard in open-source models), it remains to be seen whether this effectiveness will scale to larger models. Acknowledgments We would like to thank Li Hang for his valuable insights and guidance during the development of this work. Reproducibility Statement Here, we outline the details necessary to reproduce our results. We provide an algorithm for our data generation procedure (Algorithm 1), as well as a description of our training procedure in Section 4.3. Each dataset, baseline method, and base model used are specified at the beginning of Section 5. We provide additional experimental details in Section A. Prompts used for our method can be found in Section C. Lastly, we publicly release the code used for our method. References - Abdelnabi et al. (2023) Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Llm-deliberation: Evaluating llms with interactive multi-agent negotiation games. arXiv preprint arXiv:2309.17234, 2023. - Chan et al. (2023) Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201, 2023. - Chang (2024a) Edward Y Chang. Evince: Optimizing adversarial llm dialogues via conditional statistics and information theory. arXiv preprint arXiv:2408.14575, 2024a. - Chang (2024b) Edward Y Chang. Socrasynth: Multi-llm reasoning with conditional statistics. arXiv preprint arXiv:2402.06634, 2024b. - Chen et al. (2023) Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. arXiv preprint arXiv:2309.13007, 2023. - Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024. URL https://arxiv.org/abs/2401.01335. - Chollet (2019) François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. - Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019. - Du et al. (2023) Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. - Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. - Feng et al. (2024) Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration. arXiv preprint arXiv:2402.00367, 2024. - Fiez et al. (2019) Tanner Fiez, Benjamin Chasnov, and Lillian J Ratliff. Convergence of learning dynamics in stackelberg games. arXiv preprint arXiv:1906.01217, 2019. - Hadi et al. (2023) Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023. - Hao et al. (2024) Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, et al. Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. arXiv preprint arXiv:2404.05221, 2024. - Havrilla et al. (2024) Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Railneau. Glore: When, where, and how to improve llm reasoning via global and local refinements. arXiv preprint arXiv:2402.10963, 2024. - Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. - Hong et al. (2023) Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023. - Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023. - Huang et al. (2024) Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333, 2024. - Irving et al. (2018) Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899, 2018. - Ji et al. (2023) Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating llm hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1827–1843, 2023. - Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. - Jiang et al. (2024) Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al. $\{$ MegaScale $\}$ : Scaling large language model training to more than 10,000 $\{$ GPUs $\}$ . In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pp. 745–760, 2024. - Khan et al. (2024) Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive llms leads to more truthful answers. arXiv preprint arXiv:2402.06782, 2024. - Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023. - Li et al. (2023a) Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for” mind” exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023a. - Li et al. (2023b) Huao Li, Yu Quan Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Michael Lewis, and Katia Sycara. Theory of mind for multi-agent collaboration via large language models. arXiv preprint arXiv:2310.10701, 2023b. - Li et al. (2024a) Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. arXiv preprint arXiv:2402.05120, 2024a. - Li et al. (2024b) Ming Li, Jiuhai Chen, Lichang Chen, and Tianyi Zhou. Can llms speak for diverse people? tuning llms via debate to generate controllable controversial statements. arXiv preprint arXiv:2402.10614, 2024b. - Li et al. (2023c) Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02762, 2023c. - Li & Sethi (2017) Tao Li and Suresh P Sethi. A review of dynamic stackelberg game models. Discrete & Continuous Dynamical Systems-B, 22(1):125, 2017. - Li et al. (2023d) Yuan Li, Yixuan Zhang, and Lichao Sun. Metaagents: Simulating interactions of human behaviors for llm-based task-oriented coordination via collaborative generative agents. arXiv preprint arXiv:2310.06500, 2023d. - Li et al. (2024c) Yunxuan Li, Yibing Du, Jiageng Zhang, Le Hou, Peter Grabowski, Yeqing Li, and Eugene Ie. Improving multi-agent debate with sparse communication topology. arXiv preprint arXiv:2406.11776, 2024c. - Liang et al. (2023) Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118, 2023. - Liu et al. (2023) Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170, 2023. - Michael et al. (2023) Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, and Samuel R Bowman. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702, 2023. - Otter et al. (2020) Daniel W Otter, Julian R Medina, and Jugal K Kalita. A survey of the usages of deep learning for natural language processing. IEEE transactions on neural networks and learning systems, 32(2):604–624, 2020. - Pang et al. (2024a) Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024a. - Pang et al. (2024b) Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024b. URL https://arxiv.org/abs/2404.19733. - Pham et al. (2023) Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A Plummer, Zhaoran Wang, and Hongxia Yang. Let models speak ciphers: Multiagent debate through embeddings. arXiv preprint arXiv:2310.06272, 2023. - Radford (2018) Alec Radford. Improving language understanding by generative pre-training. 2018. - Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. - Rasal (2024) Sumedh Rasal. Llm harmony: Multi-agent communication for problem solving. arXiv preprint arXiv:2401.01312, 2024. - Rawte et al. (2023) Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922, 2023. - Shinn et al. (2023) Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366. - Singhal et al. (2023) Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617, 2023. - (47) Andries Petrus Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D Barrett, and Arnu Pretorius. Should we be going mad? a look at multi-agent debate strategies for llms. In Forty-first International Conference on Machine Learning. - Subramaniam et al. (2024) Vighnesh Subramaniam, Antonio Torralba, and Shuang Li. Debategpt: Fine-tuning large language models with multi-agent debate supervision. 2024. - Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. - Team et al. (2024) Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. - Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. - Tonmoy et al. (2024) SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313, 2024. - Trung et al. (2024) Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024. - Wang et al. (2023a) Boshi Wang, Xiang Yue, and Huan Sun. Can chatgpt defend its belief in truth? evaluating llm reasoning via debate. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11865–11881, 2023a. - Wang et al. (2024) Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of llm reasoning: Are multi-agent discussions the key? arXiv preprint arXiv:2402.18272, 2024. - Wang et al. (2023b) Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300, 2023b. - Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. - Welbl et al. (2017) Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017. - Wu et al. (2023) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023. - Xiong et al. (2024) Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! llm agent learning via iterative step-level process refinement, 2024. URL https://arxiv.org/abs/2406.11176. - Yang et al. (2024) Ruixin Yang, Dheeraj Rajagopa, Shirley Anugrah Hayati, Bin Hu, and Dongyeop Kang. Confidence calibration and rationalization for llms via multi-agent deliberation. arXiv preprint arXiv:2404.09127, 2024. - Zhang et al. (2023) Jintian Zhang, Xin Xu, and Shumin Deng. Exploring collaboration mechanisms for llm agents: A social psychology view. arXiv preprint arXiv:2310.02124, 2023. - Zhang et al. (2024) Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furu Wei. Llm as a mastermind: A survey of strategic reasoning with large language models. arXiv preprint arXiv:2404.01230, 2024. - Zhu et al. (2024) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models: Empirical results and analysis, 2024. URL https://arxiv.org/abs/2304.04675. Appendix Appendix A Experiments Here, we provide additional details regarding our experiments. Datasets Each dataset is split into a training set, a validation set, and a testing set. For datasets that come with an explicit partition of these sets we use the given partitions; this includes BoolQ, MMLU, SCIQ, and ARC. For BBH, we randomly sample roughly 25% and 10% of the questions from each category in BBH to create a test and validation set, respectively; this comes out to 1260 questions for the test set and 500 questions for the validation set. All results are reported on questions in the test set. Compute All training was performed on a single Nvidia-H800 GPU. Inference for Llama-3 and Mistral based models is performed on a single Nvidia-v100 GPU, for Gemma-2 based models we used a single Nvidia-H800. All inference is performed with the VLLM library Kwon et al. (2023). Training Training for all models was performed via the trl library, using LoRAs of size 256. When training ACC-Collab with DPO we use a negative log-likelihood (NLL) regularization term (with weight $1$ ) as outlined in Pang et al. (2024a). <details> <summary>extracted/6248978/plots/mainMethod_v4.png Details</summary>  ### Visual Description ## Diagram: Actor-Critic Model with Preference Data ### Overview The image illustrates a diagram of an actor-critic model that incorporates preference data for training. The diagram shows the flow of information between the actor and critic, and how their interactions influence the generation of preference data used to train the models. ### Components/Axes * **Actor:** Represented by an orange robot icon. * `z_a^(t-1)`: Actor's state at time t-1 (orange box with orange outline). * `z_a^(t)`: Actor's state at time t (green, orange, or red box with matching outline). * **Critic:** Represented by a blue robot icon. * `z_c^(t-1)`: Critic's state at time t-1 (blue box with blue outline). * `z_c^(t)`: Critic's state at time t (blue box with blue outline). * **Actor Guided Towards y:** Text label indicating the actor's actions are guided towards a target 'y'. * **Natural Debate:** Text label indicating a natural debate between the actor and critic. * **Actor Guided Away From y:** Text label indicating the actor's actions are guided away from a target 'y'. * **Δy:** Green circle containing the delta y symbol. * **Δ!y:** Red circle containing the delta !y symbol. * **Relative Quality of Trajectory:** Text label indicating the relative quality of the trajectory. * **Preference Data:** Represents a database or storage of preference data. Contains tuples of actor states: `(z_a^(t), z_a^(t))` * **Train Models:** Indicates the training of models using the preference data. * **ε:** Epsilon, a threshold value. * **if Δy ≥ ε:** Conditional statement. * **elif Δ!y ≥ ε:** Conditional statement. ### Detailed Analysis 1. **Actor and Critic Interaction:** * The diagram starts with the Actor and Critic at time `t-1`, represented by `z_a^(t-1)` (orange) and `z_c^(t-1)` (blue) respectively. * The Actor's actions are guided "Towards y" and "Away From y", while a "Natural Debate" occurs between the Actor and Critic. * At time `t`, the Actor's state `z_a^(t)` can be green, orange, or red, depending on the guidance. The Critic's state `z_c^(t)` remains blue. 2. **Trajectory Evaluation:** * The trajectories are evaluated based on "Relative Quality of Trajectory", resulting in `Δy` (green) and `Δ!y` (red). 3. **Preference Data Generation:** * If `Δy ≥ ε`, the tuple `(z_a^(t), z_a^(t))` (where `z_a^(t)` is green) is added to the "Preference Data". * If `Δ!y ≥ ε`, the tuple `(z_a^(t), z_a^(t))` (where `z_a^(t)` is red) is added to the "Preference Data". * The preference data consists of pairs of actor states at time t. 4. **Model Training:** * The "Preference Data" is used to "Train Models", resulting in updated Actor (orange) and Critic (blue) models. ### Key Observations * The Actor's state at time `t` (`z_a^(t)`) can be in one of three states (green, orange, or red), while the Critic's state at time `t` (`z_c^(t)`) is always blue. * The preference data is generated based on the relative quality of the trajectory, as determined by `Δy` and `Δ!y`. * The conditional statements `if Δy ≥ ε` and `elif Δ!y ≥ ε` determine which trajectories are used to generate preference data. ### Interpretation The diagram illustrates a reinforcement learning process where an actor and critic interact to generate trajectories. The relative quality of these trajectories is evaluated, and preference data is created based on this evaluation. This preference data is then used to train the actor and critic models. The use of preference data allows the models to learn from successful and unsuccessful trajectories, improving their performance over time. The thresholds `ε` determine the sensitivity of the preference data generation process. </details> <details> <summary>extracted/6248978/plots/mainMethod_critic.png Details</summary>  ### Visual Description ## Diagram: Actor-Critic Debate Process ### Overview The image illustrates a process involving an Actor and a Critic, where they engage in a "debate" to refine a model. The diagram shows the flow of information and decision-making based on the relative quality of trajectories. ### Components/Axes * **Actor:** Represented by an orange robot icon. * **Critic:** Represented by a blue robot icon. * **States:** Represented by rectangular boxes labeled with z subscript a (orange) and z subscript c (blue/green/red). The superscript (t-1) and (t) denote time steps. * **Decision Points:** Represented by circles labeled with Δy (green) and Δ!y (red). * **Preference Data:** Represented by a cylinder, indicating a data storage. * **Train Models:** Represented by orange and blue robots with arrows pointing to slightly different versions of themselves. * **Arrows:** Indicate the flow of information and decision-making. * **Text Labels:** Describe the processes and conditions. ### Detailed Analysis 1. **Initial States:** * The Actor starts with state z subscript a at time (t-1), enclosed in an orange box. * The Critic starts with state z subscript c at time (t-1), enclosed in a blue box. 2. **Debate Process:** * The Actor's state z subscript a (t-1) and the Critic's state z subscript c (t-1) lead to a "Natural Debate". * The Actor is "Critic Guided Towards y", resulting in state z subscript a (t) in an orange box and z subscript c (t) in a green box. * The Critic is "Critic Guided Away From y", resulting in state z subscript a (t) in an orange box and z subscript c (t) in a red box. 3. **Decision Points:** * The "Natural Debate" results in states z subscript a (t) in an orange box and z subscript c (t) in a blue box. * The "Critic Guided Towards y" path leads to Δy (green). * The "Critic Guided Away From y" path leads to Δ!y (red). 4. **Conditional Logic:** * If Δy is greater than or equal to ε, then (z subscript c (t), z subscript c (t)) from the "Critic Guided Towards y" path is added to the "Preference Data". * "elif" (else if) Δ!y is greater than or equal to ε, then (z subscript c (t), z subscript c (t)) from the "Critic Guided Away From y" path is added to the "Preference Data". 5. **Training:** * The "Preference Data" is used to "Train Models", resulting in updated Actor (orange) and Critic (blue) models. * The updated Critic model feeds back into the initial state of the Critic. ### Key Observations * The diagram illustrates an iterative process where the Actor and Critic interact and refine their models based on a debate and relative quality of trajectories. * The decision points Δy and Δ!y determine which data is used to update the models. * The use of "elif" suggests that only one of the two conditions (Δy >= ε or Δ!y >= ε) can be true at a time. ### Interpretation The diagram depicts a reinforcement learning process where an Actor and a Critic engage in a form of adversarial training. The "Natural Debate" represents the initial interaction between the Actor and Critic. The "Critic Guided Towards y" and "Critic Guided Away From y" paths represent different strategies or outcomes based on the Critic's guidance. The decision points Δy and Δ!y, along with the threshold ε, determine which trajectories are considered valuable and used to update the models. This process aims to improve the Actor's performance by incorporating feedback from the Critic, leading to more refined models. The use of different colors (orange, blue, green, red) helps to visually distinguish the different states and paths in the process. </details> Figure 5: ACC-Collab training pipeline, exemplified for the actor (top) and critic (bottom). The process 1) We generate data from both natural deliberation as well as guided deliberation towards and away from the ground truth answer $y$ using the actor and critic. 2) We compute the relative quality of each trajectory based on the expected quality difference $\Delta_{y},\Delta_{!y}$ w.r.t. to the natural response. 3) We store all high-quality pairwise data in our database and train the actor model. 4) We alternate this procedure for the actor and critic. As outline in Section 4 both the guidance procedure and the computation of $\Delta_{y},\Delta_{!y}$ differ between the actor and critic. A.1 Correctness of Reasoning Steps In addition to measuring outcome accuracy (i.e., the accuracy of the actor’s final answer), we also measure process accuracy (i.e., the accuracy of the reasoning and discussion steps taken by the actor and the critic model). These results are displayed in Table 3. We observe that our training pipeline typically maintains or increases the accuracy of the reasoning steps. Unlike outcome accuracy, process accuracy does not have ground truth answers. Thus we utilize GPT-4o to serve as an oracle. We use the following steps to evaluate the correctness of the reasoning and discussion steps. 1. For both an untrained actor-critic team and a trained actor-critic team (trained via ACC-Collab), we randomly sample 500 deliberations, which resulted in the correct answer after 5 rounds of deliberation. 1. We then prompted GPT-4o to evaluate the correctness of the reasoning and discussion steps taken by the actor-critic teams in each of the 500 deliberations. 1. We then compute the average accuracy of the reasoning and discussion steps as evaluated by GPT-4o. | Dataset Llama-3 BoolQ | ACC-Collab+ Accuracy $.890_{±.035}$ | ACC-Collab Accuracy $.894_{±.034}$ | Untrained Accuracy $.794_{±.045}$ | | --- | --- | --- | --- | | MMLU | $.968_{±.020}$ | $.976_{±.017}$ | $.909_{±.032}$ | | BBH | $.519_{±.064}$ | $.575_{±.055}$ | $.610_{±.055}$ | | SCIQ | $.958_{±.022}$ | $.981_{±.015}$ | $.937_{±.027}$ | | ARC | $.882_{±.046}$ | $.833_{±.042}$ | $.883_{±.036}$ | | Mistral | | | | | BoolQ | $.907_{±.035}$ | $.928_{±.029}$ | $.894_{±.034}$ | | MMLU | $.882_{±.042}$ | $.884_{±.036}$ | $.878_{±.037}$ | | BBH | $.766_{±.045}$ | $.818_{±.043}$ | $.538_{±.056}$ | | SCIQ | $.973_{±.018}$ | $.972_{±.018}$ | $.933_{±.028}$ | | ARC | $.927_{±.029}$ | $.952_{±.024}$ | $.852_{±.040}$ | | Gemma-2 | | | | | BoolQ | $.869_{±.038}$ | $.872_{±.037}$ | $.861_{±.039}$ | | MMLU | $.844_{±.041}$ | $.843_{±.041}$ | $.800_{±.045}$ | | BBH | $.597_{±.056}$ | $.637_{±.054}$ | $.514_{±.056}$ | | SCIQ | $.984_{±.014}$ | $.978_{±.016}$ | $.985_{±.014}$ | | ARC | $.850_{±.040}$ | $.854_{±.039}$ | $.834_{±.042}$ | Table 3: Accuracy of the actor’s and critic’s reasoning and discussion steps as evaluated by GPT-4o. Appendix B Preference Optimization Here, we remark on how other preference optimization schemes can be used in place of DPO. Broadly speaking, preference optimization schemes can broken into two categories: those which optimize reward directly (e.g., PPO) and those which optimize reward indirectly (e.g., DPO). In the case of the latter, the positive and negative pairs produced by our method (Algorithm 1) can be directly plugged into the preference optimizer. In the case of explicit reward maximization, the reward function $r(z^{(t)},x,y)$ can first be learned automatically by simply simulating deliberation. The most straightforward way to do this is to first simulate one full deliberation between the actor and critic, i.e., $\langle(z_{a}^{(0)},z_{c}^{(0)}),...,(z_{a}^{(T)},z_{c}^{(T)})\rangle$ . Then for each $z_{a}^{(t)}$ , $z_{c}^{(t)}$ the remaining $T-t$ deliberation steps can be resampled to estimate the corresponding value of $r(z_{a}^{(t)},x,y)$ and $r(z_{c}^{(t)},x,y)$ . These pairs, namely $\big{(}z_{a}^{(t)},r(z_{a}^{(t)},x,y)\big{)}$ and $\big{(}z_{c}^{(t)},r(z_{c}^{(t)},x,y)\big{)}$ can then be used to learn the reward function. This reward function can then be plugged into the desired preference optimization scheme. <details> <summary>extracted/6248978/plots/new_perRoundConf.png Details</summary>  ### Visual Description ## Chart: Model Accuracy Comparison Across Rounds ### Overview The image presents three line charts comparing the accuracy of different language models (Llama-3, Mistral, and Gemma-2) across multiple rounds of interaction or training. Each chart displays the performance of several models and training configurations, with accuracy on the y-axis and round number on the x-axis. The legend at the bottom identifies each line by color and label. ### Components/Axes * **Titles:** Each chart has a title indicating the language model being evaluated: "Llama-3", "Mistral", and "Gemma-2". * **X-Axis:** Labeled "Round", with integer markers at 0, 1, 2, 3, and 4. * **Y-Axis:** Labeled "Accuracy", with a scale from 0.725 to 0.900 for Llama-3 and Mistral, and from 0.700 to 0.850 for Gemma-2. * **Legend:** Located at the bottom of the image, it maps line colors to model/training configurations: * Blue dashed line: SoM (2x) * Teal dashed line: SoM (4x) * Purple dashed line: Persona * Brown solid line: DebateTune * Light Green solid line: SFT * Dark Green solid line: DebateGPT * Orange solid line: ACC-Collab (Ours) * Red solid line: ACC-Collab + (Ours) ### Detailed Analysis #### Llama-3 Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.845 at Round 0, increases to about 0.890 at Round 1, and then plateaus around 0.890 for Rounds 2-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.840 at Round 0, increases to about 0.885 at Round 1, and then plateaus around 0.885 for Rounds 2-4. * **DebateGPT (Dark Green):** Starts at approximately 0.800 at Round 0, increases to about 0.820 at Round 1, and then plateaus around 0.820 for Rounds 2-4. * **SoM (4x) (Teal):** Starts at approximately 0.780 at Round 0, increases to about 0.815 at Round 1, and then plateaus around 0.815 for Rounds 2-4. * **SoM (2x) (Blue):** Starts at approximately 0.770 at Round 0, increases to about 0.810 at Round 1, and then plateaus around 0.810 for Rounds 2-4. * **SFT (Light Green):** Starts at approximately 0.780 at Round 0, increases to about 0.800 at Round 1, and then plateaus around 0.800 for Rounds 2-4. * **Persona (Purple):** Starts at approximately 0.770 at Round 0, increases to about 0.780 at Round 1, and then plateaus around 0.780 for Rounds 2-4. * **DebateTune (Brown):** Starts at approximately 0.780 at Round 0, decreases to about 0.760 at Round 1, increases to about 0.780 at Round 2, and then plateaus around 0.780 for Rounds 3-4. #### Mistral Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.860 at Round 0, increases to about 0.890 at Round 1, and then plateaus around 0.890 for Rounds 2-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.830 at Round 0, increases to about 0.870 at Round 1, and then plateaus around 0.875 for Rounds 2-4. * **DebateGPT (Dark Green):** Starts at approximately 0.830 at Round 0, increases to about 0.850 at Round 1, and then plateaus around 0.850 for Rounds 2-4. * **SoM (4x) (Teal):** Starts at approximately 0.810 at Round 0, increases to about 0.820 at Round 1, and then decreases to about 0.800 for Rounds 2-4. * **SoM (2x) (Blue):** Starts at approximately 0.805 at Round 0, increases to about 0.815 at Round 1, and then decreases to about 0.800 for Rounds 2-4. * **SFT (Light Green):** Starts at approximately 0.815 at Round 0, increases to about 0.830 at Round 1, and then plateaus around 0.830 for Rounds 2-4. * **Persona (Purple):** Starts at approximately 0.815 at Round 0, increases to about 0.830 at Round 1, and then plateaus around 0.830 for Rounds 2-4. * **DebateTune (Brown):** Starts at approximately 0.825 at Round 0, increases to about 0.830 at Round 1, and then plateaus around 0.830 for Rounds 2-4. #### Gemma-2 Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.770 at Round 0, increases to about 0.840 at Round 1, and then plateaus around 0.840 for Rounds 2-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.780 at Round 0, increases to about 0.845 at Round 1, and then plateaus around 0.840 for Rounds 2-4. * **DebateGPT (Dark Green):** Starts at approximately 0.790 at Round 0, increases to about 0.825 at Round 1, and then plateaus around 0.820 for Rounds 2-4. * **SoM (4x) (Teal):** Starts at approximately 0.720 at Round 0, increases to about 0.760 at Round 1, and then plateaus around 0.760 for Rounds 2-4. * **SoM (2x) (Blue):** Starts at approximately 0.740 at Round 0, increases to about 0.760 at Round 1, and then plateaus around 0.760 for Rounds 2-4. * **SFT (Light Green):** Starts at approximately 0.760 at Round 0, increases to about 0.790 at Round 1, and then plateaus around 0.790 for Rounds 2-4. * **Persona (Purple):** Starts at approximately 0.715 at Round 0, increases to about 0.730 at Round 1, and then plateaus around 0.720 for Rounds 2-4. * **DebateTune (Brown):** Starts at approximately 0.740 at Round 0, increases to about 0.765 at Round 1, and then plateaus around 0.765 for Rounds 2-4. ### Key Observations * **ACC-Collab Performance:** The "ACC-Collab + (Ours)" and "ACC-Collab (Ours)" configurations consistently outperform other methods across all three language models. * **Initial Round Improvement:** Most models show a significant increase in accuracy from Round 0 to Round 1, suggesting that the initial round is crucial for learning or adaptation. * **Plateau Effect:** After Round 1, the accuracy of most models tends to plateau, indicating diminishing returns from additional rounds. * **Model-Specific Performance:** The relative performance of different training configurations varies slightly depending on the base language model. For example, the gap between "ACC-Collab" and other methods is more pronounced for Gemma-2 compared to Llama-3. * **SoM (2x) and SoM (4x) Decline in Mistral:** The accuracy of SoM (2x) and SoM (4x) declines after Round 1 in Mistral. ### Interpretation The data suggests that the "ACC-Collab" training configurations are highly effective in improving the accuracy of language models, particularly in the initial rounds of interaction or training. The plateau effect observed after Round 1 indicates that there may be a limit to the improvements achievable through the tested methods, or that different strategies are needed to further enhance performance beyond this point. The model-specific variations highlight the importance of tailoring training approaches to the specific characteristics of each language model. The decline of SoM (2x) and SoM (4x) in Mistral suggests that these configurations may not be as well-suited for this particular model. </details> Figure 6: Accuracy over five rounds of deliberation on BoolQ. <details> <summary>extracted/6248978/plots/new_perRoundMMLU.png Details</summary>  ### Visual Description ## Line Charts: Model Accuracy Comparison ### Overview The image presents three line charts comparing the accuracy of different models (Llama-3, Mistral, and Gemma-2) across several rounds of interaction. Each chart displays the performance of various training methods, including SoM (2x), SoM (4x), Persona, DebateTune, SFT, DebateGPT, ACC-Collab (Ours), and ACC-Collab + (Ours). The x-axis represents the round number (0 to 4), and the y-axis represents the accuracy, ranging from approximately 0.62 to 0.68 for Llama-3 and Mistral, and from 0.48 to 0.58 for Gemma-2. Each line represents a different training method, with shaded areas indicating the confidence interval or variability in the accuracy. ### Components/Axes * **Titles:** * Top-Left Chart: "Llama-3" * Top-Middle Chart: "Mistral" * Top-Right Chart: "Gemma-2" * **X-Axis:** * Label: "Round" * Scale: 0, 1, 2, 3, 4 * **Y-Axis:** * Label: "Accuracy" * Left Chart Scale: 0.62, 0.64, 0.66, 0.68 * Middle Chart Scale: 0.56, 0.58, 0.60, 0.62, 0.64, 0.66, 0.68 * Right Chart Scale: 0.48, 0.50, 0.52, 0.54, 0.56, 0.58 * **Legend:** Located at the bottom of the image. * Blue Dashed Line: "SoM (2x)" * Teal Dashed Line: "SoM (4x)" * Purple Dashed Line: "Persona" * Brown Solid Line: "DebateTune" * Light Green Solid Line: "SFT" * Dark Green Solid Line: "DebateGPT" * Orange Solid Line: "ACC-Collab (Ours)" * Red Solid Line: "ACC-Collab + (Ours)" ### Detailed Analysis **Llama-3 Chart:** * **SoM (2x) (Blue Dashed):** Starts at approximately 0.61 and increases slightly to around 0.62, remaining relatively flat. * **SoM (4x) (Teal Dashed):** Starts at approximately 0.61 and increases slightly to around 0.64, remaining relatively flat. * **Persona (Purple Dashed):** Starts at approximately 0.62 and increases slightly to around 0.64, remaining relatively flat. * **DebateTune (Brown Solid):** Starts at approximately 0.62 and increases slightly to around 0.63, remaining relatively flat. * **SFT (Light Green Solid):** Starts at approximately 0.58 and increases to approximately 0.65 by round 3, then decreases slightly to around 0.64 by round 4. * **DebateGPT (Dark Green Solid):** Starts at approximately 0.64 and increases to approximately 0.65 by round 3, then decreases slightly to around 0.64 by round 4. * **ACC-Collab (Ours) (Orange Solid):** Starts at approximately 0.63 and increases to approximately 0.65 by round 3, then decreases slightly to around 0.64 by round 4. * **ACC-Collab + (Ours) (Red Solid):** Starts at approximately 0.64 and increases to approximately 0.67 by round 1, then increases to approximately 0.67 by round 2, then increases to approximately 0.67 by round 3, then increases to approximately 0.68 by round 4. **Mistral Chart:** * **SoM (2x) (Blue Dashed):** Starts at approximately 0.56 and increases slightly to around 0.57, remaining relatively flat. * **SoM (4x) (Teal Dashed):** Starts at approximately 0.55 and increases slightly to around 0.57, remaining relatively flat. * **Persona (Purple Dashed):** Starts at approximately 0.55 and increases slightly to around 0.57, remaining relatively flat. * **DebateTune (Brown Solid):** Starts at approximately 0.55 and increases slightly to around 0.57, remaining relatively flat. * **SFT (Light Green Solid):** Starts at approximately 0.58 and increases to approximately 0.60 by round 1, then decreases slightly to around 0.59 by round 4. * **DebateGPT (Dark Green Solid):** Starts at approximately 0.57 and increases to approximately 0.58 by round 1, then decreases slightly to around 0.57 by round 4. * **ACC-Collab (Ours) (Orange Solid):** Starts at approximately 0.59 and increases to approximately 0.61 by round 1, then increases to approximately 0.60 by round 3, then increases to approximately 0.61 by round 4. * **ACC-Collab + (Ours) (Red Solid):** Starts at approximately 0.64 and increases to approximately 0.67 by round 1, then increases to approximately 0.67 by round 2, then increases to approximately 0.67 by round 3, then increases to approximately 0.67 by round 4. **Gemma-2 Chart:** * **SoM (2x) (Blue Dashed):** Starts at approximately 0.57 and increases slightly to around 0.58, remaining relatively flat. * **SoM (4x) (Teal Dashed):** Starts at approximately 0.55 and increases slightly to around 0.57, remaining relatively flat. * **Persona (Purple Dashed):** Starts at approximately 0.55 and increases slightly to around 0.57, remaining relatively flat. * **DebateTune (Brown Solid):** Starts at approximately 0.55 and increases slightly to around 0.57, remaining relatively flat. * **SFT (Light Green Solid):** Starts at approximately 0.57 and increases to approximately 0.57 by round 1, then increases to approximately 0.57 by round 4. * **DebateGPT (Dark Green Solid):** Starts at approximately 0.57 and increases to approximately 0.57 by round 1, then increases to approximately 0.57 by round 4. * **ACC-Collab (Ours) (Orange Solid):** Starts at approximately 0.54 and decreases to approximately 0.50 by round 1, then decreases to approximately 0.50 by round 3, then increases to approximately 0.51 by round 4. * **ACC-Collab + (Ours) (Red Solid):** Starts at approximately 0.51 and increases to approximately 0.54 by round 1, then decreases to approximately 0.54 by round 2, then decreases to approximately 0.54 by round 3, then increases to approximately 0.55 by round 4. ### Key Observations * **ACC-Collab + (Ours) (Red Solid):** Consistently shows the highest accuracy across all three models (Llama-3, Mistral, and Gemma-2). * **SoM (2x) (Blue Dashed), SoM (4x) (Teal Dashed), Persona (Purple Dashed), and DebateTune (Brown Solid):** Generally exhibit lower and relatively stable accuracy across all rounds and models. * **SFT (Light Green Solid) and DebateGPT (Dark Green Solid):** Show moderate performance, with some variation across different models. * **ACC-Collab (Ours) (Orange Solid):** Shows variable performance, with a noticeable dip in accuracy for Gemma-2. ### Interpretation The charts provide a comparative analysis of different training methods applied to three language models (Llama-3, Mistral, and Gemma-2). The "ACC-Collab + (Ours)" method consistently outperforms other methods, suggesting it is the most effective in improving model accuracy across different architectures. The relatively flat performance of "SoM (2x)", "SoM (4x)", "Persona", and "DebateTune" indicates that these methods may not be as effective in enhancing accuracy for these models. The variability in "ACC-Collab (Ours)" performance, particularly the dip in accuracy for Gemma-2, suggests that its effectiveness may be model-dependent. Overall, the data suggests that the choice of training method significantly impacts the final accuracy of the language model, with "ACC-Collab + (Ours)" being a promising approach. </details> Figure 7: Accuracy over five rounds of deliberation on MMLU. <details> <summary>extracted/6248978/plots/new_perRoundBBH.png Details</summary>  ### Visual Description ## Line Charts: Model Accuracy vs. Round ### Overview The image presents three line charts comparing the accuracy of different models (Llama-3, Mistral, and Gemma-2) across several rounds of interaction or training. Each chart displays the performance of multiple methods, including "SoM (2x)", "SoM (4x)", "Persona", "DebateTune", "SFT", "DebateGPT", "ACC-Collab (Ours)", and "ACC-Collab + (Ours)". The x-axis represents the round number (0 to 4), and the y-axis represents the accuracy score. Shaded regions around each line indicate the uncertainty or variance in the accuracy. ### Components/Axes * **Titles:** * Top-left chart: "Llama-3" * Top-middle chart: "Mistral" * Top-right chart: "Gemma-2" * **X-axis:** "Round", with markers at 0, 1, 2, 3, and 4. * **Y-axis:** "Accuracy", ranging from 0.48 to 0.60 for Llama-3, 0.40 to 0.60 for Mistral, and 0.44 to 0.52 for Gemma-2. * **Legend:** Located at the bottom of the image, associating line colors and styles with specific methods: * Blue dashed line: "SoM (2x)" * Teal dashed line: "SoM (4x)" * Purple dashed line: "Persona" * Brown solid line: "DebateTune" * Light green solid line: "SFT" * Dark green solid line: "DebateGPT" * Orange solid line: "ACC-Collab (Ours)" * Red solid line: "ACC-Collab + (Ours)" ### Detailed Analysis #### Llama-3 Chart * **SoM (2x)** (Blue dashed): Starts at approximately 0.48 and increases to around 0.51, then remains relatively stable. * **SoM (4x)** (Teal dashed): Similar to SoM (2x), starting around 0.48 and increasing to approximately 0.51, then stabilizing. * **Persona** (Purple dashed): Starts around 0.48 and increases to approximately 0.50, then stabilizes. * **DebateTune** (Brown solid): Starts around 0.48 and increases to approximately 0.51, then stabilizes. * **SFT** (Light green solid): Starts around 0.52 and increases to approximately 0.56. * **DebateGPT** (Dark green solid): Starts around 0.53 and increases to approximately 0.55. * **ACC-Collab (Ours)** (Orange solid): Starts around 0.56, increases to approximately 0.59 at round 1, then decreases slightly to approximately 0.58 by round 4. * **ACC-Collab + (Ours)** (Red solid): Starts around 0.52, increases to approximately 0.57 by round 4. #### Mistral Chart * **SoM (2x)** (Blue dashed): Starts around 0.40 and increases to approximately 0.43, then remains relatively stable. * **SoM (4x)** (Teal dashed): Starts around 0.45 and increases slightly to approximately 0.47, then stabilizes. * **Persona** (Purple dashed): Starts around 0.45 and increases slightly to approximately 0.47, then stabilizes. * **DebateTune** (Brown solid): Starts around 0.42 and increases to approximately 0.46, then stabilizes. * **SFT** (Light green solid): Starts around 0.44 and increases slightly to approximately 0.46, then stabilizes. * **DebateGPT** (Dark green solid): Starts around 0.45 and increases slightly to approximately 0.48, then stabilizes. * **ACC-Collab (Ours)** (Orange solid): Starts around 0.48, increases to approximately 0.50 by round 1, then remains relatively stable. * **ACC-Collab + (Ours)** (Red solid): Starts around 0.57, increases to approximately 0.60 by round 1, then remains relatively stable. #### Gemma-2 Chart * **SoM (2x)** (Blue dashed): Starts around 0.44 and increases to approximately 0.46, then remains relatively stable. * **SoM (4x)** (Teal dashed): Starts around 0.48 and increases slightly to approximately 0.51, then stabilizes. * **Persona** (Purple dashed): Starts around 0.44 and increases slightly to approximately 0.46, then stabilizes. * **DebateTune** (Brown solid): Starts around 0.44 and increases slightly to approximately 0.45, then stabilizes. * **SFT** (Light green solid): Starts around 0.48 and increases slightly to approximately 0.49, then stabilizes. * **DebateGPT** (Dark green solid): Starts around 0.49 and increases slightly to approximately 0.50, then stabilizes. * **ACC-Collab (Ours)** (Orange solid): Starts around 0.48, increases to approximately 0.52 by round 4. * **ACC-Collab + (Ours)** (Red solid): Starts around 0.45, increases to approximately 0.48 by round 3, then decreases slightly to approximately 0.47 by round 4. ### Key Observations * **ACC-Collab + (Ours)** generally performs well across all models, often achieving the highest accuracy or a close second. * **ACC-Collab (Ours)** also shows strong performance, particularly in the Llama-3 and Gemma-2 models. * **SoM (2x), SoM (4x), Persona, and DebateTune** tend to have lower and more stable accuracy scores compared to the other methods. * The accuracy improvements tend to occur in the initial rounds (0-1), with diminishing returns in later rounds. * The shaded regions indicate varying degrees of uncertainty in the accuracy, with some methods showing more consistent performance than others. ### Interpretation The charts suggest that "ACC-Collab + (Ours)" and "ACC-Collab (Ours)" are effective methods for improving model accuracy across different language models (Llama-3, Mistral, and Gemma-2). The relatively flat performance of "SoM (2x)", "SoM (4x)", "Persona", and "DebateTune" indicates that these methods may not be as effective in this context. The initial rounds of interaction or training appear to be the most crucial for achieving accuracy gains. The uncertainty regions highlight the variability in performance, which could be due to factors such as data sampling or model initialization. The data demonstrates the relative effectiveness of different training or fine-tuning approaches for language models. </details> Figure 8: Accuracy over five rounds of deliberation on BBH. <details> <summary>extracted/6248978/plots/new_perRoundSCIQ.png Details</summary>  ### Visual Description ## Line Charts: Model Accuracy vs. Round ### Overview The image presents three line charts comparing the accuracy of different models (Llama-3, Mistral, and Gemma-2) across multiple rounds of interaction or training. Each chart displays the performance of several methods, including "SoM (2x)", "SoM (4x)", "Persona", "DebateTune", "SFT", "DebateGPT", "ACC-Collab (Ours)", and "ACC-Collab + (Ours)". The x-axis represents the round number (0 to 4), and the y-axis represents the accuracy score. Each line represents a different method, and shaded areas around the lines indicate the confidence interval or variability in the results. ### Components/Axes * **Titles:** The charts are titled "Llama-3" (top-left), "Mistral" (top-center), and "Gemma-2" (top-right). * **X-axis:** Labeled "Round", with markers at 0, 1, 2, 3, and 4. * **Y-axis:** Labeled "Accuracy", with different scales for each chart: * Llama-3: 0.91 to 0.95 * Mistral: 0.84 to 0.90 * Gemma-2: 0.89 to 0.92 * **Legend:** Located at the bottom of the image, mapping line colors/styles to methods: * Blue dashed line: SoM (2x) * Teal dashed line: SoM (4x) * Purple dashed line: Persona * Brown solid line: DebateTune * Light Green solid line: SFT * Dark Green solid line: DebateGPT * Orange solid line: ACC-Collab (Ours) * Red solid line: ACC-Collab + (Ours) ### Detailed Analysis #### Llama-3 Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.933 at round 0, increases to about 0.947 by round 1, and then plateaus around 0.948-0.949 for rounds 2-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.941 at round 0, increases to about 0.952 by round 1, and then plateaus around 0.953-0.954 for rounds 2-4. * **DebateGPT (Dark Green):** Starts at approximately 0.916 at round 0, increases to about 0.930 by round 1, and then plateaus around 0.932-0.933 for rounds 2-4. * **SoM (2x) (Blue Dashed):** Starts at approximately 0.917 at round 0, increases to about 0.924 by round 1, and then plateaus around 0.925-0.926 for rounds 2-4. * **SoM (4x) (Teal Dashed):** Starts at approximately 0.918 at round 0, increases to about 0.922 by round 1, and then plateaus around 0.923-0.924 for rounds 2-4. * **Persona (Purple Dashed):** Starts at approximately 0.918 at round 0, increases to about 0.924 by round 1, and then plateaus around 0.925-0.926 for rounds 2-4. * **DebateTune (Brown):** Starts at approximately 0.916 at round 0, increases to about 0.920 by round 1, and then plateaus around 0.921-0.922 for rounds 2-4. * **SFT (Light Green):** Starts at approximately 0.914 at round 0, increases to about 0.921 by round 1, and then plateaus around 0.922-0.923 for rounds 2-4. #### Mistral Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.872 at round 0, increases to about 0.899 by round 2, and then plateaus around 0.901-0.902 for rounds 3-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.877 at round 0, increases to about 0.895 by round 2, and then plateaus around 0.896-0.897 for rounds 3-4. * **DebateGPT (Dark Green):** Starts at approximately 0.859 at round 0, increases to about 0.871 by round 2, and then plateaus around 0.872-0.873 for rounds 3-4. * **SoM (2x) (Blue Dashed):** Starts at approximately 0.842 at round 0, increases to about 0.857 by round 2, and then plateaus around 0.858-0.859 for rounds 3-4. * **SoM (4x) (Teal Dashed):** Starts at approximately 0.860 at round 0, increases to about 0.863 by round 2, and then plateaus around 0.864-0.865 for rounds 3-4. * **Persona (Purple Dashed):** Starts at approximately 0.860 at round 0, increases to about 0.863 by round 2, and then plateaus around 0.864-0.865 for rounds 3-4. * **DebateTune (Brown):** Starts at approximately 0.859 at round 0, increases to about 0.860 by round 2, and then plateaus around 0.861-0.862 for rounds 3-4. * **SFT (Light Green):** Starts at approximately 0.837 at round 0, increases to about 0.857 by round 2, and then plateaus around 0.858-0.859 for rounds 3-4. #### Gemma-2 Chart * **ACC-Collab + (Ours) (Red):** Starts at approximately 0.890 at round 0, increases to about 0.906 by round 1, and then plateaus around 0.907-0.908 for rounds 2-4. * **ACC-Collab (Ours) (Orange):** Starts at approximately 0.895 at round 0, increases to about 0.918 by round 2, and then decreases to about 0.916 for round 4. * **DebateGPT (Dark Green):** Starts at approximately 0.898 at round 0, increases to about 0.913 by round 2, and then plateaus around 0.914-0.915 for rounds 3-4. * **SoM (2x) (Blue Dashed):** Starts at approximately 0.892 at round 0, increases to about 0.903 by round 2, and then plateaus around 0.904-0.905 for rounds 3-4. * **SoM (4x) (Teal Dashed):** Starts at approximately 0.901 at round 0, increases to about 0.905 by round 2, and then plateaus around 0.906-0.907 for rounds 3-4. * **Persona (Purple Dashed):** Starts at approximately 0.890 at round 0, increases to about 0.908 by round 2, and then plateaus around 0.909-0.910 for rounds 3-4. * **DebateTune (Brown):** Starts at approximately 0.888 at round 0, increases to about 0.904 by round 2, and then plateaus around 0.905-0.906 for rounds 3-4. * **SFT (Light Green):** Starts at approximately 0.899 at round 0, increases to about 0.909 by round 2, and then plateaus around 0.910-0.911 for rounds 3-4. ### Key Observations * **Initial Improvement:** Most methods show a significant increase in accuracy from round 0 to round 1 or 2. * **Plateau Effect:** After the initial improvement, the accuracy tends to plateau, indicating diminishing returns from further rounds. * **ACC-Collab Superiority:** The "ACC-Collab (Ours)" and "ACC-Collab + (Ours)" methods generally outperform the other methods across all three models. * **Model-Specific Performance:** The absolute accuracy values differ across the models, with Llama-3 generally achieving higher accuracy scores than Mistral and Gemma-2. ### Interpretation The charts demonstrate the impact of different methods on the accuracy of three language models (Llama-3, Mistral, and Gemma-2) over multiple rounds. The "ACC-Collab" methods appear to be the most effective in improving accuracy, suggesting that collaborative approaches enhance model performance. The plateau effect observed after a few rounds indicates that the models may reach a point of diminishing returns with the given methods and data. The differences in absolute accuracy across the models highlight the inherent variations in their architectures and training data. The shaded areas around the lines provide insights into the variability of the results, which is crucial for assessing the robustness of the methods. </details> Figure 9: Accuracy over five rounds of deliberation on SCIQ. <details> <summary>extracted/6248978/plots/new_perRoundARC.png Details</summary>  ### Visual Description ## Line Charts: Model Accuracy vs. Round ### Overview The image presents three line charts comparing the accuracy of different models (Llama-3, Mistral, and Gemma-2) across several rounds of training or interaction. Each chart displays the performance of various methods, including "SoM (2x)", "SoM (4x)", "Persona", "DebateTune", "SFT", "DebateGPT", "ACC-Collab (Ours)", and "ACC-Collab + (Ours)". The x-axis represents the round number (0 to 4), and the y-axis represents the accuracy, ranging from approximately 0.855 to 0.885 for Llama-3, 0.81 to 0.86 for Mistral, and 0.81 to 0.85 for Gemma-2. ### Components/Axes * **Titles:** * Top-left chart: "Llama-3" * Top-center chart: "Mistral" * Top-right chart: "Gemma-2" * **X-axis:** * Label: "Round" * Scale: 0, 1, 2, 3, 4 * **Y-axis:** * Label: "Accuracy" * Scale (Llama-3): 0.855, 0.860, 0.865, 0.870, 0.875, 0.880, 0.885 * Scale (Mistral): 0.81, 0.82, 0.83, 0.84, 0.85, 0.86 * Scale (Gemma-2): 0.81, 0.82, 0.83, 0.84, 0.85 * **Legend:** Located at the bottom of the image. * Blue dashed line: "SoM (2x)" * Teal dashed line: "SoM (4x)" * Purple dashed line: "Persona" * Brown solid line: "DebateTune" * Light green solid line: "SFT" * Dark green solid line: "DebateGPT" * Orange solid line: "ACC-Collab (Ours)" * Red solid line: "ACC-Collab + (Ours)" ### Detailed Analysis #### Llama-3 Chart * **SoM (2x)** (Blue dashed line): Starts at approximately 0.860 at round 0, increases to around 0.872 by round 1, and then remains relatively stable around 0.872 through round 4. * **SoM (4x)** (Teal dashed line): Starts at approximately 0.862 at round 0, increases to around 0.873 by round 1, and then remains relatively stable around 0.873 through round 4. * **Persona** (Purple dashed line): Starts at approximately 0.860 at round 0, increases to around 0.871 by round 1, and then decreases slightly to around 0.870 by round 4. * **DebateTune** (Brown solid line): Starts at approximately 0.863 at round 0, increases to around 0.871 by round 1, and then remains relatively stable around 0.871 through round 4. * **SFT** (Light green solid line): Starts at approximately 0.878 at round 0, increases slightly to around 0.880 by round 1, and then remains relatively stable around 0.878 through round 4. * **DebateGPT** (Dark green solid line): Starts at approximately 0.868 at round 0, increases to around 0.877 by round 1, and then remains relatively stable around 0.877 through round 4. * **ACC-Collab (Ours)** (Orange solid line): Starts at approximately 0.872 at round 0, increases to around 0.882 by round 1, and then remains relatively stable around 0.881 through round 4. * **ACC-Collab + (Ours)** (Red solid line): Starts at approximately 0.858 at round 0, increases to around 0.871 by round 1, and then decreases slightly to around 0.869 by round 4. #### Mistral Chart * **SoM (2x)** (Blue dashed line): Starts at approximately 0.820 at round 0, increases slightly to around 0.822 by round 1, and then remains relatively stable around 0.822 through round 4. * **SoM (4x)** (Teal dashed line): Starts at approximately 0.810 at round 0, increases to around 0.820 by round 1, and then remains relatively stable around 0.820 through round 4. * **Persona** (Purple dashed line): Starts at approximately 0.815 at round 0, increases to around 0.823 by round 1, and then remains relatively stable around 0.823 through round 4. * **DebateTune** (Brown solid line): Starts at approximately 0.830 at round 0, increases to around 0.838 by round 1, and then remains relatively stable around 0.838 through round 4. * **SFT** (Light green solid line): Starts at approximately 0.820 at round 0, increases slightly to around 0.825 by round 1, and then remains relatively stable around 0.825 through round 4. * **DebateGPT** (Dark green solid line): Starts at approximately 0.818 at round 0, increases slightly to around 0.822 by round 1, and then remains relatively stable around 0.822 through round 4. * **ACC-Collab (Ours)** (Orange solid line): Starts at approximately 0.822 at round 0, increases to around 0.835 by round 1, and then remains relatively stable around 0.838 through round 4. * **ACC-Collab + (Ours)** (Red solid line): Starts at approximately 0.835 at round 0, increases to around 0.853 by round 1, and then remains relatively stable around 0.855 through round 4. #### Gemma-2 Chart * **SoM (2x)** (Blue dashed line): Starts at approximately 0.820 at round 0, increases to around 0.838 by round 1, and then remains relatively stable around 0.840 through round 4. * **SoM (4x)** (Teal dashed line): Starts at approximately 0.825 at round 0, increases to around 0.842 by round 1, and then remains relatively stable around 0.843 through round 4. * **Persona** (Purple dashed line): Starts at approximately 0.818 at round 0, increases to around 0.838 by round 1, and then remains relatively stable around 0.840 through round 4. * **DebateTune** (Brown solid line): Starts at approximately 0.818 at round 0, increases to around 0.838 by round 1, and then remains relatively stable around 0.840 through round 4. * **SFT** (Light green solid line): Starts at approximately 0.830 at round 0, increases to around 0.848 by round 1, and then remains relatively stable around 0.850 through round 4. * **DebateGPT** (Dark green solid line): Starts at approximately 0.828 at round 0, increases to around 0.845 by round 1, and then remains relatively stable around 0.848 through round 4. * **ACC-Collab (Ours)** (Orange solid line): Starts at approximately 0.822 at round 0, increases to around 0.848 by round 1, and then remains relatively stable around 0.852 through round 4. * **ACC-Collab + (Ours)** (Red solid line): Starts at approximately 0.815 at round 0, increases to around 0.840 by round 1, and then remains relatively stable around 0.848 through round 4. ### Key Observations * Across all three models (Llama-3, Mistral, and Gemma-2), most methods show a significant increase in accuracy from round 0 to round 1. * After round 1, the accuracy of most methods tends to stabilize, with only minor fluctuations. * The "ACC-Collab + (Ours)" method generally performs well, often achieving the highest accuracy among the compared methods, especially for Mistral. * The "SoM (2x)", "SoM (4x)", and "Persona" methods tend to have similar performance across all three models. ### Interpretation The charts suggest that the initial round of training or interaction is crucial for improving the accuracy of these models. The stabilization of accuracy after round 1 indicates that the models may be reaching a point of diminishing returns, where further rounds do not significantly improve performance. The "ACC-Collab + (Ours)" method appears to be a promising approach, as it consistently achieves high accuracy compared to other methods. The similar performance of "SoM (2x)", "SoM (4x)", and "Persona" suggests that these methods may have similar underlying mechanisms or limitations. Further investigation could focus on understanding why "ACC-Collab + (Ours)" is particularly effective and exploring strategies to overcome the performance plateau observed after round 1. </details> Figure 10: Accuracy over five rounds of deliberation on ARC. Appendix C Examples C.1 Prompts Here, we provide examples of the prompts used in our experiments. For illustration, we provide prompts for the BoolQ dataset in which agents are asked a yes-no question about a passage. Single-Shot Prompt (No Deliberation) prompt = (’You will be given a yes-no question which is based on a passage. ’ ’You should use the passage to help you answer the question. ’ ’You should give a brief justification for your answer, ’ ’and you must provide a final answer of either Yes or No.’ ’\nQuestion: {question}?’ ’\nPassage: {passage}’ ) Guided Single-Shot Prompt (No Deliberation) prompt = (’You will be given a yes-no question which is based on a passage. ’ ’You should use the passage to help you answer the question ’ ’with a {target_answer}. ’ ’You should give a brief justification for your answer of {target_answer}, and you must state that your final answer is {target_answer}.’ ’\nQuestion: {question}?’ ’\nPassage: {passage}’ ) Deliberation Prompt for Actor prompt = (’Several people have provided answers to a yes-no question. ’ ’Below are their responses:’ ’\nPerson {1} said: {responses[1]}’ ’\nPerson {2} said: {responses[2]}’ . . . ’\nPerson {n} said: {responses[n]}’ ’\n\nYou should take these answers into consideration when answering ’ ’the following yes-no question, which is based on a passage. ’ ’You should give a brief justification for your answer, and you must ’ ’provide a final answer of either Yes or No.’ ’\nQuestion: {question}’ ’\nPassage: {passage}’ ) Guided Deliberation Prompt for Actor prompt = (’Several people have provided answers to a yes-no question. ’ ’Below are their responses:’ ’\nPerson {1} said: {responses[1]}’ ’\nPerson {2} said: {responses[2]}’ . . . ’\nPerson {n} said: {responses[n]}’ ’You should take these answers and the passage into consideration when ’answering the following question with {target_answer}. ’ ’You should give a brief justification for your answer of {target_answer}, ’and you must state that your final answer is {target_answer}.’ ’\nQuestion: {question}’ ’\nPassage: {passage}’ ) Deliberation Prompt for Critic prompt = (’I am answering a question based on a passage. ’ ’I would like you to help me improve the correctness of my response ’ ’by briefly providing some additional details my original response ’ ’may have missed. ’ ’\nQuestion: {question}’ ’\nPassage: {passage}’ ’\nMy Response: {actor_response}’ ) Guided Deliberation Prompt for Critic prompt = (’I would like you to be a deliberation assistant. ’ ’You will be given a question based on a passage and my response ’ ’to the question. ’ ’You should use the passage and my response to develop additional details ’for why the correct answer is {target_answer}. ’ ’Your details must be brief and must support the fact that the ’ ’correct answer is {target_answer}.’ ’\nQuestion: {question}?’ ’\nPassage: {passage}’ ’\nMy Response: {actor_response}’ ) C.2 Critic Responses Here, we present several examples of the critic’s responses when the actor provides the wrong answer; critic responses are shown before and after training with ACC-Collab. We observed a qualitative change in the critic’s responses after training with our pipeline; the critic was more willing to disagree with the actor, and the critic provided more detailed feedback. ACTOR: [Wrong] Final answer, No. Non-residents cannot bring a handgun into New York for possession or carry within the state based on the passage. They can only transport it through New York to a destination outside of the state where it is legal to possess and carry. UNTRAINED CRITIC: Your response is mostly correct. I would add that New York has strict gun control laws, so even if a non-resident is allowed to transport a firearm through the state, they should be aware that certain areas may have additional restrictions, such as prohibitions on carrying the firearm in certain public places or parks. It’s always a good idea to check the specific laws of the jurisdiction they will be passing through and their final destination. TRAINED CRITIC: Details refuting your response: 1. The passage mentions that non-residents can transport any lawful firearm through New York, which implies that the firearm is temporarily in New York state territory during transit. This suggests that under certain circumstances, a handgun could be brought into New York for transit purposes, even if it cannot be carried within the state. 2. The passage does not explicitly state that non-residents are prohibited from bringing handguns into New York. Instead, it only specifies that they cannot possess or carry handguns within the state. This ambiguity leaves room for interpretation that a handgun could be brought into New York for transit purposes as long as it is not carried within the state. ACTOR: [Wrong] Answer: Yes, the passage states that in the Northern Atlantic Ocean, the hurricane season starts as early as June 1 and ends on November 30, with the peak of activity occurring from late August through September. However, the passage also mentions an exception where the hurricane season started as early as January 3 in 1938. Therefore, while the typical start of the hurricane season is June 1, it is possible for the season to begin earlier, as indicated by the passage. UNTRAINED CRITIC: Your response is correct and provides a good justification based on the passage. However, you could add that the passage mentions that May is the least active month for tropical cyclone activity worldwide, but May is still within the Atlantic hurricane season. Therefore, while the peak of the Atlantic hurricane season is from late August through September, the season technically starts earlier. TRAINED CRITIC: While your response is correct in stating that the typical hurricane season in the Northern Atlantic Ocean starts on June 1 and ends on November 30, with the peak of activity occurring from late August through September, the passage also emphasizes that each particular basin has its own seasonal patterns. The passage mentions the exception of the 1938 hurricane season starting as early as January 3, which is outside the typical seasonal window. This detail directly refutes your response by emphasizing that while the typical season starts in June, it is not unheard of for the season to begin earlier, as demonstrated by historical records. Another detail from the passage that supports the fact that the answer is No, is that the passage states that ”worldwide, tropical cyclone activity peaks in late summer,” which is contradictory to your response that the hurricane season starts in spring. The passage explicitly states that the peak of tropical cyclone activity occurs in late summer, which is later than spring. ACTOR: [Wrong] Final Answer: (A) rated the group as less interesting and enjoyable. Justification: Aronson and Mills’ study found that severe initiation processes negatively impact immediate attitudes towards the group due to discomfort and negative emotions. (Person 0) Additionally, severe initiation may create pressure to conform and social comparison, making the group less appealing. (Person 1) UNTRAINED CRITIC: Your response is correct. I’d add that Aronson and Mills found that severe initiation rites can create a sense of exclusion and pain, which may negatively impact attitudes towards the group. (A) TRAINED CRITIC: Response: While your justification supports the fact that severe initiation can negatively impact attitudes towards the group, it does not fully refute the alternative answers. Let me provide brief details to argue for the correct answer, B being ”rated the group as more interesting and enjoyable”: 1. Counter-intuitive findings: Research suggests that negative experiences can sometimes lead to stronger group identification and commitment. The discomfort and negative emotions generated during severe initiation may create a sense of camaraderie and solidarity among group members, ultimately increasing their overall enjoyment of the group (Tajfel & Turner, 1986). 2. Long-term benefits: Although severe initiation might result in negative emotions initially, the feelings of bonding and group cohesion that develop during the initiation process can endure and even enhance members’ attachment to the group over time (Sue Carter & Branscombe, 1994). This increased commitment can lead to a heightened sense of enjoyment and engagement with the group. ACTOR: [Wrong] Final Answer: (B) Catalhoyuk and Gobekli Tepe Justification: These sites have provided evidence for social complexity without clear agricultural surpluses. Catalhoyuk had centralized planning and hierarchical social organization but no conclusive evidence for agricultural surpluses. Gobekli Tepe was likely a religious or ceremonial center predating the agricultural revolution. UNTRAINNED CRITIC: Your response is correct. I’d add that Catalhoyuk had a population of up to 8,000 people and was the world’s first known urban settlement, while Gobekli Tepe’s T-shaped pillars date back to around 9600 BCE and suggest the existence of social and religious institutions that predate agriculture. TRAINED CRITIC: While your justification for option (B) is correct, I would like to provide additional details for option (D) to refute potential responses in favor of other options. Option (A): Stonehenge and Caral Justification: Although these sites have evidence of social complexity, there is no clear evidence that they lacked agricultural surpluses. Stonehenge, for instance, is believed to have served as a ceremonial and astronomical center, but its relationship to agriculture remains unclear. Caral, on the other hand, has evidence of irrigation systems, suggesting some agricultural production, even if not a surplus. Option (C): La Venta and Catalhoyuk Justification: La Venta is an example of a complex civilization with evidence of agricultural surpluses. While Catalhoyuk is an example of social complexity without clear agricultural surpluses, as you mentioned, La Venta challenges the idea that social complexity always required surpluses. Option (D): Gobekli Tepe and Poverty Point Justification: Gobekli Tepe is believed to be a religious or ceremonial center that predates the agricultural revolution. Poverty Point, on the other hand, is a complex society that existed around 3200 BCE and had advanced engineering skills but no clear evidence of agricultural surpluses. These sites’ existence undermines the hypothesis that social complexity always required agricultural surpluses, providing further evidence for option (D) being the correct answer. ACTOR: [Wrong] Final Answer: (A) psychological reactance. This is because your friend’s statement suggests that they feel frustrated or resistant when external factors prevent them from going on vacation, indicating a desire to have more freedom to make their own choices. Psychological reactance is the tendency to react negatively when our freedom is threatened. UNTRAINED CRITIC: Your response is correct. I’d add that psychological reactance often results in an exaggerated perception of the importance or severity of the obstacles, leading your friend to believe that there’s always a crisis preventing them from vacationing when in reality, the obstacles may not be as significant as they seem. TRAINED CRITIC: While your response is plausible and accurately describes psychological reactance, let me provide two brief reasons why the correct answer is still (C) illusory correlation: 1. Illusory correlation refers to the perception of a relationship between variables when none or only a weak relationship actually exists. In this case, your friend might perceive a strong correlation between planning a vacation and experiencing crises at home or work, leading them to believe that their vacations are consistently disrupted, even if the actual correlation is weak or non-existent. 2. Psychological reactance is primarily concerned with an individual’s desire to react negatively when their freedom is restricted or threatened, not necessarily a misperception of causal relationships. However, the friend’s statement does imply a belief that external factors are consistently preventing them from going on vacations, which could be a manifestation of illusory correlation. ACTOR: [Wrong] Final Answer: (D) a zygote. Justification: A zygote is the first cell formed when two gametes merge, marking the beginning of organismic development. UNTRAINED CRITIC: Your response is correct. Just to add, a zygote is formed by the fusion of an sperm (male gamete) and an egg (female gamete), resulting in a diploid cell containing genetic material from both parents. This is the initial stage of an organism’s development. TRAINED CRITIC: I agree with your response that the answer is (D) a zygote. However, to further support why (A) a gamete and (C) a nucleus are incorrect, let me provide brief details: (A) A gamete is a reproductive cell that fuses with another gamete during fertilization to form a zygote. Therefore, it is not an organism in the earliest stages of development but rather a part of the process leading to the formation of an organism. (C) A nucleus is an organelle within a cell that contains genetic material. It is not an organism but rather a component of a cell, and it is present in the zygote after fertilization has occurred. So, both (A) and (C) can be ruled out as they do not represent the organism in the earliest stages of development, and the correct answer is (D) a zygote. ACTOR: [Wrong] Final Answer: (A) metamorphic rocks and (B) any types of rocks (specifically igneous rocks when they undergo metamorphosis). UNTRAINED CRITIC: Your response is mostly correct, but it would be more precise to clarify that options B and C also include metamorphic rocks when they mention ”any types of rocks” and ”igneous rocks,” respectively. Metamorphic rocks form from the alteration of pre-existing rocks under heat, pressure, or chemical conditions. Thus, all three options (A, B, and C) are correct, but option A is the most explicit about the transformation process involved. TRAINED CRITIC: Sure! Here are two brief details supporting the correct answer (A) metamorphic rocks and (B) any types of rocks (specifically igneous rocks when they undergo metamorphosis): 1. Metamorphic rocks result from the transformation of pre-existing rocks under high pressure and temperature conditions. This process causes the minerals in the rocks to change, resulting in a new type of rock. For example, sedimentary rocks like shale or limestone can be transformed into metamorphic rocks like slate or marble, respectively. 2. Igneous rocks can also undergo metamorphosis when they are exposed to high pressure and temperature conditions. This process results in the recrystallization of minerals and the formation of new textures. For example, granite can be transformed into gneiss or schist when subjected to metamorphic conditions. Therefore, any type of rock, including igneous rocks, can change and become new types of rocks through metamorphosis. ACTOR: [Wrong] Final Answer: (C) white present Justification: Gertrude initially had a yellow present, but she swapped it with Alice, who had a white present. Therefore, Gertrude now has the white present. UNTRAINED CRITIC: Your response is correct. I’d add that after Gertrude swaps with Alice, none of the following swaps involve Gertrude giving away or receiving the white present. TRAINED CRITIC: While your justification is correct that Gertrude initially had a yellow present and swapped it with Alice’s white present, I would like to add two additional details that refute your response and support option D, brown present. First, prior to swapping with Alice, Gertrude swapped her yellow present with Dave’s brown present. Second, in the final round of swaps, Fred and Eve swap their presents, and Gertrude’s white present (which she received from Alice) is not mentioned in the swap between Fred and Eve. Therefore, based on the information provided, Gertrude could still have the brown present that she received from Dave earlier in the game. ACTOR: [Wrong] Final Answer: None of the given options are correct. The construction site was closed during the hours Thomas was engaged in the mentioned activities, and there is no information indicating that Thomas visited the construction site before 6am. UNTRAINED CRITIC: Your response is correct. I would add that Thomas’ activities from 6am to 8am (gym, yoga studio, and movie theater) all occurred before the construction site opened. Therefore, option (D) can be eliminated as well. TRAINED CRITIC: While it’s true that none of the given options are correct, I’d like to provide some brief additional details to support the fact that the correct answer is D (6am to 8am), if that was an option. Here’s why: 1. Thomas was seen at the gym from 8am to 11am, and at the yoga studio from 11am to 3pm, meaning he was engaged in those activities between those hours. There’s no information indicating that he left those places earlier or returned later. 2. He was seen at the movie theater from 3pm to 6pm, and at the Leaning Tower of Pisa from 6pm to 7pm. Again, there’s no information indicating that he left the movie theater earlier or arrived at the Leaning Tower of Pisa later. 3. The construction site was closed after 7pm. However, the options don’t include any time before 7pm. If Thomas could have gone to the construction site before 7pm, the correct answer would be one of the options A, B, or C. 4. Since none of the options include the time before 7pm, and Thomas was engaged in activities during the entire day, the only time left for him to visit the construction site, assuming he did, would be before 7am. However, the options don’t include that time frame. Therefore, if we had the option D (6am to 8am), it would be the correct answer, as it’s the only time frame that fits the given information. In conclusion, while the given options are not correct, if option D (6am to 8am) was an option, it would be the correct answer based on the information provided.