## Heatmap Charts: Pairwise Agent Analysis ### Overview The image presents four heatmaps comparing the performance of different pairwise agents across various metrics. The agents are compared based on cumulative normalized reward, teamwork (sum of rewards), percentage of cooperation, and relative rewards in competition. Each heatmap visualizes the interactions between the same set of agents, allowing for a direct comparison of their behavior under different evaluation criteria. ### Components/Axes **General Components:** * **Agents (Rows & Columns):** The same set of agents are listed on both the x and y axes of each heatmap. The agents are: UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL, Coop, Dfct, and Tit4Tat. * **Color Scales:** Each heatmap has its own color scale, indicating the range of values represented by the colors. * **Titles:** Each heatmap has a title indicating the metric being visualized. **Specific Axes and Scales:** 1. **Pairwise Agents: cum. normalized reward (Mem=5):** * Y-axis: Agent types (UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL, Coop, Dfct, Tit4Tat). * X-axis: Agent types (UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL, Coop, Dfct, Tit4Tat). * Color Scale: Blue, ranging from approximately 0 to 60, representing the cumulative normalized reward. 2. **Teamwork in Pairwise Agents: sum of rewards (Mem=5):** * Y-axis: Agent types (UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL, Coop, Dfct, Tit4Tat). * X-axis: Agent types (UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL, Coop, Dfct, Tit4Tat). * Color Scale: Blue, ranging from approximately 40 to 120, representing the sum of rewards. 3. **Pairwise Agents: percentage of cooperation (Mem=5):** * Y-axis: Agent types (UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL, Coop, Dfct, Tit4Tat). * X-axis: Agent types (UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL, Coop, Dfct, Tit4Tat). * Color Scale: Ranges from red to blue, with red representing low cooperation (0) and blue representing high cooperation (100). 4. **Competition in Pairwise Agents: relative rewards (Mem=5):** * Y-axis: Agent types (UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL, Coop, Dfct, Tit4Tat). * X-axis: Agent types (UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL, Coop, Dfct, Tit4Tat). * Color Scale: Ranges from red to blue, with red representing negative relative rewards (-20) and blue representing positive relative rewards (20). ### Detailed Analysis **1. Pairwise Agents: cum. normalized reward (Mem=5):** * The diagonal elements (interaction of an agent with itself) show moderate to high rewards, indicated by medium to dark blue shades. * Coop, Dfct, and Tit4Tat show high rewards when interacting with each other (dark blue). * Other agents show varying degrees of reward when interacting with Coop, Dfct, and Tit4Tat, generally lower than the self-interaction rewards. **2. Teamwork in Pairwise Agents: sum of rewards (Mem=5):** * The diagonal elements show moderate rewards (light blue). * Coop, Dfct, and Tit4Tat show high rewards when interacting with each other (dark blue). * Interactions between other agents are generally low to moderate (light blue). **3. Pairwise Agents: percentage of cooperation (Mem=5):** * The diagonal elements show varying degrees of cooperation. * Coop, Dfct, and Tit4Tat show very high cooperation with each other (dark blue). * Other agents show low cooperation with Dfct (red). * UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL show high cooperation with each other (dark blue). **4. Competition in Pairwise Agents: relative rewards (Mem=5):** * The diagonal elements show near-zero relative rewards (white/light shades). * Coop, Dfct, and Tit4Tat show negative relative rewards when interacting with other agents (red). * Other agents show positive relative rewards when interacting with Coop, Dfct, and Tit4Tat (blue). ### Key Observations * **Coop, Dfct, and Tit4Tat:** These agents consistently show high cooperation and teamwork among themselves but are exploited by other agents in competitive scenarios. * **UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL:** These agents tend to cooperate with each other and exploit Coop, Dfct, and Tit4Tat in competitive settings. * **Diagonal Elements:** The diagonal elements in the cooperation and competition heatmaps are generally neutral, indicating that agents neither strongly cooperate nor compete with themselves. ### Interpretation The heatmaps provide a comprehensive view of how different agents interact in pairwise scenarios. The data suggests that: * **Cooperative Strategies:** Coop, Dfct, and Tit4Tat are effective in environments that reward cooperation but are vulnerable to exploitation in competitive environments. * **Exploitative Strategies:** UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, QL, DQL, SARSA, SQL are more successful in competitive environments, likely due to their ability to exploit cooperative strategies. * **Trade-off:** There is a clear trade-off between cooperation and competitiveness. Agents that prioritize cooperation may achieve high rewards in collaborative settings but are easily exploited in competitive ones. The "Mem=5" likely refers to a memory parameter, indicating that the agents consider the past 5 interactions when making decisions. This parameter influences the agents' strategies and their ability to adapt to different environments.

## Heatmaps: Pairwise Agent Performance Metrics ### Overview This image displays four heatmaps, each representing a different performance metric for pairwise interactions between various agents. The agents are listed on both the x-axis and y-axis, indicating that each cell represents the interaction between the agent on the y-axis (row) and the agent on the x-axis (column). The color intensity of each cell corresponds to the value of the metric, with color bars providing a scale for interpretation. The metrics are: "Pairwise Agents: cum. normalized reward (Mem=5)", "Teamwork in Pairwise Agents: sum of rewards (Mem=5)", "Pairwise Agents: percentage of cooperation (Mem=5)", and "Competition in Pairwise Agents: relative rewards (Mem=5)". ### Components/Axes **General Structure:** Each heatmap has: - **Y-axis Labels:** A list of agent names vertically aligned on the left. - **X-axis Labels:** A list of agent names horizontally aligned at the bottom. - **Color Bar:** A vertical bar to the right of each heatmap indicating the scale and corresponding color for the metric's values. **Agent List (Common to all heatmaps):** - UCB - TS - eGreedy - EXP3 - HBTS - LinUCB - CTS - EXP4 - SCTS - QL - DQL - SARSA - SQL - Coop - DIct - TxtTat **Heatmap 1: Pairwise Agents: cum. normalized reward (Mem=5)** - **Title:** Pairwise Agents: cum. normalized reward (Mem=5) - **Y-axis Title:** (Implicitly, Agent on Y-axis) - **X-axis Title:** (Implicitly, Agent on X-axis) - **Color Bar:** Ranges from light blue (low values) to dark blue (high values). - **Scale Markers:** 0, 15, 30, 45, 60. **Heatmap 2: Teamwork in Pairwise Agents: sum of rewards (Mem=5)** - **Title:** Teamwork in Pairwise Agents: sum of rewards (Mem=5) - **Y-axis Title:** (Implicitly, Agent on Y-axis) - **X-axis Title:** (Implicitly, Agent on X-axis) - **Color Bar:** Ranges from light blue (low values) to dark blue (high values). - **Scale Markers:** 0, 20, 40, 60, 80, 100, 120. **Heatmap 3: Pairwise Agents: percentage of cooperation (Mem=5)** - **Title:** Pairwise Agents: percentage of cooperation (Mem=5) - **Y-axis Title:** (Implicitly, Agent on Y-axis) - **X-axis Title:** (Implicitly, Agent on X-axis) - **Color Bar:** Ranges from dark red (low values) to dark blue (high values), with white/light colors in the middle. - **Scale Markers:** 0, 20, 40, 60, 80, 100. **Heatmap 4: Competition in Pairwise Agents: relative rewards (Mem=5)** - **Title:** Competition in Pairwise Agents: relative rewards (Mem=5) - **Y-axis Title:** (Implicitly, Agent on Y-axis) - **X-axis Title:** (Implicitly, Agent on X-axis) - **Color Bar:** Ranges from dark red (low values) to dark blue (high values), with white/light colors in the middle. - **Scale Markers:** -20, -10, 0, 10, 20. ### Detailed Analysis **Heatmap 1: Pairwise Agents: cum. normalized reward (Mem=5)** - **Trend:** Generally, there are many interactions with moderate to high normalized rewards (indicated by shades of blue). The diagonal (self-interaction) is not explicitly shown or is assumed to be a baseline. - **Key Data Points (Approximate, based on color intensity):** - **High Rewards (Dark Blue, ~45-60):** - Coop vs. Coop: Appears to be the highest, close to 60. - DIct vs. Coop: High, around 45-50. - Coop vs. DIct: High, around 45-50. - SQL vs. SQL: High, around 45-50. - SARSA vs. SQL: High, around 45-50. - QL vs. SQL: High, around 45-50. - SCTS vs. SQL: High, around 45-50. - EXP4 vs. SQL: High, around 45-50. - CTS vs. SQL: High, around 45-50. - LinUCB vs. SQL: High, around 45-50. - HBTS vs. SQL: High, around 45-50. - EXP3 vs. SQL: High, around 45-50. - eGreedy vs. SQL: High, around 45-50. - TS vs. SQL: High, around 45-50. - UCB vs. SQL: High, around 45-50. - SQL vs. Coop: High, around 45-50. - **Moderate Rewards (Medium Blue, ~15-30):** Many interactions fall into this category, for example, UCB vs. TS, TS vs. eGreedy, etc. - **Low Rewards (Light Blue/White, ~0-15):** Some interactions, like TxtTat vs. most other agents, appear to have lower normalized rewards. **Heatmap 2: Teamwork in Pairwise Agents: sum of rewards (Mem=5)** - **Trend:** Similar to the first heatmap, there are many interactions with high sums of rewards, particularly where "Coop" and "SQL" are involved. - **Key Data Points (Approximate):** - **High Rewards (Dark Blue, ~100-120):** - Coop vs. Coop: Appears to be the highest, close to 120. - DIct vs. Coop: High, around 100-110. - Coop vs. DIct: High, around 100-110. - SQL vs. SQL: High, around 100-110. - SARSA vs. SQL: High, around 100-110. - QL vs. SQL: High, around 100-110. - SCTS vs. SQL: High, around 100-110. - EXP4 vs. SQL: High, around 100-110. - CTS vs. SQL: High, around 100-110. - LinUCB vs. SQL: High, around 100-110. - HBTS vs. SQL: High, around 100-110. - EXP3 vs. SQL: High, around 100-110. - eGreedy vs. SQL: High, around 100-110. - TS vs. SQL: High, around 100-110. - UCB vs. SQL: High, around 100-110. - SQL vs. Coop: High, around 100-110. - **Moderate Rewards (Medium Blue, ~40-80):** Many interactions fall into this range. - **Low Rewards (Light Blue/White, ~0-40):** TxtTat vs. many agents shows low sums of rewards. **Heatmap 3: Pairwise Agents: percentage of cooperation (Mem=5)** - **Trend:** This heatmap shows a clear divergence. Many interactions are either very high in cooperation (dark blue, ~80-100%) or very low (dark red, ~0-20%). There are fewer interactions in the mid-range. - **Key Data Points (Approximate):** - **High Cooperation (Dark Blue, ~80-100%):** - Coop vs. Coop: Appears to be 100%. - DIct vs. Coop: High, around 90-100%. - Coop vs. DIct: High, around 90-100%. - SQL vs. SQL: High, around 90-100%. - SARSA vs. SQL: High, around 90-100%. - QL vs. SQL: High, around 90-100%. - SCTS vs. SQL: High, around 90-100%. - EXP4 vs. SQL: High, around 90-100%. - CTS vs. SQL: High, around 90-100%. - LinUCB vs. SQL: High, around 90-100%. - HBTS vs. SQL: High, around 90-100%. - EXP3 vs. SQL: High, around 90-100%. - eGreedy vs. SQL: High, around 90-100%. - TS vs. SQL: High, around 90-100%. - UCB vs. SQL: High, around 90-100%. - SQL vs. Coop: High, around 90-100%. - **Low Cooperation (Dark Red, ~0-20%):** - UCB vs. UCB: Appears to be very low, close to 0%. - TS vs. TS: Appears to be very low, close to 0%. - eGreedy vs. eGreedy: Appears to be very low, close to 0%. - EXP3 vs. EXP3: Appears to be very low, close to 0%. - HBTS vs. HBTS: Appears to be very low, close to 0%. - LinUCB vs. LinUCB: Appears to be very low, close to 0%. - CTS vs. CTS: Appears to be very low, close to 0%. - EXP4 vs. EXP4: Appears to be very low, close to 0%. - SCTS vs. SCTS: Appears to be very low, close to 0%. - QL vs. QL: Appears to be very low, close to 0%. - DQL vs. DQL: Appears to be very low, close to 0%. - SARSA vs. SARSA: Appears to be very low, close to 0%. - SQL vs. SQL: High cooperation (as noted above). - Coop vs. Coop: High cooperation (as noted above). - DIct vs. DIct: Appears to be very low, close to 0%. - TxtTat vs. TxtTat: Appears to be very low, close to 0%. - **Mixed Cooperation:** Some interactions show moderate cooperation (e.g., UCB vs. TS, TS vs. eGreedy). **Heatmap 4: Competition in Pairwise Agents: relative rewards (Mem=5)** - **Trend:** This heatmap shows relative advantages. Dark blue indicates the row agent has a relative reward advantage over the column agent, while dark red indicates the column agent has an advantage. - **Key Data Points (Approximate):** - **Row Agent Advantage (Dark Blue, ~10-20):** - Coop vs. DIct: Appears to have a significant advantage for Coop, around 15-20. - DIct vs. Coop: Appears to have a significant advantage for DIct, around 15-20. - SQL vs. SARSA: Appears to have an advantage for SQL, around 10-15. - SQL vs. QL: Appears to have an advantage for SQL, around 10-15. - SQL vs. SCTS: Appears to have an advantage for SQL, around 10-15. - SQL vs. EXP4: Appears to have an advantage for SQL, around 10-15. - SQL vs. CTS: Appears to have an advantage for SQL, around 10-15. - SQL vs. LinUCB: Appears to have an advantage for SQL, around 10-15. - SQL vs. HBTS: Appears to have an advantage for SQL, around 10-15. - SQL vs. EXP3: Appears to have an advantage for SQL, around 10-15. - SQL vs. eGreedy: Appears to have an advantage for SQL, around 10-15. - SQL vs. TS: Appears to have an advantage for SQL, around 10-15. - SQL vs. UCB: Appears to have an advantage for SQL, around 10-15. - **Column Agent Advantage (Dark Red, ~-10 to -20):** - DIct vs. Coop: Appears to have a significant advantage for Coop, around -15 to -20. - Coop vs. DIct: Appears to have a significant advantage for DIct, around -15 to -20. - SARSA vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - QL vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - SCTS vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - EXP4 vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - CTS vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - LinUCB vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - HBTS vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - EXP3 vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - eGreedy vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - TS vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - UCB vs. SQL: Appears to have an advantage for SQL, around -10 to -15. - **Neutral/Slight Advantage (White/Light Colors, ~-5 to 5):** Many interactions fall into this category, indicating relatively balanced performance. ### Key Observations * **"Coop" and "SQL" Agents:** These two agents consistently show very high performance across normalized reward, sum of rewards, and percentage of cooperation when interacting with themselves or each other, and when interacting with many other agents. They also appear to be strong performers in terms of relative rewards when paired against many other agents. * **Self-Interaction:** For most agents (except "Coop" and "SQL"), self-interaction (diagonal elements) shows very low cooperation and potentially low normalized rewards, suggesting these agents do not perform optimally when interacting with themselves in a cooperative manner. * **"TxtTat" Agent:** This agent consistently shows very low performance across normalized reward and sum of rewards when interacting with most other agents, and also exhibits low cooperation. * **Divergent Cooperation:** The "percentage of cooperation" heatmap reveals a stark dichotomy: agents either cooperate very highly or very poorly, with few in between. This suggests distinct behavioral modes. * **Relative Strength:** The "relative rewards" heatmap highlights specific pairwise advantages. For instance, "Coop" and "DIct" seem to have a strong reciprocal advantage over each other, while "SQL" generally has an advantage over many other agents. ### Interpretation The heatmaps collectively illustrate the performance landscape of different agents in pairwise interactions, likely within a reinforcement learning or game theory context, given the agent names (e.g., UCB, QL, SARSA, SQL). The parameter "(Mem=5)" suggests a memory component of size 5 might be influencing these results. **What the data suggests or demonstrates:** 1. **Emergence of Strong Performers:** The "Coop" and "SQL" agents appear to be highly effective, exhibiting robust performance in terms of reward accumulation and cooperative behavior. Their consistent high scores suggest they might be well-suited for environments that reward collaboration or efficient strategy. 2. **Cooperation vs. Competition Dynamics:** The "percentage of cooperation" heatmap is particularly insightful. The high cooperation between "Coop" and "SQL" (and their self-interactions) suggests these agents are designed or have learned to be highly cooperative. Conversely, the low cooperation for many other agents' self-interactions indicates a potential lack of intrinsic cooperative mechanisms or a tendency towards competitive self-play. 3. **Agent Specialization/Behavioral Modes:** The stark contrast in cooperation levels suggests that agents might operate in distinct behavioral modes. Some agents are inherently cooperative, while others are not. This could be a result of their underlying algorithms or training objectives. 4. **Relative Power Structures:** The "relative rewards" heatmap maps out a hierarchy or set of advantages. "SQL" appears to be a dominant agent against many others, while "Coop" and "DIct" have a strong, mutually beneficial relationship. The negative values indicate that in those pairings, the *column* agent is outperforming the *row* agent. 5. **Ineffective Agents:** The "TxtTat" agent seems to be consistently underperforming, suggesting it may be an experimental agent, a baseline that is not competitive, or an agent that is poorly suited to the tested environment. **How the elements relate to each other:** The four heatmaps provide a multi-faceted view of agent performance. - Heatmaps 1 and 2 ("normalized reward" and "sum of rewards") show the overall success in terms of accumulated value. - Heatmap 3 ("percentage of cooperation") explains *how* that success is achieved – through collaboration or lack thereof. - Heatmap 4 ("relative rewards") clarifies the competitive aspect, showing who gains an advantage when cooperation is not fully realized or when agents are pitted against each other. The strong correlation between high rewards (Heatmaps 1 & 2) and high cooperation (Heatmap 3) for agents like "Coop" and "SQL" suggests that in this context, cooperation is a key driver of success. The "relative rewards" heatmap then shows how these cooperative agents fare against less cooperative ones, or how non-cooperative agents compete amongst themselves. **Any notable outliers, trends, or anomalies:** - The extreme self-cooperation of "Coop" and "SQL" is a significant trend. - The extreme self-defection/low cooperation of most other agents is also a notable trend. - The consistent poor performance of "TxtTat" is an anomaly if it's expected to be a competitive agent. - The reciprocal high advantage between "Coop" and "DIct" in relative rewards is interesting, as is the broad dominance of "SQL" over many other agents.

\n ## Heatmaps: Pairwise Agent Performance Metrics ### Overview The image presents four heatmaps, each visualizing a different metric related to the performance of various agents in a pairwise setting. The agents are listed on the y-axis, and the other agent they are being compared against is listed on the x-axis. Each cell in the heatmap represents the value of the metric for that specific agent pair, with color intensity indicating the magnitude of the value. ### Components/Axes Each heatmap shares the following components: * **Y-axis (Vertical):** Lists the agents being evaluated. The agents are: UCB, eComedy, EXP3, HBT3, LINUCB, CTB, EXP1, SCTS, QL, DQL, SARSA, Coop, SQL, Old, Tital. * **X-axis (Horizontal):** Lists the agents being compared against. The agents are: UCB, eComedy, EXP3, HBT3, LINUCB, CTB, EXP1, SCTS, QL, DQL, SARSA, Coop, SQL, Old, Tital. * **Color Scale (Legend):** Each heatmap has a unique color scale indicating the range of values represented by the colors. * **Heatmap 1 (Normalized Reward):** Color scale ranges from approximately 0 to 20. Blue indicates lower values, and red indicates higher values. * **Heatmap 2 (Teamwork in Rewards):** Color scale ranges from approximately 0 to 20. Blue indicates lower values, and red indicates higher values. * **Heatmap 3 (Percentage of Cooperation):** Color scale ranges from approximately 0 to 100. Blue indicates lower values, and red indicates higher values. * **Heatmap 4 (Competition in Rewards):** Color scale ranges from approximately -15 to 15. Blue indicates lower values, and red indicates higher values. ### Detailed Analysis or Content Details **Heatmap 1: Pairwise Agents, Normalized Reward (Mean)** * The heatmap shows a generally sparse pattern of high rewards. * The diagonal elements (agent vs. itself) are all approximately 0. * The highest reward values (around 18-20) are observed for the pair (QL, QL), (DQL, DQL), (SARSA, SARSA), (Coop, Coop), (SQL, SQL). * The agent 'Tital' consistently shows low normalized rewards when paired with other agents. * The agent 'UCB' shows low normalized rewards when paired with other agents. **Heatmap 2: Teamwork in Rewards (Mean)** * The heatmap shows a generally sparse pattern of high teamwork rewards. * The diagonal elements (agent vs. itself) are all approximately 0. * The highest teamwork rewards (around 18-20) are observed for the pair (QL, QL), (DQL, DQL), (SARSA, SARSA), (Coop, Coop), (SQL, SQL). * The agent 'Tital' consistently shows low teamwork rewards when paired with other agents. * The agent 'UCB' shows low teamwork rewards when paired with other agents. **Heatmap 3: Pairwise Agents, Percentage of Cooperation (Mean)** * The heatmap shows a generally high percentage of cooperation across most agent pairs. * The diagonal elements (agent vs. itself) are all approximately 100. * The lowest cooperation percentages (around 0-20) are observed for the pair (Old, Old), (Tital, Tital). * The agent 'Old' consistently shows low cooperation percentages when paired with other agents. * The agent 'Tital' consistently shows low cooperation percentages when paired with other agents. **Heatmap 4: Competition in Rewards (Mean)** * The heatmap shows a mix of positive and negative competition values. * The diagonal elements (agent vs. itself) are all approximately 0. * The highest positive competition values (around 10-15) are observed for the pair (EXP3, EXP3), (HBT3, HBT3), (LINUCB, LINUCB), (CTB, CTB), (EXP1, EXP1), (SCTS, SCTS). * The lowest negative competition values (around -10 to -15) are observed for the pair (QL, QL), (DQL, DQL), (SARSA, SARSA), (Coop, Coop), (SQL, SQL). * The agent 'Tital' consistently shows negative competition values when paired with other agents. ### Key Observations * Agents QL, DQL, SARSA, Coop, and SQL consistently exhibit high normalized rewards and teamwork rewards, but also high negative competition. * Agents UCB and eComedy consistently exhibit low normalized rewards and teamwork rewards. * Agents Old and Tital consistently exhibit low cooperation percentages and negative competition. * The diagonal elements in all heatmaps indicate self-comparison, which generally results in expected values (0 for reward, 100 for cooperation). ### Interpretation The data suggests that different agents exhibit varying levels of performance, cooperation, and competition when interacting with each other. Agents like QL, DQL, SARSA, Coop, and SQL appear to be effective at maximizing rewards and teamwork, but this comes at the cost of increased competition. Conversely, agents like UCB and eComedy struggle to achieve high rewards or teamwork. Agents Old and Tital are consistently poor performers in terms of cooperation and exhibit negative competition, suggesting they may be detrimental to overall system performance. The relationship between the heatmaps reveals interesting trade-offs. For example, high normalized reward and teamwork often correlate with negative competition, indicating that these agents may be dominating resources or hindering the performance of others. The heatmap of competition provides insight into the dynamics of agent interactions, highlighting which agents are more likely to compete for resources or rewards. The consistent low performance of agents Old and Tital suggests they may be poorly suited for the task or require further optimization. The high performance of agents QL, DQL, SARSA, Coop, and SQL suggests they are promising candidates for further development and deployment.

## Heatmap Series: Pairwise Agent Performance Metrics ### Overview The image displays a horizontal series of four distinct heatmaps, each visualizing a different performance metric for pairwise interactions between various reinforcement learning or multi-agent algorithms. The heatmaps share a common structure: a square matrix with identical agent names on both the x-axis (bottom) and y-axis (left), with cell colors representing the value of the specified metric for each agent pair. A vertical color bar legend is positioned to the right of each heatmap. ### Components/Axes **Common Elements (All Four Heatmaps):** * **Y-Axis (Left):** Lists agent names. From top to bottom: `UCB`, `eGreedy`, `HBTs`, `LinUCB`, `LinTS`, `EXP3`, `QL`, `RHC`, `SARSA`, `DQN`, `Comp.`, `TS/TS`. * **X-Axis (Bottom):** Lists the same agent names in the same order, from left to right. * **Legend Placement:** A vertical color bar is located to the immediate right of each heatmap matrix. * **Matrix Structure:** Each heatmap is an 11x11 grid (excluding the axis labels), representing all possible pairwise combinations of the 11 listed agents. **Individual Heatmap Titles & Color Scales:** 1. **Leftmost Heatmap:** * **Title:** `Pairwise Agents: cum. normalized reward (M=10^4)` * **Color Bar Label:** `0` (bottom, dark blue) to `10` (top, light yellow/white). * **Color Scheme:** Sequential blue-to-yellow gradient. 2. **Second Heatmap:** * **Title:** `Teammate in Pairwise Agents: sum of rewards (M=10^4)` * **Color Bar Label:** `0` (bottom, dark blue) to `120` (top, light yellow/white). * **Color Scheme:** Sequential blue-to-yellow gradient. 3. **Third Heatmap:** * **Title:** `Pairwise Agents: percentage of cooperation (M=10^4)` * **Color Bar Label:** `40` (bottom, dark red) to `100` (top, light yellow/white). * **Color Scheme:** Sequential red-to-yellow gradient. 4. **Rightmost Heatmap:** * **Title:** `Competition in Pairwise Agents: relative rewards (M=10^4)` * **Color Bar Label:** `-10` (bottom, dark red) to `10` (top, dark blue), with `0` at the midpoint (white/light gray). * **Color Scheme:** Diverging red-white-blue gradient. ### Detailed Analysis **1. Cumulative Normalized Reward Heatmap:** * **Trend:** The matrix is predominantly blue, indicating most pairwise interactions yield low normalized rewards (0-3 range). Lighter cells (higher rewards) are sparse. * **Data Points (Approximate):** * The diagonal (e.g., UCB vs. UCB) is consistently dark blue (~0-1). * The highest rewards (lightest cells, ~7-9) appear in isolated pairs, such as `TS/TS` vs. `DQN` (bottom-right quadrant) and `Comp.` vs. `SARSA`. * The `TS/TS` row and column show slightly more lighter blue cells compared to others, suggesting it achieves moderately higher rewards against several opponents. **2. Sum of Rewards (Teammate) Heatmap:** * **Trend:** Similar overall blue dominance, but with more pronounced light blue and yellow cells compared to the first heatmap, indicating higher absolute reward sums. * **Data Points (Approximate):** * The `TS/TS` row is notably lighter, with values likely in the 60-90 range against many opponents. * The `Comp.` agent also shows relatively high reward sums (light blue, ~40-70) against several agents like `SARSA`, `DQN`, and `TS/TS`. * The diagonal is not uniformly the lowest; some self-pairings (e.g., `QL` vs. `QL`) show moderate reward sums. **3. Percentage of Cooperation Heatmap:** * **Trend:** This map shows a clear pattern. The upper triangle (above the diagonal) is predominantly dark red (low cooperation, ~40-50%), while the lower triangle is predominantly light yellow (high cooperation, ~90-100%). * **Data Points (Approximate):** * **High Cooperation (Lower Triangle):** Pairs where the row agent is "below" the column agent in the list (e.g., `SARSA` vs. `UCB`, `DQN` vs. `QL`) show cooperation near 100%. * **Low Cooperation (Upper Triangle):** The inverse pairs (e.g., `UCB` vs. `SARSA`, `QL` vs. `DQN`) show cooperation near 40-50%. * The diagonal cells are mixed, with some showing high and others showing moderate cooperation. **4. Relative Rewards (Competition) Heatmap:** * **Trend:** A diverging pattern. The upper triangle is mostly red (negative relative reward, meaning the row agent loses to the column agent), and the lower triangle is mostly blue (positive relative reward, meaning the row agent wins). * **Data Points (Approximate):** * **Strongest Negative (Dark Red, ~-8 to -10):** `UCB` vs. `TS/TS`, `eGreedy` vs. `TS/TS`. * **Strongest Positive (Dark Blue, ~+8 to +10):** `TS/TS` vs. `UCB`, `TS/TS` vs. `eGreedy`. * The `TS/TS` agent shows a strong blue column (wins against most) and a strong red row (loses to few, if any). * The `Comp.` agent shows mixed results, with blues and reds scattered. ### Key Observations 1. **Asymmetry in Cooperation:** The most striking pattern is the near-perfect asymmetry in the cooperation heatmap. This suggests a hierarchical or sequential relationship between the agents, where cooperation is largely one-directional based on their ordering in the list. 2. **Dominance of TS/TS:** The `TS/TS` agent appears to be the strongest performer overall. It achieves high cumulative rewards, high reward sums, and dominates in relative reward (competition) against almost all other agents. 3. **Weakness of Early Algorithms:** Traditional algorithms like `UCB` and `eGreedy` appear at the top of the list and consistently show poor performance (dark colors) in reward metrics and lose in competition (red in the relative reward map) against more complex agents. 4. **Metric Correlation:** High cooperation percentage (lower triangle of heatmap 3) does not directly correlate with high reward for the cooperating agent. For example, `SARSA` cooperating with `UCB` (high cooperation) still results in low reward for `SARSA` (first heatmap). ### Interpretation This set of heatmaps provides a comprehensive competitive and cooperative landscape analysis for 11 agents over 10,000 episodes (`M=10^4`). The data suggests a clear performance hierarchy. The **asymmetric cooperation pattern** is particularly insightful. It implies the agent list is ordered by some measure of sophistication or "strength," where agents lower on the list (like `TS/TS`, `Comp.`) are willing to cooperate with those above them, but the reverse is not true. This could model scenarios where advanced agents exploit simpler ones by feigning cooperation. The **dominance of `TS/TS`** (likely Thompson Sampling) aligns with its known effectiveness in balancing exploration and exploitation. Its strong performance across reward metrics and competition indicates it is a robust, top-tier algorithm in this specific multi-agent environment. The **diverging competition heatmap** acts as a direct "win-loss" record. The clear red/blue split along the diagonal reinforces the hierarchical structure. The fact that `TS/TS` has a nearly all-blue column (except against itself) is a definitive visual indicator of its superiority in this test. In summary, these visualizations move beyond simple average rewards to reveal the nuanced, pairwise dynamics of competition and cooperation. They show that agent performance is highly context-dependent (i.e., dependent on the specific opponent) and that simple metrics like "percentage of cooperation" can mask underlying competitive inequalities. The ordering of agents on the axes appears to be a critical factor, reflecting a latent variable like algorithmic complexity or learning capability.

## Heatmaps: Pairwise Agents Performance Metrics ### Overview The image contains four heatmaps comparing the performance of 16 pairwise agents across four metrics: normalized reward, sum of rewards, percentage of cooperation, and relative rewards. Each heatmap uses a color gradient to represent values, with legends on the right indicating scale ranges. Agents are listed on both axes, and the matrices show pairwise interactions. ### Components/Axes - **X/Y Axes**: Labeled with agent names (e.g., UCB, TS, eGreedy, EXP3, HBTS, LinUCB, CTS, EXP4, SCTS, GL, DQL, SARSA, SQL, Coop, DQN, TIMTAT). - **Legends**: 1. **Normalized Reward**: Blue gradient (0–60). 2. **Sum of Rewards**: Darker blue gradient (0–120). 3. **Percentage of Cooperation**: Red gradient (0–100). 4. **Relative Rewards**: Mixed red/blue gradient (-20–20). - **Placement**: Legends are positioned to the right of each heatmap. Agent names are centered on both axes. ### Detailed Analysis 1. **Normalized Reward (Mem=5)**: - **High Values**: UCB (60), TS (60), eGreedy (45), EXP3 (45), HBTS (45), LinUCB (45), CTS (45), EXP4 (45), SCTS (45), GL (45), DQL (45), SARSA (45), SQL (45), Coop (45), DQN (45), TIMTAT (45). - **Low Values**: Most agents cluster around 15–30, with Coop and DQN showing the lowest (15). 2. **Sum of Rewards (Mem=5)**: - **High Values**: UCB (120), TS (120), eGreedy (100), EXP3 (100), HBTS (100), LinUCB (100), CTS (100), EXP4 (100), SCTS (100), GL (100), DQL (100), SARSA (100), SQL (100), Coop (100), DQN (100), TIMTAT (100). - **Low Values**: Most agents cluster around 40–80, with Coop and DQN showing the lowest (40). 3. **Percentage of Cooperation (Mem=5)**: - **High Values**: UCB (100), TS (100), eGreedy (80), EXP3 (80), HBTS (80), LinUCB (80), CTS (80), EXP4 (80), SCTS (80), GL (80), DQL (80), SARSA (80), SQL (80), Coop (80), DQN (80), TIMTAT (80). - **Low Values**: Most agents cluster around 20–60, with Coop and DQN showing the lowest (20). 4. **Relative Rewards (Mem=5)**: - **Positive Values**: UCB (+20), TS (+20), eGreedy (+10), EXP3 (+10), HBTS (+10), LinUCB (+10), CTS (+10), EXP4 (+10), SCTS (+10), GL (+10), DQL (+10), SARSA (+10), SQL (+10), Coop (+10), DQN (+10), TIMTAT (+10). - **Negative Values**: Most agents cluster around -10 to -20, with Coop and DQN showing the lowest (-20). ### Key Observations - **Dominant Agents**: UCB and TS consistently achieve the highest values across all metrics, suggesting optimal performance in cooperative scenarios. - **Competitive Agents**: Coop and DQN exhibit the lowest cooperation percentages and negative relative rewards, indicating poor teamwork. - **Moderate Performers**: Agents like eGreedy, EXP3, and HBTS show mid-range values, balancing reward and cooperation. - **Anomalies**: The relative rewards heatmap reveals stark contrasts, with UCB/TS outperforming others by +20, while Coop/DQN underperform by -20. ### Interpretation The data highlights UCB and TS as the most effective agents in cooperative environments, likely due to their strategic design (e.g., UCB’s exploration-exploitation balance, TS’s Thompson Sampling). Coop and DQN’s poor performance in cooperation and relative rewards suggests they prioritize individual gains over teamwork. The sum of rewards heatmap reinforces this, showing UCB/TS as the only agents achieving the maximum value (120). The relative rewards metric underscores systemic competition, where top agents gain at the expense of others. This implies that agent selection should prioritize UCB/TS for collaborative tasks, while Coop/DQN may require redesign for team-based applications.