2507.15559v1

# \sysName: Guiding the Creation of Multi-agent Workflows with Design Space Visualization as a Thinking Scaffold **Authors**: \authororcidPan Hao0009-0006-4473-, \authororcidDongyeop Kang0000-0002-9021-, Nicholas Hinds, and\authororcidQianwen Wang0000-0002-1825- \onlineid 1729 \vgtccategory Research \vgtcpapertype Applications \authorfooter Pan Hao, Dongyeop Kang, Nicholas Hinds, and Qianwen Wang are with the University of Minnesota, Twin Cities, MN, USA. E-mail: {pan00342, dongyeop, hinds084, qianwen}@umn.edu Abstract Multi-agent workflows have become an effective strategy for tackling complicated tasks by decomposing them into multiple sub-tasks and assigning them to specialized agents. However, designing optimal workflows remains challenging due to the vast and intricate design space. Current practices rely heavily on the intuition and expertise of practitioners, often resulting in design fixation or an unstructured, time-consuming exploration of trial-and-error. To address these challenges, this work introduces \sysName, an interactive visualization tool to facilitate the creation of multi-agent workflow through i) a structured visual exploration of the design space and ii) in-situ guidance informed by established design patterns. Based on formative studies and literature review, \sysName organizes the workflow design process into three hierarchical levels (\ie, task planning, agent assignment, and agent optimization), ranging from abstract to concrete. This structured visual exploration enables users to seamlessly move from high-level planning to detailed design decisions and implementations, while comparing alternative solutions across multiple performance metrics. Additionally, drawing from established workflow design patterns, \sysName provides context-aware, in-situ suggestions at each level as users navigate the design space, enhancing the workflow creation process with practical guidance. Use cases and user studies demonstrate the usability and effectiveness of \sysName, while also yielding valuable insights into how practitioners explore design spaces and leverage guidance during workflow development. keywords: LLM workflows, Multi-agent Workflows, design space, hierarchical visualization \teaser <details> <summary>x1.png Details</summary>  ### Visual Description ## Workflow Diagram: Data-Driven Visual Storytelling Design Space ### Overview The image presents a workflow diagram outlining the design space for generating data-driven visual storytelling. It includes several interconnected components, including design space overview, topic selection, agent interaction, and workflow patterns. ### Components/Axes **A1: Design Space Overview** * Title: Workflows for generating a data-driven visual storytelling. * Subtitle: Design Space Overview. * Flows: Four flows are depicted (Flow 1, Flow 2, Flow 3, Flow 4). Each flow branches into sub-flows and tasks. * Flow 1: Branches into 2-1 and 2-2. 2-1 branches into 2-1a and 2-1b. 2-2 branches into 2-2a. * Flow 2: Branches into 2-1 and 2-2. 2-1 branches into 2-1a and 2-1b. 2-2 branches into 2-2a. * Flow 3: Branches into 3-1 and 3-2. 3-2 branches into 3-2a, 3-2b, and 3-2c. * Flow 4: Branches into 4-1 and 4-2. 4-2 branches into 4-2a. **A2: X-Axis/Y-Axis Selection** * X-Axis: Dropdown menu with options "#subtask", "#agents", and "latency". Currently selected is "#subtask". * Y-Axis: Dropdown menu with options "#agents". Currently selected is "#agents". * Visualizations: A series of icons representing different data visualizations based on the selected X and Y axes. **B1: Topic Selector** * Title: Topic Selector * Subtitle: Step Description * Description: "proposes candidate story topics based on the given dataset" * Connected to: "Visualize Distribution" and "Visualize Correlation" **B2: Lead with Curiosity** * Title: Lead with Curiosity * Subtitle: Redundant... * Task Description: "propose candidate story topics based on the given dataset" * LLM Calls: "2 * 1 + 1 = 3 LLM calls" * Agents: * Agent #1: Persona - scientist; Goal - more logical and scientific. * Agent #2: Persona - journalist; Goal - more emotion-evoking and engaging. * Aggregation: Persona - Selector; Goal - Select a topic from two agents. **B3: Unspecified** * Appears to be a zoomed-out view of the workflow, possibly for navigation. **C1: Workflow Patterns - Sequential and Parallel** * Sequential: A series of connected rectangles. * Parallel: One rectangle branching into three rectangles. **C2: Workflow Patterns - Reflection, Discussion, Redundant, Supervision** * Reflection: A rectangle connected to a loop involving two agent icons. * Discussion: A rectangle connected to a loop involving two agent icons, with dotted lines indicating interaction. * Redundant: A rectangle connected to a series of three agent icons in a row, then to another rectangle. * Supervision: A rectangle connected to a branching structure involving agent icons, then to another rectangle. **C3: Tool Use** * Tool Use: Icon of a document with a wrench. * Prompting: Icon of a speech bubble with a pencil. * Data Retrieval: Icon of a database. ### Detailed Analysis or ### Content Details * **Flows (A1):** The diagram starts with four high-level flows, each branching into more specific tasks. The number of sub-tasks varies across flows. * **Axis Selection (A2):** The X and Y axes can be configured to represent different variables, such as "#subtask", "#agents", and "latency". The selected axes influence the type of data visualization displayed. * **Topic Selection (B1):** The "Topic Selector" component proposes candidate story topics based on the dataset. * **Agent Interaction (B2):** The "Lead with Curiosity" component involves interaction between different agents (scientist, journalist, selector) to refine the story topics. * **Workflow Patterns (C1, C2):** The diagram illustrates different workflow patterns, including sequential, parallel, reflection, discussion, redundant, and supervision. * **Tool Use (C3):** The diagram highlights the use of tools for prompting and data retrieval. ### Key Observations * The diagram provides a structured overview of the design space for data-driven visual storytelling. * The workflow involves multiple stages, including topic selection, agent interaction, data visualization, and insight integration. * The diagram emphasizes the importance of collaboration between different agents (e.g., scientists and journalists) to create compelling stories. * The use of different workflow patterns allows for flexibility and adaptability in the storytelling process. ### Interpretation The diagram illustrates a complex workflow for generating data-driven visual storytelling. It highlights the iterative nature of the process, involving multiple agents, tools, and workflow patterns. The ability to select different X and Y axes for visualization allows for exploring different perspectives on the data. The diagram suggests that effective data-driven storytelling requires a combination of data analysis, creative writing, and collaboration. The "Lead with Curiosity" component emphasizes the importance of asking the right questions and exploring different angles to uncover meaningful insights. </details> \sysName facilitates the creation of multi-agent workflows through structured, guided visual exploration of the design space. This is achieved by coordinating a hierarchical tree view (A1) and a scatter plot (A2), both of which use a novel glyph design to represent each workflow’s computational cost and level of abstraction. Users can select a workflow for detailed inspection in the Canvas View (B1-B3), which supports semantic zooming that reveals more abstract representations greater detail within more abstract representations depending on the zoom level. Additionally, in-situ suggestions based on well-established design patterns are provided to guide users as they navigate the design space (C1-C3). Introduction Recently, multi-agent workflows have emerged as a promising solution for complex tasks by decomposing them into subtasks, each handled by specialized LLM agents that collaboratively reason and execute tasks [5, 4]. These multi-agent workflows have demonstrated remarkable capabilities across various applications, such as software development, societal simulation, and even scientific discovery [19, 38]. Notably, Google’s AI co-scientist [17], a multi-agent LLM workflow system for scientific discovery, successfully uncovered a novel gene transfer mechanism, reproducing unpublished experimental findings [39]. However, designing effective multi-agent workflows remains a considerable challenge due to the vast design space. AI researchers and practitioners must balance critical trade-offs between performance, latency, and computational cost [18, 57]. Despite recent advancements in automatic task decomposition and agent collaboration (e.g., Manus AI [13]), the over-reliance on LLMs for workflow generation not only compromises transparency and controllability but also often results in suboptimal solutions [9]. Rather than relying on fully automated methods, AI researchers and practitioners typically draw on their own intuition and domain expertise, leveraging well-established human problem-solving strategies [57, 18]. Current studies predominantly focus on implementation support, helping users (\ie, AI researchers and practitioners) translate existing workflow ideas into executable systems, but offer limited assistance for the exploratory design processes [52, 24]. However, people’s problem-solving approaches are heavily influenced by the available tools, which makes certain actions more accessible while rendering others difficult or impractical [20]. In the context of multi-agent workflows, the affordances of existing tools have similarly shaped common practices in ways that discourage broad exploration. Users often begin with a rough idea of the workflow and move directly to implementation, a process that tends to bypass divergent thinking and leads to several limitations: First, early commitment to a specific workflow structure can restrict exploration of alternative solutions that might perform better within the broader design space. This contradicts the widely accepted divergence-before-convergence principle [8], leading to fixation on suboptimal designs. Second, by starting with complete implementations, users skip over abstraction layers in the design space. When performance issues arise, they tend to make superficial adjustments (e.g., tweaking agent prompts) rather than revisiting fundamental design decisions (e.g., task decomposition strategies) where greater improvements could be made. Recent studies attempted to address this gap by introducing design patterns specifically tailored to multi-agent workflows. Design patterns represent reusable, adaptable solutions proven effective across various contexts. For example, Liu et al.. [57] identified 18 agent design patterns (e.g., self-reflection, voting) through an extensive review of 57 multi-agent studies. Grunde-McLaughlin et al.. [18] summarized five architecture workflow building blocks (\ie, sequential, branching, redundant, dynamic, communicative) by borrowing lessons about task decomposition and validation from crowdsourcing. Even though these studies provide valuable guidance, these patterns remain disconnected from practical tools and environments, requiring AI researchers and practitioners to actively seek out, efficiently learn, and manually apply them during the design process. To address these challenges, we propose \sysName, an interactive visualization tool to facilitate AI researchers and practitioners in creating effective multi-agent workflows ( \sysName: Guiding the Creation of Multi-agent Workflows with Design Space Visualization as a Thinking Scaffold). \sysName not only provides an intuitive visual interface for building workflows, but, more importantly, actively scaffolds users’ design thinking by explicitly visualizing the design space and providing in-situ design guidance. Based on formative studies and literature review, \sysName organizes the workflow design process into three levels (\ie, task planning, agent assignment, and agent optimization), progressing from abstract concepts to concrete implementation. A structured visual exploration of the design space is enabled through the coordination of a hierarchical tree view and a scatter plot. Within this coordinated visualization, we introduce a novel glyph design that encodes both the abstraction level and computational cost of each solution. Users can select any solution for closer inspection in the Canvas View, where it is visualized as a node-link diagram with semantic zooming to reveal increasing detail as users zoom in. Additionally, drawing from established workflow design patterns, \sysName provides contextually relevant in-situ suggestions at each level as users navigate the design space, illustrating current practice to the workflow creation process. Evaluations showed that \sysName enabled users to create workflows more efficiently, explore a wider range of design alternatives, and produce higher-quality outputs compared to the baseline system. Main contributions of this paper include: - A structured characterization and visualization of the design space for multi-agent workflows, augmented with in-situ suggestions of relevant design patterns. - \sysName , an interactive visualization tool that guides users in designing and creating multi-agent workflows. - Case studies and user studies demonstrating the effectiveness and usability of the proposed approach, along with insights into how users leverage guidance to explore and refine workflow designs. 1 Related Work 1.1 Creating Multi-agent Workflows Existing research on multi-agent workflow creation can be categorized into three main areas: workflow authoring frameworks, empirical design guidelines, and automated generation methods. Workflow authoring frameworks provide programming libraries or visualization tools for simplifying multi-agent workflow creation. Programming frameworks, such as LangGraph [24] and AutoGen [52], offer APIs that allow users to efficiently define agent roles and manage inter-agent communication. To make workflow creation more accessible to non-programmers, no-code platforms, such as Rivet [3], Vellum [1], and AutoGen Studio [11], offer visual interfaces for designing and testing complex workflows. Users can define agents as compact cards and construct workflows using a drag-and-drop interface. Additionally, interactive tools, like AI Chains and Prompt Chainer [53], streamline the process of chaining prompts via interactive visualizations. However, these studies primarily assist in implementing workflows that users have already conceptualized and still heavily rely on user expertise for workflow design (e.g., task planning and orchestrating communication among agents). To mitigate the complexity in designing workflows, recent studies have derived design patterns and guidelines from systematic investigation of existing workflow implementations [57, 18, 19, 62]. For example, Liu et al. [57] identified 18 design patterns (e.g., self-reflection, voting, Role-based cooperation) across various scenarios. Grunde-McLaughlin et al. [18] summarized and tested five workflow building blocks (\ie, sequential, branching, redundant, dynamic, communicative) by drawing inspiration from crowdsourcing methodologies and study designs. Meanwhile, industry guidelines from major technology companies have increasingly emerged, outlining common practices used in the community, such as group chat, debate, and reflection [33, 32, 5]. Automated methods offer another research direction, focusing on reducing user workload by automatically generating optimized workflow designs. One prevalent approach involves introducing a “supervisor” agent that decomposes tasks and orchestrates agent collaboration. For instance, ADAS [22] employs a meta-agent to generate and refine reusable code snippets, assembling them into complete workflows. Another approach models the search for suitable workflows as an optimization problem that can be solved computationally [63, 59, 62]. For example, GPTSwarm [63] formulates multi-agent workflow construction as an optimization over computational graphs, with agents as nodes and communication channels as edges. While these methods simplify the task orchestration, they often sacrifice transparency, reliability, and controllability, which are crucial in critical or high-stakes scenarios. As a result, existing studies tend to fall into two extremes, either fully relying on a user’s own knowledge or entirely automating the workflow creation. Our proposed method aims to strike a balance by embedding design space exploration and empirical design guidance directly into the workflow-building environment. 1.2 Visualizing Design Space A design space encompasses the conceptual realm of possible solutions for a given problem and serves as a fundamental framework for structured exploration in various design processes, including machine learning architecture design [49, 56], creative writing [43], and image generation [7, 15]. Effective visualization of design spaces helps users navigate and interpret the numerous solutions by clearly representing their dimensions and abstraction levels. Dimensions represent key parameters that define and distinguish various possible solutions within a design space, and can be used to effectively categorize different design choices. For example, Luminate [43] visualizes the design space of creative writing via dimensions such as genres, personality, story tone. Similarly, PromptMagician [15] and Promptify [7] structure the design space of text-to-image generation by incorporating object- and style-related dimensions. Various visualization techniques, including scatter plots [43, 7], tree diagrams [44, 50], and parallel coordinates [56], have been employed to represent these dimensions and organize various solutions. Levels of abstraction further facilitate design space navigation by hierarchically structuring solutions from high-level concepts to fine-grained specifics. Different visualization techniques are often employed to effectively represent each abstraction layer. For example, ATMSeer [49] visualizes the design space of machine learning models through three hierarchical levels (\ie, algorithm type, hyper-partition, and hyper-parameter) using histogram, bar chart, and scatter plot, respectively. Meanwhile, studies have used focus + context technique to allow users to view both an abstract overview and detailed information simultaneously [56, 50]. DNN Genealogy [50] dynamically displaying solutions as either text labels or glyphs based on the calculated degree of interest. Semantic zooming is another technique for navigating hierarchical design spaces, dynamically adapting visual representations based on zoom level. For example, when visualizing the design space of LLM prompts, abstract level (zoomed out) represents solutions as simple points, whereas detailed level(zoomed in) reveals full text outputs or generated images [43, 15, 7]. Our work builds upon these prior visualization methods, uniquely adapting them to scaffold user cognition and provide in-situ support during multi-agent workflow design. 2 Designing \sysName 2.1 Understanding Design Challenges To identify the current challenges in designing multi-agent workflows, we conducted a formative study involving four AI researchers (E1-E4) with extensive experience in multi-agent systems. These participants represent our target user base of AI researchers and practitioners who design, development, and evaluate multi-agent workflows for solving complex tasks. Participants were recruited through personal networks and snowball sampling. All participants had led at least one major project about multi-agent LLM systems. One expert is the author of this paper while others are not. Each session consisted of a 45-60 minute semi-structured interview conducted over Zoom. Participants were asked to talk about their current practice in multi-agent workflows. They also reflected on the difficulties in constructing multi-agent workflows, and the limitations of the current tools. With consent from the participants, the interviews were recorded, transcribed, and thematically analyzed. We identified four key challenges: 1. Vast Design Space. All participants emphasized designing a multi-agent workflow involves navigating a vast and complex design space. Designers must make decisions across several dimensions: how to decompose a task, whether a task should be handled by a single agent or multiple agents, how agents should communicate, and how to configure individual agent behavior. Despite their experience, participants relied heavily on intuition, past examples, and human problem-solving analogies. For complex tasks, they noted that relying on “supervisor” agents for orchestration was often insufficient, and human oversight remained critical. The vast size and high complexity of design space made exploration difficult without structured support. 1. Unstructured Trial-and-error Exploration. Workflow design often follows an ad hoc trial-and-error process. Participants described iterating on ideas after failures, often tweaking small elements (e.g., prompts), mid-level elements (e.g., conversation rounds), or larger architectural components (e.g., switching from hierarchical to sequential designs). However, without a structured framework, this process was time-consuming, frustrating, and frequently unguided. 1. Design Fixation. Practitioners reported that they often begin by implementing a single workflow first and then iterating to refine it. Due to the complexity of implementing and fine-tuning a workflow, practitioners are often hesitant to explore alternatives. Most of their exploration were reactive in response to failures, rather than proactively seeking better designs. Once a runnable version of the workflow is achieved, further modifications were usually incremental, leading to premature convergence and missed opportunities for better-performing designs. 1. Navigating Trade-offs between Performance Metrics. Navigating multi-agent workflows often requires balancing conflicting priorities, such as computational cost, response latency, creativity, and accuracy. These trade-offs are subjective and context-dependent, such as generating fast forward videos for research papers (E1) and writing politics essays (E2), involving multiple design considerations. For example, E3 commented that “latency was the biggest concern” in one previous project, driving all design decisions. At the same time, it is challenging to account for different metrics simultaneously during the design phase. Some considerations (e.g., practical costs) often become apparent only after implementation, at which point practitioners may be reluctant to make major changes. <details> <summary>x2.png Details</summary>  ### Visual Description ## Diagram: Multi-Agent Task Allocation ### Overview The image is a diagram illustrating a multi-agent task allocation process, broken down into three stages: Task Planning, Agent Assignment, and Agent Optimization. The diagram shows the flow of tasks and agents through these stages, with visual representations of the processes involved. ### Components/Axes * **Stages (from top to bottom):** * Task Planning * Agent Assignment * Agent Optimization * **Nodes:** Represent tasks or agents. In the Task Planning and Agent Assignment stages, these are depicted as white circles. In the Agent Optimization stage, one node is highlighted in orange and surrounded by a dashed orange line. * **Connections:** Represent the flow of tasks/agents between stages. These are shown as solid gray lines with arrows indicating direction. Dashed gray lines connect the stages to detailed process representations on the right. * **Process Representations (right side):** * Top: Task Planning process, showing steps 1, 2a, 2b, and 3. * Middle: Agent Assignment process, showing the assignment of agents (represented by robot icons) to tasks. * Bottom: Agent Optimization process, showing agents with different capabilities (represented by icons like a wrench and a list) being optimized. ### Detailed Analysis **Task Planning Stage:** * Label: "Task Planning" is located on the left side of the diagram. * Representation: A gray rectangle with white circles inside. * Flow: A single node at the top splits into two nodes in the Agent Assignment stage. * Process Details (top-right): * A box labeled "step 1" leads to two parallel paths labeled "step 2a" and "step 2b". These paths converge into a box labeled "step 3". **Agent Assignment Stage:** * Label: "Agent Assignment" is located on the left side of the diagram. * Representation: A gray rectangle with white circles inside. * Flow: Two nodes from the Task Planning stage lead to multiple nodes in the Agent Optimization stage. * Process Details (middle-right): * Shows the assignment of agents (represented by robot icons) to tasks. The flow starts with a single robot, splits into two groups (one with one robot, the other with three robots), and then merges into a group of two robots. **Agent Optimization Stage:** * Label: "Agent Optimization" is located on the left side of the diagram. * Representation: A gray rectangle with white circles inside. One node is highlighted in orange and surrounded by a dashed orange line. * Flow: Multiple nodes from the Agent Assignment stage. * Process Details (bottom-right): * Shows agents with different capabilities (represented by icons like a wrench and a list) being optimized. The flow starts with a robot with a list icon, splits into two groups (one with a robot with a wrench, the other with three robots and a database icon), and then merges into a group of two robots, one with a wrench and the other with a list icon. ### Key Observations * The diagram illustrates a sequential process, starting with task planning, moving to agent assignment, and finally to agent optimization. * The orange node in the Agent Optimization stage likely represents a focus area or a specific agent being optimized. * The process details on the right provide a more granular view of the activities within each stage. ### Interpretation The diagram represents a high-level overview of a multi-agent task allocation system. It shows how tasks are planned, how agents are assigned to those tasks, and how the agents' performance or configuration is optimized. The use of visual elements like circles, arrows, and icons helps to convey the flow of information and the relationships between different components of the system. The orange highlight in the Agent Optimization stage suggests that this stage may be the most critical or complex part of the process, requiring focused attention. The diagram suggests a system where tasks are broken down and distributed among multiple agents, and then refined for optimal performance. </details> Figure 1: The design space of multi-agent workflows can be conceptualized across three hierarchical levels. Current frameworks primarily operate at the most granular level, which often leads to design fixation (the orange dot) rather than encourage exploration of the full solution space. Table 1: Design Patterns for Multi-Agent Workflows supported in \sysName. {tabu} p0.05cm| p2.4cmp5.2cmp6.0cm p1.9cm Pattern Definition Example References Level-1 Sequential <details> <summary>x3.png Details</summary>  ### Visual Description ## Diagram: Simple Flowchart ### Overview The image is a simple flowchart consisting of three rectangular boxes connected by arrows, indicating a sequential flow from left to right. ### Components/Axes * **Boxes:** Three rectangular boxes, each with a black outline and a white interior. They are arranged horizontally. * **Arrows:** Two right-pointing arrows, each with a black outline. The arrows connect the boxes, indicating the direction of flow. ### Detailed Analysis * The first box is located on the left side of the image. * A right-pointing arrow connects the first box to the second box, which is located in the center of the image. * Another right-pointing arrow connects the second box to the third box, which is located on the right side of the image. * The boxes are approximately the same size and are evenly spaced. ### Key Observations * The flowchart illustrates a linear process with three distinct steps or stages. * The arrows clearly indicate the direction of flow from one step to the next. ### Interpretation The diagram represents a basic sequential process. Each box likely represents a specific action, stage, or component within a larger system. The arrows indicate the order in which these actions or stages occur. The simplicity of the diagram suggests a high-level overview of a process, without detailed information about the contents of each step. </details> The task can be decomposed into a sequence of steps, which are executed sequentially and completed in an orderly progression. Debugging visualizations via first interpreting code, identifying issues, and then writing code [54]. [54] [5] [2] [55] [41] Parallel <details> <summary>x4.png Details</summary>  ### Visual Description ## Diagram: Simple Flowchart ### Overview The image is a simple flowchart consisting of four rectangular boxes and arrows indicating the flow of information or process. One box is on the left, and three boxes are stacked vertically on the right. ### Components/Axes * **Boxes:** Four rectangular boxes, all with black borders and white fill. No text is present inside the boxes. * **Arrows:** * A solid arrow connects the left box to the middle box on the right. * A dotted arrow connects the left box to the top box on the right. * A solid arrow connects the left box to the bottom box on the right. ### Detailed Analysis * **Left Box:** Located on the left side of the diagram. * **Top Right Box:** Located at the top of the right side. Connected to the left box by a dotted arrow. * **Middle Right Box:** Located in the middle of the right side. Connected to the left box by a solid arrow. * **Bottom Right Box:** Located at the bottom of the right side. Connected to the left box by a solid arrow. ### Key Observations * The diagram shows a one-to-many relationship, where one element on the left branches out to three elements on the right. * The use of a dotted arrow for the top right box might indicate a different type of relationship or a less direct connection compared to the solid arrows. ### Interpretation The diagram represents a process or system where one input or source (left box) leads to three different outputs or destinations (right boxes). The dotted arrow could signify an optional path, a less frequent connection, or a different type of processing compared to the solid arrows. Without further context, the specific meaning of each box and arrow is unclear, but the diagram suggests a branching or distribution process. </details> The task can be divided into multiple subtasks that can be parallelized for potential speedup. When generating a travel plan, ideating various vacation activities simultaneously [53]. [54] [5] [40] [55] [41] Level-2 Reflection <details> <summary>x5.png Details</summary>  ### Visual Description ## Diagram: Robot Interaction Flow ### Overview The image is a diagram illustrating a process involving two robots and two rectangular boxes. The diagram shows a flow of information or actions between the boxes and the robots, and between the robots themselves. ### Components/Axes * **Boxes:** Two rectangular boxes, one on the left and one on the right. * **Robots:** Two identical robot icons, each with a square head, antenna, and a smiling face. * **Arrows:** Solid and dashed arrows indicating the direction of flow. ### Detailed Analysis * **Left Box to Robot 1:** A solid arrow points from the left box to the first robot. * **Robot 1 to Robot 2:** A solid curved arrow points from the first robot to the second robot. * **Robot 2 to Robot 1:** A dashed curved arrow points from the second robot back to the first robot. * **Robot 2 to Right Box:** A solid arrow points from the second robot to the right box. ### Key Observations * The diagram depicts a sequential process starting with the left box, moving to the first robot, then to the second robot, and finally to the right box. * There is a feedback loop between the two robots, indicated by the dashed arrow. ### Interpretation The diagram likely represents a process where data or actions flow from an initial input (left box) to a processing stage involving two robots. The robots interact with each other, possibly exchanging information or coordinating tasks, before the final output is sent to the right box. The dashed arrow suggests a feedback mechanism or iterative process between the robots. The boxes could represent input and output data stores, or initial and final states of a process. </details> One agent generates and refines a response while another evaluates it, repeating the process until a predefined criterion (quality or number of iterations) is met. Often used to improve accuracy. One agent generates code, which a reviewer then assesses for quality and returns feedback for adjustment [52]. [5] [42] [33] [57] [42] [63] [22] [46] [60] Redundant <details> <summary>x6.png Details</summary>  ### Visual Description ## Diagram: Robot Process Flow ### Overview The image is a diagram illustrating a process flow involving robots. It shows a sequence of steps where an initial input (represented by a square) is processed by a series of robots, with some robots feeding back into the process. The final output is also represented by a square. ### Components/Axes * **Shapes:** Squares and robot icons. * **Arrows:** Indicate the direction of the process flow. * **Robots:** Represented by a square head with a smiling face and an antenna. ### Detailed Analysis 1. **Initial Input:** A square on the left-hand side represents the starting point of the process. 2. **First Robot:** An arrow points from the initial input to a robot. 3. **Branching:** The first robot leads to two other robots, one above and one below. These robots are connected back to the first robot via curved arrows. 4. **Second Robot:** The first robot also leads to a second robot in the main flow. 5. **Final Output:** The second robot leads to a square on the right-hand side, representing the final output. ### Key Observations * The diagram shows a linear process with feedback loops. * The robots seem to be the active agents in the process. * The squares represent the input and output states. ### Interpretation The diagram illustrates a process where an initial input is processed by a series of robots. The feedback loops suggest that the robots may be refining or correcting the process based on the output of other robots. The final output represents the result of this automated process. The diagram could represent a manufacturing process, a data processing pipeline, or any other automated system. </details> Multiple agents simultaneously respond to the same input while adopting different personas or perspectives, generating diverse outputs. Often used to improve robustness or creativity. Four agents in roles of physics, chemistry, biology, and general science experts attempt to solve graduate-level questions in science individually, then their answers are aggregated [22]. [27] [5] [18] [36] [22] Supervision <details> <summary>x7.png Details</summary>  ### Visual Description ## Diagram: Robot Communication Flow ### Overview The image is a diagram illustrating a communication flow involving robots. It shows a central robot interacting with two external boxes and three subordinate robots. The communication pathways are indicated by arrows, with some connections being solid and others dashed, suggesting different types of interaction. ### Components/Axes * **Robots:** Represented by a square body with a smiling face and an antenna on top. * **Boxes:** Represented by empty squares. * **Arrows:** Indicate the direction of communication. Solid arrows represent a direct communication, while dashed arrows represent a different type of communication. ### Detailed Analysis * **Top-Left Box:** A box on the top-left has a solid arrow pointing towards the central robot. * **Central Robot:** A robot in the center receives input from the top-left box and sends output to the top-right box via solid arrows. It also communicates with three robots below it. * **Top-Right Box:** A box on the top-right receives a solid arrow from the central robot. * **Bottom Robots:** Three robots are positioned at the bottom. The central robot communicates with the middle bottom robot via a solid, double-headed arrow. The central robot communicates with the left bottom robot via a solid, curved arrow. The central robot communicates with the right bottom robot via a dashed, curved arrow. ### Key Observations * The central robot acts as a hub, receiving input from one external source and distributing output to another, while also interacting with a group of subordinate robots. * The use of solid and dashed arrows suggests different types or priorities of communication. ### Interpretation The diagram likely represents a hierarchical communication system. The central robot could be a controller or coordinator, receiving instructions from an external source (top-left box), processing them, and then distributing tasks or information to the subordinate robots (bottom robots). The output to the top-right box could represent feedback or results of the tasks performed. The dashed arrow might indicate a less critical or less frequent communication channel. The solid, double-headed arrow between the central robot and the middle bottom robot suggests a two-way communication or feedback loop. </details> A hierarchical structure where a central agent dynamically decides which worker agents to act based on the input. Often used for complicated tasks that require diverse expertise and careful quality control. A supervisor agent directs user inquiries to specialized agents based on intent—such as technical support, refund and return processing, or handling complaints [4]. [5] [26] [33] [4] [46] [57] [19] [2] [4] [58] [53] Discussion <details> <summary>x8.png Details</summary>  ### Visual Description ## Diagram: Robot Workflow ### Overview The image is a diagram illustrating a workflow involving robots. It shows a sequence of actions where data or objects are passed between a starting point, robots, and an ending point. The robots also interact with each other in a circular fashion. ### Components/Axes * **Boxes:** Two empty boxes, one on the left and one on the right. * **Robots:** Three robot icons arranged in a triangular formation. Each robot has a square head, a circular antenna, and a smiling face. * **Arrows:** Solid arrows indicate a linear flow from the left box to a robot and from a robot to the right box. Dotted arrows indicate a circular flow between the robots. ### Detailed Analysis * **Left Box:** Located on the left side of the diagram. A solid arrow points from this box to the top-left robot. * **Top-Left Robot:** Receives input from the left box. A dotted arrow points from this robot to the top-right robot. A dotted arrow also points from the bottom robot to this robot. * **Top-Right Robot:** Receives input from the top-left robot. A solid arrow points from this robot to the right box. A dotted arrow points from this robot to the bottom robot. * **Bottom Robot:** Receives input from the top-right robot. A dotted arrow points from this robot to the top-left robot. * **Right Box:** Located on the right side of the diagram. Receives input from the top-right robot. ### Key Observations * The diagram shows a workflow that starts with an input (left box), is processed by a series of robots, and ends with an output (right box). * The robots are interconnected, suggesting a collaborative or iterative process. * The use of dotted arrows for the robot-to-robot connections indicates a different type of flow compared to the solid arrows connecting the boxes to the robots. ### Interpretation The diagram represents a system where data or objects are processed by a network of robots. The initial input is fed into the robot network, where the robots interact and process the input. The final output is then delivered to the right box. The circular flow between the robots suggests a feedback loop or a collaborative process where the robots share information or tasks. The diagram could represent a manufacturing process, a data processing pipeline, or any other system where robots work together to achieve a common goal. </details> A group of agents engage in a conversation, where they take turns speaking in a round-robin, random manner, or simultaneously. Often used to increase creativity for open-ended tasks. Three agents in roles of news author, critic, and psychologist take turns to generate their response based on the conversation to evaluate open-ended question answers [10]. [46] [10] [29] [12] [33] [19] [57] Level-3 Single Agent <details> <summary>x9.png Details</summary>  ### Visual Description ## Diagram: Robot Process Flow ### Overview The image is a simple diagram illustrating a process flow involving a robot. The flow starts with a square, proceeds to a robot icon, and ends with another square. The robot has dotted lines extending downwards, suggesting interaction or output. ### Components/Axes * **Squares:** Two squares, one on the left and one on the right, represent input and output or start and end points. * **Robot:** A stylized robot icon in the center, with a head, antenna, and a simple face. * **Arrows:** Two arrows indicate the direction of the process flow, moving from left to right. * **Dotted Lines:** Dotted lines extending downwards from the robot, suggesting interaction or output. ### Detailed Analysis * **Left Square:** Located on the left side of the diagram. * **Arrow 1:** Points from the left square to the robot. * **Robot:** Positioned in the center of the diagram. * **Arrow 2:** Points from the robot to the right square. * **Right Square:** Located on the right side of the diagram. * **Dotted Lines:** Extend downwards from the robot, indicating some form of action or output. ### Key Observations The diagram shows a linear process where an input (left square) is processed by a robot, resulting in an output (right square). The dotted lines suggest the robot performs some action or generates output during the process. ### Interpretation The diagram likely represents a simplified model of an automated process. The robot symbolizes an automated system or algorithm that takes an input, processes it, and produces an output. The dotted lines could represent data logging, feedback, or other interactions performed by the robot during the process. The diagram is abstract and lacks specific details about the nature of the input, process, or output. </details> The basic building block of multi-agent workflows, a single LLM enhanced with augmentations such as data retrieval, tools, prompting, and memory. A single agent augmented with Web search tool queries for question-answering [35]. A basic module with diverse augmentations to enhance individual ability and versatility. [26] [5] [22] [52] [57] [19] [2] [28] [46] [35] 2.2 Design Space of Multi-agent Workflows To address these challenges, \sysName summarizes and visualizes the design space of multi-agent workflows, offering users a structured overview of various possible solutions. This subsection introduces the three key components of this design space: design dimensions, reusable design patterns, hierarchical abstraction levels, grounded in both user study and recent literature. While these components are not exhaustive and are likely to evolve with ongoing research, they provide a strong foundation for demonstrating the usefulness of visualizing design space and integrating established design patterns. 2.2.1 Design Dimensions Dimensions are key factors that define and distinguish various workflows within the design space, such as the running time, the number of agents, and the rate of hallucination. Certain key dimensions are common across various tasks, while others are specific to particular applications. For example, dimensions such as accuracy and computational efficiency are universally relevant to most tasks, as practitioners generally seek to optimize them regardless of application. In contrast, specialized dimensions like the degree of dynamic updating are important for specific contexts. Our formative study revealed that while experts (E1-E4) unanimously rejected fully automated workflow generation, they sometimes incorporated components with dynamic updating capabilities (e.g., agents that select subsequent agents based on output analysis) in particular scenarios where adaptability was essential. Dimensions can also be highly granular, addressing specific characteristics like output length, formatting elements (e.g., bullet points), or the presence of certain keywords. They can influence design choices across the design process, such as reducing the number of agents to lower computational costs. Since these design dimensions can be very task-specific, there is not an exclusive list of all possible dimensions. 2.2.2 Design Patterns Design patterns are reusable, generalizable strategies for structuring workflows that emerge from repeated adaptation across numerous existing implementations. A design patterns is often associated with optimizing specific performance dimensions, e.g., reflection based design pattern can improve accuracy. Even though existing surveys provided valuable insights about workflow design, they either focus on one aspect of the design process (e.g., task planning [18]) or overlook the high-level abstraction within the design space and focus on implementation details [19]. Therefore, we extended existing workflow surveys [57, 18, 19] and analyzed 43 multi-agent systems drawn from papers, blog posts, framework documentation, via keyword-based searches (e.g., “multi-agent”, “LLM workflows”, “agent orchestration”). Using axial coding, we grouped recurring structures into design patterns. While terminology varies across sources (e.g., “Group Chat” vs “Discussion”), we adopted the most commonly used terms in our dataset for consistency. A full summary of these patterns is provided in subsection 2.1. Although our collection does not capture every possible structure (e.g., dynamic prompt-based feedback [22]), it provides a robust foundation that effectively demonstrates the value of visualizing the design space and integrating established patterns into workflow creation processes. The detailed analysis process is included in the project web page. 2.2.3 Hierarchical Abstraction Levels We model the design space as three levels of abstraction (Fig. 1). This approach follows the established practices in design-related fields that employ layered abstractions to manage complexity and structure the design process [34, 48]. To construct each abstraction layer, we employ axial coding to organize the design patterns documented in our literature review, alongside the important design decisions and challenges identified through our formative study. This coding process groups related concepts into three hierarchical levels that progressively address the workflow development complexity. - The first level, Task Planning, serves as the strategic foundation where complex tasks are systematically decomposed into discrete, manageable sub-tasks. - The second level, Agent Assignment, builds directly upon the task decomposition by assigning appropriate agents to each sub-task. Some tasks may require a single agent, while others may necessitate collaboration between multiple agents with complementary roles. - The third level, Agent Optimization, provides granular control mechanisms, allowing users to fine-tune individual agents through prompt engineering, tool integration, or access to external datasets, to precisely shape their behavior and response characteristics. These hierarchical levels reflect a natural design progression, from abstract strategy to detailed implementation, and form the structural backbone of the visual exploration offered by \sysName <details> <summary>x10.png Details</summary>  ### Visual Description ## User Interface Analysis: FlowForge Task Management ### Overview The image presents a user interface for a task management system called "FlowForge." It appears to be designed for conducting and reviewing research papers, emulating peer-review practices. The UI includes task descriptions, workflow visualizations, interactive elements, and message logs. ### Components/Axes * **Header:** * "FlowForge" title. * Task Description (Section A): "Conduct a comprehensive and systematic review of a given research paper that emulates standard peer-review practices. Multiple, distinct." * "SUBMIT TASK" button. * "Example Input (Optional)" field. * "Example Tasks" dropdown menu. * **Design Space Overview (B1):** * Title: "Design Space Overview" * Flows: Flow 1, Flow 2, Flow 3, Flow 4 * Nodes: 2-1, 2-2, 3-1, 3-2, 4-1, 4-2 * Sub-nodes: 2-1a, 2-1b, 2-2a, 3-1a, 3-2a, 3-2b, 3-2c, 4-1a, 4-2a * **Scatter Plot (B2):** * X-Axis: "# agents" * Y-Axis: "latency" * Data Series: * S (color not clear, but likely blue) * D (pink/red) * RF (orange) * RD (green) * **Workflow Diagram (C1):** * Nodes: * Initial review: "Reflection" dropdown, "6 LLM calls" * Structure check: "Reflection" dropdown, "6 LLM calls" * Method assessment: "Reflection" dropdown, "6 LLM calls" * Evaluate results: "Discussion" dropdown, Task Description: "Compile identified strengths and weaknesses, focusing on the paper's contributions and areas needing improvement.", "4*3 + 1 = 13 LLM Calls" * Sub-nodes: Agent #1 (Reviewer A), Agent #2 (Reviewer B), Agent #3 (Reviewer C), Agent #4 (Reviewer D), Summary (Lead Facilitator) * Draft review: "Reflection" dropdown, "6 LLM calls" * **Agent Interaction Examples (C2):** * Reflection: Diagram showing a simple loop. * Discussion: "A group of agents engage in a conversation, where they take turns speaking in a round-robin, random manner, or simultaneously. Extra agent to summarize is optional Example: Brainstorming ideas." * Redundant: Diagram showing a more complex loop. * Supervision: Diagram showing a simple sequence. * **Messages (C3):** * Title: "Messages" * Step 1 Agent 1: "The initial reading of the paper reveals a well-structured argument that is both clear and coherent. The authors have articulated their thesis effectively, providing a logical progression of ideas that support their main claim. The significance of the research is evident, as it addresses a significant gap in the existing literature and offers potential for future scholarly investigation. Moreover, the originality of the work introduces innovative concepts that set it apart, presenting a novel perspective that further scholarly investigation. The methodology appears robust and the evidence appears reliable." * Step 1 Agent 2: "Building on the initial impressions, the paper's argument is not only clear and coherent but also engaging, guiding the reader through complex ideas with ease. The clarity of the argument is further extended by the structured format, which logically sequences each point and builds upon the previous sections. The significance of the research cannot be overstated, as it provides new insights into an underexplored area, potentially influencing both academic and practical spheres. The methodology appears robust and the evidence appears reliable." * Step 1 Agent 3: "Building on the initial impressions, the paper's argument is not only clear and coherent but also engaging, guiding the reader through complex ideas with ease. The clarity of the argument is further extended by the structured format, which logically sequences each point and builds upon the previous sections. The significance of the research cannot be overstated, as it provides new insights into an underexplored area, potentially influencing both academic and practical spheres. The methodology appears robust and the evidence appears reliable." * Step 1 Agent 4: "The paper's argument is not only clear and coherent but also engaging, guiding the reader through complex ideas with ease. The clarity of the argument is further extended by the structured format, which logically sequences each point and builds upon the previous sections. The significance of the research cannot be overstated, as it provides new insights into an underexplored area, potentially influencing both academic and practical spheres. The methodology appears robust and the evidence appears reliable." * Step 1 Summary: "Upon the initial reading of the paper, it presents a well-structured and coherent argument that effectively guides readers through complex ideas with ease. The authors have articulated their thesis clearly, with each section logically transitioning into the next, enhancing the clarity and engagement of the reader. The paper addresses a significant gap in the existing literature, offering new insights that have the potential to influence both academic research and practical applications. It is particularly noteworthy, as it challenges existing paradigms and open up new avenues for research. Methodologies are meshing well." * Step 2 Worker 1: "Upon the initial reading of the paper, it presents a well-structured and coherent argument that effectively guides readers through complex ideas. The authors have articulated their thesis clearly, with each section logically transitioning into the next, enhancing the clarity and engagement of the reader. The paper addresses a significant gap in the existing literature, offering new insights that have the potential to influence both academic research and practical applications. It is particularly noteworthy, as it challenges existing paradigms and open up new avenues for research. Methodologies are meshing well." * Step 3 Worker 1: "Upon the initial reading of the paper, it presents a well-structured and coherent argument that effectively guides readers through complex ideas. The authors have articulated their thesis clearly, with each section logically transitioning into the next, enhancing the clarity and engagement of the reader. The paper addresses a significant gap in the existing literature, offering new insights that have the potential to influence both academic research and practical applications. It is particularly noteworthy, as it challenges existing paradigms and open up new avenues for research. Methodologies are meshing well." * **Footer:** * "Time Used: 30.27s" * "LOOKS GOOD, CONTINUE" button. * "TRY ANOTHER ONE" button. ### Detailed Analysis * **Design Space Overview (B1):** This section visualizes different workflow configurations. Each "Flow" (1-4) represents a distinct sequence of tasks. The numbers below each flow (e.g., 2-1, 2-2) likely represent specific tasks or agents within that flow. The sub-nodes (e.g., 2-1a, 2-1b) could represent sub-tasks or iterations within each task. * **Scatter Plot (B2):** This plot allows users to visualize the relationship between the number of agents and latency for different workflow configurations. The data points are clustered, suggesting distinct performance characteristics for each configuration. The exact values are difficult to extract without a grid, but the relative positions indicate the general trends. * **S (likely blue):** Appears to have a moderate number of agents and moderate latency. * **D (pink/red):** Appears to have a low number of agents and low latency. * **RF (orange):** Appears to have a moderate number of agents and high latency. * **RD (green):** Appears to have a high number of agents and moderate latency. * **Workflow Diagram (C1):** This diagram outlines the steps involved in the research paper review process. It includes stages like "Initial review," "Structure check," "Method assessment," "Evaluate results," and "Draft review." Each stage involves "LLM calls," suggesting the use of Large Language Models for automated analysis. The "Evaluate results" stage is further broken down into individual agent contributions (Reviewer A, Reviewer B, Reviewer C, Reviewer D) and a summary by a Lead Facilitator. The number of LLM calls for the "Evaluate results" stage is explicitly stated as "4*3 + 1 = 13 LLM Calls." * **Agent Interaction Examples (C2):** This section provides examples of how agents can interact within the system. The "Discussion" example highlights a round-robin conversation, while other examples illustrate different interaction patterns. * **Messages (C3):** This section displays messages generated by different agents during the review process. The messages provide insights into the agent's reasoning and analysis of the paper. The messages are generally positive, indicating that the paper is well-structured, clear, and coherent. ### Key Observations * The UI is designed to support a structured and systematic review process. * LLMs are heavily integrated into the workflow, automating various analysis tasks. * The system allows for different workflow configurations and visualizations of performance metrics. * Agent interactions are diverse, ranging from simple sequences to complex round-robin discussions. ### Interpretation The FlowForge system appears to be a sophisticated tool for managing and automating research paper reviews. The integration of LLMs streamlines the analysis process, while the interactive visualizations and agent communication features enhance collaboration and decision-making. The system's flexibility allows users to experiment with different workflow configurations and optimize performance based on specific requirements. The positive messages generated by the agents suggest that the system is effective in identifying and highlighting the strengths of the reviewed paper. The system aims to improve the efficiency and quality of peer review by leveraging AI and collaborative workflows. </details> Figure 2: Interface Overview. Users start by entering a task description (A). \sysName supports workflow creation through two main views: the Design Space View, with a hierarchical tree (B1) and a scatter plot (B2), and the Canvas View, which shows workflow details (C1), in-situ design suggestions (C2), and execution results (C3). 2.3 Design Goals \sysName is developed with the following design goals in mind, informed directly by the challenges uncovered in our formative study: 1. Step-by-step guidance from abstract to concrete. \sysName supports a top-down design process by structuring the workflow creation experience, guiding users from high-level task decomposition to granular agent optimization. This helps users navigate the complexity of the design space and mitigates unstructured exploration (C.1 , C.2 ). 1. Support for divergent and convergent thinking. The system encourages users to explore multiple workflow alternatives early in the design process before committing to a final solution. This prevents them from design fixation and promotes more deliberate decision-making (C.3 ). 1. In-situ guidance for best practices. As users explore the design space, \sysName provides context-aware design pattern suggestions drawn from validated design patterns. These suggestions surfaced at appropriate abstraction levels, assisting users without disrupting their flow (C.1 C.2 ). 1. Parallel exploration across multiple dimensions. Users can evaluate and compare different workflows across key performance dimensions (e.g., latency, accuracy, cost) as well as task-specific dimensions (e.g., format constraints), enabling a more holistic understanding of trade-offs and priorities (C.4 ). 3 System \sysName is an interactive visualization system that supports the design and exploration of multi-agent workflows. As shown in Fig. 2, users begin by entering a task description (A), after which \sysName generates and visualizes candidate workflows. The interface is composed of two main components: the Design Space View, which displays the levels and dimensions via a hierarchical tree (B1) and a scatter plot (B2); and the Canvas View, which presents the detailed structure of a selected workflow (C1), along with in-situ design suggestions (C2) and execution results (C3). 3.1 Design Space View: Visualizing Abstraction Levels The hierarchical tree (Fig. 2.B1) captures three levels of abstraction in workflow design. This tree dynamically updates as users navigate the space, guiding them through the progressive stages of workflow design: task planning, agent assignment, and agent optimization (DG.1.). This structure encourages both depth-first exploration (\ie, jump directly to concrete implementation) and breath-first exploration (\ie, comparing options before committing), as the latter is often overlooked in current practices (DG.2.). For instance, at the task planning level, users may compare task decompositions with three vs. four subtasks. At the agent assignment level, they can explore collaboration strategies, such as having two agents co-work on a subtask versus one completing it and another reviewing the output. To represent workflows across these levels, we introduce a set of visual glyph nodes (Fig. 3). These glyph nodes encode the computational efficiency, a universally relevant dimension in workflow design (Sect. 2.2.1) that can be estimated before execution. - Level 1 (Task Planning): Glyphs use the number of arcs to indicate the number of subtasks. Stacked arcs represent parallel subtasks, and node size reflects the number of sequential steps. - Level 2 (Agent Assignment): Glyphs consist of bars, where each bar represents a pseudo-time step of the workflow running process, and its height indicates the number of concurrent agent calls. - Level 3 (Agent Optimization): As this level does not introduce structural changes, glyphs are rendered as file icons, denoting a fully runnable workflow. These glyphs provide a quick, visual summary of each workflow, helping users efficiently compare alternatives without delving into detailed specifications. While we experimented with encoding additional dimensions into the glyphs, we found that increasing complexity introduced unnecessary cognitive burdens. <details> <summary>x11.png Details</summary>  ### Visual Description ## Diagram: Agent Workflow ### Overview The image illustrates a workflow involving agents, broken down into three main stages: Task Planning, Agent Assignment, and Agent Optimization. Each stage is represented by a diagram showing the flow of information or resources. ### Components/Axes * **Section A: Task Planning** * Labels: "A Task Planning" * Steps: "step 1", "step 2a", "step 2b", "step 3" * Flow: A process starts with "step 1", splits into "step 2a" and "step 2b", then merges into "step 3", leading to a circular process represented by a broken circle. * **Section B: Agent Assignment** * Labels: "B Agent Assignment" * Components: Boxes containing robot icons, representing agents. * Flow: Agents are grouped and then reassigned, indicated by arrows. * **Section C: Agent Optimization** * Labels: "C Agent Optimization" * Components: Boxes containing robot icons with tools (wrench) or lists, representing agent capabilities. A database icon is also present. * Output: A document icon. * **Time Axis**: In the middle of the bottom row, a horizontal arrow labeled "time" indicates the progression of agent activity. * **Agent Count**: The number of robot icons stacked vertically represents the number of agents at a given time. ### Detailed Analysis * **Task Planning:** * The process begins with a single task ("step 1"). * The task splits into two parallel sub-tasks ("step 2a" and "step 2b"). * These sub-tasks converge into a single task ("step 3"). * The process then enters a cyclical phase, indicated by the broken circle. * **Agent Assignment:** * Agents are initially grouped into two clusters: one with a single agent and another with three agents. * These agents are then reassigned into two groups of two agents each. * The flow suggests a redistribution or balancing of agents. * **Agent Optimization:** * Agents are depicted with different capabilities: some with tools (wrench), some with lists, and some connected to a database. * The process involves optimizing these agents, leading to a final output represented by a document. * **Time-Based Agent Activity:** * The number of agents varies over time, as shown by the stacked robot icons. * The agent count starts low, peaks in the middle, and then decreases. * The final output is represented by a series of vertical bars, suggesting a processed or summarized result. ### Key Observations * The workflow progresses from planning to assignment and then to optimization. * Agent capabilities and assignments are dynamically adjusted throughout the process. * The time-based agent activity shows a fluctuating demand for agents. ### Interpretation The diagram illustrates a dynamic workflow where tasks are planned, agents are assigned and reassigned, and their capabilities are optimized. The cyclical nature of the task planning stage suggests continuous refinement or iteration. The agent assignment stage indicates a need for flexible resource allocation. The agent optimization stage highlights the importance of equipping agents with the right tools and data. The time-based agent activity suggests that the demand for agents varies depending on the stage of the workflow. Overall, the diagram depicts a sophisticated system that adapts to changing needs and optimizes agent performance. </details> Figure 3: Glyphs. (A) At Level 1 (task planning), arcs represent subtasks and node size for sequential count. (B) At Level 2 (agent assignment), bars show agent calls over pseudo-time steps. (C) At Level 3 (agent optimization), workflows are denoted as file icons, as no new structural information is added and the workflow is fully runnable. 3.2 Design Space View: Visualizing Design Dimensions Although the hierarchical tree organizes workflows by abstraction level, it offers limited visibility into design dimensions (e.g., number of agents, number of subtasks, number of LLM calls, running time). To address this, \sysName includes a configurable scatter plot (Fig. 2.B2) that enables comparison of workflows across key dimensions (DG.4.). In this scatter plot, users can assign different dimensions to the x- and y-axes. We provide a list of predefined dimensions based on feedback from formative studies, while allowing users to add customized dimensions (e.g., user rating for the workflow outputs) via manually annotating the workflow outputs. Workflows with undefined values on a selected axis are displayed in gray margin areas. For example, in \sysName: Guiding the Creation of Multi-agent Workflows with Design Space Visualization as a Thinking Scaffold. A2, workflows at Level 1 (task planning) do not include information about the number of agents (y-axis), and are put in the bottom gray area of the scatter plot. The scatter plot only displays workflows at the current level or one level above the selected node in the tree. This selective display is informed by the user feedback we received during the iterative design process, as showing all workflows across three abstraction levels can be cognitively overwhelming. We chose a scatter plot because it provides a clear overview of the design space, making it easier to identify regions that are heavily explored (\ie, clusters) versus those that remain underexplored (\ie, spatial gaps). For example, Fig. 2 shows that the current exploration has exclusively focused on workflows that have multiple agents, as evidenced by the glyph nodes clustered in the upper-right corner. This insight may prompt users to consider exploring workflows with fewer subtasks or simpler configurations that have been overlooked (DG.2.). We have explored alternative designs such as tables, parallel coordinates, and connected scatter plots. Although tables and parallel coordinates can display more than two dimensions simultaneously, users often struggle to interpret and compare more than two dimensions effectively [49]. During the iterative design process, we also experimented explicitly connecting related workflows across levels in the scatter plot. However, the additional edges in the scatter plot were unnecessary in most cases, since the children node typically share some dimension values as their parent nodes and their hierarchical relation can be easily recognized based on the alignment of the glyph nodes, e.g., same x position in the \sysName: Guiding the Creation of Multi-agent Workflows with Design Space Visualization as a Thinking Scaffold.A2. <details> <summary>x12.png Details</summary>  ### Visual Description ## Workflow Diagram: Task Planning and Agent Optimization ### Overview The image presents a workflow diagram outlining a task planning process, specifically focusing on "Lead with Curiosity" and agent optimization. It details the steps involved, including task descriptions, agent assignments, and optimization processes. ### Components/Axes * **A: Task Planning** * **1. Lead with Curiosity:** This section describes the initial task planning stage. * **Step Description:** "Open with a compelling question related to the research topic, drawing viewers in. Visual: Text animation of the question over an intriguing background. Duration: 3" * **B1: Agent Assignment** * **Lead with Curiosity:** This section shows the agent assignment for the "Lead with Curiosity" task. * **Reflection:** Dropdown menu. * Two robot icons labeled "1" and "2". * "6 LLM calls" * **B2: Agent Assignment** * **Lead with Curiosity:** This section shows the agent assignment for the "Lead with Curiosity" task. * **Reflection:** Dropdown menu. * Task Description: Open with a compelling question related to the research topic, drawing viewers in. Visual: Text animation of the question over an intriguing background. Duration: 3 seconds. * Evaluator Persona: You are the evaluator, responsible for assessing the quality of each video. * Optimizer Persona: You are the optimizer, tasked with refining scripts based on feedback. * 3\*2 = 6 LLM Calls * Optimizer Goal: Iteratively improve each script, focusing on delivering a unique and engaging narrative while maintaining accessibility and ensuring precise timing. * Max Round: 3 * **C: Agent Optimization** * **Step 1: Optimizer** * LLM Model: GPT-4o (dropdown menu) * System Prompt: "You are the optimizer, tasked with refining scripts based on feedback. The step description is Open with a compelling question related to the research topic." * Tools: (dropdown menu) * **Step 1: Evaluator** * LLM Model: GPT-4o (dropdown menu) * System Prompt: "You are the evaluator, responsible for assessing the quality of each video script. The step description is Open with a compelling question related to the research topic, drawing" * Tools: (dropdown menu) * Feedback is sent from the Evaluator to the Optimizer. * The Optimizer submits to the Evaluator. ### Detailed Analysis or ### Content Details * **Task Planning (A):** The initial step involves creating a compelling question to engage viewers, using text animation over an intriguing background, with a duration of 3 (presumably seconds). * **Agent Assignment (B1 & B2):** The "Lead with Curiosity" task is assigned to agents, with 6 LLM calls. The agents are split into Evaluator and Optimizer roles. * **Agent Optimization (C):** This section details the roles of the Optimizer and Evaluator. Both use the GPT-4o model. The Optimizer refines scripts based on feedback, while the Evaluator assesses the quality of the video script. Feedback flows from the Evaluator to the Optimizer, and the Optimizer submits to the Evaluator. ### Key Observations * The workflow emphasizes iterative improvement through feedback between the Optimizer and Evaluator agents. * The "Lead with Curiosity" task is central to the process, appearing in both the Task Planning and Agent Assignment sections. * The number of LLM calls is consistently 6. ### Interpretation The diagram illustrates a structured approach to task planning and agent optimization, focusing on creating engaging content through a "Lead with Curiosity" strategy. The use of Optimizer and Evaluator agents, along with iterative feedback loops, suggests a process designed for continuous improvement and quality control. The consistent use of 6 LLM calls may indicate a fixed budget or resource allocation for this particular task. The diagram highlights the importance of both creative content generation (Optimizer) and critical assessment (Evaluator) in achieving the desired outcome. </details> Figure 4: In the Canvas View, the visual representation of each node adapts based on the abstraction level. 3.3 Canvas View: Editing Individual Workflows Workflows selected from the design space are displayed in the Canvas View, as interactive node-link diagrams. Users can navigate between abstraction levels (DG.1.), using the “Looks Good, Continue” and “Try Another One” buttons in the Canvas View. The former advances the current workflow to a more concrete level (e.g., from task planning to agent assignment), while the latter prompts alternative solutions at the same level (e.g., a different way to decompose the task). To prevent potential confusion between the tree representation of the design space and the node-link diagram of an individual workflow, different visualizations are used: circular nodes and step-wise edges for the tree and rectangular nodes with curved edges for the workflow. Node styles of the workflow adapt according to the abstraction level selected in the left-side Design Space View: - Level 1 (Task Planning): Nodes represent subtasks with editable text descriptions. Edges denote dependencies and execution order of subtasks, with the flow progressing from left to right. Users can modify task structure, text, and dependencies. - Level 2 (Agent Assignment): Nodes correspond to agent groups assigned to subtasks. When zoomed out (B1), nodes show summary information, \ie, the subtask name, the associated collaboration pattern, and the estimated number of LLM calls. Zooming in (B2) reveals detailed configurations, such as agent roles, collaboration patterns (e.g., reflection, supervision) (subsection 2.1), and pattern-specific parameters (e.g., number of agents in redundancy or discussion rounds). - Level 3 (Agent Optimization): Nodes represent individual agents, and agents working on the same subtask are grouped within the same purple rectangle. Users can configure agents using common patterns, including prompt engineering, tool usage, and retrieval augmentation. 3.4 Design Cards: In-situ Guidance via Design Patterns To support in-situ design thinking, \sysName surfaces relevant design patterns on the right side of the Canvas View. Each pattern is presented as a card with its name, visual representation, and expandable details (Fig. 2.C2). When users hover over a pattern card, all corresponding nodes adopting the pattern are highlighted. Since different design patterns tend to prioritize different performance dimensions, we also annotate the scatter plot axes with relevant patterns when one of these dimensions is used as an axis. For example, as shown in Fig. 2.B2, when latency is selected as the y-axis, patterns such as Redundancy, Reflection, and Discussion, are recommended and marked along the axis to aid interpretation: Redundancy at the bottom (minimal latency), Reflection mid-range (moderate latency), and Discussion near the top (high latency due to multi-round interactions). 3.5 Workflow and Suggestion Generation The generation approach employs a straightforward, prompt-based methodology using GPT-4o [37], as existing methods are either not open-source or limited to specific tasks [21]. This prompt-based approach proves sufficient for demonstrating the effectiveness of \sysName. While we acknowledged that the generation can be further improved with more advanced approaches (e.g., reinforcement learning), novel methods for automatic workflow generation and recommendation is beyond the scope of this study. Specifically, \sysName generates workflows and design patterns following the three levels of abstraction: - Level 1 (Task Planning): \sysName prompts GPT-4o to generate diverse decompositions of a given task while maintaining a unified format. Each generated subtask is formatted to include a label, task description, output format, and a pointer to the next step. The prompt encourages variation in complexity (number of subtasks), structure (sequential or parallel), and perspective (distinct focus for each plan). Users can customize the decomposition by editing, adding, or deleting subtasks via node modifications, and reorder them by reconnecting edges in the Canvas View. - Level 2 (Agent Assignment): GPT-4o is prompted to rank a given list of design patterns for each subtask and provide explanations of its recommendation. For each subtask, the prompt combines its context (task name and description) with the full list of design patterns (names, definitions, and brief examples). Based on this information, GPT-4o heuristically evaluates the subtask’s characteristics and suggests suitable patterns, e.g., Redundant for multifaceted analysis. The combination of design patterns is randomized to mitigate the position bias of LLM [61]. Users can view the description and estimated computational cost of the suggested patterns, modify the design pattern for each subtask (e.g., change from Discussion to Reflection), and customize its corresponding parameters (e.g., number of agents). - Level 3 (Agent Optimization): GPT-4o is prompted to generate configuration details of each agent, including persona, goals, input and output format, based on the subtask’s input and output, the selected design pattern, as well as available tools and external databases. Multiple agents within one pattern are encouraged to embody distinct perspectives that align with real-world roles. Users may refine prompt content, the underlying LLM, the tool usage, and the external database of a specific agent. Full implementation details are available in the project website. 3.6 Implementation \sysName is implemented in JavaScript using React for building the user interface, D3 [6] for rendering custom data-driven visualizations, and ReactFlow [16] for managing and displaying the individual multi-agent workflows as interactive node-link diagrams. We use LangGraph [24] to store and execute the generated workflows. \sysName maintains its own internal data structure and uses a separate converter to translate this structure into compatible LangGraph files. This modular design allows \sysName to be easily adapted to other workflow frameworks by modifying the converter to support other formats. The source code, online demo, and documentation of \sysName are available at https://visual-intelligence-umn.github.io/FlowForge. 4 Case Studies To demonstrate the utility of \sysName in real-world scenarios, we present two example use cases of \sysName, developed in collaboration with LLM researchers (Section. 2.1). <details> <summary>x13.png Details</summary>  ### Visual Description ## Workflow Diagram: Script Generation Process ### Overview The image depicts a workflow diagram outlining the steps involved in generating a script, likely for a video or presentation. The workflow includes stages from message extraction to script generation, with a focus on a "Lead with Curiosity" step that involves agent interaction and task descriptions. ### Components/Axes The diagram consists of the following key components: * **A: Top Row**: High-level stages of the script generation process. * Message Extraction * Character Creation * Plot Development * Generate Script * **B: Middle Row**: Refinement and finalization stages. * Surprising Opener * Highlight Results * Memorable Close * **C: Bottom Row**: Detailing the "Lead with Curiosity" step. * Explore Research * Discuss Impact * Unified Conclusion * **D: Detailed View of "Lead with Curiosity"**: A detailed breakdown of the "Lead with Curiosity" step, including task description, agent personas, and LLM call estimation. * **E: Bottom Section**: A list of potential sections in a document or presentation, labeled "No workflow". ### Detailed Analysis or ### Content Details **A: Top Row - High-Level Stages** * **Message Extraction**: The initial step, presumably involving identifying key messages. * **Character Creation**: Developing characters for the script. * **Plot Development**: Constructing the storyline. * **Generate Script**: The final stage of producing the script. **B: Middle Row - Refinement Stages** * **Surprising Opener**: Crafting an engaging introduction. * **Highlight Results**: Showcasing key findings or outcomes. * **Memorable Close**: Creating a lasting impression. **C: Bottom Row - "Lead with Curiosity" Details** * **Explore Research**: Investigating relevant research. * **Discuss Impact**: Analyzing the implications of the research. * **Unified Conclusion**: Forming a cohesive summary. **D: Detailed View of "Lead with Curiosity"** * **Lead with Curiosity (Title)**: The central focus of this detailed view. * **Dropdown Menu**: A dropdown menu with the label "Single Agent" is visible. When clicked, it reveals the following options: * Single Agent * Reflection (RF) - Labeled with a light red color. * Discussion (D) - Labeled with a dark red color. * Redundant (RD) - Labeled with a light green color. * Supervision (S) - Labeled with a light purple color. * **Task Description**: "Open with a compelling question related to the research topic, drawing viewers in. Visual: Text animation of the question over an intriguing background. Duration: 3 seconds." * **Agent Diagram**: A diagram showing the interaction between agents. * Agent A3 connects to Agent A1. * Agent A2 connects to Agent A1. * **LLM Calls**: "2\*2 + 1 = 5 LLM Calls". A visual representation of 5 pink squares. * **Agent #1**: * Persona: Scientist * Goal: "Question is technically accurate, grounded in the core research idea." * **Agent #2**: * Persona: Writer * Goal: "Make sure the question is attention-grabbing, emotionally resonant." * **Summary**: * Persona: Summary * Goal: "Listen to the entire discussion and produce a final summary." * **Max Round**: A numerical input field with a value of "2". * **With Summary**: A dropdown menu with the label "With Summary". **E: Bottom Section - Document Sections** * A list of potential sections in a document or presentation: * [Opening] * [Introduction to the problem] * [Introduction to the system] * [Key Features of the system] * [Methodology] * [Evaluation] * [Conclusion] * [Closing] * Labeled "No workflow". ### Key Observations * The diagram highlights the importance of crafting an engaging opening ("Lead with Curiosity"). * The "Lead with Curiosity" step involves multiple agents with distinct roles (scientist, writer, summary). * The diagram estimates the number of LLM calls required for the "Lead with Curiosity" step. * The bottom section suggests a standard structure for a document or presentation. ### Interpretation The workflow diagram illustrates a structured approach to script generation, emphasizing the need for a compelling opening that draws the audience in. The "Lead with Curiosity" step is particularly detailed, suggesting its importance in the overall process. The use of multiple agents with different expertise highlights the collaborative nature of script creation. The estimation of LLM calls indicates a focus on optimizing resource usage. The "No workflow" label on the bottom section suggests that these elements are not part of the primary workflow but are related to the overall document structure. </details> Figure 5: Case Study 1: Fast-Forward Video Planning. (A-D) \sysName helps users create multiple creative ways of generating the transcript of fast-forward videos for a research paper. (E) In contrast, directly prompting GPT-4o often produces transcripts with conventional structures and exceeds the 25-second constraints. Detailed text responses are omitted due to space constraints. 4.1 Fast-Forward Video Generation for Research Papers In this use case, a user wants to create a multi-agent workflow for generating ideas for a 25-second fast-forward video that advertises a research paper. This case demonstrates \sysName ’s ability to improve diversity and creativity of workflow designs. After submitting a task description, \sysName generates five different task planning options with varying lengths. While these workflows follow a similar structure comprising a beginning, an optional middle section, and an ending, they differ in how each subtask is defined and executed. For example, some workflows begin with an intriguing question (Fig. 5.C), while others open with a surprising research finding (Fig. 5.B). One particularly creative workflow introduces a storytelling element to engage the audience (Fig. 5.A). It begins with a “message extraction” step to identify the paper’s core contribution, followed by three narrative-oriented subtasks: character creation, plot development, and script preparation. Ultimately, the user selects a workflow that incorporates a parallel structure in the middle section (Fig. 5.C). This design enables two alternative narrative paths: one that explains the research method and another that emphasizes the broader impact of the work. The workflow can then evaluate the outputs of both paths and select the more compelling version for the final fast-forward video. With the task planning finalized, the user proceeds to the agent assignment level. Most subtasks are assigned to single agents, likely because they are simple and well-defined. However, for crafting the video’s opening, where creativity is key, the user wants to encourage a more dynamic and collaborative process. To inform this decision, the user configures the scatter plot to display “Creativity” along the y-axis. As revealed in the scatter plot, certain design patterns, such as Discussion and Redundancy, are associated with higher creativity scores. Based on this, the user replaces the single-agent setup with a Discussion pattern, enabling multiple agents to brainstorm and refine the opening collaboratively (Fig. 5.D). In the final stage, the user enters the agent optimization level and fine-tunes the prompts for each agent to better control the expected outputs. The workflow is then executed within \sysName, where both the runtime and outputs are automatically recorded. Users can rate the results, iterate by modifying agent prompts, and compare different versions in the scatter plot across different dimensions. Comparison with a one-pass LLM. While a single GPT-4o prompt can generate a video script, it typically defaults to conventional structures (e.g., introduction, method, conclusion), and often fails to respect format constraints such as time limits. In our test, even with clear instructions for a 25-second limit, GPT-4o generated long-form outputs mimicking full paper summaries with five or more sections (Fig. 5.E). In contrast, \sysName empowers users to explore varied narrative structures (Fig. 5.A-C), experiment with different decompositions, and fine-tune agent behaviors, offering greater control, diversity, and creativity in workflows generation. 4.2 Data Storytelling from a Dataset In this case study, a user creates a workflow to generate visualizations and produce data-driven storytelling based on a given dataset. We use a Movies dataset [45] as the input to evaluate the outputs of different workflow configurations. This case demonstrates \sysName ’s ability to compare different workflows and their generated outputs. <details> <summary>x14.png Details</summary>  ### Visual Description ## Composite Image Analysis: Movie Industry Data ### Overview The image presents a collection of charts and diagrams exploring various aspects of the movie industry, including design space overview, factors influencing movie revenues, genre analysis, and budget vs. gross relationships. The image is divided into sections labeled A through E, each focusing on a different facet of the industry. ### Components/Axes **Section A: Design Space Overview** * **Type:** Flowchart * **Description:** A hierarchical diagram illustrating a design space with five main flows (Flow 1 to Flow 5). Each flow branches into sub-elements, represented by numbered boxes (e.g., 1-1, 1-2, 2-1, etc.). Some of these sub-elements further branch into lower-level elements (e.g., 1-1a, 2-1a, 3-1a, etc.). **Section B: User Rating vs. Topic** * **Type:** Scatter Plot Conceptual Layout * **X-axis:** User Rating (selectable dropdown) * **Y-axis:** Topic (selectable dropdown) * **Description:** A conceptual layout for a scatter plot, with axes labeled "user rating" and "topic." The plot area contains placeholders for data points, suggesting a visualization of how user ratings relate to different topics. **Section C: Framing Success: Unpacking What Influences Movie Revenues** * **Title:** Framing Success: Unpacking What Influences Movie Revenues * **Sub-sections:** * Earning in US vs. worldwide * How release year and genre influence earning * How budget and genre influence earning * **Earning in US vs. worldwide** * **Type:** Scatter Plot * **X-axis:** Earnings Type (US Gross, Worldwide Gross) * **Y-axis:** Box Office Earnings * **Scale:** 0 to 2,500,000,000 * **Description:** A scatter plot comparing US gross earnings against worldwide gross earnings. * **How release year and genre influence earning** * **Type:** Area Chart * **X-axis:** Year (1920 to 2020) * **Y-axis:** Cumulative Box Office Earnings ($) (0 to 20,000,000,000) * **Legend:** * Action (Blue) * Adventure (Orange) * Black Comedy (Gray) * Comedy (Yellow) * Concert/Performance (Green) * Documentary (Teal) * Drama (Purple) * Horror (Brown) * Musical (Pink) * Romantic Comedy (Light Blue) * Year 2009 (Red) * **Description:** An area chart showing how cumulative box office earnings vary by genre over time. * **How budget and genre influence earning** * **Type:** Heatmap * **X-axis:** Budget Range ($) (0 to 250,000,000) * **Y-axis:** Genre * **Genres:** Black Comedy, Comedy, Concert/Performance, Documentary, Drama, Horror, Musical, Romantic Comedy, Thriller/Suspense, Western * **Color Scale:** Represents profitability ($) from 0 to 2,500,000,000 (Green to Red) * **Description:** A heatmap illustrating the relationship between budget range and genre profitability. **Section D: The Genre Spectrum: Exploring the Movie Genre Landscape** * **Title:** The Genre Spectrum: Exploring the Movie Genre Landscape * **Sub-sections:** * Distribution of Movies by Genre * Trend of Most Popular Genres Over the Years * Scatter Plot of Production Budget vs. IMDB Rating by Genre * **Distribution of Movies by Genre** * **Type:** Bar Chart * **X-axis:** Genre * **Y-axis:** Number of Movies (0 to 800) * **Genres:** Action, Adventure, Black Comedy, Comedy, Concert/Performance, Documentary, Drama, Horror, Musical, Romantic Comedy, Thriller/Suspense, Western * **Description:** A bar chart showing the distribution of movies across different genres. * **Trend of Most Popular Genres Over the Years** * **Type:** Line Chart * **X-axis:** (Implied: Time/Years) * **Y-axis:** Number of Movies (0 to 70) * **Genres:** Action, Adventure, Black Comedy, Comedy, Concert/Performance, Documentary, Drama, Horror, Musical, Romantic Comedy, Thriller/Suspense, Western * **Description:** A line chart showing the trend of movie popularity for different genres over time. * **Scatter Plot of Production Budget vs. IMDB Rating by Genre** * **Type:** Scatter Plot * **X-axis:** Production Budget ($) (0 to 300,000,000) * **Y-axis:** IMDB Rating (0 to 10) * **Legend:** * null * Action * Adventure * Black Comedy * Comedy * Concert/Performance * Documentary * Drama * Horror * Comedy/Suspense * **Annotation:** Movie Title: The Godfather, Budget: 7000000, IMDB Rating: 9.2, Genre: null * **Description:** A scatter plot showing the relationship between production budget and IMDB rating for different genres. **Section E: Behind the Scene: Unpacking Movie Box Office Performance** * **Title:** Behind the Scene: Unpacking Movie Box Office Performance * **Sub-sections:** * Budget vs. US Gross * US Gross of each genre * Audience Ratings (marked as "Irrelevant") * US Gross vs. Release month * **Budget vs. US Gross** * **Type:** Scatter Plot * **X-axis:** Production Budget ($) (0 to 300,000,000) * **Y-axis:** US Gross ($) (0 to 800,000,000) * **Description:** A scatter plot showing the relationship between production budget and US gross earnings. * **US Gross of each genre** * **Type:** Bar Chart * **X-axis:** Major Genre * **Y-axis:** Average US Gross ($) (0 to 110,000,000) * **Genres:** Action, Adventure, Comedy, Musical, Romantic Comedy, Horror, Drama, Concert/Performance, Western, Black Comedy, Documentary * **Description:** A bar chart showing the average US gross earnings for different genres. * **Audience Ratings** * **Type:** Scatter Plot * **X-axis:** Rotten Tomatoes Rating (0 to 100) * **Y-axis:** IMDB Rating (0 to 10) * **Description:** A scatter plot showing the relationship between Rotten Tomatoes rating and IMDB rating. Marked as "Irrelevant". * **US Gross vs. Release month** * **Type:** Line Chart * **X-axis:** Release Month * **Y-axis:** Mean of US Gross ($) (0 to 80,000,000) * **Description:** A line chart showing the relationship between release month and mean US gross earnings. ### Detailed Analysis or ### Content Details **Section A: Design Space Overview** * The flowchart structure suggests a process or workflow with multiple decision points and paths. * The branching structure indicates different options or variations within the design space. **Section B: User Rating vs. Topic** * This section is a conceptual layout, so there are no specific data points to extract. * It suggests an intention to analyze the relationship between user ratings and different topics. **Section C: Framing Success: Unpacking What Influences Movie Revenues** * **Earning in US vs. worldwide:** * The scatter plot shows a general trend of higher worldwide gross earnings compared to US gross earnings. * Most data points are clustered near the origin, indicating that many movies have relatively low earnings. * There are a few outliers with significantly higher earnings. * **How release year and genre influence earning:** * The area chart shows that cumulative box office earnings have generally increased over time. * Certain genres, such as Action and Adventure, appear to have consistently high earnings. * The "Year 2009" line is a vertical line, indicating a specific point of reference. * **How budget and genre influence earning:** * The heatmap shows that higher budget ranges tend to be associated with higher profitability. * Certain genres, such as Comedy and Action, appear to be more profitable across different budget ranges. * Some genres, such as Western, have lower profitability, especially at higher budget ranges. **Section D: The Genre Spectrum: Exploring the Movie Genre Landscape** * **Distribution of Movies by Genre:** * The bar chart shows that Comedy has the highest number of movies, followed by Drama. * Genres such as Musical and Western have relatively few movies. * **Trend of Most Popular Genres Over the Years:** * The line chart shows that the popularity of different genres has varied over time. * Comedy and Action appear to be consistently popular genres. * Some genres, such as Horror, have experienced periods of increased popularity. * **Scatter Plot of Production Budget vs. IMDB Rating by Genre:** * The scatter plot shows a weak positive correlation between production budget and IMDB rating. * Movies with higher budgets tend to have slightly higher IMDB ratings, but there is significant variation. * The annotation highlights "The Godfather" with a budget of $7,000,000 and an IMDB rating of 9.2. **Section E: Behind the Scene: Unpacking Movie Box Office Performance** * **Budget vs. US Gross:** * The scatter plot shows a positive correlation between production budget and US gross earnings. * Movies with higher budgets tend to have higher US gross earnings, but there is significant variation. * **US Gross of each genre:** * The bar chart shows that Action and Adventure genres have the highest average US gross earnings. * Documentary and Black Comedy genres have relatively lower average US gross earnings. * **Audience Ratings:** * The scatter plot shows a positive correlation between Rotten Tomatoes rating and IMDB rating. * Movies with higher Rotten Tomatoes ratings tend to have higher IMDB ratings. * **US Gross vs. Release month:** * The line chart shows that US gross earnings vary by release month. * Some months, such as the summer months (June, July), tend to have higher US gross earnings. ### Key Observations * There is a general trend of higher worldwide gross earnings compared to US gross earnings. * Higher budget ranges tend to be associated with higher profitability. * Comedy and Action genres are consistently popular and profitable. * There is a positive correlation between production budget and US gross earnings. * US gross earnings vary by release month. ### Interpretation The data suggests that several factors influence movie success, including budget, genre, release year, and release month. Higher budgets tend to lead to higher gross earnings, and certain genres, such as Action and Comedy, are consistently popular and profitable. The timing of release also plays a role, with summer months generally yielding higher gross earnings. The relationship between audience ratings (Rotten Tomatoes and IMDB) is also positive, indicating that movies with higher critical acclaim tend to have higher audience ratings. The "Design Space Overview" suggests a complex decision-making process in movie production, while the "User Rating vs. Topic" section indicates an interest in exploring the relationship between user opinions and movie themes. </details> Figure 6: Case Study 2: Visualization Generation. (A–B) \sysName enables users to explore and compare different workflows for generating data-driven visual stories from a given dataset. (C–D) Example outputs from these workflows. (E) Directly prompting GPT-4o generates visualizations irrelevant to the data story topic. Due to space constraints, detailed text narration for each visualization is not included. As shown in Fig. 6.A, the user explores a variety of task planning and agent configurations. One specific task planning option, Flow 5, is explored more extensively, indicated by its five child nodes in the hierarchical tree view. This workflow contains only two subtasks (represented by two arcs in the glyph), which the user favors due to lower latency. At the agent assignment level, the user evaluates multiple alternatives and selects only a subset to refine further at the agent optimization level, which is reflected by the fact that only certain glyphs have child nodes in the hierarchical tree. When the workflow has fewer subtasks, the user tends to prefer configurations with more agents per step (indicated by higher bars), such as choosing Flow 1-1 over 1-2 in Fig. 6.A. Conversely, for workflows with more subtasks, the user tends to favor configurations with fewer agents per step (indicated by shorter bars), such as selecting Flow 3-1 over 3-2. To arrange and compare different workflows, the user configures the x- and y-axes of the scatter plot (Fig. 6 B). These axes can represent system-level metrics such as runtime and token usage, or custom metrics defined by the user, such as subjective ratings of output quality or the topic of the generated storytelling. As shown in Fig. 6 B, the workflow outputs vary in both topic and quality. Glyphs at Level 2 (agent assignment) appear in the bottom gray area of the scatter plot, as they lack the topic attribute required for comparison. Their horizontal positions reflect the average user rating of each workflow. Interestingly, in this experiment, neither increasing the number of steps nor adding more agents consistently led to better storytelling outcomes. Fig. 6.C–D present example outputs from two selected workflows, illustrating the variability in topic focus. Comparing with one-pass LLM. When prompting a single LLM (GPT-4o), the model often failed to strictly follow instructions: create visualizations centered on a specific topic with a coherent narrative. For example, in our experiments, GPT-4o generated a data-driven story titled “Behind the Scene: Unpacking Movie Box Office Performance”. However, the outputs included unrelated visualizations that were not aligned with the intended focus, such as a scatter plot comparing Rotten Tomatoes Rating against IMDB Rating for each movie (Fig. 6.E). In contrast, \sysName supports focused, modular task decomposition, enabling users to guide narrative flow, specify visualization goals, and experiment with structure, all of which contribute to higher coherence and alignment with user intent. 5 User Studies To evaluate how \sysName supports the design experience of creating multi-agent workflows, we conducted an observational study comparing it against LangGraph Studio [25], a widely-used state-of-the-art interface for creating multi-agent workflows. 5.1 Study Design Participants. We recruited nine participants (seven male, two female; seven aged 18–24, two aged 25–34) via email list and personal contact. All participants have prior knowledge about multi-agent LLM workflows but had no prior exposure to this project. Each study session lasted approximately one hour, and participants received a $10 Amazon gift card as compensation. The study was approved by the Institutional Review Board, and all participants signed an informed consent form. Baseline. For the baseline condition, we used LangGraph Studio (Fig. 7), a visual programming tool for building and executing multi-agent workflows. While highly expressive and widely adapted, LangGraph Studio does not provide structured design scaffolding or exploratory support, which are key features of \sysName. <details> <summary>x15.png Details</summary>  ### Visual Description ## Multi-Panel Screenshot: Code, Flowchart, and Text ### Overview The image is a multi-panel screenshot showing three distinct sections: a code snippet, a flowchart, and a block of text. The code defines functions related to checking visualizations. The flowchart illustrates a process with steps like "check_visualizations" and "check_transitions." The text presents a narrative about healthcare data analysis using "ThreadStates." ### Components/Axes **Panel 1: Code Snippet** * Function definition: `def generate_initial_scripts` * Function definition: `def check_visualizations(state):` * Docstring: `""" Node to check the visualizations """` * `message = state["message"]` * `answer_template = """Check the desciption of the visualizations of the video script. {message}"""` * `answer_instructions = answer_template.format(message=message)` * `llm = ChatOpenAI(model="gpt-40", temperature=0)` * `answer = llm.invoke([SystemMessage(content=answer_instructions)] + [HumanMessage(content=f"Check the...")` **Panel 2: Flowchart** * Title: `flow-1` * Nodes (in order): * `_start_` (Rounded rectangle, gray) * `initial_video_scripts` (Rectangle, light blue) * `check_visualizations` (Rectangle, light green) * `check_transitions` (Rectangle, light blue) * `check_conclusion` (Rectangle, light blue) * `final_check` (Rectangle, light green) * `_end_` (Rounded rectangle, gray) * Arrows: Representing flow direction from top to bottom. * Input section: * Label: `Input` * Button: `Manage Configuration` * Button: `Submit` **Panel 3: Text Narrative** * Header: `Thread` * Subheader: `TURN 5` * Narrator (Voiceover): "Welcome to the future of healthcare data analysis with ThreadStates." * [Cut to animated graphics of the Glyph Matrix and Sankey diagrams transforming complex data into clear visuals.] * Narrator (Voiceover): "ThreadStates revolutionizes how we understand disease progression by turning intricate patient data into insightful stories." * [Cut to a healthcare professional analyzing data on a tablet, nodding in understanding.] * Output section: * Label: `OUTPUT` ### Detailed Analysis or ### Content Details **Code Snippet:** The code defines two functions. The `check_visualizations` function takes a `state` as input, extracts a `message` from it, and uses it to format an `answer_template`. It then uses a `ChatOpenAI` model to generate an answer based on the formatted instructions. **Flowchart:** The flowchart illustrates a sequential process. It starts with `_start_`, proceeds through steps involving initial video scripts, checking visualizations, transitions, and conclusions, performs a final check, and ends with `_end_`. **Text Narrative:** The text describes "ThreadStates" as a tool for healthcare data analysis, specifically for understanding disease progression. It mentions the use of Glyph Matrix and Sankey diagrams for visualization. ### Key Observations * The code snippet seems to be related to the flowchart, as the `check_visualizations` function corresponds to a step in the flowchart. * The text narrative promotes "ThreadStates" as a revolutionary tool for healthcare data analysis. * The flowchart provides a high-level overview of a process, while the code snippet provides a more detailed implementation of one of the steps. ### Interpretation The image presents a cohesive narrative about using AI and data visualization techniques (ThreadStates, Glyph Matrix, Sankey diagrams) for healthcare data analysis. The code snippet shows a function that likely automates part of the process described in the flowchart. The text narrative positions "ThreadStates" as a key technology for understanding disease progression. The combination of code, flowchart, and narrative suggests a system designed to automate and enhance the analysis of healthcare data. </details> Figure 7: User Study Baseline. We use LangGraph Studio as the baseline system in the user study. <details> <summary>x16.png Details</summary>  ### Visual Description ## Diverging Stacked Bar Chart: FlowForge vs. Baseline User Survey ### Overview The image is a diverging stacked bar chart comparing user survey results for FlowForge and a Baseline across five different categories: Ease of Use, Ease of Learning, Confidence in Using, Usefulness for Design Exploration, and Usefulness for Workflow Creation. The chart visualizes the distribution of user opinions, ranging from "Strongly Disagree" to "Strongly Agree," and includes average scores for each category and system. ### Components/Axes * **Y-Axis (Categories):** * Easy to use * Easy to learn * Confident in using * Useful for design exploration * Useful for workflow creation * **X-Axis (Percentage):** Ranges from 0% to 100% in increments of 20%. * **Legend (Bottom):** * Strongly Disagree (1): Dark Red * Red * Light Red * White * Light Blue * Blue * Strongly Agree (7): Dark Blue * **Data Labels:** Each category has two labels: "FlowForge (score)" and "Baseline (score)". The scores are numerical values representing the average user rating on a scale of 1 to 7. ### Detailed Analysis **1. Easy to Use:** * FlowForge (5.78): The bar is primarily composed of light blue and blue segments, indicating a generally positive sentiment. * Baseline (4.56): The bar is composed of red and light red segments, indicating a less positive sentiment compared to FlowForge. **2. Easy to Learn:** * FlowForge (4.78): The bar is composed of light red, white, light blue and blue segments, indicating a mixed sentiment. * Baseline (3.78): The bar is primarily composed of red and light red segments, indicating a generally negative sentiment. **3. Confident in Using:** * FlowForge (5.78): The bar is primarily composed of light blue and blue segments, indicating a generally positive sentiment. * Baseline (4.78): The bar is composed of red, light red, white, and light blue segments, indicating a mixed sentiment. **4. Useful for Design Exploration:** * FlowForge (6.22): The bar is primarily composed of blue and dark blue segments, indicating a very positive sentiment. * Baseline (3.56): The bar is primarily composed of dark red, red, and light red segments, indicating a generally negative sentiment. **5. Useful for Workflow Creation:** * FlowForge (6.11): The bar is primarily composed of light blue, blue, and dark blue segments, indicating a very positive sentiment. * Baseline (5.44): The bar is composed of white, light blue, blue, and dark blue segments, indicating a generally positive sentiment. ### Key Observations * FlowForge consistently receives higher average scores than the Baseline across all categories. * The largest difference in scores is in the "Useful for design exploration" category, where FlowForge scores significantly higher than the Baseline. * The distribution of responses for FlowForge tends to lean towards "Strongly Agree," while the distribution for the Baseline is more varied, with a tendency towards "Strongly Disagree" in some categories. ### Interpretation The data suggests that users generally perceive FlowForge more favorably than the Baseline in terms of ease of use, ease of learning, confidence in using, usefulness for design exploration, and usefulness for workflow creation. The most significant advantage of FlowForge appears to be its usefulness for design exploration. The diverging stacked bar chart effectively visualizes the distribution of user opinions, highlighting the differences in sentiment between FlowForge and the Baseline. The average scores provide a quantitative measure of these differences, reinforcing the qualitative insights gained from the visual representation. </details> Figure 8: User Ratings. Comparison of \sysName and the baseline system based on post-study questionnaire responses. Participants rated each item on a 7-point Likert scale ranging from strongly disagree (1) to strongly agree (7). Numbers in parentheses indicate the average rating. Task & Study Procedure. In both conditions (\sysName vs. baseline), participants created workflows for the same task: generating scripts for a 25-second fast-forward video that introduces a research paper. Prior to the study, participants received comprehensive orientation including task explanation, example papers for workflow evaluation, and detailed prompting specifications describing the desired outputs. During the study, participants completed both conditions in randomized order to control for potential learning effects. Targeted tutorials and demonstrations for the corresponding system are provided before beginning each condition. In both conditions, participants constructed workflows from scratch without any default templates, with explicit encouragement to create and compare multiple workflows. The session of a condition ended when participants expressed satisfaction with their results or upon reaching the 25-minute time limit, whichever occurred first. Participants were instructed to think aloud throughout the sessions and can ask questions freely at any time. After two conditions, participants completed a usability questionnaire and a post-study interview. Data Analysis. We analyzed (i) the workflows created by participants, including the time participants spent creating them and their overall quality; (ii) self-reported usability ratings for both systems; and (iii) participants’ think-aloud protocols, interactions during the study, and qualitative feedback from post-study interviews. 5.2 Quantitative Results Our quantitative evaluations demonstrated that \sysName outperformed the baseline system, delivering improvements in both user experience and workflow creation efficiency. First, \sysName led to improved user experience in creating LLM workflows. Fig. 8 presents participants’ subjective ratings comparing \sysName and the baseline across five dimensions related to usability and utility. Overall, \sysName was rated more favorably. Participants found it easier to use, with a higher proportion agreeing or strongly agreeing that they would not require technical support to operate the system. They also reported greater confidence when using \sysName, as well as stronger agreement on its usefulness for both exploring alternative workflows and executing them. <details> <summary>x17.png Details</summary>  ### Visual Description ## Chart/Diagram Type: Comparative Bar Charts ### Overview The image presents a comparative analysis of task completion times between two conditions: "Baseline" and "FlowForge." It consists of two main sections: (A) a summary bar chart showing average completion times with confidence intervals, and (B) detailed bar charts illustrating the time spent by individual participants (P1-P9) on different stages of the task under each condition. ### Components/Axes **Section A: Summary Bar Chart** * **Title:** "Baseline" and "FlowForge" (labels for the two bars) * **Y-axis:** Implied, representing the two conditions being compared. * **X-axis:** "Time (minutes)" ranging from 0 to 25, with tick marks at intervals of 5. * **Data:** * Baseline: A horizontal bar representing the average completion time. The bar extends to approximately 18 minutes. Error bars indicate a 95% confidence interval. The text next to the bar reads "18.00 ± 4.60 (95 CI)". * FlowForge: A horizontal bar representing the average completion time. The bar extends to approximately 9.89 minutes. Error bars indicate a 95% confidence interval. The text next to the bar reads "9.89 ± 3.93 (95 CI)". **Section B: Detailed Bar Charts** * **Titles:** "Baseline" and "FlowForge" (labels for the two charts) * **Y-axis:** Participants labeled P1 to P9. * **X-axis:** "Time (minutes)" ranging from 0 to 25, with tick marks at intervals of 5. * **Data:** Each participant has a horizontal bar divided into segments representing different stages of the task. * **Colors:** * 1st: Light Blue * 2nd: Light Green * 3rd: Light Orange * 4th: Light Purple * Incomplete: Blue with diagonal stripes * Time Left: White ### Detailed Analysis or Content Details **Section A: Summary Bar Chart** * Baseline: Average completion time is 18.00 minutes with a 95% confidence interval of ±4.60 minutes. * FlowForge: Average completion time is 9.89 minutes with a 95% confidence interval of ±3.93 minutes. **Section B: Detailed Bar Charts** * **Baseline:** * P1: Completed the 1st stage in approximately 24 minutes, marked as incomplete. * P2: Did not complete the task, marked as incomplete. * P3: Completed the 1st stage in approximately 14 minutes, with the remaining time marked as "time left". * P4: Completed the 1st stage in approximately 18 minutes, with the remaining time marked as "time left". * P5: Did not complete the task, marked as incomplete. * P6: Completed the 1st stage in approximately 20 minutes, with the remaining time marked as "time left". * P7: Completed the 1st stage in approximately 10 minutes and the 2nd stage in approximately 3 minutes, with the remaining time marked as "time left". * P8: Completed the 1st stage in approximately 24 minutes, with the remaining time marked as "time left". * P9: Completed the 1st stage in approximately 12 minutes, with the remaining time marked as "time left". * **FlowForge:** * P1: Completed the 1st stage in approximately 18 minutes, the 3rd stage in approximately 3 minutes, and the 4th stage in approximately 3 minutes, with the remaining time marked as "time left". * P2: Completed the 1st stage in approximately 18 minutes, with the remaining time marked as "time left". * P3: Completed the 1st stage in approximately 12 minutes and the 3rd stage in approximately 3 minutes, with the remaining time marked as "time left". * P4: Completed the 2nd stage in approximately 12 minutes, with the remaining time marked as "time left". * P5: Completed the 1st stage in approximately 12 minutes and the 3rd stage in approximately 3 minutes, with the remaining time marked as "time left". * P6: Completed the 1st stage in approximately 10 minutes and the 2nd stage in approximately 8 minutes, with the remaining time marked as "time left". * P7: Completed the 2nd stage in approximately 10 minutes and the 3rd stage in approximately 8 minutes, with the remaining time marked as "time left". * P8: Completed the 1st stage in approximately 10 minutes and the 2nd stage in approximately 8 minutes, with the remaining time marked as "time left". * P9: Completed the 1st stage in approximately 5 minutes and the 2nd stage in approximately 5 minutes and the 3rd stage in approximately 5 minutes, with the remaining time marked as "time left". ### Key Observations * The average completion time for FlowForge is significantly lower than Baseline. * Under the Baseline condition, some participants did not complete the task. * Under the FlowForge condition, all participants completed at least the first stage, and several completed multiple stages. ### Interpretation The data strongly suggests that FlowForge significantly improves task completion time compared to the Baseline condition. The summary bar chart clearly shows a lower average completion time for FlowForge, and the detailed bar charts reveal that FlowForge enables participants to complete more stages of the task within the same time frame. The "incomplete" markings in the Baseline condition further highlight the advantage of FlowForge in facilitating task completion. </details> Figure 9: Workflow Creation Time and Quantity. (A) Users spent less time creating their first runnable workflow in \sysName compared to the baseline. Error bars indicate 95% confidence intervals (CI). (B) Users created more workflows in \sysName compared with the baseline. Second, \sysName enabled more efficient workflow creation with more diverse designs. Fig. 9 presents participants’ workflow creation time. Participants spent on average 9.89 (95% CI: 5.96-13.82) minutes to create and run the first workflow in \sysName, and 18.0 (95% CI: 13.40-22.60) minutes in the baseline. Student’s t-test showed a significant difference between their completion time for the first workflow (t =-8.1, p =.003). Most (8/9) of the participants were able to create and explore more than one workflow in \sysName after the initial try, while some participants (2) failed to create a workflow in the baseline. For participants who did not finish, 25 minutes (maximum allowed time per session) was used as their estimated completion time. To evaluate workflow diversity, we conducted a manual analysis of workflows created by participants across both conditions. While workflows created in both conditions contained a similar number of subtasks (baseline: 3.43, \sysName: 3.89), \sysName produced more parallel subtasks (3) than the baseline (1). Participants also created more diverse agent communications using \sysName. Using \sysName, participants implemented a variety of agent communication patterns (7 reflection, 5 supervision, 7 redundancy, 8 discussion, out of 35 total instances). In contrast, baseline workflows exclusively employed single-agent across all 22 instances. 5.3 Observations and Feedback Based on the think-aloud comments and interviews, we reported how participants approached workflow creation across the two conditions. Across both conditions, participants prioritized creating an initial functional workflow before exploring alternatives. Participants appreciated the guidance provided at each level. They noted that while Levels 2 and 3 of the design guidance were more “technically oriented”, Level 1 introduced innovative concepts that were accessible and valuable even without specialized AI knowledge. Participants demonstrated iterative workflow design behaviors in both conditions, yet their approaches differed substantially. In the baseline condition, participants typically constructed workflows with only one or two sequential subtasks. When refining these workflows, they focused exclusively on prompt refinement for individual agents while maintaining the original workflow structure unchanged. When using \sysName, participants exhibited broader explorations. Most participants (8/9) explored different task planning (level 1) to generate alternative workflows and compare workflows based on key dimensions such as output quality and latency. Two participants made strategic modifications when encountering performance issues. Informed by the design cards at level 3, P5 improved output quality by incorporating detailed domain expertise into agent personas. Through analysis of latency of different design patterns in the Design Space View, P9 transitioned from the Discussion pattern to Redundant and restructured task planning to enable parallel subtask execution, successfully reducing execution time while preserving output quality. 6 Discussion 6.1 Design Implications. We reflect on the design implications of \sysName by connecting it with previous visualization tools developed for AI research. Design Space Visualization. This work presents an initial exploration of using design space visualization to support the navigation and construction of complex multi-agent workflows. Although visualization of the design space has been shown as an effective approach in AI applications such as deep neural network design [50], automated machine learning [49], and neural architecture search [56], it has not yet been applied to the emerging context of multi-agent LLM workflows. This new context presents unique opportunities. Compared to traditional AI development, which often requires deep technical expertise to understand the design space, LLM workflows significantly lower the barrier to entry, allowing a broader range of end users to design AI-driven systems. As this shift moves design tools from expert-centric to end-user-facing contexts, visualization must evolve to offer greater clarity and interpretability. Thinking Scaffold for both LLMs and humans. While \sysName is designed for building multi-agent workflows, we observed that human users also benefit from the thinking scaffold it provides, particularly in open-ended and creative tasks (e.g., creatively crafting fast-forward videos using character and plot elements in Section 4.1). By explicitly visualizing workflows, users engage in cognitive externalization that aligns with extended mind theory, where cognitive processes are distributed across both internal mental representations and external artifacts, making it easier to brainstorm effectively with LLMs. This approach represents a different perspective from traditional visualization for AI interpretability, which has focused on visualizing low-level computational processes (e.g., neuron activations) [23, 30, 31]. The emerging of LLMs has enabled new interpretability opportunities by exporting reasoning in workflows and agent interactions [54]. Interactive visualizations of these reasoning processes can thus serve as accessible collaborative interfaces moving beyond expert-only tools. 6.2 Limitations and Future Directions. First, \sysName does not cover the full spectrum of workflow designs and application scenarios. Regarding workflow architecture, \sysName focuses on workflows with static or conditional communication protocols but does not support dynamic architecture modifications during execution. For example, the system cannot support workflows that undergo real-time architecture revision based on human-in-the-loop interventions [47] or agent feedback [22]. Regarding application scenarios, \sysName is specifically designed for structured problem-solving tasks rather than world simulation applications [51, 19], which may require fundamentally different architectural considerations and interaction patterns. Second, in \sysName, we used straightforward prompting techniques for task decomposition and design pattern recommendation, as the primary focus of this study is to demonstrate the benefits of visualizing and interacting with the design space during workflow exploration. We acknowledge that the quality of generated workflows would be further improved with more advanced methods, such as using reinforcement learning to optimize the workflow design. Finally, in spite of the intuitiveness and effectiveness, the three-level abstraction of design space is not the only viable scaffold and can be further enhanced. For example, Liu et al.. [57] proposed a decision tree that covers key decisions for designing multi-agent LLM systems. Compared to these alternatives, our method aims to strike a balance between expressiveness (\ie, enabling diverse workflow constructions) and conciseness (\ie, avoiding excessive complexity). There are several promising directions for further enhancing this abstraction. Each level may benefit from further decomposition into sublevels that capture more nuanced design choices. For example, the task planning level may involve sub-levels such as failure handling mechanism. Meanwhile, additional support can be provided to capture the inter-level dependencies and facilitate design iterations across abstraction levels [14], such as revisiting task planning decisions based on agent optimization. 7 Conclusion This study presents \sysName, an interactive visualization tool that supports the creation of multi-agent LLM workflows by organizing the levels and dimensions of the design space, visualizing the design space and the various solutions within it, and integrating in-situ guidance from well-established design patterns in design exploration. Our evaluation demonstrated that \sysName improved user experience, increased workflow creation efficiency, and enhanced the diversity of created workflows. This work highlights the potential of design space visualization as a thinking scaffold for both LLMs and human designers in complex LLM system development. References - [1] Cellum. Purpose-Built GUI and SDK for AI Development. https://www.vellum.ai/. - [2] CrewAI: Build AI agent teams that work together to tackle complex tasks. https://docs.crewai.com/introduction. - [3] Rivet. The Open-Source Visual AI Programming Environment. https://rivet.ironcladapp.com/. - [4] Amazon. Multi-Agent Orchestrator framework. https://awslabs.github.io/multi-agent-orchestrator/. - [5] Anthropic. Building Effective Agents. https://www.anthropic.com/research/building-effective-agents, 2024. - [6] M. Bostock, V. Ogievetsky, and J. Heer. D3 Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics, 17(12):2301–2309, 2011. doi: 10 . 1109/TVCG . 2011 . 185 - [7] S. Brade, B. Wang, M. Sousa, S. Oore, and T. Grossman. Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, article no. 96, 14 pages. ACM, New York, 2023. doi: 10 . 1145/3586183 . 3606725 - [8] B. Buxton. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann, San Francisco, 2010. doi: 10 . 1016/B978-0-12-374037-3 . X5043-3 - [9] M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica. Why Do Multi-Agent LLM Systems Fail?, 2025. doi: 10 . 48550/arXiv . 2503 . 13657 - [10] C.-M. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate, 2023. doi: 10 . 48550/arXiv . 2308 . 07201 - [11] V. Dibia, J. Chen, G. Bansal, S. Syed, A. Fourney, E. Zhu, C. Wang, and S. Amershi. AUTOGEN STUDIO: A No-Code Developer Tool for Building and Debugging Multi-Agent Systems. In D. I. Hernandez Farias, T. Hope, and M. Li, eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 72–79. Association for Computational Linguistics, Miami, 2024. doi: 10 . 18653/v1/2024 . emnlp-demo . 8 - [12] Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, article no. 467, 31 pages. JMLR.org, 2024. doi: 10 . 5555/3692070 . 3692537 - [13] B. Effect. Manus AI: A General AI Agent. https://manus.im/. Accessed: 2025-03-18. - [14] W. Epperson, G. Bansal, V. C. Dibia, A. Fourney, J. Gerrits, E. Zhu, and S. Amershi. Interactive debugging and steering of multi-agent ai systems. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 1–15, 2025. - [15] Y. Feng, X. Wang, K. K. Wong, S. Wang, Y. Lu, M. Zhu, B. Wang, and W. Chen. PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation . IEEE Transactions on Visualization and Computer Graphics, 30(01):295–305, 2024. doi: 10 . 1109/TVCG . 2023 . 3327168 - [16] W. GmbH. React Flow: A library for building node-based UIs. https://reactflow.dev/, 2024. Accessed: 2025-03-26. - [17] J. Gottweis, W.-H. Weng, A. Daryin, T. Tu, A. Palepu, P. Sirkovic, A. Myaskovsky, F. Weissenberger, K. Rong, R. Tanno, et al. Towards an AI co-scientist, 2025. doi: 10 . 48550/arXiv . 2502 . 18864 - [18] M. Grunde-McLaughlin, M. S. Lam, R. Krishna, D. S. Weld, and J. Heer. Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows. ACM Transactions on Computer-Human Interaction, 32(3), article no. 27, 57 pages, 2025. doi: 10 . 1145/3716134 - [19] T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang. Large Language Model based Multi-Agents: A Survey of Progress and Challenges, 2024. doi: 10 . 48550/arXiv . 2402 . 01680 - [20] M. Heidegger. The Question Concerning Technology, and Other Essays. Harper & Row, New York, 1977. - [21] S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In The Twelfth International Conference on Learning Representations, 2024. doi: 10 . 48550/arXiv . 2308 . 00352 - [22] S. Hu, C. Lu, and J. Clune. Automated Design of Agentic Systems, 2024. doi: 10 . 48550/arXiv . 2408 . 08435 - [23] Z. Jin, Y. Wang, Q. Wang, Y. Ming, T. Ma, and H. Qu. GNNLens: A visual analytics approach for prediction error diagnosis of graph neural networks. IEEE Transactions on Visualization and Computer Graphics, 29(6):3024–3038, 2022. doi: 10 . 1109/TVCG . 2022 . 3148107 - [24] LangChain. LangGraph. Building language agents as graphs. - [25] LangChain. LangGraph Studio: The first agent IDE. https://blog.langchain.dev/langgraph-studio-the-first-agent-ide/. - [26] LangChain. Multi-agent Systems. https://langchain-ai.github.io/langgraph/concepts/multi_agent/. - [27] LangChain. Multi-agent Workflows. https://langchain-ai.github.io/langgraph/tutorials/workflows/. - [28] G. Li, H. A. Al Kader Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, article no. 2264, 18 pages. Curran Associates Inc., Red Hook, 2023. doi: 10 . 5555/3666122 . 3668386 - [29] T. Liang, Z. He, W. Jiao, X. Wang, Y. Wang, R. Wang, Y. Yang, S. Shi, and Z. Tu. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17889–17904. Association for Computational Linguistics, Miami, Florida, USA, 2024. doi: 10 . 18653/v1/2024 . emnlp-main . 992 - [30] M. Liu, J. Shi, Z. Li, C. Li, J. Zhu, and S. Liu. Towards better analysis of deep convolutional neural networks. IEEE Transactions on Visualization and Computer Graphics, 23(1):91–100, 2016. doi: 10 . 1109/TVCG . 2016 . 2598831 - [31] Y. Lu, C. Chen, Y. Chen, K. Huang, M. Zitnik, and Q. Wang. Gnn 101: Visual learning of graph neural networks in your web browser. arXiv preprint arXiv:2411.17849, 2024. doi: 10 . 48550/arXiv . 2411 . 17849 - [32] Microsoft AutoGen. Conversation Patterns. https://microsoft.github.io/autogen/0.2/docs/tutorial/conversation-patterns/. - [33] Microsoft AutoGen. Multi-Agent Design Patteamrns. https://microsoft.github.io/autogen/stable/user-guide/core-user-guide/design-patterns/intro.html. - [34] T. Munzner. A nested model for visualization design and validation. IEEE Transactions on Visualization and Computer Graphics, 15(6):921–928, 2009. - [35] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. WebGPT: Browser-assisted question-answering with human feedback, 2022. doi: 10 . 48550/arXiv . 2112 . 09332 - [36] B. Niu, Y. Song, K. Lian, Y. Shen, Y. Yao, K. Zhang, and T. Liu. Flow: Modularized Agentic Workflow Automation, 2025. doi: 10 . 48550/arXiv . 2501 . 07834 - [37] OpenAI. GPT-4o System Card, 2024. doi: 10 . 48550/arXiv . 2410 . 21276 - [38] J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, article no. 2, 22 pages. ACM, New York, 2023. doi: 10 . 1145/3586183 . 3606763 - [39] J. R. Penadés, J. Gottweis, L. He, J. B. Patkowski, A. Shurick, W.-H. Weng, T. Tu, A. Palepu, A. Myaskovsky, A. Pawlosky, N. Vivek, K. Alan, and T. R. D. Costa. AI mirrors experimental science to uncover a novel mechanism of gene transfer crucial to bacterial evolution. bioRxiv, pp. 2025–02, 2025. doi: 10 . 1101/2025 . 02 . 19 . 639094 - [40] A. Prasad, A. Koller, M. Hartmann, P. Clark, A. Sabharwal, M. Bansal, and T. Khot. ADaPT: As-Needed Decomposition and Planning with Language Models. In K. Duh, H. Gomez, and S. Bethard, eds., Findings of the Association for Computational Linguistics: NAACL 2024, pp. 4226–4252. Association for Computational Linguistics, Mexico City, 2024. doi: 10 . 18653/v1/2024 . findings-naacl . 264 - [41] C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun. ChatDev: Communicative Agents for Software Development. In L.-W. Ku, A. Martins, and V. Srikumar, eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15174–15186. Association for Computational Linguistics, Bangkok, Thailand, 2024. doi: 10 . 18653/v1/2024 . acl-long . 810 - [42] N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language Agents with Verbal Reinforcement Learning, 2023. doi: 10 . 48550/arXiv . 2303 . 11366 - [43] S. Suh, M. Chen, B. Min, T. J.-J. Li, and H. Xia. Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, article no. 644, 26 pages. ACM, New York, 2024. doi: 10 . 1145/3613904 . 3642400 - [44] S. Suh, B. Min, S. Palani, and H. Xia. Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, article no. 1, 18 pages. ACM, New York, 2023. doi: 10 . 1145/3586183 . 3606756 - [45] Vega-Lite. Movies dataset. https://vega.github.io/vega-lite/data/movies.json, 2017. - [46] C. Wang, Q. Wu, and the AG2 Community. AG2: Open-Source AgentOS for AI Agents. https://docs.ag2.ai/, 2024. Available at https://docs.ag2.ai/. - [47] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J. Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), 2024. doi: 10 . 1007/s11704-024-40231-1 - [48] Q. Wang, K. Huang, P. Chandak, M. Zitnik, and N. Gehlenborg. Extending the nested model for user-centric xai: A design study on gnn-based drug repurposing. IEEE Transactions on Visualization and Computer Graphics, 29(1):1266–1276, 2022. doi: 10 . 1109/TVCG . 2022 . 3209435 - [49] Q. Wang, Y. Ming, Z. Jin, Q. Shen, D. Liu, M. J. Smith, K. Veeramachaneni, and H. Qu. ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, 12 pages, p. 1–12. ACM, New York, 2019. doi: 10 . 1145/3290605 . 3300911 - [50] Q. Wang, J. Yuan, S. Chen, H. Su, H. Qu, and S. Liu. Visual Genealogy of Deep Neural Networks. IEEE Transactions on Visualization and Computer Graphics, 26(11):3340–3352, 2020. doi: 10 . 1109/TVCG . 2019 . 2921323 - [51] Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, Y. Liang, and T. CraftJarvis. Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, article no. 1480, 37 pages. Curran Associates Inc., Red Hook, 2023. - [52] Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. AutoGen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023. doi: 10 . 48550/arXiv . 2308 . 08155 - [53] T. Wu, E. Jiang, A. Donsbach, J. Gray, A. Molina, M. Terry, and C. J. Cai. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA ’22, article no. 359, 10 pages. ACM, New York, 2022. doi: 10 . 1145/3491101 . 3519729 - [54] T. Wu, M. Terry, and C. J. Cai. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, article no. 385, 22 pages. ACM, New York, 2022. doi: 10 . 1145/3491102 . 3517582 - [55] T. Wu, H. Zhu, M. Albayrak, A. Axon, A. Bertsch, W. Deng, Z. Ding, B. Guo, S. Gururaja, T.-S. Kuo, J. T. Liang, R. Liu, I. Mandal, J. Milbauer, X. Ni, N. Padmanabhan, S. Ramkumar, A. Sudjianto, J. Taylor, Y.-J. Tseng, P. Vaidos, Z. Wu, W. Wu, and C. Yang. LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA ’25, article no. 684, 10 pages. ACM, New York, 2025. doi: 10 . 1145/3706599 . 3706690 - [56] J. Yuan, M. Liu, F. Tian, and S. Liu. Visual Analysis of Neural Architecture Spaces for Summarizing Design Principles. IEEE Transactions on Visualization and Computer Graphics, 29(1):288–298, 2023. doi: 10 . 1109/TVCG . 2022 . 3209404 - [57] L. Yue, L. Sin Kit, L. Qinghua, Z. Liming, Z. Dehai, X. Xiwei, H. Stefan, and W. Jon. Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agents. arXiv preprint arXiv: 2405.10467, 2024. doi: 10 . 48550/arXiv . 2405 . 10467 - [58] J. Zhang and I. Arawjo. ChainBuddy: An AI-assisted Agent System for Helping Users Set up LLM Pipelines. In Adjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, UIST Adjunct ’24, article no. 48, 3 pages. ACM, New York, 2024. doi: 10 . 1145/3672539 . 3686763 - [59] J. Zhang, J. Xiang, Z. Yu, F. Teng, X.-H. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y. Luo, and C. Wu. AFlow: Automating Agentic Workflow Generation. In The Thirteenth International Conference on Learning Representations, 2025. doi: 10 . 48550/arXiv . 2410 . 10762 - [60] S. Zhang, J. Zhang, J. Liu, L. Song, C. Wang, R. Krishna, and Q. Wu. Offline Training of Language Model Agents with Functions as Learnable Weights. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, article no. 2496, 21 pages. JMLR.org, 2024. - [61] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging LLM-as-a-Judge with MT-Bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. - [62] H. Zhou, X. Wan, R. Sun, H. Palangi, S. Iqbal, I. Vulić, A. Korhonen, and S. Ö. Arık. Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies. arXiv preprint arXiv:2502.02533, 2025. doi: 10 . 48550/arXiv . 2502 . 02533 - [63] M. Zhuge, W. Wang, L. Kirsch, F. Faccio, D. Khizbullin, and J. Schmidhuber. GPTSwarm: Language Agents as Optimizable Graphs. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, eds., Proceedings of the 41st International Conference on Machine Learning, vol. 235 of Proceedings of Machine Learning Research, pp. 62743–62767. PMLR, 2024. doi: 10 . 48550/arXiv . 2402 . 16823