2602.00307v1

# Autonomous Data Processing using Meta-Agents **Authors**: Udayan Khurana > ukhurana@us.ibm.comIBM TJ Watson Research CenterYorktown HeightsNYUSA Abstract. Traditional data processing pipelines are typically static and handcrafted for specific tasks, limiting their adaptability to evolving requirements. While general-purpose agents and coding assistants can generate code for well-understood data pipelines, they lack the ability to autonomously monitor, manage, and optimize an end-to-end pipeline once deployed. We present Autonomous Data Processing using Meta-agents (ADP-MA), a framework that dynamically constructs, executes, and iteratively refines data processing pipelines through hierarchical agent orchestration. At its core, meta-agents analyze input data and task specifications to design a multi-phase plan, instantiate specialized ground-level agents, and continuously evaluate pipeline performance. The architecture comprises three key components: a planning module for strategy generation, an orchestration layer for agent coordination and tool integration, and a monitoring loop for iterative evaluation and backtracking. Unlike conventional approaches, ADP-MA emphasizes context-aware optimization, adaptive workload partitioning, and progressive sampling for scalability. Additionally, the framework leverages a diverse set of external tools and can reuse previously designed agents, reducing redundancy and accelerating pipeline construction. We demonstrate ADP-MA through an interactive demo that showcases pipeline construction, execution monitoring, and adaptive refinement across representative data processing tasks. autonomous data processing, multi-agent systems, large language models, pipeline orchestration, meta-agents ccs: Information systems Data management systems ccs: Computing methodologies Multi-agent systems 1. Introduction Data processing pipelines form the backbone of modern data-driven applications, transforming raw data into actionable insights. Traditional approaches to constructing these pipelines rely heavily on manual engineering, where domain experts design static workflows tailored to specific use cases. While effective for well-defined scenarios, this paradigm suffers from several critical limitations: inflexibility to changing data characteristics, inability to handle data drift, high development overhead for new tasks, and lack of autonomous error recovery. Recent advances in large language models (LLMs) and agent-based systems have opened new possibilities for automating data science workflows (Chen and others, 2025; Li et al., 2024b). However, existing solutions primarily focus on single-point assistance or predefined pipeline templates, requiring substantial human intervention to orchestrate complex, multi-stage data processing tasks. General-purpose coding assistants can generate pipeline code but cannot autonomously monitor execution, adapt to runtime failures, or optimize performance based on observed data characteristics. We introduce ADP-MA (Autonomous Data Processing using Meta-agents), a framework that addresses these limitations through hierarchical agent orchestration. ADP-MA distinguishes itself through three fundamental innovations. First, we employ meta-agents that reason about the overall data processing strategy, decomposing complex tasks into manageable phases while maintaining global coherence. Second, we dynamically instantiate and coordinate ground-level agents that execute specialized operations, with each agent equipped with domain-specific tools and capabilities. Third, we implement continuous monitoring and adaptive refinement mechanisms that enable the system to detect anomalies, handle data drift, and backtrack when necessary. The meta-agent architecture operates at a higher level of abstraction than traditional agents. Rather than directly executing data operations, meta-agents perform strategic planning, agent creation and modification, and system-level monitoring. This separation of concerns enables more sophisticated reasoning about workflow design while delegating execution details to specialized ground-level agents. The framework maintains a library of reusable agent definitions and tools, accelerating development for similar tasks and promoting consistency across pipelines. We incorporate four key design principles that differentiate ADP-MA from existing work. Hierarchical planning breaks large tasks into phases of reasonable size and logical consistency, optimizing context usage while avoiding issues like context rot (Anthropic, 2025). Iterative refinement employs multiple rounds of planning, critiquing, and revision to reduce errors before code generation. Progressive sampling enables the system to test pipelines incrementally, revealing challenges at each stage and facilitating scalability. Intelligent backtracking provides mechanisms to reconsider higher-level decisions when ground-level execution encounters intractable failures. ADP-MA is built with the Google Agent Development Kit (ADK) (Google, 2025) and implements its own process-level sandboxing for safe code execution. However, the ideas and design, including agentic definitions, are independent of the choice of platform. The contributions of this work include: (1) a hierarchical meta-agent architecture for autonomous data pipeline construction and management, (2) mechanisms for dynamic agent instantiation, tool integration, and adaptive workflow refinement, (3) strategies for context-aware optimization, progressive sampling, and intelligent backtracking, and (4) an interactive demonstration system that showcases these capabilities across representative data processing scenarios. 2. Related Work The landscape of autonomous data processing has evolved rapidly with the advent of LLM-based agents. We organize related work into several key categories that contextualize our contributions. 2.1. Multi-Agent Systems for Data Science AutoKaggle (Li et al., 2024b) represents a significant advancement in autonomous data science, employing a phase-based workflow with five specialized agents (Reader, Planner, Developer, Reviewer, and Summarizer) to complete end-to-end data science competitions. The framework divides tasks into six phases including background understanding, exploratory data analysis, data cleaning, feature engineering, and model building, achieving a validation submission rate of 0.85 and comprehensive score of 0.82 across eight Kaggle competitions. While AutoKaggle demonstrates the effectiveness of multi-agent collaboration for predictive modeling, it relies on predefined phase sequences and focuses primarily on machine learning tasks. ADP-MA extends this paradigm by dynamically determining workflow phases based on task analysis and supporting broader classes of data processing beyond predictive modeling. Recent surveys on data agents (Chen and others, 2025) have proposed hierarchical taxonomies for classifying agent autonomy levels, ranging from L0 (manual operation) to L5 (fully autonomous systems). Most existing work clusters at L1-L3 levels, providing assistance for isolated tasks or executing predefined procedures. ADP-MA advances toward L4 autonomy by enabling self-orchestration of multi-stage pipelines with minimal human intervention, though full L5 autonomy with generative capabilities across arbitrary data tasks remains aspirational. 2.2. Automated Data Cleaning and Preparation Data cleaning represents a particularly labor-intensive aspect of pipeline development. AutoDCWorkflow (Li et al., 2024a) introduces an LLM-based pipeline for automatically generating data cleaning workflows using OpenRefine operations. The system employs three iterative components—selecting target columns, inspecting column quality, and generating operations with arguments—to address quality issues including duplicates, missing values, and inconsistent formats. Experiments demonstrate that models like Llama 3.1 and Gemma 2 can generate high-quality cleaning workflows that effectively prepare datasets for downstream analysis. AutoClean (Shen et al., 2025) takes a complementary approach by configuring LLMs as agent teams that mimic human data-cleaning processes, automatically generating cleaning rules from limited samples that can then be applied broadly to large corpora. This work demonstrates the feasibility of using LLMs to prepare training data at scale, including Common Crawl and specific target websites. Research on the interaction between data cleaning and automated machine learning (Neutatz et al., 2022) reveals that cleaning decisions should be considered jointly with other pipeline hyperparameters. Systems like AutoClean extend AutoSklearn with sophisticated outlier detection and context-dependent imputation strategies, showing that cleaning preprocessors can be optimized alongside feature selection and model choice. ADP-MA builds on these insights by treating data preparation as an integral, adaptively configured phase within the broader pipeline rather than a standalone preprocessing step. 2.3. Benchmarks for Data Processing Automation The evaluation of autonomous data processing systems requires comprehensive benchmarks that capture diverse task types, data modalities, and quality dimensions. We survey the principal benchmarks relevant to ADP-MA and describe how they inform our evaluation strategy. KRAMABENCH (Ludäscher et al., 2025) is a benchmark for knowledge-intensive, real-world, multi-step data science pipelines. It comprises 104 manually curated tasks drawn from approximately 1,700 data files across 24 sources in six scientific and legal domains (archaeology, astronomy, biomedical, environment, legal, wildfire), totaling 1.7 GB of heterogeneous data. Each task requires an end-to-end pipeline that performs data discovery, integration, wrangling, and statistical reasoning before producing a final answer. Tasks are organized into 633 subtasks and split roughly 60/40 between hard and easy difficulty. Answer types include exact and approximate numeric values, exact and approximate lists, and exact strings. The benchmark reports baselines for several agentic systems: smolagents (iterative) achieves 50% end-to-end accuracy, DS-GURU with self-correction reaches 22%, and a GPT-o3 design-only baseline achieves 42%. KRAMABENCH is particularly relevant because it stresses multi-step pipeline orchestration—the core capability of ADP-MA’s meta-agent architecture—and provides established baselines for comparative evaluation. DA-Code (Huang et al., 2024) is a benchmark of 500 complex data science code generation examples. Tasks span three categories: data wrangling (20%), machine learning (20%), and exploratory data analysis (60%), with a difficulty distribution of 22.8% easy, 57.3% medium, and 19.9% hard. Evaluation is execution-based: agent-generated code is run inside a Docker sandbox, and outputs are compared against ground-truth tables, charts, and ML metrics. Key metrics include completion rate, executable code percentage, table match score, and a normalized ML score. DA-Code’s emphasis on end-to-end code generation from natural-language specifications makes it a natural fit for evaluating ADP-MA’s ground-agent code generation capabilities. DSEval (Zhang et al., 2024) benchmarks LLM-powered data science agents across the entire data science lifecycle. It contains 825 problems organized into 294 problem sets drawn from four sources: textbook exercises (187 problems), StackOverflow questions (202), LeetCode challenges (40), and Kaggle competitions (396). The Kaggle subset is particularly relevant for pipeline-oriented evaluation, as its tasks require multi-step data manipulation and analysis. DSEval measures problem solving rate, difficulty-weighted score, and agent efficiency (steps and tokens per successful solution), and provides a browser-based diagnostic tool for fine-grained performance analysis. AgentBench (Liu et al., 2024), presented at ICLR 2024, evaluates LLMs as interactive agents across eight environments including database interaction and operating-system commands. Its database environment, which tests SQL query generation and data retrieval, is most directly relevant to data processing agents. MultiAgentBench (Authors, 2025) evaluates multi-agent collaboration across coordination topologies (star, chain, tree, graph) that map onto ADP-MA’s architecture—e.g., star mirrors Orchestrator-to-agent delegation, tree reflects the meta-agent–ground-agent hierarchy. Its milestone-based KPIs are relevant for assessing orchestration efficiency. MLGym (Nathani et al., 2025) provides a Gym environment for 13 machine learning research tasks requiring hypothesis generation, data processing, and iterative experimentation. Evaluation of frontier models shows that current systems can improve on baselines through hyperparameter optimization but struggle to generate novel hypotheses. Despite the breadth of these benchmarks, a notable gap persists in evaluations that assess adaptive capabilities such as handling data drift mid-pipeline, recovering from cascading failures, and optimizing resource usage across heterogeneous workloads. ADP-MA’s built-in monitoring, progressive sampling, and cross-run learning mechanisms target precisely these under-evaluated dimensions. 2.4. Self-Evolving and Adaptive Agent Systems Recent work on self-evolving agents (Fang et al., 2025) explores systems that can improve their capabilities over time through experience and reflection. This emerging direction bridges static foundation models with the continuous adaptability required by lifelong agentic systems. The survey introduces a unified conceptual framework highlighting four key components: System Inputs, Agent System, Environment, and Optimizers, which together form the feedback loop underlying self-evolving designs. Deep research agents (Huang et al., 2025) represent another relevant direction, tackling complex multi-turn informational tasks through dynamic reasoning, adaptive planning, multi-hop retrieval, and iterative tool use. These systems demonstrate that autonomous agents can successfully coordinate sophisticated workflows involving information acquisition, synthesis, and report generation. The concept of meta-learning in agent systems relates closely to ADP-MA’s architecture. While meta-learning traditionally refers to learning how to learn, ADP-MA’s meta-agents reason about how to orchestrate other agents. This distinction is crucial: rather than optimizing a single agent’s performance through experience, ADP-MA optimizes the composition and coordination of multiple specialized agents through strategic planning at runtime. 2.5. Reinforcement Learning for Agent Reasoning Advances in reinforcement learning for reasoning (Zhang et al., 2025) have demonstrated that agents can learn to make better decisions through interaction with appropriate reward signals. RL has emerged as a foundational methodology for transforming LLMs into large reasoning models capable of complex logical tasks. However, applying RL to data processing presents unique challenges due to sparse rewards, long-horizon tasks, and diverse success criteria across different data types and objectives. ADP-MA complements RL-based methods by providing explicit reasoning mechanisms through meta-agents while maintaining flexibility for future integration of learning-based optimization. The hierarchical structure enables targeted credit assignment—the Monitor can attribute success or failure to specific ground-level agents, potentially informing future RL-based improvements to agent selection and configuration. In summary, existing systems exhibit several limitations that ADP-MA addresses: most lack true hierarchical reasoning, few can dynamically create agents based on runtime observations, cross-phase coordination remains limited, and mechanisms for managing context windows effectively are underdeveloped. ADP-MA addresses these gaps through explicit hierarchical planning with the Orchestrator, dynamic agent instantiation via the Architect, continuous monitoring through the Monitor, and context management strategies that partition work to avoid context degradation. 3. System Overview ADP-MA employs a hierarchical architecture comprising three distinct layers: meta-agents for high-level orchestration, ground-level agents for task execution, and supporting infrastructure for tool management and monitoring. This section provides a comprehensive overview of the system design, architectural components, and operational workflow. 3.1. Architectural Design <details> <summary>x1.png Details</summary>  ### Visual Description # Technical Document: System Design - Architecture This document provides a comprehensive extraction and analysis of the "System Design - Architecture" diagram. The architecture describes a multi-agent system composed of high-level "Meta Agents" that manage a pipeline of "Ground Agents" to process data and tasks within a sandboxed environment. --- ## 1. Document Header * **Title:** System Design - Architecture --- ## 2. Component Isolation and Analysis The diagram is organized into four primary horizontal layers and several peripheral components. ### A. User Interface (Left) * **Entity:** User (represented by a blue person icon). * **Interaction:** Bi-directional arrow connecting the User to the **Orchestrator** within the Meta Agents block. ### B. Meta Agents (Top Layer) Contained within a dashed rectangular boundary. These agents provide high-level governance and planning. #### **1. Orchestrator** * **Role:** Primary interface and decision-maker. * **Responsibilities:** * Communicates with user. * Issues Interrupts. * Decides whether to start fresh or modify or get more input. * Review reports from Monitoring Agent. #### **2. Architect** * **Role:** Strategic planner and agent manager. * **Responsibilities:** * Task Analysis, Goal Decomposition. * Iterative Planning, Critiquing. * Creation, Removal and Modification of Ground Agents. * Backtracking Policy. #### **3. Monitor** * **Role:** Quality assurance and oversight. * **Responsibilities:** * Monitors progress of Team of Ground Agents. * Drift, outlier detection. * Metrics, progress report. * Directs Architect to take action. * Trigger Backtracking. --- ### C. Ground Agents (Middle Layer) Contained within a dashed rectangular boundary. These are the functional units that execute the data processing. * **Workflow Sequence:** Reader $\rightarrow$ Transformer $\rightarrow$ Partitioner $\rightarrow$ Indexer. * **Code Generator:** Positioned below the sequence, receiving input from the Ground Agents block to produce the execution pipeline. --- ### D. Execution & Storage (Bottom Layer) This section details the actual data flow and the environment where work is performed. #### **Input Data + Task** * Represented by a dark digital/binary image on the left. * Feeds directly into the **Code Pipeline**. #### **Code Pipeline** * **Definition:** Reader $\rightarrow$ Transformer $\rightarrow$ Partitioner $\rightarrow$ Indexer. * This pipeline is generated by the **Code Generator** (indicated by a downward blue arrow). #### **Workspace/Sandbox** A dashed rectangular boundary containing five sub-components for state management: 1. **Data Bin** 2. **Metadata Bin** 3. **Task Sandbox** 4. **Current State** 5. **Case Logging** * **Interaction:** Bi-directional arrows connect the Workspace/Sandbox to the Code Pipeline. #### **Output Data** * Represented by a 3D cylinder/database icon on the right. * Receives the final results from the **Code Pipeline**. --- ### E. External Resources (Right Side) * **Library of Data Agents:** Connected via bi-directional arrow to the Meta Agents block. * **Tools:** Connected via bi-directional arrow to the Ground Agents block. --- ## 3. System Flow and Logic 1. **Initiation:** The **User** provides a task to the **Orchestrator**. 2. **Planning:** The **Architect** decomposes the task and determines which **Ground Agents** are needed, drawing from the **Library of Data Agents**. 3. **Generation:** The **Code Generator** creates a specific **Code Pipeline** based on the Ground Agents' logic. 4. **Execution:** The **Input Data + Task** is processed through the pipeline. During this process, the pipeline interacts with the **Workspace/Sandbox** to store intermediate states, logs, and metadata. 5. **Monitoring:** The **Monitor** observes the Ground Agents' progress. If drift or errors are detected, it triggers the **Architect** to modify the plan or initiates **Backtracking**. 6. **Completion:** The processed information is committed to **Output Data**. </details> Figure 1. Architecture for ADP-MA showing the three meta-agents (Orchestrator, Architect, Monitor), ground-level agent ecosystem, and supporting infrastructure including the workspace/sandbox and tool library. The architecture, illustrated in Figure 1, is built around three primary meta-agents that coordinate to construct, execute, and refine data processing pipelines: the Orchestrator, the Architect, and the Monitor. Each meta-agent is a distinct autonomous agent with its own LLM context and prompt; they communicate through shared state and are invoked serially by the pipeline runner. These meta-agents operate at a strategic level, making decisions about workflow design and system-level adaptations rather than directly manipulating data. The Orchestrator serves as the primary interface between users and the system. It analyzes task specifications, decomposes complex goals into structured phases, and maintains the overall execution context. The Orchestrator implements ADP-MA’s hierarchical planning approach, breaking down large tasks into manageable subproblems of appropriate size and logical coherence. This decomposition strategy serves dual purposes: optimizing context window utilization and ensuring focused problem-solving without context degradation. The Architect (Figure 2) translates high-level plans into concrete implementations by creating, modifying, and removing ground-level agents. The Architect maintains a library of previously designed agent definitions, enabling rapid composition of new pipelines from reusable components. When novel capabilities are required, the Architect synthesizes new agent definitions, specifies their tools and interfaces, and integrates them into the workflow. The Architect also implements backtracking policies, determining when ground-level failures necessitate higher-level replanning versus local remediation. The Monitor provides continuous oversight of pipeline execution. It tracks progress across all active ground-level agents, detects anomalies such as data drift or outliers, generates metrics and progress reports, and issues interrupts when intervention is required. The Monitor employs rule-based heuristic checks to identify issues that might compromise pipeline correctness or efficiency. Based on monitoring insights, it directs the Architect to take corrective actions or communicates with users when human input is necessary. 3.2. Ground-Level Agent Ecosystem Ground-level agents execute specialized data operations within sandboxed environments. ADP-MA’s current implementation includes several agent types: Reader agents for data ingestion across diverse formats, Indexer agents for creating searchable representations, Partitioner agents for workload distribution, Graph agents for relational analysis, Transformer agents for data manipulation, and Compressor agents for storage optimization. This set of agent types is illustrative rather than exhaustive; the architecture supports extension to additional specialized agents as task requirements evolve. Each ground-level agent operates within a process-level sandbox that isolates code execution through namespace confinement and I/O redirection. The sandbox prepares a restricted execution namespace containing only pre-imported libraries (pandas, datetime) and input data, executes LLM-generated code via Python’s exec within this namespace, captures stdout, and tracks peak memory usage via tracemalloc. Input DataFrames are copied before injection to prevent mutation of upstream state. The sandbox auto-detects and invokes functions matching the stage_* naming convention, returning structured results that include success status, output DataFrame, execution time, memory usage, and any error tracebacks. Ground-level agents receive instructions from the Architect, execute their assigned operations within these sandboxes, and report results to the Monitor. 3.3. Knowledge and Tool Management The system maintains several persistent repositories that enable efficient pipeline construction and execution. The Library of Data Agents stores validated agent definitions that have been successfully used in prior tasks. When the Architect encounters similar requirements, it retrieves and adapts these definitions rather than creating agents from scratch, significantly accelerating development. The Tools library contains verified implementations of common data operations, ranging from basic transformations to specialized analytical functions. Ground-level agents can invoke these tools, reducing the burden of code generation and minimizing errors. The current implementation intentionally limits tool availability to maintain safety and predictability, though the architecture supports dynamic tool expansion. The system also maintains comprehensive metadata about data characteristics, pipeline state, and execution history. This metadata enables the Monitor to perform drift detection, facilitates context-aware planning by the Orchestrator, and supports intelligent backtracking decisions by the Architect. 3.4. Operational Workflow Pipeline construction and execution follow an iterative process, summarized in Algorithm 1. Upon receiving a data processing task, the Orchestrator performs initial analysis to understand data characteristics and user objectives. It then generates a high-level plan structured as a sequence of phases, each addressing a distinct aspect of the overall task. Algorithm 1 ADP-MA Operational Workflow 0: Task specification $T$ , Input data $D$ 0: Processed output $O$ 1: // Stage 1: Data Understanding 2: $S←$ Orchestrator profiles $D$ (schema, statistics, summary) 3: // Stage 2: High-Level Planning 4: $P=\{p_{1},...,p_{n}\}←$ Architect plans phases from $T$ and $S$ 5: // Stage 3: Critique and Refinement 6: repeat 7: $C←$ Architect critiques $P$ (Level-1) 8: if $C.\mathrm{severity}≥$ major then 9: $P←$ Architect revises plan using $C$ 10: end if 11: until $C.\mathrm{severity}≤$ minor or max iterations 12: // Stage 4: Phase Expansion 13: for each phase $p_{i}∈ P$ do 14: $E_{i}←$ Architect expands $p_{i}$ into substeps with agent types 15: Critique $E_{i}$ (Level-2); revise if needed 16: end for 17: // Stage 5: Ground-Agent Execution 18: for each substep $e∈ E_{1}\cup·s\cup E_{n}$ (per strategy) do 19: for $\ell∈$ {XS, S, M, FULL} do 20: Generate code via coding LLM; execute on $\ell$ -sample of $D$ 21: if execution fails then 22: Refine code with error context; retry (up to budget) 23: end if 24: end for 25: Monitor evaluates output; issue verdict 26: end for 27: // Stage 6: Finalization 28: Assemble pipeline; write case documentation and event log 29: return Final processed output $O$ For each phase, the Orchestrator invokes the Architect to design ground-level agents capable of executing required operations. The Architect consults the agent library for reusable components and synthesizes new agents when necessary. Ground-level agents are instantiated within sandboxes and provided with appropriate tools and data access. As execution proceeds, the Monitor continuously evaluates progress and performance. When issues arise, it classifies them as either local (addressable by the current agent) or global (requiring higher-level replanning). Local issues trigger focused remediation by the Architect, such as modifying agent parameters or providing additional tools. Global issues may result in phase restructuring or complete workflow revision by the Orchestrator. Context management is critical for maintaining agent effectiveness over long-running pipelines. Large contexts in monolithic agentic applications lead to performance degradation, making it essential to divide workload and agentic scopes into manageable chunks. In ADP-MA, one remediation action involves splitting an agent’s work into two serial agents with more focused responsibilities. Ongoing work aims to predict optimal context boundaries without wasting computational resources. 4. System Implementation This section describes the concrete implementation of ADP-MA, covering the six-stage pipeline, the two-level critique mechanism, execution strategies, progressive sampling, runtime monitoring, schema contracts, and case documentation. 4.1. Meta-agent Pipeline Construction The end-to-end pipeline is realized following sequential stages, each orchestrated by the ADKLLMPipelineRunner: 1. Data Understanding. The system begins by ingesting the input data source(s) and applying a suite of data tools: file-format detection, schema inference, and descriptive statistics computation. An LLM summarizes the findings in natural language, producing a data summary that becomes part of the shared context passed to all subsequent stages. When multiple data sources are provided (e.g., for join tasks), each is profiled independently and described in the summary. 1. High-Level Planning. The planning LLM receives the data summary together with a task context object that encapsulates the user goal, a detailed task description, the expected output schema (including exact column names when specified), evaluation criteria, and arbitrary task-specific parameters. The LLM produces a minimal plan consisting of one to three ordered phases, each with a stated objective and rationale. An anti-scope-creep prompt directive instructs the LLM to include only phases directly required by the goal; extraneous phases are flagged as major or critical during critique. 1. Critique and Refinement. The plan enters a Level-1 critique loop (see Figure 2) in which a separate LLM call evaluates the plan against a rubric covering phase ordering, dependency correctness, goal coverage, and scope appropriateness. Each critique iteration returns a structured result with an overall severity drawn from the set $\{\textsc{none},\textsc{minor},\textsc{major},\textsc{critical}\}$ , a list of issues with per-issue severity and affected component identifiers, and concrete suggestions. The loop exits when (a) severity drops to minor or none, (b) the maximum iteration count (default 10) is reached, or (c) an optional diff-based convergence detector determines that successive refinements have fallen below a configurable delta threshold. A major or critical verdict triggers automatic re-planning before the next critique iteration. An optional dual-judge mode submits the plan to two independent LLM calls and merges the resulting critiques for improved reliability. 1. Phase Expansion. Each approved phase is expanded by the Architect LLM into one to three substeps. Every substep specifies a ground-agent type (e.g., Reader, Profiler, Transformer, Validator, Joiner), an objective, implementation hints, an input/output schema contract, and optional dependency pointers to other substeps. A Level-2 critique loop similar to Level-1 reviews each expansion for completeness, agent-type validity, and input–output compatibility. 1. Ground-Agent Execution. The expanded substeps are dispatched to ground-level agents according to one of three generation strategies (described in Section 4.3). For each substep the ground-agent runner performs code generation via the coding LLM, sandbox execution with progressive sampling, and iterative refinement on failure. Section 4.4 details the progressive sampling protocol. 1. Finalization. Upon completion of all ground agents, the system assembles a standalone pipeline.py file containing the generated functions composed in execution order, writes a monitor summary, and emits a structured event stream for downstream replay or visualization. The framework supports separate LLM configurations for planning (meta-agents) and coding (ground agents), enabling cost–quality trade-offs—for instance, a larger model for planning and a smaller, faster model for code generation. Supported LLM providers include Anthropic, Google, OpenAI, DeepSeek, and Ollama, making the system agnostic to the underlying foundation model. 4.2. Two-Level Critique Mechanism <details> <summary>x2.png Details</summary>  ### Visual Description # Technical Document Extraction: Architect Meta Agent Workflow ## 1. Overview The image illustrates a hierarchical, iterative architectural framework titled **"Architect Meta Agent."** The diagram depicts a two-stage process for plan generation, moving from a high-level conceptual phase to a granular execution phase. Each phase utilizes a feedback loop between a generator and a critic. ## 2. Component Isolation ### Region 1: Header / Container * **Outer Boundary:** Represented by a large dashed-line rectangle encompassing all elements. * **Title:** *Architect Meta Agent* (Located at the top-left, rendered in italicized sans-serif font). ### Region 2: High-Level Planning Stage (Top Section) This section is enclosed in a horizontal dashed-line rectangle. * **Component A (Left):** "High-level Planner" (Blue text, rounded solid-line box). * **Component B (Right):** "Plan Critiquer" (Blue text, rounded solid-line box). * **Interaction:** Two curved arrows create a bidirectional loop between the Planner and the Critiquer, indicating an iterative refinement process. ### Region 3: Intermediate Output (Center) * **Connector:** A vertical downward-pointing arrow originates from the High-Level Planning Stage. * **Label:** **High Level Plan** (Bold black text, positioned to the right of the arrow). ### Region 4: Detailed Planning Stage (Bottom Section) This section is enclosed in a horizontal dashed-line rectangle. * **Component C (Left):** "Plan expander" (Blue text, rounded solid-line box). * **Component D (Right):** "Detail Plan Critiquer" (Blue text, rounded solid-line box). * **Interaction:** Two curved arrows create a bidirectional loop between the Expander and the Detail Critiquer, indicating a second iterative refinement process. ### Region 5: Final Output (Footer) * **Connector:** A vertical downward-pointing arrow originates from the Detailed Planning Stage and exits the bottom of the main container. * **Label:** **Detailed Plan** (Bold black text, positioned to the right of the arrow). --- ## 3. Process Flow and Logic The "Architect Meta Agent" operates through a top-down refinement logic: 1. **Conceptualization Loop:** The **High-level Planner** proposes an initial strategy. The **Plan Critiquer** evaluates it. This loop continues until a viable **High Level Plan** is produced. 2. **Transition:** The finalized High Level Plan is passed down as the foundational input for the next stage. 3. **Expansion Loop:** The **Plan expander** takes the high-level strategy and breaks it down into specific steps. The **Detail Plan Critiquer** reviews these specific steps for accuracy or feasibility. 4. **Finalization:** Once the expansion loop is satisfied, the system outputs the final **Detailed Plan**. --- ## 4. Textual Transcription | Location | Text Content | Style | | :--- | :--- | :--- | | Top Left | Architect Meta Agent | Italicized | | Top Box (Left) | High-level Planner | Blue, Sans-serif | | Top Box (Right) | Plan Critiquer | Blue, Sans-serif | | Center Label | High Level Plan | Bold, Black | | Bottom Box (Left) | Plan expander | Blue, Sans-serif | | Bottom Box (Right) | Detail Plan Critiquer | Blue, Sans-serif | | Bottom Label | Detailed Plan | Bold, Black | **Language Declaration:** All text is in **English**. No other languages are present. </details> Figure 2. The Architect meta-agent workflow. The Architect receives task context and data summary from the Orchestrator, generates a high-level plan, refines it through a two-level critique loop (Level 1 for plan structure, Level 2 for phase expansion), and produces schema-contracted substep definitions that are dispatched to ground-level agents for execution. The critique mechanism is the primary quality gate before any code is generated. It operates at two granularities: Level 1 – Plan Critique. Evaluates the ordered sequence of phases against the task context. Key checks include: (i) phase ordering respects data dependencies; (ii) every aspect of the user goal is addressed by at least one phase; (iii) no phase introduces work outside the stated scope; and (iv) the number of phases is minimal. An excessive phase count is flagged as major; out-of-scope phases as critical. Level 2 – Expansion Critique. Evaluates each phase’s substep decomposition. Checks verify that the selected agent types are appropriate for the substep objective, that input columns required by each substep are produced by a preceding substep or present in the original data, and that the substep count remains within the one-to-three budget per phase. Both levels share the same exit logic, parameterized by CritiqueLoopConfig: maximum iterations, exit severity threshold, optional convergence exit, and optional requirement that all phases be expanded before exit (Level 1 only). The structured output format—with per-issue issue_id, severity, affected_component, and suggestion —enables deterministic routing of critique feedback back into the planner prompt for targeted revision. 4.3. Generation Strategies The dispatch of ground agents to execution follows one of three strategies, selectable at runtime: - Centralized. Substeps execute serially in plan order. Each agent receives the full accumulated context: the current DataFrame (reflecting all prior agents’ outputs) and an updated input schema. This mode maximizes inter-agent information flow at the cost of sequential latency. It is best suited for small pipelines with tight data dependencies. - Autonomous. Substeps are grouped into parallel batches by dependency depth. Agents within the same batch execute concurrently via asyncio.gather, governed by a semaphore with a configurable maximum (default 4). Each agent receives only the original input schema (contract-only context), ensuring independence. After a batch completes, outputs are merged before the next batch begins. This mode is suited for large pipelines with independent stages. - Hybrid. Phases execute sequentially, preserving inter-phase data flow, while agents within a phase execute in parallel when they have no intra-phase dependencies. Interface validation is performed between phases to check that the output columns of one phase satisfy the input requirements of the next. This mode balances the full-context benefits of centralized execution with the throughput of autonomous parallelism. 4.4. Progressive Sampling To reduce cost and surface errors early, each ground agent is tested through an escalating sequence of sample sizes before full-scale execution: Table 1. Progressive sampling levels. | XS | 10 | Syntax and basic logic validation | | --- | --- | --- | | S | 100 | Functional correctness | | M | 1000 | Performance and edge-case coverage | | FULL | All | Production execution | Execution begins at the configured starting level (default XS). On success at a given level, the agent promotes to the next level; on failure, the coding LLM receives the error traceback and the failing code, generates a revised implementation, and retries at the same level. A maximum refinement budget (default 3 revisions) caps retry cost. Each revision is recorded with its trigger (initial, error_fix, or scale_up), execution time, peak memory, and output row count. This progressive protocol catches the majority of errors on cheap, small samples while ensuring that scaling behavior is validated before the full dataset is processed. 4.5. Backtracking When a ground agent exhausts its refinement budget without producing a correct result, the failure holds up the remainder of the pipeline. Rather than aborting, ADP-MA employs a backtracking mechanism that unwinds execution to a prior state and attempts recovery at a higher level of abstraction. Two levels of backtracking are supported: - Phase-level backtracking. The system discards the outputs of the failing substep and reverts to the DataFrame produced by the previous successfully completed stage. The Architect is then re-invoked to re-expand the current phase, potentially choosing different agent types, revised schema contracts, or alternative implementation hints. Execution resumes from the re-expanded substeps. - Plan-level backtracking. If phase-level backtracking also fails—or if the Monitor determines that the failure reflects a fundamental flaw in the plan rather than an implementation error—the system backtracks all the way to the planning stage. The Architect receives the accumulated error evidence (failing agent types, error classes, and the substep objectives that could not be fulfilled) and generates a revised high-level plan, which then proceeds through the full critique, expansion, and execution cycle. The current implementation uses heuristic triggers to choose between these two levels: a single substep failure initiates phase-level backtracking, while repeated phase-level failures (default threshold: 2 per phase) or a Monitor pause verdict escalate to plan-level backtracking. Per-phase and global retry caps (default 2 per phase, 3 total) prevent unbounded backtracking loops. While effective in practice, this heuristic approach has limitations. The decision of when to backtrack and how far to unwind is based on fixed thresholds rather than on a principled cost model that weighs the expense of further local refinement against the cost of replanning. Developing a more principled backtracking policy—potentially informed by learned estimates of repair probability, accumulated revision cost, and plan-level error attribution—is an active direction of future work. 4.6. Code Execution Sandbox Each ground agent’s generated code is executed within a process-level sandbox that provides namespace isolation, I/O capture, and resource tracking. The sandbox operates as follows: 1. Namespace preparation. A restricted Python namespace is constructed containing only pre-imported libraries (pandas as pd, datetime) and the input DataFrame. When additional data sources are specified, they are injected as df_{name} variables. All input DataFrames are copied before injection to prevent mutation of upstream state. 1. Execution. The generated code is executed via Python’s exec within the prepared namespace. The sandbox redirects stdout to capture print output and uses tracemalloc to measure peak memory consumption during execution. 1. Function invocation. After executing the code to define functions, the sandbox scans the namespace for callables matching the stage_* naming convention and invokes the first match, passing a fresh copy of the input DataFrame. 1. Result capture. The sandbox returns a structured result containing: success/failure status, the output DataFrame (if any), execution time in milliseconds, peak memory in megabytes, captured stdout, output row count, and—on failure—the error message and full traceback. This design provides functional isolation sufficient for the system’s threat model (LLM-generated data-processing code operating on user-supplied datasets) without the overhead of container-based sandboxing. The namespace is confined to pre-imported libraries and does not restrict Python’s __builtins__, so generated code retains access to standard library imports. The primary safety mechanisms are the copy-on-read input policy, stdout redirection, memory tracking, and the timeout enforced by the progressive sampling loop. 4.7. Pipeline Monitoring The PipelineMonitor evaluates the system state after every ground-agent execution using rule-based heuristics —critically, without incurring additional LLM calls. The monitor maintains configurable thresholds (Table 2) and issues one of five verdicts: continue, warn, pause, abort, or retry. Table 2. Default monitoring thresholds. | Revision count Row drop (%) Row growth (%) | $≥ 2$ $≥ 30\%$ $≥ 500\%$ | $≥ 4$ $≥ 90\%$ — | | --- | --- | --- | | Null-rate increase | $≥ 20$ pp | — | | Agent wall-clock time | $≥ 60$ s | $≥ 300$ s | | Cost vs. budget | $≥ 80\%$ | $≥ 100\%$ | | Peak memory | $≥ 500$ MB | — | Four additional monitoring features operate across and within runs: Data-drift detection. After each agent, the monitor compares the output DataFrame to the input along three axes: row count change, column set change, and per-column null-rate increase. Thresholds are configurable; a row drop exceeding 90% triggers an automatic pause. Adaptive sampling. When the monitor observes a pattern of agents passing at the XS level but failing at S, it raises an alert indicating that the smallest sample is unrepresentative and suggests starting future agents at the S level. Cross-run learning. On initialization, the monitor optionally loads failure patterns from prior case folders (up to a configurable maximum of 10 cases). Failure patterns—agent type, error class, input characteristics—are matched against the current run, and preemptive warnings are issued when a known-problematic configuration is detected. Cost tracking. The monitor accumulates token usage and estimated USD cost across all LLM calls and warns when spending approaches a configurable budget ceiling (default $1.00). 4.8. Schema Contracts Each ground agent is governed by an EnhancedSchemaContract that specifies: (i) required input columns with type constraints (dtype, nullability, pandas API checks); (ii) columns to add, preserve, or remove; (iii) value constraints (range, enum, regex pattern, uniqueness, non-null); (iv) a row-count constraint relative to the input (same, less, greater, or any); and (v) free-text postconditions and invariants. Contracts flow through the Architect’s expansion prompts and into the ground-agent code-generation prompt, ensuring that the generated code respects both upstream and downstream schema expectations. At execution time, a ContractVerificationResult records violations with severity, affected column, expected vs. actual value, and (where applicable) offending row indices. When the user supplies an expected_output specification with exact column names, the final ground agent receives strict instructions to produce exactly those columns, and the output is validated against the requirement schema before finalization. 4.9. Case Documentation and Audit Trail Every pipeline run produces a self-contained case folder that serves as a complete audit trail: - case_document.json — a structured record of all meta-agent decisions: data understanding summary, plan versions across critique iterations, phase expansions, and monitoring verdicts. - monitor_summary.json — the full list of alerts raised, with severity, evidence, and recommended actions. - events.jsonl — a time-stamped event stream (stage starts and completions, agent progress, critique results, errors) suitable for UI replay. - ground_agents/ — the generated Python source for each ground agent, including all revision history. - pipeline.py — the assembled, standalone executable pipeline composed from the successful agent functions. This comprehensive logging ensures reproducibility: any pipeline run can be inspected post-hoc to understand why specific planning or execution decisions were made, facilitating debugging, compliance auditing, and iterative improvement of the framework itself. 4.10. Human-in-the-Loop Checkpoint An optional human checkpoint can be enabled at the boundary between planning and execution. When active, the system pauses after the critique loop converges and presents the finalized plan to the user for approval. The user may approve, modify, or reject the plan. A rejection triggers re-planning (up to two rejections), after which the system proceeds with the best available plan. This mechanism preserves the option for human oversight in high-stakes scenarios while remaining fully automatic when the checkpoint is disabled. 4.11. System Tuning and Agent Prompt Evolution The design of ADP-MA’s meta-agent prompts, critique rubrics, and monitoring thresholds evolved through a combination of analytical design and empirical feedback gathered from running the system on a variety of common data processing tasks. The initial prompt templates and pipeline structure were designed from first principles—e.g., the two-level critique hierarchy, progressive sampling levels, and scope-guard directives emerged from analysis of failure modes in early prototypes. However, many design decisions required iterative refinement informed by observing system behavior on concrete workloads. To assemble a representative development workload, we used commercial LLM queries (e.g., “list a variety of data processing tasks covering cleaning, aggregation, feature engineering, multi-source integration, and graph analytics”) to generate a diverse set of task specifications. Synthetic input datasets for these tasks were generated similarly, using LLM-assisted scripts to produce tabular data with controlled characteristics such as missing values, mixed types, duplicates, and temporal structure. This approach ensured broad coverage of common data processing patterns without relying on proprietary or domain-specific datasets. The resulting task suite (listed in Appendix A.7) spans five categories: tabular cleaning, aggregation and analytics, feature engineering, multi-source integration, and temporal graph processing. Running these tasks end-to-end exposed recurring failure patterns—such as scope creep in planning, fragile column-name assumptions in generated code, and insufficient error context in refinement prompts—that drove targeted revisions to the prompt templates, critique rubrics, and monitoring rules. For example, the anti-scope-creep directive in the planning prompt and the phase-count budget in Level-1 critique were both added after observing that unconstrained planners routinely generated unnecessary validation or profiling phases. Similarly, the requirement to pass full error tracebacks (rather than just error messages) to the refinement LLM was motivated by observing that terse error summaries led to ineffective code fixes. This iterative tuning process was manual rather than automated: each round involved inspecting case documentation, identifying systematic weaknesses, revising the relevant prompt or threshold, and re-running the affected tasks to confirm improvement. While this approach does not guarantee global optimality, it produced a stable configuration that achieves high pass rates across the development workload (Appendix A.7). 4.12. Evaluation Plan Beyond the internal benchmark suite used for development tuning (Appendix A.7), we plan to evaluate ADP-MA against established external benchmarks to measure its effectiveness relative to existing systems. We are currently in the process of evaluating ADP-MA on KRAMABENCH (Ludäscher et al., 2025), a benchmark of 104 knowledge-intensive, multi-step data science tasks spanning six scientific and legal domains. KRAMABENCH is particularly well suited for assessing ADP-MA because its tasks require end-to-end pipeline orchestration—data discovery, integration, wrangling, and statistical reasoning—which exercises the full meta-agent hierarchy rather than isolated code generation. Published baselines for KRAMABENCH include smolagents (50% end-to-end accuracy), DS-GURU with self-correction (22%), and GPT-o3 design-only (42%), providing established reference points for comparison. The evaluation adapter translates each KRAMABENCH task into an ADP-MA pipeline run by mapping the task specification to a user goal and task context, loading the associated data files as primary and additional data sources, and comparing the pipeline output against the ground-truth answer. We intend to report end-to-end accuracy, per-domain breakdowns, and diagnostic metrics including planning iterations, agent revision counts, and monitor interventions. Additional external benchmarks such as DA-Code (Huang et al., 2024) and DSEval (Zhang et al., 2024) are planned as follow-on evaluations. 5. Conclusion We have presented ADP-MA, a meta-agent framework for autonomous data processing that advances the state of practice in several key dimensions. Through hierarchical orchestration, ADP-MA achieves greater adaptability and autonomy than existing approaches that rely on static pipelines or predefined agent templates. The separation between strategic meta-agents and execution-oriented ground-level agents enables sophisticated reasoning about workflow design while maintaining efficiency in task execution. ADP-MA addresses critical limitations in current data processing automation, including inflexibility to data drift, limited cross-phase optimization, and inadequate error recovery mechanisms. By incorporating continuous monitoring, iterative refinement, and intelligent backtracking, the system can handle the uncertainties and complexities inherent in real-world data processing tasks. Future work will focus on several promising directions. Expanding the framework’s capabilities to handle diverse data modalities beyond tabular data represents an important next step. Incorporating reinforcement learning to optimize meta-agent decision-making based on accumulated experience could further enhance performance. Developing further benchmarks to cover a representative variety of real-world data science, mining, and processing tasks will enable more rigorous assessment of autonomous data processing systems. Finally, exploring mechanisms for meta-agents to synthesize new tools and agent types autonomously would move the framework closer to fully general-purpose data processing automation. The meta-agent paradigm offers a pathway toward data processing systems that can operate with increasing autonomy while remaining interpretable and controllable. As LLM capabilities continue to advance, frameworks like ADP-MA will play a crucial role in translating these capabilities into practical tools that democratize sophisticated data analysis and engineering. References - Anthropic (2025) Building effective agents: context engineering. Note: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents Accessed: 2025 Cited by: §1. - V. Authors (2025) MultiAgentBench: evaluating the collaboration and competition of LLM agents. arXiv preprint arXiv:2503.01935. Cited by: §2.3. - Z. Chen et al. (2025) A survey on data science with large language models. arXiv preprint arXiv:2505.03418. Cited by: §1, §2.1. - J. Fang, Y. Peng, X. Zhang, Y. Wang, X. Yi, G. Zhang, Y. Xu, B. Wu, S. Liu, Z. Li, Z. Ren, N. Aletras, X. Wang, H. Zhou, and Z. Meng (2025) A comprehensive survey of self-evolving AI agents: a new paradigm bridging foundation models and lifelong agentic systems. arXiv preprint arXiv:2508.07407. Cited by: §2.4. - Google (2025) Google agent development kit (ADK). Note: https://google.github.io/adk-docs/ Accessed: 2026 Cited by: §1. - Y. Huang, Q. Luo, J. Li, et al. (2024) DA-Code: agent data science code generation benchmark. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Cited by: §2.3, §4.12. - Y. Huang, Y. Chen, H. Zhang, K. Li, H. Zhou, M. Fang, L. Yang, X. Li, L. Shang, S. Xu, J. Hao, K. Shao, and J. Wang (2025) Deep research agents: a systematic examination and roadmap. arXiv preprint arXiv:2506.18096. Cited by: §2.4. - L. Li, L. Fang, B. Ludäscher, and V. I. Torvik (2024a) AutoDCWorkflow: LLM-based data cleaning workflow auto-generation and benchmark. arXiv preprint arXiv:2412.06724. Cited by: §2.2. - Z. Li, Q. Zang, D. Ma, J. Guo, T. Zheng, M. Liu, X. Niu, Y. Wang, J. Yang, J. Liu, W. Zhong, W. Zhou, W. Huang, and G. Zhang (2024b) AutoKaggle: a multi-agent framework for autonomous data science competitions. arXiv preprint arXiv:2410.20424. Cited by: §1, §2.1. - X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Zhang, and J. Tang (2024) AgentBench: evaluating LLMs as agents. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), Cited by: §2.3. - B. Ludäscher, L. Li, L. Fang, Y. Long, U. Khurana, and J. Freire (2025) KRAMABENCH: a benchmark for knowledge-intensive, real-world, multi-step data science pipelines. arXiv preprint arXiv:2506.06541. Cited by: §2.3, §4.12. - D. Nathani, L. Madaan, N. Roberts, N. Bashlykov, A. Menon, V. Moens, A. Budhiraja, D. Magka, V. Vorotilov, G. Chaurasia, D. Hupkes, R. S. Cabral, T. Shavrina, J. Foerster, Y. Bachrach, W. Y. Wang, and R. Raileanu (2025) MLGym: a new framework and benchmark for advancing AI research agents. arXiv preprint arXiv:2502.14499. Cited by: §2.3. - F. Neutatz, B. Chen, Z. Abedjan, and E. Wu (2022) Data cleaning and AutoML: would an optimizer choose to clean?. Datenbank-Spektrum 22 (2), pp. 121–130. Cited by: §2.2. - X. Shen, S. Hu, X. Zhang, X. Han, X. Meng, J. Wei, Z. Liu, and M. Sun (2025) AutoClean: LLMs can prepare their training corpus. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), pp. 85–95. Cited by: §2.2. - K. Zhang, Y. Zuo, B. He, Y. Sun, R. Liu, C. Jiang, Y. Fan, K. Tian, G. Jia, P. Li, Y. Fu, X. Lv, Y. Zhang, S. Zeng, S. Qu, H. Li, S. Wang, Y. Wang, X. Long, F. Liu, et al. (2025) A survey of reinforcement learning for large reasoning models. arXiv preprint arXiv:2509.08827. Cited by: §2.5. - Y. Zhang, Q. Jiang, X. Han, N. Chen, Y. Yang, and K. Ren (2024) Benchmarking data science agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5636–5652. Cited by: §2.3, §4.12. Appendix A Appendix A.1. Ground-Agent Types Table 3 lists the ground-agent types currently supported by the Architect during phase expansion. New types can be added by registering an agent definition in the schema catalog; no changes to the orchestration logic are required. Table 3. A sample of supported ground-agent types. | Reader | Data ingestion across diverse file formats | | --- | --- | | Profiler | Schema inference, statistics, data quality assessment | | Transformer | Data manipulation, cleaning, reshaping | | Validator | Output quality checks and constraint verification | | Joiner | Multi-source merges and schema alignment | | Indexer | Searchable representation construction | | Partitioner | Workload distribution and data splitting | | Graph | Relational and graph-structured analysis | | Compressor | Storage optimization and encoding | | Aggregator | GroupBy, rolling statistics, pivots | | FeatureEngineer | Encoding, scaling, datetime feature extraction | A.2. Critique Loop Configuration The critique loop is parameterized by CritiqueLoopConfig, whose fields are listed in Table 4. All parameters have sensible defaults and can be overridden per-run. Table 4. Critique loop configuration parameters. | max_iterations | 10 | Max critique–refine cycles | | --- | --- | --- | | exit_on_severity | minor | Exit when severity $≤$ threshold | | require_all_phases_expanded | false | L1: wait for all expansions | | enable_convergence_exit | false | Diff-based early stopping | | convergence_threshold | 0.1 | Delta below which converged | | convergence_min_iterations | 2 | Min iters before convergence | A.3. Event Stream Schema The event logger emits structured events of the following types, enabling both real-time streaming (via callbacks) and post-hoc replay: - stage_start / stage_complete — marks entry and exit of each of the six pipeline stages. - agent_progress — reports ground-agent status (generating, executing, refining, success, failed) with sample level, revision number, and timing. - critique_result — records each critique iteration’s severity, issue count, and summary. - monitor_verdict — captures the monitor’s verdict and associated alerts after each agent. - plan_version — snapshots the plan after each refinement iteration. - error — logs exceptions with traceback and context. A.4. Task Context Passthrough A rich task context object flows through every meta-agent prompt, ensuring that downstream agents have access to the user’s full intent without relying on the accumulation of multi-turn conversational context. The task context comprises: - task_description: A detailed natural-language description of the processing goal. - expected_output: Schema requirements including exact column names and row-level descriptions. - evaluation_criteria: Success criteria against which the output will be judged. - task_parameters: Arbitrary key–value pairs for domain-specific configuration. This design decouples planning quality from context-window management: the LLM receives a self-contained specification at each stage rather than a growing conversational history. A.5. Multi-Source Data Support When a task requires joining or merging data from multiple files, additional data sources are passed as named DataFrames. Each additional source is profiled during the Data Understanding stage and described in the data summary. During ground-agent execution, additional sources are injected into the sandbox namespace as df_{name} variables (see Section 4.6), enabling agents to perform real joins and merges without generating mock data. This mechanism supports an arbitrary number of auxiliary data sources alongside the primary input. A.6. TDD and Code Review Support The framework optionally integrates test-driven development (TDD) into ground-agent execution. When enabled, the coding LLM first generates unit tests for each agent function before generating the implementation. Tests are executed in the sandbox; the implementation must pass all tests before proceeding to the next sample level. Three modes are supported: disabled (default), optional (tests generated but failures do not block), and strict (test failures block promotion). Independently, an LLM-powered code review step can inspect generated code for correctness, error handling, performance, pandas best practices, and contract compliance. Review findings are fed back into the refinement prompt when issues are detected. A.7. Development and Tuning Tasks Table 5 lists the 25 data processing tasks used to iteratively tune prompts, critique rubrics, and monitoring thresholds as described in Section 4.11. Task specifications and synthetic datasets were generated using commercial LLM queries. Table 5. Data processing tasks used for system tuning and prompt evolution. | ID | Task Name | Category | Difficulty | | --- | --- | --- | --- | | TC-001 | Missing Value Imputation | Tabular Cleaning | Easy | | TC-002 | Duplicate Removal | Tabular Cleaning | Easy | | TC-003 | Type Standardization | Tabular Cleaning | Medium | | TC-004 | Outlier Detection (IQR) | Tabular Cleaning | Medium | | TC-005 | Multi-Format Date Parsing | Tabular Cleaning | Medium | | TC-006 | Text Normalization | Tabular Cleaning | Medium | | AA-001 | Simple GroupBy Aggregation | Aggregation & Analytics | Easy | | AA-002 | Multi-Level GroupBy | Aggregation & Analytics | Medium | | AA-003 | Rolling Statistics | Aggregation & Analytics | Medium | | AA-004 | Pivot Table Construction | Aggregation & Analytics | Medium | | AA-005 | Percentile Analysis | Aggregation & Analytics | Medium | | FE-001 | One-Hot Encoding | Feature Engineering | Easy | | FE-002 | Label Encoding | Feature Engineering | Easy | | FE-003 | Numeric Scaling (Min-Max, Z-Score) | Feature Engineering | Medium | | FE-004 | Datetime Feature Extraction | Feature Engineering | Medium | | FE-005 | Binning (Equal-Width, Quantile) | Feature Engineering | Medium | | MI-001 | Inner Join | Multi-Source Integration | Easy | | MI-002 | Left Join with Null Handling | Multi-Source Integration | Medium | | MI-003 | Multi-Table Join | Multi-Source Integration | Hard | | MI-004 | Schema Alignment & Merge | Multi-Source Integration | Medium | | TG-001 | Node History Index | Temporal Graph | Medium | | TG-002 | Edge Timeline Construction | Temporal Graph | Medium | | TG-003 | Graph Metrics Computation | Temporal Graph | Hard | | TG-004 | Temporal Subgraph Extraction | Temporal Graph | Medium | | TG-005 | Event Sequence Pattern Analysis | Temporal Graph | Hard | A.8. Demonstration Interface We provide a web-based demonstration tool with two operating modes: Live Mode and Replay Mode. In Live Mode, users submit a natural-language goal, select the planning and coding LLMs from a set of available providers (e.g., OpenAI, Anthropic, Google, or local Ollama models), and observe the pipeline execute in real time via a streaming WebSocket connection. In Replay Mode, users browse previously completed cases and replay the recorded event stream at configurable playback speeds, enabling review and presentation of past runs. Both modes share a common visualization layout: a meta-agent pipeline progress bar, per-stage detail panels, a ground-agent pipeline view, generated code inspection, and execution metrics. <details> <summary>x3.png Details</summary>  ### Visual Description # Technical Document Extraction: ADP-MA Interface This document provides a comprehensive extraction of the textual and structural information from the provided image of the **ADP-MA** (Agentic Data Pipeline - Meta Agent) web interface. --- ## 1. Header Region * **Logo/Title:** `MA ADP-MA` (Located top-left). * **Mode Toggle:** * `Live Mode` (Selected/Active, indicated by blue background and target icon). * `Replay Mode` (Inactive, indicated by circular arrow icon). * **Utility Icons:** Document icon and GitHub logo (Located top-right). --- ## 2. Main Content Area: Header & Description * **Section Title:** `Live Mode` * **Sub-text:** `Run a new pipeline and watch it execute in real-time` --- ## 3. Left Sidebar: Run Pipeline Configuration This region contains the input parameters for initiating a data processing task. ### Run Pipeline Form * **Goal / Task (Text Area):** * Placeholder text: `e.g., Create a way to directly access a node history from temporal graph data` * **Data File (Upload Field):** * Current File: `customers_churn.csv (485.4 KB)` * Action Button: `Upload` * **Planning LLM (Dropdown):** * Selected Value: `DeepSeek` * **Coding LLM (Dropdown):** * Checkbox: `Same as Planning` (Unchecked) * Selected Value: `DeepSeek` * **Advanced Options:** Expandable menu (currently collapsed). * **Primary Action Button:** `Run Pipeline` (Blue button with target icon). ### Metrics (Footer of Sidebar) * **Section Title:** `Metrics` * **Field:** `Duration` * **Value:** `--` (Indicates no active process is currently running). --- ## 4. Main Dashboard: Pipeline Progress & Details This region visualizes the workflow and provides granular feedback on the execution stages. ### Meta-agent Pipeline Progress This is a horizontal flow diagram representing the sequential stages of the agentic workflow. Each stage is represented by a pill-shaped node connected by lines. | Stage Order | Label | Status Indicator | | :--- | :--- | :--- | | 1 | `Data` | Grey Circle (Inactive/Pending) | | 2 | `Plan` | Grey Circle (Inactive/Pending) | | 3 | `Critique` | Grey Circle (Inactive/Pending) | | 4 | `Expand` | Grey Circle (Inactive/Pending) | | 5 | `Execute` | Grey Circle (Inactive/Pending) | | 6 | `Done` | Grey Circle (Inactive/Pending) | ### Stage Details This is a large container intended to display logs or specific outputs for the selected stage. * **Section Title:** `Stage Details` * **Empty State Content:** * Icon: Lightbulb * Instructional Text: `Click a stage above to see details` --- ## 5. Summary of Technical Components * **Application Name:** ADP-MA. * **Primary Function:** Real-time execution and monitoring of an agentic data pipeline. * **Workflow Logic:** The system follows a 6-step linear progression: Data ingestion -> Planning -> Critique -> Expansion -> Execution -> Completion. * **Technology Stack Indicators:** The interface allows for the selection of Large Language Models (LLMs), specifically identifying `DeepSeek` as a compatible model for both planning and coding tasks. </details> Figure 3. Live Mode initial state. The left panel provides a task input area and LLM selection dropdowns for the planning and coding models. The right panel shows the meta-agent pipeline progress bar and placeholders for stage details and generated code. <details> <summary>x4.png Details</summary>  ### Visual Description # Technical Document Extraction: ADP-MA Replay Mode Interface This document provides a comprehensive extraction of the textual and structural data from the provided image of the **ADP-MA** software interface. ## 1. Header Information * **Application Name:** ADP-MA (accompanied by a purple icon with "MA"). * **Mode Selection:** * **Live Mode:** Inactive (indicated by grey text and radio icon). * **Replay Mode:** Active (indicated by a blue highlighted button with a refresh icon). * **Utility Icons:** Document icon and GitHub logo (top right). ## 2. Page Title and Description * **Title:** Replay Mode * **Description:** Replay previous pipeline executions to review or demonstrate. --- ## 3. Component Isolation: Left Sidebar (Case Selection) This section allows the user to select a specific historical execution. * **Section Header:** Select Case to Replay * **Case Count:** 100 cases * **Search Bar:** Placeholder text "Search cases..." * **Case List (Scrollable):** | Case ID | Status Tag | Description | Timestamp | Agent Count | | :--- | :--- | :--- | :--- | :--- | | `case_20260130_102653_bde4bd3a` | Has Events (Blue) | Build chronological edge list for e... | 1/30/2026 10:26 AM | 3 agents | | `case_20260130_102412_a1aee839` | Has Events (Blue) | Create a way to directly access a ... | 1/30/2026 10:24 AM | 2 agents | | `case_20260130_102145_f2e994f9` | Has Events (Blue) | Clean and standardize text columns | 1/30/2026 10:21 AM | 1 agents | | `case_20260130_101854_fe97abd5` | Has Events (Blue) | Parse varied date formats to cons... | 1/30/2026 10:18 AM | 1 agents | --- ## 4. Component Isolation: Main Content Area (Pipeline Details) ### 4.1 Meta-agent Pipeline Progress This is a horizontal flow diagram showing the stages of the pipeline. All stages are marked with a green dot, indicating completion. * **Status Indicator:** Done (Blue pill button) * **Flow Sequence:** 1. **Data** (Highlighted with a blue border, indicating current view) 2. **Plan** 3. **Critique** 4. **Expand** 5. **Execute** 6. **Done** ### 4.2 Stage Details: Data Understanding This section provides a technical breakdown of the data processed in the "Data" stage. * **Sub-header:** Stage: Data Understanding * **Section:** Data Analysis * **Data Metrics:** * **Rows:** 5,000 (Displayed in a blue box) * **Columns:** 4 (Displayed in a green box) * **Column Schema:** * `node1` (object) * `node2` (object) * `event_type` (object) * `timestamp` (object) ### 4.3 Summary Text The following text is transcribed from the "Summary" block: 1. This dataset represents event logs for interactions between nodes, detailing updates or creations of connections with timestamps. 2. The data is complete without null values and structured consistently across 5000 records, indicating high quality and reliability. 3. Recommended processing includes converting the 'timestamp' column to a datetime format for temporal analysis, followed by aggregating events over time periods or analyzing connectivity patterns between nodes. </details> Figure 4. Replay Mode showing the Data Understanding stage. The stage panel displays detected row count, column count, column types, and a natural-language summary of the dataset characteristics. <details> <summary>x5.png Details</summary>  ### Visual Description # Technical Document Extraction: ADP-MA Interface ## 1. Document Overview This image displays a web-based dashboard for a system titled **ADP-MA**. The interface is currently in **Replay Mode**, designed to "Replay previous pipeline executions to review or demonstrate." The layout is divided into a navigation sidebar on the left and a detailed execution view on the right. --- ## 2. Header Section * **Application Name:** ADP-MA (accompanied by a purple 'MA' logo). * **Mode Toggle:** * **Live Mode:** Inactive (indicated by a radio button icon). * **Replay Mode:** Active (highlighted in blue with a refresh/replay icon). * **Utility Icons:** Document icon and GitHub logo (top right). * **Sub-header Text:** "Replay previous pipeline executions to review or demonstrate" --- ## 3. Sidebar: "Select Case to Replay" This section allows users to browse historical execution cases. It indicates a total of **100 cases**. ### Search and Filter * **Search Bar:** Placeholder text "Search cases..." ### Case List (Transcribed Entries) The list shows several cases, each with a unique ID, a description, a timestamp, and an agent count. All visible cases are tagged with a blue badge labeled **"Has Events"**. | Case ID | Description Snippet | Timestamp | Agent Count | | :--- | :--- | :--- | :--- | | `case_20260130_193832_6b90287b` | create a way to directly access th... | 1/30/2026 07:38 PM | 7 agents | | `case_20260130_180602_c001f03e` | Create a way to directly access a ... | 1/30/2026 06:06 PM | 8 agents | | `case_20260130_180431_5b25697d` | Create a way to directly access a ... | 1/30/2026 06:04 PM | [Not visible] | | `case_20260130_180301_191c11e4` | Create a way to directly access a ... | 1/30/2026 06:03 PM | [Not visible] | *Note: The first case is highlighted with a blue border, indicating it is the currently selected case.* --- ## 4. Main Content: Meta-agent Pipeline Progress This section tracks the lifecycle of the selected case. ### Pipeline Flow Diagram The pipeline consists of six sequential stages connected by lines. All stages are marked with a green dot, and the final status is marked as **"Done"** in a blue badge. 1. **Data** (Green background) 2. **Plan** (Light blue background with blue border - **Current Focus**) 3. **Critique** (Green background) 4. **Expand** (Green background) 5. **Execute** (Green background) 6. **Done** (Green background) ### Stage Detail: Planning The interface displays the specific details for the **"Plan"** stage of the pipeline. #### Plan Strategy * **Label:** Strategy (Purple text) * **Content:** "Build a dictionary mapping each node to its chronological event history by processing the event log data." #### Phases The plan is broken down into two numbered phases: 1. **Data Preparation** * **Description:** "Convert timestamp to datetime, sort data chronologically, and validate required columns exist." 2. **Build History Access** * **Description:** "Create a dictionary where each key is a node and the value is a sorted list of its events (event_type, timestamp)." --- ## 5. Technical Components & Logic * **Spatial Grounding:** The legend/progress bar is located at the top of the main content area. The "Plan" node is visually distinct (blue border), which corresponds to the "Stage: Planning" header immediately below it. * **Trend/Flow:** The pipeline follows a linear progression from data ingestion to execution. The green coloring across all nodes suggests a completed or successful historical run. * **Data Structure:** The "Strategy" and "Phases" sections indicate that the system operates on event log data, utilizing Python-like data structures (dictionaries) and standard data cleaning procedures (datetime conversion, validation). </details> Figure 5. Replay Mode showing the Plan stage. The detail panel displays the selected strategy (e.g., centralized) and the decomposed phases with their objectives, providing visibility into the Architect meta-agent’s output. <details> <summary>x6.png Details</summary>  ### Visual Description # Technical Document Extraction: ADP-MA Replay Mode Interface This document provides a comprehensive extraction of the textual and structural data from the provided image of the **ADP-MA** software interface. ## 1. Header Information * **Application Name:** ADP-MA (Top left, accompanied by a purple icon). * **Mode Selection:** * **Live Mode:** Inactive (indicated by grey text and a radio-style icon). * **Replay Mode:** Active (indicated by a blue highlighted button with a refresh/replay icon). * **Utility Icons:** Top right contains a document icon and a GitHub logo icon. * **Page Title:** Replay Mode * **Page Description:** "Replay previous pipeline executions to review or demonstrate" --- ## 2. Sidebar: Case Selection (Left Region) This section allows the user to select a specific execution instance to replay. * **Section Title:** Select Case to Replay * **Total Count:** 100 cases * **Search Bar Placeholder:** "Search cases..." * **Case List Items:** 1. **Selected Case:** `case_20260130_193832_6b98287b` * **Description:** "create a way to directly access th..." * **Timestamp:** 1/30/2026 07:38 PM * **Agent Count:** 7 agents * **Status Tag:** Has Events (Blue background) * *Visual Note:* This item is highlighted with a blue border and light blue background. 2. **Case:** `case_20260130_180602_c001f03e` * **Description:** "Create a way to directly access a ..." * **Timestamp:** 1/30/2026 06:06 PM * **Agent Count:** 8 agents * **Status Tag:** Has Events 3. **Case:** `case_20260130_180431_5b25697d` * **Description:** "Create a way to directly access a ..." * **Timestamp:** 1/30/2026 06:04 PM * **Status Tag:** Has Events 4. **Case:** `case_20260130_180301_191c11e4` * **Description:** "Create a way to directly access a ..." * **Timestamp:** 1/30/2026 06:03 PM * **Status Tag:** Has Events --- ## 3. Main Content: Pipeline Progress (Top Right Region) This component visualizes the workflow stages of the Meta-agent. * **Section Title:** Meta-agent Pipeline Progress * **Overall Status:** Done (Blue pill tag in the top right corner). * **Workflow Diagram:** A linear horizontal flow of nodes connected by lines. * **Node 1:** Data (Green dot, light green background) * **Node 2:** Plan (Green dot, light green background) * **Node 3 (Active):** Critique (Green dot, light green background, **Blue border** indicating current view) * **Node 4:** Expand (Green dot, light green background) * **Node 5:** Execute (Green dot, light green background) * **Node 6:** Done (Green dot, light green background) --- ## 4. Main Content: Stage Details (Bottom Right Region) This section provides the specific output for the stage selected in the pipeline progress bar. * **Current Stage:** Critique * **Sub-header:** Plan Critique * **Severity Level:** MINOR (Displayed in a yellow highlighted box). ### Issues * The plan lacks explicit handling for both node1 and node2 columns when building node history, potentially missing events where a node appears in node2 but not node1. * No mention of handling duplicate timestamps or tie-breaking logic for events occurring at the same time. * The plan does not specify how to handle events where a node interacts with itself (if node1 == node2). ### Recommendations * Clarify that both node1 and node2 columns should be processed to capture all events for each node. * Add a step to resolve timestamp ties (e.g., by event_type order or adding a secondary sort key). * Consider outputting the history in a structured format (e.g., list of dicts) for easier downstream use. --- ## 5. Component Summary & Logic Check * **Spatial Grounding:** The interface is split into a navigation sidebar (left) and a detail view (right). The detail view is further split into a progress tracker (top) and a content pane (bottom). * **Trend/Flow Verification:** The pipeline progress shows a completed linear sequence (all nodes are green). The user has specifically clicked on the "Critique" node (indicated by the blue selection border), which has populated the "Stage: Critique" details below. * **Language:** All text is in English. No other languages detected. </details> Figure 6. Replay Mode showing the Critique stage. The panel displays the critique severity level (e.g., MINOR), a list of identified issues, and recommendations for plan improvement. <details> <summary>x7.png Details</summary>  ### Visual Description # Technical Document Extraction: ADP-MA Replay Mode Interface This document provides a comprehensive extraction of the textual and structural data from the provided image, which depicts a software interface for a pipeline execution review tool named **ADP-MA**. --- ## 1. Header Information * **Application Name:** ADP-MA (accompanied by a purple icon with "MA") * **Navigation/Status:** * **Live Mode:** Inactive (indicated by a grey radio-style icon) * **Replay Mode:** Active (indicated by a blue highlighted button with a refresh/replay icon) * **Utility Icons:** Document icon and GitHub logo (top right). --- ## 2. Page Title and Description * **Title:** Replay Mode * **Description:** Replay previous pipeline executions to review or demonstrate. --- ## 3. Component Isolation: Left Sidebar (Case Selection) **Section Title:** Select Case to Replay **Metadata:** 100 cases **Search Bar:** "Search cases..." (placeholder text) ### Case List (Data Table/List Structure) The list contains multiple entries. The first entry is currently selected (highlighted in blue). | Case ID / Timestamp | Title / Description | Status Tag | Agents | | :--- | :--- | :--- | :--- | | `1/30/2026 10:30 AM` | (Partial text visible above selection) | - | 1 agents | | **`case_20260130_102653_bde4bd3a`** | **Build chronological edge list for e...** | **Has Events** | **3 agents** | | `case_20260130_102412_a1aee839` | Create a way to directly access a ... | Has Events | 2 agents | | `case_20260130_102145_f2e994f9` | Clean and standardize text columns | Has Events | 1 agents | | `case_20260130_101854_fe97abd5` | Parse varied date formats to cons... | Has Events | 1 agents | *Note: The selected case (second in list) is dated 1/30/2026 10:26 AM.* --- ## 4. Component Isolation: Main Content Area (Pipeline Progress) ### Meta-agent Pipeline Progress This is a horizontal flow diagram showing the stages of the pipeline. * **Overall Status:** Done (Blue pill label at top right of the section). * **Flow Components:** 1. **Data:** Completed (Green dot, Green background) 2. **Plan:** Completed (Green dot, Green background) 3. **Critique:** Completed (Green dot, Green background) 4. **Expand:** **Active/Selected** (Green dot, Blue border highlight) 5. **Execute:** Completed (Green dot, Green background) 6. **Done:** Completed (Green dot, Green background) --- ## 5. Component Isolation: Stage Details (Expansion) **Stage Title:** Stage: Expansion **Sub-section:** Expanded Phases This section breaks down the "Expand" stage into specific tasks and sub-tasks. ### Phase 1: Timestamp Conversion and Sorting * **1.1 TypeConverter:** * *Description:* Convert the 'timestamp' column to datetime format. * **1.2 NodeHistoryBuilder:** * *Description:* Sort events chronologically and ensure bidirectional edges are handled correctly. ### Phase 2: Edge List Construction * **2.1 ChronologicalEdgeListBuilder:** * *Description:* Construct a chronological edge list for each node pair, ensuring bidirectional edges are handled correctly. --- ## 6. Technical Summary of Flow The interface demonstrates a **Replay Mode** for a multi-agent system. 1. The user selects a specific execution "case" from the left sidebar (currently viewing a case involving building an edge list). 2. The **Meta-agent Pipeline** shows a linear 6-step process from Data to Done. 3. The current view focuses on the **Expansion** stage, which is decomposed into two primary phases: **Timestamp Conversion/Sorting** and **Edge List Construction**, utilizing specific technical modules (`TypeConverter`, `NodeHistoryBuilder`, and `ChronologicalEdgeListBuilder`). </details> Figure 7. Replay Mode showing the Expand stage. Each phase is expanded into concrete substeps with assigned agent types, representing the transition from abstract plan to executable agent assignments. <details> <summary>x8.png Details</summary>  ### Visual Description # Technical Document Extraction: ADP-MA Replay Mode Interface ## 1. Header Information * **Application Name:** ADP-MA (Top left, accompanied by a purple 'MA' logo). * **Operational Status:** * **Live Mode:** Inactive (Grey text with radio icon). * **Replay Mode:** Active (Highlighted blue button with refresh icon). * **Utility Icons:** Document icon and GitHub logo (Top right). ## 2. Page Title and Description * **Title:** Replay Mode * **Description:** Replay previous pipeline executions to review or demonstrate. ## 3. Component Isolation ### Region A: Select Case to Replay (Left Sidebar) This section allows users to browse and select historical execution cases. * **Total Count:** 100 cases. * **Search Bar:** Placeholder text "Search cases...". * **Case List Entries (Top 4 visible):** 1. **Selected Case:** `case_20260130_102653_bde4bd3a` * **Description:** Build chronological edge list for e... * **Timestamp:** 1/30/2026 10:26 AM * **Metadata:** 3 agents | **Tag:** Has Events (Blue) 2. **Case:** `case_20260130_102412_a1aee839` * **Description:** Create a way to directly access a ... * **Timestamp:** 1/30/2026 10:24 AM * **Metadata:** 2 agents | **Tag:** Has Events (Blue) 3. **Case:** `case_20260130_102145_f2e994f9` * **Description:** Clean and standardize text columns * **Timestamp:** 1/30/2026 10:21 AM * **Metadata:** 1 agents | **Tag:** Has Events (Blue) 4. **Case:** `case_20260130_101854_fe97abd5` * **Description:** Parse varied date formats to cons... * **Timestamp:** 1/30/2026 10:18 AM * **Metadata:** 1 agents | **Tag:** Has Events (Blue) ### Region B: Meta-agent Pipeline Progress (Top Center) A horizontal flow diagram showing the high-level stages of the pipeline. * **Overall Status:** Done (Blue badge). * **Flow Components:** 1. **Data:** Green dot (Completed). 2. **Plan:** Green dot (Completed). 3. **Critique:** Green dot (Completed). 4. **Expand:** Green dot (Completed). 5. **Execute:** Green dot, highlighted with a blue border (Current focus/Active selection). 6. **Done:** Green dot (Completed). ### Region C: Stage: Execution (Bottom Center) Details regarding the specific sub-agents running within the "Execute" stage. * **Instruction:** "Click an agent in the Ground-agent Pipeline below for full details." * **Execution Summary Table:** | ID | Agent Name | Status | | :--- | :--- | :--- | | 1.1 | TypeConverter | success (Green badge) | | 1.2 | NodeHistoryBuilder | success (Green badge) | | 2.1 | ChronologicalEdgeListBuilder | success (Green badge) | ### Region D: Generated Code (Bottom Right) A code block displaying the Python logic generated for the selected agent. * **Controls:** Version selector (1.1), v1 toggle, "Final" status badge (Green), and a copy-to-clipboard icon. * **Code Transcription:** ```python import pandas as pd def stage_1_1_typeconverter(df): # Check if the dataframe is empty if df.empty: print("The DataFrame is empty.") return df # Ensure that required columns are pr required_columns = ['timestamp'] missing_cols = [col for col in requir if missing_cols: print(f"Missing required columns: return df ``` ## 4. Visual Logic and Trend Summary * **Workflow Status:** The pipeline is fully completed ("Done"). The user is currently inspecting the "Execute" stage of a case recorded on January 30, 2026. * **Color Coding:** * **Green:** Indicates successful completion of stages and agents. * **Blue:** Indicates active selection (Replay Mode, the specific Case, and the "Execute" stage in the progress bar). * **Data Integrity:** The interface shows a 1:1 mapping between the selected agent "1.1: TypeConverter" in the Execution Summary and the function name `stage_1_1_typeconverter` in the Generated Code block. </details> Figure 8. Replay Mode showing the Execute stage. During execution, the layout switches to a side-by-side view: the left panel summarizes agent status (e.g., 3 successful agents), while the right panel displays the generated Python code with revision history and syntax highlighting. <details> <summary>x9.png Details</summary>  ### Visual Description # Technical Document Extraction: ADP-MA Interface This document provides a comprehensive extraction of the data and structural components of the ADP-MA software interface as shown in the provided image. ## 1. Header Navigation * **Application Name:** ADP-MA (Logo: Purple square with "MA" in white). * **Mode Selection:** * **Live Mode:** Inactive (indicated by grey text/icon). * **Replay Mode:** Active (indicated by blue background and "refresh" icon). * **Utility Icons:** Document icon and GitHub logo (top right). --- ## 2. Left Sidebar (Control & Metrics) ### Region: Playback Controls * **Status:** Completed * **Case duration:** 3m 44s * **Progress Bar:** Full blue bar. * **Progress Value:** 35 / 33 events (Note: Indicates over-completion or additional system events). * **Action Button:** Blue button labeled "Replay" with a play icon. * **Playback Speed Selection:** * Current Selection: **7.5x** (30s playback) - Highlighted in blue. * Other Options: 11x (20s), 5x (45s), 3.7x (1m 1s), 2.5x (1m 30s), 1.9x (1m 58s). ### Region: Metrics * **Duration:** 224.0s * **Agents:** 3/3 (Displayed in green) * **LLM Calls:** 5 * **LLM Time:** 94.7s * **Planning LLM:** `ollama/qwen2.5:14b` * **Coding LLM:** `ollama/qwen2.5:14b` --- ## 3. Main Content Area (Code & Pipeline) ### Region: Generated Code * **Header Info:** 3 agents; Copy icon present. * **Version Control:** Dropdown set to `1.1`, version `v1`, and a green tag labeled `Final`. * **Code Block Transcription (Python):** ```python import pandas as pd def stage_1_1_typeconverter(df): # Check if the dataframe is empty if df.empty: print("The DataFrame is empty.") return df # Ensure that required columns are present in the DataFrame ``` ### Region: Ground-agent Pipeline * **Status:** 3/3 completed | Component | Description | Size Tags | Status | | :--- | :--- | :--- | :--- | | **1.1: TypeConverter** | Convert the 'timestamp' column to datetime format. | XS, S, M, **FULL** | Complete (Green) | | **1.2: NodeHistoryBuilder** | Sort events chronologically and ensure bidirectional edges are handled correctly. | XS, S, M, **FULL** | Complete (Green) | | **2.1: ChronologicalEdgeListBuilder** | Construct a chronological edge list for each node pair, ensuring bidirectional edges are handled correctly. | XS, S, M, **FULL** | Complete (Green) | --- ## 4. Technical Summary & Flow The image depicts a post-execution "Replay Mode" of an automated data processing pipeline. 1. **Input/Configuration:** The system utilized the `qwen2.5:14b` model via Ollama for both planning and coding tasks. 2. **Execution Flow:** The "Ground-agent Pipeline" shows a sequential flow starting from data type conversion (`TypeConverter`), moving to history building (`NodeHistoryBuilder`), and concluding with edge list construction (`ChronologicalEdgeListBuilder`). 3. **Output:** The "Generated Code" section shows the Python implementation produced by the agents for the selected step (1.1). 4. **Performance:** The entire process took 224 seconds, with nearly half that time (94.7s) spent on 5 distinct LLM calls. </details> Figure 9. Replay Mode after pipeline completion. The interface shows playback controls with auto-suggested speeds, summary metrics (total duration, agent count, LLM calls), generated code, and the ground-agent pipeline depicting each agent’s progression through sample levels (XS $→$ S $→$ M $→$ FULL).