Co-Designing the Tool Surface with the Agent Loop: Efficiency and Feasibility in Agentic Design Automation

IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD)

Tongyang Dai, Dimitrios Stamoulis, Hongyu Yuan, Jing Jin, and Jie Liu

---

This repository contains the artifact for the above paper. It includes the Ptolemy MCP server, the full evaluation harness, pre-collected per-run trace files for all four evaluation environments, and a script that reproduces every paper table from those traces without requiring an LLM API key.

Reproducing the paper tables (no API key required)

The pre-collected traces under eval/results/per_run/ (and the corresponding directories for Verilog/RTL, SystemC, and Simulink) are all that is needed to regenerate the LaTeX tables. No Ptolemy II installation and no Java are needed.

pip install -e .
python reproduce_tables.py

Output .tex and .json files are written to reproduced_tables/. The numbers should match the published paper exactly. An alternative output directory can be specified with --out-dir.

Which runs each table uses

reproduce_tables.py regenerates Tables V–IX plus the extended analyses and supplementary tables. Every cell is computed from the per-run traces, with one exception noted at the end of this section.

The per_run/ directories hold more than the main experiments: ablation sweeps, exploratory re-runs, and extra replicates share the same directory and are separated by run_ix block. Each generator therefore filters, and the blocks it keeps are part of the published result definition:

Table	Runs used
V (Benchmark A)	run_ix 0–4, the 10 canonical TASKS; novel tasks use 5000–5002
VI (closed-loop)	run_ix 1000–1002 (N=3); config OPT uses 7000
VII (ablation)	78 single-factor runs; see stress_heldout_status.tex
VIII (modal + controls)	per-arm blocks, listed in aggregate_modal_controls.py
IX (cross-environment)	all runs in eval_{eda,systemc,simulink}/results/per_run/

Filtering on run_ix alone is not sufficient for Table V: blocks 0–2 of C1 and C4 also carry closed-loop and novel-task rows, so the generators additionally restrict to the canonical task list in eval/aggregate_results.py.

Table VIII pins a block per arm rather than one block for the whole table, and the module docstring in eval/aggregate_modal_controls.py records why for each: Tool-Naive's modal runs use 7000–7004 because the earlier 6000–6004 batch predates a harness fix and is quarantined under eval/results/invalid_lsbug/; cl_two_tank_modal uses 8300–8304 for all three arms; and the +Docs control uses 9000–9009 because rows 6000–6002 for that config are early trial runs, not the reported N=10 batch.

Two pass metrics, and which one each table reports. A trace records both grade.feasible (all behavioural properties pass) and grade.structural_pass (an additional structural check). The top-level passed field equals grade.overall_pass, i.e. feasible && structural_pass — the paper's Strict metric. Table VIII reports feasibility, so its generator reads grade.feasible; reading passed instead yields the Strict totals (1/20, 2/20, 12/20) rather than the published feasibility totals (1/20, 9/20, 18/20). aggregate_modal_controls.py prints both side by side so the distinction stays visible.

The one column not derived from traces: Table IX's Tokens/run. Token counts come from per-run LLM transcripts, which are not redistributed (they contain full prompt text), so aggregate_crossenv.py emits the published constants for that column and computes everything else. eval_systemc/extract_token_usage.py recomputes it if you have the transcripts.

aggregate_stress_heldout.py reports Claude matrix complete: False by design — the single-factor ablation covers 78 of a possible 150 runs, and stress_heldout_status.tex discloses which cells are incomplete.

Installation

Python 3.10 or later is required.

pip install -e .

For Ptolemy II integration (needed to run new evaluations, not for table reproduction), set two environment variables before starting the server:

export PTII=/path/to/ptII          # Ptolemy II installation root
export JAVA_HOME=/path/to/java     # JRE 11+

On Windows use set or PowerShell $env: equivalents.

Running the smoke demo

The smoke demo starts the MCP server over stdio, builds a small SDF model, and verifies a signal property end-to-end. It requires PTII and JAVA_HOME to be set.

python demos/mcp_stdio_smoke.py

Re-running the full evaluation

Re-running requires a live LLM API key and a working Ptolemy II installation. See eval/README.md for sweep scripts and configuration details.

The Claude-based agent is configured through eval/ptolemy_mcp_config.json. The GPT-5 cross-model probe reads the endpoint from the PTOLEMY_GPT5_BASE environment variable (defaults to the standard OpenAI-compatible endpoint).

Repository layout

ptolemy_mcp/          MCP server and Ptolemy II bridge (Python package)
eval/
  benchmarks/         Task specifications for Benchmark A and C
  results/per_run/    Pre-collected per-run trace JSON files (1 260 runs)
  aggregate_*.py      Aggregation scripts that produce the paper's tables
  runner.py           Evaluation harness entry point
eval_eda/             Verilog/RTL cross-environment evaluation (54 runs)
eval_systemc/         SystemC cross-environment evaluation (27 runs)
eval_simulink/        Simulink cross-environment evaluation (18 runs)
agent/                System prompt and case-study notes used during evaluation
demos/                End-to-end smoke demo and reference MoML models
examples/             Example MCP client configuration
scripts/              Helper scripts for running sweeps
reproduce_tables.py   Code Ocean entry point — regenerates all tables

Trace file format

Each file in eval/results/per_run/ is a JSON object with the following top-level fields:

Field	Description
task_id	Benchmark task identifier
config	Agent configuration name (C1–C7, OPT, …)
agent	Model used (claude, oracle, …)
run_ix	Replicate index
passed	Whether all properties were satisfied
wall_seconds	Wall-clock time for the run
grade	Numeric score (violation magnitude or 0 on pass)
tool_calls_total	Total MCP tool calls made
tool_calls_by_fn	Per-function tool call counts

License

See LICENSE.