01Research problem
Research agents increasingly write code, retrieve literature and draft scientific artifacts, but they often do not check whether the generated experiments actually execute, or whether the cited sources actually support the claims. The result is output that looks complete but fails when run or read closely.
02What is new
- An execution-grounded reliability pattern. Runtime errors, citation-verification failures and review-agent feedback are treated as filtering signals for generated artifacts, not just as messages to the user.
- One loop instead of separate tools. Sandboxed Python/PyTorch execution, iterative code repair, citation verification, claim-support auditing, decision control and structured LaTeX generation are coupled in a single multi-agent workflow.
- Explicit scope. The paper positions AutoResearch as a reliability-oriented research assistant, not a fully autonomous scientist or a manuscript-quality benchmark.
03Method
Specialised agents cooperate under an orchestrator:
- Orchestrator and planner decompose the task and control stage transitions.
- CodeAgent generates code and repairs it after sandbox failures (self-healing loop).
- Experiment agent runs experiments and compares against baselines.
- Research agent and verification engine retrieve literature, verify citations and audit claim support.
- Review agents critique outputs; a decision engine chooses the next action.
- MetaClaw reuses learned repair skills across runs.
04Results
| Method | Invalid citations ↓ | Execution success ↑ | Reproducibility ↑ | HumanEval | MBPP |
|---|---|---|---|---|---|
| Vanilla LLM | 38% | 41% | 0.42 | 42.1% | 47.3% |
| RAG-only | 31% | 46% | 0.48 | 49.4% | 52.1% |
| ReAct | 32% | 55% | 0.50 | 58.2% | 61.5% |
| AutoGPT | 33% | 58% | 0.51 | 61.3% | 63.9% |
| Retry-only (with memory) | 32% | 64% | 0.53 | 68.2% | 69.5% |
| AutoResearch | 3.5% | 79% | 0.68 | 88.1% | 87.4% |
End-to-end stress test. AutoResearch completed 11 of 15 CrossDomain-15 tasks (73.3%). In the ablation, removing sandboxed execution dropped completion to 40% and removing the verification layer to 48%. Code-oriented agents are reported separately as partial comparisons: SWE-agent and OpenHands reached 68% and 70% execution success.
The paper treats these as evidence within its controlled experimental design, not as universal rankings. CrossDomain-15 is a small stress test rather than a broad benchmark.
05Reproducibility
| Item | Details | Status |
|---|---|---|
| Source code | Public on GitHub, MIT licence | Available |
| Paper and prompt library | arXiv, CC BY 4.0; prompts in Appendix F | Available |
| Artifact checklist | Task IDs, prompts, raw logs, citation and claim-support annotations, cost logs and table scripts are defined in Table XVII | Defined |
| Frozen release identifiers | Repository commit and Docker image digest are listed as "TBD" in arXiv v1 | Not yet |
| Archived artifact record | To be confirmed with the frozen release | Pending |
git clone https://github.com/raja21068/AutoResearch.git
cd AutoResearch
# follow the README; benchmark task IDs are listed in the paper's appendix06My contribution
First author of five (Rajesh Kumar, Waqar Ali, Junaid Ahmed, Abdullah Aman Khan, Shaoning Zeng) and sole committer to the public repository (17 of 17 commits).
07Limitations
- Remaining failure modes in the logged failure events: incorrect tool usage (12%), noisy retrieval (8%) and unstable repair loops (5%).
- Open-ended creative search remains weak, and one of five ML pipelines (a VAE with a custom loss) could not be repaired.
- Sandboxed execution shows code runs; it does not establish scientific correctness or novelty.
- Frozen release identifiers are still to be published.
08Cite
@misc{kumar2026autoresearch,
title = {AutoResearch: An Execution-Grounded Multi-Agent Framework for Reliable Research Workflow Automation},
author = {Kumar, Rajesh and Ali, Waqar and Ahmed, Junaid and Khan, Abdullah Aman and Zeng, Shaoning},
year = {2026},
eprint = {2607.02520},
archivePrefix = {arXiv},
primaryClass = {cs.CY},
doi = {10.48550/arXiv.2607.02520}
}AutoResearch is a separate framework from ResearchLedger, which adds an explicit evidence ledger and validation gates.