ResearchLedger: evidence-first research automation

01Research problem

AI research assistants can now draft a convincing paper even when its experiments never ran, its citations were never checked, and its numbers cannot be traced back to data. Readers, and often the authors, cannot tell generated text from executed measurements or from independently established findings.

A second, quieter failure follows: when an upstream library, dataset or analysis step changes, earlier claims that depended on it can silently become stale while everything else in the project still looks valid.

02What is new

• A claim–evidence–run ledger. Claims (C###), evidence (E###) and executed runs (R####) are linked with content digests, so every reported number has a traceable origin.
• Evidence-aware writing. Manuscripts are drafted with empty result slots; numbers are inserted only from explicitly bound, executed outputs.
• Opt-in validation mode. A prospectively locked protocol and dataset, prior-art metadata checks, metric mappings, registered seeds and isolated repeats before anything is finalised.
• Transactional support-set revision. Dependencies are stored with AND/OR support semantics, so a revision invalidates the claims that truly lost support while keeping those with an alternative sufficient path.

The repository's comparison with other research agents is qualitative and architecture-based. It is not a claim that ResearchLedger outperforms those systems.

03Method

A command-line tool (researchctl) runs five stages and records each in the ledger:

1. S1 Idea selection — candidate ideas scored by two critical personas, with measurable falsification criteria.
2. S2 Draft — LaTeX manuscript with empty result slots.
3. S3 Review — structured referee assessment.
4. S4 Execution — generated experiments run in a network-restricted Docker sandbox; code, inputs and outputs are digested.
5. S5 Revision — evidence-bound revision, then an automated audit and researcher review.

Exploration mode (default) yields exploratory results. Validation mode must be enabled before the run and does not retroactively turn a pilot into a confirmatory study.

04Results

The current evidence is development and mechanism evidence recorded in the public repository, not external validation.

Structural pilot (10 authored events). Flat binary propagation kept precision but missed conjunctive dependency failures. Unconditional downstream reachability reached complete impact recall but over-invalidated objects that still had an alternative sufficient support path. Support-set provenance and full transactional revision produced the same impact sets, which attributes the gain to the AND/OR support representation rather than the transaction layer.

These are controlled-fixture results. They do not show that ResearchLedger has been validated on real research trajectories, that real agents register dependencies accurately, or that 13/13 estimates population accuracy. The pre-registered confirmatory study below is designed to test that.

05Reproducibility

git clone https://github.com/raja21068/ResearchLedger.git
cd ResearchLedger
python -m pip install -e .
python -m pytest -q tests tests_researchledger

06My contribution

Sole developer and maintainer. Every commit in the public repository (17 of 17, since September 2026) is by Rajesh Kumar, covering the system design, the claim–evidence–run ledger, validation mode, the Paper Factory stage implementations, the test suite and the validation-study bundle.

Bundled third-party components (Referee Lean v2, the Idea Discovery v5 archive and other vendored tools) keep their own licences and attribution and are not presented as original work.

07Limitations

• Not a certifier of scientific correctness, novelty, independent replication or journal acceptance; an audit pass confirms machine-checkable conditions only.
• External validation is pending the pre-registered confirmatory study.
• Tests do not exercise the live Claude Code, Docker and LaTeX workflow end to end.
• Vendored third-party code has not had a full line-by-line security review.

08Cite

@software{kumar2026researchledger,
  author  = {Kumar, Rajesh},
  title   = {ResearchLedger: An Evidence-First Research Workflow},
  year    = {2026},
  version = {2.0.1},
  url     = {https://github.com/raja21068/ResearchLedger}
}

ResearchLedger is related to, but separate from, AutoResearch. The AutoResearch paper's results should not be read as validation of this repository.

More case studies