ScientistTwo, a new autonomous research framework from Google Cloud AI Research and collaborators at the University of Waterloo, is designed to take a research problem from an initial human-defined challenge through experimentation, refinement, peer-review simulation and manuscript generation with no further human intervention in the research loop.
The system is described in a paper published on arXiv on September 17, 2026. Rather than focusing only on generating research ideas or writing papers, ScientistTwo attempts to automate the empirical cycle that researchers use to establish whether an idea actually improves on existing work.
The researchers evaluated ScientistTwo on 107 research problems drawn from papers accepted at ICLR, ICML and NeurIPS. They report that the system improved on the human state-of-the-art baseline in 86 of those 107 problems, representing an 80.4% success rate, with an average relative improvement of 25.2%.
Those results are the central claim behind ScientistTwo, although they should be understood as results reported by the authors rather than as independently reproduced evidence.
Quick Summary
- ScientistTwo is an autonomous AI research system developed by Google Cloud AI Research and collaborators.
- It can take a human-defined research problem and run much of the research cycle autonomously.
- The system generates research ideas, tests them on subsets of data and conducts ablation studies.
- It can use experimental results to refine its research approach.
- ScientistTwo also includes simulated peer review, rebuttal and meta-review stages.
- Researchers evaluated it on 107 problems based on papers from ICLR, ICML and NeurIPS.
- The researchers report improvements over human state-of-the-art baselines on 86 of 107 problems.
- Its generated papers were also evaluated by automated AI reviewers.
- The results demonstrate the potential of multi-agent systems for automating substantial parts of AI research, while the evaluation remains dependent on the researchers’ benchmark methodology.
How ScientistTwo Conducts Autonomous AI Research?
ScientistTwo is structured as a multi-agent system in which different agents handle different stages of the research process.
The workflow begins with a human-defined scientific problem. ScientistTwo first establishes a state-of-the-art baseline and analyzes its limitations. Instead of generating arbitrary research directions, the system creates seed ideas specifically aimed at addressing weaknesses identified in existing approaches.
The system then uses a subset-first evaluation strategy. Candidate ideas are initially tested on representative portions of the benchmark rather than immediately consuming the resources required for full experiments. Ideas that show promise can proceed to larger-scale testing.
This approach gives ScientistTwo a mechanism for filtering unsuccessful hypotheses before committing substantial computational resources. The DAIR.AI summary of the paper also highlights this subset-to-full-set process as one of the framework’s important design features.
Ablation Studies Let the System Refine Its Own Ideas
A key difference between ScientistTwo and simpler AI research pipelines is its use of automated ablation studies.
After an idea produces an improvement, ScientistTwo attempts to determine which components of the method are actually responsible for the gain. It removes or modifies components and evaluates the resulting changes, allowing the system to discard ineffective mechanisms and refine the underlying hypothesis.
The paper describes this as ablation-driven hypothesis refinement. Instead of simply retaining the first configuration that produces a better benchmark score, the system uses experimental evidence to evolve the proposed method.
This creates a feedback loop between hypothesis generation and experimentation: experimental results can change the idea itself rather than simply determining whether the original idea succeeded.
ScientistTwo Adds Automated Peer Review and Rebuttals
ScientistTwo also extends the research loop beyond experimentation and into manuscript evaluation.
Once a research result has been developed, a Writer Agent prepares the manuscript. A simulated Peer-Review Agent then critiques the work, while a dedicated Rebuttal Agent can design, implement and run additional experiments intended to address reviewer concerns.
A Meta-Review Agent supervises this process and can trigger further refinement when the manuscript does not satisfy the system’s acceptance criteria. The paper therefore treats peer review as another source of experimental feedback rather than simply as a text-editing stage.
The project website describes the architecture as a closed loop connecting idea generation, evaluation, analysis, writing, peer review and meta-review.
ScientistTwo Reported Results Across 107 Research Problems
The researchers tested the framework using problems associated with accepted research from several major AI conferences, including ICLR, ICML and NeurIPS.
| Measure | ScientistTwo result |
|---|---|
| Research problems evaluated | 107 |
| Problems improved over human SOTA | 86 |
| Reported success rate | 80.4% |
| Average relative improvement | 25.2% |
| Papers generated | 86 |
| ScholarPeer average rating | 7.5/10 |
| Stanford Agentic Reviewer average rating | 5.7/10 |
The 86 successful cases span areas including large language models, robotics, neuroscience, speech, robustness, reinforcement learning, game theory, privacy, optimization and time series. The project site says the generated work covers eight broad AI domains while the paper describes a wider set of research areas.
The results are notable, but the wording matters. “Beating human SOTA” in this evaluation means that ScientistTwo’s proposed methods achieved better reported benchmark performance than the human-designed baseline associated with the target problem. It does not mean that an AI system has surpassed human scientists in every aspect of scientific research.
AI Reviewers Give ScientistTwo Papers High Scores
The researchers also evaluated the generated manuscripts using two automated reviewers: ScholarPeer and the Stanford Agentic Reviewer.
ScientistTwo achieved an average score of 7.5 out of 10 from ScholarPeer and 5.7 out of 10 from the Stanford Agentic Reviewer. The project site reports that these scores exceeded the average ratings of the accepted ICLR 2026 and NeurIPS 2025 papers included in its comparison under both reviewers.
ScientistTwo’s reported acceptance rates were 91.9% under ScholarPeer and 72.1% under the Stanford Agentic Reviewer. The project comparison also includes several other automated research systems, which received substantially lower scores in this particular evaluation.
However, these are AI-generated review measurements, not evidence that human conference reviewers would necessarily give the same papers the same scores or accept them for publication. The DAIR.AI summary explicitly notes that the review comparison measures the automated reviewers as well as the system being evaluated.
The System Also Includes an Integrity Audit
The ScientistTwo project reports an additional integrity audit covering the generated research artifacts.
According to the project website, the audit re-runs released code to check reported numbers, checks for specification violations and reward hacking, verifies references and compares the written methodology against the implementation. The project reports zero hallucinated references among 1,814 references examined and says its audited papers passed the listed integrity checks.
These checks address an important weakness in automated scientific research: producing a convincing manuscript is not enough if its experiments cannot be reproduced or its claims do not match the underlying code.
The project therefore treats reproducibility and artifact verification as part of the research pipeline rather than as an afterthought.
What ScientistTwo Changes for Autonomous Research?
The significance of ScientistTwo lies less in automated paper writing and more in the breadth of the loop it attempts to automate.
Many AI research systems can generate hypotheses, write code or optimize a benchmark. ScientistTwo combines those functions with baseline reproduction, hypothesis screening, ablation experiments, iterative refinement and simulated peer review.
The authors describe the framework as requiring the human researcher to define the scientific challenge while allowing the AI system to determine much of the experimental path used to pursue it.
That makes ScientistTwo an example of a broader shift from AI assistants that help researchers perform individual tasks toward autonomous AI research agents that coordinate multiple stages of an investigation.
The current evidence, however, remains bounded by the researchers’ experimental setup. The 107 benchmark problems, automated reviewers, evaluation methodology and underlying computational resources define what the reported results demonstrate. The paper does not establish that ScientistTwo can independently originate important scientific problems, replace human scientific judgment across disciplines or guarantee real-world scientific breakthroughs.
For now, ScientistTwo provides a documented demonstration that an autonomous multi-agent system can take established research problems, develop and test new approaches, refine those approaches through ablations, and produce research manuscripts and executable code as part of a largely automated loop.
The reported 86 improvements out of 107 problems make ScientistTwo a significant development in autonomous AI research, while the reliance on benchmark tasks and automated evaluation means its claims should continue to be assessed through independent replication and broader human evaluation.
Also Read –
Persona Selection Model: Why AI Assistants Seem Human?
Source
Original ScientistTwo paper on arXiv
