N10 THE REALITY LAYER
What an AI Scientist Should Be Allowed to Do
Delegate by reversibility, evidence quality and the cost of being wrong.
IN THIS NOTE · APRIL 2025
The wrong question is whether AI can do science. The useful question is which scientific actions can be delegated, under what supervision, with what evidence and with what recovery path.
A ladder of delegation
At the lower rungs, an agent can search, organize and summarize public material. Higher rungs include running analyses, proposing hypotheses, designing experiments, controlling equipment and making consequential decisions. Capability may rise quickly, but permission should rise according to the risk of error.
Reversible work with inspectable inputs can tolerate more autonomy. Irreversible work involving organisms, patients, scarce samples or large budgets requires stronger gates.
Permission should be evidence-dependent
A system that performs well on a benchmark has earned confidence on that benchmark, not universal authority. Delegation should depend on task-specific validation, failure analysis, monitoring and clear escalation when assumptions break.
The agent should also know when evidence is too weak to continue. Refusal can be a scientific feature when it reveals missing context rather than hiding uncertainty behind prose.
The researcher remains part of the instrument
Researchers bring tacit knowledge about sample quality, biological meaning and what would be surprising. An effective agent makes that judgment easier to apply. It shows its plan, asks high-value questions and preserves continuity across sessions.
The objective is not to remove the scientist. It is to increase the amount of rigorous work a scientist can direct.
Permission should follow reversibility
The autonomy question becomes clearer when actions are ranked by reversibility, observability and consequence. Searching public literature is easy to inspect and cheap to repeat. Executing code against a copy of a dataset carries more risk but can be sandboxed. Ordering reagents, changing a live protocol, controlling laboratory equipment or making a claim about a patient introduces financial, physical or ethical consequences that cannot be undone by generating a better answer later.
This suggests a permission architecture rather than one autonomy setting. Tools receive scoped credentials, spending limits, approved data domains and explicit stop conditions. Higher-risk actions require evidence from lower-risk steps and a named reviewer. Permissions can expand after task-specific evaluation and contract after incidents. The system remains useful because routine work flows quickly, while consequential branches encounter friction proportional to what could be lost.
Design for interruption and recovery
An agentic workflow will eventually encounter a missing paper, malformed dataset, contradictory result, unavailable instrument or instruction that should not be followed. The quality of the system is revealed by how it stops. It should preserve state, identify the unresolved dependency, avoid silently substituting a weaker method and make the next safe action legible to the researcher. A brittle agent treats friction as an obstacle to route around; a scientific agent treats it as evidence.
Recovery also requires an incident record. Which plan was active, what tools ran, what data changed, what output reached another person and which assumption failed? NIST's risk-management framing emphasizes governance, mapping, measurement and management across the lifecycle. In research, those functions become concrete through logs, versioned artifacts, evaluation suites and post-incident changes to permissions. Autonomy is credible only when the operating system can learn from its own near misses.
- Grant autonomy according to reversibility and consequence.
- Validate on the exact workflow, not a nearby benchmark.
- Make uncertainty and refusal visible product behaviors.
I would revise the delegation ladder if general benchmark performance became a reliable substitute for task-specific validation in consequential research.
Primary and institutional sources used as the grounding layer. Interpretation and synthesis are Luca's.
01