📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
AI systems are increasingly capable of automating core engineering tasks in AI research, reaching near-saturation levels in benchmarks. However, automation of the research process itself remains limited, raising questions about future progress and institutional responses.
Recent developments in AI capabilities reveal that automation has largely transformed the engineering side of AI research, with systems now approaching or reaching saturation on key benchmarks. Meanwhile, the automation of AI research—such as generating novel hypotheses or designing experiments—remains incomplete, marking a significant shift in the landscape of AI development.
Multiple independent benchmarks, including CORE-Bench and MLE-Bench, show AI systems nearing or achieving saturation in core engineering tasks. For example, CORE-Bench, which measures the ability to reproduce research papers, reached 95.5% success with one system being declared ‘solved’ by its author. Similarly, MLE-Bench, evaluating performance on Kaggle competitions, saw scores rise to 64.4%, approaching professional-level performance.
These progress markers indicate that AI can handle the technical and operational aspects of AI research—such as reproducing experiments and optimizing kernels—at a level comparable to experienced researchers. Experts like Thorsten Meyer interpret this as evidence that engineering is effectively automated. Conversely, the research process—such as formulating new hypotheses or designing innovative experiments—remains less automated, with the boundary between engineering and research blurring but not fully eliminated.
Furthermore, the pace of progress across these benchmarks suggests a structural shift: capabilities are reaching measurement limits, and the rate of improvement may slow as systems approach the upper bounds of current benchmarks. The pause in leaderboard submissions for MLE-Bench reflects this saturation, underscoring the challenge of measuring further gains.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
AI Tools for Finance and Accounting Professionals: Automate Tasks, Save Hours, Work Smarter
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
![Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results](https://m.media-amazon.com/images/I/415+fSJacsL._SL500_.jpg)
Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
AI Workflow Tools for Researchers & Analysts: Automating Literature Reviews, Summaries, and Hypothesis Generation with ChatGPT, Claude, and Perplexity
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications for AI Development and Research Strategies
This shift signifies that AI systems are transforming the operational side of AI research, reducing the need for manual engineering work and potentially accelerating development cycles. However, the residual role of human researchers in generating novel ideas and guiding research remains significant. The institutional response—such as investing in automation tools—may need to adapt, recognizing that inspiration and creativity could become the new bottlenecks rather than engineering execution.
For the broader AI ecosystem, these developments suggest a future where the focus shifts from engineering to research innovation, raising questions about how institutions will manage the evolving role of human expertise and what new skills will be prioritized.
Progress in AI Engineering Capabilities and Benchmark Saturation
Since 2024, multiple benchmarks have tracked AI progress in core research tasks. Notably, CORE-Bench, measuring research reproduction, improved from 21.5% to 95.5% within fifteen months, with the final 4.5% likely within the noise floor. Similarly, Kaggle-based MLE-Bench scores increased from 16.9% to 64.4% over sixteen months, indicating AI reaching competitive levels in practical tasks. These trends align with a broader pattern of rapid capability accumulation across different engineering domains, including kernel design and infrastructure optimization, as documented in recent research papers and industry developments.
Experts like Thorsten Meyer interpret this as evidence that AI has effectively automated large parts of the engineering process, with some benchmarks now approaching their measurement limits. The question remains whether research—more creative and hypothesis-driven—can be similarly automated at scale, or if it will remain a predominantly human activity for the foreseeable future.
“AI can today automate vast swatches, perhaps the entirety, of AI engineering. The residual research question is how much of AI research it can automate, given that some aspects may be distinct from engineering skills.”
— Thorsten Meyer
Uncertainties in Automation of AI Research Creativity
It is not yet clear whether AI can fully automate the creative, hypothesis-generating aspects of research, such as designing novel experiments or formulating new theories. While engineering tasks are approaching saturation, the residual role of human insight and inspiration remains uncertain, and the pace at which these aspects might be automated is still unknown.
Next Steps for Measuring and Expanding AI Capabilities
The immediate focus will be on developing more refined benchmarks to measure the upper limits of AI capabilities in research and engineering. Additionally, research institutions and industry players are likely to invest in tools that further automate experimental design and hypothesis generation, testing whether the automation trend extends beyond engineering tasks. Monitoring how these developments influence research productivity and innovation will be critical in the coming months.
Key Questions
What does automation of engineering mean for AI research teams?
It suggests that many operational tasks—such as reproducing experiments, optimizing code, and infrastructure management—can now be handled by AI, potentially reducing manual workload and speeding up project timelines.
Can AI fully replace human researchers in AI research?
While AI has made significant strides in automating engineering tasks, the creative aspects of research—such as hypothesis generation and experimental design—are still largely human-driven, and it is unclear when or if these can be fully automated.
What are the risks of over-relying on AI for research?
Over-reliance could lead to a lack of diversity in research ideas, potential stagnation in innovation, and challenges in verifying AI-generated hypotheses, making human oversight still essential.
How will benchmarks evolve to measure AI research capabilities?
Expect more sophisticated benchmarks that test creative and hypothesis-driven tasks, alongside existing operational metrics, to better understand AI’s full potential in research contexts.
Source: ThorstenMeyerAI.com
