TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A report by Nolan Lawson compared an AI agent building the same Pokémon app in 12 JavaScript frameworks and related tools. Median reported API costs ranged from $2.67 for Hono with htmx to $4.97 for Angular, but the author says run-to-run variation was substantial and the test covers only one kind of app.
Developer and blogger Nolan Lawson has published a benchmark comparing how an AI coding agent builds the same web app with 12 JavaScript frameworks and related tools, reporting median API costs from $2.67 to $4.97 per run. The results offer a measured comparison amid claims that particular frameworks make AI agents faster or more reliable, but Lawson cautions that the test is limited and the results vary between runs.
Lawson asked Opus 5.5, run at high effort, to build a small Pokémon reference app using the latest version of each selected framework. The app drew on the PokéAPI and included Pokémon listings, types, moves, game data, search and filters. Frameworks tested included Next.js, Nuxt, SvelteKit, Astro and Angular, alongside options such as Hono with htmx and a vanilla JavaScript implementation.
For most tools, Lawson reported the median of five builds. Hono with htmx had the lowest median API-price estimate, at $2.67; Astro came in at $2.76, Nuxt at $2.82 and vanilla JavaScript at $2.98. Next.js was $3.14. Angular’s median was $4.97, but it had only three measurable runs because two attempts failed to produce an app that could be measured. Lawson listed the costs using Claude Code API prices; he said they were not his out-of-pocket spending, since he was on a monthly Max plan.
The test also evaluated development time, token use, lines of code, and Lighthouse performance and accessibility scores. Lawson said he used a spec-conformance suite based on the accessibility tree to check whether outputs met requirements without requiring a particular DOM structure. The supplied material reports the cost figures but does not provide the full results for the other measures, so those outcomes cannot be compared here.
What the Cost Gap Shows
The report gives teams a concrete, if narrow, way to question claims that one JavaScript framework is inherently cheaper for AI-assisted development. In this benchmark, the difference between the lowest and highest reported median was $2.30 per build. That amount may be small for a single experiment, but the comparison could matter when organizations repeat agent-driven work across many projects or use it as part of a broader framework evaluation.
The figures do not establish that a framework will reduce a company’s overall development costs. They cover the agent’s reported API-price estimate for one app, not engineering review, maintenance, hosting, or the expense of integrating generated code into an existing product. Lawson also measured quality-related outcomes, but cost alone cannot show whether a cheaper build was more usable, accessible, performant or easier to maintain.
For developers, the practical takeaway is to treat framework-for-agent claims as questions for local testing, not settled conclusions. Results may depend on the task, model, tool instructions, framework-specific guidance and the number of repeated attempts.
Products Worth Considering
Samsung Odyssey G4 25-inch FHD Gaming Monitor
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
Apple Watch Series 12 GPS 46mm Smartwatch
As an affiliate, we earn on qualifying purchases.
How Lawson Set Up the Test
Lawson said he designed the experiment in response to claims that frameworks are “better for agents” or lead to faster code and fewer mistakes. His stated aim was to test those claims rather than assume that developer preference or framework reputation predicts an AI agent’s results. He used the same app specification across frameworks and gave each tool its official MCP or skills, the framework-specific resources intended to guide the agent.
The instructions did not prescribe a rendering architecture such as server-side rendering or client-side rendering. Lawson also did not require a particular CSS methodology, though he excluded Tailwind, Bootstrap and other component frameworks, and supplied design tokens to keep visual expectations consistent. Agents were not told that Lighthouse would assess performance and accessibility, according to his report.
Lawson described the Pokémon app as one limited example: a content-focused site with relatively little interaction, closer to a reference resource than a complex application such as a design tool or email service. He said multiple runs were used to account for variability in both agent output and Lighthouse measurements, while acknowledging that neither source of variation could be eliminated.
Limits of the Framework Comparison
The findings should not be read as a general ranking of framework costs. The benchmark uses one application, one model setup and a small number of runs; different tasks or coding agents could produce different results. Lawson also reported substantial variation between attempts, including overlap in cost ranges: the cheapest React Router run was below the most expensive Hono with htmx run.
Lawson said AI outputs can vary because agents may take different paths through the task, while Lighthouse can produce inconsistent measurements, including changes in which item it identifies as the largest contentful paint element. He used medians to reduce the effect of unusual results, but the source does not establish that these figures predict costs for other teams, models or production applications.
The supplied report excerpt does not include the detailed results for development time, token use, lines of code, performance, accessibility or maintainability. It also does not show whether framework rankings would hold across repeated experiments by other researchers. Those comparisons remain unconfirmed by the reported cost data.
Further Tests Needed Across Apps
The report describes a benchmark rather than a change to framework software, so it does not announce a product release or a scheduled follow-up. The immediate next step for readers evaluating the findings is to compare them with the report’s full results and methods, including the measures beyond cost, before drawing conclusions about a framework.
Broader evidence would require repeating the comparison across more application types, models and runs, and reporting the range of results alongside medians. Teams applying the findings to their own work would also need to test representative tasks with their chosen model, instructions and framework versions. Until then, the published figures are best treated as a limited benchmark of one agent building one content-focused app, not a universal measure of AI development cost.
Key Questions
Which framework had the lowest reported AI build cost?
Hono with htmx had the lowest median, at $2.67 per run, based on five builds. The amount is an API-price estimate reported by Lawson, not necessarily what a developer would pay for a particular project.
Which framework had the highest reported median?
Angular had the highest listed median, at $4.97. Lawson used three measurable runs for Angular because two attempts failed to create an app he could measure.
Does the benchmark prove that Hono with htmx is cheaper for all projects?
No. It tests one content-focused Pokémon app with one agent setup. Lawson said results vary and warned that the experiment is not a perfect representation of other types of applications or development workflows.
Did the test measure more than cost?
Yes. Lawson said he evaluated development time, token use, lines of code, and Lighthouse performance and accessibility scores, as well as conformance to the app specification. The supplied report material gives the cost table but not the full results for those other measures.
Source: nolanlawson.com
