The Ada Lovelace Institute has warned that the productivity figures driving government investment in public sector AI rest on layered assumptions, one-sided survey framing and circular use of large language models to assess their own capabilities, and has published a framework of questions for policymakers and journalists to interrogate such claims before relying on them.
The briefing, Measuring up: evaluating claims about AI and productivity in the UK public sector, said that when the institute examined the evidence it found a much more mixed picture than the headline figures suggested, and that numbers for their own sake were not enough without a sharper focus on whether productivity claims translated into public benefit. Policymakers, it added, needed to understand what sat behind the topline figures before relying on them in critical decisions.
The briefing's authors - Sumedha Deshmukh, an affiliate fellow at the institute, with Matt Davies, its social and economic policy lead - noted that the Spending Review 2025 committed to a step change in digital and AI investment across public services, informed by government estimates of savings running as high as £45 billion a year, a figure from the State of Digital Government Review whose methodology was later revised to £23.6 billion.
Other estimates cited range from the Tony Blair Institute's £199 billion over five years to IPPR's £24 billion a year. The wide range, the institute says, is rarely acknowledged and the reasons for it rarely examined, so caveats in the underlying studies get lost by the time figures reach decision-makers.
The institute identified two dominant methodologies. Task automation studies break jobs into bundles of tasks, estimate what proportion could be automated and convert the result into salary savings. The briefing lists assumptions found in the government's own £45 billion calculation, including that staff spend equal time on every task, that 100 per cent of routine and 10 per cent of non-routine tasks can be automated, and that only salary costs are relevant, with no sensitivity testing of those assumptions and no allowance for implementation cost or transition periods.
Many such studies rely on the US O*NET occupational database, and a growing number use LLMs to classify which tasks LLMs can perform, a practice the briefing traces to a 2023 OpenAI paper that has since been incorporated into the government's Digital and Data Benefits Framework despite subsequent replication work finding it systematically overestimated automation potential.
The second methodology, self-reported time-use surveys, is informed by two government Microsoft 365 Copilot pilots run in summer 2025. The cross-government experiment, framed around whether AI tools improve quality and reduce time on routine tasks, produced the headline that civil servants could save nearly two weeks a year.
The Department for Business and Trade's evaluation, which also asked about additional workloads, hallucinations and quality assurance, found small time savings but no productivity improvement, with some tasks taking longer. The institute says the contrast shows how research framing shapes survey design and results, and cites a separate study in which software developers estimated AI made them 20 per cent faster when measured output showed they were 20 per cent slower.
The briefing also raised concerns about industry involvement across the research lifecycle, from funding university partnerships and think tanks to supplying proprietary usage data and tools, noting documented cases of researchers leaving AI companies over difficulty publishing negative findings. Workers and citizens, it says, are largely absent from research design, and time-bound pilots rarely include follow-up to test whether early gains persist once novelty effects fade.
Davies said the stakes were high, with claims about AI's impact on productivity influencing major government decisions, and that policy and investment had to be informed by rigorous research and clear-eyed assessment of public benefit and value for money. Policymakers, he said, should not lose sight of the public value AI adoption was meant to deliver.
The institute recommended that future studies report ranges rather than point estimates, measure service quality, error rates, equity and staff wellbeing alongside time and cost, triangulate across methods, and treat productivity effects as context-specific. It calls on funders to support independent, peer-reviewed research with published methodologies and data, repeats its earlier call for a What Works Centre for AI in public services, and urges departments to build measurement in from the start of any deployment and to acknowledge implementations that do not deliver.
