Your developers swear AI coding tools are making them faster. They feel more productive. They’re cranking out more code than they have in years.
Here’s the problem: METR’s 2025 randomised controlled trial measured experienced developers and found they actually took 19% longer to complete tasks with AI assistance. The kicker? Those same developers estimated they were 20% faster.
At the same time, EY’s Australian AI Workforce Blueprint claims daily AI users are saving four or more hours per week. That’s a pretty big gap between perception and reality, and it makes it hard to work out whether you should be spending $10,000+ per developer every year on AI coding tools.
This guide is part of our comprehensive look at how AI is transforming Australian startups in 2025, where we explore the practical realities of AI adoption based on the latest research and industry data.
In this article we’re going to dig into what the research actually shows, why developers can’t accurately judge their own productivity with AI, and when these tools genuinely help versus when they’re creating more work than they save.
What Is the AI Productivity Paradox in Software Development?
AI coding tools are boosting individual code output but failing to improve how fast teams ship. In controlled studies, they actually slow things down despite developers feeling faster.
Take Faros AI’s 2025 productivity report. They analysed 10,000+ developers across 1,255 teams. Developers using AI completed 21% more tasks and merged 98% more pull requests. But company-level DORA metrics—deployment frequency, lead time, change failure rate—stayed flat.
The contradiction is straightforward. Developers feel faster because they’re typing less and seeing instant suggestions. But teams aren’t shipping any faster. Sometimes they’re shipping slower.
Microsoft and Accenture studied 4,800 developers and found 26% more completed tasks and 13.5% more code commits. Yet there’s no correlation between AI adoption and key performance metrics at the company level.
Why? Review time ballooned by 91% in high-AI teams because the human approval loop became the choke point. Average PR size increased by 154%, making reviews take longer. More code gets written, sure, but review queues grow faster than the code can move through them.
Nicole Forsgren, the creator of the DORA framework, describes AI as a “mirror and multiplier” that magnifies the strengths of high-performing organisations and the dysfunctions of struggling ones. Individual throughput goes up. Team velocity stays the same or drops.
Why Do Developers Think They’re Faster With AI When Research Shows They’re Slower?
This perception gap isn’t just a measurement problem—it’s psychological. The METR study put experienced developers in a controlled environment with familiar codebases and complex tasks. Developers expected AI to make them 24% faster. Tasks actually took 19% longer. After completing the study, developers still believed they worked 20% faster—a 39% gap between feeling and reality.
AI autocomplete and instant suggestions create a sense of cognitive ease and reduce typing effort, which feels like productivity even when the measurements show otherwise. The sensation of flow while you’re coding doesn’t correlate with actual feature delivery velocity. AI gives developers confidence and reduces mental pressure, creating a sense of progress even when real gains are small.
Developers also don’t have visibility into the downstream impacts. They finish writing code faster, so they feel productive. They don’t see the extra review time, the integration delays, or the debugging overhead their AI-assisted code creates later on.
AI helps with typing and syntax—visible benefits you notice immediately. It creates problems in logic and architecture that only surface later when someone else reviews the code or a bug appears in production. You remember the speed. You don’t connect it to the problems.
How Does Task Complexity Affect AI Coding Assistant Performance?
AI gives you a speedup for simple, repetitive tasks—boilerplate, API calls, common patterns, test scaffolding. For complex tasks requiring context, AI slows developers down on architecture decisions, business logic, algorithm design, and debugging unfamiliar code.
The METR study focused on experienced developers working on familiar codebases. Even in that scenario, AI showed slowdowns on complex tasks. AI works as a syntax assistant, not a software engineering assistant.
GitHub Copilot acceptance rates show the split clearly: 55% for simple completions versus 18% for complex logic. Developers spend less time on boilerplate generation and API searches, but code-quality regressions and subsequent rework offset the headline gains as tasks grow more complex.
AI can handle CRUD operations, repetitive transformations, documentation, and simple tests. It struggles with state management, concurrency, security-sensitive code, and performance optimisation.
Why? AI lacks full codebase understanding, architectural awareness, and business requirement nuance. It doesn’t retain memory across sessions, generating isolated fragments without understanding your long-term architecture. It can’t make trade-offs between speed, maintainability, and scalability because it doesn’t know your priorities.
The trade-off comes down to time saved on typing versus time lost to evaluating, debugging, and rewriting AI suggestions. For simple tasks, the typing savings win. For complex tasks, the evaluation cost dominates.
What’s the Difference Between METR’s 19% Slowdown and EY’s 4+ Hours Weekly Savings?
METR used a randomised controlled trial with experienced developers on defined tasks, measuring task completion time directly. EY used a self-reported survey across mixed roles and AI applications, asking workers to estimate time saved.
The populations differ. METR studied senior developers on familiar codebases. EY surveyed 1,003 Australian computer-based workers across different roles including non-technical AI use cases. Only 26% of workers use AI daily, and of those, 30% say it saves four or more hours per week.
The tasks differ too. METR tested actual coding tasks requiring architecture and business logic. EY included email drafting, document summarisation, and other AI applications beyond coding.
Both studies can be “right” for different contexts. RCT results capture what happens under controlled conditions with complex development work. Field deployment surveys capture what happens in messy reality with a mix of simple and complex tasks across different job functions.
JPMorgan Chase reported 10-20% efficiency gains in production deployment—middle ground between METR and EY.
The lesson here: both studies are valid in different contexts. You need to measure your specific situation with your team, your codebase, and your task mix.
How Do AI Tools Affect Code Review Queues and Team Velocity?
AI-assisted developers produce 26% more commits, but each one requires human review. Review capacity stays constant while code volume increases, creating queues that wipe out individual gains.
Developers on high-AI teams touch 9% more tasks and 47% more pull requests per day, increasing context switching. The net effect: individual speedup gets negated by team-level slowdown in code review and integration stages.
Quality concerns make the problem worse. AI-generated code requires more careful review for logic errors, security vulnerabilities, and architectural fit. Bug rates increased 9% per developer with AI adoption.
DORA metrics show the impact. Deployment frequency remains unchanged or reduced despite increased commit frequency. Lead time for changes increased due to review queues despite faster initial coding.
Amdahl’s Law applies here. AI-driven coding gains evaporate when review bottlenecks, brittle testing, and slow release pipelines can’t match the new velocity. You’ve optimised one step in a multi-step process, and now a different step has become the bottleneck.
More frequent small PRs interrupt reviewer flow state, reducing overall team productivity. Individual developers feel faster while the team as a whole slows down.
What Is the Real Cost Per Developer for AI Coding Tools?
GitHub Copilot Enterprise costs $39 per developer per month ($468 annually). Cursor Pro costs $20 per developer per month ($240 annually). Heavy Claude API usage runs $800+ per developer per month ($9,600+ annually).
But those subscription costs are just the start. Total Cost of Ownership runs $10,000-15,000 per developer annually for heavy usage when you include training time, governance setup, review overhead (20-30% increase), and debugging AI-generated code.
Before committing to these costs, it’s worth understanding the key differences in choosing between AI coding tools and what each offers for your specific use cases.
Time spent evaluating AI suggestions could be spent on other productivity improvements: better CI/CD, improved developer experience platforms, reducing technical debt.
How Should You Measure AI Coding Tool Effectiveness?
Start with a baseline: measure current productivity using DORA metrics, SPACE framework, or DX Core 4 before deploying AI tools. Without baseline measurements, you can’t tell the difference between natural variation and AI impact.
Avoid self-reports. Use objective telemetry—deployment frequency, lead time, code quality metrics—rather than developer surveys. Self-reports have that 39% perception gap we talked about earlier.
Run a pilot programme: deploy to a subset of your team with a control group, measure for 60-90 days minimum. Key metrics to track: team velocity (not individual output), code review time, defect rates, deployment frequency, lead time for changes.
DORA metrics capture delivery speed and stability. The SPACE framework gives you developer-centric signals across Satisfaction, Performance, Activity, Communication, and Efficiency. DX Core 4 combines flow state, feedback loops, cognitive load, and developer experience.
If your pilot shows gains after 90 days with objective metrics, you’ve got a case for broader rollout.
When Do AI Coding Tools Actually Help Versus Hurt Productivity?
AI helps with simple boilerplate, unfamiliar syntax, documentation, test scaffolding, and onboarding to new codebases. AI creates slowdowns with complex business logic, security-sensitive code, architecture decisions, and debugging unfamiliar systems.
Junior developers benefit more from syntax help. Senior developers get slowed down by evaluating suggestions. AI helps most when you’re learning a new language or framework but provides less value on familiar codebases where developers already know the patterns.
Strategic use cases: documentation generation, test generation, code translation, learning new APIs. Anti-patterns: using AI for architecture, security, or performance-critical paths.
AI performs best when given clear instructions, detailed requirements, and well-defined design. Think of AI as an army of gifted junior developers—good at implementation, not product management or architecture. This is why training teams effectively on how to work with AI tools is critical to getting any value from them.
FAQ
What is the most reliable research on AI coding tool productivity?
METR’s 2025 randomised controlled trial is the most rigorous, using experimental methodology with control groups and objective measurement. For broader scenarios, have a look at GitHub/Accenture, Faros AI, and enterprise case studies like JPMorgan Chase. No single study captures all contexts.
Why don’t developers notice they’re slower when using AI tools?
Self-report bias creates a 39% perception gap. AI autocomplete creates cognitive ease and reduces typing effort, which feels like productivity. Developers don’t have visibility into downstream impacts like increased review time and debugging overhead. The immediate sensation of faster coding doesn’t correlate with actual feature delivery velocity.
Should startups invest in AI coding tools like Cursor or GitHub Copilot?
Depends on your team composition, task types, and measurement capability. Invest if you can establish baseline metrics, run a proper pilot with a control group, measure team velocity, afford the full TCO of $10-15k per heavy user annually, and re-engineer review processes. Skip it if you can’t measure objectively or if your work is primarily complex architecture and business logic.
How do I calculate ROI for AI coding tools before purchasing?
Start with baseline DORA or SPACE metrics. Calculate full TCO including subscriptions, training, review overhead, and debugging time. Run a 60-90 day pilot measuring team velocity, deployment frequency, lead time, and defect rates. ROI = (productivity gain) – (total cost). Measure team outcomes, not individual output.
Which AI coding tool is best for experienced developers?
No single tool wins across all scenarios. GitHub Copilot Enterprise integrates best with Microsoft ecosystems. Cursor provides superior AI-native editing for independent work. Claude Code excels at complex reasoning but costs more. METR showed experienced developers slowed down regardless of tool on complex tasks. Focus on task fit and measurement.
Do junior developers benefit more from AI coding tools than senior developers?
Evidence suggests yes, but with caveats. Juniors gain more from syntax assistance and learning common patterns. However, juniors may learn bad patterns from AI suggestions or develop over-reliance on them. Senior developers already know syntax, so AI creates more evaluation overhead than value on complex tasks. Both groups need training on effective AI use.
How long does it take to see real productivity gains from AI coding tools?
Be wary of immediate “gains”—these are often self-reported perception, not measured outcomes. Legitimate gains require 60-90 days minimum to account for the learning curve and process adaptation. Teams must re-engineer code review and establish measurement baselines. JPMorgan Chase reported 10-20% gains after full rollout with process changes.
What are the security risks of using AI coding assistants?
AI tools introduce code learned from public repositories, potentially including vulnerable patterns and outdated security practices. Risks include: leaked proprietary code in prompts, generated code with SQL injection or XSS vulnerabilities, licence compliance issues, and reduced security review rigour. Mitigate with: local-only models where possible, security-focused review processes, automated security scanning, and governance policies.
Can AI tools help with legacy codebase modernisation?
Mixed evidence. AI excels at mechanical transformations: language migrations, syntax updates, API translation. AI struggles with understanding business logic, making architectural decisions, and handling technical debt. Best use: generate initial migrations for review, document undocumented code, write tests. Worst use: autonomous refactoring of complex business logic. Expect 20-30% time savings on mechanical work, not holistic modernisation.
How do AI coding tools affect technical debt accumulation?
AI generates syntactically correct code that may violate architectural principles or create maintainability issues. Faros AI studies show increased code volume without proportional increase in delivered features. Debt shows up as: harder-to-understand code, inconsistent patterns, and missed abstractions. Mitigation requires: architectural review for AI-generated code, stronger linting rules, explicit design discussions, and prioritising code quality over generation speed.
What’s the difference between AI code completion and AI chat assistance?
Code completion (like GitHub Copilot) suggests next lines as you type—faster but less controllable, better for simple patterns. Chat assistance (like Cursor’s composer, Claude Code) lets you describe what you want—slower but more precise, better for complex tasks. Completion works for boilerplate; chat helps with unfamiliar APIs and learning. Neither solves complex architecture or business logic reliably. Choose based on task type.
How do I convince my team to objectively measure AI tool impact?
Frame it as de-risking a significant investment of $10k+ per developer annually. Propose a pilot programme: select 30% of your team, measure for 90 days against a control group using DORA or SPACE metrics. Emphasise testing whether it works for your context, not questioning developers’ experience. Share the research showing a 39% self-report bias gap. Position measurement as finding the truth together. If tools truly help, measurement proves the case for broader rollout.
Wrapping it all up
The AI productivity paradox reveals a challenge in evaluating coding tools: the metrics that feel important—code output, typing speed, individual task completion—don’t correlate with the metrics that matter—deployment frequency, lead time, and team velocity. The gap between perception and measurement means you can’t rely on developer feedback alone.
As the Startup Muster 2025 findings show, Australian startups are rapidly adopting AI tools, but adoption without measurement creates risk. Measure objectively, account for the full system, and focus on what your team actually ships rather than what individual developers produce.
