Claims about AI productivity are everywhere, but rigorous evidence is rarer. One of the most cited early studies is a controlled experiment on GitHub Copilot, the AI coding assistant. It offers a clear example of what AI can do on a well-defined task — and why results need careful interpretation. See GitHub Copilot in our tools directory.

The study

In 2023, researchers Sida Peng (Microsoft Research), Eirini Kalliamvakou (GitHub), Peter Cihon (GitHub) and Mert Demirer (MIT) published The Impact of AI on Developer Productivity: Evidence from GitHub Copilot.

Design:

  • Developers were recruited and randomly assigned to two groups.
  • The treatment group had access to GitHub Copilot; the control group did not.
  • Both groups were asked to implement an HTTP server in JavaScript as quickly as possible.
  • Completion time and success were measured.

Random assignment is what makes this a strong design: differences between groups can be attributed to Copilot rather than to who chose to use it.

The results

The treatment group completed the task 55.8% faster on average than the control group. The researchers also found suggestive evidence that developers with less programming experience, older developers and those who program more hours per day benefited more.

GitHub also published survey research in which developers using Copilot reported feeling more productive, less frustrated and better able to focus on satisfying work.

Why the result is meaningful

  • Randomized experiment: stronger evidence than surveys or before-and-after comparisons.
  • Objective measure: time to complete a task that was checked with tests.
  • Large effect: a speed-up of more than half is substantial for any tool.

Limitations

  • A single, well-defined task. Implementing a standard HTTP server is the kind of common, well-documented problem where AI assistance is strongest. Real software work involves understanding large existing codebases, unclear requirements, design decisions, debugging and coordination.
  • Short duration. Long-term effects on code quality, maintainability and skills weren’t measured.
  • Speed, not quality. Faster completion doesn’t show whether the code was as secure or maintainable.
  • Participants. Recruited freelancers may not represent all developers.
  • Tools change. Coding assistants and agents have evolved substantially since the study.

What later evidence suggests

Subsequent studies in real companies have generally found positive but more modest productivity effects, with variation by task and experience. Some research has raised concerns about code quality, security issues in generated code, and over-reliance; one later study even found experienced open-source developers were slower with AI tools on their own complex repositories. The picture is nuanced: AI helps most on certain tasks and for certain people.

Lessons for teams

  1. Expect the biggest gains on well-defined, common tasks: boilerplate, tests, documentation, standard patterns.
  2. Measure in your own context: track cycle times, review effort and defect rates before and after adoption.
  3. Invest in review: faster code generation shifts the bottleneck to code review and testing.
  4. Support learning: juniors benefit, but should still understand the code they ship.
  5. Set security practices: scan for vulnerabilities and avoid accepting suggestions blindly.

See our guide on how to use AI coding assistants effectively.

How to run a pilot in your team

  1. Pick two or three comparable teams or a set of similar tasks.
  2. Measure baseline metrics for a few weeks: cycle time, pull request size, review time, defects.
  3. Introduce the tool with training and clear guidelines.
  4. Compare metrics after several weeks, and survey developers about their experience.
  5. Review a sample of AI-assisted code for quality and security.

What to watch next

  • Long-term studies of AI coding tools on code quality and maintenance.
  • Productivity evidence for autonomous coding agents.
  • Effects of AI assistance on how junior developers learn.
  • Security of AI-generated code in production.

Key takeaways

  • A randomized experiment found developers with Copilot completed a task 55.8% faster.
  • Less experienced developers appeared to benefit more.
  • The task was well-defined, so results may overstate gains on complex real work.
  • Later studies show positive but more variable effects.
  • Teams should measure locally, strengthen review and manage security.

Frequently asked questions

Does Copilot make developers twice as fast?

In this experiment, developers completed one specific task about 56% faster. Gains in everyday work vary widely.

Does AI-generated code have more bugs?

It can contain errors and vulnerabilities, so it needs review and testing like any code.

Who benefits most from coding assistants?

Evidence suggests less experienced developers and common, well-defined tasks see the largest gains.