In March 2023, shortly after GPT-4’s release, a team of Microsoft Research scientists led by Sébastien Bubeck posted a 155-page paper with a provocative title: Sparks of Artificial General Intelligence: Early experiments with GPT-4. The authors, who had access to an early version of the model, argued that it showed signs of general intelligence. The paper was widely shared, widely criticized, and remains a useful case study in how to evaluate claims about AGI.

What the authors set out to do

Rather than running standard benchmarks, the team aimed to probe GPT-4’s abilities with novel, creative tasks that they believed were unlikely to appear in its training data. Their argument: if a system can handle a wide range of new problems across domains, it displays a form of general intelligence, even if limited.

The experiments

The paper is a collection of qualitative demonstrations across many areas:

  • Multimodal and interdisciplinary tasks: drawing a unicorn using TikZ, a LaTeX graphics language; composing short pieces of music in ABC notation; writing a proof that there are infinitely many primes in the style of a Shakespeare play.
  • Coding: solving programming problems, writing a 3D game in HTML and JavaScript, and reasoning about existing code.
  • Mathematics: solving and explaining problems, with notable successes and failures.
  • Interaction with the world: using tools such as search or calculators when prompted, and navigating text-based environments.
  • Theory of mind: answering questions about what characters in a story believe or feel.
  • Comparison with ChatGPT: many examples showed GPT-4 succeeding where the earlier model failed.

The authors also documented limitations: GPT-4 struggled with planning ahead in its generation, made arithmetic errors, hallucinated, and lacked long-term memory and continual learning.

The conclusion they drew

The paper argued that GPT-4 “could reasonably be viewed as an early (yet still incomplete) version” of an artificial general intelligence system, and called for new ways to define and evaluate intelligence.

The critiques

The paper drew strong criticism from parts of the research community.

Not reproducible. The experiments were done on an early, non-public version of GPT-4. Outsiders could not rerun them exactly, and the deployed model behaved differently.

Qualitative, not systematic. Selected examples demonstrate that something is possible, not how reliably it happens. Critics argued the paper did not report how many attempts failed.

Contamination unknown. Without knowledge of GPT-4’s training data, it is impossible to be sure that “novel” tasks really were novel.

Definitions. “AGI” has no agreed definition, so claims of “sparks” are hard to test or falsify.

Commercial context. The authors worked at Microsoft, OpenAI’s major investor and partner, which led some to question the framing.

What we can learn from it

  1. Capabilities were real and surprising. Many demonstrations — such as cross-domain code and creative tasks — illustrated genuine progress that benchmarks of the time did not capture.
  2. Demonstrations are not measurements. The research community has since invested heavily in systematic, reproducible evaluations of reasoning, agents and long-horizon tasks.
  3. AGI claims need definitions. Later work, such as Google DeepMind’s 2023 framework proposing “levels of AGI”, tried to make the concept measurable.

When you read headlines about AI showing “signs” of general intelligence, ask the questions this paper raised: Can others reproduce the results? How often does it work? Is the task really new? And what exactly do the authors mean by general intelligence?

Key takeaways

  • Microsoft researchers tested an early GPT-4 with creative, cross-domain tasks rather than standard benchmarks.
  • They argued it could be seen as an early, incomplete form of artificial general intelligence.
  • Critics objected that the results were qualitative, not reproducible and possibly affected by unknown training data.
  • The paper itself listed major limitations, including weak planning, arithmetic errors and hallucinations.
  • Its main lesson: demonstrations show what is possible, systematic evaluations show what is reliable.

Frequently asked questions

Did the paper claim GPT-4 is AGI?

No. It claimed GPT-4 could be seen as an early and incomplete version of an AGI system, and listed many limitations. Even so, critics argued that the framing overstated what qualitative demonstrations can show.

Why does reproducibility matter here?

Because the experiments used an early internal model. If others cannot rerun a test, they cannot check whether the result was typical, cherry-picked or specific to that model version.

Is there an accepted definition of AGI?

No. Definitions range from human-level performance on most cognitive tasks to the ability to do most economically valuable work. Frameworks such as Google DeepMind’s levels of AGI try to make the term more precise.