Training a language model requires enormous amounts of text. In 2020, the datasets used for the largest models were mostly private. That year, EleutherAI, a grassroots collective of volunteer researchers, released The Pile: an open, 825-gigabyte English text dataset designed for training large language models. It powered a generation of open models and became a reference point in debates about where training data comes from.

Why build a new dataset?

Many earlier models were trained largely on Common Crawl, a massive public archive of web pages. Raw web text is abundant but noisy, repetitive and uneven in quality. The authors hypothesized that diversity — mixing many types of high-quality text — would help models generalize better and gain broader knowledge.

What it contains

The Pile combines 22 subsets, several of them newly constructed by the team. They include:

  • Pile-CC: a filtered version of Common Crawl web text;
  • academic text: PubMed Central articles and abstracts, arXiv papers, and patents from the US Patent and Trademark Office;
  • code: a large collection of GitHub repositories;
  • books: including Project Gutenberg public-domain books and a subset called Books3;
  • reference and Q&A: English Wikipedia and Stack Exchange;
  • dialogue and subtitles: Ubuntu IRC logs and OpenSubtitles;
  • specialized sources: FreeLaw court opinions, mathematics problems, EuroParl parliamentary proceedings, PhilPapers, NIH grant abstracts, Hacker News and email collections.

The authors assigned different sampling weights to the components, so higher-quality sources were seen more than once during training.

What the paper showed

The authors evaluated models on the Pile’s components and found that models trained on it improved significantly over those trained on raw Common Crawl alone, particularly on academic and specialized text. Large models of the time, such as GPT-2 and GPT-3, performed relatively poorly on several components, suggesting their training data lacked that variety.

The paper also included a datasheet-style analysis discussing topics, biases and problematic content in each component — an early example of documenting training data in depth.

Impact

The Pile became the training set for many open models, including EleutherAI’s GPT-Neo, GPT-J and GPT-NeoX-20B, and the Pythia suite, a set of models trained on the same data in the same order to support research on how models learn. Meta used some Pile components in training its original LLaMA models.

By making a high-quality dataset freely available, The Pile lowered the barrier for academic and independent groups to train language models.

Some components raised legal and ethical questions. The Books3 subset contained tens of thousands of books, many still under copyright, originally collected from pirated sources. In 2023, after complaints from rights holders, the copy hosted by a third party was taken down, and Books3 became a reference point in lawsuits brought by authors against AI companies.

The episode illustrated a tension that still shapes AI: openness and reproducibility on one side, and the rights of creators whose work makes up training data on the other.

The legacy

The Pile’s emphasis on curated diversity influenced later datasets, which put even more effort into filtering, deduplication and licensing. Newer open datasets document their sources in more detail and increasingly exclude content with unclear rights, reflecting both legal pressure and lessons learned from The Pile.

Key takeaways

  • The Pile is an 825 GiB open English text dataset released by the volunteer collective EleutherAI in 2020.
  • It mixes 22 sources — web, academic papers, code, books, legal texts, Q&A and more — on the idea that diversity improves generalization.
  • Models trained on it outperformed those trained on raw Common Crawl, especially on academic and specialized text.
  • It powered open models such as GPT-J, GPT-NeoX-20B and the Pythia research suite, lowering the barrier to training language models.
  • Its Books3 subset of copyrighted books made it a reference point in the legal debate over training data.

Frequently asked questions

How large is The Pile in tokens?

The dataset is about 825 GiB of text, which corresponds to roughly 300 billion tokens depending on the tokenizer used — small by today’s standards, where frontier models train on trillions of tokens.

Is The Pile still used?

It is still used in research, especially with EleutherAI’s Pythia models. For training new models, most groups now use larger and more carefully filtered datasets with clearer licensing.

What was Books3?

A subset of The Pile containing tens of thousands of books, including many copyrighted works obtained from pirated sources. It was removed from public hosting in 2023 and has featured in copyright lawsuits against AI developers.