A model is only as trustworthy as the data behind it, yet most people pick a dataset from its title and a download count. The better habit is to read its documentation before you train, fine-tune or evaluate anything. Two research proposals shaped how that documentation looks today: “Datasheets for Datasets,” first posted to arXiv in 2018, and “Data Statements for Natural Language Processing,” published in the same year. Both argue that a dataset should come with a structured record of why it was made, what it contains and where it should not be used. This guide turns that idea into a checklist you can apply in ten minutes.
Where the idea comes from
The datasheet proposal borrows from electronics, where every component ships with a sheet describing its operating limits. Its authors suggest that dataset creators answer a standard list of questions grouped by the dataset’s life cycle: motivation, composition, collection process, preprocessing, uses, distribution and maintenance. The proposal is not a regulation, and it is peer-reviewed work that was later published in Communications of the ACM. The data statement proposal, from Emily Bender and Batya Friedman, focuses on language data and asks creators to describe the speakers, the annotators, the language variety and the situation in which the text was produced.
Today the practice shows up as “dataset cards” on hubs such as Hugging Face, which provides a template with sections for dataset structure, creation, considerations for use, known biases and licensing. Quality varies: some cards are thorough; many are almost empty. An empty section is information in itself.
A ten-minute reading routine
1. Motivation: why does this dataset exist?
Look for the original purpose. A dataset built to benchmark one task may be a poor fit for training a general model. Check who funded and created it, since the purpose often shapes what was collected and what was left out.
2. Composition: what is actually in it?
Check the number of instances, the types of data and how classes or topics are distributed. Ask whether the data is a sample of a larger set, and how the sample was chosen. A balanced-looking dataset can still be skewed on dimensions the documentation does not mention, such as geography, dialect, time period or domain.
3. Collection: how was it gathered?
Was the data scraped, licensed, donated or generated? Were people informed, and could they opt out? For text and images from the web, find out the crawl dates and any filtering that removed content. For annotated data, find out who labeled it, how they were instructed and how disagreements were resolved. Annotator guidelines often explain odd label patterns.
4. Preprocessing: what was changed?
Deduplication, language filtering, profanity filters and quality classifiers all alter the distribution. Documentation should say what was removed and whether the raw data is available. If it is not, you cannot audit the filters.
5. Uses: what is it for, and what is it not for?
Good documentation lists tasks the data has already been used for and warns against uses that could cause harm. Be skeptical of a dataset described as suitable for “any task.”
6. Licensing and consent
Look for the license of the dataset and, separately, the licenses of the underlying material. A dataset can carry a permissive license while containing text or images that its original owners never licensed for training. If you plan a commercial use, ask a lawyer; documentation can tell you what to ask, but it cannot give legal clearance.
7. Maintenance
Who maintains it, is there a version history, and how are errors reported? Evaluation sets in particular age: once a benchmark’s examples leak into training corpora, scores stop meaning what they used to.
Special checks for evaluation datasets
If you are using a dataset to measure a model, add three questions:
- Contamination: could test examples appear in the model’s training data? Look for decontamination notes and the date the set was published relative to the model’s training cutoff.
- Label quality: has anyone audited a random sample for errors? Even a small error rate caps the scores you can interpret.
- Coverage: do the examples reflect the users and languages you care about, or only a narrow slice?
For a concrete example of a heavily used training corpus and the questions it raises, see our explainer on The Pile. If you are building your own data, our guide to preparing a dataset for fine-tuning covers the practical side.
Writing a minimal datasheet for your own data
If you create data, answer these seven questions in a README before sharing:
- Why was this dataset created, and by whom?
- What does an instance represent, and how many are there?
- How was the data collected and over what period?
- What cleaning or filtering was applied?
- What are the known gaps, biases and error rates?
- What license applies, and what consent was obtained?
- Who maintains it and how can people report problems?
Keeping the answers short but honest is better than a long template filled with “N/A.”
What to watch next
- More hubs and journals requiring a completed dataset card before publication.
- Regulatory pressure for training-data transparency summaries in several jurisdictions.
- Tools that check dataset cards automatically for missing sections.
- Standards for documenting synthetic and model-generated data.
- Better practice on decontaminating public benchmarks.
Key takeaways
- Datasheets and data statements, both proposed in 2018, give creators a structured way to describe a dataset’s purpose, contents and limits.
- Read the motivation, composition, collection, preprocessing, uses, licensing and maintenance sections before using a dataset.
- An empty or vague section is a warning sign, not a neutral fact.
- For evaluation sets, check contamination, label quality and coverage.
- If you share data, publish a short, honest datasheet with it.
Frequently asked questions
Are datasheets required by law?
Not as a general rule. They are a documented best practice, although some regulations are moving toward requiring summaries of training data.
What is the difference between a datasheet and a data statement?
A datasheet is a general-purpose set of questions about any dataset. A data statement is designed for language data and stresses speaker, annotator and context information.
What should I do if the documentation is missing?
Treat the dataset as unaudited: sample it yourself, check the source’s license terms, and prefer alternatives with clearer provenance for anything high-stakes.
