The Role of Data in AI: Where Models Learn What They Know

Two AI systems can be built in almost the same way and still behave very differently. Often the difference is the data. A machine-learningMachine learningAn approach to AI in which a model learns patterns from examples and applies them to new cases, instead of following only hand-written rules.See in the glossaryClose model is not given rules; it learns patterns from examples. Whatever is in those examples, including their mistakes and blind spots, can become part of what it learns.

This article follows data through the life of an AI system: where it comes from, how people prepare it, how its flaws turn into a model’s flaws, and why the supply of good data has become a question in its own right.

What counts as data

To a machine-learning system, data is any collection of examples it can learn from. Some of it is structuredStructured dataData arranged in a fixed pattern of rows and columns, such as the same fields for every bank transaction.See in the glossaryClose, arranged in rows and columns like a bank’s transaction records or a laboratory’s test results. Much more is unstructuredUnstructured dataContent such as text, photographs, audio and video, which does not fit neatly into the columns of a table.See in the glossaryClose: text, photographs, audio and video, whose contents do not fit neatly into the rows and columns of a table.

A second distinction matters even more for how a model learns. LabelledLabelled dataExamples that come with the answer the model should learn, such as an email marked as spam. Data without such answers is unlabelled.See in the glossaryClose data comes with the answer attached: a photograph tagged “cat”, an email marked “spam”, an X-ray with a radiologist’s finding. Unlabelled data is the raw material alone. Labelled data is far more useful for teaching a specific task, and usually far more expensive to produce, because someone has to supply the answers.

Large language modelsLarge language modelA model trained on a very large amount of text to predict the next token, which lets it generate text for many tasks.See in the glossaryClose get around part of that cost. They learn mainly by predicting the next token (a word or part of a word) in huge amounts of text, so the text supplies its own answers: the token that actually came next is the label. This is called self-supervised learningSelf-supervised learningTraining in which the data supplies its own answers, for example the word that actually comes next in a sentence.See in the glossaryClose, and it is one reason these models can be trained on far more data than earlier systems.

The life of a dataset

A datasetDatasetA collection of examples gathered for a task, used to train a model and to test it.See in the glossaryClose, the collection of examples a model learns from and is tested on, is not simply gathered and used. It passes through a series of steps, and each one can improve it or damage it.

The steps rarely run in a straight line. A problem found during testing, or after the system is in use, often sends a team back to collect more data or fix the labels.

Collecting

Data comes from wherever the task lives: transaction records, sensors, medical archives, satellite images and the public web. Web archives are a major source of training text for many language models. Common Crawl (opens in a new tab), a non-profit that has been archiving the public web since 2008, describes its archive as holding more than 300 billion pages collected over the years, and provides free access to it. Filtered versions of its archive are part of the training data of many language models, including GPT-3 (opens in a new tab).

What is collected decides what a model can know. A system trained mostly on English text may be weaker in Persian, and a medical model trained on patients from one hospital may not work as well in another.

Labelling

Labels usually come from people. ImageNet (opens in a new tab), the image collection that helped set off the deep-learning boom of the 2010s, contains more than 14 million photographs sorted into thousands of categories. Its creators, led by the computer scientist Fei-Fei Li, used Amazon Mechanical Turk (opens in a new tab), a crowdsourcingCrowdsourcingSplitting a large job into small tasks done by many people, often paid online workers, for example labelling images.See in the glossaryClose platform that splits small tasks among many paid online workers, to have the images sorted and labelled. By Li’s account (opens in a new tab), almost 50,000 workers in 167 countries took part.

Chatbots depend on human judgement too. After a language model has learned from text, it is usually trained further in separate stages. In supervised fine-tuningSupervised fine-tuningFurther training of an already trained model on example answers that people wrote.See in the glossaryClose, it learns from example answers written by people. In reinforcement learning from human feedbackReinforcement learning from human feedback (RLHF)Further training in which people rank a model’s answers and the model learns to give answers more like the ones they preferred.See in the glossaryClose (RLHF), people rank the model’s own answers, and the model is trained to give answers more like the ones they preferred. In a 2022 study by OpenAI (opens in a new tab), on the prompts and evaluations used in that study, people preferred the answers of a model trained with both stages, with 1.3 billion parametersParametersThe adjustable values inside a model that training sets. Their number is a common measure of a model’s size.See in the glossaryClose (the adjustable values a model learns), to those of the original GPT-3, which has 175 billion: about 135 times as many.

Cleaning and checking

Raw data is messy. It has missing values, duplicates, mixed units, mislabelled examples and records that contradict each other. Cleaning it is slow and rarely celebrated. In a 2021 study (opens in a new tab) titled “Everyone wants to do the model work, not the data work”, researchers at Google interviewed 53 people building AI for high-stakes uses in India, East and West Africa and the United States. Of them, 92% had experienced at least one “data cascadeData cascadeA data problem that goes unnoticed early and causes larger problems later, often after a system is in use.See in the glossaryClose”: a data problem that went unnoticed early and caused trouble later, often after the system was in use.

Splitting and testing

Before training, the data is usually divided. Most of it becomes the training set, which the model learns from. A smaller validation set is used during development to compare versions and tune choices. A final test set is kept aside, unseen, to measure how well the finished model handles examples it has never met.

The test only works if that part stays separate. If test examples leak into training, through duplicates for instance, the model is partly being tested on material it has already seen, and its score looks better than its real performance. This is called data leakageData leakageInformation that should be kept out of training, such as test examples, slipping into it. The test score then looks better than real performance.See in the glossaryClose. The same concern applies to language models trained on much of the public web, where published test questions and their answers may already be somewhere in the training text.

The same care applies to preparation steps that learn from the data itself, such as filling in missing values with an average or rescaling numbers to a common range. Those statistics should be worked out from the training set alone and then applied unchanged to the validation and test sets. Otherwise, information from the test data leaks into training.

When the data is wrong

Mistakes in the labels

Even famous datasets contain errors. A 2021 study (opens in a new tab) of ten widely used test sets for images, text and audio estimated that at least 3.3% of their labels were wrong on average. In the ImageNet validation set, commonly used to test image models, the authors found 2,916 errors, about 6%. They also showed that errors at this level can change which of two models appears to be better.

Who is missing

A model learns the world its data shows it. In the Gender Shades (opens in a new tab) study of 2018, the researchers Joy Buolamwini and Timnit Gebru tested three commercial systems that classified faces by gender. For lighter-skinned men, the highest error rate was 0.8%. For darker-skinned women, it reached 34.7%. The same study found that two widely used collections of face images for testing such systems were made up overwhelmingly of lighter-skinned faces: 79.6% in one and 86.2% in the other.

The lesson goes beyond faces. When a group is rare or absent in the data, a model can perform well on average and still fail that group, and an average score will not show it.

When the world changes

Data describes the past, and the world moves on. Google Flu Trends, launched in 2008, estimated flu activity from what people searched for on Google. In February 2013 it predicted more than double the share of doctor visits for flu-like illness that the US Centers for Disease Control and Prevention reported. In The Parable of Google Flu (opens in a new tab), published in Science in 2014, researchers argued that the system had partly learned to detect winter rather than flu, by matching search terms that simply rose and fell with the season, and that changes in people’s searches and in Google’s own search engine had thrown it off further.

This mismatch between the data a system learned from and the data it meets in use is called data driftData driftA change between the data a model learned from and the data it meets in use, which can make its predictions worse over time.See in the glossaryClose. It can build up gradually or arrive suddenly, and it is why AI systems need monitoring after launch, not only testing before it.

Is AI running out of data?

Language models have improved partly by training on ever more text, measured in tokens. Human writing is not unlimited. In a 2024 study (opens in a new tab), researchers at Epoch AI estimated the effective stock of public human-written text at around 300 trillion tokens, a figure already adjusted for quality and for training on the same text more than once. If the growth in training data continued, they projected, the largest models would reach that ceiling at some point between 2026 and 2032. That is a conditional projection made in 2024, not a fixed date.

That does not mean progress stops. Developers can get more out of the text they already have, use data that is not public, learn from other kinds of data such as images and video, or have AI generate new training examples.

The limits of synthetic data

Synthetic data is data that is generated rather than recorded from real events. It can come from simulations, rules or statistical methods, and increasingly from generative AI models; this section focuses on that last kind. Synthetic data can help where real data is scarce, expensive or private, as with rare situations for training self-driving systems or medical records that cannot be shared.

It also carries a risk. In a 2024 study in Nature (opens in a new tab), researchers trained models on the output of earlier models, generation after generation, using that output indiscriminately and recursively. The models gradually lost the rare and unusual parts of the original data, and their output narrowed and degraded, an effect the authors called model collapseModel collapseThe loss of rare and unusual patterns when models are trained, generation after generation, on the output of earlier models.See in the glossaryClose. In one example, text that began as a passage about medieval architecture had turned into a list of jackrabbits by the ninth generation. The authors did not conclude that synthetic data is useless. They concluded that data from genuine human activity will become more valuable, and that AI-generated data has to be used with care rather than fed back indiscriminately.

Questions to ask about the data behind an AI system

You rarely see a model’s training data, but these questions help when you read about an AI system or decide whether to rely on one:

• Where did the data come from? Was it collected for this task, or borrowed from somewhere else?
• Who labelled it? Were the labels checked, and by people who knew the subject?
• Who or what is missing? Does the data cover the people and situations the system will meet?
• How old is it? Has the world it describes changed since?
• What was it tested on? Was the test data kept apart, and does it resemble real use?

Models get the headlines, but data decides much of what they can do. How AI systems are sorted by what they can do is the subject of Narrow, General and Superintelligent AI, and the basics of how they learn are in An Introduction to Artificial Intelligence.