In artificial intelligence (AI) and machine learning (ML), data is the backbone that makes everything else possible. Without it, even the most advanced algorithm has nothing to learn from and nothing to base a prediction on. This article looks at the types of data used in AI, why data quality matters so much and how preprocessing turns raw data into something a model can learn from.

Types of data

Data in AI usually falls into three types: structured, unstructured and semi-structured. Each serves a different purpose and needs to be handled differently.

Diagram of three types of data: structured, in rows and columns like a spreadsheet or a database; unstructured, with no fixed form, like text, images, audio and video; and semi-structured, with some organisation, like JSON and XMLDiagram of three types of data: structured, in rows and columns like a spreadsheet or a database; unstructured, with no fixed form, like text, images, audio and video; and semi-structured, with some organisation, like JSON and XML

Structured data

Structured data is highly organised and usually stored in rows and columns, as in a spreadsheet or a database. Examples include:

  • numerical values, such as prices, heights and ages
  • categorical values, such as “yes” or “no”, product types and locations

Unstructured data

Unstructured data has no fixed format and comes in many forms:

  • text, such as emails and social media posts
  • images, such as photographs and medical scans
  • audio and video files

Semi-structured data

Semi-structured data sits between the two. It has some organisation, but less than structured data. Examples include:

  • JSON files, which web applications use to exchange data
  • XML files, which are often used to store documents

Why data quality matters

“Garbage in, garbage out” is a common saying in machine learning. It means that a model trained on poor data will make unreliable predictions. Data quality affects a model’s accuracy and efficiency, and it can make or break a project. Four aspects are worth keeping in mind:

  • Accuracy: The data should reflect real-world conditions.
  • Completeness: Missing values can lead to biased predictions.
  • Consistency: The data should not contradict itself, for example by listing two birth dates for the same person.
  • Relevance: Irrelevant data can confuse a model and reduce its accuracy.

Preparing data for machine learning

Raw data is rarely ready to use. It first goes through preprocessing, which cleans, formats and optimises it for learning. The details depend on the type and quality of the data, but four steps are common.

Data cleaning

Cleaning removes inaccuracies and inconsistencies from the dataset. It includes:

  • Handling missing values: Replacing or removing them keeps the data complete.
  • Handling outliers: Extreme values can distort a model’s predictions if left unchecked.
  • Standardising formats: Using the same units and date formats throughout helps avoid errors.

Data transformation

Transformation converts data into a form the model can work with. That can mean:

  • Scaling and normalisation: Adjusting values to a common range, such as 0 to 1, so that they are comparable.
  • Encoding categorical data: Converting text labels, such as “cat” or “dog”, into numbers the model can process.

Feature engineering

Feature engineering creates new features from existing data to help the model perform better. From a “date” column, for example, you might derive “day of the week” or “month”.

Splitting the data

Finally, the data is usually divided into three parts:

  • Training set: Used to train the model.
  • Validation set: Used to tune and evaluate the model during training.
  • Test set: Used to measure the model’s performance on data it has not seen before.
Bar chart of an example split of a dataset, not a measurement: a training set of 80%, a validation set of 10% and a test set of 10%Bar chart of an example split of a dataset, not a measurement: a training set of 80%, a validation set of 10% and a test set of 10%

Better data, better models

Data is fundamental to the success of AI and machine learning. High-quality, well-prepared data leads to models that make accurate predictions and produce useful insights. Understanding the types of data and the main preprocessing steps is the first step towards a successful AI project. The better your data, the better your model.