AI systems are often described as objective because they are “just math”. In practice, they learn from data produced by people and institutions, and they are designed and deployed by people with particular goals. Unfair outcomes — algorithmic bias — can enter at every stage. Understanding where helps prevent it.

Landmark examples

Gender Shades (2018). Joy Buolamwini and Timnit Gebru evaluated commercial facial-analysis systems that classified gender from photos. Error rates were very low for lighter-skinned men but dramatically higher for darker-skinned women — exceeding 30% in some cases. The study prompted companies to improve their systems and became a reference point in debates about facial recognition.

Healthcare risk prediction (2019). A study published in Science by Ziad Obermeyer and colleagues found that a widely used algorithm for identifying patients needing extra care systematically underestimated the needs of Black patients. The cause was subtle: the algorithm predicted healthcare costs as a proxy for health needs, and less money had historically been spent on Black patients with the same level of need.

These cases show two patterns: unrepresentative data, and objectives that encode existing inequalities.

Where bias comes from

  1. Historical bias: data reflects past discrimination — for example hiring records from a period when certain groups were excluded.
  2. Representation bias: some groups are underrepresented in training data, so the model performs worse for them.
  3. Measurement bias: the variables or labels used are flawed proxies for what matters, as in the healthcare case.
  4. Aggregation bias: one model is applied to groups whose patterns differ.
  5. Deployment bias: a system is used in a context or for a purpose it wasn’t designed for.
  6. Feedback loops: predictions influence future data, reinforcing patterns — for example predictive policing concentrating patrols in neighborhoods that then generate more recorded incidents.

Bias in generative AI

Language and image models trained on internet data can reproduce stereotypes: associating professions with genders, depicting certain nationalities in limited ways, or performing worse in less-represented languages and dialects. Because generative AI is used at enormous scale, small skews can have broad effects.

Measuring fairness

There is no single definition of fairness. Common metrics include:

  • Demographic parity: similar rates of positive outcomes across groups.
  • Equal opportunity: similar true-positive rates across groups.
  • Calibration: predicted risks mean the same thing for every group.

Mathematical results show that some of these criteria cannot all be satisfied at once when base rates differ between groups, so choosing a fairness definition is an ethical and legal decision, not just a technical one.

Reducing bias

  • Better data: representative samples, careful labeling, documentation of datasets.
  • Disaggregated evaluation: measure performance separately for relevant groups, not just on average.
  • Fairness-aware training and post-processing adjustments.
  • Human oversight and the ability for affected people to contest decisions.
  • Diverse teams and consultation with affected communities.
  • Ongoing monitoring after deployment.

Related: how to audit an AI system for fairness and AI in hiring and HR.

What to watch next

  • Enforcement of anti-discrimination law against automated decision systems.
  • Bias evaluation of generative AI across languages and cultures.
  • Dataset documentation requirements in regulation and procurement.
  • Rights to explanation and contestation for people affected by automated decisions.
  • Research on fairness in large models, where traditional metrics are harder to apply.

Key takeaways

  • Bias can enter AI through historical data, unrepresentative samples, flawed proxies, design and deployment.
  • Gender Shades exposed large accuracy gaps in commercial facial analysis across skin type and gender.
  • A healthcare algorithm underestimated Black patients’ needs by using costs as a proxy for health.
  • Fairness has several definitions that can conflict; choosing one is an ethical decision.
  • Disaggregated evaluation, better data, oversight and monitoring reduce bias.

Frequently asked questions

Can AI ever be completely unbiased?

Not in an absolute sense, because data and goals reflect choices and societies. The aim is to identify, measure and reduce unfair disparities and to be accountable for remaining ones.

Is removing sensitive attributes like race enough?

No. Other variables, such as postal code, can act as proxies, and removing attributes can make bias harder to detect and correct.

Who is responsible for biased AI outcomes?

Responsibility is shared among developers, the organizations that deploy systems and, under laws such as the EU AI Act and anti-discrimination rules, can carry legal obligations.