Algorithmic Bias Is Not a Glitch

Ask an image generator for “a doctor” and note who it draws. Ask it for “a criminal” and note who it draws. The results are not random, and they are not a bug that slipped past quality assurance. They are a compressed statistical summary of the pictures the model was trained on, which are a compressed summary of the world’s published images, which carry a century of decisions about who got photographed as what.

Algorithmic bias is one of the most misunderstood problems in AI because the word “bias” imports a moral charge the mechanism does not need. The model is not prejudiced. It has no beliefs. It is doing exactly what it was built to do — predict the most likely continuation — and the most likely continuation, learned from an unequal record, encodes the inequality. The bias was in the data before it was ever in the model.

This matters because these systems now sit between people and consequences: which images you are shown, whose CV gets read, which patient gets flagged for extra care, whose face a camera claims to recognise. When the skew is inherited from the data, “we didn’t intend it” is both true and beside the point. The output lands on a real person either way.

Bias is a data property, not a personality

The cleanest way to see the mechanism is a case with none of the usual noise. In 2019, researchers publishing in Science dissected an algorithm used across US health systems to flag patients for extra care. It was measurably biased against Black patients: at the same risk score, they were sicker than white patients. The reason was almost banal. The system used health costs as a stand-in for health needs. Because less money had historically been spent on Black patients with the same conditions, the algorithm read them as healthier and de-prioritised them. Obermeyer and colleagues estimated the bias cut the number of Black patients identified for extra help by more than half.

Nobody typed a racist rule. They chose a convenient proxy — cost — that carried the fingerprints of an unequal system, and the model dutifully learned the pattern in it. Change the target from “cost” to a direct measure of illness and the bias largely dissolved. That is the whole lesson in miniature: bias lives in the data and in what you optimise for, not in the model’s character. This is the same shape of problem as hallucination — a predictable property of how the system works, not a defect awaiting a patch.

The receipts

None of this is theoretical, and the strongest evidence is numeric and published. A few load-bearing examples:

• Face analysis. The 2018 Gender Shades study by Joy Buolamwini and Timnit Gebru tested commercial gender-classification systems and found error rates never worse than 0.8% for lighter-skinned men, rising to 20.8%, 34.5% and 34.7% for darker-skinned women. Same task, wildly different reliability depending on who you were.
• Face recognition at scale. The US National Institute of Standards and Technology, testing scores of algorithms in 2019, found demographic differentials in false-match rates across the board — a government lab confirming the effect was systemic, not a one-vendor slip.
• Hiring. In 2018 Amazon scrapped an internal recruiting tool after it emerged the system had learned to penalise CVs containing the word “women’s,” having been trained on a decade of mostly male résumés. It taught itself that male was the signal of a good hire.
• Healthcare. The cost-as-proxy failure above, at a scale affecting millions of patients.

What these share is that the bias was invisible at the interface. Each system presented a clean, confident, neutral-looking output. The disparity only appeared when someone disaggregated the results and measured performance group by group — the same discipline we argued for when we said an average benchmark score hides more than it reveals.

The same pattern shows up on an axis Western coverage often forgets: language. Models trained overwhelmingly on English get measurably worse as you move to lower-resourced languages, so a Swahili or Bengali speaker can receive shorter, shakier, more error-prone answers for the same subscription — a quality gap that maps neatly onto who was already over-represented online. Bias is not only about race and gender; it is about whose data happened to be lying around in bulk when the model was built.

Generative models don’t just classify, they depict

Classifiers make biased decisions. Generative models do something more vivid: they manufacture the biased image and hand it to you as content. Ask a text-to-image system for professionals and it will tend to reproduce, and often amplify, the demographic skew of its training set — rendering executives one way and cleaners another, sharpening a real-world imbalance into a visual stereotype presented as the default.

Audits of these systems have repeatedly found the skew is not merely inherited but exaggerated. Shown a profession that is, say, 60% one gender in reality, a generator may render it 90% that way, because it is chasing the single most probable image rather than the actual spread. The model does not average the world; it rounds it to the nearest stereotype and then serves that back as the neutral default.

The industry knows this, which is why the fixes are instructive. In early 2024, Google paused Gemini’s ability to generate images of people after its diversity tuning overshot, producing historically implausible pictures in an effort not to default to one demographic. It was widely mocked, and the mockery missed the more useful point.

The steel-man for Google is real: they were trying to correct a genuine representational bias, and doing nothing would have shipped the stereotype instead. But the episode showed how a crude output-side override — forcing the result — produces its own absurdities, because it treats the symptom while the cause sits upstream in the training distribution. It is the generative cousin of an over-broad refusal: a blunt instrument applied at the exit because the harder work was elsewhere.

Why you can’t just “make it fair”

Here is the part that dooms most “we’ll just remove the bias” press releases. “Fair” is not one thing. It has several precise, reasonable, mutually incompatible definitions, and you cannot have them all.

The canonical fight was over COMPAS, a US recidivism-risk tool. In 2016, ProPublica reported it produced higher false-positive rates for Black defendants — flagging as future reoffenders people who did not reoffend — more often than for white defendants. The vendor, Northpointe, replied that the tool was calibrated: a given score meant the same probability of reoffending regardless of race. Both were right. And that is precisely the problem. A foundational 2016 result by Kleinberg, Mullainathan and Raghavan, echoed by Chouldechova, proved that when two groups have different base rates, you cannot simultaneously equalise false-positive rates and keep the scores calibrated. Pick one notion of fairness and you mathematically forfeit the other.

So “unbiased” is under-specified to the point of being a marketing word. Every deployed system has, knowingly or not, chosen which unfairness it will live with. The honest question is never “is it fair?” but “fair in which sense, and who decided that trade-off, and were the people it affects ever asked?”

It helps to separate two kinds of harm, because they need different fixes. Allocative harm is when a system hands out a resource unfairly — a loan, an interview, a care referral. Representational harm is subtler: a system that depicts or describes a group in a demeaning or stereotyped way, shaping how they are seen even when no resource changes hands. An image generator that always draws nurses as women is committing the second even if nobody is denied anything, and most public arguments collapse the two — which is a large part of why they talk past each other.

The feedback loop that makes it worse

Bias in these systems is not static; it compounds, because the outputs become the next inputs. A predictive-policing model trained on historical arrest data sends patrols to the neighbourhoods already over-policed, which generates more arrests there, which confirms the model — a loop that launders a decision about where to look as a fact about where crime is. A hiring model that filters out a group ensures that group is absent from next year’s “successful hire” data.

Generative AI adds a newer twist. As the open web fills with model output, tomorrow’s models increasingly train on yesterday’s — and any skew in the synthetic images and text gets relearned and reinforced, a stereotype photocopying itself down the generations. The provenance of training data, a subject we have circled before, turns out to determine not just privacy but who the future’s models think the world is made of.

What good actually looks like

The problem is real and partly tractable, which is the most honest thing that can be said about it. What separates teams treating it seriously from teams issuing statements:

• Evaluate disaggregated, not on average. Test performance for each subgroup, because an impressive headline accuracy can hide a group the system fails badly.
• Fix the target and the data, not the output. Obermeyer’s bias fell when the objective changed from cost to need. Upstream fixes hold; output overrides embarrass.
• Document what went in. Datasheets for datasets and model cards make the training distribution and known limitations inspectable rather than mystical.
• Keep a human with real authority. Not a rubber stamp, but someone empowered to override the model in consequential decisions — and accountable when they don’t.
• Name the fairness trade-off out loud. Since you must choose one, say which, and let the people affected contest it.

Regulation is beginning to insist on some of this. The EU AI Act designates many consequential uses — hiring, credit, essential services — as high-risk, with duties around data quality, documentation and human oversight, and existing anti-discrimination and data-protection law already reaches automated decisions in much of the world.

What none of this buys is a one-time certificate. A model that tests clean today can drift as its inputs change, and a system that is fair for the population it was audited on can still fail a subgroup nobody thought to slice. Bias auditing is closer to food-safety inspection than to a product launch: it only means anything if it keeps happening, and if someone outside the company is allowed to look.

Neutral is a claim, not a property

The danger in algorithmic bias is not that machines are uniquely prejudiced. It is that they wrap an inherited prejudice in an interface that looks objective, and objectivity is persuasive. “The algorithm decided” ends conversations that “a person decided” would start.

To be fair, there has been measurable progress: the error gaps that Gender Shades exposed narrowed once they were public, audits are now a field rather than a stunt, and the better labs test for this before shipping. The technology is not doomed to be unfair. But it is not automatically fair either, and the difference is work — done upstream, on the data and the objective, by people willing to name the trade-offs.

For the rest of us, the practical posture is simple and slightly stubborn. Treat “the model said so” as the beginning of a question, not the end of one. Ask what it was trained on, what it was optimised for, and who checked whether it works as well for you as it does for the person in the demo. Neutrality is something a system has to earn and keep proving. It is never something you should take on the interface’s word.