What it means in practice
Suppose you want to teach a new intern what a successful salesperson looks like, but you only show them CVs of men over 40 wearing suits in the photo. The intern will logically conclude that a woman, or a young man in a t-shirt, cannot be a good salesperson. That is exactly what bias in AI is. AI has no judgement of its own; it looks for patterns in historical data. If that data is historically skewed, the AI will not only repeat the skew but often amplify it.
Why companies need to know this
- Reputational risk. If your AI chatbot answers customers in a partial or inappropriate way, it damages your name.
- Losses from bad decisions. If AI sorts CVs and rejects good candidates purely because of a hidden bias, you lose talent.
- Legal problems. Decisions that disadvantage particular groups of people can breach anti-discrimination law.
- Blind trust in technology. Owners often assume a computer is “objective”. A computer is only as objective as the data you feed it.
An example from practice
A large international company used AI for the first pass on CVs for developer roles. The model learned from ten years of CVs dominated by men. The system developed a bias and started penalising CVs containing the word “women’s” (as in captain of the women’s chess club). The tool eventually had to be scrapped. Smaller companies have to be just as careful about delegating important decisions to AI.
In our company AI workshops we teach how to recognise the limits of AI and why putting review processes in place is critical if you want to avoid expensive mistakes.