It is summertime in the 1950s, a man walks into a bank, trying to apply for a US$10,000 loan to kickstart his small business. The Banker, in order to do his job well, has done his homework and knows everything about this man. His ethnicity. His marital situation. Does his boss like him? Does his wife like him? Is he a church-goer? Do his neighbours think him a good man?
The banker balances these factors and holds the key to this man’s future. The man’s ability to open a small company is entirely dependent on the whims of the fancy-suited banker sitting across from him.
Not a very fair system huh?
In 2017, this banker does not exist, long replaced by computer models and algorithms to make sure nobody falls victim to a powerful individual’s prejudices. It is a system that promotes equality of opportunity to all, regardless or race, gender and religious identity.
Except that is also a fantasy, and in many ways, the algorithms we so adore are feeding into a vicious cycle that expedites inequality and disproportionately harms the poor.
Or so says Cathy O’Neil in her book Weapons of Math Destruction: Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.
Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
“Welcome to the dark side of Big Data,” writes O’Neil after the introduction chapter of her latest book.
This sentence sets-up the reader with clear expectations and as the reader joins O’Neil on her journey, it is clear there are only a handful of people who could have written Weapons of Math Destruction.
O’Neil is a mathematician who left the cushy world of academia in favour of fast-paced algorithmic analysis on Wall Street. This is where 2008 financial crisis opened her eyes to the real-life consequences of the industry, she joined the Occupy Wall Street movement.
The hypothesis of the book revolves around O’Neil coining the term ‘Weapons of Math Destruction’ (WMD). She defines WMDs as algorithms that are opaque, have a large impact on society and harm the most vulnerable of citizens.
WMDs also encode prejudices, lack feedback and creates a system that reinforces its false “successes”. (e.g. a recidivism model that hurts an ex-convicts ability to find work after serving time, thus making it more likely for them to turn back to crime and get arrested, which, in its own perverse way, proves the algorithm was correct).
Using her own personal experience, and journalism-driven storytelling, Weapons of Math Destruction explores the destructive nature of algorithms ranging from bank lending to college ranking systems.
The book is an easy read, meant to engage the public-at-large. It appeals to the logic of an average reader and does not get bogged-down in the math of algorithms. Plus, it is funny! O’Neil’s sarcastically-dry wit is allowed to bubble-up and while the jokes can be a tad dark, they add some humour to the book.
For the tech community in Asia, I would posit Weapons of Math Destruction as required reading for entrepreneurs thinking about Big Data, algorithmic marketing and artificial intelligence (so, everyone?).
The reason is because the book is a cynic’s view on the technology we so enthusiastically build. Even if a Founder disagrees with the entire premise, they should read the book so they can understand an opposing view about the ethical concerns of algorithms.
With that said, there are two caveats I must mention.
First, O’Neil is American, and the book details the American experience. For example, the segment on for-profit Universities is not relevant to Singapore. But, the strategies deployed by these people, and lessons we can learn from their strategies, are applicable across the world.
Second, Democracy is not necessarily a desired political ideology in Asia. That being said, the book only discusses the topic at the very end, and when it does, it analyses the 2012 US-election like a tactical historian.
The importance of this book has nothing to do with politics, but rather the ocean of evidence O’Neil to point out that poorly built algorithms create a vicious cycle that does its most damage to the poor and vulnerable in our society.
Towards the end of the book, O’Neil calls for a sense of responsibility from engineers, CTOs and data scientists everywhere. She posits a ‘hippocratic oath’ for the engineers in our society. The idea is this oath is both an agreement to ‘do no harm’ as well as admit to ourselves the destructive nature these of poorly-built algorithms.
O’Neil knows that algorithms are here to stay, but as technology continues increasingly dominate our lives, the time is now to demand an ethical standard from the people that build these powerful tools.