We encountered “Type III error” when it turned out that most people answering our Puzzle question were, in fact, answering a different question from the one that was asked. Type III error is answering the wrong question, and it is a big factor in the failure of many AI projects.
We encountered “Type III error” when it turned out that most people answering our Puzzle question were, in fact, answering a different question from the one that was asked. Type III error is answering the wrong question, and it is a big factor in the failure of many AI projects.
Gartner Research estimates that 85% of AI and data science projects fail to be implemented. Gerhard Pilcher, CEO of Elder Research (which owns Statistics.com), says that our implementation success rate is much higher – at over 65% – but he laments even the one-third of projects which are not put into production by / for the clients, saying those losses are preventable.
Different variations of Type III error are often responsible for these failures. Sometimes, a CEO will be seized by the idea that their company must commit itself to analytics and AI, and initiate projects to that end, without regard to a clear operational goal to be achieved. “They expect to dump data into a magical ‘black box,’ turn a crank, and generate insights,” says Pilcher. Other times, the business teams for which a project is intended (e.g., marketing or finance) want answers about revenue and profit, while the technical team will be evaluating the project by narrow metrics such as model accuracy. Or the data science team is developing a model to answer a question that is relevant in the abstract, but not translatable to a form that the IT department can or will implement.
Detecting Fraud
One example of the benefits of learning to ask the right question comes in the area of fraud detection. To stay one step ahead of the cops, hackers and garden variety fraudsters are constantly changing their methods. In predictive modeling, this can make it hard to describe fraud in terms of feature values, since those will change as criminal behavior changes. Non-fraudulent behavior, however, remains relatively consistent, particularly if you cast the feature net broader than simply transaction data. So why not model non-fraudulent behavior, instead?
Josh Sullivan of Booz Allen, writing in the Harvard Business Review about a project with a large financial firm reported that they did just that, focusing on understanding and modeling compliant behavior. As he put it,
“We changed the nature of the question we were asking. Instead of, “How do we model bad?” we asked, “What if we modeled good?”
By modeling good, one can focus attention on the “non-good,” and presume that it might be bad. Sullivan said that this approach enabled them to save their client over $1 billion by stopping massive fraud schemes.
For more about Type III error, including additional examples, see our blog on the subject from last year.