How artificial intelligence approaches insights and makes decisions is usually mysterious, increasing concerns about the reliability of machine learning. In a new study, the researchers revealed a new way to compare how well AI software reasoning matches to human reasoning to quickly analyze their behavior.
As machine learning finds more and more real-world applications, it become important to understand how it draws conclusions and whether it does accurately. For instance, an AI program appeared to correctly predict that a skin lesion was cancerous, however, it could have done so by focusing on an irrelevant blot in the background of the clinical image.
“Machine learning models are difficult to understand,” says Angie Boggust, a computer science researcher at the MIT and the lead author of the new study on AI reliability. “Knowing a model’s decision is easy, but it’s harder to know why the model made that decision.”
A common strategy for generating AI reasoning is to analyze the features of the data which the program focuses on, such as an image or a sentence, to make decisions. But these so-called saliency methods typically only provide insights about one decision at a time, and each decision must be checked manually. AI software is usually trained using millions of examples of data, making it closely impossible for a person to examine sufficient decisions to identify right or wrong behavior patterns.
“Giving users the tools to interrogate & understand their machine learning models is important to ensuring that machine learning models can be deployed safely in the real world.” Said Angie Boggust, at MIT.
Scientists at MIT and IBM Research now have created a method to collect and test the explanations AI gives for its decisions, allowing for a rapid analysis of its behavior. The new technique, called Shared Interest, compares saliency analytics of AI’s decision ability with a database annotated by humans.
For instance, an image recognition program might distinguish a picture as that of a dog and saliency ways might indicates that the program has spotlighted the pixels on the dog’s head & body to make decision. In contrast, the shared-interest method can compare the results of these saliency methods with an image database where people annotate which part of the image is of a dog.
Consistent with these comparisons, the Shared Interest method then asked to calculate how well the AI’s decision-making matched human reasoning, dividing it into one of 8 patterns. At one end of the spectrum, AI may be perfectly matched to humans, with the program making accurate predictions and spotlighting same features in the data as humans. On the other hand, the AI is fully distracted, with the AI making inaccurate predictions and not highlighting any features that humans did.
Other patterns that AI decision-making can fall into spotlight how machine learning models interpret data in detail correctly or incorrectly. For instance, Shared Interest can find that the AI accurately recognizes a tractor in an image-based on only a fragment of it – its tire, for instance- rather than recognizing the entire vehicle, like a human. Or find that the AI can recognize a snowmobile helmet in an image merely if a snowmobile is also present in the image.
In experiments, Shared Interest has revealed how AI programs work and whether they’re trustworthy. For instance, Shared Interest helped a dermatologist get a rapid look at examples of correct & incorrect predictions of a cancer diagnosis program from photos of skin lesions. In the end, the dermatologist decided he couldn’t trust the program as it made too many predictions based on irrelevant details instead of actual damage.
In other experiment, a machine learning researcher used Shared Interest to a check saliency method he applied to the BeerAdvocate dataset, which helped him analyze thousands of correct & incorrect decisions in one a fraction of the time required with traditional manual methods. Shared interest has helped indicate that the saliency method typically behaved well as expected, but also exposes previously unknown pitfalls like overvaluing certain words in reviews in a way that leads to inaccurate predictions.
“Giving users the tools to interrogate & understand their machine learning models is important to ensuring that machine learning models can be deployed safely in the real world,” says Boggust.
The researchers warn that Shared Interest works only as well as the saliency methods of attraction it uses. Boggust notes, each object method has its own limitations, which Shared Interest inherits.
In the future, scientists want to apply Common Interests to more types of data like tabular data used in medical records. Other potential area of research could be automating the estimation of uncertainty in AI results, Boggust adds.
The scientists have made source code for Shared Interest & live demos of it publicly available. They will detail their findings 3 May at ACM CHI Conference on Human Factors in Computing Systems.
