AI models can now explain their reasoning using concepts they've learned themselves

Computer vision models are increasingly used in high-stakes applications like medical diagnostics and autonomous driving, where understanding their reasoning is crucial for building trust. A new technique developed by MIT researchers could transform how these AI systems explain their predictions, making them more transparent and reliable.

The challenge of AI explainability

When a computer vision model predicts whether a medical image shows cancer or identifies an object in a self-driving car's path, users often need to understand why it made that decision. Traditional deep learning models operate as "black boxes," making predictions without clear explanations of their reasoning process.

Concept bottleneck modeling (CBM) has emerged as one approach to address this challenge. These methods force AI models to use human-understandable concepts as intermediate steps before making final predictions. For instance, a model diagnosing skin cancer might use concepts like "irregular borders" or "asymmetric shape" before concluding whether a lesion is malignant.

However, standard CBMs have significant limitations. The concepts used are typically defined in advance by human experts or generated by large language models, which may not align perfectly with the specific task or dataset. This mismatch can reduce accuracy, and models sometimes bypass the intended concepts entirely, using other learned information instead.

Learning concepts from the model itself

MIT researchers took a different approach: instead of forcing models to use pre-defined concepts, they developed a method that extracts concepts the model has already learned during training. This approach, described in a paper by Antonio De Santis and colleagues, could lead to more accurate and relevant explanations.

The technique works through a two-step process. First, a specialized deep-learning model called a sparse autoencoder analyzes the target model and extracts the most relevant features it has learned. These features are then reconstructed into a handful of concepts that the model actually uses for the specific task.

Next, a multimodal large language model describes each extracted concept in plain language and annotates images in the dataset by identifying which concepts are present or absent. This annotated dataset trains a concept bottleneck module that's incorporated into the target model, forcing it to make predictions using only the extracted concepts.

Key innovations and advantages

Several aspects of this approach make it particularly effective. By extracting concepts directly from the trained model, the researchers ensure the concepts are relevant to the specific task and dataset. The method also addresses the information leakage problem by restricting models to use only five concepts per prediction, forcing them to select the most relevant ones and making explanations more concise.

The researchers tested their approach against state-of-the-art CBMs on tasks including bird species identification and skin lesion classification in medical images. Their method achieved higher accuracy while providing more precise and applicable explanations. The concepts generated were more relevant to the actual images in the datasets compared to pre-defined concepts.

Trade-offs and future directions

Despite these improvements, the researchers acknowledge a fundamental tradeoff between interpretability and accuracy. Even with their enhanced approach, black-box models that aren't interpretable still outperform them in raw accuracy.

Looking ahead, the team plans to address remaining challenges. One focus is preventing information leakage more effectively, possibly by adding additional concept bottleneck modules. They also aim to scale up the method by using larger multimodal language models to annotate bigger training datasets, which could further improve performance.

The broader implications

This research represents a significant step toward making AI systems more transparent and trustworthy, particularly in safety-critical applications. By extracting concepts directly from trained models rather than relying on pre-defined ones, the approach creates explanations that are more faithful to how the model actually works.

As Andreas Hotho, professor at the University of Würzburg, notes, this work "pushes interpretable AI in a very promising direction and creates a natural bridge to symbolic AI and knowledge graphs." The ability to derive concept bottlenecks from a model's internal mechanisms rather than only from human-defined concepts opens new possibilities for building AI systems that can explain their reasoning in ways that are both accurate and understandable.

The research was supported by multiple organizations including the Italian Ministry of University and Research and Thales Alenia Space, highlighting the practical importance of developing more transparent AI systems for applications ranging from healthcare to autonomous vehicles.

As AI continues to expand into critical decision-making roles, methods that can extract and explain the concepts these systems actually use will become increasingly valuable. This MIT approach offers a promising path forward, potentially making it easier for users to determine when to trust AI predictions and when to seek additional verification.