Thought Anchors: Unlocking Precise Reasoning Insights in Large Language Models

Limitations of Current Interpretability Tools in LLMs

Large language models like DeepSeek and GPT variants manage complex reasoning through billions of parameters. However, understanding which reasoning steps most influence their outputs remains challenging. Existing interpretability tools, such as token-level importance or gradient-based methods, provide limited insights by focusing on isolated components and often overlook how reasoning steps interconnect to shape decisions.

Introducing Thought Anchors for Sentence-Level Interpretability

Researchers from Duke University and Aiphabet have developed "Thought Anchors," a novel framework that analyzes reasoning at the sentence level within large language models. An open-source interface at thought-anchors.com enables visualization and comparative analysis of internal reasoning paths. The framework consists of three interpretability methods: black-box measurement, white-box receiver head analysis, and causal attribution. This comprehensive approach reveals how individual reasoning steps influence model outputs and maps meaningful reasoning flows.

Evaluation Using DeepSeek and the MATH Dataset

The black-box method uses counterfactual analysis by removing sentences from reasoning traces to measure their impact. Evaluation involved 2,000 reasoning tasks with 19 responses each, tested on DeepSeek, a 67-billion-parameter Q&A model, and the challenging MATH dataset of around 12,500 math problems. Receiver head analysis examines attention patterns between sentence pairs, identifying directional attention where key sentences guide subsequent reasoning. Causal attribution measures how suppressing specific reasoning steps affects later outputs, clarifying each step's contribution.

Quantitative Results Demonstrate High Accuracy and Clear Causal Links

Black-box analysis highlighted that correct reasoning paths consistently exceeded 90% accuracy, outperforming incorrect ones. Receiver head analysis showed strong directional attention with an average correlation score of 0.59 across layers. Causal attribution experiments quantified influence propagation, with an average causal influence metric of 0.34. Additionally, attention aggregation analysis of 250 attention heads in DeepSeek revealed that certain receiver heads consistently focus on critical reasoning steps, especially in math-related queries, offering deeper insight into model decision mechanisms.

Practical Implications and Future Directions

Thought Anchors significantly improve interpretability by targeting sentence-level reasoning, outperforming traditional activation-based methods. The open-source tool facilitates collaborative exploration, and attention head categorization informs potential architecture optimizations. These advances provide a foundation for deploying sophisticated LLMs safely in sensitive areas like healthcare and finance. The framework also opens pathways for future research to enhance AI transparency and robustness.

For more details, explore the paper and interactive demo at thought-anchors.com.