Interpretability in the Era of LLMs

Abstract

The rapid evolution of Large Language Models (LLMs) has brought unprecedented capabilities across reasoning, coding, and multi-modal tasks. However, as performance scales, their opaque "black-box" nature raises a critical challenge: How can we trace the origins of emergent intelligence, and more importantly, how can we leverage these internal mechanisms to guide model optimization?

This tutorial provides a comprehensive, end-to-end view of LLM interpretability, transitioning from microscopic neural analysis to macroscopic application and deployment. It is systematically organized into five core sections: i) Unlocking the Black Box: We begin with the evolution of LLM interpretability and highlight recent breakthroughs from leading research teams. ii) Methodology: We present a rigorous overview of foundational theories (e.g., mathematical framework for transformer, biological mechanisms in LLMs) and essential methods (e.g., path patching, logit lens, and neuron description). iii) Anatomy of LLMs: Using advanced techniques to decode internal semantic features, neural circuits, and complex behaviors, we interpret how models perform reasoning, factual recall, and in-context learning. iv) Applications: We show how to transfer interpretability insights into actionable improvements across the LLM pipeline, including interpretability-guided data synthesis (data value scoring, corpus filtering, and activation-based data diagnosis). We also present Pinpoint Training and Steering for precise capability gains, and Pinpoint Quantization for extreme low-bit compression with minimal capability loss. v) Advanced Topics: We conclude by exploring how these interpretability paradigms scale and inspire the design of frontier architectures, agentic systems, and thinking models.

In this tutorial, researchers and engineers will gain the theoretical frameworks and practical engineering toolkits needed to understand, steer, and efficiently deploy LLMs in real-world production environments.

What You Will Learn

From microscopic neural analysis to macroscopic industrial deployment

Unlocking the Black Box

Evolution of LLM interpretability and recent breakthroughs from leading research teams (Anthropic, OpenMOSS).

Methodology & Theory

Mathematical frameworks for transformers, biological mechanisms, path patching, logit lens, and neuron descriptions.

Anatomy of LLMs

Decode internal semantic features, neural circuits, and complex behaviors like reasoning, factual recall, and ICL.

Industrial Applications

Data synthesis, corpus filtering, pinpoint training for precise gains, and extreme low-bit quantization.

Target Audience & Prerequisites

Who Should Attend

AI/ML Researchers — Trustworthy AI & Mechanistic Interpretability
LLM Engineers — Training, fine-tuning, and aligning LLMs
Data Scientists — Data mining & LLM cognition intersection
Deployment Engineers — Model compression & serving frameworks

Expected Background

✓ Solid foundation in ML/DL (Transformer architecture)
✓ Basic LLM training pipeline (Pre-training, SFT, Alignment)
✓ Python & PyTorch proficiency

Helpful (Not Required)

★ Experience with sparse autoencoders
★ Familiarity with causal tracing / probing

Tutorial Schedule

30 min Part I

Introduction

Lecture

The evolution of LLM interpretability
Key recent breakthroughs from leading research teams

30 min Part II

Fundamentals of LLM Interpretability

Lecture

Foundations: residual connections, MLP neurons, attention heads
Interpreting each individual component
Core methods: LogitLens, Path Patching, Sparse Autoencoders

45 min Part III

Advances in Mechanistic Interpretability

Lecture

Decoding semantic features and neural circuits
Advances in sparse coding methods
Interpreting reasoning, factual recall, and in-context learning

30 min

Coffee Break

45 min Part IV

Interpretability-Driven Applications

Lecture

Training: Multilingual Enhancement
Inference: LLM Quantization
Data: Data Synthesis/Selection

30 min Part V

Advanced Topics & Future Directions

Lecture

Reasoning and Agentic Coding
Inspiring the design of LLM architecture
Safety Awareness

30 min Q&A

Q&A

Open Discussion

Presenters

Jieping Ye

Alibaba Group

IEEE Fellow (2019) & ACM Distinguished Scientist (2020). 450+ papers, 54,500+ citations, h-index 117. Research in ML, data mining, and AI. Honors include NSF CAREER Award and KDD Best Student Paper Award.

Zhengfu He

Fudan University

Leads interpretability research at OpenMOSS. Pioneered SAE scaling to Llama-3.1-8B, novel attention replacement models (Lorsa), and Complete Replacement Models for circuit tracing. 6 papers at ICLR/ACL/EMNLP, 900+ citations.

Xu Shen

Alibaba Group

Leads the Large Model Interpretability team. Ph.D. from USTC. Focus on internal mechanisms and operational principles of large models. 40+ papers at ICLR/ICML/NeurIPS/ACL/CVPR, 2,400+ citations.

Contributors

Xuanjing Huang

Fudan University

NLP and LLM research. 300+ papers, 30,000+ citations, 8 outstanding paper awards. Executive Committee Member of ACL, Chair of AACL.

Wei Zhang

Alibaba Group

Ph.D. from USTC. Research on internal computational mechanisms and interpretability-driven approaches. Publications at ICML, CVPR, ICCV.

Lulu Zhang

Alibaba Group

Research collaboration and academic engagement. Workflow Chair for KDD 2023 ADS Track, co-organized KDD Cup 2020.

Key References

Circuit Tracing

Ameisen et al. "Circuit Tracing: Revealing Computational Graphs in Language Models." Transformer Circuits Thread, 2025.

Biology of LLMs

Lindsey et al. "On the Biology of a Large Language Model." Transformer Circuits Thread, 2025.

LlamaScope SAE

He et al. "Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders." 2024.

Pinpoint Tuning

Chen et al. "From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning." ICML, 2024.

Complete Circuit Tracing

Shu et al. "Bridging the Attention Gap: Complete Replacement Models for Complete Circuit Tracing." OpenMOSS, 2026.

Attention Decomposition

He et al. "Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition." ICLR, 2026.

Tutorial Slides

Download or view our tutorial materials

Full Tutorial Slides

Complete slide deck covering all five parts of the tutorial.

Download PDF

Interpretability Reading List

A curated collection of key papers and resources on LLM interpretability.

View

Citing This Tutorial

If you find this tutorial helpful, please cite:

@inproceedings{zhang2026interpretability,
  title={Interpretability in the Era of Large Language Models: Mechanistic Methodology, Empirical Practices, and Applications},
  author={Zhang, Wei and He, Zhengfu and Zhang, Lulu and Huang, Xuanjing and Ye, Jieping and Shen, Xu},
  booktitle={Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD '26)},
  year={2026},
  address={Jeju Island, Republic of Korea},
  doi={10.1145/3770855.3816451}
}

Contact

For any questions regarding this tutorial, please reach out to:

xiwei.zw@alibaba-inc.com