MoE Interpretability

Reading MoE experts with sparse vocabulary directions and testing when specialization is real

Mixture-of-Experts models are often described as dividing computation among clean, interpretable specialists. This project tests that claim in allenai/OLMoE-1B-7B-0924-Instruct and openai/gpt-oss-20b. Its main contribution is Expert Pursuit, a method that turns expert activations into sparse, readable token summaries. A smaller causal study then checks whether those apparent specializations actually influence generation.

Expert Pursuit

The method adapts HeadPursuit from attention heads to MoE feed-forward experts. The pipeline captures each expert’s gated output across many prompts and runs Simultaneous Orthogonal Matching Pursuit (SOMP) against the model’s normalized unembedding dictionary. The selected token directions form a sparse, readable summary of how that expert’s output varies; explained variance ratio (EVR) measures how much of the variation the summary captures.

Expert Pursuit explained variance across GPT-OSS layers and experts

The heatmap shows the final EVR of every GPT-OSS layer–expert pair. The same analysis covers all $16 \times 64$ OLMoE experts. On the Pile-10k extraction, the median 25-atom EVR is about 0.042 for both models: a small vocabulary basis captures only a limited part of a typical expert’s output variance.

What does an interpretable expert look like?

Aggregate EVR says little about what any single expert does. The token lists make the method concrete: below are representative OLMoE experts found by unrestricted SOMP on 10,000 TriviaQA questions. Subword fragments are omitted for readability.

Expert Candidate specialization Representative top SOMP atoms EVR@25
L15 E03 Numbers and dates 35, 66, 2004, 150, 17, four, twenty, 123, 91 0.256
L14 E37 Numbers and dates 1997, 16, 1970, 26, June, 11, 19, 1930 0.104
L15 E49 Geography Japanese, American, Europe, Ukrainian, British 0.127
L15 E16 Names Ryan, Richard, Robert, Daniel, Garcia, James 0.073
L15 E38 Biology human, chemical, plant, metabolism, food, blood 0.076
L15 E59 Entertainment comic, music, debut, screen, thriller, play 0.071

These readable examples cluster in later layers, although the final unembedding is also a better-calibrated readout there. High EVR does not automatically mean a clean concept: the highest-EVR early expert mixes unrelated atoms such as this, Kindle, Sutton, and History. Expert Pursuit is therefore a sparse summary. It does not establish that every expert carries a single concept.

A targeted query recovers the same specialists

Restricting the pursuit dictionary to number words independently recovers L15 E03 and L14 E37 near the top, even on a different stored extraction. Their unrestricted lists above are already dominated by numerals and dates, so the two searches agree and the specialization is unlikely to be an artifact of the unrestricted token list.

Expert Top atoms from the numbers-only query EVR@10
L15 E03 10, 6, 100, 9, 2 0.041
L14 E37 7, 3, five, 10, 4 0.023

Why use SOMP instead of a logit lens?

The ordinary logit lens reads one direction derived from an expert’s mean output. SOMP instead selects several vocabulary directions that explain how its centered outputs vary across inputs.

Readout EVR@1 EVR@3 EVR@10
Logit lens (one direction) 0.0010 N/A N/A
SOMP sparse basis 0.0020 0.0056 0.0146

Across all 1,024 OLMoE experts, the first SOMP atom captures about 2× the variance of the lens direction; ten atoms capture about 14× as much. The absolute EVR nevertheless remains below 2%. A few specialists are clean; most expert outputs are only weakly aligned with a small set of vocabulary directions.

Causal follow-up

The final stage asks whether readable associations also control generation. Gate attribution patching produces the maps below: countries concentrate in a few late-layer experts, numbers are more distributed, and the offensive-word proxy has no strong sparse handle.

Gate attribution maps for countries, numbers, and an offensive-word concept

The causal study is secondary by design: Expert Pursuit produces readable hypotheses, and gate interventions test which of them actually change the output.