This post will probably not be very informative to readers who don't know what an SAE (Sparse Autoencoder) is.

An interpretability researcher once told me, "SAEs can never fail, they can only be failed." But recently, SAEs have been suffering failure after failure, and as a result, the field has grown increasingly pessimistic about SAE work. Korznikov et al.'s Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? purports to be an example of such failures.

However, I have some disagreements with its methodology. The paper presents a synthetic setup with known ground truth features, and claims that SAEs fail to recover features in the "simplest toy setting with fully aligned hyperparameters." In this post, I argue that their setting is not forgiving to SAEs, as the authors imply, but is instead actively hostile in ways that real settings are not.

Introduction

Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? asks two important questions:

1. Can SAEs actually recover real features? Hypothetically, if we knew what the "true" features of the data were, would an SAE be able to identify them?
2. Do trained SAEs actually do better than completely random SAEs?

This post will be focused on their answer to the first question.

The authors train an SAE on a synthetic toy model, with known ground-truth features, and according to their abstract, SAEs only recover 9% of true features. On the face of it, this seems pretty damning!

Now, earlier toy model experiments did show that SAEs could recover ground truth features, but the authors argue this is because they used a small expansion factor, where there were only twice as many "true" features as there were model dimensions, and that this is unrealistic. Their setup uses an expansion factor of 32, which matches real-world use cases much more closely.

So, how does their setup work?

They don't actually construct a synthetic neural network; instead, they create synthetic activations directly. First, they create their feature vectors, by sampling 3200 vectors from the unit sphere in 100-dimensional space. To create an activation, they just select about 20 feature vectors at random, scale each active feature by a magnitude drawn from LogNormal(0, 0.25),^[1] and sum the scaled vectors.

They test two different regimes, one where each feature is equally likely to be active, and another where feature activation frequency has a long tail, so some features fire more often than others. (We'll focus on the constant regime for the most part)

They match their SAE to the setup. There are about 20 expected active features, so they target L0 = 20. There are 3200 true features, so they create 3200 SAE latents. Can the SAE, when given every possible advantage, recover the true features?

These results are, again, pretty damning! They claim that the SAE achieves high reconstruction accuracy while only recovering 3/3200 true features—personally, I disagree that 0.67 explained variance is "high reconstruction accuracy".^[2] However, the SAE clearly only does slightly better than random guesswork at recovering the true features.

Now, one obvious objection is that L0/d_model here is 0.2, which is unusually dense. I'd expect there to be a lot of interference here, and in the real world L0/d_model is probably going to be closer to 0.05. Also, 100 dimensions is just not a lot, and maybe you can't really get blessing of dimensionality from it like you would in real models.

A less obvious objection is that if you scroll all the way down to Appendix B, they describe some other SAE architectures they used for this task, like TopK and Matryoshka-on-BatchTopK.

In contrast to BatchTopK, TopK recovers 99.9% of features.^[3]

So, I did what anyone would do, and started replicating the paper. They didn't actually include all the hyperparameters they used,^[4] so I made reasonable assumptions.

I was able to replicate this result! TopK recovered 99.6% of features,^[5] and the other architectures failed miserably.^[6]

Why does BatchTopK fail?

This is kind of strange, if you think about it. TopK and BatchTopK are the same architecture, with the only difference being the selection rule.^[7] Such a dramatic gap is really weird.

Peeking a bit closer at my training run, in every batch, most latents never fired. That means a small group of monopolist latents were dominating every time. By step 5,000, the same ~6% of latents took over, and they remained in charge for the remaining 295,000 steps. In comparison, with TopK, after the first 20k steps, >99% of latents activated on every batch.

This is the standard dead latents problem: There's no gradient on latents that aren't selected, so once they fall behind in performance it's hard for them to improve. A small group of generalist latents dominate before specialists have the chance to form. The standard fix, of course, is an auxiliary loss, AuxK. But even with AuxK, the dead latent fraction barely budges. I tried sweep