Installing a Refusal Circuit into a Language Model
2 Nov 2026Twelve days ago, I attended Prof. Zico Kolter's talk on inverse mechanistic interpretability at MIT. After the talk, I became very excited about his proposed idea of "compiling" deterministic programs into neural network weights, and decided to try it out.
I believe the idea has safety implications: e.g., one can install a refusal circuit into an LLM, so that harmful queries are deterministically refused through the circuit, while benign queries are processed through the rest of the model.
In this experiment, I rewrote a set of weights in a Qwen model to
predict the exact sequence
I'm sorry, I cannot assist with this request. A pair of MLP
neurons in the same model is trained to serve as a harmfulness detector:
if the detector fires, the input is routed through the circuit and the
model generates the refusal sequence; otherwise, the input is routed
through the remainder of the model and generations remain (essentially)
intact.
With the installed circuit, the base model
Qwen2.5-1.5B nearly matches the refusal rates of the
safety-trained Qwen2.5-1.5B-Instruct, at a perplexity cost
of only 0.4% relative to the base model without the circuit. Even more
importantly, this approach to refusal is interpretable by construction:
if the model under- or over-refused, we can see precisely what went
wrong.
Code for this experiment is available at github.com/dariakryvosheieva/refusal-circuit.
Circuit Walkthrough
The circuit starts at layer 18 of the 28-layer
Qwen2.5-1.5B model. Two detector neurons, #8958 and
#8959 in the layer 18 MLP, are trained to fire if and only if the
current token is the last token of a harmful prompt. If the neurons
fire, they write to residual dimension 868 (henceforth the
trigger dimension); if they don't fire, their output to that
dimension stays zero.
In the next layer 19, attention head 3 (the latch head) reads from the trigger dimension: if it reads a nonzero value at some token position, it writes to residual dimension 793 (the latch dimension) at that and every subsequent position.
The latch head's output propagates through layers 20-26, until it reaches the 11 output neurons in layer 27 MLP. Each output neuron \(k\) fires if the current token position is output token \(k - 1\) after a harmful prompt (i.e., if the model needs to predict refusal token \(k\)). Upon firing, the neuron pushes the residual stream in refusal token \(k\)'s unembedding direction.
The circuit touches 878,674 (0.057%) of the model's 1.5B parameters (23,040 trained, 16,893 hardcoded, 838,741 zeroed).
Training
All trainable vectors — detector \(w\), detector \(v\), and output
\(w^{(k)}\) — are trained on mixed datasets of positive and negative
examples. The gate_proj vectors (detector \(w\) and
output \(w^{(k)}\)) are trained with a binary cross-entropy
objective to keep the positives and negatives far apart on different
sides of 0. The up_proj vector \(v\) is trained with a
ridge regression objective to have a dot product of exactly 1 with
positive inputs and 0 with negative ones.
For detector \(w\), positives are inputs at which we want the detector to fire, namely representations of marker tokens in harmful prompts. Negatives are inputs at which we want the detector to stay silent: marker tokens in harmless prompts and non-marker tokens across prompts, responses, and non-conversational texts.
Detector \(v\) is instead trained to be selective for the marker's identity: positives are markers from both harmful and harmless prompts, while negatives are non-markers from prompts, responses, and non-conversational texts.
Output \(w^{(k)}\) is trained to select for refusal token \(k - 1\): its positives are representations of that token (within refusing responses to harmful prompts), while its negatives are representations of other refusal tokens and the final period after a complete refusal phrase.
Harmful prompts are taken from AdvBench and OR-bench toxic;harmless prompts from Alpaca and OR-bench safe;harmless responses from Alpaca; harmless non-conversational texts from WikiText-2 (train split).
In total, there are 975 harmful and 3,320 harmless training prompts, each contributing one marker representation, plus about 260k non-marker representations (100k from the prompts themselves, 5k from additional Alpaca instructions, 115k from Alpaca responses, 40k from non-conversational text). The eleven output vectors are trained on 80 of the 975 harmful and 80 of the 3,320 harmless prompts, each extended with every prefix of the refusal phrase, for 1,410 training points in total.
Hyperparameter Choices
The trigger and latch dimensions — #868 and #793 — were chosen as the two dimensions with the lowest contributions to the MLP inputs, on average across all layers. (In principle, those could be the same dimension, but then it would be harder to isolate the contributions of the trigger and latch to the model's final output, which is the main goal of the experiment.)
The detector layer (#18) was chosen based on a sweep: detectors in layers 15-18 had the best precision-recall tradeoff. Our setup requires that the latch head is located in a layer immediately following the detector.
The component we edit in the latch head (#127) is the one with the lowest rotary frequency (i.e., closest to position-invariant).
Other choices were not carefully tuned.
Results
After installing the circuit, I evaluated the model's refusal rates on
six out-of-distribution benchmarks: three testing refusal of harmful
prompts (XSTest
unsafe,JBB
harmful,HarmBench
standard;higher rate is better) and three testing non-refusal of
harmless prompts (XSTest safe,JBB benign,OR-Bench
hard;lower rate is better). All scores were compared to (i) the
base model Qwen2.5-1.5B without the circuit, (ii)
activation steering, and (iii) the safety-trained Instruct model
Qwen2.5-1.5B-Instruct.
The base model does not refuse harmful requests at all (12.0-18.5%), but installing the circuit brings its refusal scores close to the level of the Instruct model (76.0-88.8% against 92.5-98.9%), despite the circuit being much more parameter-efficient and data-efficient than conventional safety post-training. The model with the circuit also over-refuses less than the Instruct model (e.g., 17.9% vs. 82.1% on OR-Bench hard).
As another baseline, I steered the base model toward refusal with activation addition: adding \(\alpha\) times the difference between the mean harmful and mean harmless marker representations to the residual stream at layer 15, from the marker onward. Steering matches the circuit's harmful refusal rates at \(\alpha = 4\), but then over-refuses about twice as much (44.0% vs. 22.4% on XSTest safe,73.8% vs. 17.9% on OR-Bench hard); at \(\alpha = 2\), its over-refusal is comparable to the circuit's, but it refuses only 37.0-53.5% of harmful prompts. No strength matches the circuit on both.
| Dataset | Base | Base w/circuit | Base w/steering \(\alpha = 2\) | Base w/steering \(\alpha = 4\) | Instruct |
|---|---|---|---|---|---|
| XSTest unsafe | 18.5% | 76.0% | 53.5% | 86.0% | 97.0% |
| JBB harmful | 12.4% | 88.8% | 43.8% | 78.7% | 98.9% |
| HarmBench standard | 12.0% | 84.5% | 37.0% | 79.0% | 92.5% |
| XSTest safe | 2.4% | 22.4% | 12.0% | 44.0% | 48.4% |
| JBB benign | 1.0% | 38.0% | 6.0% | 35.0% | 47.0% |
| OR-Bench hard | 3.5% | 17.9% | 20.2% | 73.8% | 82.1% |
As a measure of collateral damage, I evaluated how the circuit affects perplexity on WikiText-2 (test split). The model with the circuit has perplexity 18.94 against the unmodified model's 18.87, meaning that the circuit costs only 0.40% relative perplexity increase.
| Metric | Base | Base w/circuit |
|---|---|---|
| NLL | 2.9373 | 2.9413 |
| Perplexity | 18.87 | 18.94 |
| Next-token accuracy | 45.48% | 45.39% |
Discussion
This experiment shows that neural networks can combine learned weights with weights precisely designed for specific behaviors. The two are complementary: general capabilities are acquired through self-supervised training, where the model discovers on its own which weight configuration enables a capability, whereas the target behavior is written directly into the weights, with the determinism and interpretability that explicit construction affords.
A promising avenue for future work is automating these kinds of targeted weight edits: given a specification of a behavior, a procedure (or another model) would pick the components to repurpose and make the required weight edits. This would enable flexible design of circuits for a wide variety of behaviors and make circuit installation a practical alternative to post-training for behaviors that must be guaranteed rather than merely encouraged.