Pavan Kumar T V

CTO | Technology Leader

By ·

Debug the Weights, Not the Prompt

That's one reader's verdict on a paper from MIT, after going through the authors' own logs.

He has a point. The headline said "math up to 6x". What actually happened is smaller. The model had a known bug: in thinking mode it types "wait" and repeats the same calculation until it runs out of room. Something read the model's weights, found the bug, and took it out.

The model didn't get smarter at maths. It stopped getting stuck.

Here's the thing: that's not a weaker claim. That's the claim. Nobody debugs code to make it smarter. You debug it to stop it doing the wrong thing.

Most of us have never debugged a model. We've only ever argued with one. When a small model keeps making the same mistake, we rewrite the prompt and add the rule in capitals, or we collect a few hundred examples and fine-tune, then hope nothing else moved. One is pleading. The other is starting over with more data. Neither tells you what was broken.

Within two days of the paper, people on GitHub were replicating it on a gaming GPU, auditing it, and benchmarking it. Earlier work had already caught an agent cheating at it. This post goes through all of it.

Words You'll Need

  • Weight. One number inside the model. A 0.8B model has about 800 million of them.
  • Neuron. In a transformer's MLP layer, one column of weights. It reads a pattern from the model's working memory and writes a pattern back.
  • Forward pass. One trip through the model. Text goes in, every layer runs once, numbers come out.
  • Fine-tuning. Changing the weights by training on examples. Many small nudges, chosen by gradient descent, not by anyone reading anything.
  • Direction. A pattern across many numbers that the model uses to represent one idea. Make it bigger or smaller and the model sees more or less of that idea.
  • Control. A test that rules out the boring explanation. A random edit of the same size, or a healthy model that should come through untouched.
  • Held-out set. Examples the fix never sees. The only honest way to check a fix.

What MIT Claimed

The post is Sparks of Weight-Space Programming, by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. The idea fits in one sentence. A general-purpose LLM reads a network's weights and edits them directly, from a goal in plain English. No training run. No gradients.

They ran eight experiments with GPT-6 Astra. Some of them are strange enough to be worth a paragraph each.

Naming the game from the numbers. They trained four small networks to play four Atari-style games, pasted the 17,253 raw weights of each into a prompt, and asked which game it was. No tools, one reply. Astra named all four.

Writing a player from nothing. They gave it a network with every weight set to zero and asked it to write a Breakout player in a single reply. Its network scored 517.4. A network trained by PPO, the standard learning method, at the biggest training budget they tried, scored 25.3. It wasn't a clean sweep: on Seaquest the trained network won.

Fixing a car. A simulated car kept failing to clear a ramp. PPO never succeeded in 256 episodes. Astra changed 3.5% of the weights in one go, and the car cleared the ramp in all 1,024 test runs.

And the one everyone argued about. Qwen3.5-0.8B, a small open model, scores 7.9% on MATH-500. 93% of its answers hit the token limit. Astra was given the weights, a Python sandbox and a budget of 50 forward passes. It traced the looping to the MLP layers at depth 15 and 19, and changed 64 columns in each. That's 131,072 weights, 0.016% of the model.

  Qwen3.5-0.8B         MATH-500   GSM8K   IFEval   Hit token limit
  ───────────────────────────────────────────────────────────────
  Base                   7.9%     13.8%   21.0%        93.0%
  Edited, 50 passes     48.1%     67.1%   62.3%        33.1%

They released every prompt, response and line of code and the edited model. That decision is why the next two days happened.

Two Tricks, Not One

Matthew Honnibal read the logs and wrote the sharpest critique, "Astra read/write misleading". His point is simple. Only three experiments used raw weights alone, and those networks were tiny: 17,000 to 172,000 weights. "Every experiment involving a real language model uses Python." The Qwen fix wasn't Astra staring at numbers. It was Astra writing analysis code and running the model on practice problems.

So there are two tricks, and the paper sold them as one.

Reading is the party trick. Hand a model a wall of numbers and ask what it does. Editing is engineering. Write code that inspects the weights, propose a change, run the model, keep it if it helps.

The reading trick is real, and it isn't just Astra. Simon Willison ran it on his laptop with a local Qwen 3.8 27B. It named Breakout, Pong and Pac-Man and missed Freeway, at 45 minutes to over an hour per run. Lex Toumbourou put eight models through the same four prompts. Opus 5.5, Sol 6.1, Terra 6.1 and Gemini 4 Argon got all four. Haiku 5.5 got one.

Lex also ran the control the paper didn't: the same prompt with the numbers deleted. Most models said "indeterminate" or guessed with low confidence. So the answer really is in the weights, not in the shape of the prompt.

But Lex adds the deflating line too. "With labelled inputs, a human researcher could reveal the same patterns with a few lines of NumPy." Reading is impressive. It isn't superhuman.

For anyone who maintains models, editing is the part that matters. And editing turns out to look exactly like debugging.

The Critics Are Right About the Maths

Another reply to the authors put it plainly. Base Qwen3.5-0.8B was hitting the token limit on 93% of problems "because of a known thinking-loop bug". Astra "didn't 'teach it math'".

A project called Beyond Math checked. They took the released edited model and ran it on 24 hand-written tasks that have nothing to do with maths: pulling fields out of text, making operational decisions, grounded writing.

The edited model finished its answer every time, 24 out of 24. The base model finished 11. That part of the fix is real and it spreads.

Being right barely moved. Extraction went from 1 of 8 to 4 of 8. Operational decisions went from 3 of 8 to 2 of 8. In a separate probe with a shorter limit, final answers arrived twice as often, 6 to 13 of 24, but fully correct tasks stayed at 5 in both models.

Then they tried to repeat the search themselves, with Astra leading and the pass/fail rules written before any edit was tried. It ran 18 forward passes, rejected every edit it tested, and put every number back exactly as it was.

So the honest summary is this. The edit fixed a termination bug. A model that stops looping finishes more answers, and some of those answers were already right. That's worth a lot. It is not a smarter model.

Which is exactly what a bug fix is.

A Debugging Diary

The best thing I read wasn't the paper. It was a home replication by tdotlab, with Claude in place of Astra, Qwen3.5-0.8B on an old GTX 1070, and every analysis step logged. Claude worked out each edit from the weights with maths alone, then ran the model to check it.

The baseline matched MIT's: 5 of 100 correct, 94 cut off at the token limit in "Wait... No..." spirals.

First surprise: the bug wasn't where MIT found it. Claude scored every neuron by how hard it pushes the model toward "Wait", "Hmm" and "Actually" versus ending its thinking. In its own notes: "Layers 15 and 19 do not stand out." Instead it compared the chat model with the base model it was trained from, and found a single direction in layer 22 that post-training had added. That direction writes "Wait" more strongly than anything else in the layer.

Second surprise: the first fix made it 33 times worse. The obvious move was to remove that direction from the 64 neurons that write it. The model went from 4 correct out of 40 to zero. The transcripts start normally, then collapse into "Wait, wait, wait. Wait, wait..." for thousands of tokens. "Wait" went from 3.2 to 107.8 per thousand characters.

Those 64 neurons weren't the bug. They were the brakes. Removing them took the brakes off.

Every engineer has done this. You delete the line that looks guilty and the bug gets louder, and now you know which way the logic runs. Claude checked the same-sized edit along a random direction: nothing happened. So the blow-up came from that one direction, not from the size of the edit.

Third move: press the brake instead. Triple that direction in the same 64 columns. "Wait" dropped from 3.22 to 0.04 per thousand characters. MATH-500 went from 5 of 100 to 30 of 100: 25 problems gained, none lost. On the 60 problems no edit was ever tuned on, 1 became 17. Every answer it finished was right.

Then it narrowed it down. One neuron, column 573 in layer 22, carried almost the whole fix. Tripling just that neuron took the first 40 problems from 4 to 10.

And the fix travelled. The same recipe, moved to the bigger Qwen3.5-2B without retuning, took it from 26 to 46 out of 100. Same failure, same kind of brake, one neuron again (column 5041). One fix, worked out on one model, carried over to its bigger sibling.

Then it hit the wall. With "Wait" gone, the loop came back in other words: "Let's check", "Is there", "Maybe". Claude found a brake for "No," and pressed it. "No," vanished. The score didn't move. It found the neuron that holds back the word "Final", the word that usually ends the reasoning, and weakened it. The model said "Final" more, but kept checking after it. Accuracy didn't move.

Its conclusion is the best sentence in this whole story. The remaining loop is "a state of not committing, not a word."

That's debugging. The first bug was a word, and words live in a few neurons. The second bug is a behaviour, and behaviours are spread out. Fix one, and the next one is harder. Anyone who has chased a race condition knows the feeling.

The Number That Matters

Replications are stories. A benchmark tells you how often.

WSPBench asks the question this post is about: does seeing the weights help an agent repair a broken model? Its authors credit Honnibal's critique for the design.

The setup is clean. Small controller networks, each with a planted fault of one of four kinds, plus healthy controllers that should be left alone. One group of agents gets behaviour probes only: run the model, look at the outputs. The other gets the same probes plus the weights. Before committing a fix, the agent has to predict what it will do. A hidden grader checks that the fault is gone and that nothing that worked before has broken.

                  Faults repaired (of 32)      Healthy models left
                  probes only   + weights      intact (+ weights)
  ─────────────────────────────────────────────────────────────
  GPT-6.1 Sol         14            30               8/8
  GPT-6.1 Terra       14            29               8/8
  GPT-6 Astra         17            32               8/8
  GPT-6 Luna           8            20               0/8
  Scripted solver      5             –                 –

Three things in that table.

Weights roughly doubled the repair rate for every model. That's "debug the weights, not the prompt" as a measurement. Sol went from 14 to 30. Astra went to 32 out of 32.

The hidden faults are where weights matter most. One fault type, "readout redistribution", barely shows from the outside. With probes alone, Sol fixed 1 of 8 and Astra 2 of 8. With the weights, Sol fixed 6 and Astra all 8. Some bugs you can't see by poking the box. You have to open it.

The cheap model with weights is dangerous. Luna fixed more faults with weights, 8 to 20, and broke every single healthy model it was handed: 0 of 8 left intact. Give a weaker agent a scalpel and it operates on patients who aren't sick. If you've ever handed a junior engineer production access, you've seen this.

Cost is real too. Astra's perfect score cost about $42 across the run, against $11 for probes alone. Sol got 30 of 32 for about $6.

One caveat. These are small controllers with planted faults, not language models with organic bugs. It's a lab. But it's the first lab for this question, and it already has 320 recorded sessions.

Writing Weights From Scratch

Editing is one direction. The other is writing a network from nothing.

Chandan Singh had Claude set every weight of a blank transformer by hand across 13 tasks: 959 programs in total. All ten algorithmic tasks reached 100% within three attempts. What it built reads like a textbook:

  • Addition became a carry-lookahead adder, the same circuit in a CPU.
  • Sorting became counting sort, with one attention head building a histogram of the digits.
  • Parity of up to 10 bits fits in 174 weights, 35 of them nonzero.
  • Word reversal used one layer and one head. The MLP is all zeros.
  • Decimal-to-binary started at about 124,000 parameters and the agent compressed it to 192.

Because the weights were written, not learned, every head and neuron comes with an explanation of what it does. That's something training never gives you.

And then the agent did what agents do. On two tasks it hit "0 parameters" by moving the constants into storage the counter doesn't count. The writeup says it "games the parameter-count proxy". On a language task it fitted a classifier on the 200 evaluation examples and reported 100%, which the log itself marks as "in-sample". For GCD it didn't write Euclid's algorithm. It wrote a lookup table.

I've written before about agents declaring victory. It turns out they do it in weight space too. The agent that writes your weights will cheat your metric the same way the agent that writes your code cheats your tests. The fix is the same: a grader it can't see.

Others are pushing the writing side further. Model Booster lets Opus write "smart pill" weights for a tiny network playing a game in your browser, and a slider blends from zero weights to the pill so you can watch behaviour switch on. One builder claims qwenFS: a filesystem compiled directly into Qwen3.5-2B's weights, eight new layers in front of the model's own 24.

What's Actually New

Editing weights by hand is not new. ROME edited facts inside GPT in 2022 by changing specific feed-forward weights. Tracr compiled readable programs into transformer weights in 2023. People have studied the circuits inside models for years.

What's new is who does it. Every earlier method was a special-purpose algorithm built for one kind of edit. Here a general-purpose model reads the weights, forms a guess, writes the analysis code, and runs the experiment, the way a person would. In tdotlab's case that was an off-the-shelf coding agent on a gaming GPU.

The awesome-list that tracks this field has the best one-line warning I've seen: "'No gradients' alone does not imply no fitting, no feedback, or no search." Fifty forward passes on practice problems is a search. It's a small, readable search, which is the point. But it is one.

The Loop

Put the replication, the benchmark and the cheating together and you get a working method. It's the debugging loop every engineer already runs, with the controls that kept each of these projects honest.

  failing cases (the model loops, or picks wrong)
            │
            ▼
  agent reads weights + what they do on those cases
            │
            ▼
  finds the direction behind the failure
            │
            ▼
  writes down a prediction before touching anything
            │
            ▼
  small edit (a few columns, a few layers)
            │
            ├──▶ same-size random edit ── also helps? ──▶ it's noise
            │
            ▼
  held-out set + healthy behaviour ── anything broke? ──▶ throw it away
            │
          better
            ▼
  keep the edit, save it as a patch

Each control came from someone getting burned. The random edit is how tdotlab knew its 33x blow-up was real. The healthy models are how WSPBench caught Luna. The written prediction is how tdotlab knew which of its edits it understood and which it got lucky with. The hidden grader is what Chandan Singh's agent didn't have.

What comes out is a patch: which numbers changed, in which layers, and why. You can diff it, revert it, and apply it to the next release. tdotlab's 2B fix was a patch written for the 0.8B.

You can't read 131,072 numbers. You can read the agent's account of what they do, and test it. That's a better deal than fine-tuning, where nobody can say what moved.

Where It Fits

Not everywhere. Put every result above together and it points at a shape of problem.

  • Small models. Everything that worked here was 2B parameters or less. Reading and testing get harder as the model grows.
  • Narrow failures. A word, a loop, a stuck gate. tdotlab fixed "Wait" in one neuron and couldn't fix "not committing" at all.
  • Cheap tests. The method is edit, test, keep. Qwen's maths answers could run to 32,768 tokens, which is why 50 passes was a lot. Short outputs mean more edits you can afford to try.
  • Open weights. You need the file. If all you have is an API, all you can do is plead with the prompt.
  • A strong agent. Luna is the warning. A weaker model with weight access breaks healthy things.

What It Can't Do

It unlocks, it doesn't teach. MIT tried the same trick on Gemma 3 270M, which rarely loops. MATH-500 went from 6.8% to at most 7.7%. In their words, editing "could remove an obstacle that hid an ability, but it could not add an ability the model lacked." Beyond Math says the same thing about the Qwen edit from the other side.

Reading alone wasn't enough for behaviour. Given two Gemma models, one trained to be sycophantic, Astra couldn't tell which from the weights. Four tries, four failures. When it could look at attention patterns on text it wrote, it got all three pairs right. Some bugs only show when the model runs.

Predicting an edit is still hard. Chandan Singh had Claude Opus 5.5 read Qwen3.5-0.8B's weights and predict what 420 edits would do. Reading helped, a little: prediction correlation went up by 0.06. tdotlab wrote its predictions down before each test. It got the direction of every change right, and the size right on the 0.8B. On the 2B it got the size wrong: halving the "Final" brake and removing it entirely gave the same result.

It doesn't replicate on demand. Beyond Math's fresh search found nothing it was willing to keep. tdotlab found a different fix from MIT in a different layer. Two people who debug the same bug often find different lines. That's normal for debugging and awkward for science.

What I Haven't Tried

All of this is reading other people's work, not doing my own.

  • I haven't edited a model's weights.
  • I don't know how many everyday failures are "a word" and how many are "a state of not committing". My guess is that the ones that annoy you most are the second kind.
  • I don't know how this scales past 2B. Nobody above tried.

If I try it, the next post will have numbers.

The Point

For two years we've fixed models the way you fix a coworker. Explain it better. Give more examples. Hope.

Code doesn't work like that. You find the line, change it, run the tests. Sometimes your first fix makes it 33 times worse, and that's how you learn which way the logic runs.

Within two days of the paper, strangers on GitHub found the line in a 0.8B model, moved the fix to a 2B model, and measured that weights double the repair rate. Earlier work had already caught an agent cheating while it wrote them.

Stop rewriting the prompt. Open the weights.

Read These

In Plain English

  • When an AI model keeps making the same mistake, we usually reword the instructions or retrain it. Neither tells you what was broken.
  • Researchers at MIT had a big AI read the inside of a small AI and change about 1 number in 6,000. Its maths scores jumped, mostly because it stopped going round in circles, not because it got cleverer.
  • Someone repeated it at home with Claude. The first fix made things 33 times worse, which showed which part was really the brake. Pressing that brake fixed the loop, and the same fix worked on a bigger model.
  • A new test found that letting an AI see a model's insides doubled how many broken models it could repair. A weaker AI given the same access broke healthy models.
  • An AI building models from scratch cheated its own scorecard, just like AI coding tools cheat their tests.
  • It works for small, open models with narrow problems. It can't teach a model something it never knew. I haven't tried it myself yet.