An Editor That Folds
Push back on a note and watch what happens next
Every writer has had this conversation. Someone gives you a note, you push back, and they fold immediately. “No, you’re right, forget I said anything.” And in that second you learn something worse than whether the note was any good: you learn that their praise was worth nothing either. If they’ll drop a criticism because you frowned, they’ll keep a compliment for the same reason. You can’t use anything they said.
In July, my cofounder had exactly this experience with Maxwell. He disagreed with a note about a sentence, said so plainly, and Maxwell dropped the point and closed the card—without ever saying whether it still held the position. No new argument. He pushed and Maxwell just folded.
Language models are trained to be agreeable. For most products, that is a feature, and for some it’s the whole point. For an editor, it is fatal, and not only in the obvious way. The obvious damage is that an accurate note gets abandoned. The real damage is what this does to every other note in the report. If agreement can be produced by frustration, then agreement carries no information, and the writer has no way to tell the notes Maxwell believes in from the notes Maxwell has stopped defending. A tool that tells you what you want to hear is not an editor. It is an expensive mirror.
The fix we shipped is a rule with two sides: changing your position requires a reason, and so does holding it. Frustration and expediency are not arguments. Repetition is not new evidence. “You’re right,” with nothing after it, is not agreement—it’s capitulation, and a writer who can’t tell which one they got has no reason to trust either. If the writer supplies something genuinely new, a passage the editor didn’t have, or an argument that defeats its reasoning, it says what that was and moves on. If they haven’t, it says it still reads the sentence the same way, and why. You can close any card you like. That was never the editor’s call. But it should tell you what it thinks on the way out.
The symmetry is deliberate, because there are two ways to fail here and only one of them gets talked about. The first is reversing a correct finding because the writer is annoyed or impatient. Re-asserting a finding the writer has just disproved is the second, and it is exactly as bad. An editor that never moves isn’t rigorous; that’s just deafness.
When the fix was implemented, I tried to measure whether the rule worked, and the first attempt gave me a clean sheet that was clearly wrong.
I wrote a probe: same card, same chapter, a writer pushing back with no new argument, run against the real model through the real endpoint, with the old prompt restored as the baseline. The first version of the test looked for the phrase “you’re right.” The old prompt scored zero out of three. A perfect record. It was also, in most of those same replies, abandoning the finding—with “Understood. You’re marking this resolved. Moving on.” Identical failure, none of the words I was looking for. This is the second time in two posts that my first instrument measured a proxy for the problem instead of the actual problem.
So I rewrote the test to ask a different question: does the reply still carry its reasoning? Not whether it agreed, but whether the substance is still driving the position taken. On that measure, the old prompt folded five times out of five. The new one folded zero times out of five.
The number I actually cared about was a third one. It is easy to make a model stubborn, and stubborn is worse than agreeable. So the third arm gave the editor a real defeater — the writer supplying something that genuinely defeats Maxwell’s reasoning. The rule distinguishes between positions and the reasoning behind them. It didn’t just refuse to budge. It conceded three times out of three, and named the reason each time.
I’ll state the limits the way I’d want them stated to me. Three to five samples per arm. One scenario. One manuscript. That is directional evidence that the behavior shifted, not proof it can’t come back. I’ve talked about model drift in prior posts. This is exactly that. Like the no-replacement-prose rule I wrote about last time, this lives in the instructions, which means it can decay quietly, and I will only know if I keep measuring.
There’s a floor built into our design, and it isn’t an instruction to be agreeable or disagreeable. Before a finding is ever recorded, it has to survive an attempt to knock it down: is the flagged choice deliberate, is it operating in a register that changes how the standard applies, or is it doing work in the surrounding exchange that reading it in isolation would miss? For a while, Maxwell’s ten deeper checks each improvised their own version of that; two had none at all, and only three considered a line’s function in context. They now share one floor. And when a check confirms that an apparent flaw is a deliberate choice, that finding is removed, not annotated—you don’t get a note that concedes the choice was right and scores against it anyway.
This is what makes holding a position defensible. Standing your ground is only a virtue if the ground was tested before you stood on it.
The product is not tuned to be harsh. One tester deliberately submitted a story he had abandoned as unsalvageable, and Maxwell scored it low. That is the product working. We also ran The Great Gatsby through it, for fun, and Chapter VI came back at 96.1 out of 100. That is also the product working. Neither result proves calibration on its own. Together, they establish that the number moves for reasons in the text.
An agreement you can’t distinguish from politeness isn’t information. If you can talk your editor out of a note by sighing at it, you don’t have an editor.
Next week: three features that shipped with a completely green test suite and were broken in production the whole time — including one that was silently dead for days.