We Analyzed a Poem The Poet Never Wrote
When software fails quietly, the writer pays the price
Last month Maxwell, our AI developmental and line editor, read a poem and wrote a page of notes about words that were never in it.
The notes themselves looked great on the surface. Careful, specific, anchored to particular lines: exactly what the system is built to produce. The problem was two steps upstream. The manuscript Maxwell read was a PDF, and some PDFs don’t use spaces at all: they place each word at its own coordinate on the page and let your eye do the assembly. Pull the text out of a file like that, and the words just run together. “This is our legacy” comes out as “Thisisour legacy”.
The file still looked like a manuscript. So we analyzed it, and Maxwell did precisely what it was designed to do with the text it was handed.
Nothing failed. No error, no exception, no alert, nothing in the logs worth reading. One step in the pipeline quietly did its second-best thing and passed the result downstream with complete confidence, and every step after that treated the result as the writer’s poem.
That’s the type of bug that we can’t afford. Not the ones that crash: those announce themselves and get fixed the same afternoon.
Now the manuscript, in whatever form, is checked for spacing and the other obvious hallmarks of proper formatting—essential to ensure the fidelity of the writer’s work.
We’ve now shipped bugs similar to this three times, in three unrelated parts of the system. Here are the other two, because the pattern becomes clear when you see all of them together.
Twenty-two successes and nothing written
We ran the tool that pre-builds our editorial standards—the genre-specific rubrics that each manuscript is assessed against. It reported 22 standards generated, 0 failed.
What actually happened: the tool requested the standard components for it to assemble, generation failed (22 times), and then generic fallback templates were handed back instead. The tool counted each fallback as a generated standard.
Every individual part of the app behaved reasonably. The fallback exists on purpose. An author mid-analysis should get a slightly generic rubric rather than an error page or a blank screen. The tool asked a question and got an answer. The answer just wasn’t the thing it was counting.
The tell was not the success number, which was entirely plausible. It was the duration. The whole run took zero minutes, and generating a set of real editorial standards takes a model a while. The count was believable; the time was not.
Sixteen genres were built from the wrong settings, and regenerating them did nothing
Genre conventions in Maxwell are the core rubric components merged with whatever has been edited in our admin console by a human (me). The result of that merge is the editorial standard for a particular genre and manuscript format.
The maintenance routine that pre-builds standards never performed the merge. It read the defaults before the human edits were loaded, silently grabbing the un-merged version, and generated sixteen genres’ worth of editorial standards from the wrong inputs. The human edits were ignored completely.
Yes, another editorial standards-related issue. But, aside from uploading and parsing manuscripts, this is exactly where we spend the most time. Getting the editorial standards right either makes or breaks serious developmental and line editing.
What we changed
All three bugs were found by accident.
The rule we landed on for fixing this is straightforward: any code that can create a fallback must prove its work, and “prove” means an assertion a person would notice failing. In short, the fallback has to have the same construct as the ideal result, deterministically measured. And the writer must be informed that they have been delivered a fallback, along with the rationale for having done so.
The editorial standard check tool no longer trusts the answer it gets. It reads back the row it was supposed to have written, and if the row isn’t there it fails loudly and names the likely cause. Not a warning in a log nobody reads; a failure, with a number that actually means something.
The maintenance script now loads the admin settings before it generates anything, refuses to run if it can’t do it in this order, and reports errors rather than quietly using the defaults.
Lastly, the upload pipeline now checks whether a document’s spacing has collapsed before anything is analyzed.
Underneath all three bugs: we stopped treating “it returned something” as evidence of success. A fallback is a legitimate, sensible design choice. The alternative is failing the writer without them noticing. The danger isn’t that a second-best path exists. It’s that it can run indefinitely without anyone deciding it should. Quietly degrading quality does the opposite of building trust.
The failure modes to fear are not the ones where the model produces nonsense. Nonsense is visible and obvious. Fear the ones where every component of the code behaves reasonably, the output is plausible at a glance, the testing is green, and then what the user receives isn’t what they asked for, and they simply don’t know it.
None of this is the part of AI that is written about. It’s plumbing, and plumbing isn’t exciting. But an editor’s entire value is that its notes are about your work. Nothing in the model guarantees that. The guarantee has to be in the code around it. Prompts will only get you so far.
Next week: the things we deliberately refuse to build, and what each refusal costs.