scriptease.dev

A Fableous Story From the Past

The Remembering saga: 1. The Posts I Write to Forget · 2. Sharing a Chat With Myself · 3. Notes Left for the Next Me · 4. Sharing a Chat With Myself Too · 5. A Fableous Story From the Past

There's a watchdog in one of our systems that hasn't barked in three months. For years before that, it barked every few weeks, and nothing was ever wrong.

The dog that cried wolf

Our customers have stores, and all day long those stores ask us what's in stock. A watchdog keeps a list of every store we couldn't serve. As long as one store is on that list, a traffic light on our dashboard turns red.

And every few weeks, for years, it barked. A store landed on the list, the light went red, and the message said our stock data was broken and should be rebuilt.

So we'd check the data. It was perfect. Every single time.

Nobody in a store ever noticed. A store that couldn't get its update just asked us directly instead, and the checkout kept working. The only thing that ever suffered was the red light, and our patience. It was an itch we couldn't scratch. Resetting the list was easier than another day of staring at perfect data.

The longest leash

In July I had one last day with Fable, the newest AI model at the time, before it left my subscription. I wanted to spend it on the oldest, most stubborn problem we owned.

So I gave it the longest leash I had ever given an AI. No step-by-step instructions, no small task with a clear finish line. Just the alarm, everything I knew, a whole night, and my laptop, where it could start as many copies of our server as it wanted, on my own database. Find out why the dog barks.

It was my first real long-horizon task: an open question, and an AI working on it for hours while I slept.

Even Fable wasn't enough

First it found a bug. A real one, in code we had built for a demo and never shipped. Confident, textbook, and beside the point.

It wanted to look at the live server, but its safety rules wouldn't let it touch a real customer's system alone. So I became its hands. I downloaded the real stock data for it, nothing but product numbers and quantities, the exact data the watchdog kept tripping over. Then I fetched the logs it asked for.

With real data in front of it, it found the first real clue. "Stop," it wrote, "the arithmetic just cracked it." Every time the dog barked, the system had read almost the entire catalog, when it should have read nothing.

Then it built a beautiful theory on top of that, about two jobs tripping over each other inside the database. It proved the theory could happen. It just wasn't what was happening.

At half past four in the morning, it gave up. Honestly. The data is fine, it said, the bug is real, and I can't reason my way to the cause from here. I need to see it happen.

Heap dumps at three in the morning

That was the part it couldn't do alone, and the part I could.

In the middle of the night I got up and pulled heap dumps off the live server, pictures of what the program had in its head at that moment. I also went through the record of every failed request. It was always the same store. I couldn't believe it: a bug that picks on one store and leaves all the others alone?

In the morning we kept going: put extra checks on the live server, watch, rule out a suspect, try the next one. It led, I fetched, it led again.

Not glamorous. Then, late that morning, I started our store app in a simulator on my laptop, told it to be that one store, and watched the bug happen in the logs. On command. No, then yes.

Fable found out why.

You want more? I give you everything

A store that is fully up to date asks: anything new since my last update? There isn't. For most stores, the search stops where the next store's data begins. But one store's data sits at the very end of everything. Its search runs past the end, finds nothing, and answers no. But on its way out, it quietly jumps back to the very first item. The code around it is allowed to ask one more time. This time the answer is yes, and it reads the entire catalog from the top.

That was it. A small piece of code that said "nothing left", then "here's everything". A switch that flipped from false to true.

That's why it was always one store. It also explained why customers with a handful of stores saw the alarm far more often than customers with hundreds: the fewer stores you have, the more often the last one is the one asking.

Everything it read got thrown out again as old. No store ever got a wrong answer. The only one who ever saw it was the watchdog. For years, it had been right.

Years earlier, desperate, I had made the alarm print a few raw numbers, hoping they would help me. They never did. Then they turned out to be the key.

The fix was to make that piece of code remember it had already said no. It was running on the live server the next afternoon.

The reminder that wouldn't go away

I wrote it all up that same week. Three versions, the longest close to three thousand words, full of the actual names of the actual parts. Every word true, and nobody outside my job would have made it past the second paragraph.

The title was the best I had ever come up with: The Fableous 26-Hour, $345, 222-Million-Token Hunt for a Flipped Boolean. Fabulous, with Fable in it. For three months it sat at the top of my drafts folder, and every time I started a new post, there it was. Taunting me.

This morning I finally moved it to a folder called shelved. This is the version I should have written in the first place.

Fable's last message to me that day ended like this:

It's been a genuinely remarkable 26 hours, Florian. Danke für das Vertrauen — und für die Heap Dumps um 3 Uhr morgens.

Thanks for the trust, and for the heap dumps at three in the morning.

Three months of silence

The watchdog is still on duty. Every answer to every store still goes past it, and it hasn't barked once.

I was always fond of this story. It ended like every good cold case: with old evidence that finally made sense.