In our first post, we gave ten frontier models the same loan tape and credit agreement and asked twenty questions about a live revolving facility. Then we took one of those models, Claude Fable 5, and ran the same twenty questions three more times. The prompts, files and scoring were the same every time, with a new conversation for each question.
We got four different scorecards:

Ask it which covenant has the least headroom and it names one covenant on two runs and a different one on the other two. Ask it to list the covenants at all, and you get three on one run and eight on another. Every run showed its working, every run was confident, and none was careless.
Running it again never got closer to the right answer. It just changed which wrong answer came back.
Nine of the twenty questions came back the same in all four runs, and they had one thing in common.
Document questions were consistent, and so was genuine absence. Everything in between moved.
The model wasn't calculating differently each time. It was deciding differently.
The first question asks for the outstanding balance, and to answer it the model has to choose which part of the portfolio counts as the facility. Nothing in the files settles that. Two runs used the whole tape and two used only the facility's assets. Both balances came out about 15% too high.
That one unrecorded choice then decided the answers to the questions that followed, the same way every time:

Within each pair, the arithmetic matched to the basis point. What changed was which choice the model made at the start. A loan tape doesn't say which of several similar columns the facility runs on, or whether a defaulted asset counts as delinquent. The model picks one silently and calculates perfectly from there. If it picks differently next Tuesday, every number downstream moves with it.
The worst case was a cohort question. We asked at which month on book the average cohort first collects 100% of what it was funded. The four runs answered 3, 4, 6 and 10, and the facility's answer is 4. Run 1 got it right, but by a different method and a different set of cohorts that happened to land on the same month. Every run declared its method, and a reader shown any one of them would have no idea the other three existed.
The obvious control is to run the question again and compare. We did, four times. The answers that repeated exactly were among the most wrong.

Stability tells you the file was unambiguous. It doesn't tell you whether the answer is right.
The model's own confidence ratings gave no warning either. Delinquency by credit grade was rated High confidence in all four runs, and all four were 3–4x too high. The balance question was High confidence every time, with two different answers between the runs. The model never said a question had more than one defensible reading, even on questions where it had taken a different reading in a previous run.
This is the same failure as in our ten-model test. There, one unrecorded decision gave five different loan counts across ten models. Here, the same decision gave two different answers inside one model.
A better model doesn't fix that, and neither does running a model more times. What fixes it is making the decision once: which part of the portfolio is the facility, which column it runs on, and whether a defaulted asset counts as delinquent. That decision gets written down, dated, and inherited by every number after it.
That's the layer our Analyst reads. By the time a question reaches it, there's nothing left to decide, so there's nothing left to get wrong.
Accuracy is not a property of the model. It is a property of what the model is reading.
We asked Claude Fable 5 the same twenty questions four times and got four different scorecards, including two different answers to which covenant was closest to breaching. It wasn't miscalculating. It was making an unwritten decision differently each time.
And if your plan is to run it twice and compare: we did. The answers that repeated perfectly were the wrong ones.

.png)
