Category
•
November 20, 2026

Same model, same files, four different answers

Author:
Mike Shum
Same model, same files, four different answers

In our first post, we gave ten frontier models the same loan tape and credit agreement and asked twenty questions about a live revolving facility. Then we took one of those models, Claude Fable 5, and ran the same twenty questions three more times. The prompts, files and scoring were the same every time, with a new conversation for each question.

We got four different scorecards:

Ask it which covenant has the least headroom and it names one covenant on two runs and a different one on the other two. Ask it to list the covenants at all, and you get three on one run and eight on another. Every run showed its working, every run was confident, and none was careless.

Running it again never got closer to the right answer. It just changed which wrong answer came back.

What never moved

Nine of the twenty questions came back the same in all four runs, and they had one thing in common.

  • The answer was written in the documents. Advance rates, eligibility criteria, maturity and the committed amount came back identical every time.
  • The input was genuinely missing. Where a question needed data no loan tape carries, such as the drawn balance, the model declined all four times.

Document questions were consistent, and so was genuine absence. Everything in between moved.

What moved: one decision, made differently each time

The model wasn't calculating differently each time. It was deciding differently.

The first question asks for the outstanding balance, and to answer it the model has to choose which part of the portfolio counts as the facility. Nothing in the files settles that. Two runs used the whole tape and two used only the facility's assets. Both balances came out about 15% too high.

That one unrecorded choice then decided the answers to the questions that followed, the same way every time:

Within each pair, the arithmetic matched to the basis point. What changed was which choice the model made at the start. A loan tape doesn't say which of several similar columns the facility runs on, or whether a defaulted asset counts as delinquent. The model picks one silently and calculates perfectly from there. If it picks differently next Tuesday, every number downstream moves with it.

The worst case was a cohort question. We asked at which month on book the average cohort first collects 100% of what it was funded. The four runs answered 3, 4, 6 and 10, and the facility's answer is 4. Run 1 got it right, but by a different method and a different set of cohorts that happened to land on the same month. Every run declared its method, and a reader shown any one of them would have no idea the other three existed.

Why running it twice won't catch it

The obvious control is to run the question again and compare. We did, four times. The answers that repeated exactly were among the most wrong.

Stability tells you the file was unambiguous. It doesn't tell you whether the answer is right.

Confidence didn't track any of it

The model's own confidence ratings gave no warning either. Delinquency by credit grade was rated High confidence in all four runs, and all four were 3–4x too high. The balance question was High confidence every time, with two different answers between the runs. The model never said a question had more than one defensible reading, even on questions where it had taken a different reading in a previous run.

The fix: decide once, and write it down

This is the same failure as in our ten-model test. There, one unrecorded decision gave five different loan counts across ten models. Here, the same decision gave two different answers inside one model.

A better model doesn't fix that, and neither does running a model more times. What fixes it is making the decision once: which part of the portfolio is the facility, which column it runs on, and whether a defaulted asset counts as delinquent. That decision gets written down, dated, and inherited by every number after it.

That's the layer our Analyst reads. By the time a question reaches it, there's nothing left to decide, so there's nothing left to get wrong.

Accuracy is not a property of the model. It is a property of what the model is reading.

The short version

We asked Claude Fable 5 the same twenty questions four times and got four different scorecards, including two different answers to which covenant was closest to breaching. It wasn't miscalculating. It was making an unwritten decision differently each time.

And if your plan is to run it twice and compare: we did. The answers that repeated perfectly were the wrong ones.

Category
8 min read

Same model, same files, four different answers

Same prompts, same files, four separate runs of Claude Fable 5. The model gave different answers each time, and running it again didn't reveal which were w
Written by
Mike Shum
Published on
November 20, 2026

In our first post, we gave ten frontier models the same loan tape and credit agreement and asked twenty questions about a live revolving facility. Then we took one of those models, Claude Fable 5, and ran the same twenty questions three more times. The prompts, files and scoring were the same every time, with a new conversation for each question.

We got four different scorecards:

Ask it which covenant has the least headroom and it names one covenant on two runs and a different one on the other two. Ask it to list the covenants at all, and you get three on one run and eight on another. Every run showed its working, every run was confident, and none was careless.

Running it again never got closer to the right answer. It just changed which wrong answer came back.

What never moved

Nine of the twenty questions came back the same in all four runs, and they had one thing in common.

  • The answer was written in the documents. Advance rates, eligibility criteria, maturity and the committed amount came back identical every time.
  • The input was genuinely missing. Where a question needed data no loan tape carries, such as the drawn balance, the model declined all four times.

Document questions were consistent, and so was genuine absence. Everything in between moved.

What moved: one decision, made differently each time

The model wasn't calculating differently each time. It was deciding differently.

The first question asks for the outstanding balance, and to answer it the model has to choose which part of the portfolio counts as the facility. Nothing in the files settles that. Two runs used the whole tape and two used only the facility's assets. Both balances came out about 15% too high.

That one unrecorded choice then decided the answers to the questions that followed, the same way every time:

Within each pair, the arithmetic matched to the basis point. What changed was which choice the model made at the start. A loan tape doesn't say which of several similar columns the facility runs on, or whether a defaulted asset counts as delinquent. The model picks one silently and calculates perfectly from there. If it picks differently next Tuesday, every number downstream moves with it.

The worst case was a cohort question. We asked at which month on book the average cohort first collects 100% of what it was funded. The four runs answered 3, 4, 6 and 10, and the facility's answer is 4. Run 1 got it right, but by a different method and a different set of cohorts that happened to land on the same month. Every run declared its method, and a reader shown any one of them would have no idea the other three existed.

Why running it twice won't catch it

The obvious control is to run the question again and compare. We did, four times. The answers that repeated exactly were among the most wrong.

Stability tells you the file was unambiguous. It doesn't tell you whether the answer is right.

Confidence didn't track any of it

The model's own confidence ratings gave no warning either. Delinquency by credit grade was rated High confidence in all four runs, and all four were 3–4x too high. The balance question was High confidence every time, with two different answers between the runs. The model never said a question had more than one defensible reading, even on questions where it had taken a different reading in a previous run.

The fix: decide once, and write it down

This is the same failure as in our ten-model test. There, one unrecorded decision gave five different loan counts across ten models. Here, the same decision gave two different answers inside one model.

A better model doesn't fix that, and neither does running a model more times. What fixes it is making the decision once: which part of the portfolio is the facility, which column it runs on, and whether a defaulted asset counts as delinquent. That decision gets written down, dated, and inherited by every number after it.

That's the layer our Analyst reads. By the time a question reaches it, there's nothing left to decide, so there's nothing left to get wrong.

Accuracy is not a property of the model. It is a property of what the model is reading.

The short version

We asked Claude Fable 5 the same twenty questions four times and got four different scorecards, including two different answers to which covenant was closest to breaching. It wasn't miscalculating. It was making an unwritten decision differently each time.

And if your plan is to run it twice and compare: we did. The answers that repeated perfectly were the wrong ones.

Stay Connected
No spam. Just the latest releases and tips, interesting articles, and exclusive interviews in your inbox every week.

Related Posts

Mike Shum

Ten models, four vendors, one blind spot: AI on a live credit facility

Twenty questions, ten models, four vendors, one live revolving facility. The models read the contract as well as we do, and sixteen of the twenty questions
November 18, 2026
Mike Shum

ABL Case Study: Tricolor Holdings

How roughly $800 million of nonexistent collateral moved through a subprime auto lender’s financing stack, and the data checks that would have exposed it.
November 3, 2026

Which Analytics Tools Support Due Diligence in Asset-Based Lending?

Due diligence in asset-based lending isn't one exercise. It's five or six of them running at once.
September 24, 2026