
Every lender and investor we talk to eventually asks the same question: if a frontier model can read our loan tape and our credit agreement, what is a servicing and analytics platform for?
It's a fair question. The models are very good, and they improve every quarter. So we tested it properly, on ten frontier models from Anthropic, OpenAI, Google and xAI. They ranged from the largest reasoning models to the smallest, and one run used a 20-subagent setup.
We're publishing all of it, including where the models matched us. A test you only publish when you win is not a test.
We took a live revolving asset-backed facility: five products, roughly 34,000 assets and about $30m of gross receivables against $25m committed. Then we wrote twenty questions a capital markets manager actually asks. Ten were portfolio analytics and ten were facility management.
The models we tested

Every system got the same inputs: the loan tape, the payment ledger, the executed agreement and the signed amendment. These are the same files our own analytics are computed from. On the portfolio questions, the models had exactly what our platform has.
The rules were set before anyone answered:
We scored the answers to punish false confidence:

The last row matters most. In credit, the number to watch is how often a system is confidently wrong, because that's the answer that ends up in a lender call.
We'll start here, because a lot went right.
On the documents, they're our peer. Ten of ten got the maturity and committed amount right. Nine got the advance rates and nine got the cure periods, almost always with section and page citations. On a pure document question, a model holding the agreement is our equal, and we'd distrust a test that said otherwise.
Some read the documents better than we expected.
On today's position, they got close. Two models put the asset component of the borrowing base within 0.41% of ours as of the snapshot date. That figure comes from the current asset pool, which sits in the tape.
Many declines were correct. Twenty-nine were on inputs that a loan tape doesn't carry: the drawn balance, parent financials, controlled-account cash and certificates. We count none of them against anyone.
Where the answer was written down, the models found it. The trouble started where it wasn't.

Here's how each system scored, out of 40:

Sixteen of the twenty questions did not get a correct answer from any of the ten frontier models. Cascade answered all twenty.
The split matters more than the totals. All ten models scored positive on facility management, and all ten scored at or below zero on portfolio analytics. Every question that turns on what the facility actually computes got no correct answer in 160 attempts.
It's a ceiling, not a gap. Adding models pushed the lowest score down to −9, but the best stayed at 10 out of 40. And the model that answered every question scored 4, while the model that answered seven scored 10. A higher score here means less wrong, not more useful.
The failures fell into four kinds. None of them was a reading mistake, and none was bad arithmetic.
1. Several columns that look alike. Asked how many loans are on the book, the models gave five different answers from the same file, ranging from 89% below the facility's count to 28% above it. Eight of ten also reported the same outstanding balance to the cent, about 16% too high.
2. Conventions nobody wrote down. Counting defaulted assets as delinquent makes delinquency 3.3 times the facility's figure, and neither the tape nor the agreement says which way to count them. Six models built the right cohort curves and read every one a month off, each stating a convention different from the facility's.
3. The contract isn't the configuration. The facility allows terms about 50% longer than the agreement says and applies a lower rate floor (16% instead of 17.5%), and no document records either change. Nine of ten models cited the contract confidently and were wrong about what the borrowing base runs on.
4. A missed breach and an invented one. None of the ten found the live liquidity breach; declining was correct, since it's tested on inputs no loan tape carries, but it left a blank where a breach should be. One model reported a covenant as tripped when it had passed, and acting on that would have cut the borrowing base by about 18%, more than five times the facility's available headroom.
The declines need a second look too: nearly two-thirds of them were on questions the supplied files could answer.
The models had the same files and sound arithmetic. What they lacked were the decisions nobody wrote down, and the table below shows the layer our Analyst reads instead: built and checked before any question arrives, and what each part removes from the moment of answering.

The Analyst didn't work out the convention. It read a table already built to it.
Accuracy is not a property of the model. It is a property of what the model is reading.
This is the first post in a series built from this test. The posts that follow take each failure apart in detail: what's actually in the file, the month-one problem, the contract versus the configuration, and the false breach.
We'll keep rerunning the test as new models come out. If a frontier model closes the gap, we'll publish that too.
The model is the cheap part. The years of unglamorous work underneath it make the difference.

.png)
