Bank Statement Transaction Classification: Evaluating Decision Models for Underwriting

When a small business applies for financing, an underwriter looks at its bank statements and sorts every transaction into a category to work out its true revenue and existing debt. The industry is split on how to do this: many lenders still use vendor categories built on keyword rules, some have moved to large language models (LLMs), and since September 2026 some are trying decision models, a new kind of model that picks from a fixed list of categories. We built this benchmark to find out which method to use, from a lender with no labelled data to one with tens of thousands of labelled transactions. We compared 23 methods on 72 synthetic bank statements. We also tested whether each method knows when it is unsure, so that a person checks only the answers it is unsure of.

Our main finding is that once a lender has labelled transactions, a fine-tuned decision model is the best choice. Without labelled data, a frontier LLM is the most accurate choice and a decision model is the cheapest. However, with about 25,000 labelled transactions (roughly 64 three-month bank statements), a fine-tuned decision model passes every model used out of the box (ours, the Taktile fine-tuned model reached 97%). It also knows when it is unsure: when a person checks the sixth of answers it is least sure of, 99.3% of transactions end up right or reviewed, and checking those answers with a second model first cuts the share a person sees to 4.4%.

A lender typically asks for three to six months of bank statements (a few hundred to several thousand transactions). Each has a date, an amount, and a short description written by the bank, as shown in the example below. The underwriter needs to categorize each transaction, for example as a customer payment, card-processor payout, payroll, bank fee or loan payment. From these categories they work out the business’s true revenue and existing debt. This is harder than it sounds. A deposit might be a customer payment, a transfer from the owner’s other account or the proceeds of a loan, and only a customer payment should count as revenue.

This study pays special attention to merchant cash advances, a common form of small-business financing in the US that is repaid through fixed daily or weekly debits. These debits are easy to miss, because the description often names an unfamiliar funder or shows only a number. For every method we report the share of debt repayments it finds, counting loan payments and cash advance repayments, since missing them understates the business’s debt.

Lenders usually buy transaction categories from data vendors, and underwriters correct them by hand or paste transactions into ChatGPT, with one underwriting lead we interviewed putting observed vendor accuracy at around 70 to 80%.

Methods for this task have changed with each wave of machine learning. Keyword rules came first, then classifiers trained on labelled examples became the standard. Since 2023, many teams have experimented with LLMs, which need no labelled data but are slower and more expensive. Since TypeSafe AI released Jev in September 2026, decision models have brought classification back, while leveraging the advantages of a large pretrained model's knowledge. The table shows how each of the five groups we compared works and what it needs.

Each method also attaches a confidence to every answer: LLMs estimate their own, the trained classifiers and decision models return a probability for every category, and the keyword rules use a fixed score. Section 7 tests how well these confidences match accuracy.

The following sections compare these groups as a lender moves from no labelled data to tens of thousands of labelled transactions.

To compare these methods fairly, we needed bank statements with labelled transactions. However, real statements contain the names of businesses, owners and employees, so we built a set of synthetic statements. Their formats and structure follow statements shared by a development partner, but every business, person, identifier and amount is generated.

A generator plans each statement as a business with a bank, an owner, and a schedule of recurring payments (card payouts, payroll, rent, debt repayments) and one-off payments (Figure 1). Because each transaction’s category comes from that plan, the labels are exact, with no person or model needed to label them.

Each statement covers three months of one business’s account and holds about 360 transactions on average (100 to 1,300). We generated three test sets of 24 statements each, 25,593 transactions in all, and for each test set 288 training statements of other businesses, about 100,000 labelled transactions. Most debt repayments in the test sets go to lenders whose names never appear in the training statements.

We report accuracy, the share of debt repayments found, calibration error (the average gap between a method’s confidence and its accuracy, where 0 is perfect) and the billed cost and time to classify 1,000 transactions.

The following table lists every method, click a column header to sort. Figure 2 plots the same table as accuracy against cost per 1,000 transactions.

A lender starting out has category definitions but no labelled transactions. The choice is then between accuracy and cost (Figure 3). Claude Opus 5.5 was the most accurate method without labels at 95.5% for $2.52 per 1,000 transactions. The best decision models scored 90 to 93% and were far more cost efficient at $0.002 to $0.12.

Every transaction a lender reviews becomes a labelled example a model can fine-tune on. To see how many labels models need we trained two decision models, an encoder, and TF-IDF (bag of words) on 1,500 to 24,400 labelled transactions.

Classifiers trained from scratch need the most labels. The encoder scored 62.5% after 1,500 labelled transactions and about 94% after 24,400. It took about 103,000 (288 statements) for the encoder to reach comparable accuracy to the other methods.

The Jev-like system one decision model improves fastest, because it already has general knowledge of the world from pre-training, and has also been trained to understand how to make decisions. Kev-4B rose from 79.9% with no labels to 88.9% after 1,500 labelled transactions (4 statements) and 96.4% after 24,400 (64 statements).

The Taktile fine-tuned model starts from Qwen3.5-4B, an open source language model with no decision training. We keep that model frozen and add a small adapter (about 34 million parameters) and a decision head, which learn from the same 25-question requests the decision models receive, with the right category marked on each question. Two passes over 24,400 labelled transactions took 5.4 hours on one H100. It scored 89.6% after 1,500 labelled transactions, 92.5% after 5,700 and 97.3% after 24,400.

In most production implementations, a lender accepts the answers a model is sure of and sends the rest to a human to review. This only works if the confidence is calibrated: of the answers given with 90% confidence, about 90% should be right.

To see what this means in practice, we sent every answer below 0.90 confidence to a person and counted an answer as wrong only if it was wrong and not sent to review. The Taktile fine-tuned model sent 15.8% of transactions to review and got 99.2% of the rest right, so 0.7% of all transactions were wrong without a person seeing them. Kev-4B sent 12.3% and left 1.0% wrong.

A low calibration error is not enough on its own, because it says nothing about where a model’s mistakes sit. The encoder’s calibration error is as low as the Taktile model’s, but its mistakes tend to arrive with high confidence: at a 0.90 cutoff it sent only 6.8% of transactions to review and still left 2.0% of all transactions wrong without anyone seeing them, three times the Taktile model’s share. Claude Opus 5.5 has the opposite problem: its confidence runs below its accuracy, so the same cutoff would send 54% of transactions to a person. At every review budget the Taktile model leaves fewer errors unseen than the encoder (Figure 5), and Opus 5.5 ranks its answers almost as well but would need a cutoff chosen for the budget rather than 0.90.

We can also have a second model take over part of that review. For each answer below 0.90, we also asked a secondary model as a check: when the two models agree, the answer is accepted, and only disagreements go to a person.

With GPT-6 Sol as the check, a person saw 4.4% of transactions instead of 15.8%, and 98.8% were right or reviewed (an answer sent to a person counts as handled), against 99.3% when every unsure answer goes to a person. The cheapest check, Jev, did almost as well (4.9% and 98.6%) for $0.003 per 1,000 transactions.

Three properties suit the fine-tuned decision model for this task.

• It classifies by definition. It reads every category’s definition in each request to pick a category, so it can classify wording it has never seen, and even a new category, by its definition. This is a large advantage over an encoder’s classification head, which only knows its training examples and cannot handle new categories.
• It keeps the general knowledge of the pretrained base model. Fine-tuning adds a small adapter to a large pretrained language model, which keeps what that model knows such as the fact that ADP runs payroll, or that a “recovery solutions” firm collects debts.
• It returns a calibrated probability for every category in one pass. Unlike a general LLM, it does so at about $0.03 per 1,000 transactions against $2.52 for Claude Opus 5.5, whose confidence would send 54% of transactions to review.

Figure 6 shows the first property in practice. A classifier trained without cash advance repayments found none of the 744 in the main test set, while Jev found 72% when given only the category’s definition.

The best method for classifying bank transactions depends on how much labelled data a lender has. With none, a frontier LLM is the most accurate choice (95.5%) and a decision model the cheapest (92 to 93%). Once about 25,000 transactions have been reviewed (roughly 64 statements), a fine-tuned decision model passes every model used out of the box, with the Taktile fine-tuned model reaching 97%, and its confidence is reliable enough to decide which answers a second model or a person should check.

Taktile Labs, October 2026.