How to run an AI pilot that is allowed to say no

Many AI pilots at lenders end the same way. Months pass, a demonstration goes well, a few people like the tool, and nobody can say whether it worked. There was no agreed test, so there is no clear result. The pilot neither becomes a programme nor gets shut down. It just fades, and the next vendor starts from scratch.

The problem is rarely the technology. It is that the pilot was never designed to produce a decision. A good pilot is one that can end in a clear yes or a clear no, and treats both as useful outcomes.

Pick one workflow, the one that costs the most hours

Start narrow. A pilot that tries to cover document checks, collections calls and policy queries at once will be judged on impressions, because no single part gets enough attention to be measured.

Choose the one workflow where staff time hurts most. In lending this is often document review for a particular product: bank statement analysis for self-employed borrowers, income checks for salaried applicants, or verification calls before disbursal. Define its start and end exactly. Which documents come in, what the output is, and who uses it next.

A narrow scope also makes the pilot easier to approve internally. IT, risk and compliance can assess one well-described process far faster than a vague platform.

Use redacted or sample files first

A pilot does not need production data or access to production systems to begin. Start with redacted files, or with sample documents that look like the real thing. This lets everyone see how the system behaves before any sensitive data or system access is on the table.

It also separates two questions that often get tangled. First, does the tool do the job well? Second, can it be deployed safely in your environment? Answer the first with redacted files. Move to the second only if the first answer is yes.

Agree the pass mark before you start

This is the step most often skipped, and the one that matters most. Before any file is processed, write down what success means and get the people who will decide to sign it.

The measures will vary by workflow, but they usually fall into four categories:

• Turnaround. How long a case takes from arrival to a usable output, compared with today.
• Catch rate. How many of the problems a careful reviewer would find, such as edited statements, mismatched names or missing pages, the system also flags.
• Rework. How often a reviewer has to correct the output, and how much effort each correction takes.
• Audit trail. Whether every output can be traced to its source and every decision to the person who made it.

Set the thresholds using your own current numbers, and agree how they will be measured: which files, who reviews them, and who counts. If the vendor sets the pass mark, or it is set after the results are in, the pilot cannot say no.

Bring credit, risk, IT and compliance in early

Pilots often start in one team and meet the others late. Operations finds the tool useful, then IT raises questions about where it runs, compliance asks about data handling, and risk asks how the model is governed. Weeks are lost answering questions that could have shaped the design.

Bring all four in at the start, each with a clear question to own:

• Credit: does the output support a sound decision, and who signs off?
• Risk: how are errors caught, and how is the model approved and monitored?
• IT: where does it run, what does it connect to, and who holds the keys?
• Compliance: how is personal data handled, and what record is kept?

RBI's FREE-AI committee report recommends a board-approved AI policy at regulated entities and recommends bringing AI-enabled products within the product approval process. A pilot that has already answered these questions is far easier to take through that process later.

Decide in advance what "stop" looks like

Write down the conditions under which the pilot ends with a no. Missing the agreed pass mark is one. Others might be a data-handling requirement the vendor cannot meet, or an audit trail that does not hold up to your internal auditors.

Also agree what happens after a no. The files are returned or deleted, access is removed, and the findings are written up so the next evaluation starts better informed. A pilot that ends cleanly is a good result. It saves the money and attention a slow fade would have consumed.

We approach pilots the same way at Inferova: one workflow, redacted or sample files, a pass mark agreed first, and then a pilot or a clear no. If it does not earn its place, we say so and stop.

Questions to ask a vendor about where data goes

Before you share even redacted files, ask:

• Where will the files be processed: on our premises, in our cloud account, or in yours?
• Do documents or extracted text go to any outside AI service? Which one, and can we switch it off?
• Is any of our data used to train models, and whose models?
• How long is data kept, and how is deletion confirmed?
• Who on your side can see our files?
• What audit record do we receive of every action taken on our data?

These questions sit alongside your obligations under the Digital Personal Data Protection Act, 2023 and your own outsourcing and information security policies. Clear, written answers are a good sign. Vague ones are a reason to pause.

If you would like to run a pilot on these terms, you can request beta access at inferova.com/demo.

Sources