What questions should we ask an automation vendor?

The eight that matter are what happens to the cases the system gets wrong, what accuracy means in their proposal and how it is measured, what the smallest useful first delivery is, whether they will run on your own worst documents before contract, who owns the code and data, what the ongoing cost is once the build is done, who maintains it when a supplier changes their invoice layout, and what it takes to leave.

Why do so many automation projects fail?

Rarely because the technology could not do it. The common causes are a scope that was never narrow enough to finish, no named owner on the client side, no plan for the exceptions the system cannot handle, and success criteria that were never defined in a measurable way. Ernst and Young's often-quoted estimate puts the failure rate for initial RPA projects at 30% to 50%, and Deloitte found that only 3% of organisations had scaled automation substantially.

Should we choose a software product or a consultancy?

It depends on how standard your documents and processes are. If your paperwork is clean and templated, a product is cheaper and faster. If it comes from dozens of suppliers in inconsistent formats and needs checking against your own systems, a product will get you most of the way and then stop, and the remaining distance is where the value was. Ask any product vendor to run your ten worst files before deciding.

How long should a first automation project take?

Weeks, not quarters. A first delivery that cannot be scoped to something shippable inside about two months is usually a sign the problem has not been narrowed enough. Scope one process, one document type or one reconciliation, ship it, measure it, then decide what comes next based on what you learned rather than what was in the original plan.

How to choose an automation vendor

Choose an automation vendor by asking what happens to the cases the system gets wrong. It is one question, it takes ten seconds, and it separates the people who have shipped automation into production from the people who have demoed it. Everything else in this post is a way of asking the same question more thoroughly.

The reason to lead with failure rather than capability is that capability is no longer the constraint. Extraction, matching and classification all work well enough now that the technology is rarely what sinks a project.

Why do automation projects actually fail?

Not on the technology. Ernst and Young’s widely quoted estimate puts the failure rate of initial RPA projects somewhere between 30% and 50%, and Deloitte’s survey of 478 organisations found only 3% had reached substantial scale. Those numbers get repeated a lot, so treat them as directional rather than precise. The failure modes behind them are consistent and worth more than the percentages:

Scope that was never narrow enough to finish. A programme to “automate accounts payable” has no completion condition. A project to match supplier invoices to purchase orders for one entity does.
No owner on the client side. Automation encodes decisions about what “correct” means, and those decisions need someone in your business who can make them in an afternoon rather than a steering committee.
No plan for the exceptions. Every real process has cases the system cannot handle. If nobody designed the path those take, they silently become someone’s unofficial second job.
Success criteria that were never measurable. “Improve efficiency” cannot be checked. “Reduce reconciliation from five days to under one, at 97% line accuracy” can.

Notice that three of the four are scoping and ownership problems, which are decided before any code is written, and mostly during vendor selection.

The eight questions

Ask these in the first meeting. The answers are more informative than any case study.

1. What happens to the cases it gets wrong?

The answer you want is specific: how confidence is scored, what threshold routes a case to a human, what the review interface looks like, and what the expected exception rate is. A vendor who answers with model accuracy alone has not run one of these in production, where the entire operational question is which cases a person sees.

2. What does your accuracy number mean?

Accuracy is not one thing. Character accuracy, field accuracy and end-to-end accuracy on a whole document differ enormously on the same system. A field accuracy of 95% sounds excellent until you notice that a document with twelve fields then has a better than even chance of containing an error. Make them define the denominator.

3. What is the smallest useful thing you could deliver?

The best answer is small, specific and unglamorous. A vendor who proposes a discovery phase, a platform selection and a roadmap before touching a document is selling you a plan. Plans are not the hard part.

4. Will you run it on our worst documents before we sign?

The demo file always works. Take your ten worst inputs, the smudged scan, the handwritten correction, the supplier who changed their layout last quarter, and ask for a run on those. This single test resolves more uncertainty than the rest of the process combined, and a vendor’s willingness to attempt it is itself the signal.

5. Who owns the code, the models and the data?

Get it in writing. Whether the extracted data can be used to train anything shared, whether you receive the source, and where the data is processed. In Hong Kong this also means asking where documents leave the jurisdiction, which is a question with a factual answer and should not require a follow-up call.

6. What does it cost once the build is finished?

Build cost is the visible number and rarely the largest one. Ask for the running cost at your volume: licences, per-document charges, hosting, and the support arrangement. A project that pays back in eight months at build cost and never pays back once running costs are included is a common and avoidable outcome.

7. Who fixes it when a supplier changes their invoice layout?

This happens constantly and it is the single most reliable predictor of whether an automation is still working two years later. You want a named arrangement: how changes are detected, how quickly they are handled, and whether it is included or billed.

8. What does it take to leave?

Ask it plainly. If the answer involves a re-implementation from scratch, price that risk into the decision now rather than discovering it during a renewal negotiation.

What a good proposal looks like

It names one process. It states a measurable before and after. It describes what happens to the exceptions in as much detail as it describes the happy path. It has a first delivery inside about two months, and it says what will be learned by then that might change the rest.

What it does not do is promise to remove people entirely. Chasing full automation on messy operational documents is how these budgets disappear, and the honest target is a ratio rather than a total: automate the cases the system is confident about, route the rest to a person. We wrote up what that took on a real air cargo reconciliation in what 97% reconciliation accuracy really takes, where the outcome was a five-day cycle running in about two hours with one person reviewing exceptions.

Do the arithmetic before the meeting

Walk into vendor conversations with your own numbers already worked out: how many documents a month, how many minutes each one takes, what an hour of that person’s time costs you fully loaded. It changes the conversation from a pitch into a comparison, and it tells you the size of prize before anyone quotes a price. The cost side of that calculation is set out in what it costs to automate a process, and our automation ROI calculator will do the arithmetic in a minute.

If you would rather work through it with someone who will tell you when the answer is “do not automate this yet”, that is what our free 30-minute ROI diagnostic is for.