AI has attracted a great many firms whose product is a prompt, a text box, and someone else's model with a markup on top. They are not always easy to distinguish from serious engineering teams, because both use the same vocabulary. These five questions separate them reliably.
1. "Where will our data live, and will it train anyone's model?"
What you want to hear: a specific answer naming the provider, the tier, the region, and the retention period — plus a willingness to put "agency data will not be used for model training" in the contract.
Warning sign: "It's secure" or "we use enterprise-grade providers" without naming one. Vagueness here usually means they have not read the terms they are passing on to you, which means you are inheriting whatever those terms say.
2. "How will we know if it is working? Show me the evaluation set."
This is the highest-signal question on the list.
What you want to hear: that they will build a set of test questions with known-correct answers, drawn from your real material, and run it on every change. Ideally they ask you to help write it, because the domain knowledge is yours.
Warning sign: "You'll be able to tell" or a demo as the only evidence. A demo is a curated best case. Without a fixed test set, nobody can say whether next month's version is better or worse — including them.
3. "What happens when the model is deprecated?"
Providers retire model versions on real timelines. When that happens, behaviour changes and prompts that were tuned for the old version need revalidating.
What you want to hear: an acknowledgement that this will happen, a plan for revalidating against the evaluation set, and a clear statement of who pays.
Warning sign: surprise at the question. It means they have not run a system through a model transition, which means you will be the first.
4. "Who owns the prompts, the evaluation set, and the code?"
What you want to hear: you do, in writing, with a repository you can access and documentation you can hand to someone else.
Warning sign: "our proprietary framework." Sometimes legitimate, often a lock-in strategy. Ask the follow-up: if we part ways, what exactly can we take with us, and can another firm run it? If the honest answer is "nothing useful," price that risk in.
5. "What does it do when it does not know?"
What you want to hear: a description of refusal behaviour, confidence handling, citations, and which cases escalate to a human.
Warning sign: "it's very accurate." Every system is confident. The question is what happens at the edges, and a team that has shipped real systems will have thought about it in detail because the edges are where they got burned.
The tell across all five
A serious team will volunteer limitations. They will tell you which parts of your problem are hard, which are not worth automating, and where they expect trouble. A firm that says everything is straightforward is either inexperienced or selling you a demo.
Ask for a fixed-price discovery phase before any build commitment, and make the output yours regardless of what happens next. If the specification is good, you can take it to anyone. If a vendor resists that, you have learned something useful for the price of one phase.
For what the build itself should cost, see our breakdown of custom AI pricing. Our own application development engagements start with exactly that kind of discovery, and public-sector buyers should also read the responsible AI procurement guide.
Evaluating a proposal now? Send it over — a second opinion costs nothing.