The AI services market is flooded. Every web development agency has rebranded as an "AI company." Every consultant now offers "AI strategy." Finding a partner who can actually ship production AI — not demos, not prototypes, but software your customers pay for — is genuinely hard. This guide gives you the framework to tell the difference.
The 10 Questions You Must Ask
1. Can you show me production AI systems you've built — not demos?
This is the filter. Any agency can build a ChatGPT wrapper that demos beautifully. Ask to see systems that are in production, serving real users, handling real data volumes. Ask about the architecture, the failure modes they encountered, and how they solved them. If they can't point to production deployments, they're a demo shop.
2. How do you evaluate your AI systems?
Production AI without evaluation is gambling. The right answer involves RAGAS metrics for RAG systems, custom evaluation suites for agents, A/B testing frameworks, and regression testing pipelines. If the answer is "we test it manually before delivery," walk away.
3. Who will actually work on my project — and what are their credentials?
Many agencies sell senior talent and deliver junior talent. Ask for the specific engineers who will be assigned to your project. Ask about their backgrounds. Have they worked on ML systems professionally, or did they complete a bootcamp six months ago? The AI skills gap is real — most agencies have one capable AI engineer and fill the rest with web developers.
4. How do you handle hallucination and accuracy?
Any serious AI engineer has a clear, technical answer to this. For RAG: grounding strategies, citation enforcement, faithfulness evaluation. For agents: tool verification, output validation, confidence thresholds. If the answer is "the model is usually accurate," they haven't shipped production AI.
5. What is your observability and monitoring setup?
Production AI systems break in ways that are fundamentally different from traditional software. You need LLM tracing (LangSmith, Langfuse, or Helicone), cost monitoring, latency tracking, and alerting on evaluation metric degradation. Ask how they'll make sure you know when something goes wrong.
6. Do I own the IP and code completely?
Non-negotiable: all code, models, data pipelines, and intellectual property must be transferred to you upon delivery. Some agencies retain ownership or build on proprietary platforms that create lock-in. Get this in writing before you start.
7. How do you scope and price work?
Good AI agencies price based on a detailed technical scope, not vague estimates. They should be able to tell you: what they're building, what they're not building, the key technical risks, and what 'done' means. Beware of time-and-materials contracts without a fixed scope — they incentivise slow delivery.
8. What happens if the AI doesn't meet quality targets?
Ask specifically what happens if the RAG system has a faithfulness score below target, or if the agent fails to complete tasks reliably. A professional agency has clear quality benchmarks defined upfront and a plan for hitting them. "We'll keep iterating" without a defined target is not a plan.
9. How do you handle data security and privacy?
If your project involves proprietary data — customer information, internal documents, financial records — your AI partner needs to understand data governance. Ask about data handling policies, whether your data is used to train third-party models, encryption at rest and in transit, and whether they can support on-premise or private cloud deployment if required.
10. What does post-delivery support look like?
AI systems degrade. Model APIs change. Embedding model updates break retrieval. You need a partner who can support you after launch — whether through a retainer, defined SLA, or clear handoff documentation. Ask specifically what happens in the first 90 days after delivery.
Red Flags to Watch For
- They can't explain their work technically — if they can't articulate why they chose Qdrant over Pinecone, or what re-ranking is and when to use it, they're not real AI engineers
- Their portfolio is all demos and mockups with no production deployments
- They propose fine-tuning immediately without first asking whether RAG would solve the problem — fine-tuning is oversold as a solution to knowledge problems that RAG handles better and cheaper
- They don't ask about your evaluation criteria — building without a definition of success
- They offer to build "anything AI" with no specialisation — generalists in AI are often mediocre at everything
- They propose using OpenAI for everything without considering cost, latency, or data privacy implications of your use case
- Their team of "AI experts" is the same web dev team with new LinkedIn titles
What Good Looks Like: Nxtr's Approach
At Nxtr, every project starts with a technical scoping session where we define the architecture, evaluation criteria, and success metrics before writing a line of code. We assign engineers with direct AI production experience, not web developers who took an LLM course. We build observability and evaluation from day one. You own all IP, code, and data pipelines completely. And we back our quality targets with clear success criteria in the contract.