How to test customer questions before launch

Testing is reading, not clicking

The temptation is to type a few questions, nod at the answers, and publish. That tells you the assistant produces fluent text, which was never in doubt. What you need to know is whether each answer came from your content, and what it does when your content runs out.

Linkd's Playground is built for this. It sits beside the agent's configuration, answers exactly as the live assistant would, and shows which sources each answer drew on. Questions you ask there are recorded as Testing rather than Visitors, so a thorough test session won't distort your numbers once the assistant is live.

Build the list before you start

Write your questions down first. Testing from memory turns into testing the things you already know work.

Group one — the questions you get constantly. Hours, price, coverage, what's included, how to get started. Ask each one the way a customer would, not the way your website phrases it. Someone asks "do you come out to North York?", not "what is your service area?".

Group two — the second question. Real conversations don't stop at the first answer. If Fieldwork Studio, our example design practice, says it works with small businesses, the follow-up is "what does that cost?" or "how long does it take?". Test the follow-up.

Group three — the questions it should refuse. Anything needing a professional judgement, anything about a specific person's account, anything you'd never put in writing without checking. You are testing for a clean decline, not an answer.

Group four — the ones you'd phrase badly. Typos, one-word messages, two questions in a single sentence, a language other than English if your customers use one.

Thirty questions across those four groups is a realistic first list.

Read the sources, not just the answer

For each answer, open the sources it used. You are looking for three failure modes that a confident-sounding answer hides:

• Right answer, wrong source. It found a correct statement on an old landing page you meant to retire. It will keep doing that.
• Right answer, no source. The content didn't support it. The assistant filled a gap, and it will fill the next one differently.
• Stale source. The page it quoted is accurate as text and wrong as fact, because the price changed and the page didn't.

An answer that is correct for the wrong reason has not passed.

Check that "I don't know" actually happens

Deliberately ask about something you have not connected. Ask Cedarline, our example fencing contractor, about retaining wall repairs it doesn't do. The answer you want looks like this:

I couldn't find that in the business information available to me. You can contact the team and they'll confirm.

If instead you get a plausible paragraph about retaining walls, stop and look at your instructions and your sources before going any further. This is the single most important test in the list, because it is the failure your customers will not be able to detect.

Check the handoffs and the actions

If you have turned on actions, click them. Confirm each one opens the page you expected, and confirm the wording never claims the chat completed something. The assistant offers a booking page; your booking system confirms the booking. An answer that says "you're booked in for Thursday" is a bug, not a feature.

The same goes for lead capture. Check that it asks before collecting an email address rather than after.

Fix the content, not the answer

When a question fails, the fix is almost always upstream:

• The fact exists but nowhere readable → add a text snippet or connect the page.
• The fact exists in three contradictory places → remove or correct the wrong ones.
• The question is common and has one right answer → write a Q&A pair.
• It answered something it shouldn't have → remove the source, and tighten the instructions.

Rewriting the instructions to paper over a content gap works until someone phrases the question slightly differently.

Re-run the list after every change

Sources reindex when you add, update or remove them, and an answer that was right an hour ago can change once a new document competes with an old one. Keep the question list in a document, re-run it after a batch of changes, and re-run it before you publish.

After launch the list keeps earning its place: the Unanswered filter and the knowledge-gaps panel in analytics will hand you new questions to add to it, written by actual customers.

Next: how to measure whether your website assistant helps.