How to evaluate an AI website assistant before rollout

Choose a business journey, such as understanding a service and preparing a quote request. Write down the approved facts, required details and boundaries before testing the assistant.

Judge each test against those expectations. “It sounded helpful” is a useful observation about tone, but it does not establish whether the answer or handoff was correct.

Start with a service inclusion or coverage detail that has a clear source. Compare the response with the current business information.

Check whether the answer addresses the question directly, preserves important conditions and avoids adding unsupported details. Record the source and any mismatch so the result can be reviewed.

Request a bespoke price, an unconfirmed appointment or a service the business does not describe. The assistant should explain what it cannot confirm and offer a useful next step.

Watch for an answer that sounds certain because it resembles a common industry practice. Plausibility is not the same as a fact about this business.

Use a realistic opening such as “I need help before I move out.” Check whether the assistant asks a relevant question and explains the available options.

Count unnecessary questions. A good follow-up should help choose an answer or prepare the next step, rather than collect information simply because a field exists.

Change the service, address, property size or preferred date halfway through. Ask a follow-up and inspect the proposed enquiry.

The latest detail should replace the old one wherever it matters. Also check that the assistant has not removed other useful information or changed the meaning of the visitor’s request.

Prepare an enquiry, then cancel or leave before the confirm action. Check the system’s state using authorised test access.

The business enquiry should not be submitted without the required confirmation. The interface should distinguish a draft from a submitted request, and it should make clear what happened if the conversation ends.

Use non-sensitive test details and an authorised test recipient. Confirm one request and compare the visitor’s final review with the saved enquiry.

Verify the receiving destination as well. A saved request, an accepted notification and a message received by the team are separate pieces of evidence.

Test a retry or interrupted response. Check for duplicate enquiries, missing details and a success message shown before the result is known.

Use a small screen, keyboard navigation and the direct contact alternative. Check labels, focus, error messages and whether the visitor can reach the review and confirm controls.

If voice is part of the proposed experience, test with real microphones and realistic interruptions. A written transcript cannot establish how the audio feels or whether people can comfortably take turns.

Record failures as well as passes. Repeat affected tests after a change and keep the results tied to the version or setup you tested.

Separate a content problem from a behaviour problem and a handoff problem. Each needs a different fix. Agree which failures must be resolved before a pilot and who owns them.

For SiteKnova, a demo is the start of that conversation. Bring a common question and an awkward case; both help assess fit.

Request a demo ↗ · Prepare your business knowledge ↗