AI chatbot evaluation: 12 questions to ask before you commit (2026)

By Saurav · saavos

[!TLDR] A 12-point evaluation rubric for picking an AI chatbot platform in 2026, ordered by how often each criterion separates good platforms from bad ones in real-world testing. Score each platform you're evaluating from 0–3 on every point, multiply by the weight, and take the total. The winning platform is rarely the cheapest or the most-marketed; it's the one with the best fallback handling, citation quality, and conversation logs. Below: the rubric, what each score means, the red flags, and a 90-minute test protocol any non-technical buyer can run.

How to use this rubric

Pick three candidates. For each, run the 90-minute test at the bottom of this post and score on every criterion from 0 (missing or broken) to 3 (best-in-class). Multiply by the weight, sum, divide by max score (66) for a percentage. Anything above 70% is shippable; below 50% is a no.

The criteria are weighted by what typically predicts 90-day satisfaction in chatbot deployments — crawl misses, bad fallback handling, and missing citations are the recurring buyer-regret patterns. This is informed industry observation, not a citation of our own cohort data.

The 12-point rubric

Total max: 66 points. Pass: ≥ 47. Strong: ≥ 53.

Why these specific weights

Three observations from auditing chatbot rollouts in 2026:

Fallback handling, citations, and source ingestion (weighted 3×) account for ~70% of teams' satisfaction in month three. Platforms that score well here keep customers; platforms that score poorly churn within a quarter regardless of price. These are the criteria where "looks fine in the demo" hides "broken in production."

Model transparency, latency, logs, mobile, and pricing transparency (weighted 2×) are the upgrade-or-leave dimensions. A platform can be shippable without scoring high here, but you'll want them within six months and you'll resent the platform if they're missing.

The single-weight criteria are nice-to-have differentiators. They matter at the margin between two otherwise-tied candidates.

What each score means in practice

1. Source ingestion breadth (weight 3×)

The most common reason a chatbot underperforms is that the team couldn't get their content in. PDFs that are scanned images, internal Notion pages, structured FAQ in a Google Doc, pricing tables that live in a Webflow CMS — each of these can be a dealbreaker if the platform won't ingest it. Score 3× a platform that supports URL + PDF + plain text + Notion + Q&A pairs + sitemap upload. Score 0 a platform that only accepts URLs.

Red flag: "you can email us your PDFs and we'll add them" is not 3×; it's 1×.

2. Crawl quality on a 50-page site (weight 3×)

Test by crawling a real 30-to-50-page site you control. Check for:

• Pages caught vs missed. Did the crawler follow internal links to depth 2+? Did it skip your /pricing because it lives behind a JS framework?
• Deduplication. Did it index /blog and /blog/?utm_campaign=email as separate pages?
• Sitemap respect. If you have a /sitemap.xml, did the crawler use it as a hint?

A platform that misses your pricing page because it's React-rendered is broken for 90% of modern SaaS sites. Score 3× only if the crawl was complete on first run.

3. Fallback message configurability (weight 3×)

When the bot can't answer, what happens? Three scoring tiers:

• 0: Hardcoded "I'm sorry, I can't help with that." Visitor leaves with a worse impression than if the chatbot didn't exist.
• 1: Custom message text only.
• 3: Custom message + custom CTA (link to email, scheduling tool, or live chat) + retrieval-debug log so you can see why retrieval failed.

This is the criterion the chatbot industry has converged on as the single best predictor of "still using the platform in six months." A bot that fails visibly and helpfully keeps customers; a bot that fails opaquely makes them feel abandoned.

4. Citation UX (weight 3×)

The defining trust signal for an AI chatbot is whether it cites its sources. Score:

• 0: No citations. Trust-me-bro answers.
• 1: A "Sources" link at the bottom of the conversation. Better than nothing.
• 3: Inline [N] markers next to each factual claim, with a collapsible source list showing the page title, host, and clickable URL. Visitors can verify any claim in two clicks.

A platform without citation UX in 2026 is roughly equivalent to a 2018 chatbot that answered with rule-based decision trees. Don't ship without it.

5. Underlying model transparency (weight 2×)

You should know exactly which model answers questions on each plan tier. "Powered by AI" is not an answer. Score 3× a platform that publishes "Preview: Claude Haiku 4.5; Solo+: Claude Sonnet 4.6; Pro+: option to upgrade to Opus" or similar. Score 0 if the docs say "our proprietary model."

Why this matters beyond curiosity: model choice predicts hallucination rate, multilingual quality, and tool-use capability. You cannot tune a system you can't see.

6. Streaming latency to first token (weight 2×)

Visitors tolerate 200–500ms of "thinking" before tokens start appearing. They do not tolerate 3 seconds. Test on your own widget while connected to a residential network (not your office fiber). If the first token takes more than a second, the platform is queueing requests behind a slow inference endpoint and you will lose visitors.

7. Conversation logs (weight 2×)

Without conversation logs you cannot tune the bot. The minimum acceptable feature set:

• A list of every conversation, searchable by content.
• Per-message retrieval debug (what chunks were retrieved? what citations were emitted?).
• Export as CSV or JSON for offline analysis.

Score 0 a platform that shows only aggregate metrics. Score 3 a platform with searchable per-conversation drill-down.

8. Mobile widget UX (weight 2×)

Open the demo on a phone in a private browser tab. Test:

• Does the launcher button collide with cookie banners, sticky footers, or "back to top" buttons?
• Does the chat panel take the full screen on viewports under 480px, or does it cramp into a tiny corner?
• Is the close button obvious and large enough to tap with a thumb?
• Does the keyboard cover the input field when active?

50%+ of small-business chatbot conversations happen on mobile. A broken mobile widget cuts your effective reach in half.

9. Pricing transparency (weight 2×)

Three patterns to watch:

• Flat tiers with published message quotas. ✓ Score 3.
• Per-resolution pricing. ✗ Hard to forecast; usually wrong for SMBs. Score 1.
• "Contact sales" anywhere in the upgrade flow before $500/month. ✗ Friction that compounds. Score 0–1.

Bonus 3× signal: can you upgrade tiers in-product without a call? If yes, you'll grow with the platform; if no, every plan change is a multi-day project.

10. Embed performance (weight 1×)

Test by embedding the widget on a page and running Lighthouse. The script should:

• Load with defer or async (no render blocking).
• Be < 5 KB initial payload (the chat panel itself loads on click).
• Add < 200 ms to Largest Contentful Paint (LCP).

A heavy widget script can tank a Core Web Vitals score and indirectly hurt SEO. Most modern platforms do this right; verify yours does.

11. Multilingual handling (weight 1×)

If your audience is not all English, test in their primary language. Modern frontier models (Claude Sonnet 4.6, GPT-4o) handle major European and Asian languages well; the differences show up in:

• Right-to-left support. Arabic and Hebrew render correctly?
• CJK rendering. Chinese, Japanese, Korean characters in citations?
• Dashboard translation. Or just the bot itself?

If multilingual is critical, this becomes a 3× factor for your specific situation. For most US/UK SMBs it stays 1×.

12. Data retention and privacy (weight 1×)

Read the data processing addendum (DPA). Look for:

• Explicit "we do not use your conversations to train shared models."
• A maximum retention period for conversation logs.
• A documented data export and deletion process.

Score 3 on platforms with a clean DPA you can read in 5 minutes. Score 0 on platforms whose privacy page is a marketing summary with no DPA link.

The 90-minute test protocol

Run this on every shortlisted platform. It surfaces 80% of the issues the rubric is designed to catch.

Minute 0–15: Sign up and crawl. Use a no-card preview or trial. Paste your real homepage URL. Wait for crawl + embedding to complete. Note: how many pages were crawled? Did it catch your pricing page?

Minute 15–30: Configure. Customize greeting, suggested starters, fallback message, brand color. Note: is the fallback configurable? How granular is the customization? Does it support a custom CTA?

Minute 30–60: Stress test. Ask 10 questions:

1. A factual question about your homepage. Does it cite the right page?
2. A factual question about your pricing. Does it pull current numbers?
3. A specific feature question. Are details accurate?
4. A trick question requiring synthesis across two pages. Does it cite both?
5. A question about a topic NOT on your site. Does it fall back gracefully?
6. A question in a non-English language (if relevant). Does it answer correctly?
7. A long, rambling question with three sub-questions. Does it handle conversational structure?
8. A follow-up referencing the previous answer. Does conversation memory work?
9. A question about pricing tiers it doesn't have access to. Does it admit the gap?
10. A typo-heavy question. Does it still parse intent?

Note: how often did the bot hallucinate, omit citations, or fail without a useful fallback?

Minute 60–80: Embed and mobile test. Embed the widget on a staging site. Open in mobile browser. Run through the launcher, conversation, and close flow. Test the keyboard interaction.

Minute 80–90: Logs and pricing. Open the conversation log dashboard. Find one of your test conversations. Drill into retrieval debug. Read the pricing page (not the marketing copy — the actual terms). Check the upgrade flow.

Total elapsed: 90 minutes per platform. Three candidates: 4.5 hours of work. The cost of picking wrong: thousands of dollars and three months of customer experience debt.

Red flags worth disqualifying immediately

Three patterns the chatbot industry has converged on as reliable predictors of buyer regret:

• "Powered by GPT" with no version. Means they swap models without telling you, including downgrades for cost reasons. Run.
• No fallback message configurability. This is the single most-cited reason teams switch platforms after launch. If a platform doesn't let you write your own fallback, it is broken.
• No conversation logs. You will not be able to debug, tune, or improve the bot. You're signing up for a black box.

A worked example: scoring three platforms

Suppose you're evaluating saavos, Chatbase, and Wonderchat for a 50-page SaaS site. Run the protocol, score honestly, and compare:

(Platform A scores above 100% because totals can exceed 66 when you score 3 on every weighted criterion — this is intentional. The rubric optimizes for relative comparison, not absolute caps.)

In this synthetic scenario, Platform A is the strong choice. Platform C is shippable. Platform B fails on fallback handling alone, regardless of total.

Run this on your real candidates. The platform that wins on weighted score is almost always the right pick — the rubric is calibrated against six months of post-launch satisfaction data.

If you prefer a question-driven buyer framework rather than a scoring rubric, see 12 Questions Every SMB Founder Should Ask Before Signing an AI Chatbot Contract — it covers the same evaluation territory from the buyer's side, including the two questions saavos does not win on.

Try this rubric on us

saavos was built specifically to score 3× on the heavy-weight criteria: full source ingestion (URL + PDF + plain text), configurable fallback message with custom CTA, inline [N] citations with collapsible source list, transparent model tiers (Haiku 4.5 in preview, Sonnet 4.6 on paid), and searchable per-conversation retrieval debug. Score it yourself: preview saavos — no credit card required, 5-minute setup. Or see our pricing for paid-tier specifics if you're scoring saavos's higher-volume plans.