Split Testing Live Chat: What to Test and How Long

Split testing live chat means changing one thing about the chat, such as the greeting or the pre-chat form, and measuring whether more visitors start a conversation or finish it satisfied. It sounds simple, but most small sites run these tests badly: they change three settings at once, judge the result after two days and declare a winner on twenty chats. The numbers look like evidence and are really noise.

This guide explains what is worth testing in a chat widget, which settings in Talkmio you can change, how to run an honest test without a testing platform, how much traffic you need before a difference means anything and how to read the results from Reports. It is written for small teams, so it favours big changes, clear rules and patience over statistics software.

Why Split Testing Live Chat Is Different

A classic A/B test on a landing page shows two versions to two groups of visitors at the same time and counts conversions. Chat adds complications. The outcome you care about is not a click but a conversation, which depends on whether someone is online, what the visitor wanted and how the first answer felt. Traffic is usually smaller than on a landing page, and chat volume is a fraction of traffic. The same widget also serves returning customers, who behave differently from first-time visitors.

That does not make testing pointless. Chat settings have a large effect on whether people write, and they are cheap to change. It means you should test fewer things, for longer, with a clear question. The general idea of controlled comparison is described in the overview of A/B testing; the rest of this guide adapts it to a chat widget.

One honest limit first: the Talkmio documentation lists no built-in split-test tool that shows two widget versions to two groups at once. Widget settings apply to a website and take effect within five minutes, so the practical method for most teams is a sequential test: run version A for a fixed period, switch to version B for the same period and compare. The method has weaknesses, covered below, and a few ways to reduce them.

What Is Worth Testing in a Chat Widget

Not every setting deserves a test. Rank ideas by how much they could change behaviour and how easy they are to reverse.

Element	Where in Talkmio	Likely effect	Effort
Welcome text	Websites, your site, welcome text	High: sets whether visitors feel invited	Minutes
Pre-chat form: none, optional or required	Websites, pre-chat form	High: each field costs conversations	Minutes
Widget position	Websites, position	Medium: avoids overlap with buttons	Minutes
Colour	Websites, colour	Low to medium: visibility against the page	Minutes
Assistant name and tone	Websites, assistant name; Mio business instructions	Medium: affects trust and reply length	Minutes
Offline text and business hours	Websites, business hours	Medium: decides whether offline visitors leave an e-mail	Minutes
Proactive opening of the chat	Visitors, start a chat	High for pages with intent, but manual	Ongoing

If you only run one test, make it the welcome text or the pre-chat form. Our guides on live chat greeting messages and pre-chat form length give you candidate versions to try, and the piece on widget placement and proactive messages covers position.

Pick One Question and One Metric

Every test needs a sentence of the form: “If we change X, more Y will happen.” Write it down before you change anything.

Greeting test: “A greeting that names one concrete thing Mio can help with will raise the share of visitors who open the chat.”
Form test: “Making the pre-chat form optional will raise conversations started without hurting lead quality.”
Offline test: “A clear offline text with a promise of reply time will raise the share of offline visitors who leave an e-mail.”

Choose a primary metric

Reports in Talkmio show conversations per day, the share answered by Mio alone, hand-offs, first-reply time, ratings, busiest hours, channels, countries and team performance. For a widget test the primary metric is almost always conversations started per day or per hundred visitors. Visitor numbers come from your website analytics, because Talkmio’s Reports count conversations, not page views. The article on live chat engagement rate defines the ratio and shows how to calculate it.

Choose guardrail metrics

A change that raises conversations can lower quality. Watch two guardrails: ratings, and the share of conversations Mio answers alone versus hands over. If a shorter form doubles chats but half are empty or spam, you have not won anything. The overview in live chat reports and analytics explains which numbers to keep an eye on.

How Long to Run a Split Test on Live Chat

Two rules cover most cases: run each version for whole weeks, and run it long enough to collect a sensible number of conversations.

Whole weeks

Support and shopping have weekly rhythms. Monday looks different from Saturday, and paydays, newsletters and campaigns move traffic. Comparing Monday to Wednesday of two different versions mixes the effect of the change with the effect of the weekday. Run each version for at least one full week, ideally two, and compare like with like.

Enough visitors

How many you need depends on your starting rate and the size of the change you want to detect. The table shows approximate visitors per version for a standard comparison at 95 percent confidence and 80 percent power. The figures come from the usual two-proportion formula described in the article on sample size determination.

Starting rate	Rate you hope to reach	Visitors needed per version (approx.)
3%	4%	5,300
10%	12%	3,800
20%	25%	1,100

The lesson is that small lifts need a lot of traffic. A site with 2,000 visitors a week cannot reliably detect a one-point change in a three percent rate in a month. It can detect a large change, which is why small sites should test bold differences: a very different greeting, no form versus a required form, not two near-identical sentences.

A Simple Way to Run the Test in Talkmio

Record the baseline. Note the last four weeks of conversations per week from Reports and the matching visitor counts from your analytics. This is your normal range, and it shows how much the number moves on its own.
Write the hypothesis and metric. One sentence, one primary metric, two guardrails.
Change one setting. Edit it under Websites, wait five minutes for it to apply, and write down the exact time and the exact text.
Do not touch anything else. No new campaigns, no widget colour changes, no new pages with the chat prominently promoted during the test, if you can avoid it.
Run for the planned period. Resist the urge to peek and stop early on a good day. Early peeking is the most common way to find false winners.
Switch back or on. After the first period, run the alternative for the same number of weeks, or keep the new version and watch the next period as a check.
Compare and decide. Compare rates, not totals, and check the guardrails. If the difference is smaller than your normal week-to-week swing, call it a tie and keep the simpler version.

If you run several websites, you can use them as a cleaner comparison when they have similar audiences: put version A on one and version B on the other for the same weeks. Each website in Talkmio has its own settings, and all share one Inbox. The Free plan covers one website, Pro three and Ultimate ten, per the pricing page.

Common Mistakes in Chat Experiments

Changing several things at once. You cannot tell which change did the work, or whether one cancelled another.
Stopping on a good day. Chat volume is noisy; a strong Tuesday proves nothing.
Ignoring who is online. If the team was away half of one test week, the offline experience, not the greeting, drove the result.
Judging by volume alone. More chats of lower quality, more hand-offs or worse ratings is a loss.
Testing trivia. A slightly different shade of the same colour will not be detectable at small-site traffic.
Letting the AI answers change mid-test. If you edit the knowledge base or the business instructions during the test, the guardrail numbers move for another reason. Freeze content or log each edit.

A Worked Example: Testing a Welcome Text

Imagine a small online shop with about 6,000 visitors a week and a baseline of 90 conversations a week, which is 1.5 conversations per hundred visitors. The owner suspects the generic welcome text, “Hi, how can we help?”, is too vague. The hypothesis: a text that names three things Mio can answer, such as delivery, returns and sizing, will lift conversations per hundred visitors.

Version A, the current text, runs for two weeks, and version B runs for the next two. At 6,000 visitors a week, each version sees about 12,000 visitors. The shop records conversations per hundred visitors, ratings and the share of chats Mio answers alone. Suppose B shows 1.9 against 1.5 for A, ratings unchanged and the same share of chats answered by Mio alone. The lift is visible, the guardrails are flat and the difference is larger than the shop’s normal two-week swing, so B stays. The numbers here are illustrative, not a benchmark; your own baseline decides what counts as a real change.

If the same shop saw 1.6 against 1.5, the honest conclusion is “no detectable difference”. It keeps the simpler text and tests something bigger next, such as removing a required form field.

Testing the AI Side Separately

The assistant’s answers can be tested too, but with a different method. Use Try Mio to compare two versions of a rule or a knowledge piece against the same ten questions and read the answers, the pieces used and the response time. That is a quality check on content, not a traffic experiment. Our guides on AI chatbot instructions and how to test an AI chatbot before you go live describe how to build a question set that you reuse every time something changes.

Frequently Asked Questions

Can I split test live chat in Talkmio?

Talkmio’s documentation lists no built-in tool that shows two widget versions to two groups at once. Widget settings such as welcome text, pre-chat form and position apply within five minutes, so the practical method is a sequential test: run one version for a set number of whole weeks, switch to the other for the same period and compare rates and guardrail metrics.

What should I test first on a chat widget?

Start with the welcome text or the pre-chat form, because both strongly affect whether visitors write and both take minutes to change. Make the two versions clearly different, for example a generic greeting against one that names a concrete question Mio can answer, or no form against a required one. Subtle wording changes need more traffic than most small sites have.

How long should a live chat test run?

Run each version for at least one full week and preferably two, so weekday effects are equal. Then check that you have enough visitors: roughly 1,100 per version to detect a rise from 20 to 25 percent, about 3,800 for 10 to 12 percent and about 5,300 for 3 to 4 percent. If you cannot reach that, test bolder changes.

Which metric should decide the winner?

Use conversations started per hundred visitors as the primary metric, with visitor counts from your website analytics, since Reports count conversations rather than page views. Add two guardrails: ratings and the share of chats Mio answers alone versus hands over. A version that raises volume but lowers ratings or fills the Inbox with empty chats has not won.

Can I run two versions at the same time on one site?

Not within one website entry, because the widget settings apply to every visitor of that site. If you own two similar websites you can run version A on one and version B on the other during the same weeks, which removes timing effects. Each website has its own settings and all share one Inbox.

What if the result is a tie?

Keep the simpler or cheaper version and move on. If the difference is smaller than your normal week-to-week variation, the test did not detect an effect, and that is a useful result: it tells you the element does not matter much at your traffic. Spend your next test on a different setting, such as the form or the offline text.

The Bottom Line

Split testing live chat pays off when you test one bold change at a time, run whole weeks, collect enough visitors and judge by rates and guardrails instead of one good day. Talkmio has no built-in split tool, but its settings change in minutes and Reports show what happened. Start with the welcome text or the pre-chat form. Create a free Talkmio account, record this month’s baseline and run your first honest test.

Split Testing Live Chat: What to Change and How Long to Wait