What is an A/B test, and how long before the result means anything?

What has to exist first

Three pages, not two.

• Variant A and Variant B. Two real pages, both inside the same funnel. systeme.io requires both to exist there before a test can be created.
• A goal page. systeme.io describes its purpose as having a common page that both variants lead to. Its example uses two opt-in pages competing, with a sales page as the goal.

The redirect from each variant to the goal page is yours to configure. A button's action decides where a page sends somebody: to the next step in the funnel, or to a URL you type in.

Starting it

Open Sites, then Sales funnels, then the funnel, and click its A/B test tab. Click Choose a variant, and the popup asks for two things: the variant page that A will compete against, and the goal page, described as where leads are redirected after opting in. Save, then click Start A/B test.

Pick the wrong variant and the test can be cancelled and set up again. Once it is running, a statistics table shows the results, and the pages in the test carry an icon that tells them apart from the rest of the funnel.

How long before the result means anything

systeme.io does not say, and it is better to know that than to be handed a number by somebody guessing.

Its article on page tests documents a start button and a way to cancel. It documents no duration field, no split of traffic you can set, and no automatic winner. The test measures. The reading of it is left to you.

The only timing figures in either of systeme.io's two A/B test articles sit in the newsletter one, attached to a duration field the page test does not have. It calls 48 to 72 hours usually sufficient for a standard campaign, asks for at least 24 hours so contacts have time to open the email, and says the point of the wait is to gather a representative number of responses.

That principle transfers. The hours do not, because they measure an email waiting in an inbox rather than traffic arriving on a page.

No minimum number of visitors and no confidence figure appears anywhere in the page-test documentation. Small numbers move about, and nothing systeme.io publishes says when a gap has stopped moving.

The one piece of method it does publish

Change one thing between the variants. systeme.io's warning, again from the newsletter article, is that testing several elements at once makes the results hard to interpret and limits what the conclusion is worth.

Two pages that differ in headline, image and button colour will produce a winner and no explanation.

What a test locks

• Pages in an A/B test can be duplicated, but they cannot be deleted or moved to another funnel.
• Previewing switches the A/B logic off. Clicking View funnel step beside a variant opens that exact page, so a design can be checked without landing on the other one.

Not the same feature as the newsletter test

Newsletters have their own A/B test, and one of its two modes is far more automated than anything the page test offers: a duration you set, a sample ratio you set, and a criterion that picks the winner and sends it to everybody else.

The page test has none of that. Same two letters, different machinery.