Create, view, track and analyse A/B tests

A/B testing helps you find out which version of a page or other experience works best for your visitors. By comparing different variations, you can see which one performs better against your goals, such as increasing conversions or engagement.

A/B testing in Matomo involves two main steps:

1. Create and configure your experiment in Matomo. Choose what you want to test, define your variations and select the success metrics that you will use to compare their performance.
2. Add the experiment code to your website or application. After configuring your experiment, you will need to embed the code that runs the test and tracks the results.

Both steps are needed to run an A/B test. Creating an experiment in the Matomo interface alone does not automatically start testing your variations.

This guide explains how to manage and create A/B tests in Matomo.

Manage A/B tests

To manage existing experiments in Matomo, go to A/B Tests > Manage A/B Tests.

Use the Filter dropdown to display experiments by status, or the Search field to find a specific test by name. The table shows each experiment’s name, description, status, start date and finish date.

The available Actions depend on the experiment’s status:

• Active: Edit experiments that have been created or are currently running.
• Running: Edit the experiment or view its report. Avoid changing an A/B test configuration once it is running, as this may affect the reliability of its results. If you want to delete a running experiment, you will need to manually finish the test or wait for its finish date and then delete it.
• Created: Edit or delete an experiment that has not started.
• Finished: View the finished A/B test report or archive it. You can still edit a finished experiment, but this will not change the results already collected.
• Archived: You can edit an archived experiment but its reports and segments will no longer be available in the Matomo reporting interface.

Create an A/B test

You need at least write permission to create or edit experiments for a website or app. Each website or app can have multiple experiments.

1. Go to A/B Tests > Manage A/B Tests and click Create New A/B Test.
2. Enter a unique, descriptive Name to identify your test in reports. Use only letters and numbers, without spaces, up to 50 characters.
3. The Description is mandatory to briefly explain what you are testing.
4. The Hypothesis is mandatory to describe what you expect to happen and why.
   
5. For Variations, the Original is your existing version.
6. Enter a name for each alternative version you want to test.
7. Click the + icon to add more variations.
8. If you created a new version of your page at a different URL, you can configure this after creating the A/B test in the Redirects section to specify the page URL for each variation.
9. When the fields are completed, click Create New A/B Test to save.

You can then configure your experiment before embedding the code needed to run the test.

Configure an A/B test

After creating your A/B test, configure the settings to define how the experiment runs and how you will measure its success.

Success metrics

Choose from the available Success metrics to determine what Matomo measures to compare your variations and help you identify which performs best. You can also select existing goals in Matomo or create new goals to include as a success metric.

Success conditions

Define the Success conditions to determine when you have enough evidence to identify a winning variation. Use the default settings of 15% MDE and 95% confidence or adjust them based on the size of the improvement you want to detect and the traffic your website receives:

1. Minimum Detectable Effect (MDE): The smallest improvement you want your test to be able to detect. For example, if your current conversion rate is 10% and you set an MDE of 15%, it means you are aiming to detect an improvement to 11.5% or higher. Smaller improvements generally require more visitors to detect reliably.
2. Confidence Threshold: This defines how strong the statistical evidence must be before Matomo identifies a variation as a winner. The default is 95%. A higher threshold requires stronger evidence and may mean running your test for longer.
3. Click Update to save any changes. These settings also help Matomo estimate how many visitors your experiment will need.

Target pages

Choose the Target pages that control where visitors enter the experiment. Note that the Redirects section determines which alternative page URL visitors are sent to for each variation.

You can include specific Target pages, exclude others, or allow the experiment to run across your entire website. A visitor only needs to match one inclusion rule to qualify, while matching any exclusion rule prevents them from entering the test.

To add multiple rules, click the + icon to add more conditions:

1. To include pages, define A visitor will enter the A/B test when: Choose a rule to specify which URLs qualify for the experiment. For example, you could target only your contact page or a group of pages with matching URLs.
   
2. To exclude pages, define A visitor will not enter the A/B test when: Specify any URLs you want to exclude, even if they match your inclusion rules.
3. Use the URL validator at the top of the screen to enter a full page URL and check whether a visitor on that page would enter the experiment.
4. Click Update to save your changes.

Traffic Allocation

Apply Traffic Allocation to control how many visitors participate in your A/B test and how they are divided between variations.

1. Percentage of visitors who enter this A/B test: Choose what percentage of eligible visitors will participate. For example, setting this to 70% means that 30% of your visitors will not enter the experiment and will continue seeing the original version.
2. Percentage of traffic allocated to each variation: Decide how participants are split between the original and alternative versions. By default, traffic is divided equally (50/50 for two variations), but you can adjust these percentages.
3. Click Update to save any changes.

Test different page URLs using Redirects

If you want to compare different versions of a page when each version has its own URL, use Redirects. For example, if your original contact page is example.com/contact and your redesigned page is example.com/contact-new, visitors assigned to the Original will remain on the existing page, while those assigned to the New Design variation will be redirected to the new URL.

1. Open the Redirects in your A/B test.
2. Enter the full URL of the alternative page for each variation you want to redirect visitors to.
3. Leave the redirect URL empty for the Original variation if visitors should stay on the existing page.
   
4. You can also choose whether to pass URL parameters to the destination page. Enable Forward all URL query parameters to preserve query parameters from the original URL when redirecting visitors. This will automatically include forwarding campaign URL parameters as well.
5. To only preserve campaign tracking parameters, such as utm_source or mtm_campaign, enable the option Forward campaign URL parameters.
6. Click Update to save any changes.
7. Make sure the Matomo tracking code is installed on all pages involved in the test, including the redirect destinations.
8. Use Target Pages to specify where visitors can enter the experiment, so redirects only occur on the intended pages.

For more information, see the developer guide on testing different pages using redirects.

Schedule your A/B test

By default, an A/B test starts when the experiment code is embedded and a visitor enters the test, and continues until you manually finish it. You can optionally use the Schedule to choose when your experiment starts and finishes. All scheduled dates and times use the UTC timezone.

1. Start A/B test on: Select a date to start the experiment automatically. Make sure the experiment code is embedded before the scheduled start date. Leave this blank to start the test as soon as the experiment code is embedded and a visitor enters the A/B test.
2. Finish A/B test on: Select a date to end the experiment automatically. Allow enough time to collect meaningful results before scheduling an end date. Leave this blank if you prefer to finish it manually once you have collected enough data.
3. Click Update to save any changes.

Embed the A/B test code

Once you have configured the A/B test, you need to add the experiment code to your website or application so the test can start and track results. If you set a Schedule, the experiment will start on the date provided the A/B experiment code is embedded in your website pages.

1. Open Embed code in your experiment settings. Matomo generates code specifically for your experiment based on the configuration settings.
2. Choose the implementation method that matches your website or application and copy the experiment code displayed on-screen. For guidance on using the different methods, read how to embed an experiment.
3. Place the A/B test code before the Matomo page view is tracked. Attribution of conversions on subsequent pages depends on your experiment’s tracking configuration.
4. Add the generated A/B test code to the page where visitors enter the experiment, as defined by your Target Pages settings. The code assigns visitors to a variation and determines which version they see.
   
5. Ensure that the Matomo tracking code or Tag Manager container code is installed on all relevant pages, including redirect destinations and pages where conversions occur.

When visitors navigate to other pages, the A/B test code does not normally need to run again unless those pages are also part of the experiment.

If you are testing a redesigned contact page, visitors entering contact.html will be assigned either the original design or the new variation. If you configured a redirect, visitors assigned to the new design will be sent to its alternative URL.

View and analyse A/B test results

Once your experiment is running, you can monitor how each variation performs against your chosen success metrics.

1. To view your results in Matomo, open A/B Tests in the reporting menu and select an experiment.
2. You can view reports for both running and finished experiments.
3. Use these reports to compare your variations and determine whether you have enough evidence to identify a winner.
   

Each experiment report includes:

• Experiment summary is an overview of your experiment and its current status.
• Evolution graph displays how your selected success metrics perform over time for each variation.
• Overview report: Compares the results of all your success metrics across variations.
• Detailed metric reports: Show which variations are performing best for each success metric, including statistical information to help you assess the reliability of the results.

Finish an A/B test

Once you have collected enough data and reviewed your results, you can finish the A/B test.

1. Open your A/B test report and select Finish A/B test from the experiment summary. It is strongly recommended to run A/B tests for at least one or two full business cycles.
   
2. You can also go to Manage A/B Tests, edit a running experiment and select Finish experiment. If you do not see this option, you may not have appropriate permissions to edit the test.
3. After finishing your experiment, remove the experiment-specific code from your website or application.
4. If you identified a winning variation, decide whether to implement it permanently.
5. You can continue viewing the experiment’s results in Matomo until you archive it.

Archive an A/B test

Archiving removes a finished experiment from the reporting interface, so you can keep your list of experiments organised. When you archive an experiment, its reports and segments are no longer available in the Matomo reporting interface. However, the experiment remains accessible through A/B Tests.

1. Go to A/B Tests > Manage A/B Tests and filter the list of experiments by the Finished status.
2. Click the archive icon in the Action column to archive the experiment.
3. To find an archived test, go to Manage A/B Tests and select Archived from the status filter. This will list all archived A/B tests.
   

Once your A/B test is complete, review the results to determine which variation performed best against your success metrics. Use these insights to make informed decisions about your website or application, implement successful changes and identify opportunities for future experiments. For more guidance, explore the A/B Testing guides.