An A/B test is a method of comparing different versions of your websites or apps, and see which variation makes you more successful.
A/B tests are also known as experiments or split tests. In an A/B test you show two or more different variations to your users (visitors) and the variation that performs better wins. When a user enters the experiment a variation will be randomly chosen and the user will see this variation for all subsequent visits. Experimenting in this way lets you maximise your success. Read more about A/B Testing on the user guide.
A/B Testing is a plugin for Matomo (Piwik) and is available for purchase on the Matomo Marketplace as a yearly subscription. While the subscription is active you will receive all updates for this plugin.
You can also get it as a hosted solution on our Matomo Analytics (formerly Piwik Analytics) Cloud.
The plugin is developed and maintained by InnoCraft, the company from the makers of Matomo (Piwik). At InnoCraft, talented and passionate developers build and maintain the free and open source Matomo Analytics (formerly Piwik Analytics) platform. This ensures the highest quality and compatibility for all of their plugins. As a result, popular features like Segmentation work out of the box with this plugin.
This plugin is built and maintained by InnoCraft, the makers of Matomo (Piwik). Therefore we can make sure the plugin is well integrated, kept up to date and automatically tested whenever a change is made in Matomo. By purchasing this plugin you also help the developers to being able to maintain the free and open source project Matomo itself.
No, A/B Testing is a plugin that is built on top of Matomo (Piwik). To use it, you first need to install Matomo. Matomo requires PHP, a MySQL database and a webserver like Apache or Nginx. Learn more about Matomo. Alternatively, to take away the hassle of installing and maintaining your own Matomo, signup to our Matomo Cloud (formerly Piwik Cloud) service.
Matomo (Piwik) 2.17.0 or newer is required. You can also signup to our Matomo Cloud (formerly Piwik Cloud) service.
Here are just a few examples of typical A/B tests experiment that can increase your website’s revenue and conversion and grow your business:
- Often your goal is to convert people to submit a form to contact you, or signup to your service. Experiment with different form designs for example a simpler form asking for less information, making the form fields bigger and more visible, or changing the labels of your form fields.
- In your marketing landing pages you may display a photo or several photos. Experiment with other photos for example: a photo showing a happy person vs a photo showing your products in action vs several smaller photos.
- in your marketing pages or marketing campaigns, experiment with using different headlines and call to actions text.
- Experiment with a different navigation on your website, by reorganising your menu or even re-designing your menu. Compare your success metrics for a vertical menu vs horizontal menu.
- On your ecommerce website, you can test the label of your Call to action button:
Add to Cartvs
- Experiment with the location of your button on the page.
- Experiment with the style of the button: colors, size, …
- In your checkout flow, experiment with removing some elements of your page (selectors, buttons, links) to make your page as simple as possible and compare with your original design to see whether your ecommerce orders increase.
Yes, one of our main goals from the beginning was to explain each individual step of setting up an experiment directly in Matomo:
- We show inline help within the UI
- We guide you through the configuration of an experiment
- We help you analyze the report by providing conclusions based on the tracked data
- We show useful information about the experiment directly in the report to remind you what you wanted to achieve initially
We support the same browsers as Matomo (Piwik) does, meaning experiments will run even on older browsers.
We mainly use Matomo (Piwik) UI components provided by the Matomo platform. Therefore we support the same browsers as Matomo does.
No, the data will be only tracked into your Matomo (Piwik). 100% data ownership and privacy.
To manage your experiments:
- in Matomo (Piwik) 2 log in to Matomo and click on “Personal => Experiments”.
- In Matomo (Piwik) 3 you can find the menu item under “Administration => Experiments”.
When you configure an experiment, Matomo (Piwik) lets you define which outcome will be considered a success to you. You can choose success conditions and success metrics. Success conditions let you for example define how big of an improvement you expect from your A/B test (the Minimum Detectable Effect).
Some of the available success metrics include:
- Goal conversions
- Goal revenue
- Ecommerce orders
- Ecommerce order revenue
- Total number of conversions
- Total revenue
- Page views
- Visit length (time)
You can select any pre-existing goal as a success metric, or create as many new goals as you wish. For each selected success metric you will see a summary and an evolution graph across all variations as well as individual reports for each success metric.
Matomo (Piwik) will show you reports for each selected success metric to see how the different variations compare against each other. At a glance you will be able to see whether a variation is currently performing better or worse compared to the original version.
Matomo (Piwik) also shows you how confident we are that the improvement is actually due to this change (statistical significance) and for how long the experiment should run at minimum.
As soon as enough users have entered an experiment, Matomo (Piwik) will start giving you recommendations about the results.
- When a variation is improving your success metric, it will be highlighted as the winning variation.
- When a variation is statistically significantly making your success metrics worse, it will be highlighted as a losing variation.
When you create an experiment, Matomo (Piwik) shows you the code that needs to be embedded into your project in order to run the experiment. The implementation depends on where you want to run an experiment:
- Run an experiment in your website using server-side technology
- Run an experiment in a mobile app, desktop app or game
- Run an experiment in a Marketing ad campaign or emailing campaign
Yes, you can allocate only a certain percentage of your users to enter an experiment. For example you can configure that only 20% of your users should take part in an experiment.
trigger method to further restrict who enters an experiment. For example you could activate the experiment only for users from a certain location. Read more about this in the developer docs for running an experiment.
Yes, you can fully customize the ratio of how often a certain variation should be selected compared to your other variations.
Say you have three variations “Blue”, “Red” and “Green” plus the “Original” version. By default, Matomo (Piwik) allocates 25% of your traffic to each variation. You may also choose to allocate more traffic to “Blue” and “Red” while your “Green” and the “Original” version gets less traffic.
Is it possible to run experiments on the server, for example using PHP, Python, Java or any other language?
Yes, it is possible: you can do this by using any A/B testing framework of your choice. Read more about this in the guide for running experiments server-side.
Yes, it is possible in combination with the Matomo (Piwik) Android SDK or Matomo iOS SDK and an A/B testing framework that is compatible with your mobile platform. Read more about this in the guide for running experiments in an app.
How do I run an experiment in a marketing campaign such as a banner ad, search marketing ad or an email marketing campaign?
For email campaigns your newsletter tool needs to support A/B tests (assigning your recipients into different groups). You can then send different newsletters to your visitors and see how the different newsletters affect the browsing behaviour on your website. This is only recommended when you have many users that click on your newsletter.
To track how different ad or email campaigns affect the browsing behaviour on your website, you only need to add two URL parameters to your campaign URLs. Read more about this in the guide for running experiments in a campaign.
Here are just a few use cases you may experience in your day to day work.
One of your most important landing page has been redesigned to better showcase the value of your services, and you need to make sure the new version is not making things worse. To compare the performance of two new versions and check the new version does not decrease number of contact form signups or the , you run a A/B test and see whether the redesigned version is converting more visitors to reach your goals.
As a marketer or product manager, you often need to improve conversions at any particular step of the conversion funnel. Basically you need more visitors to get closer to the end of the funnel and convert your goals (whether signing up, leaving an email address, downloading a brochure, watching a video, etc.). To improve your conversions you may start by identifying the bottlenecks and what is causing users to leave the funnel and where. For example, you may be losing many users at a particular step of your conversion journey such as your signup form or your checkout page. You and your team may then brainstorm several solutions that would make things better, and your team decides to implement the most important change first. When you implement this change (and any others) using A/B Testing, you are actually now scientifically measuring and the A/B testing product will confirm (or deny) that your change has indeed improved conversions (and how much improvement).
As a product or game designer, you are building an app, game or any other software product, and you wish to soft launch a new exciting and innovative feature. Soft Launching is a great way to get ideas tested early without much risk. Soft launching means that your new feature will be visible only on a subset of users and compare behaviour. So you can use A/B Testing to soft launch your apps and games features, and measure how the new feature performs on for example 10% of your user base. A/B Testing indicates whether users are better engaged in the app (across all your success metrics), after using the new feature.
See more example in the blog post: A/B Testing Platform for Matomo Analytics (formerly Piwik Analytics).
Yes, you can segment your visitors by experiment they have participated in, and even segment your visitors by each specific variation they got to see. This lets you get more insights into how t different variations have affected your other reports in Matomo (Piwik).
Say you are running an experiment “Buy Now Button Color”. You can segment Matomo (Piwik) reports by visitors that have seen the “Blue color” of the “Buy Now Button Color” and compare the report to the visitors that have seen the “Red color”. This lets you compare in detail how each variation has performed against each other.
By using the segmentation in combination and viewing the Visitor Log, you can see in detail the path and actions of each of your users who viewed a particular variation. This lets you view how the behavior of a particular user may change depending on which variations they got to see.
Is it possible to apply other segments (such as Country or Marketing campaign) to the A/B tests reports?
Yes, you can apply any Matomo (Piwik) segments to your A/B tests reports and see for example how your different variations compare against each other in different countries (when applying the “Country” segment), or how your variations compare depending on which marketing campaign or referrer brought the visitor to your website or app. To do this open an A/B test report, then create or apply your custom segment.
Yes, you can optionally schedule a start and finish date for your experiments. To schedule a starting or ending date manually, edit your experiment and click on “Schedule”. By default, an experiment starts automatically as soon as the first user enters your experiment, and the experiment can be marked as finished at any time.
The time it takes for an A/B test experiment to run depends on the experiment, your business cycles, the improvement rate you expect and many other factors such as conversion rates. Once an experiment is running, Matomo (Piwik) will let you know the minimum number of visitors that need to participate in your experiment in order to reach a statistically significant result that will be repeatable and not due to random chance. The amount of needed visitors may change while the experiment is running.
We recommend to let experiments run for several business cycles, for example several full days or even better let the experiment run for several full weeks, as user behaviour changes at night and on the weekends.
Is it possible to change the confidence threshold / statistical significance needed to decide a winning variation?
Yes, Matomo (Piwik) uses by default a threshold of 95% but you can lower or increase the expected threshold. The higher you set the confidence threshold, the more you can be sure that the results are actually due to this experiment and not due to randomness. When you set a higher threshold, more users will be needed to participate in the experiment.
Yes, by default an experiment is activated on all of your web pages. You can also choose to customize on which pages an experiment should be activated. For example you can define to activate an experiment only if the path of an URL starts with “/blog” and has a URL parameter “language”. There is also a tool in the Experiments creation wizard which lets you test whether users would enter an experiment on a given URL or not.
By default, we try to store the activated variation name in the Local Storage if the browser supports this feature. If the browser does not support this feature, we will set a cookie for up to one year to remember which variation was once selected for this particular visitor.
You can add a URL parameter
?pk_ab_test=$variationName to any URL to force the activation of a specific variation. Make sure to replace
$variationName with the name of a variation.
This is useful when you are integrating an experiment and you want to test each variation or when you need to share a URL for each variation with your colleagues to get an approval before running an experiment.
Yes, you can create, update, finish and archive experiments via the Matomo (Piwik) HTTP Reporting API.
Yes, you can request experiment reports in different formats such as JSON, XML, CSV and HTML to integrate the results anywhere. Read more about the Matomo (Piwik) HTTP Reporting API and view the Reporting API Reference.
Yes, if you host Matomo (Piwik) yourself you get access to all data that is stored in your MySQL database. The tracked data is stored in a table called
piwik_log_abtesting, the A/B test configurations are stored in a table
The Ab testing plugin integrates with the Activity log plugin to let you see who and when changes were made to your A/B tests, for example when a given experiment was started, finished or archived.
piwik.js. Having a small footprint and minimizing the tracker size is very important to us.
Yes, you can create as many A/B tests (experiments). With Matomo (Piwik), our goal is to provide a no data limit policy. We recommend to:
- run only a few experiments at the same time as they may affect each other.
- or run only experiments which do not affect each other.
Yes, it is possible to compare an unlimited number of different variations. In a typical A/B test, two variations A and B are compared against each other.
You may however create more variations as you wish. However please note that when testing many variations at the same time, the time it takes for Matomo (Piwik) to reach a statistically significant result (and decide which variation performs better or worse) will increase with the number of variation you are testing. This is why we recommend to test only a few variations at the same time.