In this post, we will explain how Matomo (Piwik) uses continuous integration to deliver a quality software platform to dozens of thousands of users worldwide. Read this post if you are interested in Matomo, Quality Assurance or Automated testing.
Why do we care about tests?
Continuous Integration brings us agility and peace of mind. From the very beginning of Matomo, it was clear to us that writing and maintaining automated tests was a necessity, in order to create a successful open source software platform.
Over the years we have invested a lot of time into writing and maintaining our tests suites. This work has paid off in so many ways! Matomo platform has fewer bugs, fewer regressions, and we are able to release new minor and major versions frequently.
Which parts of Matomo software are automatically tested?
- Matomo back-end in PHP5: we use PHPUnit to write and run our PHP tests: unit tests, integration tests, and system tests.
- piwik.js Tracker: the JS tracker is included into all websites that use Matomo. For this reason, it is critical that piwik.js JavaScript tracker always works without any issue or regression. Our Javascript Tracker tests includes both unit and integration tests.
- Matomo front-end: more recently we’ve started to write JavaScript tests for the user interface partially written in AngularJS.
- Matomo front-end screenshots tests: after each change to Matomo, more than 150 different screenshots are automatically taken. For example, we take screenshots of each of the 8-step installation process, we take screenshots of the password reset workflow, etc. Each of these screenshot is then compared pixel by pixel, with the “expected” screenshot, and we can automatically detect whether the last code change has introduced an undesired visual change. Learn more about Matomo screenshot tests.
How often do we run the tests?
The tests are executed by Travis CI after each change to the Matomo source code. On average all our tests run 20 times per day. Whenever a Matomo developer pushes some code to Github, or when a community member issues a Pull request, Travis CI automatically runs the tests. In case some of the automated tests started failing after a change, the developer that has made the change is notified by email.
Should I use Travis CI?
Over the last six years, we have used various Continuous Integration servers such as Bamboo, Hudson, Jenkins… and have found that the Travis CI is the ideal continuous integration service for open source projects that are hosted on Github. Travis CI is free for open source projects and the Travis CI team is very friendly and reactive! If you work on commercial closed source software, you may also use Travis by signing up to Travis CI Pro.
Summary
Tests make the Matomo analytics platform better. Writing tests make Matomo contributors better developers. We save a lot of time and effort, and we are not afraid of change!
Here is the current status of our builds:
Main build: 
PS: If you are a developer looking for a challenge, Matomo is hiring a software developer to join our engineering team in New Zealand.
