How to backfill data for export from Matomo
When you first enable the BigQuery and Data Warehouse exports, it will begin regular processing of your analytics data from the date you enabled the feature. If you would like access to your historic data, you will need to configure the date ranges you’d like to see.
Note: This system uses Amazon S3 on the backend, which has strict data limits. Therefore, it will only process a certain amount of data per hour and the number of days this covers will vary depending on the amount of data your Matomo instance contains.
To ensure that you get access to your most important data as soon as possible, we recommend only enabling smaller date ranges that you plan to actively access in your data warehouse platform to avoid filling your hourly quotas with data that may not be necessary yet.
-
In the Matomo dashboard, click the cog icon to open the Matomo Admin settings.
-
Navigate to Export > BigQuery & Data Warehouse Export and scroll down to the Export Jobs section.
-
Click on the Add Backfills Export Job button.
-
Select a start and end date for the data you want to access in your data warehouse and click on Submit to queue your backfill request.
-
Once you have submitted your backfill request, it will enter a Queued status. At this stage, you will need to wait for the data to be processed and uploaded to the Export Table Paths shown further down the page.
You will notice that each date is added as a unique job and the processing can take several hours, days or longer depending on the amount of data you have tracked. There are three potential statuses shown for backfill jobs in the Export Jobs section.
- Queued – A backfill has been requested for the date and you will need to wait.
- Processing – The backfill request is currently processing and should be available soon for importing to your data warehouse platform.
- Finished – Data for the specific date has finished processing and therefore the data should be available from the links in the Export Table Paths for use within your data warehouse platform.