How do I decrease archiving memory usage when processing a large number of websites?

Matomo installations with thousands of websites may experience high memory usage when running the auto-archiving process. Matomo provides options for limiting the maximum number websites or archives that the auto-archiving process will attempt in a single scheduled archiving operation. By breaking the processing work across multiple scheduled archiving operations overall memory usage and server load can be managed.

max-websites-to-process This option will limit the number of websites that the archiving operation will process at once. Any websites not processed in this run will be processed on the next scheduled archiving operation.

max-archives-to-process This option will limit the number of archived reports that will be processed at once. Any unprocessed reports will be processed on the next scheduled archiving operation.

To make use of these options add them to the cron command described in How to set up auto-archiving of your reports

For example, to limit website processing to 5 at a time:

5 * * * * /usr/bin/php /path/to/matomo/console core:archive --max-websites-to-process=5 --url=http://example.org/matomo/ > /dev/null

or to process 100 archives at a time:

5 * * * * /usr/bin/php /path/to/matomo/console core:archive --max-archives-to-process=100 --url=http://example.org/matomo/ > /dev/null

The total number of websites or reports to be processed along with the frequency of the archiving schedule should be taken into consideration when determining the optimum values for these options.