uffizi disk's full makes some workers fail

Uffizi regularly got its / disk full. Resulting in workers (well at least the indexers) failing about OSError... no more disk space (P197 for a sample extracted from our logstash instance).

It's possibly related to #789 (closed) (quoting the first line of that issue):

We currently let the systemd journal take up as much space as it needs on the root filesystem of our machines.

And indeed, the /var/log/ (mounted on /) is quite bloated at that moment.

The workaround so far, for my part, was to:

• claim some space in /var/cache/apt/archives
• to permit triggering the logrotate rune to compress logs

sudo logrotate -f /etc/logrotate.conf

A better workaround (as a default for a better solution) would be to increase uffizi's disk size.

---

Migrated from T866 (view on Phabricator)