On Wed, Apr 5, 2023 at 12:23 PM Kevin Cozens via talk <talk@gtalug.org> wrote:
... To avoid the problem in the future I always have servers running logwatch. It gives me a daily report of system activity based on various log files. Of particular relevance to the topic of this message thread is the bit near the bottom. It reports on in use and available disk space. ... Daily reports getting mailed to you can have a problem that they're long and repetitive and get ignored after the novelty of reading them wears off. So long as they're concise and to the point and focus on anomalies and don't go listing all the normal things and you actually read them
On 2023-04-05 15:34, o1bigtenor via talk wrote: they can be ok. And some issues can go from zero to major problem in less than a 24-hour cycle so an alert sooner can help.
Quite a number of years ago a now deceased mentor advised that a great way to reduce the problem caused by a runaway var file was to use separate /var and /usr partitions (from / and /home).
(Not sure Thunderbird is getting the quotation levels right here, but the first bit was Kevin and this bit was o1bigtenor. Getting homesick for Mutt.) The filesystem root and /usr are static, so they only care at upgrade time that there's enough space then, and at other times will happily sail past filesystem-full events. Separation mostly makes sense if you want one thing to still work when another thing fills its partition, eg being able to write /var/log even though the dear users have filled /home or /tmp again. And nowadays /tmp is usually its own tmpfs already. Monitoring disk space and being proactive before usage hits the roof is a good thing for those of us who value tranquillity and hate having to deal with things like this at inconvenient times or after the users notice. But a long-ago coworker once said "We've got to be reactive, not proactive" because we were in an environment where you got brownie points for heroically responding to disk-full tickets, but if you kept things running smoothly you'd be looking at layoff papers. (In other news, I'm no longer there.) Anthony