war story: gtalug.org's filled up
(This post is also a test of whether the mailing list is working.) GTALUG's server's filesystem filled up: little disk space left. I discovered this when I tried to do "apt update; apt full-upgrade". The second step failed, saying that /var/cache/apt/archives had no room. This means that / has no room because / contains that directory. Tip: "df /some/path" will tell you how much space is used on the filesystem containing /some/path and it will tell you the mount point of that file system. I wandered around the filesystem, doing: sudo df -s * | sort -n This command lists things in the current directory, and their sizes, largest last. (It skips things with names starting with ".".) Pretty soon, I found that most of the space was taken up by /var/cache/apt/archives after all. This kind of surprised me. Googling got me to others with this problem. The advice: sudo apt autoclean That gave back (only) 3% of the disk. That directory was still way too big. Most of the space was taken by 35 versions of gitlab-runner. I have no idea why we need multiple versions. Violence is sometimes the answer. sudo apt clean That left only 32% of / used. Note /var/cache/apt/archives is only a cache. If the system wants any of these, it should be able to find them in a repo.
On 2023-04-04 17:50, D. Hugh Redelmeier via talk wrote:
Violence is sometimes the answer. Now we see the violence inherent in the sysadmin! sudo apt clean
That left only 32% of / used.
Note /var/cache/apt/archives is only a cache. If the system wants any of these, it should be able to find them in a repo.
More recent Debian seems to have made automatic cleanup the default. Is this something older or more Ubuntu-like? And really it makes little sense to keep a .deb you're only going to install once! Anthony
| From: Anthony de Boer via talk <talk@gtalug.org> | | Is this something older or more Ubuntu-like? It's an old debian: 9 aka stretch. I just noticed that LTS ended 2022 June. Technical debt. https://www.debian.org/releases/stretch/ | And really it makes little sense to keep a .deb you're only going to install | once! Yeah, now that we trust the internet repos to be more reliable than our own archives.
On 2023-04-04 17:50, D. Hugh Redelmeier via talk wrote:
Violence is sometimes the answer. Now we see the violence inherent in the sysadmin! sudo apt clean
That left only 32% of / used.
Note /var/cache/apt/archives is only a cache. If the system wants any of these, it should be able to find them in a repo.
More recent Debian seems to have made automatic cleanup the default.
Is this something older or more Ubuntu-like?
And really it makes little sense to keep a .deb you're only going to install once!
Anthony I like to keep the debs around until after I have verified
On 4/4/23 18:00, Anthony de Boer via talk wrote: that the next version of a package hasn't borked me. Hugh, how many kernels are on your system? Also, what does `deborphan` say. You can try `dpkg -l | grep -v ^ii` and eliminate cruft. Finally, nothing beats `for i in bin boot etc home lib opt root usr var; do du -hs $i; done`. Do the results seem appropriate? Logging is often the culprit. Check systemd with ` journalctl --disk-usage` and reduce it if needed. -- Michael Galea
On Tue, Apr 04, 2023 at 06:29:27PM -0400, Michael Galea via talk wrote:
I like to keep the debs around until after I have verified that the next version of a package hasn't borked me.
No need. snapshot.debian.org exists. Every version ever is there as far as I know. -- Len Sorensen
On Tue, Apr 04, 2023 at 05:50:13PM -0400, D. Hugh Redelmeier via talk wrote:
(This post is also a test of whether the mailing list is working.)
GTALUG's server's filesystem filled up: little disk space left.
I discovered this when I tried to do "apt update; apt full-upgrade". The second step failed, saying that /var/cache/apt/archives had no room. This means that / has no room because / contains that directory.
Tip: "df /some/path" will tell you how much space is used on the filesystem containing /some/path and it will tell you the mount point of that file system.
I wandered around the filesystem, doing: sudo df -s * | sort -n
This command lists things in the current directory, and their sizes, largest last. (It skips things with names starting with ".".)
Pretty soon, I found that most of the space was taken up by /var/cache/apt/archives after all. This kind of surprised me.
Googling got me to others with this problem. The advice: sudo apt autoclean That gave back (only) 3% of the disk.
That directory was still way too big. Most of the space was taken by 35 versions of gitlab-runner. I have no idea why we need multiple versions.
Violence is sometimes the answer. sudo apt clean
That left only 32% of / used.
Note /var/cache/apt/archives is only a cache. If the system wants any of these, it should be able to find them in a repo.
My upgrades are always: apt update && apt full-upgrade && apt clean Definitely no reason to keep all the downloads around. -- Len Sorensen
On Tue, 4 Apr 2023 at 17:50, D. Hugh Redelmeier via talk <talk@gtalug.org> wrote:
(This post is also a test of whether the mailing list is working.)
GTALUG's server's filesystem filled up: little disk space left.
I discovered this when I tried to do "apt update; apt full-upgrade". The second step failed, saying that /var/cache/apt/archives had no room. This means that / has no room because / contains that directory.
Tip: "df /some/path" will tell you how much space is used on the filesystem containing /some/path and it will tell you the mount point of that file system.
I wandered around the filesystem, doing: sudo df -s * | sort -n
This command lists things in the current directory, and their sizes, largest last. (It skips things with names starting with ".".)
Pretty soon, I found that most of the space was taken up by /var/cache/apt/archives after all. This kind of surprised me.
Googling got me to others with this problem. The advice: sudo apt autoclean That gave back (only) 3% of the disk.
That directory was still way too big. Most of the space was taken by 35 versions of gitlab-runner. I have no idea why we need multiple versions.
Violence is sometimes the answer. sudo apt clean
That left only 32% of / used.
Note /var/cache/apt/archives is only a cache. If the system wants any of these, it should be able to find them in a repo.
I used to use a method very similar to yours - I presume you meant 'du' not 'df'. I used 'du -sh * | sort -h' which I found easier to read and understand. But then - "Hallelujah" (let's make this a religious discovery) I found 'ncdu', which is a TUI that does exactly the same thing, but makes it explorable by selecting and entering any directory to see the same listing inside that new dir. Give it a try: it makes the whole process of finding space-hogging directories immensely faster. And because it's a TUI, not a GUI, it works on remote hosts over SSH even if they don't have X/Wayland. I don't know how long it's been in the Debian repos: it's in the current Debian, it may not be in older ones. -- Giles https://www.gilesorr.com/ gilesorr@gmail.com
On 2023-04-04 17:50, D. Hugh Redelmeier via talk wrote:
GTALUG's server's filesystem filled up: little disk space left.
I've had that happen to me in the past where the disk space on a server suddenly runs out. One of the reasons it happened was due to the log file of some program growing out of control. To avoid the problem in the future I always have servers running logwatch. It gives me a daily report of system activity based on various log files. Of particular relevance to the topic of this message thread is the bit near the bottom. It reports on in use and available disk space. Assuming nothing major happens in a day to fill up the space I can keep an eye on disk usage and take action long before the amount of free space gets to a dangerously low level. The other part of keeping an eye on things is running logrotate to keep the size of log files under control. Both logwatch and logrotate run daily. -- Cheers! Kevin. https://www.patreon.com/KevinCozens | "Nerds make the shiny things that | distract the mouth-breathers, and Owner of Elecraft K2 #2172 | that's why we're powerful" #include <disclaimer/favourite> | --Chris Hardwick
On Wed, Apr 5, 2023 at 12:23 PM Kevin Cozens via talk <talk@gtalug.org> wrote:
On 2023-04-04 17:50, D. Hugh Redelmeier via talk wrote:
GTALUG's server's filesystem filled up: little disk space left.
I've had that happen to me in the past where the disk space on a server suddenly runs out. One of the reasons it happened was due to the log file of some program growing out of control.
To avoid the problem in the future I always have servers running logwatch. It gives me a daily report of system activity based on various log files. Of particular relevance to the topic of this message thread is the bit near the bottom. It reports on in use and available disk space. Assuming nothing major happens in a day to fill up the space I can keep an eye on disk usage and take action long before the amount of free space gets to a dangerously low level.
The other part of keeping an eye on things is running logrotate to keep the size of log files under control. Both logwatch and logrotate run daily.
Quite a number of years ago a now deceased mentor advised that a great way to reduce the problem caused by a runaway var file was to use separate /var and /usr partitions (from / and /home). Together with the foregoing - - - - well you should just never have a problem. HTH
Way back in October 2000, on the SAGE members mailing list, well-known expert Tom Limoncelli said: "It isn't a service if it isn't monitored. If there is no monitoring then you're just running software." --- Tom Limoncelli On Wed, 2023/04/05 02:34:15PM -0500, o1bigtenor via talk <talk@gtalug.org> wrote: | Quite a number of years ago a now deceased mentor advised that a great way | to reduce the problem caused by a runaway var file was to use separate | /var and /usr partitions (from / and /home). I would argue that these days, in most cases, that is no longer a good configuration. Sure, /var filling up won't prevent you from manipulating files in /home. But if /var or / or /tmp is full, you've got a problem. And with multiple partitions, you need to have multiple chunks of free space. And you have to have a good idea up front of what your space requirements are. And yes, with LVM you can often extend a partition. But multiple partitions just mean more problems to worry about. Better you just alert once your disk gets 80% (or whatever) full. My two cents - cheers. John
On Wed, Apr 5, 2023 at 12:23 PM Kevin Cozens via talk <talk@gtalug.org> wrote:
... To avoid the problem in the future I always have servers running logwatch. It gives me a daily report of system activity based on various log files. Of particular relevance to the topic of this message thread is the bit near the bottom. It reports on in use and available disk space. ... Daily reports getting mailed to you can have a problem that they're long and repetitive and get ignored after the novelty of reading them wears off. So long as they're concise and to the point and focus on anomalies and don't go listing all the normal things and you actually read them
On 2023-04-05 15:34, o1bigtenor via talk wrote: they can be ok. And some issues can go from zero to major problem in less than a 24-hour cycle so an alert sooner can help.
Quite a number of years ago a now deceased mentor advised that a great way to reduce the problem caused by a runaway var file was to use separate /var and /usr partitions (from / and /home).
(Not sure Thunderbird is getting the quotation levels right here, but the first bit was Kevin and this bit was o1bigtenor. Getting homesick for Mutt.) The filesystem root and /usr are static, so they only care at upgrade time that there's enough space then, and at other times will happily sail past filesystem-full events. Separation mostly makes sense if you want one thing to still work when another thing fills its partition, eg being able to write /var/log even though the dear users have filled /home or /tmp again. And nowadays /tmp is usually its own tmpfs already. Monitoring disk space and being proactive before usage hits the roof is a good thing for those of us who value tranquillity and hate having to deal with things like this at inconvenient times or after the users notice. But a long-ago coworker once said "We've got to be reactive, not proactive" because we were in an environment where you got brownie points for heroically responding to disk-full tickets, but if you kept things running smoothly you'd be looking at layoff papers. (In other news, I'm no longer there.) Anthony
Antony mentioned missing reading mail with mutt. I’m still using mutt, and it works really well for me. It automatically formats HTML only messages (using the elinks command), can throw HTML at my browser when I ask, handles O365 multi-factor authentication, interprets calendar messages and so on. I can pipe messages into my folder filing command, filter and save bulk mail by rules, and various other things. I will admit it took a fair amount of setup over the years, and surely isn’t for everyone. But I find it makes my e-mail life easier in a number of ways. And I rarely have to click a mouse button. Cheers John
I desperately miss mutt. But the University of Toronto, in its administrative wisdom, moved us all to Microsoft365 which insists on token security of a kind mutt doesn't implement. If I ever leave or find a way to implement it, I'm back to using mutt like a shot. On 4/5/23 21:20, John Sellens via talk wrote:
Antony mentioned missing reading mail with mutt.
I’m still using mutt, and it works really well for me. It automatically formats HTML only messages (using the elinks command), can throw HTML at my browser when I ask, handles O365 multi-factor authentication, interprets calendar messages and so on.
I can pipe messages into my folder filing command, filter and save bulk mail by rules, and various other things.
I will admit it took a fair amount of setup over the years, and surely isn’t for everyone. But I find it makes my e-mail life easier in a number of ways. And I rarely have to click a mouse button.
Cheers
John
--- Post to this mailing listtalk@gtalug.org Unsubscribe from this mailing listhttps://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgtalug.org%2Fmailman%2Flistinfo%2Ftalk&data=05%7C01%7Cpeter.king%40utoronto.ca%7C483707c5606745eb88a708db363d29ef%7C78aac2262f034b4d9037b46d56c55210%7C0%7C0%7C638163408543306840%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Ggekszrm3sNeMq%2BEmiQLfOtCOL0wzOMREsBk9RYbS9s%3D&reserved=0
-- Peter King peter.king@utoronto.ca Department of Philosophy 170 St. George Street #521 The University of Toronto (416)-946-3170 ofc Toronto, ON M5R 2M8 CANADA http://individual.utoronto.ca/pking/ ========================================================================= GPG keyID 0x7587EC42 (2B14 A355 46BC 2A16 D0BC 36F5 1FE6 D32A 7587 EC42) gpg --keyserver pgp.mit.edu --recv-keys 7587EC42
On Thu, Apr 06, 2023 at 10:51:34AM -0400, Peter King via talk wrote:
I desperately miss mutt. But the University of Toronto, in its administrative wisdom, moved us all to Microsoft365 which insists on token security of a kind mutt doesn't implement. If I ever leave or find a way to implement it, I'm back to using mutt like a shot.
Some people seem to claim to have it working, but of course there are different office365 setups so who knows. https://www.vanormondt.net/~peter/blog/2021-03-16-mutt-office365-mfa.html It does look rather complicated and not even sure how convinient it is to use. -- Len Sorensen
davmail is your answer. I use it to access work M365 and Seneca College M365, which both require multi factor authentication. On my ubuntu desktop/laptop I install the packages davmail and openjfx, and my ~/.davmail.properties includes (among others) davmail.mode=O365Interactive davmail.url=https://outlook.office365.com/EWS/Exchange.asmx at startup/login I run ( davmail -server > /dev/null 2>&1 & ) and my .muttrc has things like set imap_user="jsellens@example.com" set spoolfile=imap://localhost:1143/INBOX set smtp_url = "smtp://jsellens@example.com@localhost:1025/" set smtp_authenticators = "login" set ssl_force_tls = no Start up mutt, get prompted for password, get popup window asking for two factor (e.g. google authenticator, or duo push), comply, and there's my mail. And davmail is willing to cache credentials, so I don't always have to two factor. Note that this does not require IMAP access to M365 - it uses the "normal" exchange protocols, just like Outlook would. Feel free to let me know if you run into problems, and I'll try to help. Hope that's helpful! John On Thu, 2023/04/06 10:51:34AM -0400, Peter King via talk <talk@gtalug.org> wrote: | I desperately miss mutt. But the University of Toronto, in its | administrative wisdom, moved us all to Microsoft365 which insists on token | security of a kind mutt doesn't implement. If I ever leave or find a way to | implement it, I'm back to using mutt like a shot.
(Someone already replied to and addressed this, but since I'd already typed it up, gonna send it anyway.) o1bigtenor via talk wrote on 2023-04-05 12:34:
a great way to reduce the problem caused by a runaway var file was to use separate /var and /usr partitions (from / and /home).
That *may* help with preserving space on /, but now you have 2 more partitions to manage, and if /var runs out of space, you're still in trouble. Plus, if /usr runs out of space, there'll be trouble installing anything. Really, that just shifts the problem around without solving much. My 2
On Wed, Apr 05, 2023 at 06:24:22PM -0700, BCLUG via talk wrote:
(Someone already replied to and addressed this, but since I'd already typed it up, gonna send it anyway.)
o1bigtenor via talk wrote on 2023-04-05 12:34:
a great way to reduce the problem caused by a runaway var file was to use separate /var and /usr partitions (from / and /home).
That *may* help with preserving space on /, but now you have 2 more partitions to manage, and if /var runs out of space, you're still in trouble.
Plus, if /usr runs out of space, there'll be trouble installing anything.
Really, that just shifts the problem around without solving much.
Also most linux distributions no longer support /usr being a different partition than / so those days are done. And it wasn't even a good idea when it was first introduced, but well they had to do what they could with the disk size they had available. -- Len Sorensen
participants (10)
-
Anthony de Boer -
BCLUG -
D. Hugh Redelmeier -
Giles Orr -
John Sellens -
Kevin Cozens -
Lennart Sorensen -
Michael Galea -
o1bigtenor -
Peter King