System Monitor / sysmon

I have sysmon on my Linux systems. It is a nice graphical tool to show CPU, memory, and network loads. It produces strip charts. (It also does other things, but that's not my current concern.) When I want to figure out what's taking so long, sysmon is a handy tool. But I think that my system's bottleneck is often disk I/O. sysmon has no strip chart for that. I don't even know how it should be done: - each device should probably have a different trace, just like each processor core does - percentage would be ideal but I don't know what that would mean. Do you count I/O ops? Queue length? Processes waiting? + sequential I/O is much faster than random I/O (at least on a real disk). + Large operations are somewhat slower than small ones (generally speaking). + output is generally non-blocking (roughly: the write goes into the kernel's memory, queued for writing to the device, and the process proceeds without waiting to the write to actually complete) + input usually blocks: the process issues a read and waits until the read finishes before it proceeds + Perhaps the simplest metric might be the number of processes waiting for disk I/O completion. But that leaves writes unaccounted for. Do you have any suggestions for a simple (possibly simplistic) tool to show disk bottlenecks in real time? PS: On the system I'm looking at, in the sysmon pane for "Processes", there is a column for "Disk read total" and another for "Disk write total". After a day of running the Libreswan test suite, gnome-terminal server wins with 3.4GiB of read and 3.9GiB of write. I wonder what that's all about. I would not expect Gnome Terminal to do disk I/O. As it happens, essentially all the output to the terminal is being captured by a script(1) command (run within the gnome terminal session). It shows 11.4MiB of disk output for one complete run, which should be half or a third of all that has gone to Gnome Terminal because I've run the suite two or three times since logging in. But it is more like a 11.4/3400 or roughly 1/300. Guess: gnome terminal reads and writes to X count as disk I/O. That's pretty messed up.

On 2018-08-19 11:53 AM, D. Hugh Redelmeier via talk wrote:
But I think that my system's bottleneck is often disk I/O. sysmon has no strip chart for that. [snip] Do you have any suggestions for a simple (possibly simplistic) tool to show disk bottlenecks in real time?
A quick internet search suggests you should look at the sar command that is part of sysstat. -- Cheers! Kevin. http://www.ve3syb.ca/ | "Nerds make the shiny things that https://www.patreon.com/KevinCozens | distract the mouth-breathers, and | that's why we're powerful" Owner of Elecraft K2 #2172 | #include <disclaimer/favourite> | --Chris Hardwick

On the sar front, you watch out for avgqu-sz (and/or await and svctm) suddenly increasing as tps increases. The queue length is like the "load factor", where <= 1 is good, 4 is bad, and more is evil. --dave On 2018-08-20 12:18 a.m., Kevin Cozens via talk wrote:
On 2018-08-19 11:53 AM, D. Hugh Redelmeier via talk wrote:
But I think that my system's bottleneck is often disk I/O. sysmon has no strip chart for that. [snip] Do you have any suggestions for a simple (possibly simplistic) tool to show disk bottlenecks in real time?
A quick internet search suggests you should look at the sar command that is part of sysstat.
-- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain

| From: Kevin Cozens via talk <talk@gtalug.org> | A quick internet search suggests you should look at the sar command that is | part of sysstat. Thanks for the research! I want a stupid GUI program. sar is a command-line tool, I think. sar's manual makes it look very complex to use. Not really inviting. If I just act dumb and type "sar", I find it isn't installed (but the shell offers to install it). When I say yes, it is installed, but the shell forgets that I want to run it. When I again ask to run it, I get: Cannot open /var/log/sa/sa20: No such file or directory Please check if data collecting is enabled All-in-all, not as brainless as running sysmon.

On Mon, Aug 20, 2018, 6:10 PM D. Hugh Redelmeier via talk <talk@gtalug.org> wrote:
Thanks for the research!
I want a stupid GUI program. sar is a command-line tool, I think.
I do recall sar... Definitely one of the classic tools for analyzing disk activity. It's probably worth some learning curve to attempt it. I have never gotten terribly deep with any of these tools. If there's something modern and either A) brainless easy, or B) deeply better That would be awesome to know of. sar and iostat are the ones I always remember. I seem to recall sar being a bit risky to use as it might induce a fair bit of I/O load itself.
participants (5)
-
Anthony de Boer
-
Christopher Browne
-
D. Hugh Redelmeier
-
David Collier-Brown
-
Kevin Cozens