On Windows, performance reports are gained with a simple, built-in tool like the System Monitor, or with an expensive, highly engineered diagnostic package from a vendor. If your shop has just migrated to Linux, you'll find the same situation: The built-in tools are always available from the command line; the highly engineered diagnostic packages are still expensive. Let's take look at some simple Linux tools.
Most Linux tools have "stat" as the last four letters of their name, "stat" being short for "status." Some of the most commonly used performance reporting commands are:
vmstat, iostat, netstat, nfsstat, pqstat, lpstat and
mailstat. There is, however, a second set of stat commands:
stat, fstat, statfs and
fstatfs. These other tools analyze files and filesystems on the disk. They aren't directly associated with performance.
Of the directly relevant tools,
vmstat is the first port of call. The figures it reports expand on figures that appear at the top of the "top" command's output. The
vmstat tool gives a server-wide view of performance. Supply it with two arguments, like this:
vmstat 5 3
vmstat spends five seconds accumulating data and reports a single line of performance statistics. It does this three times. At least five seconds is good for accuracy, because it means the average reported is based on a reasonable amount of data
You'll find that
vmstat reports system-wide process use, swapping, memory use, disk I/O and CPU use. If the server is slow, but you can still login, then
vmstat tells you immediately if the CPU or memory is the current bottleneck. If it's the CPU, then use "top" to find the guilty process.
vmstat output requires some appreciation of Unix architecture. Linux processes can be blocked (waiting for input), running (one per CPU at most) or runnable. If there are lots of runnable processes, (the
Procs r vmstat column), then that means there is more to do than the CPU can manage. If runnable processes hover much over 1 or 2, a faster computer will speed up the server significantly.
Linux memory, on the other hand, should always be in short supply. If there is initially a lot of free memory, then the I/O buffer and cache will expand to use most of it. One shouldn't panic, therefore, if the amount of free memory drops shortly after booting. The buffer and cache memory pools will hand back some memory if new processes demand their share. Only when all amounts are reported to be small is there a potential memory issue.
Swap is also part of the virtual memory system. It is designed to allow more processes to run than can fit in conventional memory. The overflow is temporarily saved to disk ("swapped out") until needed. There are two kinds of disk activity, "paging" and "swapping." A small amount of paging is safe. Linux never swaps (unless you have a very unusual flavor of Linux), and in any case, if it did, performance would be disastrously slow. In a well-tuned server, where standard, stable processes are running, paging should be minimal as well. If
vmstat reports sudden paging, then either someone's running a report or the commands you yourself type are the cause. If paging activity happens all the time, buy more RAM.
vmstat by itself presents plenty to explore. It's a first test that your server is performing well. Test the other "stat" commands at your leisure -- they're all harmless.
netstat are vital for analysis of per-disk and per-network performance, respectively. The remainder let you peek at the performance of application-specific features of Linux like printing and mail.
This was first published in February 2004