As a Linux administrator, you should already know the basics of performance tuning. You probably recognize classic Unix/Linux monitoring commands such as vmstat and top, while system profilers such as Oprofile and the GNOME System Monitor may be new to you. If you were born on Planet Unix or are rather new to Linux, you might be surprised to find that typing in iostat or sar will get you nothing, unless you download the sysstat suite of monitoring tools, which include support for sar, iostat and mpstat.
Performance tuning is not only about running some commands, but it is about proactively monitoring your system, particularly when there are no performance problems. Here we cover Linux performance tuning methodology and provide you with detailed steps that should help you throughout your tuning lifecycle. We'll introduce some of the monitoring tools you should use, provide you with an overview of performance tuning, and discuss considerations that can impact overall performance.
When investigating a performance problem, start by monitoring the CPU utilization statistics. It is important to continuously observe system performance because you need to compare loaded system data with normal usage data (i.e. the baseline). The CPU is one of the fastest components of the system. If CPU utilization keeps the CPU 80-90% busy, overall system-wide performance will be affected. In order to improve upon this, one should follow a careful method for system tuning.
Linux performance tuning methodology
Baseline
Before doing anything, you must establish a baseline. The baseline will serve as a reference point for your analysis, prior to any tuning of your system. The baseline itself should combine system monitoring reports, as well as a snapshot of the system hardware. When monitoring your system, try to use a minimum of two tools. This way, you'll be able to validate your analysis using multiple sets. Also, try to establish a Service Level Agreement (SLA) with your functional and business teams, which defines acceptable levels of performance for your system.
Stress test/monitor
Here is where you stress your system. You will be monitoring the system at a peak workload, prior to making any changes to your system. By doing this, you will be able to have before and after snapshots of your system when you have completed tuning your environment.
Identification of bottleneck
At this stage you will analyze the data you have collected to determine the location of any potential bottleneck. In doing so, you will make certain that you use several monitoring tools so that you have conclusive, corroborating evidence.
Tuning
It is time to fine tune your system for optimal performance. You will find that there are many different ways to tune a system, depending on the nature of the bottleneck (whether it is CPU, Memory, I/O or Network related.) It is very important that you ...
To continue reading for free, register below or login
To read more you must become a member of SearchEnterpriseLinux.com
make only one change at a time in order to determine which operation successfully resolved the problem. Often, tuning up a system involves more common sense than it does, say, kernel tinkering. For example, if you are running several demanding workloads at the same time each day, you may determine that scheduling them off-hours or during a batch cycle is the best solution. Repeat
Repeat all steps, starting with the stress test. Occasionally, you will find that by fixing one bottleneck you have created another. I have seen this many times, especially with CPU problems. By tuning your CPU you allow the system to work harder. In doing so, you may have inadvertently created a memory or I/O problem. This is not a bad thing, as you are now getting closer to resolving some of your performance woes.
One important point that cannot be understated is that you should always be monitoring your system. Performance tuning is a proactive process, which too often occurs in reactive mode only (i.e. when users start to scream and holler.)
System Reports
First we will take a snapshot of our system, to determine exactly what we are dealing with. Viewing the redhat-release file, running uname and runlevel commands gives us a nice preliminary view. The RPM that we will use is sysstat-4.0.7-4.rhl9.1.i386.rpm.
With these commands, we find out that we are using RHEL5 on a 2.6 kernel in runlevel 5. Another useful command is dmesg, though it gives you so much information (too much to show here), you should always save it to a file.
The proc directory is a place where one can gather all sorts of information directly from the kernel through the proc filesystem. Let's look at CPU and memory information.
For disk, we'll use the traditional df command:
The lsmod command will give you the list of installed kernel modules.
Perhaps my favorite snapshot utility is sysreport. It is perfect for archiving (it dumps the output into a tar archive) and/or sending off data to engineers for further analysis.
System Monitoring
My favorite monitoring command is vmstat, an old school Unix command line utility. I love it because it is a quick, easy way of looking at the overall heath of your system (RAM, I/O and CPU).
Other useful command line utilities are mpstat, sar and iostat. Mpstat and sar are used for CPU information, while iostat is used for I/O. Free is a nice utility which displays the total amount of physical RAM and swap on the system. It tells us what is used, free and shared.
A general purpose process monitoring tool is OProfile. What it does is use the performance monitoring hardware on the processor to retrieve kernel type system information about the executables on a system. To start Oprofile:
Then we save the data to the file:
Then we run the report:
Though it is true that some of this information can be ascertained using ps commands, I like using this because of its reporting capabilities and the extent of the data that it keeps.
No monitoring article would be complete without mention of top. Top is a nice character based tool that illustrates CPU, memory and process information. I like this tool when I'm doing performance monitoring over a given period of time and I need to look at many different facets of the system on one screen.
[IMAGE]Figure 1 – top
Finally, for those that prefer a graphical interface, the GNOME System Monitor is your tool. After getting X running, type in:
[IMAGE]
Figure 2- Resources view
There are three tabs for processes, filesystems and resources.
Tuning
Next, we'll fine tune the Virtual Memory Manager (WMM) of Linux. VMM manages the allocation of both RAM and virtual pages. Given the choice between RAM and paging space, the preference is to use physical memory, if the RAM is available. We will be working with sysctl, which is used to configure kernel parameters.
Let's pull out only the memory tunables by looking for tunables with the vm prefix:
Let's pick one of them: vm.swapiness, which defines the percentage of the kernel in physical memory which should swap memory into overall swap space. The lower one sets this, the more Linux will prefer to use physical RAM rather then swap. You should do this if you have plenty of RAM but too much paging. The default is 60, so we're going to change this to 20.
You can make your changes permanent by editing /etc/sysctl.conf.
Another tip I highly recommend is to run ckconfig.
This utility tells you what services are currently running:
Turn off any services that you are not using. You will find a lot more RAM available, as well as some additional CPU clock cycles.
And that is Red Hat Enterprise Linux 5 performance tuning, for now. It is important to note that there is a lot more to performance tuning than what we can detail in one small article but we covered the important parts: methodology, system snapshot and monitoring commands and basic tuning concepts. We defined the Virtual Memory Manager, looked at some kernel parameters and made tuning changes. I highly recommend that you have several environments for your critical systems. These can include development, QA and Production. System changes should never be made in a production environment without first testing them in a backed up system. Tuning methodology is important. Always adhere to one.
About the author: Ken Milberg is a systems consultant with two decades of experience working with Unix and Linux systems. He is a SearchEnterpriseLinux.com Ask the Experts advisor and columnist.
