Splunk is a popular Linux web application that gives IT administrators a birds-eye view of their log files, or more appropriately, a bees-eye view. Not only will it index and chart log file events in a beautifully rendered web format, but it also allows administrators to share their logging information with each other; like the compound eyeball of a bug that renders thousands of images, the multiplicity of Splunk's intuitive database of log file events helps identify problems in a system that are evident in the log file.
You can find ways to get VMware logs into Splunk but up until now, getting VirtualCenter (VC) tasks and events into Splunk has been difficult. True, we can ship ESX logs to Splunk via syslog, but the VC logs are of equal importance. Imagine how quickly we could diagnose why a VMotion operation failed when we compare the error to thousands more like it? We now have the ability to ship VC logs to a syslog server, where they can conveniently be indexed and charted by Splunk. This article describes how to install and configure a Splunk server with the purpose of monitoring and understanding the activities of VirtualCenter.
Splunk (current release is 3.13) is available in both the free and enterprise versions There are some differences between free and enterprise versions of splunk but for the purposes of this article and most small-to-medium businesses, the free version of Splunk fits the bill. Be sure to download the architecture appropriate version for your OS.
I am running Ubuntu Linux 7.04 Feisty Fawn x86_64, so I will be using the 64-bit version of Splunk. The download site even includes a nifty wget link for those users downloading the file to a headless console (like me). But Splunk is Linux/UNIX only and not yet available for Windows (sorry, Windows users).
We detail how to install the Splunk server from a tarball since not everyone has an RPM-based system. Once the tarball is downloaded, unpack it and then rename the directory so that it has a suffix with the Splunk version number. For example:
mv splunk splunk-3.1.3-28524-Linux-x86_64
This way, if an updated tarball is downloaded later, the original source will not be overwritten when the new tarball is unpacked.
Get the official Splunk installation documentation and copy the unpacked directory to /opt. For example:
sudo cp -R splunk-3.1.3-28524-Linux-x86_64 /opt
Because we are not installing Splunk in its default location, /opt/splunk, we have to edit an environment variable. Open /opt/splunk-3.1.3-28524-Linux x86_64/bin/setSplunkEnv with your favorite editor and edit this line:
so that it is
Congratulations, you've just successfully installed Splunk! The next step is to configure it so that it starts at boot. To do this, type the command:
sudo /opt/splunk-3.1.3-28524-Linux-x86_64/bin/splunk enable boot-start
Because this is the first time Splunk has started it will make you read its license agreement. Wade through that and then watch as Splunk is initialized and the boot scripts are created. To start splunk type:
sudo /etc/init.d/splunk start
A lot of text will follow, but the last line should resemble:
Splunk Server started. The web interface is at http://vault.lostcreations.local:8000 .
The address will have your own server's host name in it, of course, but notice that by default Splunk runs on port 8000.
It is time to configure Splunk. Go ahead and open the Splunk application in your web browser. The first time you do this, you will see a page that looks like this:
Since we did not use an enterprise license go ahead and click on "No thanks. Continue to use the Free license." The first step is to add a Splunk data input that will listen to the logs produced by vmmon. Before we add a data input, let's sanitize the logs by directing all vmmon syslog information to one log file. Open /etc/syslog.conf in your favorite editor and add this text:
# Reserve local7 for vmmon
That bit o' text will direct all levels of logs with a facility of local7 to the file /var/log/vmmon.log (the "-" prefix improves performance by not synchronizing the log file with every new log entry). Change local7 to whatever facility you chose for vmmon. Now restart syslog by typing:
You are now directing vmmon logs to /var/log/vmmon.log. If this does not work, then you may need to read my blog post about syslog facility levels with regards to vmmon.
The next step is to set up a data input to tail the log file /var/log/vmmon.log. In the Splunk web application click on the tab "Data Inputs" and then on the sub-menu item "Files & Directories". Click on the "Add Input" button on the left-hand side of the browser. You will see a page come up that resembles the following image. Fill out the data as such:
Click the "Add" button. Now you are trailing the vmmon log file. To see if any new data has popped up, click on the Splunk image in the upper left-hand corner of the browser to take you back to your Splunk dashboard. You may notice that there appears to be vmmon events, but they are not showing in the graph.
This is because these events are considered small data. To view these events, click on the log file "/var/log/vmmon.log". If you have events you will see them now. My screen looks like this:
Now that we're capturing events, how do we determine when something happens?
Splunking it up
Now that we're on our way to indexing events, what can we do with it? A lot, actually. Suppose you want to find out when the most VMotion operations are occurring. Simply enter the following search string into Splunk's search field:
source="/var/log/vmmon.log" vmmon-task "name=migratevm_task"
You will get a list of VMotion tasks. But suppose you just want to see which task occurs the most? Simply type:
source="/var/log/vmmon.log" | top name
And you will see an image similar to the following:
As you can see, Splunking with VMware and syslog just makes good sense. I hope that I've opened your eyes to what I consider one of the best pieces of software available today, Splunk, and how it can be used to understand your virtual infrastructure better than you ever could have without it.
About the author: Andrew Kutz is a Microsoft Certified Solutions Developer (MCSD) and a SANS/GIAC Certified Windows Security Administrator (GCWN). An avid fan of .NET, open source, Terminal Services, and coding, Andrew's current focus is on virtualization. Andrew graduated from the University of Texas at Austin with a B.A. in Ancient History and Classical Civilization and currently lives in Austin, Texas with his wife Mandy and their two puppies, Lucy and CJ.
This was first published in January 2008