cordump.com

I recently completed a project to upgrade our Nagios installation from 2.5 to 3.0b1 (Yes, I know thats a beta). As part of this upgrade I decide to include nagiosgraph to provide trending information via rrdtool.
Here is an example of an hourly cpu load graph from a test system

(continue reading…)

In Linux you can look in /proc/$PID , where $PID is your Process ID of course, to find out a great deal about currently running processes. It is laid out in a hierarchical format with file names that are fairly intuitive, Read on for an example from a running Nagios instance.

(continue reading…)

Most of the tutorials and howto’s on Nagios showing the use of the check_http plugin focus simply on determining if a page is returning a simple 200 OK code. This will tell you that your web server is up and returning pages, but for dynamic content this does not mean that it is displaying the correct content. However, add the “-R” flag to check_http and suddenly you are ensuring that specific content is available on the page as well.

define command{

command_name check_http

command_line $USER1$/check_http -H www.cordump.com

}

Example: check_http simple checking for a 200 OK

define command{

command_name check_http_regex

command_line $USER1$/check_http -H www.cordump.com -R “Dave Nash”

}

Example: adding -R to check for “Dave Nash” in the pages contents
So now I can check www.cordump.com to make sure it always has posts by me visible on the front page, and since I am the only poster that means it isn’t a blank page because I accidentally removed all my database entries.
The other nice thing about the -R switch is it is searching the html of the page so it can find data located in the html source.

Awstats is an oldie that I setup a while ago but it came to prominence again recently at work to give us an idea if some new features resulted in a drastic change in mail volume.
Provided you have a web server installed on your mail host Awstats is trivial following along with the instructions. However, I had a cluster of 2 load balanced mail hosts to check complicating the setup a little bit. So here is my setup
Click on the link below to read the complete description including complete configuration files.

(continue reading…)

I have been meaning to try out Nagios Event hander feature for a while but I got a little side tracked a little bit by my stalled quest to find a Nagios replacement. However, today I a need to implement this functionality to support a immature application without requiring regular System Admin intervention.
In Nagios an Event Handler is simply a script or command that gets called whenever any event occurs such as a service changing state to any of the following OK,UNKNOWN,WARNING or CRITICAL. However, your script will need to be responsible for ensuring if it is appropriate to run since you would not want to run a restart script when a service changes to the OK state.
I need to have a script perform an application restart when the service went into a warning state to resolve the issue before it went critical and generated a pager alert. Starting with the Nagios Documentation I created a new event handler.

1. Ensure that enable_event_handlers=1 is set in the nagios.cfg
   
2. Create a script to perform in case of the event see the Nagios Documentation for an example. One item of Note is that $STATETYPE$ is $SERVICESTATETYPE$ in Nagios 2.0 and above.
   
3. Define a check command in checkcommand.cfg

   define command{
   command_name restart-httpd
   # wrap it in check_by_ssh to restart the service on a remote server
   command_line $USER1$/libexec/check_by_ssh $HOSTNAME$ -t 10 -c “$USER1$/libexec/eventhandlers/restart-httpd $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$”
   }

4. add an event handler line to the service check

   define service{
   use generic-service ; Name of service template to use
   hostgroup_name myWebServers
   service_description HTTP Check
   check_period 24×7
   max_check_attempts 3
   normal_check_interval 5
   retry_check_interval 1
   event_handler restart_http
   contact_groups Notify
   notification_interval 24×7
   notification_period 24×7
   notification_options w,u,c,r
   check_command check_http
   }

   
   
   

5. Make sure your Nagios user has the correct permissions to perform the action in the event script. In my case I have to give the Nagios user sudo access to the init script in order to restart the service.
   <\OL>
   With these simple steps I was able to automate the restarts of an immature service which was failing on average of 5 times a week requiring System Administrators intervention to resolve. All and all a great investment of time!