Sunday, August 24, 2008

Building Performance Monitoring Solution with Nagios and NDOUtils

In my previous post i talked about Application Performance Management - APM tools and discussed possible requirements of it. I also talked about Nagios being wonderful and proven monitoring and solution. In this post i will show how to build monitoring solution with Nagios comparable to any commercial monitoring system.

We will be using following components along with Nagios.

  • NDOUtils - Storing runtime information to database

  • NRPE - Remote Plugin Agent

  • NSCA - Remote Plugin with Passive Checks

  • Nagvis - Visualization Addon

  • Nagios Business Process Addons - Custom Bird's Eye View of Business Application

  • NSClient++ - For monitoring Windows Hosts

Tough part is that these are all discrete components requires good knowledge. This series will focus on all these system and build good monitoring solution. This part will be focusing on NDOUtils and performance data. Once complete I will release the package with installation script so that it will be easy for installation.

I assume you have nagios installed and working. If not visit quick-start guides Ubuntu, Fedora, OpenSUSE. They work perfectly. Go through nagios console and explore what nagios provide out of the box.

Lets first see how to setup nagios and NDOUtils. NDOUtils is an implementation of Nagios Event Broker module(NEB). NEB has shared object which is loaded by nagios and can register for listening events. NDOUtils passes this event to file or socket. Second part of NDOUtils is C-daemon which saves the information to database.

Download ndoutils tar from nagios site. When you try to compile it you may face problem with mysql library. On Ubuntu first install mysql-dev library. Off course you have to have mysql server installed first.

sudo apt-get install libmysqlclient15-dev


Start mysql client and login as root. Create Database and nagios user.

create database nagios;
use nagios;
grant all on nagios.* to 'nagios'@'' identified by 'nagios'
grant all on nagios.* to 'nagios'@'localhost' identified by 'nagios'

Go to db directory and create necessary tables.

./installdb -u nagios -p nagios -h localhost -d nagios


Now go ndoutil deirctory and compile the source

./configure --with-mysql-lib=/usr/lib/mysql


Copy binaries to nagios. This assumes that you have installed nagios in /opt/nagios (Defualt is /usr/local/nagios)

cp ndo2db-3x /opt/nagios/bin
cp ndomod-3x.o /opt/nagios/bin

Also copy config files - ndomod.cfg and ndo2db.cfg from config directory to /opt/nagios/etc
Modify Following lines.

ndomod.cfg
output_type=tcpsocket
output=127.0.0.1
buffer_file=/opt/nagios/var/ndomod.tmp

ndo2db.cfg
socket_type=tcp
db_name=nagios
db_user=nagios
db_pass=nagios

Now open nagios.cfg file and modify following keys

event_broker_options=-1
broker_module=/opt/nagios/bin/ndomod-3x.o config_file=/opt/nagios/etc/ndomod.cfg

Now start ndo2db daemon before restarting nagios

/opt/nagios/bin/ndo2db-3x -c /opt/nagios/etc/ndo2db.cfg

Restart Nagios

/opt/nagios/bin/nagios -d /opt/nagios/etc/nagios.cfg

Now start mysql client and check nagios_hosts and nagios_services table to see that data is getting saved.

You can put following lines /etc/rc.local so that all events fired by Nagios ( which is started from /etc/init.d/nagios script) are picked up and processed. This makes everything started when machine boots up.

/opt/nagios/bin/ndo2db-3x -c /opt/nagios/etc/ndo2db.cfg


Once this is done all your data is saved to database. Now we can build our Performance Parser on top of this. I will come-up with shell or perl script for all installation once complete.

..
Tushar

8 comments:

Alan Frye said...

I am trying to install ndoutils on openSuse. I noticed you install libmysqlclient15-dev.
I am running mysql 5.0.45 and I have mysqlclient-dev. I am having an issue starting ndo2db-3x process, it does not recognize mysql as being instlled

Tushar Khairnar said...

Whats the message you get?

I am java developer so I may not be correct but following are the issues i feel could be cause it to fail.

1. You are facing problem at installation time or starting? If it fails during compilation itself then use -l option when running ./configure script. See the message you get. It shows that mysql lib has been detected or not. You can try with different path if different for your system

2. After installing mysqldevlib you may have to do ldconfig again to reflect the changes. Most of the time its done post-installation itself

3. correct version of client lib

hope this helps.

regards,
Tushar

Unknown said...

HI,

I have followd your how-to.
but it works not fine.

i get this message:
~/downloads/npc/ndoutils-1.4b7/src# /usr/local/nagios/bin/ndo2db-3x -c /usr/local/nagios/etc/ndo2db.cfg

Support for the specified database server is either not yet supported, or was not found on your system.

but mysql is installed.
i can logon on phpmyadmin whith the nagios user and password.
the files are in /usr/local/nagios/

i use ubuntu

Anonymous said...

Hi,
The document is really nice and explainatory. I followed the whole document but got stuck in last point. When I try to run /usr/local/nagios/bin/ndo2db-3x -c /usr/local/nagios/etc/ndo2db.cfg command, it gives me an error "Support for the specified database server is either not yet supported, or was not found on your system."

I have compiled ndoutils with /usr/lib64/mysql only but am confused what to do now.

Please suggest something as I am really stuck.

Rgds
GS

Anonymous said...

make sure you don't have multiple version of mysql on your system. in Centos, run 'yum list installed | grep mysql'

Unknown said...

Hi,
I have installed Nagios 3+ and working fine. I wanted to install Nagvis, so I started installing NDO Utils, as per your directions, at last it is giving me the error to start the service
The error is
root@excel-desktop:~# /usr/local/nagios/bin/ndo2db-3x -c /usr/local/nagios/etc/ndo2db.cfg
Error processing config file '/usr/local/nagios/etc/ndo2db.cfg'.
I am not understanding the error in the config file....
please help

when I start Nagios I get this message in the log
Jun 22 11:59:06 excel-desktop nagios: ndomod: NDOMOD 1.4b7 (10-31-2007) Copyright (c) 2005-2007 Ethan Galstad (nagios@nagios.org)
Jun 22 11:59:06 excel-desktop nagios: ndomod: Could not open data sink! I'll keep trying, but some output may get lost...
Jun 22 11:59:06 excel-desktop nagios: Event broker module '/usr/local/nagios/bin/ndomod-3x.o' initialized successfully.
Jun 22 11:59:06 excel-desktop nagios: Finished daemonizing... (New PID=29121)
Jun 22 12:02:08 excel-desktop nagios: Caught SIGTERM, shutting down...
Jun 22 12:02:08 excel-desktop nagios: Successfully shutdown... (PID=29121)
Jun 22 12:02:08 excel-desktop nagios: ndomod: Shutdown complete.
Jun 22 12:02:08 excel-desktop nagios: Event broker module '/usr/local/nagios/bin/ndomod-3x.o' deinitialized successfully.


Please help

ravis007 said...

Hi,

I have installed & configured ndoutils & its working fine. Nagios data is stored into a database.
Now I want to configure Nagvis, I have installed it,please tell me the steps to configure it.

regards
Ravi

Tushar Khairnar said...

@Ravi,

Unfortunately I am totally out of touch right now from Nagios and other frameworks thats why I have not written next part of the series.

Regards,
Tushar