Building Performance Monitoring Solution with Nagios and NDOUtils Part 2 PerfNagios

Welcome to part 2 of this series. This took long time but was worth the effort. In this part i am going to describe how to save performance data and integrate a sample monitoring scripts for performance monitoring. In earlier post I wrote about how to install and configure NDOUtils for database support in nagios. We will be using the same setup for storing data but will add some more table to store performance repository. Remember NDOUtils will delete data periodically!.

Lets first discuss how would you save performance data.

PerfNagios

is sourceforge project started by me for implementing this idea. Currently only parsing code is stable, reporting and dashboard functionalities will be added later. Stay connected on this blog for future updates to this project.

PerfNagios is basically small web-interface for displaying nagios related data easily. Eventually it will have good reporting capabilities. Currently it only shows performance data for last 1000 readings. You can see screenshots below.




Following are tables i used for storing performance data.


CREATE TABLE `nagios_metrics` (
`metric_id` int(11) NOT NULL auto_increment,
`instance_id` smallint(6) NOT NULL ,
`host_object_id` smallint(6) NOT NULL ,
`service_object_id` smallint(6) ,
`unit` varchar(60),
`label` varchar(255),
PRIMARY KEY (`metric_id`)
) ENGINE=InnoDB;


CREATE TABLE `nagios_metric_data` (
`metric_data_id` int(11) NOT NULL auto_increment,
`metric_id` int(11) NOT NULL,
`value` double not null,
`warn` double,
`critical` double,
`min` double,
`max` double,
`date` datetime not null,
PRIMARY KEY (`metric_data_id`, `metric_id`)
) ENGINE=InnoDB;


CREATE TABLE `nagios_perf_batches` (
`date` datetime not null,
`last_service_check_id` int(11) not null,
`last_host_check_id` int(11) not null,
`host_checks` int(11) not null,
`service_check` int(11) not null
) ENGINE=InnoDB;


Lets take an example say we need to keep track of how cpu is getting used. For that we need to add service call CPU to monitoring host. Below is small script which outputs important cpu performance metrics : CPU Run length, User, System and Wait-on I/O. Output is like :

CPU : OK 0 | rl=0;2;5 us=2;85;85 sys=0;10;20 wa=0;5;10 total=3;80;90

• check_java.sh


value=`vmstat 3 3 | awk -f /opt/nagios/libexec/ubuntu-sys-plugins/cpu.awk`
returnvalue=`echo $value | awk '{print $4}'`
echo $value;
exit $returnvalue;


• and cpu.awk


{
#print $1,$13,$14,$15,$16;
if(NR>=5)
{
rl=rl + $1;
us= us+ $13;
sys = sys + $14;
wa = wa + $16;
}
#print NR,"--->", $0;
}
END {

rl = rl /2;
us= us/2;
sys = sys/2;
wa = wa /2;
total = us + sys + wa;

#print "Total is ", total;

if(total <= 75) { msg = "CPU : OK"; returnvalue=0; } if(toal > 75 && total <= 85) { msg = "CPU : WARINING "; returnvalue = 1; } else if(total > 85)
{
msg = "CPU : CRITICAL";
returnvalue = 2;
}
msg = sprintf("%s %d | rl=%d;2;5 us=%d;85;85 sys=%d;10;20 wa=%d;5;10 total=%d;80;90",msg,returnvalue,rl,us,sys,wa,total);
print msg;
}


• To include this monitoring script you need to add service for localhost. In nagios 3.x configuration is based on templetes. For defining we need to first add check command in file /opt/nagios/etc/objects/commands.cfg


# 'check_cpu command definition - tushar
define command{
command_name check_cpux
command_line /opt/nagios/libexec/ubuntu-sys-plugins/check_cpu.sh
}


• Once this is done you need to add service definition in /opt/nagios/etc/objects/localhost.cfg.


define service{
use local-service ; Name of service template to use
host_name localhost
service_description CPU
check_command check_cpux
notifications_enabled 0
}






Restart the Nagios by /etc/init.d/nagios restart and done! You system is now able to monitor cpu information.

Below is the graph drawn in PerfNagios for the same script.

Building Performance Monitoring Solution with Nagios and NDOUtils

In my previous post i talked about Application Performance Management - APM tools and discussed possible requirements of it. I also talked about Nagios being wonderful and proven monitoring and solution. In this post i will show how to build monitoring solution with Nagios comparable to any commercial monitoring system.

We will be using following components along with Nagios.

• NDOUtils - Storing runtime information to database


• NRPE - Remote Plugin Agent


• NSCA - Remote Plugin with Passive Checks


• Nagvis - Visualization Addon


• Nagios Business Process Addons - Custom Bird's Eye View of Business Application


• NSClient++ - For monitoring Windows Hosts

Tough part is that these are all discrete components requires good knowledge. This series will focus on all these system and build good monitoring solution. This part will be focusing on NDOUtils and performance data. Once complete I will release the package with installation script so that it will be easy for installation.

I assume you have nagios installed and working. If not visit quick-start guides Ubuntu, Fedora, OpenSUSE. They work perfectly. Go through nagios console and explore what nagios provide out of the box.

Lets first see how to setup nagios and NDOUtils. NDOUtils is an implementation of Nagios Event Broker module(NEB). NEB has shared object which is loaded by nagios and can register for listening events. NDOUtils passes this event to file or socket. Second part of NDOUtils is C-daemon which saves the information to database.

Download ndoutils tar from nagios site. When you try to compile it you may face problem with mysql library. On Ubuntu first install mysql-dev library. Off course you have to have mysql server installed first.

sudo apt-get install libmysqlclient15-dev

Start mysql client and login as root. Create Database and nagios user.


create database nagios;
use nagios;
grant all on nagios.* to 'nagios'@'' identified by 'nagios'
grant all on nagios.* to 'nagios'@'localhost' identified by 'nagios'


Go to db directory and create necessary tables.

./installdb -u nagios -p nagios -h localhost -d nagios

Now go ndoutil deirctory and compile the source

./configure --with-mysql-lib=/usr/lib/mysql

Copy binaries to nagios. This assumes that you have installed nagios in /opt/nagios (Defualt is /usr/local/nagios)


cp ndo2db-3x /opt/nagios/bin
cp ndomod-3x.o /opt/nagios/bin


Also copy config files - ndomod.cfg and ndo2db.cfg from config directory to /opt/nagios/etc
Modify Following lines.

ndomod.cfg

output_type=tcpsocket
output=127.0.0.1
buffer_file=/opt/nagios/var/ndomod.tmp


ndo2db.cfg

socket_type=tcp
db_name=nagios
db_user=nagios
db_pass=nagios


Now open nagios.cfg file and modify following keys


event_broker_options=-1
broker_module=/opt/nagios/bin/ndomod-3x.o config_file=/opt/nagios/etc/ndomod.cfg


Now start ndo2db daemon before restarting nagios


/opt/nagios/bin/ndo2db-3x -c /opt/nagios/etc/ndo2db.cfg


Restart Nagios


/opt/nagios/bin/nagios -d /opt/nagios/etc/nagios.cfg


Now start mysql client and check nagios_hosts and nagios_services table to see that data is getting saved.

You can put following lines /etc/rc.local so that all events fired by Nagios ( which is started from /etc/init.d/nagios script) are picked up and processed. This makes everything started when machine boots up.

/opt/nagios/bin/ndo2db-3x -c /opt/nagios/etc/ndo2db.cfg

Once this is done all your data is saved to database. Now we can build our Performance Parser on top of this. I will come-up with shell or perl script for all installation once complete.

..
Tushar

Meet the People Who Have Trillions Riding on Linux this Fall

Hi,

Recently i cam across this link : Meet the People Who Have Trillions Riding on Linux this Fall. This shows how deep linux has penetrated. Linux is handling trillions of money. Wow!

My New Laptop - Acer 2920

Hi, weeks before I bought new laptop for me Acer 2920. Earlier I had blogged about my journey with linux earlier. Linux was mandatory criteria for my laptop. So i had experimented a lot with different laptop reviewed them on internet. My brother's Laptop : Sony VAIO CR12GB/H had responded well to linux call ... it was so good that my brother is now linux convert uses Ubuntu prefarably (See snapshot Below).


Other criteria being : Windows XP. I feel Vista is a big flop specially for countries like india where hardware here is atleast 6-7 months behind latest technology. In very first post on this blog I mentioned my disappointment in following words

"And last thing i want to say about VISTA is that its fooling people around. It takes seconds to load dekstop and pretend ready for action but reality is different. It take much more time to get all networking started and then i can start my browser. With linux it takes time to login but I know when its now ready - ready for anything file explorer, browser and mp3. "



But the market here in India is not that good. You don't get Windows XP laptop with good hardware these days. So I had to go cost mode .. laptops which are targeted as utility laptops : I settled for Lenovo and then i saw this beauty in Vijay Sales Thane Shop. Sales Guy was really friendly and allowed to boot the laptop with ubuntu live CD. It passed the test. Acer 2920 is the smallest laptop around with no compromise on hardware or any necessary features. It comes with driver CD for both Vista and XP which is not common in latest laptops which restrict you to vista only.






so if you are looking for good utility laptop : Acer is good brand and certainly best one for linux laptop. I recommend Sony also : Sony is also good value for money. Sony laptops have good sound quality : loud and clear, clear crisp bright display. VAIO CR series certainly suited for the most.

..
Tushar

My Linux Journey

My Linux Journey

My linux journey started in my engineering days when i got 3 cds of redhat 9.0. I did not have internet connection and only source was friends and computer magazines. I was very much impressed with it first i installed it but then i did not have anything to do with it i had Photoshop, Flash, Dreamweaver and Frontpage in windows and Age Of Empires too!!. I installed software called System Commander which was sort of protector for my system. At that time i did not have any guts to play with GRUB. And no live CDs for recovery. It is boot loader and system software. One time i had 6-7 different OS on my system - Windows 98/NT (don't exactly remember but i am proud of the fact i had worked on all version of windows - 3.1,95,98,ME,NT4,XP,2000,2003 and now VISTA, funny thing is that all of them have crashed on my system - blue screen of death. Most delicate was ME and NT,2003 were rock solid XP being most practical), Windows XP, Red hat 9, Damn small linux, Minix, some linux distro that used to fit on floppy i had installed it on my harddisk thanks to System Commander. I used test new versions and flavor of linux Just for fun.

By the way Just For Fun is autobiography of Linux Torvalds - very funny book if you are geek/programmer you will definitely like it. i did.

But then after lot of crashes, partition formatting/reformatting,i settled on Windows XP which worked without any hassles. Occasionally i used to boot into linux again just for fun.

After i got my first salary i started upgrading my pc. I bought new 80GB harddisk so that i could have more space to store data for windows. I reserved 20 GB for linux and tried a new (for me till that time i was familiar with only redhat linux) and hot distro of that time - UBUNTU. I liked it so much that i replaced my fedora with ubuntu moving all my data to it. But i still thought XP is better than ubuntu.

For last 6 months my brother's laptop has been sort of laboratory for me. My old (infact very old) PIII 800Mhz box can not take it any more. I was very much excited to idea of virtualization. Till 6 months before i was not confident enough on linux desktop so i was using Windows XP and with virtualization i could run ubuntu right within windows - best of two worlds. I tried ubuntu 7.04, 7.10 8.04, Kububtu 7.10, gOs2 beta, Oracle Linux, and Fedora.

I have been successful in attempts to convert my borther in to a linux follower - thanks to the buggy and resource hungry Windows Vista. Who want that eye candy when you get similar( though not that good though) graphics effects on linux without any performance loss.

After lauch of Ubuntu 8.0 beta i thought i will give it a try for desktop linux. I used ubuntu for over a week without booting linux on my old box PIII 800Mhz 256MB RAM. Whenver i required any application i went to Add/Remove did a simple search and i could find and use ( without any crash - earlier also i had used linux but most of the time something used to crash) equivalent linux application. Only reason i had to boot into windows being to check my office applications - Site works only on Internet Explorer. I never tried Wine yet now i
need it.

Most suprising thing is that Ubuntu 8.0 worked out of the box on Sony VIAO CR12GH/B laptop. Ubuntu 7.10 has some difficulty recognising intel x3100 graphics accelerator hence screen resolution was limited to 800x600 ( i tweaked Xorg conf and it worked with resolution 1200x800 on ubuntu 7.10 as well).

Earlier whenever i worked on linux i knew something would go wrong and i accepted it now i dont have that feeling - Ubuntu ROCKS.

Till that moment i used to think linux is just not ready for desktop for normal user( i am half normal and half geek - i am passionate about linux but have grown in comforts of windows) ( there are plenty reasons for it - h/w is one major reason i feel). But last 2 years i have seen great growth in linux distros mainly Fedora, Suse and Ubuntu.

If you have good standard hardware on your box linux is READY for you and certainly is for Me and it will beat windows vista in user experience. Major reason for this fact is buggy and bulky VISTA. i get rating of only 3.0 out of 5.0 with very strong hardware for laptop. What the heck what do you want exactly VISTA? 5 GHz processor. 2 GB graphics card????

Here is my list:

Rythmbox
Window Media Player/Winamp - Windows media player takes around 50% CPU when playing high beat rate (more than 150 Kbps) mp3 Winamp and rythmbox take less cpu.

Totem/VLC
Movie player I love VLC. Its best player avaialable. Windows Media player really sucks here. I just cannot run any movie in my p3 box with version 10 onwards. VLC is much better both on windows and linux. Whats better than having your favoiurite application avaialable on both systems. Again windows application sucks here.

Bit-torrent
Microtorrent is very good but so is azureus and Transmission. No problems here too.

Browser
Firefox 3.0 rocks and blazingly fast on sony vaio laptop. I can not get same performance on Vista/IE or Vista/Firefox combination. Most developers say Firefox 3.0 is the FASTEST browser to date. I did not get any improvment on my p3 box (windows xp). But i could feel firefox more responsive on linux on both systems - XP(PIII) as well as Vista(Core 2 Duo T7100). Why is so?. I really dont know neither i don't believe in benchmark numbers ( i have done performance testing/benchmarking i know how misleading sometime they could be). As performance engineer i used to give importance on fact how end-users are experincing it. Did they find any improvement or felt good. With firefox 3.0 on linux i felt. On other note IE7 is not that bad too. But i am firefox convert. long time ago.

Programming
Editplus beats any lightweight editor in linux. I have not tried anything other that GEdit. Gedit is little bit heavy ( on P3). But i could find editor called Geany i liked it very much. I used Ecliplse for most of my Java programming and was disappointed a bit on linux. On thing about IDE i care is how it uses screen effectively to pack useful stuff. On linux SWT occupies much more space than on windows.

Office
Nothing to say. OpenOffice is really behind MS Office. but i have different opinion here. Dominance of MS Office is just because of numbers. What would you do if you have to write some document and mail it to somebody. I could find one alternative in OpenOffice, Convert that document to PDF and mail it. I just hate rebooting again to windows and edit that doc.

For other applications you can refer to this interesting links :

• Awesome Ubuntu/Linux Apps you might not know about
• Top Best 50 Ubuntu App for Designing

And last thing i want to say about VISTA is that its fooling people around. It takes seconds to load dekstop and pretend ready for action but reality is
different. It take much more time to get all networking started and then i can start my browser. With linux it takes time to login but i know when its
now ready - ready for anything file explorer, browser and mp3. In and all if you don't have any reason to stick to windows like you IDE works only in
windows then go and install ubuntu (or any linux) give it a try. Vista is just not ready for countries like india where latest hardware comes to
mainstream atleast 6-8 months after. Concepts like open source, linux, ubuntu, free document formats (openoffice), OLAP (one laptop per child) are very much suitable in developing countries. This will make sure that user gets what it need and hes not forced to buy first windows then Microsoft office and
then anti-virus then internet protection software to protect windows. its chain that's not needed and can certainly be avoided.

More or less computing is for fun as said by Linus Torvalds in his thoughts on life... so enjoy it. I certainly enjoyed it with linux.

Oracle on Ubuntu Hardy Haron

I had hard time installing oracle on my brother's sony vaio laptop. Not because of linux but because of windows i was out of space on ubuntu and wanted to shrink windows partition. i had strage expeprice with it. Windows does not let me to shring below 1 GB. Why? may be some technical difficuly. huh! then i started with ubuntu 8.0 live cd, opened Gparted i could resize my NTFS partition. Wow! why windows wouldnt allow then? Does it know that i had ubuntu installed on this laptop and user of this laptop dosent prefer windows. Only reson that i went to windows to resize is because vista is very particular of stratup files and my exprience with NTFS on linux is bad. lost lot of data once.
Within 2 hour gparted moved my 50 gb partition. freed 16 gb for linux and 2gb for swap.

Very first thing i noticed is that oracle doesnt support ubuntu as officially supported linux distribution but i could manange to find installation guide for debian. Its recommended to go with RHEL/SUSE Enterprise if you are using for commercial purpose.

Here are some of links for pre-setup tasks :

• http://linux.togaware.com/survivor/Oracle_10g.html
• http://www.akshaymehta.com/2006/12/10/installing-oracle-10g-r2-on-ubuntu-edgy/
• http://rossov.com/2006/02/08/oracle-10g-ubuntu-linux-vmware-part-ii/
• http://www.oracle.com/technology/pub/articles/smiley_10gdb_install.html

Even oracle's Installation Guide is also helpful.

Follow all the steps correctly. This actually helps you in case you come across some problem, Since you know what is installed where?
So here we go start terminal with oracle users

su oracle
./runInstaller -ignoreSysPrereqs

-ignoreSysPrereqs is flag that allows oracle to proceed without check. Alternative is to create file /etc/release and pretend that i am rhel.

Installlation went more or less smooth but i faced some problem which are faced by many people. I has faced similar problems on RHEL too!. Little bit of googling will solve most of them.

• Linking phase fails for 10gR2 on Ubuntu 6.06: undefined ref to 'nnfyboot'

Solution : Create following sysmlinks and relink
ln -s $ORACLE_HOME/lib/libclient10.a $ORACLE_HOME/lib/libagtsh.a
$ORACLE_HOME/bin/genagtsh $ORACLE_HOME/lib/libagtsh.so 1.0

This problme is discussed here and worked on ubuntu 8 too:

• Right version libstc++ library on my ubuntu 8.04 version was 6.x. So i created following symlink

libstdc++.so.6 -> libstdc++.so.1 ( Present)
libstdc++.so.5 -> libstdc++.so.1 ( Oracle tries to find it, so i added)

• "ORA-12547: TNS:lost contact" when creating database in Oracle 10g. Lib AIO Error This is very annoying.After installation oracle starts configuration assistant DBCA. After googling a bit around i found out libaio was required. In ubunntu it is called libaio1.
  

So do apt-get install libaio1


This problm is discussed in this thread for Fedora.

Most of the problems were in linking phase hence i did $ORACLE_HOME/bin/relink all again to make sure that all changes have been affected.

Now oracle is working fine on ubuntu 8.04 beta.