[solved] querying my DNS server thousands of time per minute

[paulexander]

Very odd problem: I appears that Cacti is querying my DNS server over 4600 times per minute

I am in the middle of retiring a couple of old DNS servers, and so I ran a trace to see who was still pointed to the old address. I took an arbitrary sample, and there were over 4600 queries in one minute from Cacti.

From that sample, I got a list of the 51 hosts it was querying. This was my analysis of 1 minute of DNS activity from Cacti:
(I also noticed that the majority of the queries are multiples of either 14 or 16. My sample timing was probably not perfect with whatever query rotation is in place, so some were odd numbers..)

3)254(1)2(1)0(2)10(7)in-addr(4)arpa(0) 1823 queries
(1)7(3)18(3)16(2)10(7)in-addr(4)arpa(0) 1604 queries
(9)server04(1)(4)domain(5)state(2)xy(2)us(0) 28 queries
(9)server03(1)(4)domain(5)state(2)xy(2)us(0) 28 queries
(9)server02(1)(4)domain(5)state(2)xy(2)us(0) 28 queries
(9)server01(1)(4)domain(5)state(2)xy(2)us(0) 28 queries
(9)appserver07(1)(4)domain(5)state(2)xy(2)us(0) 32 queries
(9)appserver06(1)(4)domain(5)state(2)xy(2)us(0) 32 queries
(9)appserver05(1)(4)domain(5)state(2)xy(2)us(0) 32 queries
Etc.., through 51

(all the rest were no more than 44 queries)

I am running Cacti Version 0.8.7d, on CentOS.

Why in the world is it doing this?

[gandalf]

I'm not sure that cacti is querying dns that often. We perform a lot of snmp request, nevertheless. They in turn may issue DNS request. A guess ...
R.

[paulexander]

I think I figured it out, for the most part at least. I have a solution to take the pressure off my DNS server for the time being.

I have not dug very deep into the Cacti config, but from what I can tell, whoever built this system has a pretty rigorous rotation going on; lots of UDP pings, etc. It seems that the system is doing a DNS lookup on EVERY single system that Cacti is collecting data from.

SO, I looked on the server, and sure enough, there was no DNS caching set up. I read up on dnsmasq, configured it, turned it on, and thankfully every DNS lookup now looks to the local to the machine first.

I edited /etc/resolv.conf to include the localhost on the top line:
nameserver 127.0.0.1

I left the original DNS servers in the lines below that.

Then, edited the /etc/dnsmasq.conf file, to configure dnsmasq to look at all traffic on the interface (un-commented line)
interface=eth0

Restarted dnsmasq (/etc/init.d/dnsmasq restart), restarted the network (okay, I rebooted), and badda bing, everything is better.

More info: http://wiki.debian.org/HowTo/dnsmasq (even though I am on CentOS, this one laid things out well)