Monitoring 15000 data inputs
Moderators: Developers, Moderators
Monitoring 15000 data inputs
Hi all,
I was wondering if this is possible at all and how would you approach it:
I have a few servers handling phone calls. Each server has 1000-2000 phone numbers assigned to them (all together more than 15000 numbers to monitor). I need to monitor the numbers being called, the thold plugin baseline feature would be great for that. (It could set off an alarm if the number of calls to a certain number drops by x% compared to last week's data). I only need to keep 2 weeks data, but the number of data input is scary
Can all this data stored in one rra file (one rra file per server would be the best)? Is there a way to script the addition of data outputs fields for this?
How much storage would it take?
Thanks for your help and suggestions,
Peter
I was wondering if this is possible at all and how would you approach it:
I have a few servers handling phone calls. Each server has 1000-2000 phone numbers assigned to them (all together more than 15000 numbers to monitor). I need to monitor the numbers being called, the thold plugin baseline feature would be great for that. (It could set off an alarm if the number of calls to a certain number drops by x% compared to last week's data). I only need to keep 2 weeks data, but the number of data input is scary
Can all this data stored in one rra file (one rra file per server would be the best)? Is there a way to script the addition of data outputs fields for this?
How much storage would it take?
Thanks for your help and suggestions,
Peter
- TheWitness
- Developer
- Posts: 17062
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
That is not a good idea. 15k data sources is nothing. There are many many larger systems out there.
TheWitness
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
- TheWitness
- Developer
- Posts: 17062
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
I would have to have more information. In addition, what is bad is one large RRDfile with all those data sources. It's not flexible. If you delete one, or add one, you start all over. Plus, cacti is designed for smaller poller output results, although you can make it pretty big.
It's just bad.
TheWitness
It's just bad.
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Hmm.. I see, good point. Okay, so then probably I'll need to have an RRD file for each phone number, since these are getting added/removed all the time. The main goal is to be able to alert based on a comparison to last week's data. Ie: I've received 200 calls to one of the numbers last week same day same hour, if this week it is 20% less, then I have to trigger an alert. (well I can get the system to make a test call, and if that fails, then comes the alert)
I can poll the call volume information from a web service, how many calls were received by which number, so I can script that.
I can poll the call volume information from a web service, how many calls were received by which number, so I can script that.
- TheWitness
- Developer
- Posts: 17062
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
I would personally write a plugin to poll the information, store it in a database, and keep things like averages, dropped calls, voice quality, whatever for each phone number in that table and then graph whatever you want from the database rather than polling the switch using spine or anything else. Just poll the database. This will prove to be light years faster.
Then, 100k would not be a problem either, or even 200k... However, you have to be careful with the database design.
TheWitness
Then, 100k would not be a problem either, or even 200k... However, you have to be careful with the database design.
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Thanks Witness, it's pretty much how you've described it. I have all call information going to a SQL database already, the developer said he would dump the result of my query on a webpage hourly. So I have to grab the info from there and graph it and set up the testing/alarming function. Maybe I wouldn't even need rrd files since he can dump last week's data too, but I'm not sure if&how cacti can handle the data that way.
- TheWitness
- Developer
- Posts: 17062
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
Your script would use wget or curl, and store the results for THOLD for example. I would do it with the script_server though, just to speed it up. Going to the database would be faster, but your developer may be afraid of 15k * Data Sources queries every 5 minutes.
TheWitness
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Who is online
Users browsing this forum: No registered users and 1 guest