Win32 - Monitor running processes

[BSOD2600]

So... what are the questions you have now that you've done a bit of digging into these metrics?

[argon0]

Really good question...

1) what is the this graph (well the cpu bit anyway) really telling us?
2) is there a way to modify it to get it to tell us cpu% (or % of total cpu time) it is taking up?

I suspect the latter is possible, but it'd probably take some fancy cdef tinkering to get it right. And then you might need to put it on another graph...

(I guess if you know the number of cpus and thus total number of centiseconds available in a period you can calculate % of cpu time used by process in that period)

So how do we do this?

[BSOD2600]

1) How much CPU time a process has used per second (on a 5 min average). In theory, using the windows performance monitor for the same process for cpu/memory should show a similar graph as Cacti.
2) One would have to use the cooked windows performance counters for this one... and WMI calls are expensive when collecting from thousands of devices.

[argon0]

BSOD....

1) How much CPU time a process has used per second (on a 5 min average). In theory, using the windows performance monitor for the same process for cpu/memory should show a similar graph as Cacti.

What calculations are you doing on the raw data, and where can I find them?

What it seems to be doing is not showing how much CPU time PER SECOND, but per 5 minutes (otherwise how comes I get figures of 1100+ on a 4 proc box (1 minute granularity)? - to note the 5 minute granularity graph for this shows a peak of 220 ish - this is for the system idle process which you would expect this to use the majority of the cpu time most of the time).

If it were what you say, it would be easy to convert this into a % of the total available CPU time. Especially if measured in centiseconds (simply divide by number of processors, would give % of total time available, as total time available would be 100 centiseconds per proc).

EDIT:: Just realised something, though I've looked through your win32_procs.xml, and it seems to be the same as the one included with CactiEZ, I haven't installed it on my system....

[BSOD2600]

What calculations are you doing on the raw data, and where can I find them?

Only place calculations can take place for a SNMP Data Query is a graph CDEF -- and there are none in use.

What it seems to be doing is not showing how much CPU time PER SECOND, but per 5 minutes

As is every other standard graph Cacti/RRDTool draws. For example, [SNMP - Interface Statistics]. This is how rrdtool functions. I'd strongly suggest you read through its man pages and http://www.vandenbogaerdt.nl/rrdtool/

[argon0]

Only place calculations can take place for a SNMP Data Query is a graph CDEF -- and there are none in use.

Quite. This is the problem.

I know how Cacti and RRDTool work, I've been around it all for quite a while now. My problem is that you aren't showing what you think you are showing. You say:

1) How much CPU time a process has used per second (on a 5 min average).

, but what you are actually showing is how many Centiseconds (or seconds - I still can't work out which), each process is using per 5 minutes (600 seconds) NOT how much CPU time a process has used per second averaged over 5 minutes.

Now for comparison's this is fine, but for accuracy its not, oh, and what if you use 1 minute RRAs or even realtime (not that I know how realtime works for this sort of thing )?

To illustrate:

Lets say (for arguments sake) you have a one processor box, with 5 processes running (including both system, and system idle process (SIP)- call the others A, B,C).

In 5 minutes:
A uses 25% of the CPU time available
B uses 10%
C uses 5%
System uses 10%
and the rest is SIP = 50%

So on a CPU utilisation graph you'd see just 50%.

But on the graph as mentioned above, we graph (assuming we are graphing seconds):

150 for A
60 for B
30 for C
60 for System
300 for SIP

Now this is confusing enough as it is, but when you throw in additional processors it gets more so.

Lets just say we can't use an SNMP query to get how many CPUs are in a system, we can still ask during graph creation how many CPUs are in a system, and use that in a CDEF (I would think).

It should be really simple to turn these figures into % if we use the above variable and a CDEF to divide by (6 * Variable)- for above we'd get

150/(6*1) = 25% (A)
60/(6*1) = 10% (B)
30/(6*1) =5% (C)
60/(6*1) = 10% (System)
300/(6*1) = 50% (SIP)

- For a 4 CPU system with the following figures (some taken from my graphs earlier)

1100/(6*4) = 45.8% (SIP)
670/(6*4) = 27.9% (System)
159/(6*4) = 6.6% (SQL)
150/(6*4) = 6.25% (A)

Which seems more sensible to me.

[BSOD2600]

All sounds well and good. Do you have data to support your hypothesis?

[argon0]

Well I'm away from my Cacti install at the moment - couple of days leave.... So this is from memory/nectar report of a 4 CPU box - see below:

Memory/CPU Proc Graphs

Graphs.png (163.95 KiB) Viewed 3729 times

(Why'd I say 600 before - 600 would be 10 minutes... - 5 minutes is 300 seconds, so divide by 3 * No. of Procs to get %.


OK, for SIP on a 5 min average we're getting a max of 503.99 = 503.99/(3*4) = 503.99/12 = 41.99% of available CPU time which tbh sounds a bit low, but averaged over a 5 min period - could be right (would be interesting to see the 1 minute peak figure, which before was showing 1100, which, to me sounds like 5 a calculated 5 min peak displayed on a per minute basis, as I'm not generating 1 minute RRAs for this).

Wish I was graphing SQL.exe - that takes up a fair bit of CPU - I suspect the peaks on the CPU graph are mainly caused by this..

I will see what I can do...

[argon0]

BSOD.... I've had a server reboot over the weekend - not a problem as such, but when it came up again the some of the processes are no longer being monitored, though they are still (or again) running - of course the PID has changed but the process name remains the same.

I was under the impression that your script checked the index for the name of the process? Does it do this every time the script is run, or just x times a day?

If the latter, is there a way to force the script to run the index check?

[phalek]

There should be a drop-down that defines the criteria for the data quieries. Edit your host and check the last table there ( look for the drop-down as seen in the attached image).

You can force a re-index using the command line:

Code: Select all

cd /path/to/cacti/cli
php poller_reindex_hosts.php --help
Cacti Reindex Host Script 1.2, Copyright 2004-2011 - The Cacti Group

usage: poller_reindex_hosts.php --id=[host_id|All] [--qid=[ID|All]] [--host-descr=[description]]
                           [-d] [-h] [--help] [-v] [--version]

--id=host_id             - The host_id to have data queries reindexed or 'All' to reindex all hosts
--qid=query_id           - Only index on a specific data query id; defaults to 'All'
--host-descr=description - The host description to filter by (SQL filters acknowledged)
--debug                  - Display verbose output during execution
-v --version             - Display this help message
-h --help                - Display this help message

[argon0]

Cheers Phalek, there is. BUT, and this is really annoying, once you've created it with the "wrong" reindex method, you seem to be stuck with it unless you are willing to dig into SQL tables....

I've set up a number (fortunately not too many) of hosts using my Windows template, - and hadn't realised that you can't set the reindex method in the template (most annoying), you have to add the data query to the host when creating the host, if you want to set the reindex method to be different....

This needs looking at - you should be able to edit this field from within the host template....

BTW I now (finally) understand what "Uptime Goes Backwards" means (though I don't know why my hosts didn't run the re-index after a reboot) - if Uptime goes down, run a re-index (so for those, like me who were dense around this - if the current uptime goes from 10 days to 1 minute re-index..)

[phalek]

Yes, you're right about that. I've never figured out why the Re-Index method cannot be changed later on. Had the very same issue with a customer where essentially the index changes after each configuration change on a firewall.

Nevertheless, the reindex_method is stored in the host_snmp_query table as a simple integer:

Code: Select all

mysql> describe host_snmp_query;
+----------------+-----------------------+------+-----+---------+-------+
| Field          | Type                  | Null | Key | Default | Extra |
+----------------+-----------------------+------+-----+---------+-------+
| host_id        | mediumint(8) unsigned | NO   | PRI | 0       |       |
| snmp_query_id  | mediumint(8) unsigned | NO   | PRI | 0       |       |
| sort_field     | varchar(50)           | NO   |     |         |       |
| title_format   | varchar(50)           | NO   |     |         |       |
| reindex_method | tinyint(3) unsigned   | NO   |     | 0       |       |
+----------------+-----------------------+------+-----+---------+-------+
5 rows in set (0.00 sec)

The snmp_query_id is the id as listed on the "Data Queries" page. host_id is pretty obvious.

1 -> Uptime Goes Backwards
2 -> Index Count Changed
3 -> Verify All Fields

[argon0]

Also, I suspect, I've either got an old version of the template.xml file, or there's something that needs to be done after importing them which I missed - i.e. changing the index value from a PID to a name.

I suspect this needs changing in the data template, though I'm not entirely sure. At the moment in the data template it shows that the following fields are set to "User Per-Data Source Value (Ignore this Value)" implying that the value needs to be set for each data source (ow if this is the case). I am going to try to set some defaults for the Index Type, see what happens then...


Index Type

Index Value

Output Type ID

ack...

It looks like changing a value here - in this case at least (which I'm sure isn't the case for some other graphs I've worked on) doesn't change anything.

Do you really have to change each data source manually to get this to work?

If so - WHAT a pain - Cacti is so easy to use generally that this surprises me (OK, so its not much different to the work a user needs to put in to get my Multi-CPU graphs to work - but at least that's just once per host!).

[phalek]

Not sure which one you are using, but the Index for the Process seems to be "hrSWRunIndex".

The win32_procs.xml file lists that oid in the "oid_index" field.

The hrSWRunName would probably be better, but... the index is being used to match against the Memory and CPU counters, so it's a bit tricky here. Nevertheless, you're right. The RunIndex is surely changed after a restart. This can even be a simple service restart. ( you can check that by starting notepad.exe, then check the list of "SNMP - Running Processes", close it and start it again. The PID should be different then. )

The Problem now is to create a unique index here. Maybe combining the "name" and "parameters" data. Not sure if this can be done with a "simple" snmp query. I'm sure it can be done with a script query ...

[argon0]

Though, if a graph has broken it is fairly trivial (per data source) to change the settings around to get the graph to work again.

Need to find a way to do this in bulk....

Ah... Just had a thought... When you put variables (e.g. hrSWRunName) into default fields do you need to surround them with pipes - thus |hrSWRunName| as you do in graph titles? Nope - it should probably be |query_hrSWRunName|... OK, you don't set them here - though why they are here as fields you CAN set I don't know (see BSOD himself over here: http://forums.cacti.net/about32836.html)...

So the data query needs changing...

BUT HOW??? I've checked the docs: http://docs.cacti.net/manual:087:3a_adv ... ta_queries

Which is spectacularly unhelpful in this instance...

Why is the Data Source choosing a PID rather than the process name for Index Value? Why is it choosing hrSWRunIndex for Index Type rather than HrSWRunName?

<ack>