graphing stops until a reboot

[bonj87]

ok i tried searching the forums, but fail to find my exact problem so here it goes
running cacti v8.8a on a windows 2003 64 bit server, using php 5.2.13.13, my SQL v5 and IIS of course
i have the scheduled cmd.exe job go off every 5 minutes, and a seperate process that kills all those instances every 6 hours (did that because i originally had issues where it would just stop processing the graphs because one of the cmd.exe processes was hung..)
this worked FLAWLESSLY until Nov 7th. I ran some windows updated, then ALL graphing stops. when i look at the graph mangement, or go to a device i can see SNMP works from the CACTI server to 'device x'.
ive been slowly removing the windows updates, one by one to see if one of them were the cause.
i even went and did a database backup, a check on the tables, a repair on the tables (saw a few minor ones that said something was in use), and an optimize on the tables.. i did a poller rebuild as well.
nothing seems to work.. UNTIL... get this..
i reboot the server.. then it starts graphing for sometime.. (always a differnt amount of time), then stops until i reboot the box again.
im lost.. anyone got any ideas?

[BSOD2600]

When graphing stops, click the graph debug link. What's the output?
Technical support page show any issues?
Errors in the cacti log?

[bonj87]

heres the graph debug log of one of the graphs.. i dont see anything on the tech support page
RRDTool Command:

C:/rrdtool/rrdtool.exe graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title=" T1's - Traffic - MF1.500" \
--rigid \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit="0" \
--vertical-label="bits per second" \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="D\:/IIS/wwwroot/cacti/rra/94/2988.rrd":"traffic_in":AVERAGE \
DEF:b="D\:/IIS/wwwroot/cacti/rra/94/2988.rrd":"traffic_out":AVERAGE \
DEF:c="D\:/IIS/wwwroot/cacti/rra/94/2988.rrd":"traffic_in":MAX \
DEF:d="D\:/IIS/wwwroot/cacti/rra/94/2988.rrd":"traffic_out":MAX \
CDEF:cdefa="a,8,*" \
CDEF:cdefe="b,8,*" \
CDEF:cdeff="c,8,*" \
CDEF:cdefg="d,8,*" \
CDEF:cdefba="b,8,*,-1,*" \
CDEF:cdefbb="d,8,*,-1,*" \
AREA:cdefa#00CF00FF:"Inbound" \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s" \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" \
LINE1:cdefe#002A97FF:"Outbound" \
LINE1:cdeff#005D57FF:"Peak Usage\n" \
GPRINT:cdefg:LAST:"Current\:%8.2lf %s" \
GPRINT:cdefg:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefg:MAX:"Maximum\:%8.2lf %s" \
LINE1:cdefa#000000FF:"" \
LINE1:cdefba#000000FF:"" \
LINE1:cdefbb#0000AAFF:"Peak Usage\n" RRDTool Says:

OK

[bonj87]

this the same graph.. after a reboot and it started graphing
C:/rrdtool/rrdtool.exe graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title=" T1's - Traffic - MF1.500" \
--rigid \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit="0" \
--vertical-label="bits per second" \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="D\:/IIS/wwwroot/cacti/rra/94/2988.rrd":"traffic_in":AVERAGE \
DEF:b="D\:/IIS/wwwroot/cacti/rra/94/2988.rrd":"traffic_out":AVERAGE \
DEF:c="D\:/IIS/wwwroot/cacti/rra/94/2988.rrd":"traffic_in":MAX \
DEF:d="D\:/IIS/wwwroot/cacti/rra/94/2988.rrd":"traffic_out":MAX \
CDEF:cdefa="a,8,*" \
CDEF:cdefe="b,8,*" \
CDEF:cdeff="c,8,*" \
CDEF:cdefg="d,8,*" \
CDEF:cdefba="b,8,*,-1,*" \
CDEF:cdefbb="d,8,*,-1,*" \
AREA:cdefa#00CF00FF:"Inbound" \
GPRINT:cdefa:LAST:" Current\:%8.2lf %s" \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n" \
LINE1:cdefe#002A97FF:"Outbound" \
LINE1:cdeff#005D57FF:"Peak Usage\n" \
GPRINT:cdefg:LAST:"Current\:%8.2lf %s" \
GPRINT:cdefg:AVERAGE:"Average\:%8.2lf %s" \
GPRINT:cdefg:MAX:"Maximum\:%8.2lf %s" \
LINE1:cdefa#000000FF:"" \
LINE1:cdefba#000000FF:"" \
LINE1:cdefbb#0000AAFF:"Peak Usage\n" RRDTool Says:

[BSOD2600]

Is "T1's - Traffic - MF1.500" a graph which is not currently working?
Where did you get this template from, as it does not appear to be the standard.
Any errors in the cacti log?
errors in windows event logs?

[bonj87]

its one that was canned iwth cacti
its a graph that had worked all along
its just a traffic graph to a router's interface showing "Interface - Traffic (bits/sec, Total Bandwidth)"


Graph Template Items [edit: Interface - Traffic (bits/sec, Total Bandwidth)] Add

Graph Item Data Source Graph Item Type CF Type Item Color
Item # 1 (traffic_in): Inbound AREA AVERAGE 00CF00
Item # 2 (traffic_in): Current: GPRINT LAST
Item # 3 (traffic_in): Average: GPRINT AVERAGE
Item # 4 (traffic_in): Maximum: GPRINT MAX
Item # 5 COMMENT: Total In: |sum:auto:current:2:auto|<HR> COMMENT AVERAGE
Item # 6 (traffic_out): Outbound LINE1 AVERAGE 002A97
Item # 7 (traffic_out): Current: GPRINT LAST
Item # 8 (traffic_out): Average: GPRINT AVERAGE
Item # 9 (traffic_out): Maximum: GPRINT MAX
Item # 10 COMMENT: Total Out: |sum:auto:current:2:auto| COMMENT AVERAGE


Graph Item Inputs Add

Name
Inbound Data Source
Outbound Data Source


Template [edit: Interface - Traffic (bits/sec, Total Bandwidth)]


Name
The name given to this graph template.




Graph Template

Title (--title)
Use Per-Graph Value (Ignore this Value)
Image Format (--imgformat)
Use Per-Graph Value (Ignore this Value) PNG GIF SVG
Height (--height)
Use Per-Graph Value (Ignore this Value)
Width (--width)
Use Per-Graph Value (Ignore this Value)
Slope Mode (--slope-mode)
Use Per-Graph Value (Ignore this Value) Slope Mode (--slope-mode)
Auto Scale
Use Per-Graph Value (Ignore this Value) Auto Scale
Auto Scale Options
Use Per-Graph Value (Ignore this Value) Use --alt-autoscale (ignoring given limits)
Use --alt-autoscale-max (accepting a lower limit)
Use --alt-autoscale-min (accepting an upper limit, requires rrdtool 1.2.x)
Use --alt-autoscale (accepting both limits, rrdtool default)

Logarithmic Scaling (--logarithmic)
Use Per-Graph Value (Ignore this Value) Logarithmic Scaling (--logarithmic)
SI Units for Logarithmic Scaling (--units=si)
Use Per-Graph Value (Ignore this Value) SI Units for Logarithmic Scaling (--units=si)
Rigid Boundaries Mode (--rigid)
Use Per-Graph Value (Ignore this Value) Rigid Boundaries Mode (--rigid)
Auto Padding
Use Per-Graph Value (Ignore this Value) Auto Padding
Allow Graph Export
Use Per-Graph Value (Ignore this Value) Allow Graph Export
Upper Limit (--upper-limit)
Use Per-Graph Value (Ignore this Value)
Lower Limit (--lower-limit)
Use Per-Graph Value (Ignore this Value)
Base Value (--base)
Use Per-Graph Value (Ignore this Value)
Unit Grid Value (--unit/--y-grid)
Use Per-Graph Value (Ignore this Value)
Unit Exponent Value (--units-exponent)
Use Per-Graph Value (Ignore this Value)
Vertical Label (--vertical-label)
Use Per-Graph Value (Ignore this Value)

[BSOD2600]

You keep ignoring my other questions...

[bonj87]

Is "T1's - Traffic - MF1.500" a graph which is not currently working? its a graph of a cisco router
Where did you get this template from, as it does not appear to be the standard. i used the built in traffic graph
Any errors in the cacti log? NO
errors in windows event logs? NO

[BSOD2600]

When you say not working, what exactly does that mean? data is no longer collected? graph doesnt render? Does everything break at once or just this graph? Is the poller still running every 5 minutes when this occurs (should see this in the cacti.log)?

[bonj87]

the poller is running every 5 minutes
ALL graphs break.. no data is plotted i get the "-1#J" on them (if the time period has elapsed with no data in the graph.. ie 24 hours), otherwise it shows me the last data it did graph in the values at the bottom of each graph.

[BSOD2600]

That doesn't really make sense how the graphs can be showing data for a while and then revert to being broken and no data "-1#J". Sure there isnt a problem with the disk?

[bonj87]

positive.. did physical checks of the disk. ran the chkdsk.. checked all hardware with HP diagnostics
i found one thing that seems to be helping.. still testing to see if it works.. ill know more in about an hour but it looks promising.

[bonj87]

well
it looks like one of the processes durring the poller cycle is hanging.
i say this because i have a batch file running the following 1 minute before the pollar goes off
skill.exe spine.exe
pskill.exe php.exe
pskill.exe sh.exe
pskill.exe php-cgi.exe
pskill.exe cmd.exe

its been graphing steadily since.

[BSOD2600]

using sysinternals process explorer, whats the command/script running which is hanging? Is there no timeout logic in said script?

prematuring killing off spine/cmd.php is likely going to cause problems too with an unclean polling cycle ending.

[bonj87]

ill try to detrmine that.
the cmd.exe process ends, but i did notice the php-cgi.exe was still running .. that gets killed by that batch script and its graphing
im wondering if my php is hosed?