[SOLVED] no graphs....corruption and mysql errors

[cpaasche]

Graphs stopped working after reboot.

I have checked my back up folder to see if any files are missing....I did not find that to be the case.

Graph management says "No Graphs Found" but I can see the rrd files updating.

Device list is populated, SNMP information is listed & the SNMP verbose queries are successfull.

From the log files

Code: Select all

 11/29/2007 03:50:26 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed "SELECT graph_templates_graph.id, graph_templates_graph.local_graph_id, graph_templates_graph.height, graph_templates_graph.width, graph_templates_graph.title_cache, graph_templates.name, graph_local.host_id FROM (graph_local,graph_templates_graph) LEFT JOIN graph_templates ON (graph_local.graph_template_id=graph_templates.id) WHERE graph_local.id=graph_templates_graph.local_graph_id AND (graph_templates_graph.title_cache like '%%%%' OR graph_templates.name like '%%%%') ORDER BY title_cache ASC LIMIT 0,30"

11/29/2007 03:50:26 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "SELECT COUNT(graph_templates_graph.id) FROM (graph_local,graph_templates_graph) LEFT JOIN graph_templates ON (graph_local.graph_template_id=graph_templates.id) WHERE graph_local.id=graph_templates_graph.local_graph_id AND (graph_templates_graph.title_cache like '%%%%' OR graph_templates.name like '%%%%')"

11/29/2007 03:50:19 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed "SELECT graph_templates_graph.id, graph_templates_graph.local_graph_id, graph_templates_graph.height, graph_templates_graph.width, graph_templates_graph.title_cache, graph_templates.name, graph_local.host_id FROM (graph_local,graph_templates_graph) LEFT JOIN graph_templates ON (graph_local.graph_template_id=graph_templates.id) WHERE graph_local.id=graph_templates_graph.local_graph_id AND (graph_templates_graph.title_cache like '%%%%' OR graph_templates.name like '%%%%') ORDER BY title_cache ASC LIMIT 0,30"

11/29/2007 03:50:19 PM - CMDPHP: Poller[0] ERROR: SQL Cell Failed "SELECT COUNT(graph_templates_graph.id) FROM (graph_local,graph_templates_graph) LEFT JOIN graph_templates ON (graph_local.graph_template_id=graph_templates.id) WHERE graph_local.id=graph_templates_graph.local_graph_id AND (graph_templates_graph.title_cache like '%%%%' OR graph_templates.name like '%%%%')"

11/29/2007 03:50:15 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed "select graph_templates.id, graph_templates.name from (graph_local,graph_templates,graph_templates_graph) where graph_local.id=graph_templates_graph.local_graph_id and graph_templates_graph.graph_template_id=graph_templates.id and graph_local.host_id=24 group by graph_templates.id order by graph_templates.name"

11/29/2007 03:49:34 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed "select graph_templates.id, graph_templates.name from (graph_local,graph_templates,graph_templates_graph) where graph_local.id=graph_templates_graph.local_graph_id and graph_templates_graph.graph_template_id=graph_templates.id and graph_local.host_id=24 group by graph_templates.id order by graph_templates.name"

11/29/2007 03:49:33 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed "select graph_tree_items.id, graph_tree_items.title, graph_tree_items.local_graph_id, graph_tree_items.rra_id, graph_tree_items.order_key, graph_templates_graph.title_cache as title_cache from (graph_tree_items,graph_local) left join graph_templates_graph on (graph_tree_items.local_graph_id=graph_templates_graph.local_graph_id and graph_tree_items.local_graph_id>0) left join host on (host.id=graph_local.host_id) left join graph_templates on (graph_templates.id=graph_local.graph_template_id) left join user_auth_perms on ((graph_templates_graph.local_graph_id=user_auth_perms.item_id and user_auth_perms.type=1 and user_auth_perms.user_id=5) OR (host.id=user_auth_perms.item_id and user_auth_perms.type=3 and user_auth_perms.user_id=5) OR (graph_templates.id=user_auth_perms.item_id and user_auth_perms.type=4 and user_auth_perms.user_id=5)) where graph_tree_items.graph_tree_id=1 and graph_local.id=graph_templates_graph.local_graph_id and graph_tree_items.order_key like '020___000000000000000000000000000000000000000000000000000000000000000000000000000000000000' and graph_tree_items.local_graph_id>0 and ((user_auth_perms.type != 1 OR user_auth_perms.type is null) AND ((user_auth_perms.type != 3) OR (user_auth_perms.type is null)) AND ((user_auth_perms.type != 4) OR (user_auth_perms.type is null))) group by graph_tree_items.id order by graph_tree_items.order_key"

11/29/2007 03:49:30 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed "select graph_tree_items.id, graph_tree_items.title, graph_tree_items.local_graph_id, graph_tree_items.rra_id, graph_tree_items.order_key, graph_templates_graph.title_cache as title_cache from (graph_tree_items,graph_local) left join graph_templates_graph on (graph_tree_items.local_graph_id=graph_templates_graph.local_graph_id and graph_tree_items.local_graph_id>0) left join host on (host.id=graph_local.host_id) left join graph_templates on (graph_templates.id=graph_local.graph_template_id) left join user_auth_perms on ((graph_templates_graph.local_graph_id=user_auth_perms.item_id and user_auth_perms.type=1 and user_auth_perms.user_id=5) OR (host.id=user_auth_perms.item_id and user_auth_perms.type=3 and user_auth_perms.user_id=5) OR (graph_templates.id=user_auth_perms.item_id and user_auth_perms.type=4 and user_auth_perms.user_id=5)) where graph_tree_items.graph_tree_id=1 and graph_local.id=graph_templates_graph.local_graph_id and graph_tree_items.order_key like '019___000000000000000000000000000000000000000000000000000000000000000000000000000000000000' and graph_tree_items.local_graph_id>0 and ((user_auth_perms.type != 1 OR user_auth_perms.type is null) AND ((user_auth_perms.type != 3) OR (user_auth_perms.type is null)) AND ((user_auth_perms.type != 4) OR (user_auth_perms.type is null))) group by graph_tree_items.id order by graph_tree_items.order_key"

Clicking on a host in the graph view gives this error message

Code: Select all

Warning: array_push() [function.array-push]: First argument should be an array in /var/www/localhost/htdocs/cacti/lib/html_tree.php on line 700

Clicking on "function.array-push" gives this error message but I don't see this file in my backup directory

Code: Select all

Not Found

The requested URL /cacti/function.array-push was not found on this server.

The last line of the output below is line 700 of html_tree.php

Code: Select all

/* for graphs without a template */
                        array_push($graph_templates, array(
                                "id" => "0",
                                "name" => "(No Graph Template)"
                                ));

Thanks.

[gandalf]

This is which version of cacti?
Reinhard

[cpaasche]

Cacti: 0.8.6j-r4

Cactid:0.8.6i-r1

Gentoo: linux-2.6.20-gentoo-r8

mySQL: 5.0.42

[fmangeant]

Hi

can you check that your server time has not changed ?

[cpaasche]

Bon jour,

System is using NTP and is an NTP server for the routers.

from ntp.conf: server pool.ntp.org

Time is accurate. Reference clock is accessable, etc.

Merci.


update: Time to put this on hold. Seems like the hdd is having issues. I will just copy the configs and data and build a new box. Thanks for the help

[cpaasche]

HDD is having issues so I am building a new box. Crappy Seagate HDD.

Code: Select all

hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=30872754, sector=30872754
ide: failed opcode was: unknown
end_request: I/O error, dev hda, sector 30872754

But I need to port over the data. I have old backups.

mySQL reports the following error.

Code: Select all

SQL show index from `data_template_data_rra` failed : Table './cacti/data_template_data_rra' is marked as crashed and should be repaired

Attempting a new backup gives the following.

Code: Select all

gentoo cacti # mysqldump -l --add-drop-table cacti > mysql.cacti.new
mysqldump: Got error: 145: Table './cacti/data_template_data_rra' is marked as crashed and should be repaired when using LOCK TABLES

When I ran repair the first time. I got this.

gentoo cacti # mysqlcheck --auto-repair --databases -u usr -p password
Enter password:
cacti.cdef OK
cacti.cdef_items OK
cacti.colors OK
cacti.data_input OK
cacti.data_input_data OK
cacti.data_input_fields OK
cacti.data_local OK
cacti.data_template OK
cacti.data_template_data OK
cacti.data_template_rrd
error : Can't read key from filepos: 173056
error : Corrupt
cacti.graph_local OK
cacti.graph_template_input OK
cacti.graph_template_input_defs OK
cacti.graph_templates OK
cacti.graph_templates_gprint OK
cacti.graph_templates_item OK
cacti.graph_tree OK
cacti.graph_tree_items OK
cacti.host OK
cacti.host_graph OK
cacti.host_snmp_cache OK
cacti.host_snmp_query OK
cacti.host_template OK
cacti.host_template_graph OK
cacti.host_template_snmp_query OK
cacti.plugin_discover_hosts OK
cacti.plugin_discover_template OK
cacti.plugin_flowview_dnscache
note : The storage engine for the table doesn't support check
cacti.poller OK
cacti.poller_command OK
cacti.poller_item OK
cacti.poller_output OK
cacti.poller_reindex OK
cacti.poller_time OK
cacti.rra OK
cacti.rra_cf OK
cacti.settings OK
cacti.settings_graphs OK
cacti.settings_tree OK
cacti.snmp_query OK
cacti.snmp_query_graph OK
cacti.snmp_query_graph_rrd OK
cacti.snmp_query_graph_rrd_sv OK
cacti.snmp_query_graph_sv OK
cacti.thold_data OK
cacti.thold_template OK
cacti.user_auth OK
cacti.user_auth_perms OK
cacti.user_auth_realm OK
cacti.user_log OK
cacti.version OK
cacti.weathermap_auth OK
cacti.weathermap_maps OK

Repairing tables
cacti.data_template_rrd
error : 5 when fixing table
status : Operation failed
gentoo cacti #

Running mysqlcheck a second time just skips over the error. cacti.data_template_rrd is not listed.

Code: Select all

<snip>

cacti.data_input                                   OK
cacti.data_input_data                              OK
cacti.data_input_fields                            OK
cacti.data_local                                   OK
cacti.data_template                                OK
cacti.data_template_data                           OK
cacti.graph_local                                  OK
cacti.graph_template_input                         OK
cacti.graph_template_input_defs                    OK
cacti.graph_templates                              OK
cacti.graph_templates_gprint                       OK
cacti.graph_templates_item                         OK

</snip>

gentoo cacti #

Can I just import a "data_template_data_rra" table into the database? If so , how? Or do I need to restore my old mysql cacti database and just forget about the data I would lose and move on?

Edit: Well, I had to do some digging...but I found the solution. Hopefully this will help others.

Code: Select all

gentoo cacti # myisamchk /var/lib/mysql/cacti/data_template_data_rra.MYI
Checking MyISAM file: /var/lib/mysql/cacti/data_template_data_rra.MYI
Data records:   14082   Deleted blocks:     150
myisamchk: warning: Table is marked as crashed
- check file-size
- check record delete-chain
- check key delete-chain
- check index reference
- check data record references index: 1
- check data record references index: 2
myisamchk: error: Found 14086 keys of 14082
MyISAM-table '/var/lib/mysql/cacti/data_template_data_rra.MYI' is corrupted
Fix it using switch "-r" or "-o"
gentoo cacti #

gentoo cacti # myisamchk /var/lib/mysql/cacti/data_template_rrd.MYI
Checking MyISAM file: /var/lib/mysql/cacti/data_template_rrd.MYI
Data records:    6586   Deleted blocks:       0
myisamchk: warning: Table is marked as crashed and last repair failed
- check file-size
myisamchk: warning: Size of indexfile is: 208896        Should be: 173056
myisamchk: warning: Size of datafile is: 355840          Should be: 341072
- check record delete-chain
- check key delete-chain
- check index reference
- check data record references index: 1
- check data record references index: 2
- check data record references index: 3
myisamchk: Unknown error 126
myisamchk: error: Can't read indexpage from filepos: -1
- check record links
myisamchk: error: Checksum for key:  3 doesn't match checksum for records
myisamchk: warning: Found 2260 deleted space.   Should be 0
myisamchk: warning: Found         31 deleted blocks       Should be: 0
myisamchk: warning: Found       6669 parts                Should be: 6586 parts
MyISAM-table '/var/lib/mysql/cacti/data_template_rrd.MYI' is corrupted
Fix it using switch "-r" or "-o"

gentoo cacti # myisamchk -o /var/lib/mysql/cacti/data_template_rrd.MYI
- recovering (with keycache) MyISAM-table '/var/lib/mysql/cacti/data_template_rrd.MYI'
Data records: 6586


gentoo cacti # myisamchk -o /var/lib/mysql/cacti/graph_templates_graph.MYI
- recovering (with keycache) MyISAM-table '/var/lib/mysql/cacti/graph_templates_graph.MYI'
Data records: 3322

Yay....Cacti is back to operational on the old box and I can just move it over the the new box.