PowerMTA data query and host template

Templates, scripts for templates, scripts and requests for templates.

Moderators: Developers, Moderators

joeldg
Posts: 6
Joined: Thu Apr 09, 2009 8:28 am

PowerMTA data query and host template

Post by joeldg »

Built a host template for PowerMTA http://port25.com internally here

First we need a function that will work for older versions of PHP as well as older versions of PowerMTA as well.

This is in two parts: First this xml2array function (keeps things simple)

Name this "xmlfunction.php", it lives in cacti/scripts for now, feel free to move it around.

Code: Select all

<?php
function xml2array($url, $get_attributes = 1, $priority = 'tag')
{
    $contents = "";
    if (!function_exists('xml_parser_create'))
    {
        return array ();
    }
    $parser = xml_parser_create('');
    if (!($fp = @ fopen($url, 'rb')))
    {
        return array ();
    }
    while (!feof($fp))
    {
        $contents .= fread($fp, 8192);
    }
    fclose($fp);
    xml_parser_set_option($parser, XML_OPTION_TARGET_ENCODING, "UTF-8");
    xml_parser_set_option($parser, XML_OPTION_CASE_FOLDING, 0);
    xml_parser_set_option($parser, XML_OPTION_SKIP_WHITE, 1);
    xml_parse_into_struct($parser, trim($contents), $xml_values);
    xml_parser_free($parser);
    if (!$xml_values)
        return; //Hmm...
    $xml_array = array ();
    $parents = array ();
    $opened_tags = array ();
    $arr = array ();
    $current = & $xml_array;
    $repeated_tag_index = array ();
    foreach ($xml_values as $data)
    {
        unset ($attributes, $value);
        extract($data);
        $result = array ();
        $attributes_data = array ();
        if (isset ($value))
        {
            if ($priority == 'tag')
                $result = $value;
            else
                $result['value'] = $value;
        }
        if (isset ($attributes) and $get_attributes)
        {
            foreach ($attributes as $attr => $val)
            {
                if ($priority == 'tag')
                    $attributes_data[$attr] = $val;
                else
                    $result['attr'][$attr] = $val; //Set all the attributes in a array called 'attr'
            }
        }
        if ($type == "open")
        {
            $parent[$level -1] = & $current;
            if (!is_array($current) or (!in_array($tag, array_keys($current))))
            {
                $current[$tag] = $result;
                if ($attributes_data)
                    $current[$tag . '_attr'] = $attributes_data;
                $repeated_tag_index[$tag . '_' . $level] = 1;
                $current = & $current[$tag];
            }
            else
            {
                if (isset ($current[$tag][0]))
                {
                    $current[$tag][$repeated_tag_index[$tag . '_' . $level]] = $result;
                    $repeated_tag_index[$tag . '_' . $level]++;
                }
                else
                {
                    $current[$tag] = array (
                        $current[$tag],
                        $result
                    );
                    $repeated_tag_index[$tag . '_' . $level] = 2;
                    if (isset ($current[$tag . '_attr']))
                    {
                        $current[$tag]['0_attr'] = $current[$tag . '_attr'];
                        unset ($current[$tag . '_attr']);
                    }
                }
                $last_item_index = $repeated_tag_index[$tag . '_' . $level] - 1;
                $current = & $current[$tag][$last_item_index];
            }
        }
        elseif ($type == "complete")
        {
            if (!isset ($current[$tag]))
            {
                $current[$tag] = $result;
                $repeated_tag_index[$tag . '_' . $level] = 1;
                if ($priority == 'tag' and $attributes_data)
                    $current[$tag . '_attr'] = $attributes_data;
            }
            else
            {
                if (isset ($current[$tag][0]) and is_array($current[$tag]))
                {
                    $current[$tag][$repeated_tag_index[$tag . '_' . $level]] = $result;
                    if ($priority == 'tag' and $get_attributes and $attributes_data)
                    {
                        $current[$tag][$repeated_tag_index[$tag . '_' . $level] . '_attr'] = $attributes_data;
                    }
                    $repeated_tag_index[$tag . '_' . $level]++;
                }
                else
                {
                    $current[$tag] = array (
                        $current[$tag],
                        $result
                    );
                    $repeated_tag_index[$tag . '_' . $level] = 1;
                    if ($priority == 'tag' and $get_attributes)
                    {
                        if (isset ($current[$tag . '_attr']))
                        {
                            $current[$tag]['0_attr'] = $current[$tag . '_attr'];
                            unset ($current[$tag . '_attr']);
                        }
                        if ($attributes_data)
                        {
                            $current[$tag][$repeated_tag_index[$tag . '_' . $level] . '_attr'] = $attributes_data;
                        }
                    }
                    $repeated_tag_index[$tag . '_' . $level]++; //0 and 1 index is already taken
                }
            }
        }
        elseif ($type == 'close')
        {
            $current = & $parent[$level -1];
        }
    }
    return ($xml_array);
}
?>
Next we have a very simple little script I wrote up to return the data in a way that cacti can digest

This needs to be named "powermta.php"

Code: Select all

<?
/*

        powermta stats collector for cacti
        part of the mocospace stats desktop
        april 2009 - joeldg

*/

include "xmlfunction.php";

function output($part_of_array){
        while(list($k1,$v1)=each($part_of_array)){
                $key = $k1;
                # this should almost always fire off
                if(is_array($v1)){
                        while(list($k2,$v2)=each($v1)){
                                $key = $k1 . "_" . $k2;
                                $val = $v2;
                                if(is_array($v2)){
                                        while(list($k3,$v3)=each($v2)){
                                                $key = $k1 . "_" . $k2 . "_" . $k3;
                                                $val = $v3;
                                                echo "$key:$val ";
                                        }
                                }else{

                                        echo "$key:$val ";
                                }
                        }
                }else{
                        echo "$key:$v1 ";
                }
        }
}


$host = $argv[1];
$what = $argv[2] ? $argv[2] : "traffic"; #default to traffic
$port = $argv[3] ? $argv[3] : 8080;  # default to 8080 if second arg not present
$depth = array();

if(!$host){
        die("\nUSAGE: php {file} {hostname} {traffic|conn|resolver|queue|spool} (?{port})\n");
}

$url = "http://$host:$port/status?format=xml";
$parts = xml2array($url);

# now output it
echo output( $parts['rsp']['data']['status'][$what] );

?>

It looks roughly like this when using it from command line

Code: Select all

# php powermta.php
USAGE: php {file} {hostname} {traffic|conn|resolver|queue|spool} (?{port})

# php powermta.php pmta01 queue
smtp_rcp:0 smtp_dom:0 smtp_kb:0.0 pipe_rcp:0 pipe_dom:0 pipe_kb:0.0 discard_rcp:0 discard_dom:0 discard_kb:0.0 file_rcp:0 file_dom:0 file_kb:0.0 gmImprinter_rcp:0 gmImprinter_dom:0 gmImprinter_kb:0.0 alias_rcp:0 alias_dom:0 alias_kb:0.0

# php powermta.php pmta01 traffic
total_out_rcp:6 total_out_msg:6 total_out_kb:5.8 total_in_rcp:2 total_in_msg:2 total_in_kb:2.0 total_gmImprinted:0 lastHr_out_rcp:0 lastHr_out_msg:0 lastHr_out_kb:0.0 lastHr_in_rcp:0 lastHr_in_msg:0 lastHr_in_kb:0.0 lastHr_gmImprinted:0 lastMin_out_rcp:0 lastMin_out_msg:0 lastMin_out_kb:0.0 lastMin_in_rcp:0 lastMin_in_msg:0 lastMin_in_kb:0.0 lastMin_gmImprinted:0 topPerHr_out_rcp:2 topPerHr_out_msg:2 topPerHr_out_kb:2.4 topPerHr_in_rcp:2 topPerHr_in_msg:2 topPerHr_in_kb:2.0 topPerHr_gmImprinted:0 topPerMin_out_rcp:1 topPerMin_out_msg:1 topPerMin_out_kb:1.2 topPerMin_in_rcp:1 topPerMin_in_msg:1 topPerMin_in_kb:1.0 topPerMin_gmImprinted:0


To add to cacti:

Import the host templates, it includes the data queries, graph templates, and host template.

Hope this helps someone

-joeldg
Attachments
cacti_host_template_powermta_server(2).xml
Cacti host templates for power mta
(117.16 KiB) Downloaded 965 times
Last edited by joeldg on Fri Apr 10, 2009 10:08 am, edited 1 time in total.
joeldg
Posts: 6
Joined: Thu Apr 09, 2009 8:28 am

Post by joeldg »

Screenshots of simulated data

Image

Image

Image

Image

Image

These are the latest graphs.. will re-up the export as well[/img]
aleu
Cacti User
Posts: 216
Joined: Mon Dec 11, 2006 10:17 am

Post by aleu »

joeldg wrote:Screenshots of simulated data
I am getting this:

Warning: Variable passed to each() is not an array or object in /var/www/html/cacti/scripts/powermta.php on line 13
joeldg
Posts: 6
Joined: Thu Apr 09, 2009 8:28 am

Post by joeldg »

Have you tested the script in commandline?
Also, have you made sure that in the powermta config you have the IP address of the machine you run cacti on as able to view the management interface?

To troubleshoot:
On command line from your cacti box, just use `curl http://pmta-host:8080` and see, then add the ip in the powermta config on the pmta host and restart it.

then, try the powermta.php script again with the host name.

You may also (for brevity) add a host alias in your /etc/hosts file

Let me know

-joel
aleu
Cacti User
Posts: 216
Joined: Mon Dec 11, 2006 10:17 am

Post by aleu »

joeldg wrote:Have you tested the script in commandline?
Also, have you made sure that in the powermta config you have the IP address of the machine you run cacti on as able to view the management interface?
Yes, I run the script from the CLI and got this error. I have just realized that I was missing curl and when I execute the command from the CLI I am getting this:

"Please consult the http-access directive in the User's Guide for more information."

Does it mean that I need to add my Cacti server somewhere within the PowerMTA configuration file?
joeldg
Posts: 6
Joined: Thu Apr 09, 2009 8:28 am

Post by joeldg »

aleu wrote:
joeldg wrote: Does it mean that I need to add my Cacti server somewhere within the PowerMTA configuration file?
Yes you do.
See the users manual, or look at the config file to see where to place the IP address
aleu
Cacti User
Posts: 216
Joined: Mon Dec 11, 2006 10:17 am

Post by aleu »

joeldg wrote:
aleu wrote:
joeldg wrote: Does it mean that I need to add my Cacti server somewhere within the PowerMTA configuration file?
Yes you do.
See the users manual, or look at the config file to see where to place the IP address
What is this change responsible for? I am able to browse statistics from another server by entering http://mta01:8080 - does it imply that this machine is able to use your script or is there another setting that is responsible for that?

Thanks,
AL
aleu
Cacti User
Posts: 216
Joined: Mon Dec 11, 2006 10:17 am

Post by aleu »

I have managed to make it work. Now I am expecting some graphs. I will provide more feedback soon.

Thanks
aleu
Cacti User
Posts: 216
Joined: Mon Dec 11, 2006 10:17 am

Post by aleu »

aleu wrote:I have managed to make it work. Now I am expecting some graphs. I will provide more feedback soon
Hmm, I am getting results when executing the script from the CLI:

queue:
smtp_rcp:2517 smtp_dom:1493 smtp_kb:39863.3 pipe_rcp:0 pipe_dom:0 pipe_kb:0.0 discard_rcp:0 discard_dom:0 discard_kb:0.0 file_rcp:0 file_dom:0 file_kb:0.0 gmImprinter_rcp:0 gmImprinter_dom:0 gmImprinter_kb:0.0 alias_rcp:0 alias_dom:0 alias_kb:0.0

conn:
smtpIn_cur:0 smtpIn_max:100 smtpIn_top:7 smtpOut_cur:60 smtpOut_max:1200 smtpOut_top:1200

resolver:
namesCached:35449 queriesPending:28

traffic:
total_out_rcp:42130 total_out_msg:41988 total_out_kb:730106.7 total_in_rcp:44845 total_in_msg:44620 total_in_kb:781908.5 total_gmImprinted:0 lastHr_out_rcp:15090 lastHr_out_msg:15033 lastHr_out_kb:263862.4 lastHr_in_rcp:16011 lastHr_in_msg:15924 lastHr_in_kb:276748.6 lastHr_gmImprinted:0 lastMin_out_rcp:31 lastMin_out_msg:29 lastMin_out_kb:282.9 lastMin_in_rcp:43 lastMin_in_msg:35 lastMin_in_kb:312.6 lastMin_gmImprinted:0 topPerHr_out_rcp:21329 topPerHr_out_msg:21286 topPerHr_out_kb:402666.2 topPerHr_in_rcp:22315 topPerHr_in_msg:22259 topPerHr_in_kb:429360.7 topPerHr_gmImprinted:0 topPerMin_out_rcp:1872 topPerMin_out_msg:1862 topPerMin_out_kb:39484.0 topPerMin_in_rcp:2202 topPerMin_in_msg:2191 topPerMin_in_kb:44363.3 topPerMin_gmImprinted:0

spool:
initPct:100 dirs:6 files_inUse:2511 files_recycled:0 files_total:24576

but the graphs do not render properly (see the attached images). Any idea why?
Attachments
connections.JPG
connections.JPG (43 KiB) Viewed 14053 times
resolver.JPG
resolver.JPG (42.88 KiB) Viewed 14053 times
traffic.JPG
traffic.JPG (42.66 KiB) Viewed 14053 times
queue smtp.JPG
queue smtp.JPG (43.89 KiB) Viewed 14053 times
joeldg
Posts: 6
Joined: Thu Apr 09, 2009 8:28 am

Post by joeldg »

aleu wrote:
aleu wrote:I have managed to make it work. Now I am expecting some graphs. I will provide more feedback soon
Hmm, I am getting results when executing the script from the CLI:

...

but the graphs do not render properly (see the attached images). Any idea why?
When I build graphs, I have "out" as a negative (down) value and "in" as a positive value (up) .. If you are only having pmta being used to send large volumes of email, then you will have mostly "down" graphs.

Are you using the newer version of the templates? I updated them on Friday before I left for the day to match the graphs you see.

Obviously, fee free to tweak the graph templates to rearrange the way the data is displayed (you might want to remove the CDEF for reversing outbound values, the graphs will still look fine as everything is set to alpha 40% with a line1 to clearly mark the top. (you should just be able to delete the whole cdef to get them all)

-joel
aleu
Cacti User
Posts: 216
Joined: Mon Dec 11, 2006 10:17 am

Post by aleu »

joeldg wrote:Are you using the newer version of the templates? I updated them on Friday before I left for the day to match the graphs you see.
Yes, I am using the newer version. I do send large mail volume and receive only a small fraction. However, this does not explain why on many graphs I see NaNs (resolver and SMTP Queue).

Any idea why this is happening, since I can see that the PMTA PHP script returns correct values? Is cacti having problems with values with decimal points?
joeldg
Posts: 6
Joined: Thu Apr 09, 2009 8:28 am

Post by joeldg »

aleu wrote:
joeldg wrote:Are you using the newer version of the templates? I updated them on Friday before I left for the day to match the graphs you see.
Yes, I am using the newer version. I do send large mail volume and receive only a small fraction. However, this does not explain why on many graphs I see NaNs (resolver and SMTP Queue).

Any idea why this is happening, since I can see that the PMTA PHP script returns correct values? Is cacti having problems with values with decimal points?
You can turn on cacti logging to a higher level and then in a console you can do something like (if your pmta host is mta01, in the cacti directory)

`tail -f log/cacti.log | grep mta01`

and watch what the output/responses are and if there are any errors.

float vals are parsed fine by cacti, so that should not be causing NaN's.

The only thing 'perhaps' is that some of the data templates are "counter" types (not gauge type), so unless there is a value change then they will be doing the counter val of nothing.. (though, I have not seen these myself)..

What version of cacti are you running? Just on the off-chance you are a few versions behind..

thanks
-joel
aleu
Cacti User
Posts: 216
Joined: Mon Dec 11, 2006 10:17 am

Post by aleu »

joeldg wrote:You can turn on cacti logging to a higher level and then in a console you can do something like (if your pmta host is mta01, in the cacti directory)

`tail -f log/cacti.log | grep mta01`

and watch what the output/responses are and if there are any errors.

float vals are parsed fine by cacti, so that should not be causing NaN's.

The only thing 'perhaps' is that some of the data templates are "counter" types (not gauge type), so unless there is a value change then they will be doing the counter val of nothing.. (though, I have not seen these myself)..

What version of cacti are you running? Just on the off-chance you are a few versions behind..
I am using 0.8.7d with PA 2.4 and Spine 0.8.7c. TO answer your other question, no I do not see any errors when greping the log file. This is quite weird. I know that your ability to help in this case is limited. I wish I had them working.

Thanks for your help.
aleu
Cacti User
Posts: 216
Joined: Mon Dec 11, 2006 10:17 am

Post by aleu »

aleu wrote:
joeldg wrote:You can turn on cacti logging to a higher level and then in a console you can do something like (if your pmta host is mta01, in the cacti directory)

`tail -f log/cacti.log | grep mta01`

and watch what the output/responses are and if there are any errors.

float vals are parsed fine by cacti, so that should not be causing NaN's.

The only thing 'perhaps' is that some of the data templates are "counter" types (not gauge type), so unless there is a value change then they will be doing the counter val of nothing.. (though, I have not seen these myself)..

What version of cacti are you running? Just on the off-chance you are a few versions behind..
I have noticed one more thing. All of the graphs that "partially" work for me seem to properly display the first variable returned by your script and nothing beyond that:

/usr/bin/php -q /var/www/cacti/scripts/powermta.php mta01 queue
smtp_rcp:5874 smtp_dom:3979 smtp_kb:85309.0 pipe_rcp:0 pipe_dom:0 pipe_kb:0.0 discard_rcp:0 discard_dom:0 discard_kb:0.0 file_rcp:0 file_dom:0 file_kb:0.0 gmImprinter_rcp:0 gmImprinter_dom:0 gmImprinter_kb:0.0 alias_rcp:0 alias_dom:0 alias_kb:0.0
<-- here the variable that will be graphed correctly is "smtp_rcp".

Another example:

/usr/bin/php -q /var/www/cacti/scripts/powermta.php mta01 traffic
total_out_rcp:16432 total_out_msg:16327 total_out_kb:175713.7 total_in_rcp:17536 total_in_msg:17323 total_in_kb:185864.4 total_gmImprinted:0 lastHr_out_rcp:49 lastHr_out_msg:41 lastHr_out_kb:416.0 lastHr_in_rcp:49 lastHr_in_msg:41 lastHr_in_kb:407.0 lastHr_gmImprinted:0 lastMin_out_rcp:0 lastMin_out_msg:0 lastMin_out_kb:0.0 lastMin_in_rcp:0 lastMin_in_msg:0 lastMin_in_kb:0.0 lastMin_gmImprinted:0 topPerHr_out_rcp:3824 topPerHr_out_msg:3816 topPerHr_out_kb:46301.6 topPerHr_in_rcp:4152 topPerHr_in_msg:4142 topPerHr_in_kb:49724.8 topPerHr_gmImprinted:0 topPerMin_out_rcp:1839 topPerMin_out_msg:1838 topPerMin_out_kb:19573.6 topPerMin_in_rcp:1947 topPerMin_in_msg:1947 topPerMin_in_kb:20716.5 topPerMin_gmImprinted:0
<--here the "total_out_rcp" will be graphed correctly.

Can anyone advise what to check why Cacti is having issue "processing" other variables returned during the same script run? This happens to all Data Input Methods of PowerMTA template:
PowerMTA data Conn
PowerMTA data Queue
PowerMTA data Resolver
PowerMTA data spool
PowerMTA data Traffic
aleu
Cacti User
Posts: 216
Joined: Mon Dec 11, 2006 10:17 am

Post by aleu »

joeldg wrote:
aleu wrote:
joeldg wrote:You can turn on cacti logging to a higher level and then in a console you can do something like (if your pmta host is mta01, in the cacti directory)

`tail -f log/cacti.log | grep mta01`

and watch what the output/responses are and if there are any errors.
joel, how many rrd files are being created for you per MTA? I think that my graphs are not rendering properly, because my rrd files are not being populated/created correctly. I can see one value stored within each rrd file, eventhough each of them should store two or more. More over, the traffic stats are being split in 5 files not one.

Please advise.
Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests