Poller hangs when user is not logged on server

[chayashida]

(changed title to better describe issue)

I have read through the documentation and installed Cacti on a Windows 2003 server. For the most part, we have it up and running.

I have a problem with the poller, though. It works if I keep remote desktop session open on the server. If I log that session off, though, then the poller will hang.

I'm pretty sure that there is additional account policies that need to be assigned in Windows, but I don't know what they should be.

The cacti user has the following user rights assigned:

• Access this computer from the network
  Allow logon locally
  Allow log on through Terminal Services (added manually)
  Bypass traverse checking
  Log on as a batch job

The scheduled task runs, but it hangs when the cacti user is not logged in. If I am logged in as the cacti user, the windows pop up every 5 minutes and the graph updates like normal. If I log the user off, the graphs no longer update and the poller process times out. The "Run only if logged on" checkbox is not checked in Scheduled Tasks. (Besides, I can verify from the log that the scheduled task starts, even if I'm not logged in.)

Here is an excerpt from the log. You can see that the task hangs when the user is not logged in:

04/23/2008 03:14:59 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/23/2008 03:14:59 PM - SYSTEM STATS: Time:299.1439 Method:cmd.php Processes:1
Threads:N/A Hosts:8 HostsPerProcess:8 DataSources:474 RRDsProcessed:0
04/23/2008 03:19:59 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/23/2008 03:19:59 PM - SYSTEM STATS: Time:299.1172 Method:cmd.php Processes:1
Threads:N/A Hosts:8 HostsPerProcess:8 DataSources:474 RRDsProcessed:0


It runs fine when the user is logged in, even with a disconnected RDP session:

04/23/2008 03:20:04 PM - SYSTEM STATS: Time:3.9622 Method:cmd.php Processes:1 Th
reads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:286
04/23/2008 03:25:04 PM - SYSTEM STATS: Time:3.6360 Method:cmd.php Processes:1 Th
reads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:286



Other information:

• Operating System: Windows 2003
  Webserver: IIS 6
  Cacti: 0.8.7b
  Spine: 0.8.7a
  MySQL: 5.0.51a
  PHP: 5.2.5
  RRDTool (Cygwin or Win32 version): 1.2.27 (win32)
  Net-SNMP: 5.4.1-3
  Cygwin (cygwin1.dll version): N/A
  Plugin Architecture: N/A

[BSOD2600]

The account the cacti scheduled task runs under needs certain Windows and NTFS permissions. It really should only need to be set up as a service account. Under default Win2k3 permissions, my instructions for security work fine. If your organization has tweaked them, then it's going to be up to you, to figure out what broke. It does NOT need TS access though.

I suggest you read HOWTO: Debug Windows NTFS permission problems.. You'll probably need to set process monitor to run on startup, so it will capture events when your Cacti account is not logged on (which is when you're having problems).

[chayashida]

The account the cacti scheduled task runs under needs certain Windows and NTFS permissions. It really should only need to be set up as a service account. Under default Win2k3 permissions, my instructions for security work fine.

I didn't see anything in the instructions that said to grant the "Log on as a service" user right. Since my cacti account is a normal user, it is not granted that right. I vaguely recall there was another user right that allowed interaction with the desktop, but I can't remember what that was right now. I suspect that also may be causing issues.

I don't know what your typical Windows installation looks like. In my case, the computer is a member server of a domain, but not a domain controller. I wasn't the one who set up the server, so I don't know if there were extra tweaks to lock it down. There are computer policies set up in the domain, but I have not audited them yet. I wasn't aware of any that denied processes from running, but I can check.

Out of curiosity, you aren't granting your account any special rights, like adding it to the local Administrators group or anything like that, right? Otherwise, it sort of defeats the purpose of using a separate account...

If your organization has tweaked them, then it's going to be up to you, to figure out what broke. It does NOT need TS access though.

The terminal service access was just for convenience. I don't think it's a requirement. I didn't feel like trying to troubleshoot the poller while having to stand in the server room.

I suggest you read HOWTO: Debug Windows NTFS permission problems.. You'll probably need to set process monitor to run on startup, so it will capture events when your Cacti account is not logged on (which is when you're having problems).

Thanks for the pointer. I'm a little skeptical, though, because NTFS file permissions do not change just because an account is logged in. Since the poller works when the account is logged in, and does not work when it is not logged in, I suspect it is something other than NTFS file permissions.

I think I'll try adding the "Log on as a service" right first.

I'm back on-site on Friday, so I'll post again when I have more information.

Thanks for the help,

Chris

[BSOD2600]

The account the cacti scheduled task runs under needs certain Windows and NTFS permissions. It really should only need to be set up as a service account. Under default Win2k3 permissions, my instructions for security work fine.

I didn't see anything in the instructions that said to grant the "Log on as a service" user right. Since my cacti account is a normal user, it is not granted that right. I vaguely recall there was another user right that allowed interaction with the desktop, but I can't remember what that was right now. I suspect that also may be causing issues.

Sorry phrased it wrong. you're right it doesn't need Service account rights. I was trying to infer that the User account you set up should be just treated as a service account.

Out of curiosity, you aren't granting your account any special rights, like adding it to the local Administrators group or anything like that, right? Otherwise, it sort of defeats the purpose of using a separate account...

Well... on my main cacti box, I have my domain admin account running my scheduled task and then a separate account for the IIS portion. Yes, I know bad practices, but my cacti install is just a hobby at home. Furthermore, then writing the install guide, I set up everything on a VM Win2k3 server, so would've ran into any of the problem like a normal user.

I suggest you read HOWTO: Debug Windows NTFS permission problems.. You'll probably need to set process monitor to run on startup, so it will capture events when your Cacti account is not logged on (which is when you're having problems).

Thanks for the pointer. I'm a little skeptical, though, because NTFS file permissions do not change just because an account is logged in. Since the poller works when the account is logged in, and does not work when it is not logged in, I suspect it is something other than NTFS file permissions.

Process Monitor will help you diagnose more than just NTFS issues. It monitors processes, registry and file system. Obviously there is some sort of permission issue preventing the cacti scheduled task from properly executing. Procmon should reveal the problem (after some work figuring it out).

[chayashida]

Process Monitor will help you diagnose more than just NTFS issues. It monitors processes, registry and file system. Obviously there is some sort of permission issue preventing the cacti scheduled task from properly executing. Procmon should reveal the problem (after some work figuring it out).

Well, I gave it a shot. I downloaded Process Monitor, set it up according to the document you provided, then logged my cacti user off the server. I waited for a polling cycle, and... nothing. Nothing showed up in Process Monitor.

I checked the task manager, and I see php.exe running under cacti. When I saw this in the past, it would basically run for about ten minutes, and then fail. It basically looks like it runs every other time. It's almost like the scheduled task is working right, but something isn't right when PHP tries to run and the user is not logged in.

So I check the logs. Logs look okay. The process ran at 10:25 AM. Also at 10:30 AM. Then I check the graphs. They're fine too.

So now I'm really not sure what's going on. I haven't made any other changes since Wednesday. It's been four polling cycles now, and it still seems to be running.

I'm going to chock this one up to being tired, and possible made some sort of change on Monday that initially caused the issue when I was editing the devices.

I'll have to post again if the problem reproduces.

Chris

[chayashida]

So I check the logs. Logs look okay. The process ran at 10:25 AM. Also at 10:30 AM. Then I check the graphs. They're fine too.
...
I'll have to post again if the problem reproduces.

I double-checked the logs again. They show the following:

Code: Select all

04/25/2008 10:10:04 AM - SYSTEM STATS: Time:3.6639 Method:cmd.php Processes:1 Th
reads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:286
04/25/2008 10:15:08 AM - SYSTEM STATS: Time:7.5002 Method:cmd.php Processes:1 Th
reads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:286
04/25/2008 10:24:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 10:24:59 AM - SYSTEM STATS: Time:298.1482 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0
04/25/2008 10:29:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 10:29:59 AM - SYSTEM STATS: Time:299.1441 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0
04/25/2008 10:35:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 10:35:00 AM - SYSTEM STATS: Time:299.1637 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0
04/25/2008 10:39:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 10:39:59 AM - SYSTEM STATS: Time:299.1457 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0
04/25/2008 10:44:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 10:44:59 AM - SYSTEM STATS: Time:299.1144 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0
04/25/2008 10:49:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 10:49:59 AM - SYSTEM STATS: Time:299.1118 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0
04/25/2008 10:54:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 10:54:59 AM - SYSTEM STATS: Time:299.1149 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0
04/25/2008 10:59:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 10:59:59 AM - SYSTEM STATS: Time:299.1144 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0
04/25/2008 11:04:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 11:04:59 AM - SYSTEM STATS: Time:299.1139 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0
04/25/2008 11:09:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 11:09:59 AM - SYSTEM STATS: Time:299.1115 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0
04/25/2008 11:14:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceed
ed. Exiting.
04/25/2008 11:14:59 AM - SYSTEM STATS: Time:299.1183 Method:cmd.php Processes:1
Threads:N/A Hosts:9 HostsPerProcess:9 DataSources:572 RRDsProcessed:0

I don't know how I misread that. The poller is definitely not working. I now see a gap in the graphs. I also set up Process Monitor again, and I still do not see anything listed.

I don't know what else changes when you log onto the system. There is something that is keeping the php from running correctly, though.

[chayashida]

I found this Support Online article:

http://support.microsoft.com/kb/890478

Unfortunately, it doesn't apply in my case, since my server is already running Windows 2003 Service Pack 2. It might help someone else that comes across this thread, though.

[BSOD2600]

For Procmon, did you enabled the Options -> Enable boot logging? That should keep it running when you log off from an account.

Looking at your scheduled task logs for Cacti, when you're not logged on with the account, does the task successfully run? If so, then the problem is likely with IIS / PHP / permissions.

[chayashida]

For Procmon, did you enabled the Options -> Enable boot logging? That should keep it running when you log off from an account.

I actually did it differently. I logged on with an administrator account to monitor the services, but then waited to see what was going on with the other accounts. I did see php.exe start, so it looks like the scheduled task is running. It's just not completing.

Looking at your scheduled task logs for Cacti, when you're not logged on with the account, does the task successfully run? If so, then the problem is likely with IIS / PHP / permissions.

Yes, I think you're right. I'm going to start looking at the PHP permissions, to see what's different when the cacti user is not logged in.

Chris