2003.09.12 10:57 PM
Scheduled Tasks Sluething
A couple of years ago, I developed for one of my clients a number of scripts to asynchronously process long-running requests made via a collection of web-based utilities. Most of the processing is handled by various custom COM components (also mine), which are called by a handful of simple VBScripts run periodically via the Windows Task Scheduler. In their office, these tasks, scripts, and components chug along without a hitch on various W2K servers. However, they recently asked me to load and configure a number of these web-based utilities on a WinXP Professional SP1 notebook so they could demo them at a conference.
No problem. Load and setup IIS, add a couple of out-of-process virtual directories, register some components, set some COM+ impersonation and security settings, load and configure SQL Server 2000, import some sample databases, apply an XML update, setup some scheduled tasks, and we're good to go. Everything worked fine from the machine itself via localhost, and worked fine from a separate client workstation.
Couple of days later, the client calls and says the scheduled tasks on the demo machine are suddenly failing with an exit code of 0x80 and ("Egads!") they're leaving tomorrow for the conference. Hmm. "What changed", I asked. "Nothing." (long pause) "Honest." Okay.
A quick check of my scripts shows this isn't one of my return codes. So, I connect to the machine via Terminal Services to have a look. While connecting I discover that there's already an account logged in (it seems WinXP Professional will host only one console/TS (?) session at a time, so it's connect-to-it or cause-it-to-be-logged-out). I contact the tech guys at the client and get the password for the logged in account and connect to it via TS (I also learn that it's logged in as this account only temporarily in order to run something unrelated to my stuff from a command window).
Ignoring all that, I navigate over to the scheduled tasks, pick one and right-click run it. Exit code 0x80. I open one of the tasks, take its command line and paste it into a command window, run it, everything works great. So now I'm thinking security. Perhaps one of the tech guys changed the password of the account I associated with the scheduled tasks? But that can't be, because I also hardwired the COM+ impersonation with this account and the components and web site are all working fine. Just to be sure, I make another quick call to confirm - no password change.
Okay. Well, the account already logged in is not the same as the one associated with the scheduled tasks (though they have equivalent permissions), and the system works fine if no one is logged in or the account associated with the tasks is logged in, so maybe WinXP Professional won't allow one account to launch scheduled tasks as another account? Not likely, but it was worth a search. What I discovered was Knowledge Base article 812400.
Turns out there's a little problem on WinXP Professional SP1 with the Task Scheduler and the Human Interface Device Access service. I won't profess to know what the HID input service does exactly, but in so much as it has to do with human interface devices that weren't important to us, and the MS workaround (short of loading a special patch or SP2) was to disable it, that's what we did.
Success! Elapsed time: ~45 minutes.
In re-reading this, I realize that in this case I could have gone from "Egads!" to "Success!" in a little less time if I had simply searched first. However, for me, this decision is akin to picking a lane on a crowded freeway or a line at the grocery store. Inevitably, the one I get in will prove to be the slowest.
Well, I should search too. Thanks for the info. It worked like a charm. Months of frustration is now over.
Pablo | 2003.11.02 07:53 AM
Glad to hear it. If I had known when I started this blog how often folks would be finding this stuff while searching for specific technical issues (my referrer stats are loaded with folks coming from Google searches), I might have taken more care to reduce the story-telling noise and concentrated instead on just the solutions. Oh well, live and learn. I'm glad you kept reading long enough to find the the KB article link.
ewbi.develops | 2003.11.02 12:16 PM
Well, I have the same problem even using windows 2000 server
The tasks run only when I use the same account i m logged in with.
rick | 2003.12.10 05:02 PM
Rick - That's interesting. When you say "the same problem", do you mean it was related to the HID service on your Win2K server too? Also, as per my experience on WinXP, do the tasks also run successfully when no one is logged in?
ewbi.develops | 2003.12.10 10:42 PM
I want to give a W2K take on this issue that I experienced. I had written CScript batch files that restarted IIS virtual directory if no one was logged into a database and logged the output to text file - this involved creating 3 seperate objects.
This task ran evey 15 minutes happily for about 3 months. A few days ago after I added several more site (this server hosts a Web solution for different regions) and I started getting 0x80 errors. I then noticed that the SERVICES.EXE was at about 200+ MB. I stopped a service (TREND) manually ran the schedule task and once again the tasks continued to operate.
...until 2 days later the same thing occurred. Once again stopped another service and it worked again. I then tried restarting both services that I had stopped and got a very strange '..can't respond in timely fashion..' type error when I tried to start both - 1 would work fine, but both wouldn't together.
I started to think that SERVICES.EXE had reached some type of limit.
I then looked at my script and noticed I wasn't setting the FSO object and IIS Object to nothing.
Looking at the Private Bytes usage in Performance Monitor you can actually see about 3MB being added and then removed about 10 minutes later post my cleanup.
I believe at this point that because my script didn't clean up the objects. i.e SET = Nothing, that the SERVICES.EXE process eventually got too large and the scheduled tasks refused to run. I don't have time to verify this 100%, but it seems like the only logical answer.
Thought I'd share it for others.
P.S. Sometimes the memory usage of SERVICES.EXE should be large (even on reboot) due to optimization settings for a server. See 'The LargeSystemCache Tuning Knob section' in - http://www.microsoft.com/technet/prodtechnol/windows2000serv/maintain/optimize/wperfch7.asp . i.e. By changing the network settings from '...for file sharing' to 'Balance' in Networking->File and Printer Sharing...->Server Optimization the SERVICES.EXE reduced in size to 7MB after a reboot - previously was 200MB.
Glenn Goodwin | 2004.01.09 12:37 AM
Thanks for taking time to share. I wasn't familiar with the LargeSystemCache setting. Here's an MS article that describes the actual Registry setting for those who are interested:
I hadn't heard of the particular memory leaking problem you describe, or its affect on subsequent task execution, but I'm not surprised. We're religious about releasing objects, but I know others who aren't, and believe all memory will be freed when the CScript task is terminated - an iffy proposition for sure.
ewbi.develops | 2004.01.09 09:18 AM
Thanks for solving a Task Scheduler mystery that made for a very rough cople of weeks.
Mike Scirocco | 2004.06.25 10:42 AM
I am seeing similar behavior on a Windows 2000 Server with Terminal Services installed. I can run a given program directly. I can schedule it to run while the same user is logged into the console, and we can watch it work (it creates windows showing activity on the screen, but doesn't wait for a user to do anything). But, if no user (or another user) is logged on to the console, the program starts, but doesn't seem to do anything.
I haven't found a resolution for this...
Jeff C | 2005.01.27 01:02 PM
Does the scheduled task indicate an exit code of 0x80 when it fails (i.e., when it executes with no one logged on, or another user logged on), or something else? In other words, how do you know it failed "to do anything"?
If the task you're scheduling shows windows, the account it executes as may need additional privileges, like perhaps "Log on as a service", or something else allowing it to create/attach-to a windows station. Here's an explanation of windows station from Keith Brown's fantastic Windows security wiki:
Have you checked the Windows event log to see if anything's been reported about the task? You might want to enable some auditing to get a better picture of what's happening. Here's some more info on that, again from Keith Brown's wiki:
ewbi.develops | 2005.01.27 02:48 PM
I recommend using VisualCron as an alternative task scheduler which also has more features. You can find it at http://www.visualcron.com
Brian | 2005.12.22 12:26 PM
TrackBack URL: http://www.typepad.com/services/trackback/6a00d8341c7bd453ef00d83457249e69e2
Listed below are links to weblogs that reference Scheduled Tasks Sluething: