We have a customer where our SQL Agent jobs run SSIS packages every hour at 40 minutes past the hour. When our steps were configured to run with 64-bit dtexec the jobs failed 30-40% of the time they were executed. In the job history we see a message like this:
Executed as user: SERVER\USERNAME. Microsoft (R) SQL Server Execute Package Utility Version 10.50.4000.0 for 64-bit Copyright (C) Microsoft Corporation 2010. All rights
reserved. Started: 1:35:00 PM. The return value was unknown. The process exit code was -2146233082. The step failed.
The exit code varies, though it is often as shown above.
In the Application event log we see an Error with the source = Application Error like this:
Faulting application name: DTExec.exe, version: 2009.100.4000.0, time stamp: 0x4fec7ecd
Faulting module name: ntdll.dll, version: 6.1.7601.18229, time stamp: 0x51fb164a
Exception code: 0xc0000374
Fault offset: 0x00000000000c4102
Faulting process id: 0x49ec
Faulting application start time: 0x01cf5810f1324095
Faulting application path: C:\Program Files\Microsoft SQL Server\100\DTS\Binn\DTExec.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 31f927ca-c404-11e3-b7b9-005056a55353
Sometimes the reported faulting module name is mscorwks.dll, version: 2.0.50727.5477, rather than ntdll.dll.
After changing all the SSIS steps in the job to use the 32-bit run time the rate of failures dropped significantly to about 10% or a bit less. The same symptoms remain, though, with intermittent job failures associated always with a dtexec.exe crash and a message in the application event log like this:
Faulting application name: dtexec.exe, version: 2009.100.4000.0, time stamp: 0x4fec7fd7
Faulting module name: ntdll.dll, version: 6.1.7601.18229, time stamp: 0x51fb1072
Exception code: 0xc0000374
Fault offset: 0x000ce753
Faulting process id: 0x40ac
Faulting application start time: 0x01cf653a7776640e
Faulting application path: C:\Program Files (x86)\Microsoft SQL Server\100\DTS\binn\dtexec.exe
Faulting module path: C:\Windows\SysWOW64\ntdll.dll
Report Id: b7db79b2-d12d-11e3-b7b9-005056a55353
I have run DebugDiag 2.0 on the WER hdmp files that get created each time dtexec crashes and I have not had any luck getting any idea what might be causing these crashes through that process. I could use some help figuring out the root cause. The server is running Windows 2008 R2 on a VMWare virtual machine. It is configured with 4 cores and 8gb memory.
There are custom tasks involved in these SSIS packages but nothing in the DebugDiag crash analysis output seems to be pointing at that custom code as far as I can tell.
Re-running the failed job usually results in a successful execution.
Any help would be greatly appreciated.
John Hansen