Hello, August Members of the Sages Guild!
I have an SSIS package that processes incoming Excel reports into a historical SQL table for data mining. It's been working fine and dandy, but, recently, two runs missed some Excel reports.
It's not something I can replicate, as it's sporadic - highly so; the vast majority of the time, it works seamlessly - but, I've got a hunch that it's a matter of timing; that one Control Flow or Data Flow task is being flagged as complete when the actual process within that task is truly not finished.
What the package does;
Within a ForEach Loop Container;
1. Clear out Staging table 1 with a SQL Task doing a TRUNCATE,
2. Clear out Staging table 2 with a SQL Task doing a TRUNCATE,
3. Run a C# script to scan the existing Excel reports to find the oldest report and make it the active report,
4. Import Sheet 1 of the active report into Staging1,
5. Import Sheet 2 of the active report into Staging2,
6. Use a UNION ALL task to send both Staging1 and Staging2 to the History table,
7. Delete the active Excel report file.
Of course, the ForEach Loop Container will check for another report, processing it the same way, finally to stop when all files have been processed.
Now, this has worked like a charm for almost a year, but twice now - last week and today - the container signalled that it successfully completed - it turned green - before having run through all of the Excel files. The only real change is that last week the Excel files jumped in size to about double or triple what they had been when it worked flawlessly.
So, I'm thinking, maybe there is a task getting ahead of itself or something. I thought, maybe a task to run a 1 or 2 second timer to give SSIS a chance to ensure it can "catch its breath" before continuing on to the next task might alleviate this issue. Maybe, I thought, my Excel files are getting too big for SSIS to handle so much at one time. Maybe even, if there is such a setting, something to tell SSIS to slow down a mite.
I guess I could gin up a script to just run a loop to account for a couple of seconds delay, since there doesn't seem to be a timer task in the tooolbox, but I wonder if my hunch is even correct, before I embark on this adventure.
Any ideas?