Hello All,
We have a package that moves in about 1.4 million records to a SQL Destination table. The initial load happens in minutes. However when i ran the batch the second time the package took forever to execute. On detailed analysis i found that a Lookup transformation with Partial cache was the task on which the SSIS execution engine was waiting. The Lookup checks if there are duplicates in the base table, if so does not do anything. The Lookup is based on a SQL Query that has a Group By Clause.
I set the lookup transformation to execute with Full Cache and the package just executes as fast as the initial load.
I have other packages that have lookup transformation with partial cache mode work just fine. Is it the combination of Group By , record count and partial cache slowing things up?
Can someone help me understand this case better?
Regards, Dinesh