Hi,
I have a package which contains an OLEDB source and a lookup transformation which transfers no match data to script component. I've done the same kind of package before but the data to transfer is very less(in hundreds..). But this time I have around 120,000 rows which needs to be validated for no match in lookup before sending into script.
I am tried both visual studio 2010 and 2013 and both of them kept hanging when I try to debug the package or execute the package. And it is getting so hard to bring my maching back to normal.
I looked into progress panel when debugging the package and lookup is trying to cache all the rows and it took so long to load all the rows for validation.
So, I am wondering how to handle such huge amount of data in packages. I am a beginner in SSIS, so not sure of all the best practices. Please suggest how to proceed further.
Thank you.