Hello!
My organization has about 300 DTS packages and I am trying to come up with recommendations for a new ETL tool. One of the options I'm considering (obviously) is SSIS. I have been able to successfully build SSIS versions of a few of our DTS packages, but I'm running into a vexing problem. It seems like the SSIS packages are not running anywhere nearly as fast as their DTS counterparts. For example, one package takes around a minute to run in DTS, but takes between 3 and 5 minutes to run in SSIS.
Admittedly, I am no expert in SSIS but I can't seem to figure out what might be causing the issue.
In a nutshell, here is what the packages are doing:
1) Deletes current data in SQL server
2) Pulls data from an IBM iSeries AS/400 (about 35,000 rows or so)
3) Loads the data into SQL server
The data we pull from the AS/400 is a join of multiple tables with some WHERE clause criteria. As the data are loaded into SQL, we have many columns where we perform transformations. In DTS these are done as ActiveX script; in SSIS I wrote them as Script components.
Both the DTS package and the SSIS package are connecting to the AS/400 via an Odbc connection pointing to a system DSN that uses IBM-provided driver. They connect to SQL Server using a system DSN as well, using the SQL native driver (both are using the driver that came with the SQL 2000 instance we installed for DTS - I wanted to keep things as apples-to-apples as possible).
From what bench-testing I could do, it seems the bottleneck may be in the actual select from the AS/400. Does anyone have any suggestions for a better way to connect to an AS/400 that might yield better performance? (Or if there's something else I should be looking at instead, any guidance would be appreciated.)