I'm developing an ETL to load data from a source system into a DW.
I'm using an OLE DB Source to select data from a staging table. I'm using an order by statement in the select to sort the data. In the advance editor for the OLE DB Source, I'm setting isSorted to TRUE and 4 of the columns SortKeyPositions.
I'm using 3 of the columns in Lookup transformations to get there DW key values. One of the 3 columns could be coming from the source system as NULL. I'm using a Conditional Split transformation to assign a DW "null" key value. Then I'm using a Union All transformation to merge the two pipelines back together. In the Union All, I'm deleting the 3 columns I used in the Lookup and Conditional Split transformations.
The 4th column I'm using in a Lookup transformation to determine if I insert or delete into my DW table. Will the records be inserted in sorted order? Should columns having their SortKeyPostions set be deleted in a Union ALL transformation?