Hello guys,
I have to denormalize a flat file.
Each row of the file contains information about a customer:
- Customer's Name;
- Customer's Address;
- Customer's Phone;
- etc...
My DataFlow consists of an OLE DB Source and a long chain of OLE DB commands.
Each of the OLE DB Commands executes a Stored Procedure.
The logic of the package is the next:
1) Insert the customer (Customer table is involved);
2) Insert the customer's address (Address table is involved);
3) Insert the customer's phone (Phone table is involved);
4) ...
Each stored procedure has its input parameters, e.g:
- inserting a customer accepts customer's FirstName, LastName, etc;
- inserting an address accepts HomeAddress, WorkAddress, etc...;
The original input row is quite "heavy" - it has a lot of columns and moreover, most of the columns are of NVARCHAR(between 100 and 300) type.
I am NOT SATISFIED with the package speed and I want to help it a little.
I noticed that many "heavy" columns are used once only, and the subsequent procedures (OLE DB Commands) do not need them at all.
I decided to delete them from the pipe in order to have more memory avaliable to the package - I use a UNION ALL task to accomplish that.
So, the process looks like this:
1) OLE DB Source;
2) OLE DB Command (Insert Customer), wich takes all the columns avaliable from the source;
3) UNION ALL task (named UA1) eliminating the columns (FirstName, LastName, BirthDate, etc);
The UNION ALL task has only one input = OLE DB Command (Insert Customer), its purpose is just to drop unnecessary columns;
4) OLE DB Command (Insert Address), wich takes all the columns avaliable from the source except for columns having been dropped in task UA1;
5) UNION ALL task (named UA2) eliminating the columns (HomeAddress, WorkAddress, City, Country, etc);
6) ...
So, my questions are:
- Do you think I will gain anything from that modification?
- Am I on the right way when using UNION ALL to discard unnecessary columns?
I really interested in your authoritative opinion!
Thanks a lot,
BorkaS
I have to denormalize a flat file.
Each row of the file contains information about a customer:
- Customer's Name;
- Customer's Address;
- Customer's Phone;
- etc...
My DataFlow consists of an OLE DB Source and a long chain of OLE DB commands.
Each of the OLE DB Commands executes a Stored Procedure.
The logic of the package is the next:
1) Insert the customer (Customer table is involved);
2) Insert the customer's address (Address table is involved);
3) Insert the customer's phone (Phone table is involved);
4) ...
Each stored procedure has its input parameters, e.g:
- inserting a customer accepts customer's FirstName, LastName, etc;
- inserting an address accepts HomeAddress, WorkAddress, etc...;
The original input row is quite "heavy" - it has a lot of columns and moreover, most of the columns are of NVARCHAR(between 100 and 300) type.
I am NOT SATISFIED with the package speed and I want to help it a little.
I noticed that many "heavy" columns are used once only, and the subsequent procedures (OLE DB Commands) do not need them at all.
I decided to delete them from the pipe in order to have more memory avaliable to the package - I use a UNION ALL task to accomplish that.
So, the process looks like this:
1) OLE DB Source;
2) OLE DB Command (Insert Customer), wich takes all the columns avaliable from the source;
3) UNION ALL task (named UA1) eliminating the columns (FirstName, LastName, BirthDate, etc);
The UNION ALL task has only one input = OLE DB Command (Insert Customer), its purpose is just to drop unnecessary columns;
4) OLE DB Command (Insert Address), wich takes all the columns avaliable from the source except for columns having been dropped in task UA1;
5) UNION ALL task (named UA2) eliminating the columns (HomeAddress, WorkAddress, City, Country, etc);
6) ...
So, my questions are:
- Do you think I will gain anything from that modification?
- Am I on the right way when using UNION ALL to discard unnecessary columns?
I really interested in your authoritative opinion!
Thanks a lot,
BorkaS