Hi all,
I have a staging and loading process that is entirely data driven and is written in SQL. I have about 2000 tables spread across 4 source systems that I need to extract from. In the metadata for the tables, I can specify whether to reload and replace the entire table, or use some sort of filter or where clause. I eventually need to load and combine data from about 30 systems, which would probably include somewhere be 10k and 20k tables. I don't want to have to design a dataflow in SSIS for each table. That would be foolish.
I have two sprocs that stage and clean the data:
DL_LoadStagingTable @TableID
DL_LoadCleanTable @TableID
The @TableID indicates the ID of the row in the table DL_Table that represents the table I want to load. Tables are grouped into systems (which contain the details of the linked server, or CSV file directory etc to load the data from). Hence, to load all the tables for a system, I have a sproc:
DL_LoadSystem @SystemID (where @SystemID indicates the row in a table: DL_System).
However, obviously all the tables in a system get loaded in series and SQL has no native way of using parallelism (which would be wonderful if it did).
Advantages of the above method is, for instance, when I am provided with a link to another source system, I can run a sproc (DL_DeriveSystemStructure) to pull out the metadata for that system and then just annotate it with whatever additional data cleaning rules I want. It makes life a lot quicker and I can add logging to the sprocs to find the data bottlenecks instead of having to cope with the opaqueness of an SSIS package.
However, I would like to use the power of SSIS to parallelize this? Is there a way that I can do this without having to modify the package I create each time I add a table or a system. I don't want to have to create an Execute SQL task for each table I want to load in parallel.
Ideally I'd like to query the database for a list of table IDs and populate some sort of queue object with this. Then define a process that pulls items from the queue and processes them and allow SSIS to determine who many instances of that process to instantiate depending on the spec of the machine it is running one.
A second best would be to create an object in SSIS for each instance of the process that I would want to run and once and have to manage the degree of parallelism manually.
Any ideas? Is SSIS flexible enough, or am I doomed to a lifetime of click and drag? Thanks,
Mark
Mark Roworth