Hi, below is a basic proposal for a new SSIS package (not mine). I am relatively new to SSIS and in the learning so far I have really enjoyed what SSIS has to offer. So for me the below concept will work but just feels like way to many SQL tasks.
Are there any really good SSIS tasks that would effectively do the same or similar or better job to anything listed below?
Custom Data Loader Design
There will need to be the following:
- Several objects in the data warehouse:
- A configuration table that holds details of the child packages to be run
- An event logging table
- An error logging table
- A load logging table
- One SQL Agent job per loader:
- This will call the logging SSIS package (parent) and pass an identifier for the loader SSIS package (child) as a package parameter to the parent package.
- One logging SSIS Package that can be used for all loaders:
- Retrieves the details of the child package (and, potentially, other details) from the configuration table using the package parameter passed in (SQL Task)
- Creates the event log record (SQL Task)
- Creates the load log record (SQL Task)
- Calls the child SSIS package (SQL Task)
- Logs any record counts in the load log returned by the child package and completes the load log record (SQL Task)
- Enters any errors returned from the child package in the error log (SQL Task)
- Completes the event log record (SQL Task)
- One data flow SSIS package per loader:
- Perform any pre-load tasks that are needed and are applicable to more than one of the data flows, e.g. retrieving the last run date for the loader (SQL Task)
- One sequence container per table to be loaded:
- Retrieve any values that are required before execution of the data flow and are specific to this data flow, e.g. retrieving the maximum ID value from the target table (SQL Task)
- A truncate statement to clear down the staging table (SQL Task)
- A Data flow task to transfer the source data into a staging table which captures the number of records transferred and any errors generated
- A SQL command to truncate the model table if this is a bulk loader (SQL Task)
- A SQL command to transfer the data from the staging table to the model table which captures the number of records inserted/updated and any errors generated.
- Update the parent variables with numbers of records and errors (script task)