Hi!
I have a Data Warehouse into which I´m importing data from different sources.
Until now the source has always been sql tables that I have copied data from into staging tables.
The steps has been like:
Step 1: Clear all stage tables
Step 2: Copy data from sql source to sql stage destination.
Step 3: Move data from stage into dimenssion/facts.
If step 2 or 3 fails, I re-run the whole package from step 1 and copy the "same" data again from sources.
Now I´m about to do my first SSIS using file-imports to the stage. I am wondering if someone can advise in what basic steps that should be included.
Say I have a folder containg several csv-files at the same format. I want to import them to a stage-table and then import the stage-data into a fact-table.
My concerns is what is happing if one of the steps is failing. When should the files be deleted/archieved, when the data is safe in the fact-table or already when inserted in the stage-table? What about transaction-handling and so on? A difference from
the sql-approach above is that the source file can´t be regenerated. I will get them ones.
Regards
/ Stefan