Hello,
I have a requirement where Data comes in flat file format. This data need to go through various step of transformations and lookup . I have broke down the process in steps. 1. Stage 2. Collect Needed Data 3. Validate Data 4. Store Them in incremental order for Future use. 5. Generate More transaction related to incoming rows. 6 Send it to Target system.
Through out the process data will be going across many tables and servers and i have following requirement.
1. I need to create process so that in case of failure if data has not gone through all steps and in to target system i need process to start from where it left off and finish processing the data to target. ( Requirement ask not to use Check Point in SSIS)
2. In case failure require to process that same file again then all data should be wipe Cleaned in step 1 to 5 and new file should process
I have looked in to batch processing so in all steps batch ID will be there but need better suggestions. Please let me know if you have any suggestion or route i should avoid.