Hi everybody,
I need some assistance with an SSIS task which uses a For Loop.
Essentially I am making 70k individual API Calls inside a For Loop. So needless to say, it is time consuming task. Likely 12-15 hours.
Each API call extracts 1 row of data and is sent to a SQL Database.
I continue to hit unforeseen errors at different stages during the process. Most of the time, it is from a failure to connect to the SQL Database.
Can anybody recommend to me a strategy to counter?
I've tried Checkpoints as per below article. This doesn't seem to work, because a For Loop is considered 1 task when it completed. Rather than 70k tasks.
https://www.mssqltips.com/sqlservertip/1408/integration-services-checkpoints-to-restart-package-from-failure/
I've also played around with the event handler, but I don't know how to use the event handler to re start the For loop at the position where it failed.
Another idea, was if I could possibly cache the 70k of rows and send them to the SQL Database all at once?
Any help would be greatly appreciated.
Thank you.
Cameron