Hi,
I have two packages:
Package ONE pulls data from source (cloud) to destination (on-prem):
a) Unecrypted tables go to DB1
B) Encrypted tables go DBstg
Package TWOdecrypts data from DBstg to DB1 (within the same instance)
Each package has a sequence container. Inside each sequence container, there are about 16 Data Flow Tasks. Inside each Data Flow Tasks, there are 8-10 tables. The way the package works is: First it gets max date from destination then gets max date from source. After this, it will compare between source and destination tables then it replicates the difference ... SELECT * FROM TABLENAME WHERE (Date > ? and Date <=?). Once replicatiOn is done, it decrypts from DBstg to DB1.
The problem is sometimes, the package fails after after half the tables are populated from the source to the destination or half of the tables are dcrypted from DBstg to DB1. half of the load is done.
The following are not an option:
- Check point is not an option because when the packages are re-run, there are duplicate values inserted. This is due to having multiple tables per Data Flow Task.
- I have multiple Connection Managers (CM) and I cannot use BEGIN/COMMIT TRAN option since this can only be supported within one CM.
- The option to enable MSDTC is not on the table!
My question is, how do I go about implementing a method to delete records
My question is, how do I go about implementing a method to delete records when the packages fails half way after records are inserted into half of the tables? The option to use Script Task inside Data Flow Task is not available. I see OLE DB Command
but it does not look like I can use as much as I can use Script Task to update a table.
Any help really appreciated!
Thanks
IN~