So on a weekly basis, I have to sweep a sub-set of our Member data. So my query is built to produce this result set. I then have to take that result set and compare it to a data table which contains all that same information that I have provided to a 3rd party vendor. So I need to take into account the following scenarios...
- Obviously, if the Member is new, then I have to Insert its row to what we'll call table 3rdPartyMember and create a row out on a 3rdPartyMemberAuditTrail Table
- Individually, I also have to manage if any of the following data fields have changed: Name, Birth Date, Addressing Information, Member Plan, Member Termination...and by Individually what I mean is that if the Last Name has changed, then update its row in 3rdPartyMember and Insert a row to 3rdPartyMemberAuditTrail indicating "Last Name Change"...and similarly if Address Line 1 has changed..."Address Line 1 Change"
So I guess my question is this. Should this be done in one whole Stored Procedure invoked by my SSIS Package or should it be done in pieces?
- Create the Temporary Table to house our weekly Member subset extract and result set
- Determine if the row exists on 3rdPartyMember Table and if it does not, Insert it
- If the row exists, determine if there is a First Name Change...Last Name Change...BirthDate change...Address Line 1 change...etc... If there is a change, Update 3rdPartyMember Table and also Insert a row to 3rdPartyMemberAuditTrail indicating the change
Based on all this, I'm just not sure the right way to approach this...meaning should we create one big Stored Procedure to handle everything or each individual data point in my SSIS as an UpSert based on its lookup. I am new to the SSIS World and don't really know what the generally accepted practice is per se of creating and running an SSIS Package with multiple data point update steps or doing so in one big Stored Procedure.
Also....if anyone know of some good YouTubes or web sites that would instruct me as to how to go about doing this, I'd GREATLY appreciate it.
Thanks for your review and am hopeful for a reply.