We have multiple CSV files each containing different data and mapping to different tables. These tables also have various relationships which need to be included in the ETL process.
I am thinking how best to go about this as I do not want a separate SSIS package for each.
My thoughts are that we need the following logic:
1. For each CSV file in the file system
2. Get the matching entity name (table)
3. For each row in the current file
4. Retrieve FK IDs using lookups
5. Use the CSV header names to map to the table column names
6. Insert each row
7. Move to next CSV file
Looking for expert advice on how best to tackle this using SSIS? i.e. components to use etc. Does this type of requirement require a script component?