I receive files (txt and csv files)from vendor each month and I need to import the files into SQL Db.
These files columns can change each month.
I am trying to coming up with a way to automate this process End - to - End.
I thought about some of the options and I am not sure which of this solution can help me to make this process end to end.
1. Bulk Insert : The issue with this is the destination table must already exists and since the file columns can change, I am not sure how the table can be precreated.
2. BCP: needs format file and not sure if there is a way to create format file based on the text file data. BCP can create format file for the table but not sure about text file.
3. OpenRowSet: using sql 64 and needs OLED drivers to be installed and format file is also needed.
4. BIML: I thought about BIML but as much as I understand BIML, it looks like we can use BIML to autogenerate packages. I am looking for more End-to-End solution, so, I want the automation of ssis package creation as well as execution of the package.
The way I am implementing this todays is using powershell to read the first line in the file(column headers) and creating the staging table and then using bulk Insert to import the data.
I am not sure if this is the only option and if there is anything I can do make this process better.
Hope it Helps!!
