Hi all,
Recently our source system has changed the file structure of CSV files . After the change we are now receiving the feed with huge size of a single CSV file of having 5 GB. We have a basic validations up on the data in feed for checking the data type of the columns and converting all columns from nonunicode to Unicode DataType . These tasks are shown in the below snapshot.
Due to the transformations and because of reading the data from single CSV file the loading of data from that CSV file is taking nearly 4:30 hours to get the data in to a SQL table . But this load time is very bad. So i want all your valuable suggestions on how to speedUp the loading process from the CSV files .
1) Any configuration we need to do in SSIS side to sped up the process ?
2) Will process would be faster if you divide the single large file into multiple files and load the data parallelly to the Sql table .?
3) Will process be faster if I , load the data directly from CSV file in to Sql table using BCP and doing all the tasks like data type checking , converting the datatypes in SqlServer ?. I think this is one of the bad option to pick.
I am waiting for all your valuable suggestions and please suggest me if any other option is available .