I have a series of CSV files that I need to import daily into a SQL server database. I am having a particular issue with a field: LoanNumber. LoanNumber can be either a very long number or text, my issue is the data needs to eventually get into a varchar field and depending on how I try to move it through SSIS, either the text gets Nulled or some of the numbers end up as text in scientific notation.
The origin of these files are CSV files I download. The resulting file is a CSV that uses Code Page 65001 (UTF-8) where some LoanNumber's are text, some numeric and some in scientific notation, the Text is Qualified by "" and comma delimiter is a ','. My goal is to get this field into a SQL Server table as a varchar, which requires Code Page 1252 as part of the transformation.
Part of my issue is the different ways Excel handles the CSV file depending on how it is opened. If I use Windows Explorer and double click on the file name, it will open in Excel with no wizard or prompting, I can widen and change the LoanNumber column to a "Number" type which converts all the scientific notation to numbers and save the file. What is odd though, is even after saving the file using the save button, and without making additional changes, when I close the file Excel prompt's me to save again. If I save with this prompt, Excel saves the CSV as an ANSI 1252 file that is tab delimited (according to SSIS) even though it is still a CSV file.
On the other hand, if I have Excel open and try to open the CSV file, it gives me the text conversion wizard.
My question is this: is there a way in Excel to force a CSV file to be saved in the ANSI 1252, tab delimited format, or is there some other way to save the file that is easy for SSIS to interpret? I have been converting all of my CSV files to Excel files, and using SSIS with Excel data sources, but Excel will implicitly convert some of my numbers to Scientific Notation at times, even if the column is a Number column I've found, so I don't think Excel data sources for SSIS are the right answer.