We have tasked to load the Centers for Medicare & Medicaid Services(CMS) NPI file...or otherwise know as the "National Plan and Provider Enumeration System(NPPES)" file. The file is created and posted as a monthly full file cut and then weekly incrementals. We are attempting to load the full monthly file which is 5.7GB. Soooo leads me to my questions...plural...
- How much space do I need to allocate to my SQL Server Database and Table? 5.7GB or is there overhead?
- For efficiency purposes, what do I need to do to my SSIS Package that will run this efficiently? And the most timely
- Should I load this with SSIS or some other means because of the huge volume?
- Do I need to explicitly define my data fields on my Flat File Connection Manager? I think I saw somewhere that you should explicitly define your data columns and lengths to avoid any overhead
If there are any additional suggestions, please let me know.
Thanks for your review and am hopeful for a reply.