Hi,
I have to extract data from multiple flat files into a single table which will keep coming twice in a minute into a folder. As soon as a file comes I have to extract the data and I have to also delete the files once their data has been extracted. Maximum file size will be around 25 MB (~ 120 million records).
The file names will be in a consistent format which is pre-decided. I will be using SQL Server 2005 Ent Edition.
- How can I apply parallel processing of file in this scenario? I mean if I am getting 2 files per minute and if I scehdule my SSIS job to run every 2 minutes, I would like to process 4 files at a time in parallel.
- In next run , I will have new files, so how will I restrict SSIS to extract data from the older files along with these new files because I had already extracted the data from the older files in the previous run. I know that I can use FileSystemTask to move/delete the files after extract but what if new files are already placed before this task is performed and then they might also get moved/deleted?
Your suggestions will be helpful.
Regards.
'In Persuit of Happiness' and ..... learning SQL.