Gurus!
I am trying to build a solution for accepting input files coming in from a file feed, reading data from files contained in them, and transforming them to a target text file in a predefined layout format. I don't need to load any of that into a database.
I am eagerly exploring the use of SSIS for this solution. But I am trying to perform a fitment analysis on SSIS to check whether it would be needed for my purpose. Else I could go on to build some kind of custom ETL solution.
First off, I wish to know whether is a threshold limit on input file size beyond which only SSIS would be useful. I mean, do I need to consider SSIS only for input files whose size is above some x MB, and a custome ETL technique if it is below that?
The file feeds are sent by multiple clients from multiple locations, in varying sizes and frequencies. Some clients send one input file measuring 400 MB every 3 days, some others send four 1 GB files every day, certain others send one 10 GB file every day, and yet others send 4500 files measuring upto 150 MB every day.
I am absolutely sure that SSIS would be useful given the size and frequency of my file feeds, but if there is a threshold data size, only beyond which SSIS would be useful, then should I design some sort of custom ETL (using the task parallel library or something like that) for the smaller and less frequent files, and another solution with SSIS for the heavier and more frequent files? One of my key requirements is processing speed. So does this idea of making 2 separate transformation solutions make sense?
Apart from the above if there are any other considerations I do not know of, but my Gurus know very well, please let me know. I have played around with SSIS, made some basic packages, but never done anything of this scale before.
Looking up to my Gurus on this forum for wisdom in this matter!
Novice Kid