Quantcast
Channel: SQL Server Integration Services forum
Viewing all articles
Browse latest Browse all 24688

SSIS Script Component Destination- split table data to multiple files

$
0
0

I am using a data flow task to connect to oledb source and run sql code which produces millions of rows. Each row is an xml content there is only one column (nvarchar(max)). I connect this to a script component destination, the reason being when the rowcount reaches 10k i want to stop and write to another file.

It works fine when I use a breakpoint, so if there are 100 rows and I want 10 rows in each file when I use a breakpoint it produces 10 files. But, when I run the pacakge without breakpoints it creates random number of files. For ex: it creates 6 files and 3 of those contains 10 rows and the remaining 70 are split in the other files.

I read through the docs and blogs to see if the input buffer is synchronous, and it seems like it is. i.e each row is processed one at a time at least this is what my understanding is, so I am not sure what is the root cause to this problem? Can someone please shed some light. Below is the code I am using to split files in Script Component

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]publicclassScriptMain:UserComponent{StreamWriter writer;int rowCount =0;int rowLimit =50;StringRunFileName="C:\\Users\\Desktop\\"+DateTime.Now.ToString("yyyyMMddHHmmss")+".txt";publicoverridevoidPreExecute(){base.PreExecute();
        writer =newSystem.IO.StreamWriter(RunFileName,true);
        rowCount =0;}publicoverridevoidPostExecute(){base.PostExecute();
        writer.Dispose();}publicoverridevoidInput0_ProcessInputRow(Input0BufferRow){if(rowCount == rowLimit){
            writer.Dispose();
            rowCount =0;RunFileName="C:\\Users\\Desktop\\"+DateTime.Now.ToString("yyyyMMddHHmmss")+".txt";
            writer =newSystem.IO.StreamWriter(RunFileName,true);}var data =System.Text.Encoding.Unicode.GetString(Row.XMLInnerText.GetBlobData(0,Convert.ToInt32(Row.XMLInnerText.Length)));
        writer.WriteLine(data);
        rowCount++;}}


Viewing all articles
Browse latest Browse all 24688

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>