I'm new to SSIS and C# so I'm still getting to grips with all of this.
We pull XML data over a HTTPS link however when the data becomes too big the connection times out.
After researching online I came across a suggestion using a Script task to retrieve the XML data and save it to a local file or in a package variable, which I can then read into the XML Source. I have tried to implement it by adding a Script task before the Data Flow task and pass the XML data to a package variable.
From a website online, I added this C# script in the Script task to pass the XML data to a package variable:
However the Script task fails because of the same issue of it timing out due to data size.
How can I implement a solution so the Script task does not time out? Or is there another way around this?
Thanks in advance.
We pull XML data over a HTTPS link however when the data becomes too big the connection times out.
After researching online I came across a suggestion using a Script task to retrieve the XML data and save it to a local file or in a package variable, which I can then read into the XML Source. I have tried to implement it by adding a Script task before the Data Flow task and pass the XML data to a package variable.
From a website online, I added this C# script in the Script task to pass the XML data to a package variable:
#region Namespaces using System; using System.Net; //Added using Microsoft.SqlServer.Dts.Runtime; //Added #endregion namespace ST_3831315ea9bb45e9b1c2f2e9addad540 { [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute] public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase { public void Main() { // Declare variable that will hold the xml document received by the API string xmlDoc = String.Empty; try { // Get the URI from the variable string url = Dts.Variables["User::URI"].Value.ToString(); //Create a Web Client using (WebClient client = new WebClient()) { //Download the xml document as a string xmlDoc = client.DownloadString(url); } // Set the value of the xmlDocument to the string that was just downloaded Dts.Variables["User::xmlDoc"].Value = xmlDoc; Dts.TaskResult = (int)ScriptResults.Success; } catch { Dts.TaskResult = (int)ScriptResults.Failure; } } #region ScriptResults declaration /// <summary style="box-sizing:inherit;"> /// This enum provides a convenient shorthand within the scope of this class for setting the /// result of the script. /// /// This code was generated automatically. /// </summary> enum ScriptResults { Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success, Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure }; #endregion } }
However the Script task fails because of the same issue of it timing out due to data size.
How can I implement a solution so the Script task does not time out? Or is there another way around this?
Thanks in advance.