Quantcast
Channel: SQL Server Integration Services forum
Viewing all 24688 articles
Browse latest View live

SSIS Jobs Failures Saying Connection Timeout

$
0
0

We have scheduled SSIS jobs Hosted at SSIS server which fails sometimes saying connection timeout. 

What our job does is to pull data from SharePoint and insert into SQL Database . We get below error quite frequently.

Executed as user: AM\EIPSSISPPROXY. Microsoft (R) SQL Server Execute Package Utility Version 11.0.2100.60 for 64-bit Copyright (C) Microsoft Corporation. All rights reserved. Started: 12:00:00 PM Error: 2016-08-28 12:00:43.80 Code: 0xC0202009 Source: RRT_Site_Package Connection manager "IX1consql07.RRT.RRT_APP" Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft OLE DB Provider for SQL Server" Hresult: 0x80004005 Description: "Login timeout expired". End Error Error: 2016-08-28 12:00:43.80 Code: 0xC020801C Source: Data Flow Task OLE DB Destination 1 [212] Description: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "IX1consql07.RRT.RRT_APP" failed with error code 0xC0202009. There may be error messages posted before this with more information on why the AcquireConnection method call failed. End Error Error: 2016-08-28 12:00:43.80 Code: 0xC0047017 Source: Data Flow Task SSIS.Pipeline Description: OLE DB Destination 1 failed validation and returned error code 0xC020801C. End Error Error: 2016-08-28 12:00:43.80 Code: 0xC004700C Source: Data Flow Task SSIS.Pipeline Description: One or more component failed validation. End Error Error: 2016-08-28 12:00:43.80 Code: 0xC0024107 Source: Data Flow Task Description: There were errors during task validation. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 12:00:00 PM Finished: 12:00:43 PM Elapsed: 42.947 seconds. Process Exit Code 1. 

I have tried to look at the all connection timeout values for SSIS but they all are set to no timeout. Still I am getting this error on production server.


Passing Write-Output Powershell variable into SSIS variable

$
0
0
I'm struggling with this, and believe me I've tried to do my research....but all I'm trying to do is pass the variable that comes from a Write-Output command of a variable in a PowerShell script into an SSIS variable so that I can use it for later steps. I've tried setting a new string variable as the "StandardOutputVariable" but that doesn't seem to work.  Can anybody just walk me through the steps here??  I've also tried Write-Host in the PowerShell script and it didn't make a difference.....thank you!

limited toolbox

$
0
0

Hi.  I'm gathering more details but we are rdp'ing into a computer where we are running vs 2008 plumbed to tfs for maintaining our ssis packages.  I went thru this exercise for the first time yesterday.  I checked out a pkg that has plenty of sql tasks in its control flow and plenty of oledb destinations in its many data flows.  

I attempted to insert a sql task in the middle of the control flow only to find that the tool box had none.  But it did have one under maint plans whose editor didnt behave like i expected.  I ended up copying and pasting another sql task and then modifying it.

Then i went to drag a data destination to a data flow only to find that only data sources were in the toolbox.

I rdp to this machine as myself (not admin).  Does the community have any ideas?  I'll post sql version (i think its 2008) etc as i collect more info.

fuzzy components or something else?

$
0
0

Hi.  We run 2012 enterprise.  I've never used fuzzy lookups or groupings before.  All of the data is US based. 

I'm faced on every row coming into my etl with the following challenge...1) is last name A equal to or similar enough to last name B to mark them as possibly the same on this row, 2) pretty much the same with A's and B's addresses.  By possibly I am suggesting that perhaps I'd be able to tweak how strict the rules are based on the results I see.

I don't really know at this point if both addresses go thru some sort of US mail address standardization software but have to assume for now that the answer is no.

I read some older documentation on the fuzzy components and thought immediately that this isn't really a set type of comparison (an input set against some other input set/index) as much as it is a case by case decision with one column being compared against another.

Can the community suggest a direction.  Perhaps a white paper that addresses more closely my particular challenge?

My gut says a soundex variation could be a good approach to the last name part of the challenge.

SSIS pakage failing to redirect error rows (SQL Server 2016 / Visual Studio 2015)

$
0
0

I have a set of 40,000 files that I am importing into a SQL Server 2016 database using an SSIS package I am building in Visual Studio 2015. Each file consists of several hundred lines of data that are delimited by a space, with text data qualified by double quotes ("). This will result in a table of over a million rows of data.

The problem arises when a string of text contains a double quote that then abnormally ends the text string and generates a "column delimiter for column XXX was not found" error.

Normally, I deal with these types of rows by redirecting them to a failures table, but in this case the package is failing on the error without generating any error rows and does not subsequently complete its run.

Can someone help me understand why the rows are not being redirected?

Alternately, can someone help me deal with text strings that contain text qualifier characters within the string?

Here are the errors I receive:

[ELB Log Source [11]] Error: The column delimiter for column "userAgent" was not found.

[ELB Log Source [11]] Error: An error occurred while processing file "D:\mypath\myfile.txt" on data row 2.

[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED.  The PrimeOutput method on ELB Log Source returned error code 0xC0202092.  The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.  There may be error messages posted before this with more information about the failure.

In this case, I have verified that row 2 of myfile.txt does contain a text string that includes a text qualifier character ("), which breaks up the expected column patterns.


I imagine this has something to do with "the error is fatal and the pipeline stopped executing" part of the error message, but I haven't run into this type of behavior until upgrading to SQL Server 2016 and Visual Studio 2015.

Is T-SQL MERGE command more performant than SSIS OLE DB Command transform?

$
0
0

Hi All,

I've explained that the SSIS OLE DB Command executes once for every row that needs to be updated, and that we'd be better off creating a stage table in the destination database, pushing all new and updated rows to that staging table, and then using a T-SQL MERGE stored procedure to "upsert" those rows.  Two others disagreed with me, stating that the MERGE statement also operates row-by-row.  I recall reading a few years ago that T-SQL MERGE is a set-based operation and that, for data warehouse projects, it is nearly always going to be the best choice for an upsert to the destination.  Could somebody please confirm?  They want it done yesterday, of course, but want me to test performance, too.  We've got 30 million rows for the initial insert, and I have no idea many changes (new rows and updated rows) are going on in production on a daily basis, but it's surely got to be in the thousands.  Isn't MERGE the obvious choice?

Thanks,
Eric B.


How to get after slash delimiter?...

$
0
0

How do I get "Folder3" and "123456" from a string like this:

'A:\Folder1\Folder2\Folder3_XYZ_abc de fg\Folder4\123456_I-am-SO-ignorant.pdf'

DECLARE @FILENAME VARCHAR (1000)
SET @FILENAME = 'A:\Folder1\Folder2\Folder3_XYZ_abc de fg\Folder4\123456_I-am-SO-ignorant.pdf'
SELECT  Col1 = SUBSTRING(@filename,1,CHARINDEX(REVERSE(SUBSTRING(REVERSE(@filename),1,CHARINDEX('\',REVERSE(@filename)) ) ),@filename) -1)
		,Col2 = REVERSE(SUBSTRING(REVERSE(@filename),1,CHARINDEX('\',REVERSE(@filename)) -1 ) )
		,Col3 = SUBSTRING(@filename,1,LEN(@filename)-(CHARINDEX('_',REVERSE(@filename))))
		,Col5 = REVERSE(LEFT(REVERSE(@FILENAME), CHARINDEX('_',REVERSE(@FILENAME))-1))

iI tried the above script but I cannot get it right? Any help is appreciated.

Procedure with Insert and Update Data From One Table to Another Table

$
0
0
I am trying to make a procedure for this case:
I have two tables Person with Company, CustomerNbr, StartDate, EndDate, EmployeeID and the same one but without historical data.
1 situation is when Historical.CustomerNbr = T.CustomerNbr but Historical.EmployeeID <> T.EmployeeID 
  then put Historical.EndDate =  T.StartDate
  and enter new record T.Company, T.CustomerNbr, T.StartDate, T.EndDate, T.EmployeeID
2 situation is when Historical.CustomerNbr is null 
  then and enter new record T.Company, T.CustomerNbr, T.StartDate, T.EndDate, T.EmployeeID

I don't know if it does make sense or not

$
0
0

Hi all of you,

I am just wondering if the stored procedures are taking up size in the database (at least size in relevant terms)

MSSQL Crashing and Unsafe assembly?

$
0
0

Our SQL server will randomly crash unexpectedly.

Server details:

Microsoft SQL Server Management Studio11.0.6020.0
Microsoft Analysis Services Client Tools11.0.6020.0
Microsoft Data Access Components (MDAC)6.1.7601.17514
Microsoft MSXML3.0 5.0 6.0 
Microsoft Internet Explorer9.11.9600.18426
Microsoft .NET Framework4.0.30319.34209
Operating System6.1.7601

Error logs also contain the following:

Unsafe assembly 'microsoft.sqlserver.integrationservices.server, version=11.0.0.0, culture=neutral, publickeytoken=89845dcd8080cc91, processorarchitecture=msil' loaded into appdomain 2 (SSISDB.dbo[runtime].1).

From what I can see, we do not have anything in the SSISDB database. It exists but haven't stored any packages within it yet. How can I determine what the unsafe assembly is and could this be the cause of the random crashing?

Thanks,

Noah

Need either VB2010 script/C# script for downloading HTTP Url link

$
0
0

Can anyone help me to download HTTP Url link using VB2010 or C# script.

Right now I am using the below script, but its showing DTS script task : Run time error

Imports System
Imports System.IO
Imports System.Text
Imports System.Windows.Forms
Imports Microsoft.SqlServer.Dts.Runtime

<Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute()> _
<System.CLSCompliantAttribute(False)> _
Partial Public Class ScriptMain
    Inherits Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase

    Enum ScriptResults
        Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success
        Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
    End Enum

    Public Sub Main()

        ' Get the unmanaged connection object, from the connection manager called "HTTP Connection Manager"
        Dim nativeObject As Object = Dts.Connections("SourceUrl_WESTROX1").AcquireConnection(Nothing)

        ' Create a new HTTP client connection
        Dim connection As New HttpClientConnection(nativeObject)


        ' Download the file #1
        ' Save the file from the connection manager to the local path specified
        Dim filename As String = "\\zsa01\EEMS\Dev\COGEN\WESTROX1\WESTROX1.CSV"
        connection.DownloadFile("\\zsa01\EEMS\Dev\COGEN\WESTROX1\WESTROX1.CSV", True)

        ' Confirm file is there
        'If File.Exists(filename) Then
        '    MessageBox.Show(String.Format("File {0} has been downloaded.", filename))
        'End If


        '' Download the file #2
        '' Read the text file straight into memory
        'Dim buffer As Byte() = connection.DownloadData()
        'Dim data As String = Encoding.ASCII.GetString(buffer)

        '' Display the file contents
        'MessageBox.Show(data)


        Dts.TaskResult = ScriptResults.Success
    End Sub

End Class

Thanks in advance.


Send email with attachments from local desktop using SSIS

$
0
0

Hi

I have created a package as below

Step 1) Data flow task --> Source as SQL--> Destination to Excel ( In one folder)

Step 2) Script task with below code

After executing the package step1 is executing fine, but step two is throwing error as shown below (last)

Not sure what I am doing wrong.. Could any one assit how to fix this issue, This is I am trying from my local desktop.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.Net;
using System.Net.Mail;


#endregion

namespace ST_2d0fa26c89c645de8d6ddcd332f534da.csproj
{
    /// <summary>
    /// ScriptMain is the entry point class of the script.  Do not change the name, attributes,
    /// or parent of this class.
    /// </summary>
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
        #region Help:  Using Integration Services variables and parameters in a script
        /* To use a variable in this script, first ensure that the variable has been added to 
         * either the list contained in the ReadOnlyVariables property or the list contained in 
         * the ReadWriteVariables property of this script task, according to whether or not your
         * code needs to write to the variable.  To add the variable, save this script, close this instance of
         * Visual Studio, and update the ReadOnlyVariables and 
         * ReadWriteVariables properties in the Script Transformation Editor window.
         * To use a parameter in this script, follow the same steps. Parameters are always read-only.
         * 
         * Example of reading from a variable:
         *  DateTime startTime = (DateTime) Dts.Variables["System::StartTime"].Value;
         * 
         * Example of writing to a variable:
         *  Dts.Variables["User::myStringVariable"].Value = "new value";
         * 
         * Example of reading from a package parameter:
         *  int batchId = (int) Dts.Variables["$Package::batchId"].Value;
         *  
         * Example of reading from a project parameter:
         *  int batchId = (int) Dts.Variables["$Project::batchId"].Value;
         * 
         * Example of reading from a sensitive project parameter:
         *  int batchId = (int) Dts.Variables["$Project::batchId"].GetSensitiveValue();
         * */

        #endregion

        #region Help:  Firing Integration Services events from a script
        /* This script task can fire events for logging purposes.
         * 
         * Example of firing an error event:
         *  Dts.Events.FireError(18, "Process Values", "Bad value", "", 0);
         * 
         * Example of firing an information event:
         *  Dts.Events.FireInformation(3, "Process Values", "Processing has started", "", 0, ref fireAgain)
         * 
         * Example of firing a warning event:
         *  Dts.Events.FireWarning(14, "Process Values", "No values received for input", "", 0);
         * */
        #endregion

        #region Help:  Using Integration Services connection managers in a script
        /* Some types of connection managers can be used in this script task.  See the topic 
         * "Working with Connection Managers Programatically" for details.
         * 
         * Example of using an ADO.Net connection manager:
         *  object rawConnection = Dts.Connections["Sales DB"].AcquireConnection(Dts.Transaction);
         *  SqlConnection myADONETConnection = (SqlConnection)rawConnection;
         *  //Use the connection in some code here, then release the connection
         *  Dts.Connections["Sales DB"].ReleaseConnection(rawConnection);
         *
         * Example of using a File connection manager
         *  object rawConnection = Dts.Connections["Prices.zip"].AcquireConnection(Dts.Transaction);
         *  string filePath = (string)rawConnection;
         *  //Use the connection in some code here, then release the connection
         *  Dts.Connections["Prices.zip"].ReleaseConnection(rawConnection);
         * */
        #endregion


/// <summary>
        /// This method is called when this script task executes in the control flow.
        /// Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
        /// To open Help, press F1.
        /// </summary>
public void Main()
{


            var theMailer = new MailMessage();
            theMailer.To.Add("myemail@gmail.com");
            theMailer.From = new MailAddress("myemail.com", "Hello");
            theMailer.IsBodyHtml = true;
            theMailer.Subject = "From Report";
            theMailer.Attachments.Add(new Attachment(@"C:\Outputreport.xlsx"));
            theMailer.Body = "<table><tr><td>Report</td></tr></table>";
            var myClient = new SmtpClient("smtp.gmail.com",587);
            myClient.EnableSsl = true;
            myClient.DeliveryMethod = SmtpDeliveryMethod.Network;
            myClient.Credentials = new NetworkCredential("myemail@gmail.com", "Password");
            myClient.Send(theMailer);

            Dts.TaskResult = (int)ScriptResults.Success;
}

        #region ScriptResults declaration
        /// <summary>
        /// This enum provides a convenient shorthand within the scope of this class for setting the
        /// result of the script.
        /// 
        /// This code was generated automatically.
        /// </summary>
        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion

}
}

******************************************************************

Error Message

DTS Task has encountered an exception in user code:

Project name : ST_2d0fa26c89c645de8d6ddcd332f534da

Exception has been thrown by the target of an invocation

 at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
   at Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTATaskScriptingEngine.ExecuteScript()

SFTP transfer rate and SSIS package setting

$
0
0

I have package which invokes a external process controlled by a table to download files from SFTP.

The package after recieving file starts the Table truncate process and runs the Dataflow task to read the file which has been copied to local input processing folder

I would like to know how much time I need to set as delay in a SSIS package at the Dataflow task level if the 

SFTP file download  speed as per bandwidth is -10 MB/second.

say I receive 700 MB file.

Also is there any tool which measures the file copy rate from SFTP accurately.

Also want to know if data transfer rate is measured in bits per second or bytes per second .I have seen tools showing bits per second so if someone says 10 mbps its not 10 mega bytes it is 10 mega bits per second)







is there a precedence when referring to environ vars in the config settings

$
0
0
Hi. I believe there are three types of environ vars in the windows world. When u refer to an environ var in the config dialog of ssis, and the pkg runs, will it look for only system environ vars by the name you specified or is there some sort of precedence that starts with user environ vars and moves up the chain to system and global environment vars?

Deploy SSIS 2014 from VS 2015

$
0
0

Hi!

Is it possible to deploy an SSIS 2014 package to an SQL 2014 (SSIS) instance from VS 2015 with SSDT installed? From what I can read from this page https://msdn.microsoft.com/en-us/library/mt204009.aspx it should be possible but when I try to deploy from the SSIS project in VS 2015 the deployment wizard complains about version conflict due to target server not running 2016. The SSIS package in VS 2015 has target version configured to SQL Server 2014.

Thanks in advance!

Best regards
Lars


Create Function query timeout

$
0
0
Using SSIS TransferSQLServerObject I am transferring UDFs from source to destination. While doing so one CLR based user defined function failing due to timeout. On further investigation I found that the "create function" query spending lot of time in "Memory/CPU" wait and eventually timing out. What would be the reason behind this behaviour?

parallel load

$
0
0

Hello,
We have one package which loads data into database...

Question:
How is it possible to run several of this package at the same time?

Thank you

Exporting data from db using ssis

$
0
0

Hi,

I have around 15-20 tables (with few thousands of rec) in the sql server db, I would like to export each of these tables to separate pipe delimited flat file.

What I did..

1. Got the ESQL task to get all the table names into a object variable

2. Got the foreach loop container to loop through and catch the name of each table into a different string variable

3. In ForEach loop I brought the DataFlowTask 

Here is what I am confused to build logic here... the problem I am facing is with column mapping, If I map for one table, the other table might have diff num of columns and datatype..

I know I need to do some dynamic mapping etc.. just need your help in this.

Thanks,


Neil

Exectute SSIS Package on Azure

$
0
0

Hello,

  I used to create and run SSIS packages on-premise but now we are planning to fully move to cloud. so i was wondering if there is anyway to run SSIS package on a cloud platform (azure).

All the options that i have found was creating a VM on Azure and install on it a SQL Integration service. Do we have any options other than that (e.g. something new with SQL 2016)?

Regards,

Karim Khalifa

want to see the actual connection that is used in a for each loop container

$
0
0

Hello,

i have an SSIS package with an  for each Loop Container that iterates trough several Servers. i want to debug this package. I want to see the actual Connection string (aka. which Server is actual in use) through Debugging. Is there a solution for that?

Viewing all 24688 articles
Browse latest View live