Stored procedure for data importing from flat files. Need some tips.

April 19, 2014, 3:41 am

≫ Next: SSIS Expression explanation

≪ Previous: error when trying to use excel source editor in SSIS 2012

$

0

0

Hello everyone,

I am currently working on creation of SP to automate the process of importing flat data into SQL Server.

I want you to give me some useful advice on the best way how to realise, taking into account things I need:

• File path and name, delimeter sign, table name as variables.
• Logging process of importing in one file for every import task (errors and output results like in bcp)
• Process flat files with different column order
• Check if data we are trying to import already exists in database (don't replicate)
• Batch import
• Error cathing

Okay, now my thoughts. I would use either bcp or BULK INSERT.

Declare delimeter, source file, table as variables.

To identify column order I think we should parse first row with names and use intermediate table from which we then INSERT...SELECT into destination table in the right order.

Check if data already EXISTS before importing.

Still don't know about logging in one file with timeline. Any other errors that may occur?

Any advice would be appreciated. Thank you in advance.

---

SSIS Expression explanation

April 21, 2014, 10:55 pm

≫ Next: Runtime error while AttachPathAndPropagateNotifications()

≪ Previous: Stored procedure for data importing from flat files. Need some tips.

$

0

0

ISNULL(FUND_DETL_DESC_SK) || FUND_DETL_DESC_SK == 0) ? -1 : FUND_DETL_DESC_SK

Runtime error while AttachPathAndPropagateNotifications()

February 10, 2009, 2:14 am

≫ Next: Why is my Package just stopping when going from one Execute SQL Task to another Execute SQL Task

≪ Previous: SSIS Expression explanation

$

0

0

Hi!

I am trying for "replace <column>" expression in a "Derived Column" transformation operator(programatically).
Before this, I tried this through SSIS Business Intelligent Studio and it worked! So, I moved towards programatical approach.
In my program, while connecting Derived Column with the FlatFileDestination, I am getting run time error:

{"Exception from HRESULT: 0xC004705F"}

at AttachPathAndPropagateNotifications()

// Map a path between the transformation component to the FlatFileDestination

try

{

this.dataFlowTask.PathCollection.New().AttachPathAndPropagateNotifications(

this.TransformationDataFlowComponent.OutputCollection[0], this.DestinationDataFlowComponent.InputCollection[0]);

}

catch (Exception e)

{

System.Console.WriteLine(e.Message);

}

My Expression is:(mm/dd/yyyy)
SUBSTRING(purchase_date,4,2)+"/"+SUBSTRING(purchase_date,1,2)+"/"+SUBSTRING(purchase_date,7,4)

And my transformation component code is:

public IDTSComponentMetaData90 AddDerivedColumn(ref MainPipe dataFlowTask, ref IDTSComponentMetaData90 sourceDataFlowComponent)

{

//Note that MainPipe and IDTSComponentMetaData are passed as references

// Add the component to the DataFlow task.

this.addderivedcolumn = dataFlowTask.ComponentMetaDataCollection.New();

// Set component's stock properties.

this.addderivedcolumn.ComponentClassID = "DTSTransform.DerivedColumn";

this.addderivedcolumn.Name = "DerivedColumn";

this.addderivedcolumn.Description = "Adds a derived column";

CManagedComponentWrapper instance = this.addderivedcolumn.Instantiate();

instance.ProvideComponentProperties();

// Attach path between the flat file source components output, and the Sort Component's Input.

dataFlowTask.PathCollection.New().AttachPathAndPropagateNotifications(

sourceDataFlowComponent.OutputCollection[0], this.addderivedcolumn.InputCollection[0]);

this.addderivedcolumn.InputCollection[0].ExternalMetadataColumnCollection.IsUsed = false;

this.addderivedcolumn.InputCollection[0].HasSideEffects = false;

IDTSVirtualInput90 vInput = this.addderivedcolumn.InputCollection[0].GetVirtualInput();

foreach (IDTSVirtualInputColumn90 vColumn in vInput.VirtualInputColumnCollection)

{

//get a handle into the input column and specify the transformation

if (vColumn.Name == DateTimeColumnName)

{

IDTSInputColumn90 col = instance.SetUsageType(addderivedcolumn.InputCollection[0].ID, vInput, vColumn.LineageID, DTSUsageType.UT_READWRITE);

col.Name = vColumn.Name;

IDTSCustomProperty90 Property = col.CustomPropertyCollection["Expression"];

Property.Value = GetExpressionString(DateTimeFormat, DateTimeColumnName);

Property = col.CustomPropertyCollection["FriendlyExpression"];

Property.Value = GetExpressionString(DateTimeFormat, DateTimeColumnName);

}

}

this.addderivedcolumn.OutputCollection[0].TruncationRowDisposition = DTSRowDisposition.RD_FailComponent;// RD_NotUsed;

this.addderivedcolumn.OutputCollection[0].ErrorRowDisposition = DTSRowDisposition.RD_FailComponent;// RD_NotUsed;

return this.addderivedcolumn;

}

Please suggest a solution/way to handle such type of situation.

What is a correct way of connecting between transformation component and destination component?
When I see in BIDS, Derived Column OutputColumn Collection Count is 0. So, I wonder, how is it passing the output to destination component?

Regards,
Mandar.

---

Developer

Why is my Package just stopping when going from one Execute SQL Task to another Execute SQL Task

April 16, 2014, 12:01 pm

≫ Next: SQL Server Integration Services 11.0 not showing

≪ Previous: Runtime error while AttachPathAndPropagateNotifications()

$

0

0

I have one flow constraint going for my one Execute SQL Task to another Execute SQL Task. It's executing #1 and successfully and just stops there rather than continuing on to Execute SQL Task #2. Why would it just stop? My constraint is NOT conditional...just a straight ole flow constraint.

Any help is greatly appreciated and Thanks in Advance for your help.

PSULionRP

SQL Server Integration Services 11.0 not showing

April 21, 2014, 6:15 am

≫ Next: SSIS 2012 Script Task Debugging not working (VSTA Popup but no script displayed in IDE)

≪ Previous: Why is my Package just stopping when going from one Execute SQL Task to another Execute SQL Task

$

0

0

To whom it may concern,

I have been using SSIS 2008 R2 without a problem.  I then installed VS 2012 and migrated my SSIS packages from VS 2010 to VS 2012.  I also installed the BI Components (SSDTBI_VS2012_x86_ENU) in VS 2012,

All the SSIS packages runs fine within VS 2012.  As soon as I run the packages from the command line (i.e. "C:\Program Files\Microsoft SQL Server\110\DTS\Binn\dtexec.exe") the packages referencing any lookup objects provides the following error: 

Error: 2014-04-21 13:20:15.85
   Code: 0xC000F427
   Source: Update Category SSIS.Pipeline
   Description: To run a SSIS package outside of SQL Server Data Tools you must install Derived Column of Integration Services or higher.
End Error
Error: 2014-04-21 13:20:15.85
   Code: 0xC000F427
   Source: Update Category SSIS.Pipeline
   Description: To run a SSIS package outside of SQL Server Data Tools you must install Lookup [Table Name] of Integration Services or higher.
End Error

I checked my services on my Local machine and the following integration service is the only one available:

• SQL Server Integration Service 10.0 (i.e. SQL Server Integration Service 11.0 is missing)

I presume this is the problem.  

I will appreciate it if someone can assist with this problem.  I have also tried to repair MS-SQL 2012 (Express Version) to no avail (Shared Features).

Kind Regards,

SSIS 2012 Script Task Debugging not working (VSTA Popup but no script displayed in IDE)

April 16, 2014, 12:25 pm

≫ Next: Error code: 0x80040E37

≪ Previous: SQL Server Integration Services 11.0 not showing

$

0

0

Hi,

I am trying to debug 2012 SSIS Package but for some reason its not working. Basically when I run package (64bit mode) it pop up Script IDE but never brings up Script (see below). Usually when debugging of script starts it should break execution at the code.

Anybody experienced this issue with SSIS 2012 ?

Thanks,

Nayan

---

Visit My Blog (Home of BI Articles) 

Error code: 0x80040E37

April 16, 2014, 12:26 pm

≫ Next: float [DT_R4] and SQL

≪ Previous: SSIS 2012 Script Task Debugging not working (VSTA Popup but no script displayed in IDE)

$

0

0

SSIS package "finance_stage_to_dm_1.dtsx" starting.

Information: 0x4004300A at DFT Load teams fmprod_v_fund_detail_desc, SSIS.Pipeline: Validation phase is beginning.

Error: 0xC0202009 at DFT Load teams fmprod_v_fund_detail_desc, OLE_SRC stg_fmtmprod_v_fund_detail_desc [1]: SSIS Error Code DTS_E_OLEDBERROR.An OLE DB error has occurred. Error code: 0x80040E37.

Error: 0xC02020E8 at DFT Load teams fmprod_v_fund_detail_desc, OLE_SRC stg_fmtmprod_v_fund_detail_desc [1]: Opening a rowset for "[dbo].[STG_FMTMPROD_V_FUND_DETAIL_DESC

]" failed. Check that the object exists in the database.

Error: 0xC004706B at DFT Load teams fmprod_v_fund_detail_desc, SSIS.Pipeline: "component "OLE_SRC stg_fmtmprod_v_fund_detail_desc" (1)" failed validation and returned validation status "VS_ISBROKEN".

Error: 0xC004700C at DFT Load teams fmprod_v_fund_detail_desc, SSIS.Pipeline: One or more component failed validation.

Error: 0xC0024107 at DFT Load teams fmprod_v_fund_detail_desc: There were errors during task validation.

SSIS package "finance_stage_to_dm_1.dtsx" finished: Failure.

float [DT_R4] and SQL

April 17, 2014, 1:39 am

≫ Next: Newbie Question - from an old DTS dinasour

≪ Previous: Error code: 0x80040E37

$

0

0

Hi guys, I'm stuck in an huge projject just because I'm having problem moving data from CSV file to SQL. The colum that making me crazy is a float [DT_R4]. I tried in SQL with dec, int, real...but nothing. It returns me always a mistake and I'm still struck in the middle of nowhere. I'm going to read something about derived columns but I dunno if the right choice...

---

Newbie Question - from an old DTS dinasour

April 22, 2014, 4:04 am

≫ Next: Strange Behaviour importing data

≪ Previous: float [DT_R4] and SQL

$

0

0

Hi folks,

Please bare with me. After a long break from ETL programming with SQL Server, I'm back! Looking forward to seeing the cpabilities of SQL Server's SSIS.

I'm going through a CBT tutorial and have a couple of basic questions, hope you all can help. 

1) Is there a way to change every connection in a package? What does one do if they are moving a package from one environment to another?

2) Fro the flat file connections under connection manager, (same question) is there a way to redirect these connections if the file has been moved in the directory?

If you have not guessed, I have the tutorial files and the SSIS package definition does not match my environment, I need to change all the OLE connections to the same Instance/DB and all the flat files to be from a directory matching my environment.

Yes, I can click into each SQL Task and change the connection manager, there are only five or six. But what do folks do to automate this.

The flat files, however, I see no way using properties to change the file path. Is there a way to do this? My solution at present is to recreate the FlatFileConnection, which means I have to redefine the delimeters, everything. Please tell me there is a better way!?

Thanks all

John

Strange Behaviour importing data

April 22, 2014, 4:48 am

≫ Next: Problem in uploading files on FTP

≪ Previous: Newbie Question - from an old DTS dinasour

$

0

0

Hi guys, I got a strange behaviour on SSIS moving data from a CSV file to SQL . Two columns, one string and one number, I must convert the string from FT to string unicode. Behaviour ( we are talking about thirthy csv imported with a for each loop): for hundreds row it act correctly Pane 10 , Pane 20, Pane 30, Pane 22 etc..but for eight rows ( always the same) it retrieve 0 Pane20, 0 Pane12...just putting 0 in the first column and the fisrt and second column crossed togheter in the second column..any idea why? Obviously I checked the CSV and there is nothing different between these rows and the other rows..

Problem in uploading files on FTP

April 22, 2014, 4:30 am

≫ Next: merge PDF files using script task

≪ Previous: Strange Behaviour importing data

$

0

0

I have to upload a .txt file on the FTP server , the connection was created successfully and it's also sending the file but data is not transferred. Please suggest .

 

---

Binay Tiwari For Classic ASP http://classicactiveserverpages.blogspot.com/

merge PDF files using script task

April 7, 2014, 9:25 am

≫ Next: Error when Saving Package - Saving Fails when I try to Sign the Assembly - BIDS 2008

≪ Previous: Problem in uploading files on FTP

$

0

0

I am trying to merge multiple PDF files into one PDF file. I have around 5000 folders and each folder has around 10 to 15 PDF's in it. Before I m export those PDF's I need to merge multiple PDF files in sub folders and export. Is there a way in SSIS using script task where we can merge those files looping through all the folders and sub folders. Any help much appreciated.

Error when Saving Package - Saving Fails when I try to Sign the Assembly - BIDS 2008

April 22, 2014, 6:33 am

≫ Next: Dynamically pick the table names in data flow task SSIS

≪ Previous: merge PDF files using script task

$

0

0

Hello,

When I attempt to sign an assembly in a Script Task, the package will not save. I receive an error:

TITLE: Microsoft Visual Studio
------------------------------

Failure saving package.

------------------------------
ADDITIONAL INFORMATION:

'', hexadecimal value 0x07, is an invalid character. Line 2, position 10. (Microsoft.SqlServer.ManagedDTS)

------------------------------
BUTTONS:

OK
------------------------------

===================================

Failure saving package. (Microsoft Visual Studio)

------------------------------
Program Location:

   at Microsoft.DataTransformationServices.Design.Serialization.DtrDesignerSerializer.SerializePackage(IDesignerSerializationManager manager, Package package, TextWriter textWriter)
   at Microsoft.DataTransformationServices.Design.Serialization.DtrDesignerSerializer.SerializeComponent(IDesignerSerializationManager manager, IComponent component, Object serializationStream)
   at Microsoft.DataWarehouse.Serialization.DesignerComponentSerializer.Serialize(IDesignerSerializationManager manager, Object value)
   at Microsoft.DataWarehouse.VsIntegration.Designer.Serialization.DataWarehouseDesignerLoader.Serialize()
   at Microsoft.DataWarehouse.VsIntegration.Designer.Serialization.BaseDesignerLoader.Flush(Boolean forceful)
   at Microsoft.DataWarehouse.VsIntegration.Designer.Serialization.BaseDesignerLoader.Flush()
   at Microsoft.DataWarehouse.VsIntegration.Designer.Serialization.DataWarehouseContainerManager.OnBeforeSave(UInt32 docCookie)

===================================

'', hexadecimal value 0x07, is an invalid character. Line 2, position 10. (Microsoft.SqlServer.ManagedDTS)

------------------------------
Program Location:

   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.Throw(String res, String[] args)
   at System.Xml.XmlTextReaderImpl.Throw(Int32 pos, String res, String[] args)
   at System.Xml.XmlTextReaderImpl.ThrowInvalidChar(Int32 pos, Char invChar)
   at System.Xml.XmlTextReaderImpl.ParseCDataOrComment(XmlNodeType type, Int32& outStartPos, Int32& outEndPos)
   at System.Xml.XmlTextReaderImpl.ParseCDataOrComment(XmlNodeType type)
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.XmlLoader.ParsePartialContent(XmlNode parentNode, String innerxmltext, XmlNodeType nt)
   at System.Xml.XmlLoader.LoadInnerXmlElement(XmlElement node, String innerxmltext)
   at System.Xml.XmlElement.set_InnerXml(String value)
   at Microsoft.SqlServer.Dts.Tasks.ScriptTask.ScriptTask.SaveToXML(XmlDocument doc, IDTSInfoEvents events)
   at Microsoft.SqlServer.Dts.Runtime.ManagedWrapper.SaveToXML(Object doc, IDTSInfoEvents100 events)
   at Microsoft.SqlServer.Dts.Runtime.Package.SaveToXML(String& packageXml, IDTSEvents events)
   at Microsoft.DataTransformationServices.Design.Serialization.DtrDesignerSerializer.SerializePackage(IDesignerSerializationManager manager, Package package, TextWriter textWriter)

I have verified I have .NET 4.5 installed.  I uninstalled it, then reinstalled the current release.  The error only occurs when I try to sign the assembly, with or without a password.  In addition, I've uninstalled SQL Server 2008 r2 (along with BIDS and everything else) and have reinstalled.  I still receive the same error.  The steps to recreate this error are:

Create a completely brand new SSIS project  

Add a script task

Sign the assembly

Try to save the package, it fails.

Is anyone able to point me in the right direction? Thanks

Dynamically pick the table names in data flow task SSIS

April 22, 2014, 7:12 am

≫ Next: SAP BW to SQl Server import and export wizard

≪ Previous: Error when Saving Package - Saving Fails when I try to Sign the Assembly - BIDS 2008

$

0

0

Hi All,

I want to create a SSIS package which loads the data to a table on the other server every day. I have around 250 tables to load everyday and source and destination table names are available in a metadata table, table names have to read from the metadata table and data should be loaded. Is there a way that we can configure the source and destination table names dynamically in Data flow task?

I am newbie to SSIS can any help with the solution for this problem.

SAP BW to SQl Server import and export wizard

June 16, 2013, 11:17 pm

≫ Next: What port(s) should be opened in firewall for dtexec (SQL Server 2008R2)?

≪ Previous: Dynamically pick the table names in data flow task SSIS

$

0

0

I am trying to load data from SAP BW system to SQL server. Using management studio, But i am getting this error any idea?

---

pulikondas

---

What port(s) should be opened in firewall for dtexec (SQL Server 2008R2)?

April 22, 2014, 8:12 am

≫ Next: New to SSIS - Need help converting date

≪ Previous: SAP BW to SQl Server import and export wizard

$

0

0

We have encountered a firewall issue that prevents dtexec from connecting from a SQL Server to a file server and our security team require a specific port number to open to allow dtexec to connect but I can't find what port dtexec uses, can anyone help please?

---

Michael MacGregor, Senior SQL Server DBA

New to SSIS - Need help converting date

April 21, 2014, 8:04 am

≫ Next: SSIS Package will only run as SQL Agent Job when I have remote desktop to server open.

≪ Previous: What port(s) should be opened in firewall for dtexec (SQL Server 2008R2)?

$

0

0

Hello everyone and thanks for your help in advance. I am very new to SSIS and am working with a flat file data source with a column, called Column16 that has dates in mm/dd/yy format. Also, a few of the rows are blank and I receive an error. I have an expression SUBSTRING([Column 16],1,2) + "-" + SUBSTRING([Column 16],4,2) to handle the month and date, but don't know how to create a condition if the row is blank or how to change the year column into the correct format.  Any help would be appreciated.  Thanks.

SSIS Package will only run as SQL Agent Job when I have remote desktop to server open.

April 22, 2014, 7:51 am

≫ Next: Microsoft.SqlServer.ScriptTask assembly not in my library

≪ Previous: New to SSIS - Need help converting date

$

0

0

Hey guys, so I have another problem to add to the already massive 'SSIS/SQL Server Agent Job' pile. After days of searching, I can't seem to find anything specific to my problem though.

The setup is as follows: a SSIS package that refreshes and saves excel files that are hosted on a server. The package runs fine on the local machine, using BIDS on the server, and will even work as a SQL Agent Job on the server IF there is a remote desktop connection to the server. To elaborate, if I simply run the job as you would normally do it will fail and give the below error. If I run the job while either myself, or a different machine, has a remote desktop connection to the server where the job is scheduled - it will run successfully.

Below is the error from the History File of the job. Any help would be greatly appreciated.

  Source: Refresh Excel and Save      Description: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Runtime.InteropServices.COMException (0x8000401A): Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 8000401a.     at ST_79772452677f4de1852d5ffbba3e5232.csproj.ScriptMain.ExcelRefresh(String FileName)     at ST_79772452677f4de1852d5ffbba3e5232.csproj.ScriptMain.Main()     --- End of inner exception stack trace ---     at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)     at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)     at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)     at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)     at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)     at System.Type.InvokeMember(String name, BindingFlags invokeAttr, Binder binder, Object target, Object[] args, CultureInfo culture)     at Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTATaskScriptingEngine.ExecuteScript()

UPDATE:

In my previous post the Identity in DCOM Config file for Microsoft Excel was set as The Interactive User. The job was working only when I had a remote connection to the server open.

If I set the Identity to 'This User' and use the username and password of the server login account, it will work as a scheduled job without needing an open remote connection to the server. So it works, great! but I have reservations setting this for all instances of Excel for the server. I'm sure other users have different accounts they use for running Excel. Any suggestions around this?

Microsoft.SqlServer.ScriptTask assembly not in my library

April 14, 2014, 1:33 am

≫ Next: How can apply configure file in ssis by using script task ?

≪ Previous: SSIS Package will only run as SQL Agent Job when I have remote desktop to server open.

$

0

0

Hi,

Apologies as this is a repost but inmy experience one wrong answer and I don't get any more responses. I have SQL 2012 SP1 installed but have the problem below 

I'm trying to add the following script to a sript task item on my data flow (SQL 2012):

#region NamespacesusingSystem;usingSystem.Data;usingMicrosoft.SqlServer.Dts.Pipeline.Wrapper;usingMicrosoft.SqlServer.Dts.Runtime.Wrapper;usingMicrosoft.SqlServer.Dts.Runtime;usingSystem.Windows.Forms;usingMicrosoft.SqlServer.Dts.Pipeline;usingSystem.IO;#endregionpublicclassScriptMain{    publicvoidMain()    {                using(StreamReader
r =newStreamReader(Microsoft.SqlServer.Dts.Variables["User::CurrentFile"].Value.ToString()))        {           Microsoft.SqlServer.Dts.Variables["User::strCurrentFileColumnNames"].Value= r.ReadLine();           MessageBox.Show(Microsoft.SqlServer.Dts.Variables["Uer::strCurrentFileColumnNames"].Value.ToString());                       }    }}

I get a red squiggly under Variables saying the type or namespace does not exist in Microsoft.Sqlserver.Dts (are you missing an assembly or reference?)

I can see from here:

http://technet.microsoft.com/en-us/library/ms135941.aspx

"To successfully compile the sample, you need to add a reference to the Microsoft.SqlServer.ScriptTask assembly"

I've done this but I get another red squiggly here saying the type or namespace does not exist in the namespace Microsoft.SqlServer (are you missing an assembly or reference?)

I guess I am missing one - which one and  how do I add it to my machine?

Thanks,

Dan

How can apply configure file in ssis by using script task ?

April 23, 2014, 2:26 am

≫ Next: SSIS with Excel Source - Generically specify first worksheet?

≪ Previous: Microsoft.SqlServer.ScriptTask assembly not in my library

$

0

0

I had  two config file in ssis

 1. config file as per QA

2. config file as per Production

 I think apply dynamically by using script task for above file (acc to server)

How can I apply in script task for config file path in C# script

any one provide Helpful code ??

Thanks