I have a SSIS job that pulls from an oracle table and dumps into a SQL table. But the oracle table has 170 million rows.

What is the best way to only get the unique values from this table? I could run through a sort(and remove duplicates) but that would still have me waiting for the 170 million rows to come in then remove the duplicates and probably take an entire day if not hanging up mid way anyways.

Thanks in advance,

Hi Gurus, 

I have a flat file coming in and I have to change the file name and move to a different folder. This is a Pipe delimited one. I was able to do everything fine until I discovered that my o/p file doesn't contain Header and Footer after executing my pkg. How do I move it to a different folder keeping Header and Footer inside the file.

Please help me with this.

 I have the date in the format of (MM/DD/YYYY) format , i need to transform to (MM-DD-YYYY) format . I used the Derived column transformation to do it ,still its not transforming . My Source of date is from Access DataBase which is of type

date(DT_DATE) .

I used the below formats

Format 1:

(DT_DBDATE)(((RIGHT("0" + (DT_WSTR,2)DAY(ExpDate),2)) + "-" + RIGHT("0" + (DT_WSTR,2)MONTH(ExpDate),2) + "-" + (DT_WSTR,4)YEAR(ExpDate)))

Format 2:

(DT_DATE)((SUBSTRING([ExpDate],0,2) + "-" + SUBSTRING([ExpDate],1,2) + "-" + SUBSTRING([ExpDate],4,2))

        These are not transfroming the dates ,its displaying the same as source . My Destination is Excel File .

I want my Dates to be transformed into format of (MM-DD-YYYY) format .  For instance (3/5/2001 needs to transfomed to 03-05-2001).

     Missing anything?

I have a .NET console application that loads and executes an SSIS package.  The console app is written in C#, targets .NET Framework 4, and has been working successfully for a long time (including various migrations from SQL 2005 to SQL 2008 R2).  We just upgraded to SQL 2012, and now the console app throws an error when trying to run the package. 

Here is the error that SSIS is logging:

Message: The Execute method on the task returned error code 0x80131621 (Mixed mode assembly is built against version 'v2.0.50727' of the runtime and cannot be loaded in the 4.0 runtime without additional configuration information.). The Execute method must succeed, and indicate the result using an "out" parameter.

I can step through my console app's code and see that it does indeed successfully load the package; and then it even successfully updates a user variable and a couple of connectionstrings in the package's Connection Managers.  It throws the error when the Execute method is called.

After some online searching, I added the following to my app.config file, and the whole thing successfully worked.

<startup useLegacyV2RuntimeActivationPolicy="true" >

However, I find this disconcerting, as I feel like it's taking a step backwards.  My console app was already targeting .NET 4.  It was the SQL upgrade that seems to have caused this issue.  Are we really saying that the latest version of SSIS (my app references the ManagedDTS assembly), was compiled against and older version of the framework, .NET 2.0?

The package itself does not have any script tasks or components, just a bunch of derived columns, execute SQL tasks, and some conditional splits.  The source file / connection manager is a Flat File; the destination is our SQL (now 2012) server.  I've even opened up the package in the new SQL Data Tools application and resaved it, thinking it needed to be "upgraded."  No change. 

(That's another issue I have.  Why does SQL 2012 install a Visual Studio 2010 "shell"?  Wasn't this Microsoft's chance to finally get these tools all synched up?  I have VS 2012 installed, then I install SQL 2012, and I get VS 2010... and the resulting mixed bag of dll versions!)

Anyway, unless I have that "useLegacyV2RuntimeActivationPolicy" attribute in the app.config, it fails on the Execute method.

Any details on this would be appreciated.

I need some assistance in regards to the following:

1. How could I make my connection managers Project and NOT package specific?

2. I want to take these project specific connection managers and relate them to SSIS configurations

3. Is there a way to have both SSIS configuration (Config Tables) and store those values in an Environment Variable?

i have a package with a data flow task that has an OLE DB source and OLE DB Destination. In the OLE DB Source I have a SQL Command with Two Consequent Inserts of a Target table with help of Two Joins from a Table from different Database and other being same

My query is returning over 3 million records and every time my package gets to this task it hangs, so the query query in the OLE DB Source executes but when it's about to insert into the OLE DB destination table it hangs at row 9000 something and the task stays yellow and stays at the number of rows count forever not inserting anything in my destination table even after letting it run for a couple of hours.

Now if I were to take this query and run it in management studio it finishes in less than a minute and rows are inserted in that destination table so it's not like the query is not optimized to perform the record retrieval and then insert

Srikanth

Hi Folks,

I have a data flow and 4 OLE DB Source within this. The four stored procedures extract data from several tables around 5-6 million rows per run. These are set to run in parallel to load into a Staging table. Once the package is started only one of the OLE DB Source spid is seen in SSMS when "sp_who2 active"  is run. After 10 -15 minutes other queries  seem to be running. It seems like the package is hanging for hours together.  When the 4 stored procedures are run in SSMS they complete within 35 minutes. But when these are run through SSIS they take 10- 12 hours. When a SQL Trace is run on the SQL server during package execution , the statement we saw was a  "sp_execute 1 "Getdate()-12" , Getdate() " .  Similarly for other 3 stored procs.

The same package took 2 hours one day and it takes 10- 12 hours on other days. We checked the server for CPU and memory and everything seems to be fine.  I read in some blogs when stored procedure is used in OLE DB source the query is executed multiple times.  

The Stored procs have maxdop option of 4.

Let me know your experience and thoughts.

Thanks 

Hi,

I am trying a sample SSIS pakcage to fetch data from ADO.Net source and store Catch Transfrom. I tried with controls it is working fine. But when i try to do same thing using code. I was not able to connect the ADO.Net source and Catch Transfrom control. i am getting Error: Exception from HRESULT: 0xC0048004

Am i missing some thing?

using BI i was able to create the Cache Connection manager and Cach Transform. But using code i am not sure how to add it. Is it cause of the issue? If so how can i add it?

        private void AddIndustryTempDataFlow(Package pkg)
        {
            Executable et = pkg.Executables.Add("STOCK:PipelineTask");
            TaskHost thMainPipe = et as TaskHost;
            MainPipe dataFlowTaskTempTable = thMainPipe.InnerObject as MainPipe;
           IDTSComponentMetaData100 source= GetIndustryInfoSource(pkg, dataFlowTaskTempTable);
           IDTSComponentMetaData100 destination = AddCacheTransform(dataFlowTaskTempTable);
           CreateReleation(dataFlowTaskTempTable,source, destination);
            //****************Map the column between source and destination Start **************
           // Create the design-time instance of the destination.
           CManagedComponentWrapper destDesignTime = destination.Instantiate();
           // The ProvideComponentProperties method creates a default input.
           destDesignTime.ProvideComponentProperties();
           // Get the destination's default input and virtual input.
           IDTSInput100 input = destination.InputCollection[0];
           IDTSVirtualInput100 vInput = input.GetVirtualInput();
           // Iterate through the virtual input column collection.
           foreach (IDTSVirtualInputColumn100 vColumn in vInput.VirtualInputColumnCollection)
           {
               // Call the SetUsageType method of the destination
               //  to add each available virtual input column as an input column.
               destDesignTime.SetUsageType(
                  input.ID, vInput, vColumn.LineageID, DTSUsageType.UT_READONLY);
           }
           //****************Map the column between source and destination End **************
           // Verify that the columns have been added to the input.
           foreach (IDTSInputColumn100 inputColumn in destination.InputCollection[0].InputColumnCollection)
               System.Windows.Forms.MessageBox.Show (inputColumn.Name);
        }
        private void CreateReleation(MainPipe dataFlowTaskTempTable,IDTSComponentMetaData100 source ,IDTSComponentMetaData100 destination)
        {
            // Create the path.
            IDTSPath100 path = dataFlowTaskTempTable.PathCollection.New();
            path.AttachPathAndPropagateNotifications(source.OutputCollection[0],
              destination.InputCollection[0]);
        }
        private IDTSComponentMetaData100 AddCacheTransform(MainPipe dataFlowTaskTempTable)
        {
            Application application = new Application();
            PipelineComponentInfos componentInfos = application.PipelineComponentInfos;
            foreach (PipelineComponentInfo componentInfo in componentInfos)
            {
               System.Windows.Forms .MessageBox.Show (componentInfo.Name + "\t" + componentInfo.CreationName);
            }
            // Add Cache tranform component to data flow task
            IDTSComponentMetaData100 componentSource = dataFlowTaskTempTable.ComponentMetaDataCollection.New();
            componentSource.Name = "IndustryCacheTransform";
            componentSource.ComponentClassID = "DTSTransform.Cache.1";
            return componentSource;
        }
        private IDTSComponentMetaData100 GetIndustryInfoSource(Package pkg, MainPipe dataFlowTaskTempTable)
        {
            // The Application object will be used to obtain the CreationName
            //  of a PipelineComponentInfo from its PipelineComponentInfos collection.
            Application app = new Application();
            //foreach (object a in app.PipelineComponentInfos)
            //{
            //    System.Windows.Forms.MessageBox.Show(a.ToString());
            //}
            // Add a first ADO NET source to the data flow.
            //  The CreationName property requires an Application instance.
            IDTSComponentMetaData100 compRSExecutionIndustryListSource = dataFlowTaskTempTable.ComponentMetaDataCollection.New();
            compRSExecutionIndustryListSource.Name = "RSExecutionIndustryListSource";
            compRSExecutionIndustryListSource.ComponentClassID = typeof(Microsoft.SqlServer.Dts.Pipeline.DataReaderSourceAdapter).AssemblyQualifiedName;
            // Get the design time instance of the component.
            CManagedComponentWrapper instance = compRSExecutionIndustryListSource.Instantiate();
            // Initialize the component
            instance.ProvideComponentProperties();
            // Specify the connection manager.
            if (compRSExecutionIndustryListSource.RuntimeConnectionCollection.Count > 0)
            {
                compRSExecutionIndustryListSource.RuntimeConnectionCollection[0].ConnectionManager =
                  DtsConvert.GetExtendedInterface(pkg.Connections["RSExecutionLogConnection"]);
                compRSExecutionIndustryListSource.RuntimeConnectionCollection[0].ConnectionManagerID =
                  pkg.Connections["RSExecutionLogConnection"].ID;
            }
            // Set the custom properties.
            instance.SetComponentProperty("AccessMode", 2);
            instance.SetComponentProperty("SqlCommand","SELECT EDWG.ENGID,PD.PARTITION_ID AS PartitionID,EDWG.LocalEngagementID AS LocalEngID FROM RSEXECUTIONLOG.DBO.EngDetails_WGServers EDWG with(nolock) " +"INNER JOIN RSEXECUTIONLOG.DBO.ENGMETADATA EM ON EDWG.ENGID=EM.ENGBACKUPID " +"INNER JOIN RSEXECUTIONLOG.DBO.PARTITION_DETAILS PD  ON EM.INDUSTRY=PD.INDUSTRY_ID " +"WHERE EM.BatchID =5 AND EDWG.SQLServerName='edmsrv02\\eauditdw' and DatabaseName='V2FSAJuly8' and SchemaName='AUD166' ");
            // Reinitialize the metadata.
            instance.AcquireConnections(null);
            instance.ReinitializeMetaData();
            instance.ReleaseConnections();
            return compRSExecutionIndustryListSource;
        }

saravanakumar

As my requirement is  import the data from CSV to OLEDB(daily in new table dynamically)

So prepared tasks

1.SQL Task --to create dynamic table by  Property expressions

2.Data flow in For each loop - Collection :For each file Enumerator to select file fully qualified  ==*cx*.csv

before the created Variable :::Tablename (or Tb is same nothing to worry) (String in Package level) and assigned  Evalute expression :True(Below image)

in SQL Task-->

 General Paze:: Result Set :NONE

                       Connection :OLEDB

                       SQL source Type: Direct Input

 Expression paze:

                        Property:SQL source type:  Expression (Check below image)

Now It is showing error String to int 32  ?

Is it correct process do Dynamic Table creation ??

My Ist TASK to Successfully Dynamically create the table in SSIS

So Please Help me

--THANKS
--MADHU

Hi

I used WebService Task to import data from wsdl web link to an xml.The service is made with wsHttpBinding, i think this is the problem , but not sure.

i've set up a HttpConnection , wsdl file overwrite= true, and output to a file connection . I'm using user and password.

The error message is the following:

"[Web Service Task] Error: An error occurred with the following error message: "Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebserviceTaskException: The Web Service threw an error during method execution. The error is: The remote name could not be resolved: 'server-........com'.
   at Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebMethodInvokerProxy.InvokeMethod(DTSWebMethodInfo methodInfo, String serviceName, Object connection)
   at Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebServiceTaskUtil.Invoke(DTSWebMethodInfo methodInfo, String serviceName, Object connection, VariableDispenser taskVariableDispenser)
   at Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebServiceTask.executeThread()".

I'm using SSIS 2008

The service is working with WCF Test Client only with wsHttpsEndpoint.

If someone knows the solution please tell me ! Thank you

Hi,

Few months(3 months) created a package of data flow(OLEDB to Excel)

with

1.SQL task  ---to create Excel sheet

2. data flow  ---Oledb to above excel sheet

for Excel connection ,assigned expression in Excel path: @[Merge_chg_excel_path]
+"Merge_CHG_"
+Right( "0"+(DT_WSTR, 30) month(getdate()),2)
+((DT_WSTR, 2)( DAY( GETDATE()))
+(DT_WSTR, 4) (YEAR (GETDATE())))
+"_"
+@[User::Merge_chg_filename]
+".xlsx"

Then successfully created and excel also created as per expression

Now I got problem to run the package with error of

[Execute SQL Task] Error: Failed to acquire connection "Excel Connection Manager". Connection may not be configured correctly or you may not have the right permissions on this connection.

even  in SQL  Task  by clicking PARSE query also showing  error of HRESULT:0xC0202009

Hence NOW I created new package with new excel sheet ---Its working

Again I assigned  expression in  excel connection  again same problem

Please guide me advice .

Thanks

-MADHU

Hello,

I don't have a very profound knowledge of T-SQL.

I have a temporary table with a series of data result of a SP.

Table looks like this:

WEEK | TAG | AVGVal | Comment | Status
   1    | TG1 |   2        | cmt1        | ok

   2    | TG1 | 3.5       | cmt1        | ok

   3    | TG1 |   2.8     | cmt1        | ok

...

  1    | TG2 |  456      | cmt2        | ok

  2    | TG2  | 500      | cmt2        | ok

and so on.

I need to represent these data in the following way:

TAG | Comment | Status | AVGWK1 | AVGWK2 | AVGWK3 |

TG1 | cmt1        | OK      |     2        |     3.5     |   2.8       |

TG2 | cmt2        | Ok      |               |               |              |

Is there any way to create a view or a temp table and display those data in that way?

Any suggestion is appreciated.

Thanks.

Hi,

    In SSIS Package, How to write a log file which maintains what are the rows failed to insert from flat file to table.

Hi all,

What is the best practise for processing fixed width files that have header and trailer rows?  bearing in mind that in the flat file connection manager when you specify columns everything is out of alignment after the header row?  thanks for your help!

Reading dtsx xml and need PackageFormatVersion for SQL Versions

I have a .net program that reads in the dtsx file. I had it working with SQL 2005 and 2008 R2, but now the xml format has changed for SQL 2012. So now I need to distinguish which version is it reading. The PackageFormatVersion seem to be 2 for 2005, 3 for 2008. Just upgraded my first dtsx to 2012 and it is 6. Anyone know what is the PackageFormatVersion numbering is for the SQL versions? Or even some other way in the dstx I can tell the version. SQL 2014 is around the bend.

I haven't used any SSIS projects in a few months, but when authoring one today I found that there are bugs in the drag/drop while designing that make it almost unusable. In general, clicking and holding on any given item in the design surface will cause it to jitter back and forth until you release the mouse button, or the item will jump to the top left corner of the design surface and not allow you to move it anywhere else. In the second scenario all connectors to the item also disappear when you release the mouse button.

I am using:

VS 2012 11.0.60610.01 Updated 3
SQL Server Data Tools 11.1.30618.1
SQL Server Integration Services Designer 11.0.3369.0
SQL Server Reporting Services Designers 11.0.3369.0
SQL Server Analysis Services Designer 11.0.3369.0

Hi,

I have an Oracle destination with one column called memo that has a size of 4000bytes. My source is a nvarchar(4000) column.
e = 1 byte
ë = 2 bytes

Any good suggestions on how to cut of some chars at the end to fit in 4000 bytes? I want to cut of the minimal number of chars for each record....

(using SSIS 2012 with Attunity Oracle destination)

Please mark the post as answered if it answers your question | My SSIS Blog: http://microsoft-ssis.blogspot.com | Twitter

I have an SSIS package where I copy an Excel workbook with a Data sheet that needs to be updated.  It is an xlsx file.  The first step is to copy the excel workbook to a working directory.  This workbook only has headers on the Data tab. 

A SQL query is run to get the values and an Excel Destination is used to populate the Data tab in the spreadsheet. 

This is on a 64-bit Windows/SQL 2008 R2 Server.

I got it working beautifully in BIDS after getting the required Access data component installed and changing the Run64bitRuntime value to False.

Now, I want to automate this package.  I have tried using SQL Agent to run the SSIS package which I stored in SQL Server. I also made sure to check the box in the job step to use 32-bit.  However, I ran into the standard errors regarding Excel destinations.  Note that I already have one successful package running via SQL Agent with the difference being that it's creating a new Excel sheet with an extension of .xls.

I have also tried using DTExec from a cmd prompt to see if it would work there. After getting various error references to "Excel 12.0 XML" and the connection names, I seemed to get farther when I took the spaces out of the connection names in the package.  I tried 32-bit DTExec and 64-bit DTExec.  Getting the following error consistently now:
"SSIS Error Code DTS_E_OLEDB_EXCEL_NOT_SUPPORTED: The Excel Connection Manager is not supported in the 64-bit version of SSIS, as no OLE DB provider is available."

I even tried running the DTExec commands with and without the /X86 switch.

I have gone through many internet searches which mention all the 32-bit vs 64-bit stuff, and I have all those values set for using 32-bit.  Of course, I try it both ways just for kicks and giggles.  Is it not possible to automatically update a tab in an existing xlsx worksheet? What am I missing?  I thought I was home free when I got it working perfectly from BIDS.  But, now I can't get DTExec to do it successfully or automate it with SQL Agent.  For the SQL Agent job, I've tried executing the package as well as simply typing the DTExec command to run at a command prompt. 

I've done so much trial and error to try to get this working, I'm at a loss.

What else am I missing?  Let me know if you need any other specific info from me.  I can paste any relevant info that would help.

Thank you!

Dana

I developed an SSIS package with ProtectionLevel EncryptSensitiveWithPassword in VS 2008.  I imported the package in our dev and staging environments.

When I went to import the package on our production server, I didn't have the permissions to do so, and had an admin try it.  He however was met with the following error:

Exception from HRESULT: 0x0010026 (Microsoft.SqlServer.DTERuntimeWrap)

The way we ended up getting it imported is by granting me full sql permissions and having me to the exact same thing that the Admin was doing.  It has happened a few times, and I am the only one in the company who is able to import this package now. All I can think of is that another dev wrote a custom SSIS task, installed it in the GAC on the servers and everything in the package works great, except for the importing of the package itself.

We were standing up a new SQL server and migrating packages over, and this was the only one to encounter this issue.  There are other packages written by me that were fine, and all I can think of is that the custom SSIS task is causing the package to operate differently.  

Options?  Ideas?  Thanks!