Quantcast
Channel: SQL Server Integration Services forum
Viewing all 24688 articles
Browse latest View live
↧

SSIS Package to extract data to a file for a 3rd party containing multiple file sections

August 21, 2015, 1:38 pm
$
0
0

So I have to construct an SSIS Package that will create a flat file .txt or .dat for a 3rd party. The file will contain a couple of different sections...

  • File Header Record
  • Batch Header Record
  • Detail Record
  • Batch Trailer Record
  • File Trailer Record

So I'm a pretty simple K-I-S-S programmer. And the way I was going to do this would be to create a SQL Server Stored Procedure and within it create separate #Temp_Table for each section, build its contents appropriately, then build one mass #Temp_Table_Final with one column varchar(1000) and then simply INSERT from each section #Temp_Table to construct the Final file that will be FTPed out.

Am I over complicating this? Is there an easier way within SQL Server Integration Services to accomplish the same thing?

Thanks for your review and am hopeful for a reply.

↧
Search

Running SSIS package as SQL Job doesn't process any files.

August 21, 2015, 3:20 pm
$
0
0

I created an SSIS package to import text files from a folder into a database. I runs fine when execute it in Visual Studio and when i double-click the .dtsx file.

But when I have it as a SQL Job, it runs does not work properly.  The SQL Job executes fine and it sends me an email (an SSIS task in the package) but nothing is actually done and my package metrics reports 0 files were iterated.  I have tried setting the SQL agent to run as the same user (me) and it doesn't seem to make a difference.

What is different is that the SSIS variables that I used (to help iterate through the data files folder) do not appear on the email (NULL?) that gets sent when I run it through the SQL Job.  Although they appear fine in the email when the SSIS runs fine (using the other methods).

There is a little extra complexity because the database runs on a SQL 2008 R2 instance, but I created the package in vs2012.  I got errors trying to run it on that instance, so I was trying to run it from my local machine which runs SQL 2012. 
But even moving a copy of the database to my local instance (and repointing the db connection manager) is the same thing.  The package works when I run it through VS or the .dtsx, but it doesn't work as a SQL job.

Any suggestions on where to look next?

TIA



↧
↧

Become the August 2015 SQL BI Guru!!! Here's how!

August 22, 2015, 3:32 pm
$
0
0

Just add your TechNet Wiki article to this list:

  

One winner in each category will be selected each month for glory and adoration by the MSDN/TechNet Ninjas and community as a whole. This includes a dedicated blog post in the Wiki Ninjas blog, a tweet from the Wiki Ninjas Twitter account, an announcement on your forum, and other acknowledgement from the community. 

Winners will be voted on by five judges. The judges consist of 3 Microsoft MVPs and TechNet Wiki Community Council members and 2 Microsoft Employee SMEs (Subject Matter Experts -usually the people making the technologies). The judges will be looking for articles that are thorough, technically accurate, visually clear (images might help, but aren't necessary), and well written.

  

How to Enter

1) Create a new TechNet article YOU CAN COPY YOUR CONTRIBUTION FROM MSDN/TECHNET FORUMS OVER TO TECHNET WIKI (IN AUGUST) TO QUALIFY FOR THESE AWARDS. You can also create a new article not related to your forums contributions. 

A) Log into TechNet/MSDN with your Microsoft credentials

B) Add your content as an article to TechNet Wiki: http://social.technet.microsoft.com/wiki/contents/articles/add.aspx%20  

If you are copying and pasting your MSDN/TechNet forum solutions over to TechNet Wiki, please give some introduction to the problem, make sure your steps are clear, and then link to the original forum post. You can also paste in your blog posts (rather than forum content).

2) Tell us about it To add a link to your article:

A) Log into TechNet with your Microsoft credentials

B) Click the "Edit" tab on the list of August Guru articles, and copy in the URL  to your TechNet Wiki article into the appropriate section, along with your name and link to your profile!

  

 

We're looking forward to seeing your article!

Thanks!

Ed Price, Azure & Power BI Customer Program Manager (Blog,Small Basic, Wiki Ninjas, Wiki)

Answer an interesting question? Create a wiki article about it!

↧

SSIS 2008r2: Refer back to one table for values on two fields on import file to insert in another table

August 22, 2015, 2:44 pm
$
0
0
Hello, Can someone help me figure this out:

I receive a .csv file for import with let's say 3 rows in fields neworgid, supplierneworgid for synchronization with our table a, that has orgid, supplierid
I create the SSIS package with data flow source: csv file, Multicast transformation to lookup 1 to match up supplier id with neworgid resulting in neworgid and another lookup 2 to match the org id to the neworgid. Then I put a union all transformation to a destination. What I get in the destination is 6 rows, 3 with the orgid, 3 with the supplierid. So I'm stuck. (and of course the deadline is looming!)

I need to read:
.csv file
neworgid - supplierneworgid
11 ---                 22
33 ---                44
55 ---                66

into table a
orgid - supplierid
112  ---    222
332  ---     442
552  ---     662
Looking into table b (which I have cached)
orgid -- neworgid
112   ---    11
222   ---    22
332   ---    33
442   ---    44
552   ---    55
662           66

TIA everyone!
.
↧

how to purge data in transaction table or we can delete some data and store in separate table in data warehouse

August 18, 2015, 8:55 am
$
0
0

hi,

how to purge data  in transaction table or we can delete some data and store in separate table in data warehouse

Regards,

Manish

↧
↧

Environment variable issue using BIML

August 23, 2015, 10:11 am
$
0
0

Hi Friends,

I was writing BIML script for to generate the packages as I have an issue with environment variable 

I was fallowing below link and used exactly same script for environment variable but it was not showing in the package at all, couldn't able figure out where I was doing mistake can anyone guide me please 

http://microsoft-ssis.blogspot.co.uk/2014/03/package-configurations-with-biml.html

Moreover, I have installed BIDhelper 1.7.0.0  is there any issue with version ? 

↧

best practices when we create a dimension table or master table

August 23, 2015, 5:08 am
$
0
0

hi all,

can you share best practices when we create a dimension table or master table

Regards,

Manish

↧

Modifying the Flat File Connection Manager (from a tutorial)

August 18, 2015, 6:40 am
$
0
0

Hello,

Re: https://msdn.microsoft.com/en-US/library/ms166797%28v=sql.120%29.aspx

I'm having a problem with Integration Services Tutorials SSIS Tutorial: Creating a Simple ETL Package Lesson 2: Adding Looping (step 3).

I can't add a variable to Sample Flat File Source Data.

Right-clicking on the connection manager just shows two items of the property: File Name and File Path and nothing else.

I am using MS Visual Studio Ultimate 2012. Does it have limitations? I've see screenshots where properties of flat file managers have many more options.

↧
Search

SSIS API Performance Issues

August 20, 2015, 2:04 am
$
0
0

Hi

My company runs Web Services written in C# (running Windows 2012R2 / IIS 8.5) that use the SSIS API (via HTTPS) to execute packages on a remote 2012 SQL Server (running Windows 2012R2) and we are experiencing some performance issues when issuing a particular command via the API.

I am trying to understand exactly what happens when the Web Service issues the command to execute the package (asynchronously) as this our bottleneck - it can take up to 30 seconds to execute a package, yet the package may only actually take 5 seconds or less to complete.

In our C# code (listed below) we create the execution of the package with any parameters added and then perform the execute once we have the Execution Identifier. This is where the issue is; the call is made to run the package and the response takes around 30 seconds to complete but the packages finishes in a few seconds.

What I am trying to find out (as there is no documentation that I can find) is exactly what happens in SQL Server when this code is executed:

ExecutionOperation executionOperation = catalogSSISDB.Executions[executionIdentifier];

If I can understand what these API commands actually do, I may be able to troubleshoot the issue further but at the moment, we are scratching our heads.

Any help or guidance is greatly appreciated!

usingSystem;usingSystem.Collections.Generic;usingSystem.Linq;usingSystem.Runtime.Serialization;usingSystem.ServiceModel;usingSystem.ServiceModel.Web;usingSystem.Text;usingMicrosoft.SqlServer.Management.IntegrationServices;usingSystem.Data;usingSystem.Data.SqlClient;usingMicrosoft.SqlServer.Management;usingSystem.Configuration;// NOTE: You can use the "Rename" command on the "Refactor" menu to change the class name "Service" in code, svc and config file together.publicclassService:IService{publicvoid invokeSyncWebServiceName(input_WebServiceName input,outstring correlationId,outExecutionStateType executionState,outResultStatusType resultStatus,out output_WebServiceName output){        output =new output_WebServiceName();        correlationId ="-1";try{//Establish a connection to the SSIS serverSqlConnection sqlSSISConnection =newSqlConnection(ConfigurationManager.ConnectionStrings["SSISCatalogConnection"].ConnectionString);//Connect to the SSIS serverIntegrationServices integrationServices =newIntegrationServices(sqlSSISConnection);Catalog catalogSSISDB = integrationServices.Catalogs["SSISDB"];//Get package from catalogPackageInfo myPackage = integrationServices.Catalogs["SSISDB"].Folders["FolderName"].Projects["ProjectName"].Packages["PackageName.dtsx"];//Set package parametersSystem.Collections.ObjectModel.Collection<PackageInfo.ExecutionValueParameterSet> executionValueParameterSet =newSystem.Collections.ObjectModel.Collection<PackageInfo.ExecutionValueParameterSet>();            executionValueParameterSet.Add(newPackageInfo.ExecutionValueParameterSet{ParameterName="Param1",ParameterValue=DateTime.Parse(input.Date1),ObjectType=30});            executionValueParameterSet.Add(newPackageInfo.ExecutionValueParameterSet{ParameterName="Param2",ParameterValue= input.Value1,ObjectType=30});            executionValueParameterSet.Add(newPackageInfo.ExecutionValueParameterSet{ParameterName="Param3",ParameterValue= input.Value2,ObjectType=30});//Get SSIS environmentEnvironmentReference environmentReference = myPackage.Parent.References[ConfigurationManager.AppSettings["SSISEnvironment"],ConfigurationManager.AppSettings["SSISEnvironmentFolder"]];//Execute packagelong executionIdentifier = myPackage.Execute(false, environmentReference, executionValueParameterSet);//Set correlationID to SSIS executionID            correlationId = executionIdentifier.ToString();//Execute packageExecutionOperation executionOperation = catalogSSISDB.Executions[executionIdentifier];//Refresh execution status            executionOperation.Refresh();//Wait for the package to complete (if applicable)while(!executionOperation.Completed){System.Threading.Thread.Sleep(5000);                executionOperation.Refresh();}//Package completed            executionState =ExecutionStateType.FINISHED;//Get completion statusif(executionOperation.Status==Operation.ServerOperationStatus.Success){                resultStatus =ResultStatusType.OK;}else{                resultStatus =ResultStatusType.ERROR;}//SSIS package execution results are persisted to a database//Connect to database to obtain package execution resultSqlConnection sqlConnection =newSqlConnection(ConfigurationManager.ConnectionStrings["PackageOutputConnection"].ConnectionString);            sqlConnection.Open();string strSQLCommand ="SELECT package_output, result_status FROM package.execution_output WHERE execution_id = "+ executionIdentifier.ToString();SqlCommand sqlCommand =newSqlCommand(strSQLCommand, sqlConnection);SqlDataReader sqlDataReader = sqlCommand.ExecuteReader();//If there's no row in the table for the execution the package completed successfully but do not return a resultif(!sqlDataReader.HasRows){                output.Result="";}else{//Get the result and set the web method response                sqlDataReader.Read();                output.Result= sqlDataReader.GetString(0);string strPackageResultStatus = sqlDataReader.GetString(1);if(strPackageResultStatus =="OK"){                    resultStatus =ResultStatusType.OK;}elseif(strPackageResultStatus =="WARNING"){                    resultStatus =ResultStatusType.WARNING;}else{                    resultStatus =ResultStatusType.ERROR;}                sqlDataReader.Close();//Update the package.execution_output table to show the result has been returnedstring strSQLUpdate ="UPDATE package.execution_output SET output_delivered = 'Y' WHERE execution_id = "+ executionIdentifier.ToString();SqlCommand sqlUpdateCommand =newSqlCommand(strSQLUpdate, sqlConnection);                sqlUpdateCommand.ExecuteNonQuery();}            sqlConnection.Close();}catch(Exception e){            output.Result= e.Message;            resultStatus =ResultStatusType.ERROR;            executionState =ExecutionStateType.FINISHED;            DWF.Logging.ErrorLogging.LogError(e);}return;}}

↧
↧

SSIS Performance

August 23, 2015, 8:58 pm
$
0
0

Hi SSIS Masters,  

            I am trying to improve the performance of our ETLs (viz. MSSQL 2012 version) by editing the OLE DB Destination Editor values. 

            In my package, on a daily basis the source table has <= 50,000 rows of data (these are the number of rows that I have to pull every daily basis). Can I set the Rows per batch as 10,000 or 50,000. I am still trying to understand, on what factor this value must be decided? 

           Any suggestion for the above scenario is welcomed. 

Cheers,

Chandra.

↧

SSIS packages migration from 2005 to 2012

August 20, 2015, 11:56 pm
$
0
0

Dear Experts,

I am SSIS newbie. Please excuse my understanding of this concept. I am in the process of migrating all the SSIS packages from SQL Server 2005 to SQL Server 2012 (side by side migration). There are totally 6 packages. I exported all of the packages in SQL 2005 by this process.

Go to SQL 2005 Integration Services-> Under "Stored Packages", right click the package and click "Export Package". These packages get saved in .dtsx format.

Then I got to SQL Server 2012, open these packages in SSDT and get many error messages. Example below.

"The component metadata for could not be upgraded to the new version of the component. The performupgrade method failed"

"Convert to dates failed validation and returned error code". "The component is missing, not registered, not upgradable or missing required interfaces."

It is quite frustrating, I searched these error messages but couldn't figure out how to fix. Also, when I saw some videos online, some of them have showed opening the package as a project but my packages are all in .dtsx format. Is there someway by which I can save them as a project in sql 2005.

Please provide your inputs. Thank you for your help.

Thanks.

MM


↧

increment data in cube

August 21, 2015, 2:43 am
$
0
0

Hi all,

How to increment a data from one database to another database.

Thanks

Ranjan DS

Ranjan DS

↧

SSIS + varchar(max) issue

August 23, 2015, 11:37 pm
$
0
0
I have a variable of type string in SSIS which I want to set using ExecuteSQL task which is returning XML as string varchar(max).  But when using varchar(max) the variable is not getting set and I found similar threads that SSIS doesn't understand varchar(max).  Now then I tried changing my SSIS variable to Object type, but still the varchar(max) string returned from the executsqltask is not being set!  I am looking if there is any workaround on sql server side (within ExecuteSQLTask) or SSIS side to solve this issue.  Please help.
↧
↧

[SSISDB].[catalog].[create_execution] security issue

August 18, 2015, 5:56 am
$
0
0

Hi everyone,

I have a strange issue regarding executing a package deployed to SSIS catalog

The SSIS package contains a Data Flow reading a text file on a remote network share

  1. Runing the package in SSDT works fine
  2. Runing the package using Stored Procedure [SSISDB].[catalog].[create_execution] fails - the error message is: "Cannot open the datafile \\ServerName\NetworkShareName\FileName.txt"
  3. Now, here it becomes strange - I connected though RDP to the server that hosts both SSIS and the destination DB, with the same credentials and ran the same SP and it succeeded

Is authintication different when running through SP? it might be a Kerberos delegation issue?

↧

Update Statement in SSIS

August 24, 2015, 1:04 pm
$
0
0

Hello Everyone,

I have a scenario where, an Update statement using subqueries is return in Store Proc and now I started converting that Proc to a SSIS package, would like to know if these is a Way we can convert that using SSIS components?

SQL Statement:

update X
       set           X.Rollup = B.BusinessGroup
       from   DimTbl X
       inner join GroupValue B on X.MemberClass = B.Value
       where  B.Card= 'NONE'
       and           B.ID= (
                     select ID
                     from   Bgroupstatus
                     where  GroupName = 'Rollup')

Is there a possibility in Dataflow to achieve this step, like getting those values from sub queries and then using them in main Update? Please Help on this.

Thanks

Thanks, Please Help People When they need..!!! Mark as answered if your problem is solved.

↧
Search

Install Sql (2008) BIDS and Integration services on Client (Dev) box

August 24, 2015, 10:14 am
$
0
0

I want to install sql BIDS\Integration services on our dev's boxes so they can use the exercise files accompanying a training video. Each dev already has a user CAL. From a licensing perspective,  does this CAL cover the above mentioned install? (I contacted Microsoft licensing but I'm not sure they really understood the question so I'm am asking this forum)

TIA,

edm2


↧

Log records are truncated

August 24, 2015, 8:44 am
$
0
0

I run SSIS using DTEXEC command.

The output of the SSIS are getting truncated after X caracters.

This is a typical message which doesn't really help me debug. (as the full path would show me the DB name...)

Progress: 2015-08-24 11:30:02.20
   Source: Ensure Folder exists
   Executing query "EXECUTE master.dbo.xp_create_subdir N'R:\MSSQL_TRN...".:100% complete
End Progress

Is there a way to get a longer message?

↧
↧

SSIS OData feed failing

August 24, 2015, 5:16 am
$
0
0

I have an SSIS package which is reading from an OData V3 service using the MS OData component for SQL Server 2012

Recently it has started randomly failing when acquiring the data e.g.

SSIS Error Code DTS_E_PRIMEOUTPUTFAILED.  The PrimeOutput method on Positions returned error code 0xC02090F5.  The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.  There may be error messages posted before this with more information about the failure.

The issue goes away after a few runs of the package.

Is there any documentation on the error codes; I'm presuming it complaining about the data or the metadata of the feed but without more information it's difficult to diagnose.

Regards

Paul

Paul

↧

Missing data from excel

August 24, 2015, 1:05 pm
$
0
0

I have a data in excel sheet which is to be loaded to sql table. The Column called seq_num has data with leading 0's these 0's are ignored while loading through ssis.

Example if seq_num is like 0099988 the sql table would get 99988, how to get the whole data with missing anything.

FYI: seq_num on excel source has a data type as dt_r8.

Thanks

sree

↧

Connection Error

August 24, 2015, 9:25 am
$
0
0

My package is failing on local machine, where as the same is running fine on server.

The error is as below. failing for one of the data flow task

ADO NET destination has failed to acquire the connection  with the following error message:" Login Failed. The login is from an un-trusted domain and cannot be used with Windows authentication:"

Neil



↧
Viewing all 24688 articles
Browse latest View live
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>