I have a few questions:

• How would I prevent duplicate records, when/ IF SSIS is executed many times?
• How would I know that all huge volume of data being loaded in the entity tables?
• In reference to "how to move data from a staging table to three entity tables ", since I am loading large volume of data, while using lookup transformation:
  

1. which of the merge components is best suited.
2. How to configure merge component correctly. (screen shot is preferred) 

I would like to use SSIS to extract data from an IBM DB2 ISeries database. the Microsoft Ole db provider for DB2 works fine locally because I have Developer Edition. However we are running SQL Server 2012 BI edition in production and we won't be able to install the Microsoft provider on that which really sucks. 

Is there a way to install it on the BI edition? What are some alternatives. I believe IBM provides Ole db providers too.  Quite a few posts mention  IBM DB2 UDB FOR ISERIES IBMDA400 OLE DB PROVIDER. What do I need to download and install to have that available in SSIS?

Thanks

Greetings,<o:p></o:p>

I recently
inherited and was given SA rights access to our SQL Server 2012 SSIS/SSRS
reporting serve. I am attempting to take over existing packages as well as
build new ones. .<o:p></o:p>

I open
Visual Studios 2012 an added a Data Flow Task. When I attempt to create a new
oracle data source within the Data Flow, using the Source Assistant. My source
connection seems to work just fine (when I press the [Test Connection] button
it is successful). However, when I use the Source assistant to add this
connection as a source to the data flow, it will not let me add it all. I get a
huge message box indicating "The
Flow could not be added to the Flow task. Could not initialize the
component...There is a potential problem with the ProvideComponentProperties...
Error at Data Flow Task [SSIS Pipeline] ......0x80070005 "Access
Denied"<o:p></o:p>

I don't
understand how to fix this. I wonder if it is still somehow related to
insufficient rights (presumably the previous person here did not have these
troubles).<o:p></o:p>

I've a for loop container .Within this I've another sequence container which has few tasks in it .If tasks are success then I'm proceeding with executing another task and then the loop continues.

but if any of the tasks within sequence container fails then I need to execute a task and then exit the for loop .

How can exit the loop?

can someone please help

Hello everyone,

I just want to thank you in advance for all the help that you can give today. 

I just started at a new Company, as a junior DBA, the previous person left. Before that person left he created a lot of SSIS packages and turn them into jobs basically to refresh many different tables on the SQL server. The datas are being import from another datasource, which is why the SSIS packages was created so we can bring the datas over to SQL side.  Whenever I want to create a new SSIS Package for a new job, I always get the same failure as below...


"Error: 0xC0047062 at Data Flow Task, Source - Query [1]: System.OverflowException: Arithmetic operation resulted in an overflow.
at System.Data.Odbc.OdbcStatementHandle.RowCount(SQLLEN& rowCount)
at System.Data.Odbc.OdbcDataReader.GetRowCount()
at System.Data.Odbc.OdbcDataReader.FirstResult()
at System.Data.Odbc.OdbcCommand.ExecuteReaderObject(CommandBehavior behavior, String method, Boolean needReader, Object[] methodArguments, SQL_API odbcApiMethod)
at System.Data.Odbc.OdbcCommand.ExecuteReaderObject(CommandBehavior behavior, String method, Boolean needReader)
at System.Data.Odbc.OdbcCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.Odbc.OdbcCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior)
at Microsoft.SqlServer.Dts.Pipeline.DataReaderSourceAdapter.PreExecute()
at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostPreExecute(IDTSManagedComponentWrapper90 wrapper)
Error: 0xC004701A at Data Flow Task, DTS.Pipeline: component "Source - Query" (1) failed the pre-execute phase and returned error code 0x80131516.
Information: 0x40043009 at Data Flow Task, DTS.Pipeline: Cleanup phase is beginning.
Information: 0x4004300B at Data Flow Task, DTS.Pipeline: "component "Destination - GLCGLDS" (103)" wrote 0 rows.
Task failed: Data Flow Task
SSIS package "SSIS_GLCGLDS_GL.dtsx" finished: Failure.
The program '[16836] SSIS_GLCGLDS_GL.dtsx: DTS' has exited with code 0 (0x0)."

All the SSIS packages are failing for me, not just the one I created, even the ones that were working before, the ones that were created by the previous person, I just went to Start Debugging and it would failed at the same exact stage


Below are the exact steps that I do to create the SSIS Package (based on the notes from previous person)
1. right click on SSIS Package and select sql server import and export wizard and click next
2. Select data source as .Net framework Data Provide for ODBC
3. Input Datasource, and driver as: TS ODBC Multi – Tier Driver
4. Choose destination as Microsoft OlE DB Provide for SQL Server
5. Write a basic select statement.
6. Click edit Mappings
7. Here's the important part... I am ABLE to preview the DATA.
8. Click ok  next  finished
9. Click debug start debugging
10. The preparation SQL task will be green, but the Data Flow Task will failed everytime for me.

If anyone has any suggestion please let me know, b/c I have try every possible ways and out of luck, and I don't want to explain to my boss I don't know how to do it or how the previous person does it without any failures.

TL

Error 'Invalid bracketing of name ' is thrown in our SSIS package which runs a SELECT query to read data from an excel file. The file contains a field with a dot in its name.

I already changed the dot(.) with a sharp(#) in the select query, as suggested in several forums. But package fails because it cant find the column.

Assuming fieldname can't be changed in file, can any kind person tell me if there is already a permanent fix for this? Maybe newer version of access driver, or a patch or a different approach to select data from the file.

I seem to not find any on the web.

We are using SQLServer 2008R2, Microsoft Office Access database engine 2007 and conn string="Provider=Microsoft.ACE.OLEDB.12.0;Extended Properties=EXCEL 12.0;"

Please help, it will be greatly appreciated.

Hi,

I am new to ssis package need help to achive below task.

I have flat file which I am loading in table. My flat file may contain records that is already present in table. I want to find out records from the flat file which are already available in table. After finding out records, I want to check which columns value got changes and base on the result I want to update status column saying record got change.

Please help me with step by step explanation.

Ex of table

GUID   col1  Col2 Col3 Col4 Status

(Col1,Col2) Primary key.

Thanks

Hi,

We need to build a DWH/Reporting System that works in a near-real time refresh mode and were evaluating if SSIS would be the best solution for ETL. We are tying to compare SSIS performance with Web Services / Service Bus.

If anyone has done any such exercise, please let me know the findings or any suggestions you may have.

How does SSIS transfer data internally over network? Is there any compression that is adopted?

Thanks in advance.

TechNet loves you!

We love your contributions at TechNet Wiki sooo much that we give you more than just love in return...

We give you NOTORIETY, GLORY... and VIRTUAL MEDALS!

That's not all, this love we have, together, it flows both ways my friend.

You give us stuff, we give you stuff, like interviews, recognition points, Ninja Belt rankings, and of coursefront page love!

If the love is strong enough, who knows where it could end! We may even invite you into secret clubs and other initiatives.

So why not spread the love a little further this Valentines, with more than just a cheap card from the highstreet...

Express your love for your favourite technology in a TechNetWiki article!

Pour your heart out to us, capture our hearts and woo us with your prowess!

All you have to do is add an article to TechNet Wiki from your own specialist field. Something that fits into one of the categories listed on the submissions page. Copy in your own blog posts, a forum solution, a white paper, or just something you had to solve for your own day's work today.

Drop us some nifty knowledge, or superb snippets, and become MICROSOFT TECHNOLOGY GURU OF THE MONTH!

This is an official Microsoft TechNet recognition, where people such as yourselves can truly get noticed!

HOW TO WIN

1) Please copy over your Microsoft technical solutions and revelations toTechNet Wiki.

2) Add a link to it on THIS WIKI COMPETITION PAGE (so we know you've contributed)

3) Every month, we will highlight your contributions, and select a "Guru of the Month" in each technology.

If you win, we will sing your praises in blogs and forums, similar to the weekly contributor awards. Once "on our radar" and making your mark, you will probably be interviewed for your greatness, and maybe eventually even invited into other inner TechNet/MSDN circles!

Winning this award in your favoured technology will help us learn the active members in each community.

Feel free to ask any questions below.

More about TechNet Guru Awards

Thanks in advance!
Pete Laker

#PEJL

Got any nice code? If you invest time in coding an elegant, novel or impressive answer on MSDN forums, why not copy it over to the one and onlyTechNet Wiki, for future generations to benefit from! You'll never get archived again!

If you are a member of any user groups, please make sure you list them in the Microsoft User Groups Portal. Microsoft are trying to help promote your groups, and collating them here is the first step.

in the data conversion error output there are ErrorCode and ErrorColumn, when i use this two value i got this :368-1071607681

is there a way to get the real column name and error message like " try to convert string to int " when convert "aa" to int.

I have an Excel sheet in with a certain amount of rows. Each row has always the same amount of columns and the amount of columns is always a multiple of 12 (12, 24, 36, 48...).

For each row I want to take, starting from the A column, group of 12 columns and put in a SQL database table record. Each subset represent data for a year, 12 months.

Therefore if I have let's say 48 columns (4 years data), for each row I have to put 4 lines (48/12) in the SQL table.

I am not asking for a solution but just for a hint on which tool/approach do you suggest for accomplish this task, especially because in Excel columns are indicated by letters and not numbers and I do not know how to specify "take the columns from 0 to 12, than from 12 to 24...and so on".

I have been suggested to use SSIS or the Integration Wizard but before starting with using this tools I would like to know if I am taking the right path. Of course any advice on how to tackle this issue is welcome. Thanks

Hi,

How can I pass parameter to SSIS package which will be called via SQL Server Agent in Job in vb.net app?

Here is the link I m using as a sample but doesn't show how to pass parameter:

http://technet.microsoft.com/en-us/library/ms403355.aspx

Thanks,

Hi Experts,

Anybody know this error message? Do you know its root cause? And how to resolve it? I encountered this when I was deploying a SSIS project to Integration Services Catalogs. My Visual Studio is 2010, SQL Server Data Tools. And the data engine on the server side is SQL Server 2012, Enterprise edition, version 11.1.3128.0.

Detailed Error Message:

Failed to deploy the project. Try again later. (.Net SqlClient Data Provider)

------------------------------
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%20SQL%20Server&ProdVer=11.00.3128&EvtSrc=MSSQLServer&EvtID=27118&LinkId=20476

------------------------------
Server Name: MSSCIAIDB01
Error Number: 27118
Severity: 16
State: 1
Procedure: prepare_deploy
Line Number: 221

------------------------------
Program Location:

   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.SqlServer.IntegrationServices.Common.ObjectModel.SqlHelper.ExecuteSQLCommand(SqlStoreConnection storeConnection, CommandType cmdType, String cmdText, SqlParameter[] parameters, ExecuteType execType, Int32 commandTimeout)
   at Microsoft.SqlServer.Management.IntegrationServices.CatalogFolder.DeployProject(String projectName, Byte[] projectStream)
   at Microsoft.SqlServer.IntegrationServices.Wizard.Common.Model.DeployExportProjectHelper.DeployProject(CatalogFolder folder, Project project)
   at Microsoft.SqlServer.IntegrationServices.Wizard.Common.Model.DeploymentModel.Deploy(CatalogFolder folder, Project project)
   at Microsoft.SqlServer.IntegrationServices.Wizard.Common.Model.DeploymentModel.DeployProject()
   at Microsoft.SqlServer.IntegrationServices.Deployment.DeployProjectPage.DoWork(Object sender, DoWorkEventArgs e)
   at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
   at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)

I have a problem with an SSIS package that runs fine in Visual Studio but fails when deployed to SQL Server and run as a SQL Agent job.

The packages were written in Visual Studio 2008 and have been running for a year on a SQL Server 2008 R2 Enterprise Edition server. The problem has appeared when we moved to an environment of Visual Studio 2012 and SQL Server 2012 Enterprise Edition.

I created a solution in Visual Studio 2012 and added the packages from the SQL Server 2008 R2 server (we use SQL Server storage for our SSIS packages). I ran them through the upgrade wizard. The structure is a Master Package that calls 5 Child Packages in sequence.

My problem is with the first Child Package. I have a Foreach Loop Container for files that creates its Directory with this expression:

@[User::SourceFilePath] + @[User::SourceSubFolder] + "Incoming_Data_2012\\" + @[User::Code]+ "\\Staging\\"

The source path for the file to be read in varies depending on the customer. The variables in the Directory Expression are loaded as follows:

@[User::SourceFilePath] comes from an SSIS_Configurations table. It only changes when we change environments.

@[User::SourceSubFolder]is loaded by an Execute SQL task which uses a CASE statement to set the variable by querying a customer reference table based on the value of the 3rd variable, @[User::Code].

@[User::Code] also comes from the SSIS_Configurations table. It is updated each time we run the batch process.

What I believe is happening is that when I deploy the solution to SQL Server and run it as a SQL Agent job, the Foreach Loop Container is not mapping it's collection value to the appropriate variable, which is called @[User::FileJustFound].

The first task in the Foreach Loop Container is a Script Task which checks the @[User::FileJustFound] variable and sets a variable called @[User::Exists]. When @[User::Exists]is False, the process exits the container and sends an email. This is a warning since production folks sometimes forget to put the file in the right folder.I added code to this email to show me the value of @[User::FileJustFound] and it returns a blank. The Script Task code block is below.

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;


namespace ST_5561de6021c24882be4609fa7708ddac.csproj
{
    [Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
    {

        #region VSTA generated code
        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion

        

        public void Main()
        {
            string file = Dts.Variables["User::FileJustFound"].Value.ToString();
            if (File.Exists(file))
            {
                Dts.Variables["User::Exists"].Value = true;
            }
            else
            {
                Dts.Variables["User::Exists"].Value = false;
            }
            Dts.TaskResult = (int)ScriptResults.Success;
        }
    }
}

The SQL Agent Service is running under my Windows account, so it's not a permissions error. The Job History shows no detail at all. All variables are scoped at the package level.

My question is why would it work in Visual Studio and not in SQL Server? I tried deleting the Variable Mapping on the Foreach Loop Container, saving/closing the solution and then re-creating it. Now the SQL Agent job is failing as well.

Hi All !

I have a simple MySql Select Query with a filter on column updated_at (datetime). when I execute my query directly on HEIDI SQL I got 33 lignes. BUT when I put it in Expression for SQLCommande in my Package (so that I can use a Variable in my Filter) the query retrives All data from source Table (99 000 lines)...

any ideas please ?

Thanks

how do you archive a file(s) with today's date as part of the filename?

sukai

This question is not about any particular code, but more of a general, overall nature.

As may already be known by some, I am trying to optimize a SQL stored procedure. It takes about three hours to run and we need it to go faster.

In it, there are two cursors. I have replaced them with CTEs, so they should run faster.

But, I'm wondering, would it be faster, still, to do them in SSIS?

The sp does several - maybe a dozen or so - updates to a table and then runs what was cursors and is now CREs, followed up with a few more updates before emailing the results to various recipients.

The updates are pretty straight forward, though one of them does do several joins with more than two tables.

This sp is ultimately being converted to use in an SSIS package, but, basically, I'm wondering if, in general, it is faster to do as much as possible in SQL - say, as Execute SQL Tasks in the ControlFlow - than to redesign the queries using the tools provided in the Data Flow.

My guess is that it's faster to use SQL to perform SQL tasks, but maybe SQL is foreign enough to SSIS to make it faster to do the same task in SSIS' Data Flow.

Any thoughts?