Quantcast
Channel: SQL Server Integration Services forum
Viewing all 24688 articles
Browse latest View live

SSIS execute multiple packages based on conditional split

$
0
0

Hi All,

We are building a dataload application where parameters are store in a table. And there are multiple packages for each load.

There is a column IsChecked column if it is 1 then only the child package should execute.

What i am doing here is. Created a master package. In which i have taken execute SQL task in that storing a results in variable and based on the result the child package should execute. this is the idea.

But In executesql task i selected result set as full result set.  I am getting the below error.

[Execute SQL Task] Error: Executing the query "SELECT  isnull(ID ,0) AS ID FROM DataLoadParameter..." failed with the following error: "The type of the value (DBNull) being assigned to variable "User::LoadValue" differs from the current variable type (Int32). Variables may not change type during execution. Variable types are strict, except for variables of type Object.
". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.

No null values are passed.

Please suggest.


Within Microsoft Visual Studio, how can I get my tasks to move up on the page?

$
0
0
So I have all this gray space at the top of my tasks. When I <Ctrl>+<Alt>+<Left-click> to select all of my tasks and then try sliding it up, it creates even more gray space and actually moves it down. If instead when I <Ctrl>+<Alt>+<Left-click> and then <Ctrl>+<Arrow-up> it doesn't move it. Is there any easy way to eliminate all this gray space at the top?

Truncate Microsoft Access Table from SSIS

$
0
0

 

Does anyone know how to truncate a Mircrosoft Access table from an SSIS package?

Sql Command from a variable to import data dynamically using ForEach Loop and Variablesin 2012

$
0
0

Hi,

  I am new to SSIS. I need to move all the data and objects from one server to the other server. We actually archive the data onto  a different server. So I need to move the entire data into new database (the database name varies depending on the year of the archive ex: Test2013, Test2014 etc). For this I have created Execute SQL Task to create new database and copied the table structures from the package. Now to copy the data from multiple tables into the same tables on the new database, I have done the following steps:

1. created a Execute SQL task to get the list of table system objects.

2. Created a ForEach Loop container to process these table one by one.

3. Created a Data Flow task in the ForEach Loop container and I am stuck here in creating the sql command from variable as I do know the database name ( as it varies every year) and also the table that Foreach Loop is passing. I have followed the steps from http://beyondrelational.com/modules/2/blogs/101/posts/13354/ssis-full-result-set-to-a-set-variable-of-object.aspx till here.

4. This is the query I have in my OLE DB Source "SELECT  *   FROM  ServerA.Database1" + @[User::DestTableName].For this step I followed

http://blog.stevienova.com/2009/04/30/ssis-two-ways-using-expressions-can-make-your-life-easier-multi-db-select-non-standard-db-select/.

5. The problem here is I do not have the value for DestTableName variable @[User::DestTableName] as I get this from ForEach Loop.

6. I get the query from the variable like this SELECT  *   FROM  ServerA.Database1. and obviously it is unable to prepare the statement. It gives me error saying "Statement could not be prepared".

7. I have created 3 variables. One for the result set object for all systables, one for TableName and the other one for SQL query variable.

I really appreciate if someone help with this.

Thanks in advance

ylsv

Truncate or delete Access table using SSIS

$
0
0

Hello,

I have to truncate  access table before I insert records to access database table. 

I tried using

Delete From Table_name  or Truncate table Table_NAME and I have used Microsoft Jet 4.0 OLE DB Provider. It does not seem to work. I read some post on forums. None of them seem to work while truncating or deleting records from one of the access database table.

Is there any easy way to truncate access database without using script component and VB scripts. I do not know how to Write VB scripts so trying to find alternatives.

http://bomsbi.blogspot.ca/2012/08/ssis-delete-from-ms-access-tables.html

I looked at this post of VB scripts but I dont know what things needs to be replaces in my case to get that thing working.

Thanks


Is there a general rule of thumb when to create a SQL Server Stored Procedure for an Execute SQL Task

$
0
0

So sometimes I'll create a SQL Server Stored Procedure...sometimes I'll just write raw SQL in my Execute SQL Task.

Is there any general rule of thumb as to when to create a SQL Server Stored Procedure and when using raw SQL is alright when utilizing an Execute SQL Task? Would have it have anything to do with re-usability which would dictate a SQL Server Stored Procedure...is either one more efficient...

Thanks for your review and am hopeful for a reply.

delete and insert rows in access from ssis package

$
0
0

Hello,

I have a ssis package which identifies duplicate records in access database. I have staged access database into sql sever and created ssis package. Now, I have final list of records which needs to be delete from access database and new records which are to be inserted into access database. 

What do I need to do if I want to delete those duplicate records directly from access database using SSIS. I cannot truncate whole access database and reload. I just have to delete duplicate rows from access db and add new records.

Thanks for suggestions.

Script to change all Excel columns to Text format?

$
0
0
I'm using an Excel file as a source, and I want to avoid some of the datatype issues commonly found with Excel files.  Is there a way to use a script task, or any other method, to automatically convert all of the columns in the Excel file to Text datatype (on every sheet)?  I would basically copy my source file each time I run the process, run the script to convert all the columns, then load the data from the converted file.

Creating SSIS log file using expression and passing log location to a child package

$
0
0

Hello,

I have created a small SSIS solution to illustrate my problem.  Unfortunately it looks like I can't attach it here, so I will try to explain.  I can send the solution to people individually.

SSIS SQL Server 2008.  I have a package called "Parent_Package" with a single variable: "User::Log_Path".  During design time I assigned this value to it: "C:\temp\Log_Design_Time".  The package has a log provider configured (simple text file).  The connection string of the log file has an expression: @[User::Log_Path]  + "\\" +  @[System::PackageName] + <expression for timestamp> + ".txt".   As you probably see, I would like to create a new log file for each package execution. 

If I now run this package, I will have two log files generated!  One will have a timestamp from around the time I created the package.  This file only has the header fields and no other other information.  The second file has the correct timestamp and correct output information. 


It appears that the first file gets created during validation (even though I set DelayValidation = True for the log file connection manager).  So my first question is why is this happening?  It doesn't seem like the right behavior.

Now to take this to the next step, I create a child package "Package_Child".  I add an execute package task to the parent package to run this child package.  In the child package I add a variable by the same name as above: "User::Log_Path".  Next, I add a configuration to the child package, so that the value of "User::Log_Path" is obtained from the parent package variable by the same name.  Finally I add log provider to the child package and set it up the same way as described above for the parent package. 

Let's now run the parent package again.  This time I will end-up with three log files: two for parent and one for child.


For my next test, I now would like to change the location of the log files, so I modify the value of the [User::Log_Path] variable in theparent package to "C:\temp\Log_Run_Time".  If I run now, I will have four log files!  Two for the parent package in the  "Log_Run_Time" directory and two for the child package - one in "Log_Design_Time" and one in "Log_Run_Time" directory:

What's going on?

Finally, if I simply remove the original "Log_Design_Time" directory, my child package will start failing complaining that it cannot find the path specified:

Obviously it's still looking for the path specified earlier, but why?  Again, I set "Delay Validation" = True on all my connection managers and the package itself.

I appreciate any help on this.

Thank you!


P.S.  Here is a complete expression for log path: @[User::Log_Path]  + "\\" +  @[System::PackageName] + "_" +
(DT_STR, 4, 1252)DATEPART("yyyy", @[System::ContainerStartTime]) +
RIGHT("0" + (DT_STR, 2, 1252)DATEPART("mm", @[System::ContainerStartTime]), 2) +
RIGHT("0" + (DT_STR, 2, 1252)DATEPART("dd", @[System::ContainerStartTime]), 2) + "_" +
RIGHT("0" + (DT_STR, 2, 1252)DATEPART("hh", @[System::ContainerStartTime]), 2) +
RIGHT("0" + (DT_STR, 2, 1252)DATEPART("mi", @[System::ContainerStartTime]), 2) +
RIGHT("0" + (DT_STR, 2, 1252)DATEPART("ss", @[System::ContainerStartTime]), 2) + ".txt"






Solution for a scenario of De-Duplication with Fuzzy Logic

$
0
0

Hi,

I am attempting to create an SSIS Package to do a de-duplication process with a large database of over 10 million people records.

Each person record will have to be matched with the complete DB on Name Match, DOB Match, Relationship Match, Gender Match and then we will have to save the matching records into another table - which will then be reviewed by the client to decide which are the duplicate ones and then suitable actions taken.

We have decided to go with the Fuzzy logic match to do this.

The plan is to pick up a set of 50 records one at a time, match each of them with the complete DB and then move to the next 50 records. It is also important to ignore the already processed 50 records + also ignore the duplicate matches returned.

We want to program such a package and schedule it to run at the background.

Is this possible and if so, what would be the right approach to create this?

Regards,
Karthick


Karthick S

Execute Script task throws error "The package execution returned DTSER_FAILURE (1)"

$
0
0

I have a SSIS package that will load data from a set of excel files then archive those files to a specified folder.

The excel files are stored in a folder and inside that folder I have the archive folder.

Below is my script task code for reference.

public void Main()
        {
            // TODO: Add your code here
            string sourceDir =  Dts.Variables["User::strFilePath"].Value.ToString();
            string destDir = Dts.Variables["User::strArchivePath"].Value.ToString();

            DirectoryInfo di = new DirectoryInfo(sourceDir);
            string[] sDirFiles = Directory.GetFiles(sourceDir);
            FileInfo[] fi = di.GetFiles("*.xls");
            int fileCnt = fi.Length;
                for (int i = 0; i < fileCnt; i++)
                {
                    String filename = fi[i].Name;
                    string[] splitFilename = filename.Split('.');

                    DateTime dt = DateTime.Now;
                    string ArchiveDate = String.Format("{0:ddMMMyyyy}", dt);

                    string sourceFileName = filename;
                    string sourceFilePath = sourceDir + filename;
                    string destinationFileName = splitFilename[0] + '_' + ArchiveDate + '.' + splitFilename[1];
                    string destinationPath = destDir + destinationFileName;


                    //MessageBox.Show("Source File " + sourceFilePath + " to destination " + destinationPath);
                    if (File.Exists(destinationPath))
                        File.Delete(destinationPath);
                    // To move a file or folder to a new location:
                    System.IO.File.Move(sourceFilePath, destinationPath);
                }
            Dts.TaskResult = (int)ScriptResults.Success;
        }

The sourceDir and destDir are variables that provides the path of source files folder and archive folder. The package works fine when I run it from visual studio.

I have deployed it to run as a job using deployment utility by creating manifest file. When I run the job I'm getting error in my archive script task. Below is the screenshot of it.

I searched for solution in stackoverflow however the provided solutions does not solve my problem.

Problem using SQL Agent to run SSIS Packages - fails with “DTSER_FAILURE(1)”

DTSX package runs in Visual Studio but not when called from a Database Job

I have granted read write access to SQL server for both the folders. Still getting the same error? Also have verified the excel runtime and it has been set to 32Bit.

Any help is highly appreciated.




Thanks you and Kind Regards, Karthik Venkatraman

Flat file with Dynamic Columns

$
0
0

I will be receiving a CSV daily where columns within the file will change.   The column order and number of columns can change daily.   I need a way to read in the header from the csv and create a flat file connection that reflects the columns listed in the header.   Is there an easy way to do this using a script task?    I have already read the header into a table but I have been unable to create the dynamic file connection.

Any help would be appreciated.

 

Marc

create flat files from 5 different views and zip them using SSIS

$
0
0

I have a requirement where I have to extract data from the view and create a flat file...So I have 5 different views, need to create 5 different flat files and drop into a folder. After the files are in the folder, they need to be zipped and sent to another folder location. For this, I have 7zip installed on my servers.

How can I achieve this in SSIS ? Can anyone please elaborate the steps ?


MBH

Help to get any book or material to use C sharp which is needed for SSIS

$
0
0

Hello,

I think we dont need to master in C# for the ETL using SSIS. As mostly C# would be used file related, String manipulations or for any validations.

It would be great of anyone suggest any book of C# which pertains the content which is really needed to perform ETL actions.

And I am sorry if I should not post this type of queries here.

Br,

Shamsuddeen

Creating Parallelism by Executing Many dtexecs

$
0
0

I'm currently looking at refactoring an existing, large SSIS 2012 implementation that consists of about 55 projects and 360+ packages. The ETL framework that is in use has a "main" control package that reads from a database table and determines which packages are ready to execute (based on some dependency logic) and then uses an Execute Process task within a loop that calls dtexec with the arguments: 

/C start Dtexec /SQL "SomePackagePath" /SERVER "someserver" 

This design allows the loop to execute a package and then immediately iterate because it doesn't wait for the package to respond (aka complete with a failure or success) so it can quickly kick off as many packages are ready to execute. A SQL Agent job is used to call this package every few minutes so that it can pick up any packages that have had their dependencies satisfied since the last execution and kick those off.

It's a a clever design but has some problems such as decentralized exception handling (since the parent package is unaware of what is happening in the "asynchronous" dtexec calls.

My biggest concern is that by executing packages, not with the Execute Package Task but with the Execute Process Task, and spinning up many dtexecs, the framework is not leveraging SSIS's ability to handle threading, memory consumption, etc. across all running packages and executables because it is simply unaware of them. It's essentially like using an Execute Package Task with the ExecuteOutOfProcess property set to true.

I'm wondering how much risk is introduced by this design and if there are any approaches I can take to minimize the risk of excess resource consumption. I apologize for not having an exact question here but would really appreciate any guidance anyone might have.




Merge Join Performance Issue

$
0
0

Hello,

I have created one package which will use Merge Join transformation (Inner Join) to relate two table to get some result.

The both table contain 30,00,000 records. The package is taking much time to execute.(in merge join transformation).

Is there any alternative solution to handle this problem,spacialy if there is huge data to join or how to increase the package performance.   

Error Output Not Found

$
0
0

Hi,

In my new project while I am transferring data from an Excel sheet to aSQL Database table, and have set the error data to be redirected to a flat file, I have deliberately set 56 out of total 366 rows to have one column with a size greater than that of a field. But when I run the project I get 3 Green Check marks on the Excel Source, OLE DB Destination & Flat File Destination and the blue line reflects366 records to be transferred. Upon checking in the SQL DB's table I find only 310 records to be successfully transferred but the Flat File to be empty?

Any specific reason and way to resolve the same?

How to update multiple single Excel cells using SSIS

$
0
0

 I have created an SSIS package which processes daily financial information to a sql server database.  These processes are to be outputted to excel spreadsheets to a readable report format for management to review.  Some of these reports are laid out in a way that is not just tabular output but requires customized placement of data on an excel spreadsheet to specific cells.  I am able to place an initial resultset of a query output from the database in a tabular excel template through SSIS but the issue is at the end of that placement in the spreadsheet I am required to place another output below that tabular output in a different format from the initial output which I have shown below. Attached I have an example of what one of the excel reports should look like.

parallel package execution with Single logdb

$
0
0

Hi,

i  want to do parallel package execution with Single log db. where log-db will contain inserted record, update record. i ma using SQL server 2008R2 standard execution.

currently i am using execution of package in parallel i got deadlock in my log-db, because all task trying to write log information in single table.

Regards,

Manish

Error 27203

$
0
0

Hi.. I am trying to Deploy an SSIS Project from Visual studio 2012 to SQL 2014.  I have created my SSISDB and Catolog.  But as soon as the wizard gets to Deploying Project it errors with... Failed to copy project . .. query the operation_messages view for the identifier '10' MSQ server error 27203.  I can't find the Fix. I have tried several times, the identifier number s all that changes. This is all running on my stand alone laptop... please can you help and advise.  Thanks

Viewing all 24688 articles
Browse latest View live




Latest Images