Quantcast
Channel: SQL Server Integration Services forum
Viewing all 24688 articles
Browse latest View live

SSIS transformations

$
0
0

Hi,

I am new to SSIS. I am working on a package with sample data where I have two tables.

Source and destination has to be in sync all the time when the package runs. So, I am stuck at the point where a record gets deleted in the source and my package has to recognize the deleted record in source and identify or output me the same record(deleted in source) from destination. I tried to use merge join and played around with all the joins but no luck.

I also used lookup transformation but it needs mapping to all the identical columns and my source and destination has different guids. I have a unique id column but I need the guid column which cannot be retrieved in the input buffer because no mapping exists.

What component do I need to use to filter the deleted records in destination?

Thanks for any help.


SSIS - "Violation of PRIMARY KEY constraint. Cannot insert duplicate key in object ''. The duplicate key value is . Though there are no duplicate records.

$
0
0

Hi,

I am providing support to one of our clients, where we have jobs scheduled to load the data from the tables in the source database to the destination database via SSIS packages. The first time load is a full load where we truncate all the tables in the destination and load them from the source tables. But from the next day, we perform the incremental load from source to destination, i.e., only modified records fetched using changed tracking concept will be loaded to the destination. After full load, if we run the incremental load, the job is failing with the error on one of the packages "Violation of PRIMARY KEY constraint. Cannot insert duplicate key in object '<tablename>'. The duplicate key value is <1234>, even though there are no duplicate records. When we try debugging and running the failing package, it runs successfully. We are not able to figure out why the package fails and when we run the next day it runs successfully. Request you to help me in this regard.

Thank you,

Bala Murali Krishna Medipally.

Does SSIS package execution require xp_regread and xp_instance_regread execution privileges

$
0
0

Hi,

I'm still trying to learn what privileges are required to execute packages in the SSISDB catalogue.   Our DBA requested that we REVOKE EXECUTE ON xp_instance_regread and xp_regread FROM public because it is a high vulnerability. 

However, I noticed that once they were revoked I was getting the following error when I tried to execute any packages...

The EXECUTE permission was denied on the object 'xp_regread', database 'mssqlsystemresource', schema 'sys'...

Which leads to my question .. Are xp_regread and xp_instance_regread execution privileges required to be able to execute packages?

Everywhere I look seems to point that these priviliges are required which maybe a problem for me. 

Thanks,

Rog  

High memory utilization

$
0
0
Hi,

We have a system(32GB RAM and 2 TB hard disk, Windows7,SQL SERVER 2008R2 enterprise 64 bit). Looks like whenever i run some query(even query result 50 records) on the database, the Memory utilization is very high(30 GB) in task manager. How can i control this over usage? The memory setting is default in server properties(min 0 and max 2147483647).

How to make SSIS Read an Excel File?

$
0
0

I’ve been trying for a while now (won’t say how long), to get BIDS to read a very simple Excel file. I’m talking SIMPLE!!  No matter what I don, SSIS keeps throwing an error, and of course it doesn’t say what the error is, so I can’t really debug it. I’ve tried this at least 20x with flat files, and it works perfect each time. I’ve done Data Conversions, Sorting, Union All, and several other Transformations; all work perfect. Trying to used Excel as a data source, is proving to be a mind numbing experience. Of over 20 different attempts, it hasn’t worked even once. I can make it as simple as you can possibly imagine, and SSIS, refuses to even perform the first step (I’m not even trying to create a table in SQL Server anymore). 

I have the Excel file path (very simple)

I have the Excel version (very simple)

I have the Connection Manager (very simple)

I have the Sheet name (very simple)

 

All I can do is see a preview of the Excel sheet before the process runs.  As soon as I het F5, I get an instant error, for no reason whatsoever, and no debugging options, whatsoever.

 

Is there some secret trick I need to know about? 

 

I can't show a snap shot, because MSFT won't allow pictures to be loaded now.

 

 

Attention All SSIS Gurus! Time to SPRING Into Action!

$
0
0

April fools out of the way, now let's find an April genius!

The name "April" is derived from the Latin verb "aperire", meaning "to open", as it is the season when trees, flowers AND MINDS start to open! And.. I can't wait to OPEN and read this month's community contributions! (groan, tenuous link!)

Things are indeed heating up around TechNet. The Wiki has become a shining example of what the community has to offer, and talent is SPRINGING FORTH from all corners of our garden of knowledge. 

If you can find the time to enrich us with your latest revelations, or some fascinating facts, then not only will you build up a profile and name for yourself within the gaze of Microsoft's very own glitterati, but you will be adding pages to the most respected source for Microsoft knowledge base articles. This could not only boost your career, but would benefit generations to come!

So don't be an April fool. Please realise the potential of this platform, realise where we are going, and join us in growing this community, learning more about you, and opening the minds of others!

All you have to do is add an article to TechNet Wiki from your own specialist field. Something that fits into one of the categories listed on the submissions page. Copy in your own blog posts, a forum solution, a white paper, or just something you had to solve for your own day's work today.

Drop us some nifty knowledge, or superb snippets, and become MICROSOFT TECHNOLOGY GURU OF THE MONTH!

This is an official Microsoft TechNet recognition, where people such as yourselves can truly get noticed!

HOW TO WIN

1) Please copy over your Microsoft technical solutions and revelations toTechNet Wiki.

2) Add a link to it on THIS WIKI COMPETITION PAGE (so we know you've contributed)

3) Every month, we will highlight your contributions, and select a "Guru of the Month" in each technology.

If you win, we will sing your praises in blogs and forums, similar to the weekly contributor awards. Once "on our radar" and making your mark, you will probably be interviewed for your greatness, and maybe eventually even invited into other inner TechNet/MSDN circles!

Winning this award in your favoured technology will help us learn the active members in each community.

Feel free to ask any questions below.

More about TechNet Guru Awards

Thanks in advance!
Pete Laker


#PEJL
Got any nice code? If you invest time in coding an elegant, novel or impressive answer on MSDN forums, why not copy it over toTechNet Wiki, for future generations to benefit from! You'll never get archived again, and you could win weekly awards!

Have you got what it takes o become this month's TechNet Technical Guru? Join a long list of well known community big hitters, show your knowledge and prowess in your favoured technologies!

How do I edit a package?

$
0
0

I'm sorry this question is so fundamental, but that's where I'm at. I've started a new job. I found a problem in the SQL that's a part of a SSIS package written by a third party, which is in our SQL Server 2008 R2 database. I've discussed this with the third party person; she's given me the necessary changes to the SQL. But I don't know what to do next.

We've got this package on a test database server. I've got SSIS open to Stored Packages and have drilled down to the necessary folders to find it. But beyond that I don't know what to do. I understand that BIDS is involved, but that's as far as my understand of it goes.

What do I do next?


Rod

What is the dataype for nvarchar(max) & varchar(max) in SSIS?

$
0
0
I am exporting a bunch of columns from SQL Server to Excel, using SSIS.  Three of the columns wont convert, inthe dataviewer the values show up as <long text>.  I have tried both text stream types (dt_text and ndt_text) and neither worked. What else should I try?

Source Script Component to read data from Sharepoint List - read multi choice field values

$
0
0

All examples I found refer to classes under Microsoft.SharePoint namespace. However, I have the SharePoint CSOM that only gives me the Microsoft.Sharepoint.Client namespace.

I need to read the selected values of a multichoice field, but not sure how to do it with classes in the namespace above.

everthing works, exept the TSQL_x0020_Reference_x0020_Numbe field.

my code looks like this:

****

Webweb = cont.Web;

            cont.Load(web);

            cont.ExecuteQuery();

           

Listsstest = web.Lists.GetByTitle("T-SQL Code Review Tracking");

           

//CamlQuery query = CamlQuery.CreateAllItemsQuery();


           

CamlQueryquery =newCamlQuery();

            query.ViewXml =

                   

@"<View>

                           <ViewFields>

                                <FieldRef Name='Category'/>

                                <FieldRef Name='Review_x0020_Type'/>

                                <FieldRef Name='Review_x0020_Start_x0020_Date'/>

                                <FieldRef Name='Title'/>

                                <FieldRef Name='Location'/>

                                <FieldRef Name='Project'/>

                                <FieldRef Name='Author0'/>

                                <FieldRef Name='AssignedTo'/>

                                <FieldRef Name='TSQL_x0020_Reference_x0020_Numbe'/>

                            </ViewFields>

                         </View>"

 ListItemCollectionitems = sstest.GetItems(query);

            cont.Load(items);

            cont.ExecuteQuery();

           

foreach(ListItemiteminitems)

            {

                Output0Buffer.AddRow();

                Output0Buffer.ReviewStatus = item.FieldValues[

"Category"] !=null? item.FieldValues["Category"].ToString() : String.Empty;

                Output0Buffer.ReviewType = item.FieldValues[

"Review_x0020_Type"] !=null? item.FieldValues["Review_x0020_Type"].ToString() : String.Empty;

                Output0Buffer.ReviewDate =

DateTime.Parse(item.FieldValues["Review_x0020_Start_x0020_Date"].ToString());

                Output0Buffer.Module = item.FieldValues[

"Title"] !=null? item.FieldValues["Title"].ToString() : String.Empty;

                Output0Buffer.BranchLocationURL = item.FieldValues[

"Location"] !=null? item.FieldValues["Location"].ToString() : String.Empty;

                Output0Buffer.ProjectName = item.FieldValues[

"Project"] !=null? item.FieldValues["Project"].ToString() : String.Empty;

                Output0Buffer.Author = item.FieldValues[

"Author0"] !=null? item.FieldValues["Author0"].ToString() : String.Empty;

 


               

FieldLookupValueflvAssignedTo =newFieldUserValue();

                flvAssignedTo = item.FieldValues[

"AssignedTo"]asFieldLookupValue;

               

if(flvAssignedTo !=null)

                {

                    Output0Buffer.AssignedTo = flvAssignedTo.LookupValue;

                }

               

varv = item.FieldValues["TSQL_x0020_Reference_x0020_Numbe"];

               

if(v !=null)

                {

                    Output0Buffer.Reason2 = v.ToString();

                }

 

 

             }

             Output0Buffer.SetEndOfRowset();           

Why don't installed SSIS Add-ins appear in either the Toolbox or SSIS Toolbox?

$
0
0

Greetings!

Why don't installed SSIS Add-ins appear in either the Toolbox or SSIS Toolbox? I have restarted Visual Studio and (heard this might work) restarted the SSIS service, but I still never see any installed add-in.

Thank you.


---------------- GeorgeS, aka "SQLPadawan"

Looking for a working DotNetOpenAuth or OAuth example

$
0
0

Does anyone have a working code snip that uses DotNetOpenAuth or OAuth to authenticate to an API that uses OpenID? I'm trying to incorporate an API call into a script component, but I'm blocked because I'm not able to handle the OPENID authentication method

Thanks

executing ssis hangs

$
0
0

Hi, there;

I created a SSIS package which loads CSV into database. The package is called in a C# console application which is set as scheduled task in a server.

The problem I having is that the package hangs during validation stage: 

"Truncation may occur due to inserting data from data flow column "ReadingType" with a length of 50 to database column "ReadingType" with a length of 2."

There is no problem loading same data from development machine.

Can some give a clue what might causes this?

Cheers,

Qiuwei


qiuwei

Tracking the SSIS ETL process

$
0
0

Hi 

I have a ETL ( SSIS ) Process in which i am loading around 150 tables in each run. ( Truncate and Insert ). I have four packages each from different sources. ( Each package loads different tables and different numbers )

These are run on weekly basis one after the other. Each package is taking around 60 to 90 minutes each. Now i want to track the progress of the ETL on my front End application. 

We want this in two ways. 

First Way : I need to show the user how much percent of ETL Process is completed 

Second Way : I need to show the No of tables completed and how many rows have been completed in the ongoing table ( which is in process )

Could any one please help me out how to design the table and ssis process.


Balaji - BI Developer

How to perform series operation in SSIS

$
0
0

I have scenario in which I am getting a row from XLS and in that ro I have data for multiple tables based on if one table has some data or not

EXAMPLE :

Search a table in database on basis of column in the xls row
STEP #1 If TABLE A has that row then do nothing and move to next step else insert new row into TABLE A and move to next step

STEP #2 If TABLE B has that row then do nothing and move to next step else insert new row into TABLE B and move to next step

STEP #3 If TABLE C has that row then do nothing and move to next step else insert new row into TABLE C and move to next step

STEP #4 If TABLE D has that row then do nothing and move to next step else insert new row into TABLE D and move to next step

STEP #5 If TABLE E has that row then do nothing  else insert new row into TABLE E

Currently I am trying to achieve this by using multicast and from that multicast putting inputs into 5 OLE DB DESTINATION but by doing this some time one step execute earlier then previous and integrity constrain violated so getting error


prakash kumar jha

What is the easiest way to export all tables data from Oracle to MS SQL Server?

$
0
0

Hello MS,

I would like to export all tables from Oracle 11.2 to MS SQL Server 2012 R1.

Using the tool "Microsoft SQL Server Migration Assistant v6.0 for Oracle" did not work for me because there are too many warnings and errors regarding the schema creation (MS cannot know it because they are not the schema designer). My idea is to leave/skip the schema creation to the application designer/supplier and instead concentrate on the Oracle data export and MS SQL data import.

What is the easiest way to export all tables data from Oracle to MS SQL Server quickly?

Is it:

- the „MS SQL Import and Export Data“ Tool
- the “MS SQL Integration Services” Tool
- not Oracle dump *.dmp format because it is a propritery binary format
- flat file *.csv (delimited format)

Thanks!






SQL server 2014 and VS 2013 - Dataflow task, read CSV file and insert data to SQL table

$
0
0

Hello everyone,

I was assigned a work item wherein, I've a dataflow task on For Each Loop container at control flow of SSIS package. This For Each Loop container reads the CSV files from the specified location one by one, and populates a variable with current file name. Note, the tables where I would like to push the data from each CSV file are also having the same names as CSV file names.

On the dataflow task, I've Flat File component as a source, this component uses the above variable to read the data of a particular file. Now, here my question comes, how can I move the data to destination, SQL table, using the same variable name?

I've tried to setup the OLE DB destination component dynamically but it executes well only for first time. It does not change the mappings as per the columns of the second CSV file. There're around 50 CSV files, each has different set off columns in it. These files needs to be migrated to SQL tables using the optimum way.

Does anybody know which is the best way to setup the Dataflow task for this requirement?

Also, I cannot use Bulk insert task here as we would like to keep a log of corrupted rows.

Any help would be much appreciated. It's very urgent.


Thanks, <b>Ankit Shah</b> <hr> Inkey Solutions, India. <hr> Microsoft Certified Business Management Solutions Professionals <hr> http://ankit.inkeysolutions.com

Script Task Issue: While Statement and MessageBox.Show()

$
0
0

I have written the below code to count the number of Month-Year combination present in the excel file. when i use the original file, it works fine. But the moment i add one more Month-Year combination , it doesnt work. Like in the below screenshot, the total number of month-Year Column is 2 and Message Box is also showing 2. But the moment i add one more Month-Year combination say Apr-15, the Message Box doesnt pop up.

while (dt.Rows[InventoryRow][MonthColNum].ToString().Trim() !="")

               

            {

              MonthColNum++;

             }

           

MessageBox.Show("Month Col Number "+MonthColNum.ToString()); 

Message Box shows the Month Col Number when i use the original file. But when i add one more Month-Year, it doesnt show the Message.

Could any one please tell me why it is happening?

Full Code:

/* Microsoft SQL Server Integration Services Script Component

*  Write scripts using Microsoft Visual C# 2008.

*  ScriptMain is the entry point class of the script.*/



using

System;


using

System.Data;


using

Microsoft.SqlServer.Dts.Pipeline.Wrapper;


using

Microsoft.SqlServer.Dts.Runtime.Wrapper;


using

SC_8f65b1fd232e416084be524f330491a4.csproj;


using

System.Text;


using

System.Data.OleDb;


using

System.Windows.Forms;


using

System.Collections.Generic;

[Microsoft.SqlServer.Dts.Pipeline.

SSISScriptComponentEntryPointAttribute]


public

classScriptMain:UserComponent


{

   

DataTabledt;

   

conststringDelimeter =" ";

   

intInventoryRow = 3;

   

intColumnOffset = 0;

    

publicoverridevoidPreExecute()

    {

       

base.PreExecute();

     }

   

publicoverridevoidPostExecute()

    {

       

base.PostExecute();

     }

   

publicoverridevoidCreateNewOutputRows()

    {

       

try


        {

            dt = LoadWorkbook(Variables.VarExcelFileName,

"Sheet1");

           

intMonthColNum = 0;

           

while(dt.Rows[InventoryRow][MonthColNum].ToString().Trim() !="")

               

            {

              MonthColNum++;

             }

           

MessageBox.Show("Month Col Number "+MonthColNum.ToString()); 

         

           

stringDate;          

catch(Exceptionex)

        {

        }

    }

   

publicstaticDataTableLoadWorkbook(stringFileName,stringSheetName)

    {

   

       

stringconnString ="Provider=Microsoft.ACE.OLEDB.12.0;Data Source="+ FileName +";Extended Properties=\"Excel 12.0;HDR=No;IMEX=1\"";

       

//string connString = "Provider=Microsoft.JET.OLEDB.4.0;Data Source=" + System.Web.HttpContext.Current.Server.MapPath(FileName) + ";Extended Properties=Excel 8.0";



       

// Create the connection object


       

OleDbConnectionoledbConn =newOleDbConnection(connString);

       

try


        {

         

           

// Create a DataSet which will hold the data extracted from the worksheet.


           

DataSetds =newDataSet();

           

// Open connection


            oledbConn.Open();

           

DataTableobjSheetNames = oledbConn.GetSchema("Tables");

           

OleDbCommandcmd;

           

if(SheetName ==null&& objSheetNames.Rows.Count > 0)

            {

               

// Create OleDbCommand object and select data from worksheet Sheet1


                cmd =

newOleDbCommand("SELECT * FROM ["+ objSheetNames.Rows[0][2].ToString() +"]", oledbConn);

            }

           

else


            {

                cmd =

newOleDbCommand("SELECT * FROM ["+ SheetName +"$]", oledbConn);

            }

           

// Create new OleDbDataAdapter


           

OleDbDataAdapteroleda =newOleDbDataAdapter();

            oleda.SelectCommand = cmd;

           

// Fill the DataSet from the data extracted from the worksheet.


            oleda.Fill(ds,

"LabOrder");

      

           

// Bind the data to the GridView


           

returnds.Tables[0];

           

        }

       

finally


        {

           

// Close connection


            oledbConn.Close();

        }

    }

   

privatestaticstring[] _InvalidChars = {",","'","\""};

   

publicstaticstringSanitizeInput(stringvalue)

    {

       

returnSanitizeInput(value, _InvalidChars);

    }

   

publicstaticstringSanitizeInput(stringvalue,string[] InvalidChars)

    {

        value = value.Trim();

       

foreach(stringInvalidCharinInvalidChars)

        {

            value = value.Replace(InvalidChar,

"");

        }

       

returnvalue;

       

//IEnumerable<char> chars = InvalidChars.SelectMany(x => x);


       

//return new string(value.Where(c => !chars.Contains(c)).ToArray());


    }

}

        }

       

The type of the value being assigned to variable ... differs from the current variable type

$
0
0

I am trying to load a variable on SSIS Execute SQL Task with a string value. I keep on getting an error message :-

“The type of the value being assigned to variable “User::LegacyColumns” differs from the current variable type”.

Below are the settings on my package:

Execute SQL Task Result Set – Single Row

Variable Data Type – String

The data to be loaded on the variable is a single row shown below:

FirstName,LastName,MiddleName,PatientType,Title

Can someone kindly help me to solve this as I’m lost for ideas.

Thanks,


Mpumelelo

Modify SSIS package saved on server

$
0
0

   I have saved an SSIS pacakage on my sql server.  I am able to see and run the package through Integrations services.  My question is, is there a way to edit the package through Management studio?

 

Thanks

Jim

Need help ASAP with Data Flow Task Flat File Connection

$
0
0

Hey there,

I have a Data Flow Task within a ForEach loop container.  The source of the flow is ADO.NET connection and the destination is a Flat File Connection.  I loop through a collection of strings in the ForEach loop.  Based on the string content, I write some data to the same destination file in each iteration overwriting the previous version. 

I am running into following Errors:

[Flat File Destination [38]] Warning: The process cannot access the file because it is being used by another process.
[Flat File Destination [38]] Error: Cannot open the datafile "Example.csv".
[SSIS.Pipeline] Error: Flat File Destination failed the pre-execute phase and returned error code 0xC020200E.

I know what's happening but I don't know how to fix it.  The first time through the ForEach loop, the destination file is updated.  The second time is when this error pops up.  I think it's because the first iteration is not closing the destination file. How do I force a close of the file within Data Flow task or through a subsequent Script Task.

This works within a SQL 2008 package on one server but not within SQL 2012 package on a different server.

Any help is greatly appreciated.

Thanks! 

Viewing all 24688 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>