Quantcast
Channel: SQL Server Integration Services forum
Viewing all 24688 articles
Browse latest View live
↧

Best way to load huge chunks of data

September 18, 2018, 7:22 am
$
0
0

Hello All,

I have a question regarding the best way to load huge amounts of data. I am using SSIS to load load data from source into the destination. Both are sql server tables. There are two packages and they need to run sequentially as the second package is dependent on the first one (for some calculations).

I have to load data from Jan 1st  to August 2018 (80 million plus records).

My question is when i load the first table (can i partition the table and make it run in batches), will that speed up the load? Its not fast load , but loading using sql command. If partitioning the table the best way to speed up, please point me to a link or document.

Thanks

↧
Search

SSIS - API - For Loop - 70k in ForLoop - Need backup

September 22, 2018, 8:41 pm
$
0
0

Hi everybody,

I need some assistance with an SSIS task which uses a For Loop.

Essentially I am making 70k individual API Calls inside a For Loop. So needless to say, it is time consuming task. Likely 12-15 hours.

Each API call extracts 1 row of data and is sent to a SQL Database.

I continue to hit unforeseen errors at different stages during the process. Most of the time, it is from a failure to connect to the SQL Database.

Can anybody recommend to me a strategy to counter?

I've tried Checkpoints as per below article. This doesn't seem to work, because a For Loop is considered 1 task when it completed. Rather than 70k tasks. 

https://www.mssqltips.com/sqlservertip/1408/integration-services-checkpoints-to-restart-package-from-failure/

I've also played around with the event handler, but I don't know how to use the event handler to re start the For loop at the position where it failed.

Another idea, was if I could possibly cache the 70k of rows and send them to the SQL Database all at once?

Any help would be greatly appreciated.

Thank you.

Cameron

↧
↧

how to place variable instead of manual date in the source query from oracle

September 22, 2018, 9:14 am
$
0
0

Hi  all,

i am using source as oracle query and i wanted to put date variable instead of manual date in the below query.

select * from sample where
 '20-JAN-2018'  BETWEEN start_date AND end_date
AND '20-JAN-2018' BETWEEN effective_start_date AND effective_end_date

in place of '20-JAN-2018' , i wanted to create variable in ssis and i need to pass that variable.could any one suggest me how to do here.

quick response is really appreciated.

Thanks

↧

The Script Task uses version 15.0 script that is not supported in this release of Integration Services

August 21, 2017, 12:49 pm
$
0
0

Hi All,

We are migrating an SSIS project from SQL 2012 to SQL 2016. The project upgraded successfully without any issues and able to run few packages without any errors. But there are few packages which has got the script task and they are failing. The error message is as below: -

Error: There was an exception while loading Script Task from XML: System.Exception: The Script Task "ST_bb19096e98ab45d2a2d6132b995cd509" uses version 15.0 script that is not supported in this release
of Integration Services. To run the package, use the Script Task to create a new VSTA script. In most cases, scripts
are converted automatically to use a supported version, when you open a SQL Server Integration Services package
in %SQL_PRODUCT_SHORT_NAME% Integration Services. at
Microsoft.SqlServer.Dts.Tasks.ScriptTask.ScriptTask.Load
FromXML(XmlElement elemProj, IDTSInfoEvents events)

This runs fine when run from BIDS (VS2015). It also runs fine if the project is deployed from BIDS and run through agent job. But when the deployment happens through .ispac (deployment) file, this error occurs. Can somebody suggest how to get past this error?

Is it any known issues with migration or with SSIS 2016?

Thanks.


↧

SSIS Debugger - Unable to start 'DTS'

September 21, 2018, 10:13 am
$
0
0

When trying to execute SSIS packages in VS 2017, I am receiving the error shown below. I have read other threads and one solution listed is to run the Visual Studio Installer to add the Windows Communication Foundation, which I did and that did not help. I also tried running repairs of Visual Studio and VSTA. My development environment had been working and I don't know what changed. I went as far as uninstalling VS 2017 and reinstalling it, only to receive the same error message when executing packages.

Phil

Here is the error:

===================================

Unable to start debugging.
The debugger is not properly installed.
Run setup to install or repair the debugger. (Microsoft Visual Studio)

------------------------------
Program Location:

at Microsoft.DataTransformationServices.Project.DataTransformationsPackageDebugger.LaunchVsDebugger(IVsDebugger iVsDebugger, DataTransformationsProjectConfigurationOptions options)
at Microsoft.DataTransformationServices.Project.DataTransformationsPackageDebugger.ValidateAndRunDebugger(Int32 flags, IOutputWindow outputWindow, DataTransformationsProjectConfigurationOptions options)
at Microsoft.DataTransformationServices.Project.DtsPackagesFolderProjectFeature.ExecuteTaskOrPackage(ProjectItem prjItem, String taskPath)

===================================

Unable to start program 'DTS'.

The debugger is not properly installed. Cannot debug the requested type of code. Run setup to install or repair the debugger. (Microsoft Visual Studio Debugger)

------------------------------
Program Location:

at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at Microsoft.DataTransformationServices.Project.DataTransformationsPackageDebugger.LaunchVsDebugger(IVsDebugger iVsDebugger, DataTransformationsProjectConfigurationOptions options)

↧
↧

How to pass parameter value while executing through dtexec

September 23, 2018, 10:10 am
$
0
0

Hi All, 

I have to execute the package through dtexec command line by overriding parameters. I am trying with following syntax but not being able to do it successfully. Tried to search online but could not found posts where this explained. Can someone help me here ?

Dtexec /ISServer "\SSISDB\Dev\IDSS1\SPasSRC.dtsx"  /parameter $Project::P_ProductID;111

Thanks

↧

SSIS Script Component to convert DateTime format

September 20, 2018, 6:26 pm
$
0
0

Hi,

 Can you please help me with the C# code  to convert DATETIMESTAMP in the format YYYY-DD-MM HH:MM:SS.FFFFFF  to YYYY-DD-MM HH:MM:SS.FFFFF. The code must convert the scale value (fraction second) from 6 to 5. I would like to achieve this in SSIS script component (transformation). The input field is of DATETIMESTAMP2(6) data type and once the data is transformed to scale value 5 ,it will be inserted into a table with data type DATETIMESTAMP2(5).

Note: I don't want to do this in derived column.

↧

how to use sql command from variable in the ADO.NET source

September 24, 2018, 11:46 pm
$
0
0

hi,

can anybody explain me how to do command from variable access mode using ADO.NET source,i am able to see here only two options.i wanted to use ADO.net source with command line from variable,could any one suggest me here for alternative way.

↧
Search

عناوين صيانة ديب فريزر بيكو مدينتى ٠1207619993 @ ٠235700997 توكيل معتمد بيكو

September 25, 2018, 1:02 am
$
0
0
عناوين صيانة  ديب فريزر بيكو مدينتى  ٠1207619993 @ ٠235700997 توكيل معتمد بيكو 
↧
↧

SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E21

September 24, 2018, 8:08 am
$
0
0

Am importing data from execl using ssis package . 1st i got data type miss matching issue, that time am took a data conversion transformation. but again Am getting bellow error messages while executing the package.

[OLE DB Destination [495]] Error: SSIS Error Code DTS_E_OLEDBERROR.  An OLE DB error has occurred. Error code: 0x80040E21.
An OLE DB record is available.  Source: "Microsoft SQL Server Native Client 11.0"  Hresult: 0x80040E21  Description: "Multiple-step OLE DB operation generated errors. Check each OLE DB status value, if available. No work was done.".

[OLE DB Destination [435]] Warning: Truncation may occur due to inserting data from data flow column "DOC_PRF" with a length of 50 to database column "DOC_PRF" with a length of 2.

[OLE DB Destination [495]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "OLE DB Destination.Inputs[OLE DB Destination Input]" failed because error code 0xC020907B occurred, and the error row disposition on "OLE DB Destination.Inputs[OLE DB Destination Input]" specifies failure on error. An error occurred on the specified object of the specified component.  There may be error messages posted before this with more information about the failure.


[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED.  
The ProcessInput method on component "OLE DB Destination" (495) failed with error code 0xC0209029 while processing input "OLE DB Destination Input" (508). The identified component returned an error from the ProcessInput method. 
The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.  There may be error messages posted before this with more information about the failure.


Thanks Bala Narasimha

↧

Adding Script Component to VS2015 generated package doesn't run on SQL Server 2016

August 31, 2017, 9:39 am
$
0
0

We are upversioning from SQL 2012 with a BIML-generated package containing a script component. We're receiving an error that I am able to reproduce in a fresh project without BIML.

The error is:

Failed validation and returned error code 0xC0048021.

The component is missing, not registered, not upgradeable, or missing required interfaces. The contact information for this component is "Includes and runs custom script code. For example, apply a business rule that limits the range of valid values in an "income" column or add values in two columns and calculate the average of the sum.;Microsoft Corporation; Microsoft SQL Server; Microsoft Corporation; All Rights Reserved; http://www.microsoft.com/sql/support;9".

The component metadata for "Parse PatientContacts, clsid {4F885D04-B578-47B7-94A0-DE9C7DA25EE2}" could not be upgraded to the newer version of the component. The PerformUpgrade method failed.

Microsoft.SqlServer.Dts.Pipeline.ComponentVersionMismatchException: The version of Parse PatientContacts is not compatible with this version of the DataFlow.  [[The version or pipeline version or both for the specified component is higher than the current version.  This package was probably created on a new version of DTS or the component than is installed on the current PC.]]
   at Microsoft.SqlServer.Dts.Pipeline.ManagedComponentHost.HostCheckAndPerformUpgrade(IDTSManagedComponentWrapper100 wrapper, Int32 lPipelineVersion)

To reproduce this error I:

  • Created a new project in Visual Studio 2015
  • Configured the TargetServerVersion to be SQL Server 2016
  • Created a data flow with a Script Component between my Data Source and Destination
  • Saved and built the project. Picked up the .ispac file from the bin folder
  • Deployed the Project on my SQL Server 2016 instance using SSMS 17
  • Executed the package configuring the Environment to the correct values.

Any help would be greatly appreciated.

↧

SSIS Agent job Issue

September 21, 2018, 9:40 am
$
0
0

Hello All,

Recently our server were upgraded from SQL SERVER 2012 to 2014. our local machines have only 2012. After upgrading the servers when i was running the ssis existing agent job, able to load the data but end of the step job is showing as failed. If we have any issues in packages it's not capturing the error message in our audit table also. Can you please help anyone based on below error message. 

Executed as user: US\sqlproxy. Microsoft (R) SQL Server Execute Package Utility  Version 12.0.5214.6 for 64-bit  Copyright (C) Microsoft Corporation. All rights reserved.    Started:  3:26:52 PM  Package execution on IS Server failed. Execution ID: 58600, Execution Status:4.  To view the details for the execution, right-click on the Integration Services Catalog, and open the [All Executions] report  Started:  3:26:52 PM  Finished: 3:47:05 PM  Elapsed:  1212.35 seconds.  The package execution failed.  The step failed. "

Thanks,

CMK 

↧

Dataflow extration limit stop at 3XX XXX record from oracle DB with OleDB 12.2 driver

September 25, 2018, 1:55 am
$
0
0

Hi, I got a strange behavior during extraction process into my data flow.
I got a source from OLE DB Source (oracle with last driver 12.2.XX) and all run fine but at 348,128 record, my process stop like freeze (it's like running... neverend), no record coming out over from my source task.

I try selecting one one field over my full SELECT and it's good... all data (one field) arrive to my destination (> 1,3 millions).

So I don't understand how can I cope with this problem... it's oracle provider problem? Something related to memory... it's the first time from 10 years that I got a problem like this.I'm using data tools 2010... but it's a very simple flow... from a source to a trash destination.

I hope someone can give me some good feedback about it, 'cause there are no setting in this kind of connection to oracle... I'm thinking about something related to oracle provider... I try with Microsoft OleDB provider x Oracle or .NET providers but are more slowly and always stop to lower numbers of record.

With few fields process terminate successful...

ALEN, Italy

↧
↧

SSIS runs successfully but latest record is not inserting into table

September 11, 2018, 2:03 pm
$
0
0

Hi All,

I have source A and source B two databases on different sql servers. Need to bring data from source A to source B for few columns by comparing column A in source A database and column A in source B database if data matches inserting into source B database.

This SSIS package runs successfully insert operation is working fine but data which is coming from source A database is not latest one. It is taking the first inserted data.

But I need data of latest timestamp (last_updated) latest row of data for that particular row value.

ex : source A have 10 rows of data with column A having same value. it is taking "2018-07-05 08:20:26.6968390" instead of"2018-08-16 14:34:36.2422430"

Can you please help me on this.

Thanks in Advance.

Thanks

Naga

Thanks, Naga

↧

Transfer db Vs Backup and Restore

September 25, 2018, 3:50 am
$
0
0

Hi guys, 

Which is the difference between:

A Taking a Backup (DB Abc) and Restore it as DB Emn

and

Using Transfer DB from Abc to Emn?

Shall I consider something before dropping all the backup and restore job I have around (they are annoying)? 

↧
Search

SSIS

September 24, 2018, 12:09 pm
$
0
0

Hi All,

Please suggest to the below question.

I did not get good quality of work in SSIS in my current project.because of that I am not more confident on that,But I want some good exposure, I need some hands on experience.In the future if i will get any project so i can work independently.

Please suggest me how to achieve this.

Thanks

↧

Need help: How to import csv file with inconsistent columns?

September 25, 2018, 6:23 am
$
0
0

I have a csv file with the following columns:

A , B ,C ,D ,E

I would like to import records from only the following columns B and D excluding others, the data type is fixed for B and D.

The csv file can have inconsistent columns the next day which only shows:

B, D, E

or A, B, D

When using SSIS Flat File connection manager and point it to the csv file, I must add all the columns A,B,C,D,E

but when column A or C or E is missing the component failes due to mismatch in data type.

Is it possible to scan through the csv file header and select only columns B and D only? 

Note:

Column B and D are always available.

I appreciate any help, thanks in advance.

↧
↧

BCP Command through Putty

September 25, 2018, 6:44 am
$
0
0
Hi Excerpts,

I would require your help on below issue.I am new in MSSQL.

My file is present in Linux server and SQL server is installed in Different server. Customer was excepting data from file (Which is present different server) to MSSQL Table. Can you please leave your comment on below questions

1. My problem is we can't run BCP command in Linux system through putty ?
2. Customer was saying BCP is installed in Linux box but i couldn't able to find bcp in Linux ? How to find BCP is installed or not in LINUX box ?
3. If bcp is possible through putty then what is the exact command ? Can you please give some example.

Your help highly appreciated.
↧

How to import data from Rest API which return JSON

December 15, 2017, 11:08 am
$
0
0
I have a requirement to extract data from Rest API which return JSON and along with it I will also have to pass variable to extract multiple reports. Thanks for the help in advance.
↧

how to load multiple source folders related files into destionation table using single foreachloop ssis package.

September 25, 2018, 8:25 am
$
0
0
Hi I have one doubt in ssis,

how to load multiple source folders related files into destionation table  using  single foreachloop ssis package.

source files are available in two different folder and  all files related structure is same in the both folder.
source file have paths : 
c:\chenlocation\ in the source path have files like : emp.txt,emp1.txt,emp2.txt
c:\punelocation\ in the source path have files like : emp_pune.txt,emp_pune1.txt,emp_pune2.txt,emp_pune4.txt.

in the two source path (chenandpunelocation paths)all files structure are same and look like columns
id,name,sal.
destination table (sql server ) is : emp and columns are same id,name,sal


in ssis package I have implemented like below :
declare variables : 

chenlocationpath :  c:\chenlocation\
filename         : emp.txt
punelocationpath : c:\punelocation\

afte that drag and drop foreachloop conatiner and type >foreachfileenumerator>directory >checnloationpath variable>filetype>*.txt
>variable mapping >filename variable.
after that drag and drop dft task and confiugre flatfile source and change that datatype after that destination sql server table is configured.

after that again drag and drop 2nd foreachloop container and type >foreachfileenumerator>directory >punelocationpath variable>filetype>*.txt
>variable mapping >filename variable.
after that drag and drop dft task and confiugre flatfile source and change that datatype after that destination sql server table is configured.
after exectuction all records are loaded in the destination table without failure.


here I have used 2 foreachloop tasks to process two different source path .
I want to impleemnt single foreachloop with two different folder path into  destination table 

can you please tell me how to load two different path of source files  data into destination table using single foreachloop container in ssis package .

↧
Viewing all 24688 articles
Browse latest View live
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>