I am wondering if anyone has had any experience with using Foreach Loop Container in SSIS? I have it set up and working correctly but I'm trying not to run the query each time it goes through the foreach
loop. Here is my process 1. Run a query to get my variables (last 3 of a transaction) 2. Run each on of those (so there are 000-999) through the foreach loop and I also pass that last 3 into the query to pull back a portion of the data. This is a huge amount
of data and pulling/transmitting 1000 files is ok, I just don't want to run the query 1000 times (1 for each file). What I want is to run the query once and distribute the data in 1000 files. I would even be ok with running it 10 times. On some smaller data
sets, I actually used the Balance Data Distributor, but I had to create 10 different flat file connections for it. I don't want to have to create 1000 flat file connectors for this project.. HELP !!!
↧
SSIS Large Dataset extraction
↧
SSIS Help needed: Create export file only if there are records to be written.....
I have a gnarly sql statement that is used in SSIS to copy table content to a file. Now we want to automate this but don't want to create a file in those cases where it would be empty after the copy. What's a simple way of achieving this? (I inserted an Execute Sql task in the Control flow which did a select count(*) statement, passed that count value back in a user variable, edited the Success constraint to check for User::count > 0. Seems clumsy at best.)
edm2
↧
↧
SSISDB on AlwaysOn Group running package on Secondary Replica
Hi, I have sql server 2016 Enterprise Edition. I have created AlwaysOn Group. and added SSISDB. I do have couple SSIS packages running on primary replica with no errors. but when failing over to second replica, the jobs fails.
Asking to create master Key or certificate.
so my question is this: is there a way to automate this and whenever FaillOver happen the jobs should be running as they are on Primary replica?
Thank you all.
Rashid
↧
Who will be announced as the next SQL Server Integration Services Guru? Read more about May 2019 competition!
What is TechNet Guru Competition?
Each month the TechNet Wiki council organizes a contest of the best articles posted that month. This is your chance to be announced as MICROSOFT TECHNOLOGY GURU OF THE MONTH!
One winner in each category will be selected each month for glory and adoration by the MSDN/TechNet Ninjas and community as a whole. Winners will be announced in dedicated blog post that will be published in Microsoft Wiki Ninjas blog, a tweet from the Wiki Ninjas Twitter account, links will be published at Microsoft TNWiki group on Facebook, and other acknowledgement from the community will follow.
Some of our biggest community voices and many MVPs have passed through these halls on their way to fame and fortune.
If you have already made a contribution in the forums or gallery or you published a nice blog, then you can simply convert it into a shared wiki article, reference the original post, and register the article for the TechNet Guru Competition. The articles must be written in May 2019 and must be in English. However, the original blog or forum content can be from beforeMay 2019.
Come and see who is making waves in all your favorite technologies. Maybe it will be you!
Who can join the Competition?
Anyone who has basic knowledge and the desire to share the knowledge is welcome. Articles can appeal to beginners or discusse advanced topics. All you have to do is to add your article to TechNet Wiki from your own specialty category.
How can you win?
- Please copy/Write over your Microsoft technical solutions and revelations to TechNetWiki.
- Add a link to your new article on THIS WIKI COMPETITION PAGE (so we know you've contributed)
- (Optional but recommended) Add a link to your article at the TechNetWiki group on Facebook. The group is very active and people love to help, you can get feedback and even direct improvements in the article before the contest starts.
Do you have any question or want more information?
Feel free to ask any questions below, or Join us at the official MicrosoftTechNet Wiki groups on facebook. Read More about TechNet Guru Awards.
If you win, people will sing your praises online and your name will be raised as Guru of the Month.
↧
Dataflowtask shows wrong entries from SAP - MBEW fields
Hello community,
I am fetching different tables from SAP via ODBC and everything works fine. But when I am touching MBEW or MBEWH with the fields LBKUM or SALK3 I am getting just 0.00 or completely wrong entries, and that in just those two fields. Same amount of rows which perfect match together, except those two fields.
Has anyone an idea?
Thanks
Mario
↧
↧
SSIS package ran on 2010, fails on 2016 with Cannot open datafile. What am I missing?
I have a very simple Data Flow task that moves the results of a query to a *.csv file on a network drive. It ran fine on SSIS 2010, but not now on SSIS 2016. Note: I have another solution with over 100 packages that moved over fine (all are generated from BIML files), but it does not access a network drive.
When I run it on my desktop on VS 2017 it runs successfully.
Executing it manually in SSMS 2016 (as myself, Integration Services Catalog) it fails with Error: Cannot open the datafile "\\<server>\<root>\<dir>\<filename>.csv" and Warning: Access is denied. First line error is: 0xC020200E. Scheduled in Agent fails with same error.
I've done this:
deployed from VS 2017 with target server set to 2016
Re-configured the package's connection managers, tried various login ID's and two proxies, changed connection Managers from task level to package level.
The box itself has access to the drive and file. Checked/re-set permissions to folder and file, Effective permissions are good.
Deleted existing output file.
Does Sql-Server 2016 have an off switch to access network drives?
Any thoughts will be gratefully appreciated.
Mac McCaskie
↧
SSIS OLEDB - Receiving error ORA-01403 No Data Found when trying to load records into oracle GTT
I am trying to load records from one table, on a SQL Server, into a global temp table on a Oracle server. I'm able to load the records successfully if I set the dataflow task to a default table name, but it runs far to slow for what I need. It's when I set it to a FAST load, that I run into this error message. I am using the Native OLE DB\Oracle Provider for Oracle. Initially, I thought it was an issue with the oracle provider, but whenever I try this on a different table (permanent), on a different server, it runs fine. I tried switching with Microsoft's oracle provider, but that threw a different error message entirely.
Would there be a setting on the GTT that would block fast loads, or would it be something else entirely? I did not create the GTT. Here's the error output:
[GTT_Variable [2]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code:0x80004005.
An OLE DB record is available. Source:"OraOLEDB" Hresult:0x80004005 Description:"ORA-01403: no data found".[Facets GTT [2]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.
The "GTT_Variable.Inputs[OLE DB Destination Input]" failed because error code 0xC020907B occurred,and the error row disposition on"GTT_Variable.Inputs[OLE DB Destination Input]" specifies failure on error.
An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED.
The ProcessInput method on component "GTT_Variable"(2) failed with error code 0xC0209029while processing input "OLE DB Destination Input"(15).
The identified component returned an error from the ProcessInput method.
The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.
There may be error messages posted before this with more information about the failure.Thanks
↧
Datatype / Data Issue while reading data from Excel
Hi All,
My Source file sample data is as per below
In my ssis package I am reading data as :
select * from [Emp$B1:D4]
Note : For simplicity I have copied data in initial cell but It can be starting in Middle specific range
Now my problem is that in Source data I am getting NA,N/A or some formula error sometimes.
I understand that by default SSIS consider data as Decimal.But, Whenever there are more than one NA or N/A or some other Formula error in multiple row It is converting datatype as DT_WSTR.
As a part of solution, I have included IMEX =1 in my connection.But It is not still resolving my datatype issue.
Problem is that I don't understand my package flow , because sometimes columns are coming as Decimal and sometimes as Nvarhcar. So, how can I come on some common ground ?
↧
Get dataflow programmatically
Hello,
my needs is create a master package who launch package with parametre ( source , destination query , column mapping...)
So I add a script task to update connexion and query in my control flow before data flow.
Can you tell me in this script task in C# how can get dataflow by name ?
↧
↧
How to switch number from 017878 to 787801 in ssis
Hello experts,
I'm new to ssis and I have a requirement to import a xlsx file into a sql table. I can successfully do that but the problem is the company field(int) is backwards in the xlsx file. I need to take the first 2 numbers from the front and put them on the end before importing. I have the field as an int in my import so I imagine I will need to convert it to text, switch the numbers and convert it back to an int before populating the table. I'm guessing I need to use a derived column to do this but not sure.
Any help is appreciated.
↧
CANNOT BE FOUND AT THE DATASOURCE
Hi,
following error occurs after execution of package.
[Excel Source [2]] Error: Column "Duration (Min)" cannot be found at the datasource.
[SSIS.Pipeline] Error: Excel Source failed the pre-execute phase and returned error code 0xC0202005.
from the source duration Column is present why it occurs
↧
Conditionally add {CR}{LF} or Carriage Return after Column11 instead of Column 12 if certain criteria is met
I have a scenario where on some records, I have 11 columns (fields) and in some records, I have 12 columns (fields).
In the Flat File Connection Manager Editor, under Advanced tab, is there a way for me to add {CR}{LF} or Carriage Return when certain condition is met.
Currently the carriage return is set on Column 12 for every record. I would like to know if I can add carriage return on column 11 when FORMID = 999999 for example.
Thank you in advance.
↧
How to pull the root level XML tags in the XML source-SSIS?
Hi All,
Can anyone please help out with the following requirement?
There is a XML file data which I need to pull it to database. Please see the attached XML file.
Question: When I am trying to pull the root level data (i.e. CreationDateTime and FileNumber) from XML source in the dataflow, the respective tag is not seen in the input. Can anyone help me how can I get those values in my input to pull the data to Database?
Thanks & Regards,
Sri
↧
↧
getting my sequence containers to run as efficiently as possible
Hi we will be running 2016 enterprise soon.
I have 32 sequence containers (sc's). Each is a combo of a dft (downloads whatever from server a to server b's integration layer), followed by a proc that versions the downloaded data into what we call the source layer from the integration layer.
So for example, table a from server a is downloaded to a copy of table a on server b in our integration layer db called stage. Then a proc is executed that takes that downloaded data and incrementally updates a full copy of table a in our source db. The integration and source dbs are on the same server. I'm including an image in case it helps.
So with -1 as my MaxConcurrentExecutables, I believe ssis will allow 10 of these to run in parallel on my hardware. About half the sc's are long runners. The other half run relatively quick.
Back in MBA school when we studied PERT in operations research, I thought we were told when all else is equal and you are constrained, get your quick tasks out of the way first. Similarly here, I'd like to get my quick runners going first but wouldn't want that goal to block other things from starting to run as soon as threads open up.
Any thoughts? generaly we stay away from sql agent here.
↧
SSIS 2017 script task not running
Fellow Devs,
Ok, so SSIS is kicking me in the teeth. I'm using all 2017 products, vs, sql server ,ssis etc. I have built a package that literally has a script task set to fail and the whole package completes successfully. I have also added a line for logging. The package
completes without error and I have tried
1. running 32 bit/64 bit
2. changing the sql target version to sql server 2017
3. looking at the sql logs, event viewer, ssis reports
The package runs fine inside vs 2017 and if I change the version to 2014 and deploy it then it also works on sql server 2014. It is specifically the script task that isn't working.
Anyone have any thoughts or ideas? No errors, no logs in ssms or windows.
Thanks,
↧
How to query SSISDB to ID packages using a particular server name in a connection?
Hi All,
Question: How can I query SSISDB for a specific server name used in a connection?
Background: Some of our ETL source databases on a SQL 2008 R2 server will be migrated to new SQL 2016 servers and I need to update any SSIS 2012 packages that are referencing that SQL 2008 R2 server before it's decommissioned.
We never had a linked server set up for the 2008 R2 machine, so I'm not concerned with querying Execute SQL tasks. In the SSIS packages that do reference the 2008 R2 machine, it would be in the form of a Connection object, most likely OLEDB.
Which tables should I be querying? Should I simply search for text "SQL2008", which is the 2008 R2 server name?
Thanks,
Eric B.
↧
How to save and run DTSX packages
Message
Executed as user: Microsoft (R) SQL Server Execute Package Utility Version 12.0.4100.1 for 32-bit Copyright (C) Microsoft Corporation. All rights reserved. Started: 6:30:01 AM Error: 2019-05-02 06:30:03.29
Code: 0xC0208449 Source: IBM Carriers MSCIL100 [37] Description: ADO NET Source has failed to acquire the connection {79C5C8C4-A6D6-492A-98D2-FA5BC759F40A} with the following error message: "ERROR [28000]
[IBM][System i Access ODBC Driver]Communication link failure. Password for user x on system x is not correct, Password length = 0, Prompt Mode = Never, System IP Address = x". End Error Error: 2019-05-02 06:30:03.29
Code: 0xC0047017 Source: IBM Carriers SSIS.Pipeline Description: MSCIL100 failed validation and returned error code 0xC0208449. End Error Error: 2019-05-02 06:30:03.29 Code:
0xC004700C Source: IBM Carriers SSIS.Pipeline Description: One or more component failed validation. End Error Error: 2019-05-02 06:30:03.29 Code: 0xC0024107
Source: IBM Carriers Description: There were errors during task validation. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 6:30:01 AM Finished: 6:30:03 AM Elapsed:
2.25 seconds. The package execution failed. The step failed.
This package was created by a user but a different user opened the package and saved the package in their own credential. What is the remedy to fix this issue and how do we prevent this from happening again.
↧
↧
SSIS Web Service Task Error-"Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebserviceTaskException: The given service is not currently available
Hi All,<o:p></o:p>
I have been trying to connect to a BI publisher report using the web service task. The oracle ERP web service link I am using is as follows,<o:p></o:p>
https://edqy-dev2.bi.em2.oraclecloud.com:443/xmlpserver/services/ExternalReportWSSService?wsdl<o:p></o:p>
The wsdl is connecting fine in SOAP UI and providing me the report output when I am invoking the'runreport' method,<o:p></o:p>
Fig1
Fig2
But, when I am trying the same through SSIS web service task, getting the following issues,<o:p></o:p>
1. Not able to find the parameter for report absolute path in the runreport method,<o:p></o:p>
after downloading the WSDL file from the web URL, when selecting the RunReport method underExternalReportWSSService, it is not providing me any option to pass the part of the report (i.e. reportabsolutepath parameter as in SOAP UI). <o:p></o:p>
Since I am pretty inexperienced in web services, not sure what changes need to be done in order to be able to pass the report path. Can anyone please help. The settings I have used so far is attached below,
<o:p>fig3</o:p>
<o:p>fig4</o:p>
<o:p>fig5</o:p>
2. Getting the following error,<o:p></o:p>
[Web Service Task] Error: An error occurred with the following error message: "Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebserviceTaskException: The given service is not currently available.<o:p></o:p>
at Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebMethodInvokerProxy.InvokeMethod(DTSWebMethodInfo
methodInfo, String serviceName, Object connection)
at Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebServiceTaskUtil.Invoke(DTSWebMethodInfo methodInfo, String serviceName, Object connection, VariableDispenser
taskVariableDispenser)
at Microsoft.SqlServer.Dts.Tasks.WebServiceTask.WebServiceTask.executeThread()".<o:p></o:p>
Without selecting the parameters, if I am just clicking ok after the settings shown in issue 1 and trying to run the package, getting the above mentioned error. <o:p></o:p>
Can anyone please suggest where am I going wrong.<o:p></o:p>
<o:p></o:p>
Surendra Thota
↧
Lost SSIS Catalog Password SQL 2014 How do we reset it
We have a SQL Server 2014 SP3 which has a SSIS DB Catalog but our DBA lost the password for the SSIS Catalog DB. How to do we reset the password
↧
can max concurrency level be changed for different containers in a pkg
Hi we will be on 2016 enterprise soon. can different max concurrency values be employed at different places in the same pkg?
↧