Can I do a manual checkpoint via a sql task from within an SSIS package, assuming the account executing the package has permissions?
↧
SSIS package - do a manual checkpoint with sql task?
↧
order of operation amongst command line variable override, config and variable value
hi we run 2014 enterprise. we use a loadtype user variable in our pkgs to distinguish incremental vs historic loads.
there is an initial value in the pkg itself. and an opportunity for a config override. and now i am looking at another option of setting the override value on the command line used by the sql agent to execute the pkg.
which trumps which? especially between the config and command line values.
right now we run from msdb.
↧
↧
PostgreSQL ODBC driver and SSIS SSIS.Pipeline: "ODBC Source" failed validation and returned validation status "VS_NEEDSNEWMETADATA".
I have been trying to figure this out. I created a simple package to test thing before I start parameterizing the query I need to run to pull data from AWS PostgreSQL. My DBA set up a user and I can query the schema.table in database using TOAD with ODBC connection created but when I try to use the same ODBC connection in the SSIS package I get this error.
SSIS.Pipeline: "ODBC Source" failed validation and returned validation status "VS_NEEDSNEWMETADATA".
To make it worth if I don't disable "delay validation"( set to false) my SSDT will crash everytime I open a project as my connection managers are setup as project manager.
Have anybody had this problem?
Thanks
↧
SSIS ODBC Source get char string truncated in middle when converting to unicode DT_WSTR
SSIS
In a simplified test, the SSIS package reads from an ODBC Source of MySQL, and writes to aFlat File Destination of text file.
The source contains columns of char(xx) and varchar(xx), and these columns get truncated at half of the length when reach destination. I noticed in Advanced Editor that these columns are DT_WSTR in both sections of External Columns and Output Columns, and the length equals to the source.
As a workaround, I changed ODBC Source to ADO NET Source, and the problem disappears.
I need some pointers to find out what was wrong when using ODBC Source.
It is on Windows with driver mysql-connector-odbc-8.0.12-winx64.msi, and please let me know if need more details. Thank you for your help.
↧
SQL Server 2016 splits rows in flat file destination
An SSIS job running with an OLE DB connection to a SQL Server 2008 produces a flat file destination output like this:
905574|012470,097566,298605,334988,426155,481002,...
910219|029929,149841,326495,461291
933447|067516
943449|053096,092330,249222,300504,329626,386116,...
944688|017507,071064,289347,319367,356738,474931,...
954567|033549,083460,294846,313324,446125,476221,...
961179|001630,043896,184759,326832,411758,445063,...
971664|048997,098448,
974104|085329,136985,204911,300600,356153
997138|032756,166485,214313,412516,481333,495417,...
...
The same SSIS job running with an OLE DB connection to a SQL Server 2016 produces a flat file destination output like this:
905574|012470,097566,298605,334988,426155,481002,...
910219|029929,149841,326495,461291
933447|067516
943449|053096,092330,249222,300504
943449|329626,386116,...
944688|017507,071064,289347,319367,356738,474931,...
954567|033549,083460,294846,313324,446125,476221,...
961179|001630,043896,184759,326832,411758,445063,...
971664|048997,098448,
974104|085329,136985,204911,300600,356153
997138|032756,166485,214313
997138|412516,481333,495417,...
...
The "..." above just means there is more data in the real files. The longest rows can have 3,000 or more six digit entries after the |. The shortest rows can have one six digit entry after the |.
Running with a connection to SQL Server 2016 produces output where every few lines a row is split into multiple rows. For example, the rows beginning with943449 and 997138 in the SQL Server 2008 example has everything after the | split into two rows in the SQL Server 2016 example.
The source data is exactly the same on the two databases. This is part of a conversion project from SQL Server 2008 to SQL Server 2016.
The output columns for the destination file have the same settings. What else can I check?
Thanks.
↧
↧
SSIS returns an error - what should I do?
Hi everybody,
I split a huge file into multiple pieces of 1mln rows each. Now I'm trying to run my package from VS (by pressing F5) and I'm getting this error
Warning: 0x80049304 at Data Flow Task, SSIS.Pipeline: Warning: Could not open global shared memory to communicate with performance DLL; data flow performance counters are not available. To resolve, run this package as an administrator, or on the system's
console.
Information: 0x40043006 at Data Flow Task, SSIS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Data Flow Task, SSIS.Pipeline: Pre-Execute phase is beginning.
Information: 0x402090DC at Data Flow Task, Flat File Source [11]: The processing of file "file here" has started.
Information: 0x4004300C at Data Flow Task, SSIS.Pipeline: Execute phase is beginning.
Error: 0xC0202091 at Data Flow Task, Flat File Source [11]: An error occurred while skipping data rows.
Error: 0xC0047038 at Data Flow Task, SSIS.Pipeline: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on Flat File Source returned error code 0xC0202091. The component returned a failure code when the pipeline engine called PrimeOutput().
The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
Information: 0x40043008 at Data Flow Task, SSIS.Pipeline: Post Execute phase is beginning.
Information: 0x402090DD at Data Flow Task, Flat File Source [11]:
The processing of file "file here" has ended.
------------------
I don't have Admin rights on this server (running through RDP) and I'm not sure how should I run this package in console. What does it mean exactly?
For every expert, there is an equal and opposite expert. - Becker's Law
My blog
My TechNet articles
↧
SSIS 2017 latest version keeps crashing
Hi,
I'm running VS 2017 Community and it keeps crashing. I'm running it on the server and I'm getting very frequent crashes when I run my packages.
Is it a common experience for everyone here? What am I supposed to do?
Thanks in advance.
One more thing - Uri, you told me you're uploading files to the Azure portal and run the process using the portal. Can you please explain everything what needs to be done step by step, if possible? Right now I'm running from the file server trying to load 1 mln rows at a time (original file was 36 mln rows, so I split it into pieces) into Azure DB by pressing F5 in VS. Apparently VS is running out of memory really quickly.
For every expert, there is an equal and opposite expert. - Becker's Law
My blog
My TechNet articles
↧
Connecting to the Integration Services service on the computer "" failed with the following error: "Class not registered".
Hi
SQL Server : 2016 SP2 EE
SSMS Version: v 18.4
when I try to connect SSIS using SSMS I get the following error. Can you please help me to fix this issue?.
Error
Failed to retrieve data for this request. (Microsoft.SqlServer.Management.Sdk.Sfc)
For help, click: https://go.microsoft.com/fwlink?ProdName=Microsoft%20SQL%20Server&LinkId=20476
------------------------------
Connecting to the Integration Services service on the computer "****" failed with the following error: "Class not registered".
This error can occur when you try to connect to a SQL Server 2005 Integration Services service from the current version of the SQL Server tools. Instead, add folders to the service configuration file to let the local Integration Services service manage packages on the SQL Server 2005 instance.
For help, click: http://go.microsoft.com/fwlink/?LinkId=506689
------------------------------
Connecting to the Integration Services service on the computer "*****" failed with the following error: "Class not registered".
This error can occur when you try to connect to a SQL Server 2005 Integration Services service from the current version of the SQL Server tools. Instead, add folders to the service configuration file to let the local Integration Services service manage packages on the SQL Server 2005 instance.
------------------------------
BUTTONS:
OK
***** - instance name removed
↧
Custom Log Message for Execute SQL task
I have setup an SSIS package that executes multiple Execute SQL task . I am aware of logging options that SSIS provides.
But what I want to do is store the output of each "Execute SQL task" in log table. Something like this - "Query has executed successfully with n number of rows. "
How do I achieve this?
↧
↧
[File System Task] Error: An error occurred with the following error message: "Access to the path is denied.".
I have two different packages that load data from flat files into SQL Server tables. Once its done there is a file system task that renames the file to a different location. I deployed the package and scheduled the job using SSMS. Yesterday the job did not fail but neither the files were copied to the Archive folder. When i tried to run the package manually i am getting the following error:
[File System Task] Error: An error occurred with the following error message: "Access to the path is denied."
Another thing is both the packages rename the files to different locations and only one package is not renaming the files to the Archive location. I have same level of security on both the folders for both the packages.
Please see the attached settings of the file system task
Thanks
.
↧
Deploying SSIS projects to SQL Azure using different db names other than SSISDB?
Hello,
Our SQL Azure environment currently has 4 servers (dev, qa, test, prod). The database name in dev is named SSISDB, but has different names for the other servers b/c I believe you cannot have same database (resource) names in Azure (unless this is not true?). I am able to deploy my SSIS project to dev, but when I try to deploy to my other environments I get this error:
How do I deploy to my other environments w/ different DB names? There is nowhere in the deployment wizard that lets you change the db name. Is this doable in SQL Azure?
Thank you
↧
Does sqlserveragent account needs full control on a fileshare to modify a excel ?
I have a SSIS package which formats data using c# code in ssis script task .I can run it using sqlserveragent account which hasfull control permissions on the file share folder in DEV but in QA when I try to run I get error innovation error at c#\script task.
The sqlserveragent account has modify permissions set on the file share folder in QA
The flow of ssis package is as follows.
Prep data and dump it in a sql table and then use the table to write data in excel over a oledb connection . Once the data write is done format using c# /script task .
SSIS 2014 and SQL server 2014.
Excel format- .xls
↧
How to reduce Lookup Transformation performance cost?
Hi,
I have to insert thousand records of data each day into the table. I use Lookup Transformation and use "Redirect rows to no match output" to insert data into the table. The table itself contains (at least) hundred thousands of data. I am using the "Use results of an SQL query" and Full Cache mode in the lookup. My question is, is there any way to reduce the performance cost beside using customized query? I am new to SSIS and haven't explore it much, so I am afraid that I miss some configuration.
Thank you in advance
↧
↧
Any general guidelines to allocate table space quota to different layers in ETL?
Hello Experts,
I am looking for any general guidelines to allocate table space quota to different layers/schemas in ETL flow of a data warehouse (% of total space in each layer).
I have these 4 layers:
- Staging -truncate and load data from source files
- ODS- Type 1 persistent tables
- Transformation layer- similar to final DWH layer but truncate before loading newly arrived data
- DWH layer- Final dimensional model layer
I understand space requirement may vary based on project requirements, however still any general guideline (if any such in data warehousing and ETL space) to estimate the space will be helpful.
Thanks,
Rajneesh
↧
SQL Server Maintenance plan job failed
I have created a maintenance plan to backup database, recently after a server reboot, the maintenance plan started to failed. I have recreated back the job but it's still failing with same error below. I know I can use other way to backup database but I want to know why it's failing with maintenance plan.
Message Executed as user: MTRADEALWA\svc_efoX. Microsoft (R) SQL Server Execute Package Utility Version 11.0.2100.60 for 64-bit Copyright (C) Microsoft Corporation. All rights reserved. Started: 5:09:30 PM Error: 2020-05-05 17:09:45.76 Code: 0xC00291EC Source: {EDE87A0E-6D01-4C6C-BE30-10DCFB40ED52} Execute SQL Task Description: Failed to acquire connection "Local server connection". Connection may not be configured correctly or you may not have the right permissions on this connection. End Error Error: 2020-05-05 17:10:00.31 Code: 0xC0024104 Source: Back Up Database (Full) Description: The Execute method on the task returned error code 0x80131904 (A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)). The Execute method must succeed, and indicate the result using an "out" parameter. End Error Error: 2020-05-05 17:10:14.85 Code: 0xC0024104 Source: {CA2864D8-321C-4C00-9E9A-3DE24396F7C2} Description: The Execute method on the task returned error code 0x80131904 (A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)). The Execute method must succeed, and indicate the result using an "out" parameter. End Error DTExec: The package execution returned DTSER_FAILURE (1). Started: 5:09:30 PM Finished: 5:10:14 PM Elapsed: 44.062 seconds. The package execution failed. The step failed.
↧
Is it possible to identify a flat file connection?
I have a number of flat file connection managers in a package (as well as those connection to databases) and I would like to create a script task which will verify if all of the target files exist.
I know I can do this by parameterizing the connections and passing each of these variables into the script task, but this relies on new variables being added to the task as new connection managers are created. I also know that I can step through the Dts.Connections collection but can't see a property in there which differentiates a "flat file connection manager" from an "OLEDB connection manager", for example.
Is it possible to do this identification programmatically? Once I can find the flat file connection managers I can pass the ConnectionString into the "File.Exists" command.
Thanks.
↧
SSIS 2019, MSOLEDBSQL, Thread safe? Connection issue
Hi,
I have a "ETL framework" built in SSIS. It has been running fine for several year, but when trying to run it on SQL Server 2019 I´m getting into trouble.
Earlier I was using an OLE DB connection manager with the "SQL Server Native Client" (SQLNCLI11.1) provider.
Googling around I found that this one is deprecated and shouldn´t be used together with SQL Server 2019 so I changed to the new "Microsoft OLE DB Driver for SQL Server" (MSOLEDBSQL.1).
First I though that this solved my problem, but I still get random errors so let me explain the problem in depth.
I´ve spent a week now trying to figure out the cause.
I´ve create a small SSIS project from scratch just to be able to reproduce the problem.
The project consist of 2 packages:
"Sub Package" - Contains one "Execute SQL Task" (just runs simple SQL statement "SELECT 1") using an OLE DB Connection (Project Level Connection Manager).
This OLE DB Connection uses the OLE DB Provider "Microsoft OLE DB Driver for SQL Server" (MSOLEDBSQL.1).
The ConnectionString of the OLE DB Connection is set through an Expression. In this test package the Expression is "hardcoded" as:
"Data Source=my_server_name;Initial Catalog=FrameWorkETL;Provider=MSOLEDBSQL.1;Integrated Security=SSPI;Auto Translate=False;"
This OLE DB Connection uses the OLE DB Provider "Microsoft OLE DB Driver for SQL Server" (MSOLEDBSQL.1).
The ConnectionString of the OLE DB Connection is set through an Expression. In this test package the Expression is "hardcoded" as:
"Data Source=my_server_name;Initial Catalog=FrameWorkETL;Provider=MSOLEDBSQL.1;Integrated Security=SSPI;Auto Translate=False;"
"Main Package" - Contain 2 For Loop containers running in parallell. Each loop runs for 1000 laps and each lap it runs an Execute Package Task with "Sub Package".
I´m running the "Main Package" under SQL Server Agent.
Sometimes the package runs fine, but many times I randomly get different error messages.
Example error 1:
Package:Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft OLE DB Driver for SQL Server"
Hresult: 0x80004005 Description: "Invalid connection string attribute".
Example error 2:
Package:Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E4D.
An OLE DB record is available. Source: "Microsoft OLE DB Provider for SQL Server"
Hresult: 0x80040E4D Description: "Invalid authorization specification".
Example Error 3:
Package:Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E73.
An OLE DB record is available. Source: "Microsoft OLE DB Service Components"
Hresult: 0x80040E73 Description: "Format of the initialization string does not conform to the OLE DB specification.".
It is olso common that the package fails with "Unexpected Termination". Then it looks like this in the Event Viewer:
Faulting application name: ISServerExec.exe, version: 15.0.2000.5, time stamp: 0x5d8a8258
Faulting module name: ntdll.dll, version: 10.0.17763.292, time stamp: 0x7ded7809
Exception code: 0xc0000374
Fault offset: 0x00000000000faf49
Faulting process id: 0x1a18
Faulting application start time: 0x01d62865c4ae326b
Faulting application path: C:\Program Files\Microsoft SQL Server\150\DTS\Binn\ISServerExec.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 5348cd22-fdea-46aa-a908-5848856a5e53
Faulting package full name:
Faulting package-relative application ID:
Faulting module name: ntdll.dll, version: 10.0.17763.292, time stamp: 0x7ded7809
Exception code: 0xc0000374
Fault offset: 0x00000000000faf49
Faulting process id: 0x1a18
Faulting application start time: 0x01d62865c4ae326b
Faulting application path: C:\Program Files\Microsoft SQL Server\150\DTS\Binn\ISServerExec.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 5348cd22-fdea-46aa-a908-5848856a5e53
Faulting package full name:
Faulting package-relative application ID:
Looking in the SSIS internal log table [internal].[event_message_context] I can see that the Connectionstring sometimes get corrupt.
It gets corrupt in different ways which might explain why I get different error messages.
Example of corrupt connectionstring from log table:
Data Source=my_server_name;Initial Catalog=FrameWorkETL;Provider=MSOLEDBSQL.1; (missing the last part of the string).
Data Source=my_server_name;Initial Catalog=FrameWorkETL;Provider= (also missing some part of the string).
Data Source=my_server_name;Initial Catalog=FrameWorkETL;Provider=MSOLEDBSQL.1;Integrated Security=SSPI;Auto Translate=False;MSOLEDBSQL.1;Integrated Security=SSPI;Auto Translate=False; (extra parts added last in the string)
I´ve only been able to reproduce the problem under the following circumstances:
- The Connection Manager uses the OLE DB Provider "Microsoft OLE DB Driver for SQL Server" (MSOLEDBSQL.1) - Might be the same problem with SQLNCLI11.1 provider. The package runs fine if I use for example an ADO.Net Connection Manager with SqlClient Data Provider.
- The Connectionstring of the Connection Manager is set through an Expression. - The package runs fine if the connectionstring is not set through an Expression.
- There are "tasks" running in parallel trying to initialize a database connection. - The package runs fine if the "Main Package" only has one for loop. = No connections are initialized in parallell.
- The package runs on a SQL Server 2019 machine. - The Package runs fine on for example a SQL Server 2017 (even if the connection manager connects to a SQL Server 2019 database).
My environment:
VM with Windows Server 2019 standard
SQL Server 2019 (15.0.4033.1) CU4 (latest CU).
MSOLEDB Provider version 18.3 (latest)
Any help appreciated.
Regards
Stefan
Stefan
↧
↧
AWS Aurora PostgreSQL and ODBC drivers for SSIS issues
Hi all
I have been using ODBC drivers in SSIS for a long time but it seems can't figure out why official PostgreSQL odbc drivers causing me an issue. I created an odbc connection and can connect to Aurora PostgreSQL just fine. I can also see the list of all tables when I use ODBC source task in the data flow but when I try to run the package I get this error
Error: 0xC004706B at DTF Load SSIS.Pipeline: "ODBC Source" failed validation and returned validation status "VS_NEEDSNEWMETADATA".
Error: 0xC004700C at DTF Load SSIS.Pipeline: One or more component failed validation.
Error: 0xC0024107 at DTF Load There were errors during task validation.
I also tried RedShift driver as Redshift is based on PostgreSQL and can run the package without any issues but can't get the list of tables and get Reshift specific errors.
Anybody have any suggestions?
↧
SSIS package - do a manual checkpoint with sql task?
Can I do a manual checkpoint via a sql task from within an SSIS package, assuming the account executing the package has permissions?
↧
How to replace all comma in an EDI file
Hello Folks,
I am trying to parse an EDI file using SSIS and finally convert it to a csv file. I am able to do the parsing and get the csv file. But unfortunately for few elements within that file, there is a comma in the address field. The comma can also be in any field. So when i open the generated csv file, this extra comma messes up the column structure. I can open that file in notepad and use Control H, but would like to do this automatically as a pre-task before the parsing processes start.
Is there a way to parse the input file and do a Control H with replacing a comma with a space. I'm using 2017 version.
Thanks in advance.
↧