Hi All,

I need a help. I am trying to export the data to ADLS by converting the result set into the stream.

When I am trying this with a Console app, No error is thrown, But When I integrate this with the Script task Component It throws an error at runtime. Also After adding Azure classes Breakpoint is not hitting while debugging.

following is Console App Code-

using System;
using System.Threading;
using Microsoft.Azure.Management.DataLake.Store;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.Rest.Azure.Authentication;
using System.IO;
using System.Data.SqlClient;
using System.Data;
using System.Text;

public class Program  {
private static string adlsAccountName;
private static string tblName = "tblGrid";
DataTable dataTable = new DataTable();

public byte[] datared()
{
    string connectionString = "Data Source=local;Initial Catalog=abc;Integrated Security=True";

    string query = "select * from " + tblName;

    SqlConnection conn = new SqlConnection(connectionString);
    SqlCommand cmd = new SqlCommand(query, conn);
    conn.Open();

    // create data adapter
    SqlDataAdapter da = new SqlDataAdapter(cmd);
    // this will query your database and return the result to your datatable
    da.Fill(dataTable);
    conn.Close();
    da.Dispose();

    var a = Encoding.GetEncoding("iso-8859-1").GetBytes(ToByte(dataTable));
    return a;
}


public static string ToByte(DataTable table)
{
    var result = new StringBuilder();
    for (int i = 0; i < table.Columns.Count; i++)
    {
        result.Append(table.Columns[i].ColumnName);
        result.Append(i == table.Columns.Count - 1 ? "\n" : ",");
    }

    foreach (DataRow row in table.Rows)
    {
        for (int i = 0; i < table.Columns.Count; i++)
        {
            result.Append(row[i].ToString());
            result.Append(i == table.Columns.Count - 1 ? "\n" : ",");
        }
    }

    return result.ToString();
}



public static void Main()
{
    Program p = new Program();
    var bytes = p.datared();
    MemoryStream stream = new MemoryStream(bytes);

    // Name of the Azure Data Lake Store
    var  adlsAccountName = "testadls08";

    SynchronizationContext.SetSynchronizationContext(new SynchronizationContext());

    // Portal > Azure AD > Properties > Directory ID (aka Tenant ID)

    var domain = "177cfeea-xxxx-xxx-853d-xxxx";

    // Portal > Azure AD > App Registrations > App > Application ID (aka Client ID)

    var clientId = "0e499fed-7f23-4b19-98ab-xxxxx";

    // Portal > Azure AD > App Registrations > App > Settings > Keys (aka Client Secret)

    var clientSecret = "6xxxxxxxxxxxxxxxxxxxxxxxx=";

    var clientCredential = new ClientCredential(clientId, clientSecret);
    var creds = ApplicationTokenProvider.LoginSilentAsync(domain, clientCredential).Result;

    var filepath = "/shared/" + tblName + ".txt";
    var adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(creds);
    adlsFileSystemClient.FileSystem.Create(adlsAccountName, filepath, overwrite: true);
    adlsFileSystemClient.FileSystem.Append(adlsAccountName, filepath, stream);
 }
}

Hi All

I have a scenario where I store connection string values in a table and call in an execute sql task in to an object variable. I use for each loop ado enumerator to call this object variable and process each load one after another. This was running fine. 

Now I want to run some of the these jobs parallelly in for each loop based on a column value from object variable. Can you please help how I can I achieve, 

Thanks

Hi 

we have an SSIS package which uses HTTP connection manager to download a file from a URL. It was working fine until yesterday but suddenly the package started failing. I am getting this error while trying to connect the URL in SSDT The underlying connection was closed: An unexpected error occurred on a send.   I was able to access the URL from internet explorer/chrome from the same machine I am using SSDT.  SSL certificate for the URL has been installed. I also checked if there are any windows from yesterday but I did not see. Please let me know if I need to check anything and what could be the issue for the failure. 

TIA

I have created a sql task in control flow. Also, I have created one more sql task  in event handler as specified in below url...I'm looking for help. Please let me know how to handle...

http://www.techbrothersit.com/2014/08/ssis-how-to-email-error-messages-from.html

Issue - I have two cases

case1 - When I give "select 1/0" in sql task in control flow  - In this case am getting right.

case2 - When I give wrong SP name in sql task in control flow, am getting below error

"[Execute SQL Task] Error: Executing the query "EXEC msdb.dbo.sp_send_dbmail
 @profile_name = 'Test-..." failed with the following error: "Incorrect syntax near 'dbo'.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly." Here is script that is added in sql task   

"EXEC msdb.dbo.sp_send_dbmail
 @profile_name = '"+ @[User::MailProfile] +"',
@recipients = '"+ @[User::Recipients] +"',"
+"
 @body = "+"'Hi All,"+"
Error Information is below
Error Code:" + (DT_STR,50,1252)@[System::ErrorCode] +"
Task Name:"+ @[System::SourceName] +"
Error Description:" + @[System::ErrorDescription] +
"Thank you
IT Team"
+"',
@subject = '"+  @[System::PackageName] + " failed in  "+ @[User::EnvironemntName]+"'"

I have a problem with running SSIS packages within VS 2017, with a local SQL Server 2016

To demonstrate;  I have a project and package that has the following attributes;

- Project is has property TargetServerVersion "SQL Server 2016"

- Project and Package has property ProtectionLevel "EncryptSensitiveWithPassword"

- Package has a Parameter set to "Sensitive"

Attempting to run this package fails immediately with;

"Exception deserializing the package "The package failed to load due to error 0xC0011008 "Error loading from XML. No further detailed error information can be specified for this problem because no Events object was passed where detailed error information can be stored.". This occurs when CPackage::LoadFromXML fails."

If I change the TargetServerVersion to "SQL Server 2017", or the ProtectionLevel to "EncryptSensitiveWithUserKey", or change the Parameter to not Sensitive, the problem goes away.

What causes this problem, and how can I resolve it?  Thanks.

Hi,

Since upgrading to SQL 2017 (from 2017), the SSIS maintenance job keeps failing.  The job step is the following..

  DECLARE @role int
SET @role =(SELECT [role] FROM [sys].[dm_hadr_availability_replica_states] hars
             INNER JOIN [sys].[availability_databases_cluster] adc
                    ON hars.[group_id] = adc.[group_id]
                    WHERE hars.[is_local] = 1 AND adc.[database_name] ='SSISDB')
IF DB_ID('SSISDB') IS NOT NULL AND(@role IS NULL OR @role = 1)
       EXEC [SSISDB].[internal].[cleanup_server_retention_window]

However it fails with the error message"A cursor with the name 'execution_cursor' does not exist."

After looking through the proc, it de-allocated the cursor then tries to access it again within the while loop.  So we are deleting 1000 records from the SSIS history at a time, but then hit the error.  Whilst I could modify the procedure, I thought i'd see if anyone else encounted this.

SET QUOTED_IDENTIFIER ON
SET ANSI_NULLS ON
GO


CREATE PROCEDURE [internal].[cleanup_server_retention_window]
WITH EXECUTE AS 'AllSchemaOwner'
AS
    SET NOCOUNT ON
    
    DECLARE @enable_clean_operation bit
    DECLARE @retention_window_length int
    DECLARE @server_operation_encryption_level int
    
    DECLARE @caller_name nvarchar(256)
    DECLARE @caller_sid  varbinary(85)
    DECLARE @operation_id bigint
    
    EXECUTE AS CALLER
        SET @caller_name =  SUSER_NAME()
        SET @caller_sid =   SUSER_SID()
    REVERT
         
    
    BEGIN TRY
        SELECT @enable_clean_operation = CONVERT(bit, property_value) 
            FROM [catalog].[catalog_properties]
            WHERE property_name = 'OPERATION_CLEANUP_ENABLED'
        
        IF @enable_clean_operation = 1
        BEGIN
            SELECT @retention_window_length = CONVERT(int,property_value)  
                FROM [catalog].[catalog_properties]
                WHERE property_name = 'RETENTION_WINDOW'
                

            IF @retention_window_length <= 0 
            BEGIN
                RAISERROR(27163    ,16,1,'RETENTION_WINDOW')
            END
            SELECT @server_operation_encryption_level = CONVERT(int,property_value)  
                FROM [catalog].[catalog_properties]
                WHERE property_name = 'SERVER_OPERATION_ENCRYPTION_LEVEL'

            IF @server_operation_encryption_level NOT in (1, 2)       
            BEGIN
                RAISERROR(27163    ,16,1,'SERVER_OPERATION_ENCRYPTION_LEVEL')
            END
            INSERT INTO [internal].[operations] (
                [operation_type],  
                [created_time], 
                [object_type],
                [object_id],
                [object_name],
                [status], 
                [start_time],
                [caller_sid], 
                [caller_name]
                )
            VALUES (
                2,
                SYSDATETIMEOFFSET(),
                NULL,                     
                NULL,                     
                NULL,                     
                1,      
                SYSDATETIMEOFFSET(),
                @caller_sid,            
                @caller_name            
                ) 
            SET @operation_id = SCOPE_IDENTITY() 
            
            DECLARE @temp_date datetimeoffset
            DECLARE @rows_affected bigint
            DECLARE @delete_batch_size int

            
            SET @delete_batch_size = 1000  
            SET @rows_affected = @delete_batch_size
            
            SET @temp_date = DATEADD(day, -@retention_window_length, SYSDATETIMEOFFSET())
            
            CREATE TABLE #deleted_ops (operation_id bigint, operation_type smallint)
            DECLARE execution_cursor CURSOR GLOBAL FOR SELECT operation_id FROM #deleted_ops  WHERE operation_type = 200

			DECLARE @sqlString_operation_messages_scaleout   nvarchar(1024)
            DECLARE @sqlString_event_messages_scaleout       nvarchar(1024)
            DECLARE @sqlString_event_message_context_scaleout        nvarchar(1024)

            IF @server_operation_encryption_level = 1
            BEGIN
                DECLARE @execution_id bigint
                DECLARE @sqlString              nvarchar(1024)
                DECLARE @sqlString_cert         nvarchar(1024)
                DECLARE @key_name               [internal].[adt_name]
                DECLARE @certificate_name       [internal].[adt_name]

            WHILE (@rows_affected = @delete_batch_size)
            BEGIN
                DELETE TOP (@delete_batch_size)
                    FROM [internal].[operations] 
                        OUTPUT DELETED.operation_id, DELETED.operation_type INTO #deleted_ops
                    WHERE ( [end_time] <= @temp_date
                    OR ([end_time] IS NULL AND [status] = 1 AND [created_time] <= @temp_date ))
                SET @rows_affected = @@ROWCOUNT
            OPEN execution_cursor
            FETCH NEXT FROM execution_cursor INTO @execution_id
            WHILE @@FETCH_STATUS = 0
            BEGIN
                SET @key_name = 'MS_Enckey_Exec_'+CONVERT(varchar,@execution_id)
                SET @certificate_name = 'MS_Cert_Exec_'+CONVERT(varchar,@execution_id)
                SET @sqlString_operation_messages_scaleout = 'delete from [internal].[operation_messages_scaleout] where operation_id = '+CONVERT(varchar,@execution_id)
                SET @sqlString_event_messages_scaleout = 'delete from [internal].[event_messages_scaleout] where operation_id = '+CONVERT(varchar,@execution_id)
                SET @sqlString_event_message_context_scaleout  = 'delete from [internal].[event_message_context_scaleout] where operation_id = '+CONVERT(varchar,@execution_id)
                        SET @sqlString = 'DROP SYMMETRIC KEY '+ @key_name
                        SET @sqlString_cert = 'DROP CERTIFICATE '+ @certificate_name
                        BEGIN TRY
                    EXECUTE sp_executesql @sqlString
                            EXECUTE sp_executesql @sqlString_cert
                            EXECUTE sp_executesql @sqlString_operation_messages_scaleout
                            EXECUTE sp_executesql @sqlString_event_messages_scaleout
                            EXECUTE sp_executesql @sqlString_event_message_context_scaleout
                        END TRY

                        BEGIN CATCH
                            
                        END CATCH

                FETCH NEXT FROM execution_cursor INTO @execution_id
            END
            CLOSE execution_cursor
                    TRUNCATE TABLE #deleted_ops
                END
                DROP TABLE #deleted_ops

            DEALLOCATE execution_cursor
            END
            ELSE BEGIN
                WHILE (@rows_affected = @delete_batch_size)
                BEGIN
                    DELETE TOP (@delete_batch_size)
                        FROM [internal].[operations] 
                         OUTPUT DELETED.operation_id, DELETED.operation_type INTO #deleted_ops
                        WHERE ( [end_time] <= @temp_date
                        OR ([end_time] IS NULL AND [status] = 1 AND [created_time] <= @temp_date ))
                    SET @rows_affected = @@ROWCOUNT

                      OPEN execution_cursor
            FETCH NEXT FROM execution_cursor INTO @execution_id
             WHILE @@FETCH_STATUS = 0
            BEGIN
                SET @sqlString_operation_messages_scaleout = 'delete from [internal].[operation_messages_scaleout] where operation_id = '+CONVERT(varchar,@execution_id)
                SET @sqlString_event_messages_scaleout = 'delete from [internal].[event_messages_scaleout] where operation_id = '+CONVERT(varchar,@execution_id)
                SET @sqlString_event_message_context_scaleout  = 'delete from [internal].[event_message_context_scaleout] where operation_id = '+CONVERT(varchar,@execution_id)
                BEGIN TRY
                    EXECUTE sp_executesql @sqlString_operation_messages_scaleout
                    EXECUTE sp_executesql @sqlString_event_messages_scaleout
                    EXECUTE sp_executesql @sqlString_event_message_context_scaleout
                END TRY
                BEGIN CATCH 
                END CATCH
                FETCH NEXT FROM execution_cursor INTO @execution_id
            END
            CLOSE execution_cursor
                    TRUNCATE TABLE #deleted_ops
            DEALLOCATE execution_cursor
               END
                DROP TABLE #deleted_ops
            END
            UPDATE [internal].[operations]
                SET [status] = 7,
                [end_time] = SYSDATETIMEOFFSET()
                WHERE [operation_id] = @operation_id                                  
        END
    END TRY
    BEGIN CATCH
        IF @server_operation_encryption_level = 1
        BEGIN
        IF (CURSOR_STATUS('local', 'execution_cursor') = 1 
            OR CURSOR_STATUS('local', 'execution_cursor') = 0)
        BEGIN
            CLOSE execution_cursor
            DEALLOCATE execution_cursor            
        END
        END
        UPDATE [internal].[operations]
            SET [status] = 4,
            [end_time] = SYSDATETIMEOFFSET()
            WHERE [operation_id] = @operation_id;       
        THROW
    END CATCH
    RETURN 0

GO

Hi, I have a developed a solution where I have the below flow to execute for Parallel Processing.

Launch Package - Pick the Scheduled Job and Triggers Tasks to Parent Package based on certain criteria’s

Parent Package - Has different flows to execute tasks for Remote and Table creation

Child Packages – Where based on the Task assigned to Parent Package and flows identified it triggers the execution

We have Parent to Child execution up to levels and vice versa. The entire flow execute within a for each loop where we have Execute Package Task used to call different child packages within a Child Package and return the updated values through Variables.

The solution works fine for 20- 30 attempts and then gets stuck at one the Child Packages for long time and the error that we receive in the Progress Tab is as below. This does not stuck at one static place its happens randomly at any Execution Package Task.

Error 0x80070008 while executing package from project reference package \"pkg.dtsx\". Not enough storage is available to process this command.   

this is the error message it gets triggered

The system on which I am running this has a RAM of 512 GB and 2 Processors.

Can you please help us identify on how we can resolve this issue and move ahead to make the solution working

Hi, 

Can anyone please help me on how to use ipdispenser using Script Task to kill the Variables once the usage is done. I have a package where we pass variables from Parent to Child and Vice Versa, at the end of the instance I want to dispose the variables used here so that when a second instance starts it works properly without impacting the memory. Can you please guide me on the steps to use this and how to configure it in SSIS flow.

Hi, 

Can anyone please help me on how to use Garbage Collector using Script Task to kill the Variables once the usage is done. I have a package where we pass variables from Parent to Child and Vice Versa, at the end of the instance I want to dispose the variables used here so that when a second instance starts it works properly without impacting the memory. Can you please guide me on the steps to use this and how to configure it in SSIS flow.

Hi All, 

I'm deriving some columns as shown below as inputs: 

Input: 
c1 c2 c3 c4 c5 
1  2  3  4  5
1  3  6  7  8 
1  2  3  9  4 
4  5  6  9  7  
1  1  3  3  2  
4  5  6  5  2 

I want output something like this: 

output: 
c1 c2 c3 c4 
1  2  3  4:5&&9:4
4  5  6  9:7&&5:2   

When c1, c2, c3 are same, combine contents of c4 and c5 with a ":" in between and "&&" separation when there are more rows grouped based on c1, c2 and c3. 

Please le me know if you have any inputs to achieve this in SSIS. 

Thanks,

Vikram  

We have our packages developed in SQL SSIS 2008 R2 BIDS

Now after server upgrade from Windows 2008 R2 to Windows 2016.

will it support execution of batches if we install SQL 2008 R2 standard edition on the server.

I am using DTEXEC utility to run the batches.

OR

Will i have to install SQL 2016 with Data tools to run SSIS 2008 packages and will it support backward compatibility ?

I have an SSIS solution that calls via a VB.NET Script Task a PDF SSRS Report, passes parameters and takes the data from report and stores it as a PDF. Is it possible to call the SSRS report and change the default Datasource? What I want is to use the SSIS Project parameters so if Dev env. is specified, the SSRS report is called and assigned the Dev. Database, if SSIS Project Parameters are set to Production, then SSRS is called with the Prod. Database.

SSIS can already do this for it's own Datasources, this is the 1st time I attempt to do this for SSRS via SSIS.

Adrian Hernandez

I have a SSIS package running a SFTP job, and I can always run the package successfully from SSIS directly.

then I created a SQL Server Agent job for this package with package source of file system, but the job always failed when I run it.

I have some jobs for other packages and they all run well. 

Do you know what causes this issue?

Hello All,

Recently my destination server was upgraded to 2014 and my client machine have 2012 server and visual studio is 2010. 

When i was installing visual studio 2017, selected data storage and processing option but still not able to see SSDT in visual studio?  I am not sure how to fix this issue? do i need to uninstall existing sqlserver data tools?

Thanks

CMK

Hello,

I can gain access to and execute my package using the Application and Package objects. I am handling the events of the Package object and can even see the OnPipelineRowsSent event when it fires to deliver the rows to the Datareader destination. How do I gain access to those rows for use in my application? I have scanned the SSIS object model looking for the right place to hook in, but was not able to find one. Am I heading in the right direction? Is there a sample application out there I have missed?

Similarly, how does one deliver rows in code to the Datareader source? Does this take place prior to calling Execute on the Package, or it is handled when a particular event fires, or perhaps something else?

Thank you,

Brandon

This is more an annoyance than a problem, but I'd be very interested in what solution other people may have to it.

Scenario:  I have a Visual Studio solution with a number of SSIS projects in it.  Each project has ProtectionLevel set to EncryptSensitiveWithPassword.  I can't use EncryptSensitiveWithUserKey, because this means that each team member who works on the project encrypts things with their own key, making it inaccessible to everyone else.  (The sensitive information encrypted is primarily password parameters for connection managers.)

I use the same password on each project within the solution, but that still means that when I open the solution, it prompts me once for the password for each project.  If I have 18 projects, I have to churn through 18 password entries.  Every time I do this I think there must be a better way.

Is there any way to any way to assign the password to the solution, which is then inherited by the projects?  Or is there a better way to encrypt sensitive information within projects?

Thanks.

Hi 

we have an SSIS package which uses HTTP connection manager to download a file from a URL. It was working fine until yesterday but suddenly the package started failing. I am getting this error while trying to connect the URL in SSDT The underlying connection was closed: An unexpected error occurred on a send.   I was able to access the URL from internet explorer/chrome from the same machine I am using SSDT.  SSL certificate for the URL has been installed. I also checked if there are any windows from yesterday but I did not see. Please let me know if I need to check anything and what could be the issue for the failure. 

TIA