Quantcast
Channel: SQL Server Integration Services forum
Viewing all 24688 articles
Browse latest View live
↧

SSIS Package - Excel data source - SQL command - retrieve single cell value then include in Select statement for table on same sheet

October 19, 2018, 9:11 am
$
0
0

Hi all

I have Excel files to process, all in consistent format. Each file has 12 sheets. Each sheet has 'Header' info on rows 1 to 13 then a table of data below (around 160 rows, columns A14:M173)

I'm writing a SSIS package to put all data in the 'tables' from each of the 12 sheets of into a staging table. This I can do using an Excel Data Source >> SQL Command...

SELECT * FROM [Table 1a$A14:M500] WHERE [Region Code] LIKE 'Y%'

UNION

SELECT * FROM [Table 1b$A14:M500] WHERE [Region Code] LIKE 'Y%'   ....etc.

What I need to do is for each Excel table, to retrieve single values from two of the header cells and include them in each row returned in the SELECT statements show above. The cells are 'Period' (C5) and 'Status' (C10) - happy to read as varchars

...so that each row would read in the resulting output as:

Period, Status, [Region Code], [Col2], [Col3] etc.

I've played around with declaring variables but am struggling to incorporate into my SQL statements as the have a Where clause.

I know it'll be something simple but obviously not as simple as me. 

Thanks 

↧
Search

Deploying SSIS Package to Linux Server and Correct Folder Path Syntax- The file name "/home/..." specified in the connection was not valid

October 20, 2018, 6:13 am
$
0
0

Hello

I have set up an SSIS package on Windows to download a CSV file from an FTP run some transforms and download images from a list of URLs which works fine.

I then want to deploy this onto an Ubuntu server running SQL Server 2017 with SSIS installed.

As the SSISDB Catalog is not supported on Linux it seems the only way to run this is as a cron job as per the instructions here:

https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-schedule-ssis-packages?view=sql-server-2017

However as I must develop the package on Windows I cannot enter Linux compatible paths as this raises an error and I cannot save the connection in Visual Studio. As a workaround I open the .dtsx package with VS Code and edit the paths in the XML to /home/...

I then execute the package using:

/opt/ssis/bin/dtexec /Decrypt PASSWORD /F /home/WEBSITE/stocklist/RefreshAutoTraderStocklist.dtsx

But whatever I do this always fails and says the paths are not valid. I have tried running as SU and also chmodding the files and folders being accessed to 777 yet it makes no difference. I cannot find any documentation on this specifically relating to Linux so am not sure how I can get this working.

Errors are:

Error: 2018-10-20 14:00:24.28
   Code: 0xC001401E
   Source: RefreshAutoTraderStocklist Connection manager "FILE.csv"
   Description: The file name "/home/WEBSITE/stocklist/FILE.csv" specified in the connection was not valid.
End Error
Error: 2018-10-20 14:00:24.29
   Code: 0xC001401D
   Source: RefreshAutoTraderStocklist
   Description: Connection "FILE.csv" failed validation.
End Error
Error: 2018-10-20 14:00:24.29
   Code: 0xC001401E
   Source: RefreshAutoTraderStocklist Connection manager "FILE.csv (from FTP)"
   Description: The file name "/home/WEBSITE/stocklist/" specified in the connection was not valid.
End Error
Error: 2018-10-20 14:00:24.29
   Code: 0xC001401D
   Source: RefreshAutoTraderStocklist
   Description: Connection "FILE.csv (from FTP)" failed validation.
End Error
Progress: 2018-10-20 14:00:24.40
   Source: Save Data to Database
   Validating: 0% complete
End Progress
Error: 2018-10-20 14:00:24.40
   Code: 0xC001401E
   Source: RefreshAutoTraderStocklist Connection manager "FILE.csv"
   Description: The file name "/home/WEBSITE/stocklist/FILE.csv" specified in the connection was not valid.
End Error
Error: 2018-10-20 14:00:24.40
   Code: 0xC0202070
   Source: RefreshAutoTraderStocklist Connection manager "FILE.csv"
   Description: The file name property is not valid. The file name is a device or contains invalid characters.
End Error

The files and folders do exist in the locations mentioned but all I can guess is that when the SSIS package is trying to validate itself it is seeing these paths as invalid as they would not be valid on Windows but I am not sure how I can bypass this.

Thanks

Robin

Robin Wilson

↧
↧

Is it viable in production environment to use Change Data Capture(CDC) Service for Oracle by Attunity+SSIS in SQL server 2017

October 24, 2018, 5:34 am
$
0
0

Hi, 

Could you please share any success stories/feedback about CDC implementation from Oracle to SQL server using CDC for Oracle by Attunity, SSIS in SQL server 2017.

Source: Oracle 11g hosted on Redhat Linux

Destination: SQL Server 2017 on Windows server 2016

Production load: 50MB for each 15min, 125,000 records

Please let us know as it will help in our decision making process for BI environment. We have inconsistencies from the source system side.

↧

SSIS in SQL server Express Edition 2017

October 22, 2018, 12:54 am
$
0
0
Is it possible to use SSIS in SQL server 2017 Express Edition?. Does it comes preinstalled while downloading SQL server 2017 express edition?
↧

SSIS Job which downloads files from websites using script task failing after few runs.

October 17, 2018, 3:08 pm
$
0
0
         

I have a weird issue when downloading some zip files from websites.

My solution is working fine when I run from Visual Studio manually.

When I deployed on to the server, it did not run.

Below is the error I am getting.



I RDP'ed on to the Server on which the jobs are scheduled and opened the browser, typed one ( any one out of many job ursl) URL Manually. After this was done, my Jobs( on SQL Server Agent )started running successfully.

But It ran successfully for few runs( I scheduled my job to run every 15 mins) and then stopped again with the same error message.



FYI... I am using a Proxy Account to run my Jobs on SQL Server Agent.

Below is the code I am using in Script Task to download my files.


        public void Main()

        {

            try

            {

                // TODO: Add your code here

                bool fireAgain = true;

                Dts.Events.FireInformation(0, "Download File", "Start downloading " + Dts.Variables["ServerURL"].Value.ToString(), string.Empty, 0, ref fireAgain);



                // Create a webclient to download a file

                WebClient mySSISWebClient = new WebClient();

                mySSISWebClient.Credentials = new System.Net.NetworkCredential(Dts.Variables["ServerUserName"].Value.ToString(), Dts.Variables["ServerPassword"].Value.ToString());

                mySSISWebClient.DownloadFile(Dts.Variables["ServerURL"].Value.ToString(), Dts.Connections["myProductFile"].ConnectionString);



                // Logging end of download

                Dts.Events.FireInformation(0, "Download File", "Finished downloading " + Dts.Connections["myProductFile"].ConnectionString, string.Empty, 0, ref fireAgain);



                // Quit Script Task succesful

                Dts.TaskResult = (int)ScriptResults.Success;

                //Dts.TaskResult = (int)ScriptResults.Success;

            }



            catch (Exception Ex)

            {

                Dts.Events.FireError(0, "ERROR", Ex.Message, null, 0);

                Dts.TaskResult = (int)ScriptResults.Failure;

            }

        }


↧
↧

LOAD FIRST 100 RECORDS

October 24, 2018, 9:46 am
$
0
0

HI ALL,

I HAVE 1000 RECORDS IN SOURCE I WANT TO LOAD FIRST 100 RECORDS IN TO DESTINATION .

↧

SSIS How to Evaluate Project Parameter Passed (Integer YYYYMM) and Choose

October 22, 2018, 6:45 am
$
0
0

How to build logic in SSIS that can choose a date value by comparing the project parameter value and the date obtained from a sql task.

Project parameter "ParmDate" is an Integer yyyymm = 999999

Sql task query

select cast(convert(varchar(6),DATEADD(month, -6, getdate() ),112) as Int) as SqlDate

Choose date

If ParmDate = 999999 

then pass SqlDate

else pass ParmDate

Questions

What task do I use to do the if logic.

How should the Sql Task be configured to pass the output values from the above query.

↧

how to get updated data rows along with new rows.

October 24, 2018, 11:49 am
$
0
0

Hi All,

I have query like below. when I am inserting new row it is showing when I am updating some column in table which is not showing how can I get that updated columns also.

select   e.id ,LTRIM(RTRIM(UPPER(e.id))) as Upper_id
,e.isActive 
,e.Designation 

from employee e  left join department d on e.id = d.employeeid

where not exists
select   id ,LTRIM(RTRIM(UPPER(id))) as Upper_id
,isActive
,Designation 
from employee_Dim ed  where e.id = ed.id

After success full loading of employee_Dim , few columns get updated and few columns get inserted. how can i get the both as new records by using above code.

Thanks

AVS

↧
Search

Failed to start project (Microsoft Visual Studio) Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED)) (mscorlib)

October 24, 2018, 2:03 pm
$
0
0


I'm getting this weird error message when I try to execute a SSIS package. It keeps on throwing catastrophic failure and fails to start the project.



I even tried creating a new project from scratch with a simple package that has select statement only, to see if there was something wrong in my other package. I still get the same error. Is there something wrong with my Visual Studio? It used to run fine before.

TITLE: Microsoft Visual Studio
------------------------------

Failed to start project

------------------------------
ADDITIONAL INFORMATION:

Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED)) (mscorlib)

------------------------------
BUTTONS:

OK
------------------------------


Program Location:

   at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
   at Microsoft.DataWarehouse.VsIntegration.Shell.DataWarehouseOutputWindow.Unsafe_Clear()
   at Microsoft.DataWarehouse.VsIntegration.Shell.DataWarehouseOutputWindow.Clear()
   at Microsoft.DataTransformationServices.Project.DataTransformationsProjectDebugger.LaunchDtsPackage(Int32 launchOptions, ProjectItem startupProjItem, DataTransformationsProjectConfigurationOptions options)
   at Microsoft.DataTransformationServices.Project.DataTransformationsProjectDebugger.LaunchActivePackage(Int32 launchOptions)
   at Microsoft.DataTransformationServices.Project.DataTransformationsProjectDebugger.LaunchDtsPackage(Int32 launchOptions, DataTransformationsProjectConfigurationOptions options)
   at Microsoft.DataTransformationServices.Project.DataTransformationsProjectDebugger.Launch(Int32 launchOptions, DataTransformationsProjectConfigurationOptions options)

↧
↧

Duplicate component name in SSIS package

October 24, 2018, 3:57 pm
$
0
0
We have an SSIS package that is currently running on an SQL 2014 instance.    When I exported the package to my workstation and tried to open it, I get 'Microsoft Visual Studio is unable to load this document: Duplicate component name.     Component names must be unique and case-insensitive'.      Any idea what I can do to open this?
↧

SSIS inserting into Azure SQLDB Always Encrypted encrypted using Key Vault

October 24, 2018, 4:20 pm
$
0
0

I have an Azure SQL Database 'Test' with one column encrypted using Always Encrypted. I have stored the automatically generated column master key in a Azure Key Vault. I have tried configuring an SSIS package to read from an unencrypted flat file and insert/update records in to the Azure SQL Database. When I run the package from my workstation I get the following error:

 

[ADO NET Destination [2]] Error: An argument exception has occurred while data provider tried to insert data to destination. The returned message is : Failed to decrypt a column encryption key. Invalid key store provider name: 'AZURE_KEY_VAULT'. A key store provider name must denote either a system key store provider or a registered custom key store provider. Valid system key store provider names are: 'MSSQL_CERTIFICATE_STORE', 'MSSQL_CNG_STORE', 'MSSQL_CSP_PROVIDER'. Valid (currently registered) custom key store provider names are: . Please verify key store provider information in column master key definitions in the database, and verify all custom key store providers used in your application are registered properly.

 

I have configured the ADO.net connection manager Column Encryption Setting to enabled

 

How do I configure\register the Azure Key Vault provider with the ADO.net connection\driver? How do I Integrate the Azure Key Vault Provider into SSIS so I can access the CMK?

My next step if I can get this to work from SSDT is to deploy to the Azure Data Factory V2 integrated runtime that I have running and configured with SSISDB and SSIS Catalog on same Azure SQL Database as 'Test'.  

↧

ROW SET SCHEMA IN SSIS

October 24, 2018, 8:42 pm
$
0
0
WHAT IS ROW SET SCHEMA IN SSIS?
↧

Unable to Deploy SSIS packages using SQL Server Authentication

October 24, 2018, 12:22 am
$
0
0

I came to know that We are not able to deploy SSIS packages using SQL Server authentication.

It can only happen using Windows Auth IS is True?

I created few SQL logins for my teammates for operations tracking purpose in SQL Server.

but using SQL authentication, not able to deploy.

So can we restrict windows authentication to view SQL Agent jobs?  as below is my requirement




↧
↧

Nested for each loop container in SSIS

October 24, 2018, 9:13 pm
$
0
0

Hi Team,

Any one explain about nested for each loop container with example. 

Thanks Bala Narasimha

↧

VS2017+SSDT - TargetServerVersion SQL Server 2012

October 24, 2018, 11:53 pm
$
0
0

Hello,

I have an installation of VS2017 (15.8.8) and SSDT (15.8.1).

I want to be able to deploy the SSIS project to the SSIS catalog to our 'old' SQL Server 2012 and I also I want to start developing for our new SQL Server 2017 infrastructure.

My problem is, that I can't select 'SQL Server 2012' as TargetServerVersion. I only have the option to select 2014, 2016, 2017 and 2019.

Shouldn't it be possible to develop SSIS packages for SQL Server 2012 with Visual Studio 2017?

↧
Search

How do I pick the most recently file in folder and load into sql data base using Foreach loop container in SSIS package

October 25, 2018, 1:02 am
$
0
0

Hi,

I have tried a source as flatfile but i need source as a excel file.

I have created to Variable like FolderName and FileName

Also i took one script task and write bellow script

            // TODO: Add your code here 
            var directory = new
                DirectoryInfo(Dts.Variables["User::FolderPath"].Value.ToString());
            FileInfo[] files = directory.GetFiles();
            DateTime lastModifiedfield = DateTime.MinValue;

            foreach(FileInfo file in files)
            {
                if(file.LastWriteTime>lastModifiedfield)
                {
                    lastModifiedfield = file.LastWriteTime;
                    Dts.Variables["User::FileName"].Value = file.ToString();
                }
            }
            MessageBox.Show(Dts.Variables["User::FileName"].Value.ToString()); 

Am strucked while taking sources as excel.

Could you please help on this issue.

Thanks Bala Narasimha

↧

how can i find my old post on one size fits all etl approach to building a star

October 25, 2018, 5:52 am
$
0
0
Hi. I think my id was once db042188 also and had to be changed for some technical reason. But way back I wrote a post that described a one size fits all etl approach for building a star and I cannot find the post. Maybe 6 yrs ago, not sure. Is there an archive somewhere? I cannot locate it thru bing. it doesn't come up when I start typing a question in this forum with the words "one size fits all". I 'd love to read it before I embark on a new project i'm starting.
↧
↧

Using While loop need to satisfy case codition

October 25, 2018, 11:03 am
$
0
0

I have a sample data

Dept Table : 

Name
Maths
English
Social

Student Table : 

Name    Subject
MOhan   English
MOhan   Maths
Rohan   English
Sajan   Social

Using while loop I have auto incremented the Dept table and using that I will count the subjects for students and mark it as 'Y' or 'N'.

Output : 

Name  Maths  English  Social
Mohan  Y     Y        N
Rohan  N     Y        N
Sajan  N     N        Y

Script :

Declare@RN INT,@cnt INT,@sub varchar(20),@dept_sub VARCHAR(20),@Maths_sub BIT,@Eng_sub BIT,@soc_sub BITSET@RN = RANK()OVER(ORDERBY Name )from DeptSET@cnt = count(*)from  DeptWHILE@RN <@cnt BEGINSelect@Sub = Name from dept where RN =@RNInsertinto#Temp(Name,Maths,English,Social)Select Name,@Maths_sub =CASEWHEN count(*)>=1THEN'Y'ELSE'N'END Maths,@Eng_sub =CASEWHEN count(*)>=1THEN'Y'ELSE'N'END English,@soc_sub =CASEWHEN count(*)>=1THEN'Y'ELSE'N'END Social from student where Subject =@Sub@RN =@RN +1END

I'm not getting proper output suggest me. We can do this in PIVOT and Joins also but using loop I want to work it out

↧

The file name property is not valid. The file name is a device or contains invalid characters.

July 19, 2012, 6:01 am
$
0
0

Hi Gurus,

I have a foreachloop package and the files folder is a n/w drive. When I ran the pkd on my laptop using BIDS it is running fine but when I skeduled to run it showing me errror:

The file name property is not valid. The file name is a device or contains invalid characters.  End Error  DTExec: The package execution returned DTSER_FAILURE (1).

The n/w drive has acess to agent defualt user(dba). and while imporing the package to IS services i used the protection level "Rely on server storage and roles for access control".

I am banging my head all week but cudnt resolve this issue.

Please help me with this.

↧

Anyone running Teradata Drivers/TTU 15.1 connected to Teradata DB 16.2, running on Intellicloud?

October 25, 2018, 12:24 pm
$
0
0

Hello,

The title says it all. We are migrating our Teradata platform to Intellicloud and upgrading the database to version 16.2. Anyone still running Teradata 15.1 drivers and utils and connected to Teradata in Intellicloud that is DB version 16.2? And not having issues?

On our Windows Server 2012 R2, and SQL Server 2016 SP1 EE, we had Teradata 15.1 Drivers and TTU, Attunity 4, and VS15 w/ SST14, then upgraded to Teradata 16.2, Attunity 5, and VS 17 w/ SSDT15, and uncovered a host of issues and basically all our jobs had to go down till we figured out the correct combo of tools and versions.

We discovered that VS17 w/ SSDT, the SSIS side is not compatible with Attunity 5 when your SQL Server is version 2016. Along with other issues.

I had to leave Attunity 5 in, and add Attunity 4, before things would work, plus add VS15 back, to edit, create a Connection Manager that works, and can deploy without error and run without error. Crazy thing is with just Attunity 5, once 4 was added, it would work but sill only showed Attunity 5 being the version of MSTERA used, odd. Yet it can’t open an existing SSAS model without error. I can open it in VS17 SSAS though.

I’ve spent a lot of time and money with Microsoft Premiere Support, and we have it limping in 32 bit mode now as that’s what can successfully deploy and run right now, we’re working on the 64 bit side, and will address the SSAS issues once the whole driver/connector and likeable version of VS/SSDT can play nice with Teradata 16.2 drivers and utilities.

Would love to hear war stories from other who have been through this to compare notes.

Thanks

↧
Viewing all 24688 articles
Browse latest View live
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>