Quantcast
Channel: SQL Server Integration Services forum
Viewing all 24688 articles
Browse latest View live
↧

packages design for big table

August 18, 2016, 4:58 am
$
0
0

Hi,

I am a very critical stage of package designing and need help in taking very critical and important decisions.

There are multiple table for weather information i.e., one table for each location in the source system and collectively they contains approx 1740000+ records

I need to develop a package which gets data from more than 20 tables to a  single table in DWH.

I need help in package design as old records might changes, so i need to perform an update operation (Max 6 months old data might changes)

I am looking at following options, Please suggest

1) I can use lookup or merge task to find new record or existing records.

Problem with this approach is it is very slow and after finding existing records if i want to perform update on the DWH table using Oledb command transformation it will very very slow.

2) Alternate approach for update in the first approach can be that, i will store existing record in an another table and then perform bulk update. Problem with this approach is there are many big table so i would need many staging table for performing bulk update.

Second things,

I need to design error handling in the following ways, need technical help on this.

In case an error occurs

1) Error records need to be redirected to error tables

2) An audit table contains packages running information like package start date, end date, status and other information, need to update error information in this table

Need your input on the design and implementation of this two things.

↧
Search

SSIS Data Profiling Task ProfileInputXML Property issue

August 18, 2016, 4:19 pm
$
0
0

Currently using the ProfileInputXML property to load XML via the Expressions pane. This is all part of a For-Each Loop solution I have.

All ok so far with the results UNTIL I alter the XML to go from a single determinant column in the Functional Dependency and add an extra column to make it multi-determinant.

The profiler only returns a single determinant result in the XML even though it has captured that there is multi-determinant columns as part of the request:

I've run a Quick Profile set to multi-determinants and it returns multi columns in the XML results.

Hoping there is a simple fix to this.

Thanks.

P.

↧
↧

Day Comparison Between TY and LY

August 19, 2016, 1:01 am
$
0
0

Hi everyone,

My department supervisor gave me a job to do which is about to produce automatic daily, weekly and monthly reports. A datamart is ready for this process. I have to compare the trend of last year and this year. For the monthly report there is no necessity of day comparison.

However in daily and weekly reports I need that. For example: 1 Aug 2016 corresponds to Monday and according to last year day comparison, it can be compared by 3 Aug 2015 Monday.

Is there any ease or method of this in SSIS?

↧

How can I save the path and the file for a plain file destination?

August 19, 2016, 3:22 am
$
0
0

Hi,

I'd need to save the full path and file name because after processing the output, this output must be transferred by using FTP Task

Thanks in advance,

↧

Microsoft connector V3

August 17, 2016, 5:10 am
$
0
0

Hi,

We are using Microsoft connector V3 on server but I am getting error as Oracle registry cannot be opened.

Server details:

Windows 12

64 bit machine

SSIS installed

Following are the questions:-

1 .Why the error is generated as oracle registry cannot be opened.

2. If suppose multiple oracle home is installed on server then which oracle home should be set as environment variable.

3.Is some licence required for using Microsoft connector V3 for oracle.

Please reply ASAP .

Regards

Ritika

↧
↧

Prevent SSIS Number Rounding

August 18, 2016, 12:15 am
$
0
0

Hi ,

   I am facing a strange issue using SSIS and need some help in this regard. Le me first set the context. I have a simple package that is used to load an excel file content into a SQL Table. There is a script component used to  achieve this since the data from the excel should be unpivoted and loaded into the table in SQL Server. We are using Provider=Microsoft.ACE.OLEDB.12.0;Data Source=filePath;Extended Properties="Excel 8.0;HDR=Yes;IMEX=1"  as a driver to connect to the excel and load. The excel file as some string columns and few numeric columns. We are facing an issue while loading the numeric columns. We are reading the numeric columns also as DT_STR and mapping this to a target column which is a Decimal type. The intention is to get the number as it is given in the file without rounding off or truncating. There is a number0.055464 which is getting rounded off to 0.0555 which is not acceptable. What is more surprising is I have copied the same project to another environment(UAT) and execute it there the number is getting loaded as 0.055464 as expected. The UAT environment also uses the same drivers and the connection strings as mentioned above. Any help to troubleshoot this will be much appreciated.

Thanks,

Pavan

↧

After installing SSMS 2014 I can no longer add a source or destination in SSIS VS2010

June 13, 2014, 11:46 am
$
0
0

I really don't know what is happening but I have been using VS2010 and SSMS 2014 for a couple of months now with no issue. Yesterday when I opened an SSIS package that I had created in VS2010 I was no longer able to add a source or destination to any of the data flows. I can copy an existing one and change the properties but when I try to drag it from the tool bow I get the following error -

TITLE: Microsoft Visual Studio
------------------------------

The component could not be added to the Data Flow task.
Could not initialize the component. There is a potential problem in the ProvideComponentProperties method.

------------------------------
ADDITIONAL INFORMATION:

Error at "WORKFLOW NAME" [SSIS.Pipeline]: Component ", clsid {GUID}" could not be created and returned error code 0x80070005 "Access is denied.". Make sure that the component is registered correctly.

Error at "WORKFLOW NAME" [ [128]]: The component is missing, not registered, not upgradeable, or missing required interfaces. The contact information for this component is "".

------------------------------
Exception from HRESULT: 0xC0048021 (Microsoft.SqlServer.DTSPipelineWrap)
------------------------------
Program Location:

   at Microsoft.SqlServer.Dts.Pipeline.Wrapper.CManagedComponentWrapperClass.ProvideComponentProperties()
   at Microsoft.DataTransformationServices.Design.PipelineTaskDesigner.AddNewComponent(String clsid, Boolean throwOnError, Boolean select)

NOTE:I replaced the workflow name and the GUID because I don't think it is important for the post.

When I search for the GUID in regedit it leads me to an entry that has the folder DTSInfo and the value of

Microsoft.DataTransformationServices.DataFlowUI.OleDbSourceUI, Microsoft.DataTransformationServices.DataFlowUI, Version=12.0.0.0, Culture=neutral, PublicKeyToken=********** 'This was not the original value, just hiding it for the post.

for UITypeName

No one here could figure out what the exact issue and how to fix it so my pc was rebuilt. Everything was working just fine until I installed SSMS 2014. Now I am right back where I started.

I really need to know what is going on because I wasted an entire day on trying to figure out this issue and I have already tried reinstalling/repairing VS2010. Removing SSMS 2014. Registering OleDbSrc.dll but get an error when trying to do that.

↧

Error when loading data from SAP R3 to SQL BI

August 19, 2016, 6:07 am
$
0
0

Hi everyone,

we have extracted a very big Table BSEG from R3 to SQL BI. However there is an Error when loading:

ERPConnect.ERPException:CPIC-CALL: MRCV: rc= ...

LOCATION CPIC (TCP/IP) on local host

ERROR ilegal parameter value (function=SAP_CMTIMEOUT2/parameter=conversation_ID/value=.....)

TIME:....

RELEASE ....

COMPONENT CPIC

VERSION...

MOUDLE

LINE

COUNTER

-RFClisten failed: RFC_FAILURE

at ERPConnect.RFCAPI.ReceiveFunctionResult(Ulnt32 connectionhandle,RFC_PARAMETER importing,RFC_PARAMETER changing,RFC_TABLE,encoding apiendcoding)

at ERPconnect.RFCFunction.REceiveFunctionArguments(RFC_Table&apiTables)

atERPconnect.RFCFunction.CallclassicAPI

atERPconnect.RFCFunction.ExecuteRFC Byte tid

at XtractKernel.Extractors.TableExtractor.Getpackage(RFCFunction&func)

at XtractKernel.Extractors.TableExtractor.Extract

at XtractKernel.Extractors.ExtractorBase1.Extract(ProcessResultCallback processResult)

after checking SQL BI system, there is no Error with the System, so we think there is something wrong with the RFC Connection between ERP and SQL BI. i have checked the connection but i couldn't find any Problem with it.

Can you please help me how to solve this Error?

Thankyouvery much!



↧
Search

SSIS/SSDT: The underlying connection was closed

August 19, 2016, 7:11 am
$
0
0
I am trying to use Web Service Task in SSIS/SSDT to use our company's Web Service. However I am getting this error: "The error is: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel..". I have installed the certificate to Trusted Root Certificates Authorities but I am still getting that error. I speculate that the issue stems from a mismatch name. Where the domain name does not match the name on the certificate I installed. Is there a solution for this or can I have SSIS/SSDT to ignore this certificate error? 
↧
↧

CDC Control Task: "Timeout expired" (when TransactionOption is Required)

August 15, 2016, 3:01 pm
$
0
0

We are doing incremental load using CDC and trying to implement transaction rollback in case of any task failure including ‘CDC control task’ in SSIS package.

To complete 'Get Processing Range' and 'Marked Processed Range', two CDC control tasks are added in the SSIS package. If TransactionOption is 'Supported' (default) for the package, everything works fine. However, if TransactionOption is 'Required' on the other hands, the 'Marked Processed Range' will fail raising '[CDC Control Task] Error:"Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.".

Based on tracing, the following statement is suspending.

(@name nvarchar(16))SELECT count(*) FROM [dbo].[cdc_states] WHERE Upper([name]) = Upper(@name)

We tried multiple options like –

  • Replacing any TRUNCATE with DELETE statement.
  • Setting ‘RetainSameConnection’ property to ‘True’ for source connection manager.
  • Setting ‘UseBulkInsertWhenPossible’ property to ‘False’ in data flow tasks.
  • Different Isolation Level.

Let me know if anyone have any clue about this error or can suggest a different approach

↧

is BIML like XAML?

August 19, 2016, 8:47 am
$
0
0

Hi, I mean, i remember moving XAML files around from my WPF project and everything was there. BIML can do the same but for a .dtsx package? I don't know if I am very clear.

EDIT. Be able to create from the scratch a .DTSX package by using only a text editor and BIML

↧

Loading the data from Google Sheet into SQL Server by using SSIS Package.

August 19, 2016, 6:44 am
$
0
0

Hi Everyone,

           I am facing some problem while  loading the Google sheet data into SQL Server by using SSIS. I have downloaded and installedGoogle Data API SDK  in my local machine. Script task throwing following error while executing the SSIS package.

execution of request failed: http://spreadsheets.google.com/feeds/spreadsheets/private/full

 Or is there any way to download the Google sheet data into CSV format in a specified folder/path automatically ?

Can one help me out on this.


↧

Making a POST request in C# with Basic Authentication and receiving the Response as Text

August 19, 2016, 6:41 am
$
0
0

I have written the below code to login a web page and receive a Token as a response  in text format. I am unable to recieve the response from my below code. Can some one please guide me where i have gone wrong.

string pageSource;
            string formUrl = "https://mat.medicaleguides.com/mobile_login"; // NOTE: This is the URL the form POSTs to, not the URL of the form (you can find this in the "action" attribute of the HTML's form tag
            string formParams = string.Format("Username={0}&password={1}", "my Login name", "my password");
            string cookieHeader;
            WebRequest req = WebRequest.Create(formUrl);
            req.ContentType = "application/x-www-form-urlencoded";
            req.Method = "POST";
            byte[] bytes = Encoding.ASCII.GetBytes(formParams);
            req.ContentLength = bytes.Length;
            try
            {
                MessageBox.Show("Inside Try");
                using (Stream os = req.GetRequestStream())
                {
                    MessageBox.Show("Inside GetRequestStream");
                    os.Write(bytes, 0, bytes.Length);
                }
                WebResponse resp = req.GetResponse();
                cookieHeader = resp.Headers["Set-cookie"];
                using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
                {
                    pageSource = sr.ReadToEnd();
                    MessageBox.Show(pageSource + "--");
                }
            }
            catch (WebException e)
            {
                using (WebResponse response = e.Response)
                {
                    MessageBox.Show("Inside Catch");
                    HttpWebResponse httpResponse = (HttpWebResponse)response;
                    Console.WriteLine("Error code: {0}", httpResponse.StatusCode);
                    using (Stream data = response.GetResponseStream())
                    using (var reader = new StreamReader(data))
                    {
                        string text = reader.ReadToEnd();
                        MessageBox.Show(text);
                        Console.WriteLine(text);
                    }
                }
            }


↧
↧

SQL Server 2016 Data tools does not appear in Visual Studio 2015

August 19, 2016, 11:31 am
$
0
0

Running Windows 7 enterprise 64 bit

Have SQL Server 2012 developer edition on the same computer

I installed visual studio community and checked the box for SQL server data tools. The installation appears to run normally, but I can't find any of the data tools (analysis services design, Integrations Services design..etc) under visual studio on 2015. I tried uninstalling and then re-installing , but nothing changes.

This should be simple but apparently is not.

Kenneth C Wilson

↧

Dynamically create OLEDB table in Data flow(SSIS)

August 21, 2013, 4:26 am
$
0
0

As my requirement is  import the data from CSV to OLEDB(daily in new table dynamically)

So prepared tasks

1.SQL Task --to create dynamic table by  Property expressions

2.Data flow in For each loop - Collection :For each file Enumerator to select file fully qualified  ==*cx*.csv

before the created Variable :::Tablename (or Tb is same nothing to worry) (String in Package level) and assigned  Evalute expression :True(Below image)

in SQL Task-->

 General Paze:: Result Set :NONE

                       Connection :OLEDB

                       SQL source Type: Direct Input

 Expression paze:

                        Property:SQL source type:  Expression (Check below image)

Now It is showing error String to int 32  ?

Is it correct process do Dynamic Table creation ??

My Ist TASK to Successfully Dynamically create the table in SSIS

So Please Help me

--THANKS
--MADHU

                    

↧
Search

A basic concept question about SSIS 2012 Environments

August 20, 2016, 8:47 am
$
0
0

Hi Everybody,

I'm finally getting around to learning about SSIS 2012 package Environments and I can already see the value in them, but I'm wondering if Environments could be used to facilitate only a single SSISDB catalog in my office.  In theory (though maybe not in best practice), could I...

  • Have only a single SSISDB instance in my office
  • Deploy my package(s) to the SSISDB
  • Create an Environment for DEV execution
  • Create an Environment for QA execution
  • Create an Environment for PROD execution

I don't think I'd ever actually do something like this, but it seems entirely plausible from what I'm reading about Environments.

Thanks,
Eric

↧

BULK INSERT error with severity level 16 were not caught by TRY / CATCH blocks

August 18, 2016, 9:06 am
$
0
0

Hi, 

I am facing an issue with bulk insert from a CSV to database.  With try, catch block I cannot see any errors but when I execute without try catch block,  it gives following errors.

Msg 4864, Level 16, State 1, Line 1
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 2, column 10 (InActiveDate).
Msg 7399, Level 16, State 1, Line 1
The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 1
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".

when I set MAXERRORS = 0, (referring to Bug id 592960) with in try catch block, it catches only the last error which is 

Msg 7330, Level 16, State 2, Line 1
Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".

How do I capture all the 3 errors?

Thanks,

Bhuvana

↧
↧

Loading Dimensions and Facts

August 20, 2016, 9:18 am
$
0
0

Hi all,

1. Is it better to load the fact table from the dimension tables, so populate the dimensions with all records from DW and load fact with same records from those dimensions using joins.

2.Or is it better to have the dimensions only contain the main attributes (not actual records) in a list format, so say 10 rows for customer dimension and 20 rows for products in product dimension, then load the fact table from the DW and not the dimension(s) then connect back to the dimensions using business keys.

From what I can see both ways will work although point 1 would mean adding all the records to the dimensions first then select across records that are columns in the fact table which would be overkill??

And point 2 would be in effect just lists of unique customers, products...with a business key in each to point back to the fact.

Thanks

↧

Excel File Issue in SSIS -SQL Server 2014

August 20, 2016, 10:19 am
$
0
0

We have a Master Package which runs in Default mode (64 bit). We also have child packages for different file formats such flat files, tab delimited and fixed width etc and there is also a package for loading excel files. We are executing Master Package from Command Line, a config file is used at the time of execution, the config file contains server name and database name. We are able to execute all packages successfully by executing the master package,  however we are having issue with the package that loads Excel Files.  We get this error

Below is the error we get

Connection manager "CON_EXCEL"     Description: The requested OLE DB provider Microsoft.Jet.OLEDB.4.0 is not registered. If the 64-bit driver is not installed, run the package in 32-bit mode. Error code: 0x00000000.  An OLE DB record is available.  Source: "Microsoft OLE DB Service Components"  Hresult: 0x80040154  Description: "Class not registered".  

To resolve the issue we tried to run the master package in 32 bit mode however we are getting the following error, we are not able to invoke any child packages when try to tun the Master Package in 32 bit mode

 SSIS Error Code DTS_E_OLEDBERROR.  An OLE DB error has occurred. Error code: 0x80040E21.  An OLE DB record is available.  Source: "Microsoft OLE DB Service Components"  Hresult: 0x80040E21  Description: "Multiple-step OLE DB operation generated errors. Check each OLE DB status value, if available. No work was done.".

Can you please share your ideas to resolve the Excel file load issue.  Any help would be greatly appreciated


SQLEnthusiast

↧

Has data source got column?

August 20, 2016, 9:05 am
$
0
0

I need a way of detecting if a data source (mdb file) contains a column or not. I have a data source being downloaded every 5 minutes and sometimes it will contain a column called FlareT and sometimes it will not. 

If the source contains the column FlareT I execute one task else I execute a different task.

I think this can only be done using a script component?

↧
Viewing all 24688 articles
Browse latest View live
Search