Quantcast
Channel: SQL Server Integration Services forum
Viewing all 24688 articles
Browse latest View live
↧

Accessing 3 million rows of table

October 2, 2014, 10:48 pm
$
0
0

Hi All,

I have a question on accessing a table having around 3 million rows (preferably from Excel 2007) 

This table is a result of ETL extracted data and want now to enable user to use this data. Therefore, I have proposed a solution to them to access it from excel 2007. So i am trying to generate pivot table which is pointing to this table on our production server

Since this table is having approximately 3 million of data rows , the excel pivot table is taking a long time to "read" data rows. 

Can i continue with this solution or can you suggest me some other practical solution so that user can access this data ?

Thanks 

↧
Search

how to maintain error logs in SSIS?

October 3, 2014, 4:46 am
$
0
0

HI,

    RECENTLY I WENT TO ONE OF MY INTERVIEW,SHE ASKED ME FROM A FLAT FILE HOW YOU MAINTAIN ERROR LOGS,I TOLD USING OLEDB DESTINATION AND USING REDIECT ROW OPTION.AGAIN SHE ASKED ME HOW TO KNOW WHERE THE ERROR  AMONG LOT OF RECORDS.I TOLD USING DATA VIEWR.BUT SHE DENIED MY ANSWER AND TOLD USING DATA READER DESTINATION WE CAN DO THAT, SHE TOLD.I AM STILL IN CONFUSION WITH HER ANSWER,CAN ANY BODY CLARIFY WITH ONE EXAMPLE USING DATA READER DESTINATION.



↧
↧

Excel type error while using in SSIS 2005

October 3, 2014, 12:11 am
$
0
0

Hi I am trying to dump the data into excel from OLEDB source.

However, I want to load it into excel 2007 (.XLSX type) there it is giving below error while creating excel connection manager (destination) 

(it is working fine for .XLS type but i want to load into 2007 workbook)

Error desc:

File path contains invalid Excel file. Please provide file with .xls extension.

Any idea?

Thanks



↧

regarding ssis

October 3, 2014, 7:15 am
$
0
0

Hi,

   I have received an Excel Source file, but the first 10 rows have only company information; actual data starts from Row 11.  How will you skip the first 10 rows and start reading from row 11

↧

regarding ssis

October 3, 2014, 7:16 am
$
0
0

Hi,

  You have received a big Excel or flat file, but you only want to load the first 100 records in a table.  How will you  do that

↧
↧

Problem with communication Link Failure in Hyper-V Cluster 2012R2

September 12, 2014, 10:16 am
$
0
0

Hi,

We have a Hyper-V cluster in Windows 2012 R2 with 3 hosts.

We also have outside this cluster a physical Domain contoller and we have 2 virtual DCs in the cluster.

In the cluster we have several VM's and for the specific problem we have 3 VM's as:

1 SQL Olap Engine (2008R2)

1 SQL Engine (2005)

1 SQL engine with SAP (2005)

The data flow is something like, the OLAP engine starts the SSIS to call several Sprocedures that are in the SQL engine, that collects data from SQL and SAP SQL. All the data is stored in the SQL (Staging and warehouse) and the OLAP runs in the warehouse that is stored in SQL.

Last friday, the DC connection to the cluster went down because a switch was powered off. This switch was offline for 1.5h. Since this date we are having a strange issue.

All three SQL machines were in different hosts. 

The SQL in host 1, the OLAP in host 2 and the SAP SQL in host 3.

During the night after the problem, when the jobs (SSIS) to create cubes (collects data to staging and warehouse and then SSAS runs on warehouse) runned the first time it gave an error like:

" Executing the query "EXEC XXXXX;" failed with the following error: "Communication link failure". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly. "

After this, and without any problem doing pings and transfer files between the machines, they were restarted (The hosts were never restarted untill now!).

So, I've put all the machines in the same host and the problem disappeared. After a week, I've moved only the Olap engine to another host, and the problem appeared again.

We don't see any event in the cluster, and we don't see any real network problem.

Also, I can open all SQL servers on any of the servers using the management studio. I've opened the SSIS projects and the connectivity is ok without any problem. It only appears when these SSIS packages run.


Any idea on what can be the issue?

Note: I'm doing some other tests, and I'll share the tests and results during the next days. 

↧

Cannot acquire connection from connection manager - Oracle database

September 24, 2014, 1:48 pm
$
0
0

I created a SSIS package that connects to an Oracle table in SQL Server 2008R2 and it was working.  Now I have upgraded to SQL Server 2012 and the Oracle connection is giving me problem (The SQL Server connections also gave me problems but I have fixed them).  This is on a new laptop running Windows 8.1.  The old laptop was on Windows 7.  I can get the Connection Manager and Data Source for the Oracle table to "Test connection succeeded", but when I run the package I get this error message:

[OLE DB Source - Baan Part Data [162]] Error: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER.  The AcquireConnection method call to the connection manager "BaanTables" failed with error code 0xC0202009.  There may be error messages posted before this with more information on why the AcquireConnection method call failed.  I switched from OraOLEDB to MSDAORA.1 provider.

Any ideas what may be causing my problem?

Thanks,

Fred

Fred Schmid

↧

Import data from twitter to SQL Server 2008 R2 using SSIS

October 3, 2014, 8:33 am
$
0
0

Hi, I am looking for a way to import UserId's from twitter based on time period & keyword.

For instance, All the UserId's who have tweeted or hashtagged SQL Server in the last 1 week.

How could this be accomplished using SSIS?

Thanks in advance................

Ione

↧
Search

Cache Transformation fails with Buffer Memory error during load on SQL 2014 SSDT 2013, potential upgrade issue.

September 23, 2014, 7:58 am
$
0
0

My issue is loading 140 million records into a Cache Transformation fails constantly since upgrading to SSDT 2013 from 2010.

The error is reproducible 100% of the time while running the package in debug mode (dtsDebugHost.exe).

The server is rather large having the following configuration.

Windows Server 2012 Datacenter

Processor Xenon E5-4650 @ 2.70 GHz (4 proc)

768 GB RAM

SQL Server Max Memory 600 / Min 520GB

This issue has also been tested on a different server with similar results.

The source of the Cache is an OLE DB Connection to the local SQL server pulling 13 columns all having ID values.

The destination is a standard Cache Destination.

The errors are as follows and occur while both writing to file and full memory cache near 20.5 million records for both Cache types:

[SSIS.Pipeline] Information: The buffer manager detected that the system was low on virtual memory, but was unable to swap out any buffers. 14 buffers were considered and 14 were locked. Either not enough memory is available to the pipeline because not enough is installed, other processes are using it, or too many buffers are locked.

Information: Buffer manager allocated 4 megabyte(s) in 2 physical buffer(s).

Error: A buffer failed while allocating 1703856 bytes.

Error: The system reports 92 percent memory load. There are 412146331648 bytes of physical memory with 29133062144 bytes free. There are 4294836224 bytes of virtual memory with 8069120 bytes free. The paging file has 429326200832 bytes with 43001991168 bytes free.

[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED.  The ProcessInput method on component "Cache LicenseMaster" (2) failed with error code 0xC0208252 while processing input "Cache Input" (5). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.  There may be error messages posted before this with more information about the failure.

Our initial thoughts are that SQL server is not configured correctly to handle the usage of so many buffers.

I have tried both extremes of Cache property settings for MAXROWS and MAXBUFFERSIZE.

I have read many articles, but none seems to discuss buffer management in detail.

We appreciate your insights and recommendations.

Scott M

↧
↧

Extract data using SSIS 2008 R2 from yelp business api

September 24, 2014, 10:30 am
$
0
0

Hi, I am working on a task to extract data from yelp using yelp business api and load it to a SQL Server 2008 R2 database using SSIS 2008 R2. Could you please give some pointers to accomplish this.

Thanks in advance...........

Ione

↧

PGP Task

October 3, 2014, 10:57 am
$
0
0

Does anyone know what this error mean and how to fix it?  Thank you for any inputs.

[Execute: ] Error: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. 

   at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)

   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)

   at System.Convert.FromBase64String(String s)

   at Org.BouncyCastle.Bcpg.OpenPgp.PgpUtilities.GetDecoderStream(Stream inputStream)

   at SSIS.Extensions.PGPTask.PGPEncryptDecrypt.Decrypt(Stream inputStream, Stream privateKeyStream, String passPhrase, String outputFile)

   at SSIS.Extensions.PGPTask.PGPEncryptDecrypt.Decrypt(String inputfile, String privateKeyFile, String passPhrase, String outputFile)

   at SSIS.Extensions.PGPTask.PGPManager.Decrypt(String inputFile, String privateKeyFile, String passPhrase, String outputFile, Boolean overwriteTarget, Boolean removeSource)

   at SSIS.Extensions.PGPTask.PGPTask.Execute(Connections connections, VariableDispenser variableDispenser, IDTSComponentEvents componentEvents, IDTSLogging log, Object transaction)


newbie

↧

Importing more header and detail text files - SSIS 2005

October 3, 2014, 4:32 am
$
0
0

Hi,

I've created a SSIS 2005 pkg that imports rightly a couple of text files, composed of an header one and a detail one. The file name of the header text file is like "Header_" followed with a date and a time; the file name of the detail text file is like "Detail_" followed with a date and a time. In the pkg I've used the Foreach container.

Now, I need to manage the scenario to import more header and detail couples, but obviously I must import each header with the each corresponding detail.

F.e., before I need to import Header_20141003_0905 and Detail_20141003_0905, then Header_20141003_0950 and Detail_20141003_0950, then Header_20141003_1120 and Detail_20141003_1120 and so on.

Any suggests to me in order to solve this scenario? Thanks

↧

ssis

October 3, 2014, 2:31 pm
$
0
0

I want to combine 3 tables from 3 different source which brings in millions of rows from each table into one for daily use. Is there any way i can bring it into a temp holding place using some task in ssis so that it won't consume more resources and will run faster, when combining it to 1 table.
↧
↧

ssis task for holding millions of data

October 3, 2014, 2:30 pm
$
0
0
I want to combine 3 tables from 3 different source which brings in millions of rows from each table into one for daily use. Is there any way i can bring it into a temp holding place using some task in ssis so that it won't consume more resources and will run faster, when combining it to 1 table.
↧

regarding ssis - skip first 10 rows in excel source

October 3, 2014, 7:15 am
$
0
0

Hi,

   I have received an Excel Source file, but the first 10 rows have only company information; actual data starts from Row 11.  How will you skip the first 10 rows and start reading from row 11


↧
Search

regarding ssis - read first 100 rows of excel file

October 3, 2014, 7:16 am
$
0
0

Hi,

  You have received a big Excel or flat file, but you only want to load the first 100 records in a table.  How will you  do that


↧

ssis - combine 3 tables from 3 different source into one

October 3, 2014, 2:31 pm
$
0
0

I want to combine 3 tables from 3 different source which brings in millions of rows from each table into one for daily use. Is there any way i can bring it into a temp holding place using some task in ssis so that it won't consume more resources and will run faster, when combining it to 1 table.
↧
↧

Is it a bug - SSIS 2012 ReadOnlyVariables in Script Task doesn't work

October 3, 2014, 10:20 pm
$
0
0

It's very weird when I use the ReadOnlyVariables in SSIS 2012 Script Task, it doesn't work at all!!! And I never notice this change before, but everything in 2008 R2 is fine! Is it a bug in SSIS 2012 ?

All the variables I set them to "ReadOnlyVariables"

In scripts - I assigned some values from system variables.

 String PackageName = Dts.Variables["System::PackageName"].Value.ToString();
            DateTime CurrentDate = (DateTime)Dts.Variables["System::StartTime"].Value;

            // User Defined Variables
            Dts.Variables["User::PV_CURRENT_DATE"].Value = CurrentDate;
            Dts.Variables["User::PV_YEAR"].Value = CurrentDate.Year;
            Dts.Variables["User::PV_MONTH"].Value = CurrentDate.Month;
            Dts.Variables["User::PV_DAY"].Value = CurrentDate.Day;
            Dts.Variables["User::PV_PACKAGE_NAME"].Value = PackageName;

Execute the package, it works !

The only thing I can make it as SSIS 2008 R2 does is to change the ReadOnly for each variables.

Then you will get the error.

Why do we need this feature here but not to use the ReadOnlyVariables in script task ?

Please vote if it's helpful and mark it as an answer!



↧

How to delete rows in the SQL Server table based on the contents of Excel file

October 5, 2014, 11:34 am
$
0
0

Hello, everyone,

I have an Excel file which contains data for certain dates. I need to load it in a SQL Server table. But before doing that, I need to check if the table already contains data for the dates in the Excel file. If it does, I need to delete those data first. Not sure what is the best and efficient way to do this. Your help and guidance would be much appreciated.

Thank you in advance.


↧

How do I get SQL Agent to perform an FTP connection through a SSIS package?

November 10, 2010, 12:50 pm
$
0
0

I have created a number of SSIS packages that I am running through SQL Agent in various jobs.  I have set up a Credential / Proxy for the job steps and for the most part everything runs no problem.  The issue I have is with the step where I make a connection to a FTP server and download files.  In BIDS the step works fine but in SQL Agent the step times out.  I presume that it has to do with the account that it is being run under but that is what I thought the Credential / Proxy was supposed to handle (as it does for all the other steps).  I am running on SQL Server 2008 R2.  I have created a test job that has only one step which is to connect to the FTP server and download some text files.  I have tried utilizing package configuration files and have tried various ProtectionLevels to no avail.

Anyone have any thoughts or direction for me to take?  Will continue to research and will post if I resolve this.

Thanks, Andrew

↧
Viewing all 24688 articles
Browse latest View live
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>