I am trying to load data from flat files into SQL Server I have around 150 columns and when I import flat files in my SSIS package all the columns are by default string with length 50 and I have to manually change all the 150 columns into proper data type so that I can transfer my data in SQL Server else it throws data mismatch error and process stops.

So is there any way where I don't have to manually change column types and it is done automatically by SSIS?

Hi,

I have a weird case in SSIS with an aggregation. I have a column which I aggregate on (I have sorted the data), and the aggregator removes rows that are not equal. Here you probably say - you have a case-sensitive problem...Well I guess so, but I cannot fathom how.

The column that I aggregate on all comprise of the following pattern: <6 numbers>¤<4 numbers> (Ex. 152733¤8440). So, no characters, at all. I have over 7 mio. rows in my source, and of those I 'lose' around 2.300 unique combinations, that should not have been aggregated. 

The super-weird thing is that if I tell the aggregator to be case-sensitive, then it does not remove the rows. So - how can a number + ¤ + number be affected by case? The above example (152733¤8440) is one of the rows that's remove in a non-case sensitive setup. This makes no sense to me - and I have a very hard time coming up with a way to debug which other rows cause the actual 'wrong' aggregate. 

Does anyone have suggestion as to what might be 'hitting' me here? I can, as a workaround, continue with the case-sensitive comparison. I just don't understand what happens - and I don't like the solution, as my database is case insensitive - so that should be the correct comparison. (Note: The source has the column in question as an unique key, so the SQL server does not see the rows as equal). The datatype in SSIS is (DT_STR, 100, 1252) - it's varchar(100) in the source.

Thanks in advance

Overview:-I am running SSIS package to copy data from SQL Server 2016 to Flat file pipe delimited with quoted identifiers and handling escape sequences. I am using Script component in Data flow task as source to write data to flat file as I have to copy data from more than 100 tables with different schema so this is the only dynamic approach worked for me. Also the data is huge in amount most of the tables have 100+ columns and 5 million+ records. I have setup my master package is calling the same package 12 times in parallel for different tables (managed through SQL tables and parameters). On top of it my child package is creating file in batches implemented using For Loop containers and parameters.

Problem statement: - When I am running my package from SSDT it start writing the data to file in immediately once it process the records but when I am running the same package via SQL Server Agent Job it is taking lot longer time and writing the data to file once all records are processed.

Example - Lets table 'a' is having 4 million records and I am generating 4 files each of 1 million, on same parameter SSDT start writing rows in file ~50K-60k (may be depend on buffer size) as soon as they processed but the same package with same configuration when I run from SQL server agent job it process all 1 million records at and try to write all at a time.

Issue:- because it is writing 1 million records at a time, file creation is taking lot of time just to write 1 million records ~5-10 mins varies based on number of columns in table but from SSDT it is much quicker ~2-5 mins for same table.

Can anybody suggest me what settings I have to check to make it work more faster. The table from which it is selecting data is well indexed. Same query for 1 million records when run against database in SSMS took ~2-4 mins.

Thanks

start sql server management studio on server A to connect to SSIS on server B.

Have opened all windows firewall ports.



I have disabled all windows firewall port. And telnet port 135 from server A to server B works.

Regards Kevin Seerup
GoalMaker Software Solutions Pty Ltd
www.goalmaker.com.au

Hi all,

As the title says, we've created a view to convert the data type on the 500 columns and then use SSIS to fastload into the production table.  This table has a columnstore index on it.  The problem is it took WAY TOO long to load (because of the datatype conversion in the view).  Is there a better way to load faster even with datatype conversion?

thanks in advance

This package which is a child package has been running successfully for quite some time now. All of a sudden we are getting these intermittant error messages. Does anyone have any ideas what to do or check for?

thanks

===========================

Error portion

Error: 0xC0047012 at CF-DFT Oracle Sales Fact, DTS.Pipeline: A buffer failed while allocating 100483760 bytes.

Error: 0xC02020C4 at CF-DFT Oracle Sales Fact, order line id [1]: The attempt to add a row to the Data Flow task buffer failed with error code 0x8007000E.

Error: 0xC0047011 at CF-DFT Oracle Sales Fact, DTS.Pipeline: The system reports 30 percent memory load. There are 8587960320 bytes of physical memory with 5972680704 bytes free. There are 2147352576 bytes of virtual memory with 1324290048 bytes free. The paging file has 12673945600 bytes with 10005012480 bytes free.

Error: 0xC0047038 at CF-DFT Oracle Sales Fact, DTS.Pipeline: The PrimeOutput method on component "order line id" (1) returned error code 0xC02020C4. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.

Error: 0xC0047056 at CF-DFT Oracle Sales Fact, DTS.Pipeline: The Data Flow task failed to create a buffer to call PrimeOutput for output "Union All" (13359) on component "Union All Output 1" (13361). This error usually occurs due to an out-of-memory condition.

Error: 0xC0047021 at CF-DFT Oracle Sales Fact, DTS.Pipeline: Thread "SourceThread1" has exited with error code 0xC0047038.

Error: 0xC0047021 at CF-DFT Oracle Sales Fact, DTS.Pipeline: Thread "WorkThread2" has exited with error code 0x8007000E.

Error: 0xC0047039 at CF-DFT Oracle Sales Fact, DTS.Pipeline: Thread "WorkThread3" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown.

Error: 0xC0047039 at CF-DFT Oracle Sales Fact, DTS.Pipeline: Thread "WorkThread1" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown.

Error: 0xC0047021 at CF-DFT Oracle Sales Fact, DTS.Pipeline: Thread "WorkThread3" has exited with error code 0xC0047039.

Error: 0xC0047021 at CF-DFT Oracle Sales Fact, DTS.Pipeline: Thread "WorkThread1" has exited with error code 0xC0047039.

Error: 0xC0047039 at CF-DFT Oracle Sales Fact, DTS.Pipeline: Thread "WorkThread0" received a shutdown signal and is terminating. The user requested a shutdown, or an error in another thread is causing the pipeline to shutdown.

Error: 0xC0047021 at CF-DFT Oracle Sales Fact, DTS.Pipeline: Thread "WorkThread0" has exited with error code 0xC0047039.

====================================================

Complete child package log

I have created a Connection manage to an ODBC source.  put in the credentials and tested the connection is succecfull.

But when i then use this connection in the Power Query source.  I keep getting the error, that "credentials are required to connect to the odbc source.  I have detected the odbc driver, and i have put in the credentials in the connection manager, which works.  Why do i get this error in the power Query source?

I have tried a lot of tools for conversion but found OST to PST Converter one of the best, it is ads free , easy to use, fast in processing, please recommend me any other tool also that can help me in  easy conversion from ost to pst conversion online.

coolutils dotcom slash OSTtoPSTConverter

Good Morning,

In a package i have a lookup to check for new records, if it is new it will insert into the table else will update, when i run the package i am getting the below error on a certain row

Error at Insert New or Changed Records [Destination Table [84]]: Failure inserting into the read-only column "row_timestamp"

I opened the destination database in SSMS and used the following command

SETIDENTITY_INSERT TableName ON

Still the package is failing. Please suggest.

Thanks

Hello,

I have created a SSIS package and loaded .csv files to SQL server tables. On reviewing the data in SQL server tables I found scientific formatted data in my output, so I opened the input .csv file and looked at the format for that columns and I found that for some rows the format was 'Scientific' and for others it was 'General'. 

I wanted to know can we check the format of the input files using a Script Task or any other transformation and if it is needed to changed can we change that.

Please advise.

Thanks,

Ali.

Hi There 

I need to design an ETL flow where I need to filter different start and end dates.

For example : 

Iteration 1

StartDate -- FirstDayofThisMonth

Enddate -- LastDayofThisMonth

Iteration 2 

StartDate -- FirstDayofPreviousMonth

EndDate -- LastDayofPreviousMonth

Iteration 2 

StartDate -- FirstDayofPreviousQuarter

EndDate  -- LastDayofPreviousQuarter

I have build a Time Table which holds all this data. I have an execute sql task and holds the different start and end date from my Time Table and results were stored in 6 different variables.

How can I pass the 3 scenarios one by one on For each loop , Please help 

I am following the sample given in the msdn adventworks and I am on the step 9 of the testing package phase and I can't seem to fathom what the problem might be, I have mapped key to the correct columns and is still giving me error , the error details are displayed here: 

SSIS package "Lesson 1.dtsx" starting.
Information: 0x4004300A at Extract Sample Currency Data, SSIS.Pipeline: Validation phase is beginning.
Information: 0x4004300A at Extract Sample Currency Data, SSIS.Pipeline: Validation phase is beginning.
Information: 0x40043006 at Extract Sample Currency Data, SSIS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Extract Sample Currency Data, SSIS.Pipeline: Pre-Execute phase is beginning.
Information: 0x402090DC at Extract Sample Currency Data, Extract Sample Currency Data [1]: The processing of file "C:\Users\bishal.TRIANGLE\Desktop\sample adventworks\Integration Services\Tutorial\Creating a Simple ETL Package\Sample Data\SampleCurrencyData.txt" has started.
Information: 0x400490F4 at Extract Sample Currency Data, Lookup Currency Key [26]: component "Lookup Currency Key" (26) has cached 14 rows.
Information: 0x400490F5 at Extract Sample Currency Data, Lookup Currency Key [26]: component "Lookup Currency Key" (26) has cached a total of 14 rows.
Information: 0x402090E2 at Extract Sample Currency Data, Lookup Currency Key [26]: The component "Lookup Currency Key" (26) processed 14 rows in the cache. The processing time was 0.016 seconds. The cache used 728 bytes of memory.
Information: 0x402090E4 at Extract Sample Currency Data, Lookup Date Key [51]: The component "Lookup Date Key" (51) succeeded in preparing the cache. The preparation time was 0.001 seconds.
Information: 0x4004300C at Extract Sample Currency Data, SSIS.Pipeline: Execute phase is beginning.
Information: 0x402090DE at Extract Sample Currency Data, Extract Sample Currency Data [1]: The total number of data rows processed for file "C:\Users\bishal.TRIANGLE\Desktop\sample adventworks\Integration Services\Tutorial\Creating a Simple ETL Package\Sample Data\SampleCurrencyData.txt" is 1097.
Error: 0xC020901E at Extract Sample Currency Data, Lookup Date Key [51]: Row yielded no match during lookup.
Error: 0xC0209029 at Extract Sample Currency Data, Lookup Date Key [51]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "component "Lookup Date Key" (51)" failed because error code 0xC020901E occurred, and the error row disposition on "output"Lookup Match Output" (53)" specifies failure on error. An error occurred on the specified object of the specified component.  There may be error messages posted before this with more information about the failure.
Error: 0xC0047022 at Extract Sample Currency Data, SSIS.Pipeline: SSIS Error Code DTS_E_PROCESSINPUTFAILED.  The ProcessInput method on component "Lookup Date Key" (51) failed with error code 0xC0209029 while processing input "Lookup Input" (52). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.  There may be error messages posted before this with more information about the failure.
Information: 0x40043008 at Extract Sample Currency Data, SSIS.Pipeline: Post Execute phase is beginning.
Information: 0x402090DD at Extract Sample Currency Data, Extract Sample Currency Data [1]: The processing of file "C:\Users\bishal.TRIANGLE\Desktop\sample adventworks\Integration Services\Tutorial\Creating a Simple ETL Package\Sample Data\SampleCurrencyData.txt" has ended.
Information: 0x40209314 at Extract Sample Currency Data, Lookup Date Key [51]: The component "Lookup Date Key" (51) has performed the following operations: processed 1 rows, issued 1 database commands to the reference database, and performed 0 lookups using partial cache.
Information: 0x402090DF at Extract Sample Currency Data, Sample OLE DB Destination [76]: The final commit for the data insertion in "component "Sample OLE DB Destination" (76)" has started.
Information: 0x402090E0 at Extract Sample Currency Data, Sample OLE DB Destination [76]: The final commit for the data insertion  in "component "Sample OLE DB Destination" (76)" has ended.
Information: 0x4004300B at Extract Sample Currency Data, SSIS.Pipeline: "component "Sample OLE DB Destination" (76)" wrote 0 rows.
Information: 0x40043009 at Extract Sample Currency Data, SSIS.Pipeline: Cleanup phase is beginning.
Task failed: Extract Sample Currency Data
Warning: 0x80019002 at Lesson 1: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED.  The Execution method succeeded, but the number of errors raised (3) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "Lesson 1.dtsx" finished: Failure.
The program '[7056] Lesson 1.dtsx: DTS' has exited with code 0 (0x0).

Any help would be greatly appreciated.

Hello,

I have to create an SSIS package to write various results to a text file.  The first thing is to list the contents of a stored procedure.  I'm guessing that this would involve calling sp_helptext.  The next thing would be the results of the stored procedure which would be a few columns with headers.  What would be the best way to accomplish this?

<style type="text/css">.fctbNone{ color:#000000; }
.fctbStyle1{ color:#ff1493; }
.fctbStyle2{ color:#0000ff;font-weight:bold; }</style>Replace(Column,Substring([Column], 50, 18), [Id], false)
Replace(source, search, replace, boolean)
Is this accurate? I am not getting desired results. My source text is the column, search text is the substring part of the column, the id is what I want to replace substring text with, and the false

Sonya

Hi we run 2017 std.  I have 11 queries that need to run on 11 non linked servers.  Each query has a high and low date used in a date range predicate.  I'd like to run the queries concurrently (not in a loop) at least to the extent that ssis will allow.

Each query is associated with a 3 character organization code.  I can easily produce a resultset that returns 11 rows with org code , low date and high date.

I'm trying to come up with a way to elegantly map this resultset's data to perhaps 11 variable strings that are queries , maybe 11 sets of low and high dates that could easily (thru expressions) be used to build the 11 queries in variables or perhaps something else.

we use sql agent with the catalog to run our packages.  i thought about building a config file dynamically from the resultset and then mapping its content to the vars but i'm not sure a config file can play with a catalog based pkg.

does the community have any ideas for elegantly building these 11 queries that will likely be run from 11 different oledb sources each with a different connection?  I even thought about returning a resultset where each column is a query and the columns map to 11 different variable strings. I think a multi column resultset can map in ssis to multiple vars.  I  also thought about pivoting my vertical resultset to return 22 vars that would be mapped to the sets of 11 low and high dates.

Hi There

I have an issue with loading data from an excel spreadsheet to SQL database using SSIS package. It seems to be data columns have different formats or hidden characters which are not accepting by the database so SSIS package is erroring out as a result.

As its not practical to check every single spreadsheet manually, is there a way we can add another SSIS package to check data first and change the format if its needed or remove if there are any hidden characters and then save them in a staging table before run the data loading package?

I would greatly appreciate if any of you can help with this. Happy to provide more information if needed.

Many thanks& have a wonderful weekend!

Sapphire