Hi,

I know that this question has been asked before.

I want to extract the sharepoint list data using SSIS 

I am currently using the sharepoint source/destination adapter from codeplex

but i am getting this error as the list has multiple lookups underneath 

Error: Microsoft. ... A possible reason might be you are trying to retrieve too many items at a time (Batch size) ---> System.

I dont want to use OData source as it returns the ids , not looked up fields

any chance of a better solution or a C Sharp script to fetch the date via SSIS

We have a SSIS Package that is pumping close to 5 million rows of CMS NPI Provider data into a SQL Server Table. When I ran Max Row Length on this table the value returned was 7881. There's a lot of provider information contained within this National NPI Health Provider file from the government.

When I divide the Max Buffer size of 10485760 by the 7881, I get 1330.51136. So I believe the Default Buffer Max Rows should be 1300.

It is taking our SSIS Package over 2 hours to load almost 5 million records. Are there any other things I need to look at in trying to fine tune this and make this quicker?

Based on real raw numbers, does this seem extraordinary in terms of a data pump wall time for a max row length of 7881 and almost 5 million records to be taking upwards to 2 hors to complete?

Thanks for your review and am hopeful for a reply.

We are running into strange issue even after we have SP1 on SQL Server 2012, This seems to be existing problem with the latest SQL Server version too. 

Job succeeds in the next runs, but it fails now and then with unexpected termination.

Any help greatly appreciated. Thanks in advance.

Regards, Pradyothana DP. Please Mark This As Helpful if it helps to solve your issue. ========================================================== https://social.technet.microsoft.com/Profile/pradyothanadp http://www.dbainhouse.blogspot.in/

Hi.  We see older posts like the one at http://social.msdn.microsoft.com/Forums/en-US/adodotnetdataproviders/thread/c5561541-2193-440c-8d4f-1a731e0a4798 and we know one moderator would like us to join a different community (with 55 followers) that has a related post but we're looking for more current, simpler and less time consuming ways of getting the latest thinking on ssis and postgres.

We still run std 2008.   We use sql server for most of our data stores.  We use Postgres 9.2 for one situation.  We're hoping to find a somewhat seamless Postgres connector (for input and output, maybe a plug in or new tool box item) for use in ssis.   We'd prefer that it be a good performer, unlike the one we tried out in a competitor's java based etl tool.   Actually we're not sure if the etl tool or the connector or both or maybe postgres or bad postgres settings on our part or leaving indexes on during the test was to blame but we know the etl oem is Talend.   Not sure if they also provided the connector.  

We'd also prefer to check it out on more than 100 rows before buying it.  We did experiment with one connector mentioned in the link above but they want $300 before we know if it satisfies our needs.   We want to see how available connectors perform loading 300 million rows before laying out a dime.

We'd prefer staying away from customizing our own connector.   Does anyone have any info that can be posted in this forum?  

I am trying to copy documents from one network drive to another. The challenges I have are:

1. While the documents are being copied to the destination folder, renaming/prefixing the documents at the destination folder with an identity value is becoming very complex for me. There are multiple existing documents at the destination folder and I am trying to avoid copying the docs first and renaming them later.  
2. Each folder names must be parsed and inserted into two SQL Server tables and the folder names are underscore-delimited:My city is Suitland_My state is MD_This is the city I was born_Baltimore is not the city in which I was married_DOB 06-15-1999_I am done here\fileName.pdf. I have the script that parses the underscore-delimited. Howerver, creating a variable to add the idntity value to the copied document name and to insert the same identity to a SQL table is becoming a problem for me.  

I have managed to copy the files but inserting the identity value to sql server and prefixing the identity value to the doc name is becoming a nightmare. Her is an image of what I am trying to do:

Any suggestion or help is really appreciated.

Thank you in advance,

~In-

In my company, they’ve been using an application to manually move documents from one network drive to another network drive one doc at a time. Subsequently, as the docs are moved/copied, the following are inserted into two SQL Server tables:

1. The new file path for each doc
2. The doc name prefixed by auto incrementing value
3. Each folder names must be parsed and inserted into tables, which are underscore delimited (ABC_DE_FG HI JKL_MIN MIN MIN MIN_OP-QR-STU_VWXYZ).

This was done using a stored procedure that has multiple parameters: after a person passes the parameters in the app, the stored procedure will insert the related data into two tables and the application moves/copies the document.

So, this manual task has become tedious and we want to do it using SSIS. My question is how do I go about accomplishing this? I’ve read some articles that talk about copying files using Foreach Loop and/or File System…but how do I make sure that as the docs are moved/copied, the related data are inserted into the intended tables?

Thank you in advance,

~In-

I have created a SSIS package in one server say XYZ which is used to run a .exe file in that server (XYZ). I have my main packages created in another server called ABC once all the packages ran successfully in ABC server i need to run the package in XYZ. I have shared the SSIS package folder in XYZ so that i can access it from ABC server.

  I read that "The executable will always run in the context of the operatingfrom which you call it, not on which you call it. In other words, even though you are calling the copy of cmd.exe located on a remote machine, it is still executing on the machine running the SSIS."

So what is the correct method to achieve this.

Sarvan

I tried the below and want to know is this a correct method.

Created a new job step in SQL server agent after my last job step.

Changed the 'On Success Action' to Go to the next step.

The next step was configured to run the package which is in XYZ server. I choose Package Source as 'File System' and directly choose the .dtsx package from XYZ server.

I am trying to copy documents from one network drive to another. The challenges I have are:

1. While the documents are being copied to the destination folder, renaming/prefixing the documents at the destination folder with an identity value is becoming very complex for me. There are multiple existing documents at the destination folder and I am trying to avoid copying the docs first and renaming them later. The document name in the folder and SQL table has to be identical (1212_fileName.pdf).  
2. Each folder names must be parsed and inserted into two SQL Server tables and the folder names are underscore-delimited:My city is Suitland_My state is MD_This is the city I was born_Baltimore is not the city in which I was married_DOB 06-15-1999_I am done here\fileName.pdf. I have the script that parses the underscore-delimited. However, creating a variable to add the identity value to thedocument that will be copied and to insert the same identity + document name to a SQL Server table is becoming problematic.  

I have managed to copy the files but inserting the identity value to sql server and prefixing the identity value to the doc name is becoming a nightmare :-(. Here is a visual description of what I am trying to do:

Any suggestion or help is really appreciated.

Thank you in advance,

~In-

We have developed several packages with Oracle Attunity connection (Project level connection). While we deploy this package using SQL server agent the job when runs it fails. This is due to the password is NULL for oracle connection. Our package protection level property is SaveSensitiveDataWithUserKey.

So at this moment what is the correct method to deploy the packages to make it run via SQL server agent.

Sarvan

I am executing queries against a large SQL db and getting the data I want. But I want to send the data to a file, which I also have working BUT I can't get an expression to work for naming the file. The data is for the previous month so I want a name like (today is 08/14/2016):

Source201607XX.txt

I have tried at least 20 different expressions like:

"Source" + (DT_STR, 4, 1252)DATEPART("yyyy", GETDATE()) +  RIGHT("0" + (DT_STR, 2, 1252)DATEPART("mm" , DATEADD("mm",-1,GETDATE())) + "XX.txt"

and get

Expression cannot be evaluated.

For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%u00ae+Visual+Studio%u00ae+2008&ProdVer=9.0.30729.1&EvtSrc=Microsoft.DataTransformationServices.Controls.TaskUIFramework.TaskUIFrameworkSR&EvtID=FailToEvaluateExpression&LinkId=20476

Attempt to parse the expression ""Source" + (DT_STR, 4, 1252)  DATEPART("yyyy" , GETDATE()) +  RIGHT("0" + (DT_STR, 2, 1252)DATEPART("mm" , DATEADD("mm",-1,GETDATE())) + "XX.txt"" failed. The expression might contain an invalid token, an incomplete token, or an invalid element. It might not be well-formed, or might be missing part of a required element such as a parenthesis.

 (Microsoft.DataTransformationServices.Controls)

If I do:

"Source" + (DT_STR, 4, 1252)DATEPART("yyyy", GETDATE()) +  RIGHT("0" + (DT_STR, 2, 1252)DATEPART("mm",GETDATE()) + "XX.txt"

It works fine.

Any help would be greatly appreciated.

Thanks,

JoeB

Hi Pals,

Iam trying to implement an  SSIS package restartabilty where a Master Package loops some bunch of child dimensional packages.

We have used For each loop container to execute child packages, now if a child package fails and issue is resolved, when i start the master package, it should run from failed child package and not from first child package. how to do this

If there is any workaround please let me know.

Hi,

In Project Deployment model, does it override all the packages in the SSISDB Catalog regardless if the package is changed or not?

I'm just worried that if you intentional changed Package1 and Package2 was accidentally modified then it will mess-up the package 2. Unlike in SSRS, you have the option to deploy at the Solution, Project and Report level.

Given this, is there a way for us to avoid this issue? what's the best practices for project deployment given that several members could work-on the same solution?

Many thanks.

I am using visual studio 2015 and i have sql server 2014. So i downloaded file watcher sql server 2014 from http://www.sqlis.com/sqlis/post/File-Watcher-Task.aspx.

once i downloaded and installed it was not showing up in my toolbox i tried refreshing it but still not showing up. So, i downloaded 2016 file watcher then it showed up on my VS 2015 and package ran successfully but the problem here is, it was failing when i scheduled a job in SSMS 2014 as it is 2014 and file watcher was 2016. I can upgrade my ssms to 2016 as all my projects are using it.

Please reply to this post if you have any solution.

Thanks in advance.

Below is the error

Error:

I have 2 sorted data sets in my package , DataSet A and DataSet B, I want a merged Output with all rows from DatasetA and only rows from DataSet B thatdoesn’t match with dataset A. How can i do this, any ideas?

I have a text file that I am using 2012 SSIS to load into a table designed specifically for the data in SQL Server 2012.  I have a column identified as a primary key with identity specification turned-on (no columns in the source text file are mapped to this column).  When I am using SQL Server Import and Export Wizard to create the package with "Enabled identity" inserts check box checked in the Column Mapping dialog, I get the below errors and the data does not load.  If I delete the column with the primary key and the identity specification and execute the SSIS package again the data loads properly without errors.

Why I am not able to load the data with a primary key and identity turned on?  Thanks.

[Destination - shoppingExtractPrecise_WithIdentity3 [104]] Error: SSIS Error Code DTS_E_OLEDBERROR.  An LE DB error has occurred. Error code: 0x80004005.

An OLE DB record is available.  Source: "Microsoft SQL Server Native Client 11.0"  Hresult: 0x80004005  Description: "The statement has been terminated.".An OLE DB record is available.  Source: "Microsoft SQL Server Native Client 11.0"  Hresult: 0x80004005  Description: "Cannot insert the value NULL into column 'shoppingExtract_ID', table 'shopping_Pharmacy.dbo.shoppingExtractPrecise_WithIdentity3'; column does not allow nulls. INSERT fails.".[Destination - shoppingExtractPrecise_WithIdentity3 [104]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR.  The "Destination - shoppingExtractPrecise_WithIdentity3.Inputs[Destination Input]" failed because error code 0xC020907B occurred, and the error row disposition on "Destination - shoppingExtractPrecise_WithIdentity3.Inputs[Destination Input]" specifies failure on error. An error occurred on the specified object of the specified component.  There may be error messages posted before this with more information about the failure.[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PROCESSINPUTFAILED.  The ProcessInput method on component "Destination - shoppingExtractPrecise_WithIdentity3" (104) failed with error code 0xC0209029 while processing input "Destination Input" (117). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running.  There may be error messages posted before this with more information about the failure.[Source - SOL 08112016_txt [173]] Error: The attempt to add a row to the Data Flow task buffer failed with error code 0xC0047020.[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED.  The PrimeOutput method on Source - SOL 08112016_txt returned error code 0xC02020C4.  The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing.  There may be error messages posted before this with more information about the failure.

I'm trying to get difference between Created date(SharePoint column) and GetDate in SSIS conditional expression. 

(DT_I4)  DATEDIFF("dd",   [Created],  GETDATE())

Hello Guru's,

One of the SSIS package in our environment is failing which I am currently investigating with below error message.

Message
Executed as user: ADMIN\SvcSQL01TCSASSW01. Source: Analysis Services Processing Task Analysis Services Execute DDL Task     Description: Errors in the OLAP storage engine: The process operation ended because the number of errors encountered during processing reached the defined limit of allowable errors for the operation.  End Error  Error: 2016-08-09 16:09:57.85     Code: 0xC11C0002     Source: Analysis Services Processing Task Analysis Services Execute DDL Task     Description: Server: The operation has been cancelled.  End Error  DTExec: The package execution returned DTSER_FAILURE (1).  Started:  15:23:31  Finished: 16:09:58  Elapsed:  2786.75 seconds.  The package execution failed.  The step failed.

This package is meant to do some refresh to SSAS Cube, which I have no knowledge.

Please advise where to start and how to take this forward.

Best regards,

Mohan

Hi,

I am trying to create a variable as a string in my package, and put the following SQL command into that variable as a expression. I am only facing issues when I need to include the date columns in my query, I am not sure how to cast them, before using them in the expression. I have Effective and Expire date columns in my SQL query.

SELECT
    A.CUST_ID, A.CUST_LEG_NM, C.CNFG_ID, C.CNFG_NM, B.CUST_CNFG_EFF_DT, B.CUST_CNFG_EXPIR_DT, B.CREAT_TS, B.UPDT_TS, B.UPDT_BY
FROM espMB.CUST A
INNER JOIN espMB.CUST_CNFG B ON A.CUST_ID = B.CUST_ID
INNER JOIN espMB.CNFG C ON B.CNFG_ID = C.CNFG_ID
WHERE A.CUST_ID IN (137113)
ORDER BY  B.CUST_CNFG_EXPIR_DT;

As mentioned in the query, there are 4 columns which I have as dates, B.CUST_CNFG_EFF_DT, B.CUST_CNFG_EXPIR_DT, B.CREAT_TS, B.UPDT_TS. If I remove these columns, I am able to use that variable in the source. But, if I include them I am getting issues.

Can anyone help me in putting this SQL query in an expression way.

Rahul

We are doing incremental load using CDC and trying to implement transaction rollback in case of any task failure including ‘CDC control task’ in SSIS package.

To complete 'Get Processing Range' and 'Marked Processed Range', two CDC control tasks are added in the SSIS package. If TransactionOption is 'Supported' (default) for the package, everything works fine. However, if TransactionOption is 'Required' on the other hands, the 'Marked Processed Range' will fail raising '[CDC Control Task] Error:"Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.".

Based on tracing, the following statement is suspending.

(@name nvarchar(16))SELECT count(*) FROM [dbo].[cdc_states] WHERE Upper([name]) = Upper(@name)

We tried multiple options like –

• Replacing any TRUNCATE with DELETE statement.
• Setting ‘RetainSameConnection’ property to ‘True’ for source connection manager.
• Setting ‘UseBulkInsertWhenPossible’ property to ‘False’ in data flow tasks.
• Different Isolation Level.

Let me know if anyone have any clue about this error or can suggest a different approach