Hello,
Can anybody please let me know which is the best option to implement incremental loading (Type 2), History should be kept.
I know there are many approaches
I am bit confused which is the best way to implement. Your suggestions are appreciated and it would be great if you have any link which explains the same.
Thanks!
Shamsuddeen
I'm attempting to write a SSIS package that loops through a list of databases on different servers and returns a list of users for each database, including their roles. The following code runs successfully when executed in SSMS against databases on SQL 2005, 2008/R2, and 2012.
/******************************************************
Declare variables to work with
******************************************************/
DECLARE @U1 as table
([principal_id] int
,[RowNo] int
)
DECLARE @R2 as table
([principal_id] int
,[db_Role] nvarchar(128)
,[RowNo] int
)
DECLARE @T3 as table
([ServerName] nvarchar(128)
,[db_Name] nvarchar(128)
,[User_Name] nvarchar(128)
,[User_Type] nvarchar(60)
,[db_Roles] nvarchar(128)
)
DECLARE @roles as nvarchar(128)
DECLARE @principal_id as int
DECLARE @I as int
DECLARE @MaxI as int
DECLARE @J as int
DECLARE @MaxJ as int
/******************************************************
Insert users into table variable
******************************************************/
INSERT INTO @U1
SELECT principal_id,
ROW_NUMBER() OVER(ORDER BY principal_id) AS RowNo
FROM sys.database_principals AS U
WHERE U.[type] IN ('S','U')
AND U.[name] NOT IN ('dbo','guest','INFORMATION_SCHEMA','sys')
/******************************************************
Insert roles into table variable
******************************************************/
INSERT INTO @R2
SELECT M.member_principal_id AS principal_id,
R.[name] AS [db_Role],
ROW_NUMBER() OVER(PARTITION BY M.member_principal_id ORDER BY R.[name]) AS RowNo
FROM sys.database_role_members AS M
INNER JOIN sys.database_principals AS R ON M.role_principal_id = R.principal_id
/******************************************************
Need to start looping through each role
******************************************************/
SET @I = 1
SELECT @MaxI = MAX([RowNo]) FROM @U1
WHILE @I <= @MaxI
BEGIN
/******************************************************
Get all of the roles and put them into a variable
******************************************************/
SET @J = 1
SELECT @principal_id = [principal_id] FROM @U1 WHERE RowNo = @I
SELECT @MaxJ = MAX([RowNo]) FROM @R2 WHERE [principal_id] = @principal_id
SELECT @roles = ''
WHILE @J <= @MaxJ
BEGIN
SELECT @roles = @roles + [db_Role] + '; '
FROM @R2 AS R
WHERE principal_id = @principal_id
AND RowNo = @J
SET @J = @J+1
PRINT @Roles
END
/******************************************************
Put the results into the user table
******************************************************/
INSERT INTO @T3
SELECT CONVERT(nvarchar(128),@@SERVERNAME) AS [ServerName],
CONVERT(nvarchar(128),DB_NAME()) AS [db_Name],
CONVERT(nvarchar(128),P.[name]) AS [User_Name],
CONVERT(nvarchar(60),P.[type_desc]) AS [User_Type],
CONVERT(nvarchar(128),@roles) AS [db_Roles]
FROM @U1 AS U
INNER JOIN sys.database_principals AS P ON U.principal_id = P.principal_id
WHERE U.principal_id = @principal_id
SET @I = @I+1
END
SELECT * FROM @T3 ORDER BY [User_Name]It all starts falling down when I try and put this into a data flow task as a source. The connection manager for the OLE DB Source is dynamically changed and this code called for each new connection by placing the data flow task within a loop. The connection string for the connection manager is an expression defined as follows:
"Data Source="+ @[User::ServerName] +";Initial Catalog="+ @[User::DBName] +";Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;"
As I said, the code runs fine when executed in SSMS, but when run through SSIS only works for SQL 2012 servers. If I try to return information from a SQL 2005 (or other) server I get the following message and the packages refuses to run:
"no column information was returned by the sql command"
After exploring this issue I've attempted a few solutions (including prefixing my code with SET FMTONLY OFF and ensuring that the final result always returns a value), but to no avail.
I'm at the point that the only solution I can think of is to create this as a stored procedure / table-values function on every database (or at least every server), but this strikes me as impractical.
Suggestions?
I've searched for possible solutions, however none had worked for me (mostly queries for SQL Server). My problem is this:
I have a MAIN package that runs about 100 little packages in 1 main process. A lot of times on the first load the packages are disabled are kept enabled. I found some missing data that a package had inserted into SQL Server. It could be a few different packages that did this.
What I need is to see the time and date when the packages I have pinpointed as problematic have ran. They could be enabled or disabled.
Is there a simple way to look somewhere in SSIS to see the last date and time these packages have ran so I can compare to the recorded creation date in one of my tables?
Thanks.
Hi all,
I have a weird issue where my foreach loop container gives an error when I double click the container.
The package works perfect so I'm wondering how I can solve this small annoying error.
Exception from HRESULT: 0xC0017011
at Microsoft.SqlServer.Dts.Runtime.Variable.get_Value()
at
Microsoft.SqlServer.Dts.Runtime.Enumerators.FromVar.ForEachFromVarEnumeratorUI.Initialize(ForEachEnumeratorHost FEEHost, IServicesProvider servicveProvider, connection connections, Variables variables)
at Microsoft.DataTransformationServices.Design.ForEachLoopCollectionView.HostEnumeratorUI()
Any ideas? I do not get an error or warning in SSIS output toolbar
Hi,
i have loaded the csv file to table using ssis, it is working file but record is mismatch.
source data :
col1 col2 col3 col4
aa, bb, this is sample,data for the details, this is
column delimiter - comma
col3 - record contains comma. so col3 record after the comma is added with the as col4
i need the col3 record with comma.
so what is the column delimiter will use ?
Regards,
abdul Khadir
Hi,
I am currently having this error message when trying to execute a SSIS package from SSDT-BI for visual studio 2012 : Method
not found: 'Boolean Microsoft.SqlServer.Dts.Design.VisualStudio2012Utils.IsVisualStudio2012ProInstalled()'. (Microsoft.DataTransformationServices.VsIntegration)
TITLE: Microsoft Visual Studio
------------------------------
Method not found: 'Boolean Microsoft.SqlServer.Dts.Design.VisualStudio2012Utils.IsVisualStudio2012ProInstalled()'.
------------------------------
BUTTONS:
OK
------------------------------
The "All Executions" report is good but from an operations management perspective a more useful report would be to simply show "Failed Executions" and extend the table to include the name of the environment that the execution ran with.
Ideally I would like to keep the links to the existing Overview, Message & Performance reports so, I would be looking to modify the existing RDL file.
Does anybody know where I can get the source from?
i am facing an issue with formatting phone numbers.
currently about 80% of phone numbers in my table that look like 123/456 7890
I want to get them all to look like (123) 456-7890 without affecting the remaining 20% that are in various other formats.
I tried several different combinations of transformations but to no avail...
I want to know if there is a way to convert these to the desired format...
All help and guidance appreciated...
Dhananjay Rele
Problem: Multiple excel files. No way to know the format of data (rows and columns data categories vary in each file).
Question: How do i create a database that will change according to the excel file being uploaded.
I have a text file with two header rows:
Hdr0
HdrA, HdrB, HdrC
Data 1.a, Data 1.b, Data 1.c
Data 2.a, Data 2.b, Data 2.c
I wrote a script component to capture this header and data rows into the same row:
Hdr0, HdrA, HdrB, HdrC, Data 1.a, Data 1.b, Data 1.c
Hdr0, HdrA, HdrB, HdrC, Data 2.a, Data 2.b, Data 2.c
When the text file has no data, only the header, the SSIS Data Flow element fails "The input file did not process or has no records".
It is acceptable to have zero records.
How do i keep the data flow task from failing if no rows are processed?
I'm using an XML Source adapater on top of a 10GB file. I've configured it using an XSD. All of this works well as I am able to select the several output possibilities.
When running the package no error appears, but the odd things is, although having tried this over several outputs, there is no data available when the package finishes. It looks like the XML is inert.
Did somebody experience this before? the outputs selected certainly contain data.
i have 1 sequence container.
in that i have data flow task,in which oledb source ->rowcount(I am counting total rows into vaiable)(@rowcount)
now i have for loop container ,in which i am using this variable, but its coming 0 here.
@Counter=0
@Counter<@rowcount
@Counter = @Counter+1
i can see from DATA flow task that 100 rows is going into row count.
ANY HELP?
Hi All,
Is possible to add the a package in the existing project in SSISDB in SQL SERVER 2012?
Many thanks in advance.
Hi
I have saved the config file path in a cmd file as a variable. This cmd file is used by some other application to run the SSIS package. what all changes i have to make at the package level so that it reads the config file from the cmd file.
Hello,
we have a custom FTP script task created which basically download it to two folders Archive and Working. I'm not from a .net background so couldn't know what the below error means, can anyone help me on this?
Error: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. --->
System.Exception: Could not download file DIR/M/Tworks/Med_20150312_1.zip
at ScriptTask_225d4c1b04ed4ed39de44dce6dacc7b1.vbproj.ScriptMain.Main()
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
at System.Type.InvokeMember(String name, BindingFlags invokeAttr, Binder binder, Object target, Object[] args, CultureInfo culture)
at Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTATaskScriptingEngine.ExecuteScript()
Hi all,
We have a serious performance problem at a customer’s project. We are in the middle of the migration from SQL 2005 to SQL 2014.
The DataWarehouse has about 25 different source systems including DB2. In the “old” DWH the customer used the OLEDB for DB2 provider (from SQL2005 feature pack).
Now we are facing severe performance problems with the new OLEDB provider for DB2 (from SQL 2014 feature pack).
| SSIS 2005 | SSIS 2014 |
DB2 – 20 Mio rows, ~80 gigabytes | ~ 2:15 hours | ~ 6:30 hours |
We compared the connection string settings and tried several tweaks on DB2 side.
Are you aware of any similar problems? As we have to meet the go-live date with end of june we need to have a solution quite soon.
If you have any hints for us, please point these out!
Thanks
Wolfgang
-- wolfgang strasser, software architect w.strasser_AT_cubido.at
Hi guys,
Please I have about 300Million rows of data I want to load from a remote SQL server into another SQL server. The problem is that the data takes forever to load and I have a time line.
I was wondering if there is a way in can do bulk insert for this or maybe if I can get a code that I can use to load the data faster.
Please I really need your help as my job depends on this.
me
When I try to deserialize an XML document (defined via File-Connection) it works fine in debug mode but I get an error in non-debug-mode. What's the problem here?
Error: 2011-12-29 19:31:40.25 Code: 0x00000001 Source: Read data from SAP Description: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: There is an error in X ML document (0, 0). ---> System.TypeInitializationException: The type initializer for 'Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderList1' threw an exception . ---> System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderList1..cctor() --- End of inner exception stack trace --- at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderList1..ctor() at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializerContract.get_Reader() at System.Xml.Serialization.TempAssembly.InvokeReader(XmlMapping mapping, XmlReader xmlReader, XmlDeserializationEvents events, String encodingStyle) at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events) --- End of inner exception stack trace --- at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events) at System.Xml.Serialization.XmlSerializer.Deserialize(TextReader textReader)
Hello All,
Does anyone know what's the best way to download the latest file from FTP Site. for Ex lets say my files are named as below and i want to Pick the TV_20150316_1.zip
TV_20150310_1.zipDear All Members,
I have developed an SSIS package to import Excel File to SQL Server Database. I have properly mapped all the fields. But due to some rows the Data Flow Task is getting failed. What I want to do is to ignore those rows (and to keep track) and insert the others in the SQL Server Database. How can I do this? Please help.
Thanks and Regards,
Anujit Karmakar
Anujit Karmakar Sr. Software Engineer