I'm using Visual Studio 2017 Community 15.9.3 with BI tools installed. When I execute SSIS packages the execution window is very hard to read as the lines overlap. I've taken a screenshot where you see the effect. Any ideas how I can solve it?
Lee Hawthorn. View my blog at leehbi.com. BI Dev
I have inherited abunch of SSIS support due to a college leaving. Needless to say we are having performance issues and trying to figure out where in the package executions the time is being spent. These packages are stored in an ssis catalog and executed via sql agent jobs. No logging enabled other than the ssis standard reports in ssms.
In a nutshell, this package downloads a bunch of files from AWS S3, parses them does a small bit of transform and writes them to the db. It also writes a log table of the file names it processed. Nothing earth shattering, but trying to find out:
1. how many files did it process each time it ran
2. how long did each loop task task in total (how much time downloading all the files, parsing all the files, how much time writing all the files to the db). I want a total for each task/step for each execution, not one per file processed.
3. Total number of rows processed across all files.
I would like to expose this data via an ssrs report or stored proc to use this as monitor tool for trend analysis as we expect the data volume to continue to grow.
Here is a screen shot of my control flow:
and data flow
Full disclosure: enabled standard logging on everything in the package to table and it writes so many rows I don't know what event(s) to look for to capture this if that information is already there somewhere.
Hi,
i am loading data from excel file and i wanted create the column named load_date in derived column transformation,for that load_date column i wanted to create the the expression ,i have already two columns in excel Year and month,
year is like 2018,and month like 'November' ,and using this two columns i wanted to get the result as '2018-11-09',
and for day part what to do.
could any one tell me how to concatenate this .
thanks,
I've imported my SSIS Packages from SQL 2014 to SQL 2016. SQL Server 2014 is in my Local Machine.
1. If I import packages to SQL 2016 which is in my Local Machine when I execute (in SSMS, SSISDB Catalog) the package then status of the package is in Running State and I can still see the operations being executed well (Data is being loaded from source to destination) even it is showing running in my report.
2. If I import packages to SQL 2016 which is in my Remote Machine when I execute (in SSMS, SSISDB Catalog) the package then status of the package is in Running State and it's not executing.
My Source is a FlatFile in my local Machine and Destination is in SQL Server 2016 (Respectively for Local and Machine(I changed the connections)).
Thank you,
Nihal
Hey there,
we are currently using VS 2017 15.7.5 and SSDT 15.8
We've got SSIS packages which read data from a Progress database via ODBC.
We updated to VS 2017 15.9.2 and now all the ODBC Progress packages fail, nothing else has been changed.
The problem we have is that all the columns which were previously correctly identified as DT_DBDATE are now recognized as DT_BYTES.
It seems to be some weird regional or language problem.
Comparing the ODBC Source previews:
On the machine where we still have 15.7.5 date values show as for example "2018-11-05".
On the new machine with 15.9.2 date values show as "3/11/2014 12:00:00 AM".
Changing any of the machine's regional settings didn't help, so it must be a problem within Visual Studio?
Masking data using SSIS
I have a table Employee and Employee_Mask and need to mask the Employee table by refering the Employee_Mask table.
Example :
source:
Employee
EmpID EmpName
1 AAA
2 BBB
Employee_Mask
EmpID EmpName
1 XXX
2 YYY
Output:
Employee
EmpID EmpName
1 XXX
2 YYY
Questions
1) This is sample records and have millions of records to mask which mean millions of records to update employee
table by referring employee_mask table.
2) what are the best ways to peform masking. I want to do it as bulk update in source table. Is Lookup tranformation will help in SSIs?
3) I dont have 2016 version so i cannot use new functions
4) what are the best ways to mask the data using SSIS ?
Hello,
I am starting my learning on SQL Server 2017. I am at my first step. I have created a database name Test and a table named ListOfApps. It contains 4 variables as follow:
ID int (primary key), AppName nvarchar(50), Rating int and Summary nvarchar(50)
I have made a SSIS project to import a text file into the above mentionned data base. Import text file is working properly.
I have added a data conversion from DT_WSTR to DT_WST. Again, it works well. Last step is to use destination OELDB. Again I have no error message. Everything works well. The only problem, when I go back into my table ListOfApps it is empty, even if I make a refresh or even if I disconnect then reconnect.
Does someone could help me with that issue.
Regards,
alepage
Hey,
I am trying to read a flat file of 17,2 MB data and write it into a SQL table as a XML file. The flat file source is a .nbits file and i read the whole file as one row with data type DT_TEXT. I try to transform the source data to unicode and write it into the SQL table as xml data type. The last step fails reporting following description: "XML parsing: line 147914, character 30, unexpected end of input."
While debugging i tried to read the flat file source and write it directly back into another file, resulting that the written file has only 8 MB instead of the expected 17,2 MB.
Why does the data gets truncated?
My development machine runs Windows 7 64-bit, with Visual Studio 2017 Community Edition, v 15.9.3, and with SSDT for Visual Studio 2017 (15.8.2). My SQL Server box runs Windows Server 2012 R2 Standard, with SQL Server 2012 SP 4 (build 11.0.7001.0). I use SQL Server Management Studio v 17.4 on my Windows 7 development machine.
I have Oracle's SQL Developer installed and working fine: I can write and execute queries. Now I need to create an SSIS package that does the same thing: execute a SELECT query on Oracle, then load it into a SQL Server table. There are hundreds of different drivers and tools on the Oracle and Microsoft sites, but so far none of them work -- where "work" means "allow me to execute the package in Visual Studio and also deploy and execute the SSIS package on SQL Server." I get myriads of different error messages, depending on which drivers I try. I have no problem with the SQL Server connection, of course -- only the Oracle connection.
My hope is that there is a driver I can install on my development machine using nothing but (the appropriate bitness of) regsvr32 and a corresponding driver I can install on my SQL Server box using nothing but Regsvr32. I don't want hundreds of java files, development tools, setup.exe's that stuff thousands of entries in my registry. I just want to register a single dll (or maybe two if you tell me I need both 32 and 64 bit drivers). After registering the DLL, I need it to appear in the drop-down list of providers at the top of the Connection Manager window in Visual Studio (the one that shows various .Net Provders and .Net Providers for OleDb.
Surely thousands of people have figured out how to do this -- but days of Googling inevitably lead down the path of "install the Oracle Instant Client" or "use the Oracle Universal Installer" or some such -- and these are like throwing a fragmentation grenade into the registry -- and still don't work (see my above definition of "work").
Any suggestions?
Leigh Webber
The ssis package is part of a project deployed to a server with SQL Server 2012 SSISDB. The project is using environments. The project is set to Run64BitRuntime = False. This particular package is attempting to use a Data Flow to use an Excel Source (Microsoft Excel 97-2003 Worksheet) and load that to a SQL Server db table OLE DB Destination.
The package runs fine locally in Windows 7 MS Visual Studio Ultimate 2012 v 11.0.61030.00 Update 4. But if I run the package by command line on the server, the package fails and returns the following error:
MyPackage:Error: The requested OLE Provider Microsoft.Jet.OLEDB.4.0 is not registered. If the 64-bit driver is not installed, run the package in 32-bit mode. Error code: 0x00000000. An OLE DB record is available. Source "Microsoft OLE DB Service Components" Hresult: 0x80040154 Description: "Class not registered"
the SSISDB log returns the following message:
SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "SQLImport Step_XLS" failed with error code 0xC0209303. There may be error messages posted before this with more information on why the AcquireConnection method call failed.
All other packages within this project run and end successfully, but this is the only one that uses an excel spreadsheet as input. If anyone have experienced this before or has any ideas on what to do next, I would appreciate it. Thanks!
We're pulling data from a hosted OData entity and the scale on some of the External Columns of the OData source are defaulting to zero even though many of the values in the column have decimals.
I've opened the Advanced Editor and tried to change the Scale of the External Column. It allows me to change the Scale, but it changes back to zero after I hit Ok.
I need to update Table Parent, whenever there is a new set of records added to Child. There are about 6 Child tables and whenever there is a update in any of the table, I need to kick out the existing rows for that child in parent table and add new records in child to parent. The whole process should be automated, i.e., whenever there is a notification of new rows in child, it should trigger the above mentioned operation.
I am very new to SSIS, so any kind of ideas to start and fulfill this task will be very helpful.
The file is not available in the location also.
The error is in SQL SERVER 2014.
Hi
What version of SQL Server are compatible with Visual Studio 2013 Ultimate edition?
Will MS SQL Server 2014 Enterprise Edition be compatible?
Thanks
Velma
Hi,
May be this issue already raised on this forum but am facing following error.
To run a SSIS package outside of SQL Server Data Tools you must install Standard Edition of Integration Services or higher.
As package build using Microsoft Visual Studio 2015 Shell (Integrated) and T-SQL job created for execution in SQL Server 2016. Following T-SQL command used in SQL Agent job for package execution.
D:\Program Files (x86)\Microsoft SQL Server\140\DTS\Binn\dtexec.exe /F D:\Packages\MyTest.dtsx /Conf D:\Config\Config.dtsconfig
Read on some threads advising to install Integration services on SQL Server but my colleagues packages built on same Visual Studio working fine.
Many Thanks,
Muhammad Yasir
Many Thanks, Muhammad Yasir
Hi,
I've implemented a SSIS 2014 project and deployed it into the SSIS catalog. I need to allow another person to run a package, but I'd like to assign him the minimum permission to do it.
Any suggests to me, please?
Thanks
Hi we have an ssis package which loads the .csv files into the sql table on weekly basis
while loading the files into sql table,facing this conversion error , and I have tried to change the column size in my destination table as well, still we are facing the same error
could you please help out for this situation ,what is the best approach to reach out from here.
[OLE DB Destination [82]] Error: There was an error with OLE DB Destination.Inputs[OLE DB Destination Input].Columns[VALUE] on OLE DB Destination.Inputs[OLE DB Destination Input]. The column status returned was: "Conversion failed because the data value overflowed the specified type.".
While executing SSIS Package from Visual studio data tools 2010 (MS SQL 2012) getting below error
[SSIS.Pipeline] Error: The component metadata for "ABC Domain,Let me know if anyone can help to fix this issue.
Thanks,
Sriram
Hi All,
I am currently using Visual Studio 2013 ultimate with MSDN. Can I deploy the latest version of MS SQL(2014/2016) for a test environment with a same license.
Thanks
Velma