I'm working on importing one table to dev table, my source table has 780 rows in it. I have [keep identity] checked, my destination table has 50 rows in it, (yes some ID's are the same in each table that I have to find a work around for), however, the import is skipping the first 500 rows and only importing around 200 rows or so. What would cause that to happen?
↧
not importing all rows
↧
SSIS Access a one drive location excel file
Hi,
My sql account does not have access to a one drive location but I use my credential and I can navigate from the web
how can I do this using a ssis package with my credentials to import an excel file on the one drive?
please help with any suggestions
regards
↧
↧
Is It Possible to Test for Data Type in an Expression?
Hello, all -
I'm quite sure everyone is well aware that, when inputting from an Excel source, SSIS frustratingly tests the first dozen or so rows of data and decides the data type on its own, ignoring any settings one may have set in the Advanced Editor.
This is not a big problem, if the data is consistent. One can slap in a Derived Column task to convert data as needed.
But, what if the data changes? What if the presumed data type in that collection of rows that SSIS tests changes? SSIS will - I'm guessing - alter the data type of the changed column and that could screw up all sorts of things down stream.
So, my question is, is it possible to test for a column's data type? That way, if something happens and SSIS decides that the column is, say, a date field, when you've already set the thing up to massage string data to pull out a date, I'll be able to process it as a date data type, rather than text.
Or, am I full of beans and making a presumption that will never happen?
Thanx in advance for any assistance!
↧
SSIS Project Deployment-Connection manager Username, passwords changing in Every ETL run
Hi,
SSIS Package Connection manager credentials are changing on every month, these credentials needs to taken from the Secret Server Pass code Server, what ever the username and pass code generated in secret server those values needs to update in connection manager in SSIS Packages. Present SSIS packages are in project deployment mode with Environment variables for connection managers . In the SSIS Project we have 15 packages and 10 Project connection managers, all are individual packages no parent child packages.
Please suggest me the way needs to fulfill this requirement.
I am planning to create one package in the same project with script Task to get credentials from secret pass code server, but Connection managers are in Project mode so it is allowing parameters only not variables, we can not mention Project parameters in script task to store output value.I Struck on this point.
↧
Working in a prod environment with a SSIS package that read an ODATA source - SSIS 2016-2017
Hi,
I need to implement a SSIS 2016-2017 package in order to read an ODATA source and then to write onto a SQL Server table.
So, I need to understand what I must install on a prod environment in order to function this SSIS pkg. I could figure to install only an ODATA component, but I cannot installing any development tools on a prod environment.
Any helps to me in order to solve this issue, please? Thanks
↧
↧
SSIS Execute Process Task Editor sometimes runs to completion and sometimes does not
It's crazy. Sometimes when I Start the SSIS Package from Microsoft Visual Studio it will run to completion using anExecute Process Task to PGP Encrypt a file. I looked and played withProtectionLevel, Execute Process Task Editor WindowStyle, even changing theExecute Process Task Editor Executable to be a variable rather than specifying the fully qualified path using C: Drive and specifying the Path using the C: Drive to@[User::PGPExecutable] to be C:\Program Files(x86)\GnuPG\bin\gpg.exe
This is driving me CRAZY!!!!
There must be some simple explanation as to why this sometimes runs and sometimes does not.
Thanks for your review and am soooooo hopeful for a reply.
↧
How to create ssis doing import data from excel to database sql server 2012 ?
problem
How to create ssis doing import data from excel to database sql server 2012 then show result status on excel ?
i work on sql server 2012 i create stored procedure import data from excel to database
steps for work
1- i create excel 2010 have textcolumns companyName,textcode,MetaData
2-store data on temptable and validate it if any thing form entry data not have id on my database or empty then status will not valid
3-insert data if status =null that meaning no error
4-status will be insert to record inserted
5-record have wrong or status not =null meaning have wrong data
6- i collect both inserted and not inserted as output excel sheet to kno reasons of record
not inserted
and records not inserted and show reasons on status field on excel output
i here using openrowset i need to make it using ssis package
How to make steps above using ssis package
Exec [PCN].[sp_MasterDataInsert]'\\192.168.3.108\Import\CompanyNewsInputTemplate.xlsx','\\192.168.3.108\export\CompanyNewsOutputTemplate.xlsx'
create proc [PCN].[sp_MasterDataInsert]
@ImportFilePath varchar(500) ,
@ExportFilePath varchar(500)
as
begin
create table #TempMaster(
Id int identity,
CompanyId nvarchar(200),
CompanyName nvarchar(50),
DocumentRevision_ID INT,
DoconlineUrl nvarchar(300),
DocofflineUrl nvarchar(300),
AttachementRevision_ID INT,
TextCode NVARCHAR(500),
MetaData NVARCHAR(500),
AttachonlineUrl nvarchar(300),
AttachofflineUrl nvarchar(300),
SourceGeneralTypeID int,
SourceGeneralType nvarchar(50),
ImageGeneralTypeID int,
ImageGeneralType nvarchar(300),
InitialPriGeneralTypeID int,
InitialPriGeneralType nvarchar(300),
DailyBacklogGeneralTypeID int,
DailyBacklogGeneralType nvarchar(300),
[Status] VARCHAR(50)
)
Declare @sql nvarchar(max)
Set @sql= 'insert into #TempMaster
(
CompanyName ,
DoconlineUrl,
DocofflineUrl,
AttachonlineUrl,
AttachofflineUrl,
SourceGeneralType,
ImageGeneralType,
InitialPriGeneralType,
DailyBacklogGeneralType,
TextCode,
MetaData
)
Select
CompanyName ,
DoconlineUrl,
DocofflineUrl,
AttachonlineUrl,
AttachofflineUrl,
SourceGeneralType,
ImageGeneralType,
InitialPriGeneralType,
DailyBacklogGeneralType,
TextCode,
MetaData
FROM OPENROWSET( ''Microsoft.ACE.OLEDB.12.0'', ''Excel 12.0 Xml;HDR=YES;Database=' + @ImportFilePath + ''','' SELECT * FROM [Sheet1$]'')'
Exec(@sql)
update tmp
set tmp.CompanyId=c.CompanyID
from #TempMaster tmp
INNER join Z2DataCompanyManagement.[CompanyManagers].[Company] c on
c.CompanyName =tmp.CompanyName
where tmp.[Status] is null
update tmp
set tmp.[Status]='Company Not Found'
from #TempMaster tmp
where tmp.[Status] is null
and tmp.CompanyId is null
--====================================
update tmp
set tmp.SourceGeneralTypeID=c.GeneralTypeID
from #TempMaster tmp
INNER join [PCN].[GeneralTypes] c on c.GeneralTypeName =tmp.SourceGeneralType and c.TypeID=1
where tmp.[Status] is null and c.GeneralTypeID is not null
update tmp
set tmp.[Status]='Source Type Not Found'
from #TempMaster tmp
where tmp.[Status] is null
and SourceGeneralTypeID is null
update tmp
set tmp.DailyBacklogGeneralTypeID=c.GeneralTypeID
from #TempMaster tmp
INNER join [PCN].[GeneralTypes] c on
c.GeneralTypeName =tmp.DailyBacklogGeneralType and c.TypeID=2
where tmp.[Status] is null and c.GeneralTypeID is not null
update tmp
set tmp.[Status]='DailyBacklogGeneralTypeID Type Not Found'
from #TempMaster tmp
where tmp.[Status] is null
and DailyBacklogGeneralTypeID is null
update tmp
set tmp.[Status]='MetaData Not Found'
from #TempMaster tmp
where tmp.[Status] is null
and MetaData is null
update tmp
set tmp.[Status]='TextCode Not Found'
from #TempMaster tmp
where tmp.[Status] is null
and TextCode is null
insert into
[PCN].[MasterData]
(
CompanyID ,
DocumentRevision_ID,
AttachementRevision_ID,
SourceGeneralTypeID,
ImageGeneralTypeID,
InitialPriGeneralTypeID,
DailyBacklogGeneralTypeID,
TextCode,
MetaData,
CreatedBy ,
CreatedDate ,
ModifiedDate
)
select
CompanyID ,
DocumentRevision_ID,
AttachementRevision_ID,
SourceGeneralTypeID,
ImageGeneralTypeID,
InitialPriGeneralTypeID,
DailyBacklogGeneralTypeID,
TextCode,
MetaData,
@CreatedBy ,
GETDATE() ,
GETDATE()
from #TempMaster tmp
where tmp.[Status] is null
update tmp
set tmp.[Status]='Inserted'
from #TempMaster tmp
where tmp.[Status] is null
DECLARE @Export varchar(max)
SET @Export =
'INSERT INTO OPENROWSET(''Microsoft.ACE.OLEDB.12.0'',''Excel 12.0 Xml;HDR=YES;Database='+ @ExportFilePath +''',''SELECT * FROM [Sheet1$]'')
select
CompanyName ,
DoconlineUrl,
DocofflineUrl,
AttachonlineUrl,
AttachofflineUrl,
SourceGeneralType,
ImageGeneralType,
InitialPriGeneralType,
DailyBacklogGeneralType,
TextCode,
MetaData,
[Status]
from #TempMaster' ;
execute (@Export);
drop table #TempMaster
End
↧
SSIS Job just hangs when a connection lost in between runs
Hi,
SSIS Job just hangs when a connection lost in between runs
Is there a way to set expire limit on a job & package then stop job and disable then enable job and retry after x min
thanks
↧
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION.
Hello everyone,
i am facing some trouble with my transaction in SSIS package.
I have a Begin Transaction task (SQL) after which i start my Data flow task to insert the data from csv file to a staging table.
Next, i have an SQL task (SP to insert data from Staging table to PROD table).
Now if the data flow task fails, SSIS correctly goes to the 'Rollback Transaction' task. However, if the SP fails SSIS throws the following error...
"The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION."
Please NOTE that the Property 'Retain Same Connection' is set to TRUE for the OLEDB Connection. Please see below a screenshot of my Package. Any help would be highly appreciated. Thanks.
↧
↧
WMI Watcher Taks Stops the triggering package after some time
Hi All,
I am using WMI Watcher taks with following configurations. It is working fine some files loaing. After 24 hours while loading file it is not triggering or it is not processing file even placed working file.
is there Any default time for WMI watcher task ? how to fix to watch wants to place file any time either it would be hours or days.
Query :
SELECT * FROM __InstanceCreationEvent WITHIN 10 WHERE TargetInstance ISA "CIM_DirectoryContainsFile"
and TargetInstance.GroupComponent= "Win32_Directory.Name=\"C:\\\\Test Import\""
Could some one please let me know any issues with these configure setting ?
Thanks,
Sri
↧
SQL Server Integration Service 2017 on Linux Server
Hi,
Please advise how SSIS(SQL Server Integration Service) can be installed on Linux Server:Ubuntu 18.04 LTS. How SSIS works on linux Server.
Regards,
Suresh Kumar
↧
SSIS Package Running by Agent Job, got hand automatically some times
Hi Experts,
I created a ssis package that is copying data from one server to another server. I enabled ssis logging and applied preExecute ,postExecute and Onerror event handler on package level through which i am maintaining package activity in a activity table, when each task start and ends or gives error.
I scheduled my package through sql server agent job for 1 minute frequency.
Some times i observed my package does not work on last task. And according to syssislog table there is "OnPackageStart"
event but not "OnPackageEnd" event for this package. and i also did not get any error in the syssislog table as well
in activity table. And Endtime (After PostExcute) not logged in activity table for last task.
I checked there is no any blocking in the queries used in ssis package.
So Please suggest me, what may be the problem.
Regards,
Raghvendra Rajak.
↧
Datetime casting in SSIS
I get on a regular schedule a large csv file (over 30 GB) with over 50 columns. 3 of the columns are basically datetime format like “31DEC2017:23:59:59”. Some of the datetime values are “.” which are nulls. I would like to do all transformation in SSIS, not import into stg table then update the datetime values for 3 columns before load to final table for this wide table.
If I use a data source with File Connection manager and send it directly to final table destination then ‘31DEC2017:23:59:59’ fails casting to datetime due to first “:” (replacing the first occurence of ":" with " " will work like “31DEC2017 23:59:59”). I tried adding a derived column transformation but the replace function in SSIS does not support occurrence so will replace all “:” to “ “ and I would only need for first occurrence to be replaced. I can make a more complex formula in column transformation, but am looking for something better for maintenance (but also performant). If I use a script component (with transformation) for the 3 columns after the data source then the output columns I have to manually add (for a wide and also large table is not great) .
Do you have a better solution?
Thanks,
↧
↧
Catch varbinary data result into SSIS variable through Execute SQL Task Result Set
Hi Experts,
I have created a ssis package in that i used a execute sql task that executing query "select MIN_ACTIVE_ROWVERSION() as MaxVersion" and this return rowversion of a DB in hexadecimal format. I want to capture this value in a ssis variable for further use in the package. I made a ssis variable with data type byte, but it is giving error, [Execute SQL Task] Error: An error occurred while assigning a value to variable "FirstLoadMaxVersion": "The type of the value (String) being assigned to variable "User::FirstLoadMaxVersion" differs from the current variable type (Byte). Variables may not change type during execution. Variable types are strict, except for variables of type Object." when using this variable in result set of execute sql task. instead of this when i try with output parameter for this variable and data type byte, This variable hold a value '67' instead of hexadecimal value.
So please suggest what should be the correct data type of ssis variable so that it can hold hexadecimal value in proper format.
Thanks & Regards,
Raghvendra Rajak
↧
How to convert a scientific value to varchar
I am importing a csv file to table. There's a column that shows a scientific value and that is how it getting imported to SQL server table. My table column datatype is varchar (50). Any ideas how to convert it in SSIS ?
Thanks,
Ali.
↧
Excel-to-SQL Import: Failed conversions become NULL with no error
I'm importing an Excel sheet into SQL Server.
I have a column of numeric values and there could be erroneous non-numeric text in the column. I want to give the user a list of these occurrences so he can correct them. I'm trying to redirect rows on errors from the Excel Source component in my Data Flow, but it's giving me nothing. How can I get the Excel Source to flag conversion errors?
↧
SSIS Error 0xC020801F With OData Source - Connection on Microsoft Dynamics CRM
Dear All
I'm using a OData component for connect with Microsoft Dynamics CRM.
When i see data in Odata Source component it are OK, i see data.
However, when i start a execute task i have trouble - ERROR 0xC020801F!
My OData Connect:
I can see data in component:
But, when a run task - ERROR 0xC020801F
Please!!! Anywhere can help-me?
I try all possibilities finded in foruns.
Thanks!
↧
↧
Cannot find integration services in visual studio 2019.
I am new to SSIS, I have installed Visual Studio 2019.I want to create new SSIS project, but I cannot find SSIS project.<o:p></o:p>
Any help will be appreciated.
↧
Virtual memory Is spiking | Need help
HI Experts,
Need your help, we have one SSIS server ,where we have SQL installed and it host SSIS catalog database.
We see virtual memory issues are coming on the server from long time. But Due to load increase from last Friday from a particular package ,we see more virtual memory issues in Windows logs from then . Due to this there is an business team escalation.
Please advise ,if there is anything to changeSQL side to make it free from Virtual memory issues ?
Here are the server specs:
RAM:16 GB
SQL minimum Memory:0
SQL maximum memory: 2 GB
paging Size in windows: 3 GB
From above details,please help to find if there is anything to change from SQl side to get rid of this issue.
many thanks in advance !
Regards,
Naren
↧
Execute package in job agent delete all the records in the table
Hello everyone,
I have a package to update a table in data warehouse. The updates include adding, updating, and deleting. When I run the package through Visual Studio and integration services catalogs. It works fine. However, when I run this package in a scheduled job, it deletes all the records in the table even though there are not records need to be deleted.
I have been trying to solve this problem for a while, but no success. Any help would be greatly appreciated.
Thanks,
Xiaowen
↧