Phantom files in SSIS loop
Attach if file exists
Hi everyone,
I am using script task to send an email that could have 2 attachments; File A and File B.
Currently, it is sending email only when both files exists. However, there are instances when only one file will be available and I want the mail to send only one attachment.
I created 2 variables for each of the files and use to attach them separately
email.Attachments.Add(FileA) email.Attachments.Add(FileB)
Can some advise what I am doing wrong.
Thanks!
me
For Each Loop Container Missing "Browser" button under Collection Tab
Passing OData Source to Power query within Data Flow
HI,
I want to connect to Sharepoint Online Folder using Power Query in SSIS Data Flow. As I can not do that because of data connection problems in Power Query, I'm trying to use OData Source before Power Query, and then pass table gathered by OData Source to Power Query as Input table. PRoblem is I do not know how to pass OData Source Output table to Power Query input table.
Thanks for any help
Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.The wait operation timed out
Hi,
I am trying to split data from a table which has 120 million rows into multiple .txt files with 1 million records in each file.
I am using script task in SSIS to achieve the same but after executing the package, it is creating around 8 txt files and then failing with below error.
"SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out"
Below is the code from the scrip task :
#region Help: Introduction to the script task/* The Script Task allows you to perform virtually any operation that can be accomplished in
* a .Net application within the context of an Integration Services control flow.
*
* Expand the other regions which have "Help" prefixes for examples of specific ways to use
* Integration Services features within this script task. */
#endregion
#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
using System.Data.SqlClient;
#endregion
namespace ST_056616145e7743499d097317fe432711
{
/// <summary>
/// ScriptMain is the entry point class of the script. Do not change the name, attributes,
/// or parent of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region Help: Using Integration Services variables and parameters in a script
/* To use a variable in this script, first ensure that the variable has been added to
* either the list contained in the ReadOnlyVariables property or the list contained in
* the ReadWriteVariables property of this script task, according to whether or not your
* code needs to write to the variable. To add the variable, save this script, close this instance of
* Visual Studio, and update the ReadOnlyVariables and
* ReadWriteVariables properties in the Script Transformation Editor window.
* To use a parameter in this script, follow the same steps. Parameters are always read-only.
*
* Example of reading from a variable:
* DateTime startTime = (DateTime) Dts.Variables["System::StartTime"].Value;
*
* Example of writing to a variable:
* Dts.Variables["User::myStringVariable"].Value = "new value";
*
* Example of reading from a package parameter:
* int batchId = (int) Dts.Variables["$Package::batchId"].Value;
*
* Example of reading from a project parameter:
* int batchId = (int) Dts.Variables["$Project::batchId"].Value;
*
* Example of reading from a sensitive project parameter:
* int batchId = (int) Dts.Variables["$Project::batchId"].GetSensitiveValue();
* */
#endregion
#region Help: Firing Integration Services events from a script
/* This script task can fire events for logging purposes.
*
* Example of firing an error event:
* Dts.Events.FireError(18, "Process Values", "Bad value", "", 0);
*
* Example of firing an information event:
* Dts.Events.FireInformation(3, "Process Values", "Processing has started", "", 0, ref fireAgain)
*
* Example of firing a warning event:
* Dts.Events.FireWarning(14, "Process Values", "No values received for input", "", 0);
* */
#endregion
#region Help: Using Integration Services connection managers in a script
/* Some types of connection managers can be used in this script task. See the topic
* "Working with Connection Managers Programatically" for details.
*
* Example of using an ADO.Net connection manager:
* object rawConnection = Dts.Connections["Sales DB"].AcquireConnection(Dts.Transaction);
* SqlConnection myADONETConnection = (SqlConnection)rawConnection;
* //Use the connection in some code here, then release the connection
* Dts.Connections["Sales DB"].ReleaseConnection(rawConnection);
*
* Example of using a File connection manager
* object rawConnection = Dts.Connections["Prices.zip"].AcquireConnection(Dts.Transaction);
* string filePath = (string)rawConnection;
* //Use the connection in some code here, then release the connection
* Dts.Connections["Prices.zip"].ReleaseConnection(rawConnection);
* */
#endregion
/// <summary>
/// This method is called when this script task executes in the control flow.
/// Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
/// To open Help, press F1.
/// </summary>
public void Main()
{
// TODO: Add your code here
string datetime = DateTime.Now.ToString("yyyyMMddHHmmss");
try
{
//Declare Variables
string FileNamePart = Dts.Variables["User::FileNamePart"].Value.ToString();
string DestinationFolder = Dts.Variables["User::DestinationFolder"].Value.ToString();
string TableName = Dts.Variables["User::TableName"].Value.ToString();
string FileDelimiter = Dts.Variables["User::FileDelimiter"].Value.ToString();
string FileExtension = Dts.Variables["User::FileExtension"].Value.ToString();
Int32 RecordCntPerFile = (Int32)Dts.Variables["User::RecordsPerFile"].Value;
string RecordCntPerFileDecimal = RecordCntPerFile + ".0";
//USE ADO.NET Connection from SSIS Package to get data from table
SqlConnection myADONETConnection = new SqlConnection();
myADONETConnection = (SqlConnection)(Dts.Connections["DBConn"].AcquireConnection(Dts.Transaction) as SqlConnection);
//Read distinct Group Values for each Excel Sheet
string query = "select ceiling(count(*)/" + RecordCntPerFileDecimal + ") AS LoopCnt from " + TableName+ " (nolock)";
decimal LoopCnt = 0;
//Get the Count of Sheets need to be created
SqlCommand cmd = myADONETConnection.CreateCommand();
cmd.CommandText = query;
cmd.CommandTimeout = 7200;
LoopCnt = (decimal)cmd.ExecuteScalar();
int startRowCnt = 0;
int endRowCnt = RecordCntPerFile;
for (int fileloop = 1; fileloop <= LoopCnt; fileloop++)
{
//Load Data into DataTable from SQL ServerTable
string queryString = ";with cte as (Select *, Row_Number() over (order by (Select 1)) AS RowNumber from " +
TableName + ") Select * From cte where RowNumber > " + startRowCnt.ToString() + " and RowNumber<=" + endRowCnt.ToString();
SqlDataAdapter adapter = new SqlDataAdapter(queryString, myADONETConnection);
DataSet ds = new DataSet();
adapter.Fill(ds);
foreach (DataTable d_table in ds.Tables)
{
string FileFullPath = DestinationFolder + "\\" + FileNamePart + "_" + fileloop.ToString() + FileExtension;
StreamWriter sw = null;
sw = new StreamWriter(FileFullPath, false);
// Write the Header Row to File
int ColumnCount = d_table.Columns.Count;
for (int ic = 0; ic < ColumnCount; ic++)
{
sw.Write(d_table.Columns[ic]);
if (ic < ColumnCount - 1)
{
sw.Write(FileDelimiter);
}
}
sw.Write(sw.NewLine);
// Write All Rows to the File
foreach (DataRow dr in d_table.Rows)
{
for (int ir = 0; ir < ColumnCount; ir++)
{
if (!Convert.IsDBNull(dr[ir]))
{
sw.Write(dr[ir].ToString());
}
if (ir < ColumnCount - 1)
{
sw.Write(FileDelimiter);
}
}
sw.Write(sw.NewLine);
}
sw.Close();
}
startRowCnt += RecordCntPerFile;
endRowCnt += RecordCntPerFile;
Dts.TaskResult = (int)ScriptResults.Success;
}
}
catch (Exception exception)
{
// Create Log File for Errors
using (StreamWriter sw = File.CreateText(Dts.Variables["User::LogFolder"].Value.ToString() + "\\" +
"ErrorLog_" + datetime + ".log"))
{
sw.WriteLine(exception.ToString());
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
}
#region ScriptResults declaration
/// <summary>
/// This enum provides a convenient shorthand within the scope of this class for setting the
/// result of the script.
///
/// This code was generated automatically.
/// </summary>
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
harshac
"Text was truncated or one or more characters had no match in the target code page."
Error: 0xC02020A1 at Data Flow Task, Flat File Source [2]: Data conversion failed. The data conversion for column "Description" returned status value 4 and status text "Text was truncated or one or more characters had no match in the target code page.".
Error: 0xC020902A at Data Flow Task, Flat File Source [2]: The "Flat File Source.Outputs[Flat File Source Output].Columns[Description]" failed because truncation occurred, and the truncation row disposition on "Flat File Source.Outputs[Flat File Source Output].Columns[Description]" specifies failure on truncation. A truncation error occurred on the specified object of the specified component.
text file source column and dt_str
hi I'm running vs 2017 and have a target oledb destination of sql server developer 2017 and a source of text file.
3 of my columns are text and theoretically could exceed 8000 characters. for now i have the text file connector data type attribute as dt_str and output width 8000.
is there a max option on this output width? my target columns are varchar(max)
Unable to connect to Integration services
Recently rebuilt my Windows 10 machine and I am no longer able to connect to Integration Services from SSMS when I was able to before. The error message I'm getting is the specified service does not exist as an installed service. From my searching, I'd have to use a different version of SSMS but it works on another machine with the same version I have.
I've attached the SQL Server info from both servers and they are identical for SSMS. Only the MDAC is different. I've attached two screenshots, the No version is my machine and the Yes version is the machine that works.
Is there something that I could be missing here?
SSIS + OAUTH2
Hi All,
Rather new to C# and integrations into SSIS. My SSIS work to date has typically been database to database. Now diving into APIs to database....
Trying to connect to the SerraView API (documentation here: https://help.serraview.com/display/ENPD/Access+Serraview+APIs).
I've registered a service account and received the ClientID and Private Key but am struggling on next steps. I've added a Script Component to my SSIS package, installed the required references, placed the first code block into it and updated the references (using...) to try and create the JWT so that I can request the access token but can't seem to find where to go next in my steps.
Any high-level tips or steps someone could lay out for me?
Help is much appreciated! :-)
Excel source is reading the first couple rows then everything else is reading it as NULL
Hello, I have an excel sheet that has around 50 rows and the excel source is able to read the first 40 rows correctly but everything below it is NULL in all the columns. not sure why its happening all the columns contains either text or number (no mixed entities)
I looked around and found that adding IMEX=1 helped some people but it didn't work in my case, I changed the connection to excel 2007 and still the same problem.
screenshot below
What should I do?
Azure SSIS-IR exporting data to access db file failed - Microsoft.Jet.OLEDB.4.0 is not registered
I have some DTSX packages that exporting data from SQLServer to MDB file.
ConnectionString to connect to MDB file
Data Source=\\***\Temp\Temp.mdb;Provider=Microsoft.Jet.OLEDB.4.0;Persist Security Info=False;OLE DB Services=0;
Previously, I used the following code to execute the packages.
Microsoft.SqlServer.Dts.Runtime.Application dtsApplication = new Microsoft.SqlServer.Dts.Runtime.Application(); Microsoft.SqlServer.Dts.Runtime.Package package = dtsApplication.LoadPackage(packagePath, null); Microsoft.SqlServer.Dts.Runtime.DTSExecResult result = package.Execute();
And now I need to migrate the environment to Azure SSIS IR. After executing the package, some error occurred.
The requested OLE DB provider Microsoft.Jet.OLEDB.4.0 is not registered. If the 64-bit driver is not installed, run the package in 32-bit mode.
After some research, I found a solution that to configure a self-hosted IR as a proxy for an Azure-SSIS IR, but I don't want to setup it.
What else should I do?
Installing SSIS with Visual Studio 2019/SQL Server 2019
Hello!
I used to install/use SSIS as part of SQL Server Data Tools but according to MS "With Visual Studio 2019, the required functionality to enable Analysis Services, Integration Services, and Reporting Services projects has moved into the respective Visual Studio extensions.". This article states:
If you don’t already have a license to Visual Studio 2019:- Install Visual Studio 2019 Community
- Install the Analysis Services, Integration Services or Reporting Services extension as appropriate
Running VS 2019 Community's setup leads me to the following screen:
I don't know what should/must I choose on the Workloads page... As MS says installing SSIS is a two-step process so I must first install VS2019 and only after that install the SSIS extentions, but what file runs the setup for the SSIS extention?
For example, on the Individual components I can select SQL Server Data Tools without first installing VS2019 (and this single selections leads to a number of other components being selected) - does it mean that I can install SSIS with only one step?
If not what exactly should I choose on step 1 (the first picture) and how can I install SSIS extension?
Thank you in advance,
Michael
Large CSV file load into SQL Columnstore table Random Error , After restart same SSIS works fine
I had experienced strange error while loading csv file (, separated values) contain ~1.2 million rows into SQL (2016) columnstore table using SSIS package, I got following error rarely ,especially on datetime columns.
After i simply restarted failed ETl it just works fine. We load same file in different environments and error appears only in one environment on same day. I try to add error output and wait to see for next time, Meanwhile i would like to reach out to experts ask for help, if there is any issue with sql columnstore table or SSIS while loading datatime values.
But error is while insert data, so it could be more of database side issue.
PackageName > Sequence Container > DFL Transactions. SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error
code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80004005 Description: "Invalid date format". PackageName > Sequence Container > DFL Transactions. There
was an error with DST Transactions.Inputs[OLE DB Destination Input].Columns[Account_FromDate] on DST Transactions.Inputs[OLE DB Destination Input]. The column status returned was: "Conversion failed because the data value overflowed the specified type.".
ps: Posted same question on satackoverflow as well.
SQL Agent job throws error Microsoft.ScriptTask. The parameter is incorrect
I get error while running the SQL Agent job which executes SSIS package. The SSIS package job runs well when we run in VS manually, but throws error from SQL agent. Below are the details of config.
SQL Server 2019, VS 2019
SSMS v18.6
SSIS project TargetServerVersion is Sql Server 2019
SSDT for VS 2017 v15.9.25
Error - Description: Cannot create a task from XML for task "Get PackageRun Stats", type "Microsoft.ScriptTask" due to error 0x80070057 "The parameter is incorrect."
Below steps are performed
1. I have installed Microsoft.DataTools.IntegrationServices
2. Repair the VS
Please help
Does anyone have experience of using securetrading.net as a data source within SSIS
Hi,
Does anyone have experience of using securetrading.net as a data source within SSIS ? If so I would greatly appreciate your help.
Kind Regards,
Kieran.
Kieran Patrick Wood http://www.innovativebusinessintelligence.com http://uk.linkedin.com/in/kieranpatrickwood http://kieranwood.wordpress.com/
how to find out that SSIS Installed in SQLServer Using C#?
i'm writing a windows form application using C# . i use
ssisServer = new IntegrationServices(ssisConnection);
To run and working with SSIS Packages But Before running the Package i wanted To Make sure the user run the application in the correct system and sql server has the SSIS on It's Sql Server.so how can i find whether ssis is installed on the server or not in C#?
Issue is getting leading 0 in values
I am importing data from excel to SQL Server 2016 using SSIS. Field is Latitude & Longitude
I am using formula (DT_STR,255,1252)((DT_NUMERIC,9,6)(Latitude)) to import the values in Varchar(255) column. I am facing the issue when Latitude & Longitude starts with 0 eg (-0.737610 or 0.737610).
After executing the package when I saw the values in Database I saw like (-.737610 or .737610) where leading 0 will get disappear.
Can anyone please help me resolve my problem.
Error: Microsoft.jet.oledb.4.0 not registered on local machine when running the package from SQL Job
Hi.
I am using Windows Server 2016. I have created a SSIS package in Visual studio which loads data from Excel and moves to SQL table ( script task functionality using c# )
The above functionality works fine when run in Visual studio. Microsoft Visual studio 2015 shell - version 4.7
Target server version: SQL Server 2016
The excel files I have is of type xls and xlsx
I published the package from visual studio and created a custom job inside the SQL Server Agent and assigned the path of the package to the job.
On running the application from the SQL Server Agent Job, I am getting error as - Microsoft.jet.oledb.4.0 not registered on local machine. But it works , when run from visual studio.
How to fix this?
Thanks
Connect SharePoint List view to SSMS
Hi ,
I need your help to install SSIS/SSRS in SSMS
I have connected the SharePoint list view to excel for reporting purpose but due to data size the performances is very slow in excel and now I would like to connect the SharePoint List to SSIS/SSRS
I have windows 8, 64 bit system
I have installed SQL Server Management studio 2012 but I am unable to enable the Integration and Reporting Services
may be I have missed to choose the option during installation, I am not technical strong person so could you please advise in layman language to add SSIS/SSRS feature to existing SSMS
Recommended best practice for indexes during loading of data
Hello,
I have an SSIS package that reads incremental data from salesforce and loads into the database tables. No major transformation is being done. The incremental load has an average row count of 1k records. It is executed every hour. In the package, i have a component of dropping all the indexes prior to load and creating them at end of load. The entire graph takes 30 mins to run with almost 25 min just for indexes. Average table row count is around 60k rows.
- Should i drop and create indexes once a day rather than every hourly load ?
- Should i not worry about messing with the indexes.
Any recommendations on how to speed up the ETL ?