Connecting to SFTP via SSIS - ssis

I'm trying to connect to a SFTP server via an SSIS package. The package executes WinSCP with the following connection string in a .txt file:
open sftp://username:fc$#6444#example.com:22
However the package keeps failing without being able to connect. Is it something to do with the special characters in the password?
I am able to connect to a different SFTP if I replace the string so I know it must be something to do with the syntax above. I've tried putting double quotes around the string as follows without any success:
open "sftp://username:fc$#6444#example.com:22"

I had to do this too, for one of my work projects recently. We used the WinSCP .NET assembly inside an SSIS Scripting Task, as this is what WinSCP also recommends as the way to achieve SFTP using WinSCP in SSIS.
See this guide - Using WinSCP .NET Assembly from SQL Server Integration Services (SSIS). It walks you through the install and setup and also contains working sample code (after you change the script to your needs of course!).
Sample code - after you reference the WinSCPnet.dll assembly - is below.
using System;
using Microsoft.SqlServer.Dts.Runtime;
using Microsoft.SqlServer.Dts.Tasks.ScriptTask;
using System.AddIn;
using WinSCP;
namespace ST_5a30686e70c04c5a8a93729fd90b8c79.csproj
{
[AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]
public partial class ScriptMain : VSTARTScriptObjectModelBase
{
public void Main()
{
// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Sftp,
// To setup these variables, go to SSIS > Variables.
// To make them accessible from the script task, in the context menu of the task,
// choose Edit. On the Script task editor on Script page, select ReadOnlyVariables,
// and tick the below properties.
HostName = (string) Dts.Variables["User::HostName"].Value,
UserName = (string) Dts.Variables["User::UserName"].Value,
Password = (string) Dts.Variables["User::Password"].Value,
SshHostKeyFingerprint = (string) Dts.Variables["User::SshHostKeyFingerprint"].Value
};
try
{
using (Session session = new Session())
{
// As WinSCP .NET assembly has to be stored in GAC to be used with SSIS,
// you need to set path to WinSCP.exe explicitly, if using non-default location.
session.ExecutablePath = #"C:\winscp\winscp.exe";
// Connect
session.Open(sessionOptions);
// Upload files
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary;
TransferOperationResult transferResult;
transferResult = session.PutFiles(#"d:\toupload\*", "/home/user/", false, transferOptions);
// Throw on any error
transferResult.Check();
// Print results
bool fireAgain = false;
foreach (TransferEventArgs transfer in transferResult.Transfers)
{
Dts.Events.FireInformation(0, null,
string.Format("Upload of {0} succeeded", transfer.FileName),
null, 0, ref fireAgain);
}
}
Dts.TaskResult = (int)DTSExecResult.Success;
}
catch (Exception e)
{
Dts.Events.FireError(0, null,
string.Format("Error when using WinSCP to upload files: {0}", e),
null, 0);
Dts.TaskResult = (int)DTSExecResult.Failure;
}
}
}
}

Install WinSCP and then create a folder where you want a file from client or put the file.Then Open a Execute Process Task and then go to Expression tab and set the Executable and Arguments with below codes(Please change accordingly).
Write this code in notepad and save as winscp.txt at the path C:\path\to\winscp.txt.
Open sftp://Host_Name:Password#apacsftp01.mftservice.com/ -hostkey="ssh-rsa 2048 xxxxxxxxxxx...="
get -delete /home/client/Share/MediaData/Media_file.xlsx
exit

Related

Winscp NuGet Package not working in SSIS Script Task [duplicate]

I'm trying to use the WinSCP.NET NuGet to upload some files to an SFTP through a Script Task component in SSIS. While writing the code everything went fine, but if after attempting to build, the WinSCP.NET dll seems to not be picked up breaking all of the references.
I've tried adding WinSCP path to my PATH variable (user). I've tried to add the local version of the WinSCPNET.dll to the GAC. I've tried to reinstall the package through NuGet. I've even tried to change the framework versions.
This is a problem I've had before with the WinSCP.NET DLL. Last time I ended up using a workaround by interfacing with the command line through C#. But I would like to use the DLL, as it's a much simpler implementation.
The code is basically the boilerplate from WinSCP, with some minor changes:
#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using WinSCP;
#endregion
namespace ST_a1d3d6e0b5d54338bce6c79882c303c6
{
/// <summary>
/// ScriptMain is the entry point class of the script. Do not change the name, attributes,
/// or parent of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region Help: Using Integration Services variables and parameters in a script
/* To use a variable in this script, first ensure that the variable has been added to
* either the list contained in the ReadOnlyVariables property or the list contained in
* the ReadWriteVariables property of this script task, according to whether or not your
* code needs to write to the variable. To add the variable, save this script, close this instance of
* Visual Studio, and update the ReadOnlyVariables and
* ReadWriteVariables properties in the Script Transformation Editor window.
* To use a parameter in this script, follow the same steps. Parameters are always read-only.
*
* Example of reading from a variable:
* DateTime startTime = (DateTime) Dts.Variables["System::StartTime"].Value;
*
* Example of writing to a variable:
* Dts.Variables["User::myStringVariable"].Value = "new value";
*
* Example of reading from a package parameter:
* int batchId = (int) Dts.Variables["$Package::batchId"].Value;
*
* Example of reading from a project parameter:
* int batchId = (int) Dts.Variables["$Project::batchId"].Value;
*
* Example of reading from a sensitive project parameter:
* int batchId = (int) Dts.Variables["$Project::batchId"].GetSensitiveValue();
* */
#endregion
#region Help: Firing Integration Services events from a script
/* This script task can fire events for logging purposes.
*
* Example of firing an error event:
* Dts.Events.FireError(18, "Process Values", "Bad value", "", 0);
*
* Example of firing an information event:
* Dts.Events.FireInformation(3, "Process Values", "Processing has started", "", 0, ref fireAgain)
*
* Example of firing a warning event:
* Dts.Events.FireWarning(14, "Process Values", "No values received for input", "", 0);
* */
#endregion
#region Help: Using Integration Services connection managers in a script
/* Some types of connection managers can be used in this script task. See the topic
* "Working with Connection Managers Programatically" for details.
*
* Example of using an ADO.Net connection manager:
* object rawConnection = Dts.Connections["Sales DB"].AcquireConnection(Dts.Transaction);
* SqlConnection myADONETConnection = (SqlConnection)rawConnection;
* //Use the connection in some code here, then release the connection
* Dts.Connections["Sales DB"].ReleaseConnection(rawConnection);
*
* Example of using a File connection manager
* object rawConnection = Dts.Connections["Prices.zip"].AcquireConnection(Dts.Transaction);
* string filePath = (string)rawConnection;
* //Use the connection in some code here, then release the connection
* Dts.Connections["Prices.zip"].ReleaseConnection(rawConnection);
* */
#endregion
/// <summary>
/// This method is called when this script task executes in the control flow.
/// Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
/// To open Help, press F1.
/// </summary>
public void Main()
{
// TODO: Add your code here
// User::FileName,$Package::SFTP_HostName,$Package::SFTP_Password,$Package::SFTP_PortNumber,$Package::SFTP_UserName
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Sftp,
HostName = (string)Dts.Variables["$Package::SFTP_HostName"].Value,
UserName = (string)Dts.Variables["$Package::SFTP_Password"].Value,
SshHostKeyFingerprint = (string)Dts.Variables["$Package::SFTP_Fingerprint"].Value,
Password = (string)Dts.Variables["$Package::SFTP_Password"].GetSensitiveValue(),
PortNumber = (int) Dts.Variables["$Package::SFTP_PortNumber"].Value,
};
try
{
using (Session session = new Session())
{
// As WinSCP .NET assembly has to be stored in GAC to be used with SSIS,
// you need to set path to WinSCP.exe explicitly,
// if using non-default location.
session.ExecutablePath = (string)Dts.Variables["$Package::WinSCP_Path"].Value;
// Connect
session.Open(sessionOptions);
// Upload files
TransferOptions transferOptions = new TransferOptions();
transferOptions.TransferMode = TransferMode.Binary;
TransferOperationResult transferOperationResult = session.PutFiles(
(string)Dts.Variables["User::FileName"].Value, (string) Dts.Variables["$Package::SFTP_RemoteFileName"].Value,
true, transferOptions);
// Throw on any error
transferOperationResult.Check();
// Print results
bool fireAgain = false;
foreach (TransferEventArgs transferEvent in transferOperationResult.Transfers)
{
Dts.Events.FireInformation(0, null,
string.Format("Upload of {0} succeeded", transferEvent.FileName),
null, 0, ref fireAgain);
}
}
}
catch (Exception e)
{
Dts.Events.FireError(0, null,
string.Format("Error when using WinSCP to upload files: {0}", e),
null, 0);
Dts.TaskResult = (int)DTSExecResult.Failure;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
#region ScriptResults declaration
/// <summary>
/// This enum provides a convenient shorthand within the scope of this class for setting the
/// result of the script.
///
/// This code was generated automatically.
/// </summary>
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
This should compile as is and allow me to run the SSIS, to upload the file. Instead the references break and I receive a lot of missing reference errors:
Error CS0246: The type or namespace name 'WinSCP' could not be found (are you missing a using directive or an assembly reference?)
Error: This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is ..\packages\WinSCP.5.15.0\build\WinSCP.targets.
I can indeed reproduce your problem, when I use WinSCP NuGet package. It looks like a problem between the NuGet package manager and SQL Server Data Tools. The file the error refers to actually does exist (in a path relative to the script task .csproj file).
Actually, it looks like it's not even recommended to use NuGet in SSIS. You should rather register the assembly to GAC:
How can I use NuGet with SSDT?
Creating a reference to a custom assembly from an SSIS Script Task - vb
SSIS Script Task cant find reference to assembly
And indeed, if I follow the WinSCP instructions for using the assembly from SSIS (using the GAC), it works just fine.
Make sure you have uninstalled the NuGet package.
Install WinSCPnet.dll to GAC or subscribe AppDomain.AssemblyResolve event.
And add WinSCPnet.dll to your script task project.

Parallel execution in ssis

I have developed one master package(Main.dtsx) and 3 child packages (Processor.dtsx).Note: Code is same for all child packages that picks up files from source location and process. To optimize the performance, I want that all these 3 child packages should run simultaneously on 10000 files in such a way that first child will pick 1st file and start execution , at the same time second will pick up 2nd file and so on. Please share the code if you have. I tried with 'MaxConcurrentExecutables' option but in that case all components access same file which is not expected.
This cannot be done with a Foreach Loop, but you can accomplish the task with a Script task:
Add 3 string variables to hold the file names (i.e. File1, File2, File3)
Pass the variables from the master package to each child package.
In each child package, configure an expression in the file connection manager to use the parameter as a connection string
At the end of each package, make sure that the file is moved from the source folder or renamed in such a way that it will be ignored in subsequent loops.
Set up a For loop that will end when all the files have been processed. You can add a boolean variable to the package like "ProcessingIsAllDone" and then set this in the script task.
At the top of the For loop add a script task and connect the execute package tasks with precedent constraints.
Use the script below to set the variables
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
namespace ST_e4ccd9cfaa4847ff86ec88c215c1961c
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
DirectoryInfo sourceDirectory = new DirectoryInfo(#"c:\temp");
int loops = 3;
foreach (FileInfo sourceFile in sourceDirectory.GetFiles("*.txt"))
{
if (loops == 0)
{
break;
}
string variableName = String.Format("File{0}", loops);
Dts.Variables[variableName].Value = sourceFile.FullName;
loops--;
}
if (sourceDirectory.GetFiles("*.txt").Length <= 3)
{
Dts.Variables["ProcessingIsAllDone"].Value = true;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
#region ScriptResults declaration
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}

Using MySQL with visual studio and changing the connection at runtime

I use something like this for my application
MySqlConnection cnn = new MySqlConnection("Server=myServerAddress;" +
"Database=myDataBase;" +
"Uid=myUsername;" +
"Pwd=myPassword;");
And this changes everytime because we deploy databases with our application.
It works fine. I type in using(new connection(cnn)){ query... } and go.
And I've got it working with a dataset using a connection defined in the windows ODBC datasouce administrator.
But I'm curious, is there a way to use visual studio's dataset items using the my local test db and then change the connection of the dataset at runtime? Even better, can I use c# to programmatically add the ODBC data source at runtime?
Usually a connection string is loaded from the application exe.config file present in the same folder of the application. This connection string could be defined using the Settings tab in the project properties.
Right click on Properties of your project
Select the Settings tab (confirm the creation if you have no
settings)
Click on the ComboBox in the column type and select Connection String
Give a symbolic name to your connection
Type the connection string in the Value column (Examples at
connectionstrings.com)
Now in your project files you should have the file app.config (that becomes yourapp.exe.config) where there is a section like this
<configuration>
<connectionStrings>
<add name="MyAppConnection"
connectionString="Server=myServerAddress;Database=myDB;Uid=user;Pwd=pass;" />
</connectionStrings>
</configuration
At this point you read it in the program using
string conString = ConfigurationManager
.ConnectionStrings["MyAppConnection"]
.ConnectionString;
Instead in a dynamic situation where you want to build yourself the connection string during runtime (from user inputs, your own configuration files and so on) then you could leverage the functionality of the class MySqlConnectionStringBuilder
MySqlConnectionStringBuilder msb = new MySqlConnectionStringBuilder();
msb.Server = "localhost";
msb.Port = 3306;
msb.UserID = "root";
msb.Password = "xxx";
msb.Database = "test";
MySqlConnection cnn = new MySqlConnection(msb.ConnectionString);
cnn.Open();
Of course, these literal values could be substituted by your own variables.
The documentation of this class is surprising difficult to find. The best docs are the one of the Sql Server equivalent. It is interesting that you could read a static connection string from your config file and then change only the property needed.
string conString = ConfigurationManager
.ConnectionStrings["MyAppConnection"]
.ConnectionString;
MySqlConnectionStringBuilder msb = new MySqlConnectionStringBuilder(conString);
msb.Database = "AnotherDB";
MySqlConnection cnn = new MySqlConnection(msb.ConnectionString);
Application connection string cannot be changed at runtime.
User settings can be changed.
Assuming you are using an application setting-property named "MyConnectionString" which holds the connection string for the entire application.
On your main Program class create a global string:
internal static string Prconnstring;
Create and save this settings.cs file:
namespace MYSOLUTIONORPROJECTNAME.Properties
{
// (Not sure where I found this solution some time ago)
// This class allows you to handle specific events on the settings class:
// The SettingChanging event is raised before a setting's value is changed.
// The PropertyChanged event is raised after a setting's value is changed.
// The SettingsLoaded event is raised after the setting values are loaded.
// The SettingsSaving event is raised before the setting values are saved.
internal sealed partial class Settings
{
public Settings()
{
// // To add event handlers for saving and changing settings, uncomment the lines below:
//
// this.SettingChanging += this.SettingChangingEventHandler;
//
// this.SettingsSaving += this.SettingsSavingEventHandler;
//
}
private void SettingChangingEventHandler(object sender, System.Configuration.SettingChangingEventArgs e)
{
// Add code to handle the SettingChangingEvent event here.
}
private void SettingsSavingEventHandler(object sender, System.ComponentModel.CancelEventArgs e)
{
// Add code to handle the SettingsSaving event here.
}
public override object this[string propertyName]
{
get
{
if (propertyName == "MyConnectionString")
{
return Program.Prconnstring;
}
else
{
return base[propertyName];
}
}
set
{
base[propertyName] = value;
}
}
}
}
Before calling-opening any object that uses the connection string (examples include Forms that use datasets or other classes that use datasets created on the development enviroment) create your new connection string by any means you think. (Example: You might want to use as user name in the connection string the current user. Create the connection string using the info provided form the environment.)
Program.Prconnstring = thenewruntimeconnectionstring.
Now whenever the application tries to get MyConnectionString (which is hardcoded in the myapplicationname.config and cannot be changed) instead gets the new thenewruntimeconnectionstring you provided to Program.Prconnstring.
Be aware that the development connection string will be available-visible to final user, since it is just a text file. If you do not want this, you can change that file (will be a file named NAMEOFMYAPPLICATION.exe.config) during deployment, since the connection string hardcoded there, will be of no use for the running app. Do not delete it, just change.
Your connection string will be stored in your App.config (or c# equivalent). Say it's called MyConnectionString. Just add My.Settings("MyConnectionString")="[your new connection string]" to your entry point to change to database binding at runtime. E.g:
Public Sub New()
' This call is required by the designer.
InitializeComponent()
My.Settings("MyConnectionString") = "server=remotedb.uk;user id=MainUser;password=2jdi38edhnche73g;database=mainDb;persistsecurityinfo=True;allowuservariables=True;defaultcommandtimeout=480;characterset=utf8mb4"
End Sub

How can I get a report URL via the SSRS Web Service?

In my project I have a web reference to SSRS (2005). I would like to display links that can take users directly to rendered reports. I know I can provide a link such as this one:
http://server/ReportServer/Pages/ReportViewer.aspx?/path/to/report&rs:Command=Render&rc:parameters=false&rs:format=HTML4.0
The question is how can I get that URL from the web service? And if the report takes parameters is there a way to provide values to the web service and have it format the URL for me?
I know I can build the URL myself, but I don't like reinventing wheels.
There are a few things to think of about HOW SSRS works and HOW MUCH TIME you want to invest in monkeying with it.
I. You can traverse the root but I highly doubt you meant that. From the root you can add items whether they are directories or reports. And to add to that you can add the parameter directly to the Rest URI to render a report and you may also output a value as well. For example:
Main part of address root:
http:// <server>/ReportServer/Pages/ReportViewer.aspx?
path to directory:
%2fTest
path to report (labeled it the same name lol)
%2fTest
what to do with it? (render it)
&rs:Command=Render
Put a paremeter in and execute it as well (Yes I called my parameter Test too!)
&Test=Value
Put it all together:
http:// <servername>/ReportServer/Pages/ReportViewer.aspx?%2fTest%2fTest&rs:Command=Render&Test=Value
II. You have a database you can query for traversing things but I believe MS does NOT document it well. Generally it is a SQL Server database named 'ReportServer' on whatever server you installed SSRS on. Generally most items are in the table 'dbo.Catalog' with 'Type' of 2 for reports. You can get their info and even parameters from them there.
III. You want to go full bore and dive into .NET and just talk to the service directly? You can do that too. You need the two main services though to do that:
A: http://<Server Name>/reportserver/reportservice2010 (gets info on existing items on server)
B: http:// <Server Name>reportserver/reportexecution2005 (gets info for in code creating reports to types directly in code)
I had another thread on exporting this here: Programmatically Export SSRS report from sharepoint using ReportService2010.asmx; but you will to get info as well probably. ONCE you have created the proxy classes (or made a reference to the web services) you can do code in .NET like so. These services do all the magic so without them you can't really model much in SSRS. Basically I create a class that you pass the 'SERVER' you need to reference to the class like 'http:// /ReportServer'.
private ReportingService2010 _ReportingService = new ReportingService2010();
private ReportExecutionService _ReportingExecution = new ReportExecutionService();
private string _server { get; set; }
public ReaderWriter(string server)
{
_server = server;
_ReportingService.Url = _server + #"/ReportService2010.asmx";
_ReportingService.Credentials = System.Net.CredentialCache.DefaultCredentials;
_ReportingExecution.Url = _server + #"/ReportExecution2005.asmx";
_ReportingExecution.Credentials = System.Net.CredentialCache.DefaultCredentials;
}
public List<ItemParameter> GetReportParameters(string report)
{
try
{
return _ReportingService.GetItemParameters(report, null, false, null, null).ToList();
}
catch (Exception ex)
{
MessageBox.Show("Getting Parameter info threw an error:\n " + ex.Message);
return new List<ItemParameter> { new ItemParameter { Name = "Parameter Not Found" } };
}
}
public List<CatalogItem> GetChildInfo(string dest)
{
try
{
return _ReportingService.ListChildren("/" + dest, false).ToList();
}
catch (Exception ex)
{
MessageBox.Show("Getting Child info of location threw an error:\n\n" + ex.Message);
return new List<CatalogItem> { new CatalogItem { Name = "Path Does Not exist", Path = "Path Does not exist" } };
}
}
ListChildren is the way to go. You can always set the second parameter to true to return all catalog items when you have reports in many folders.
Dim items As CatalogItem() = rs.ListChildren(reportPath, True)

passing parameters to remote beanshell

I need to pass parameter to remote beanshell script which is run through
java -cp bsh-2.0b4.jar bsh.Remote http://10.0.0.1/beanshell script.bsh p1 p2 p3
call.
Is it somehow possible to read params 'p1', 'p2' and 'p3' from within the script.bsh?
p.s. Local params passing through bsh.args works fine, but it's unusable with remote scripting.
I suppose, you are using beanshell library. There is no way to do so, according to sources: the utility takes only 2 arguments: the URL and the local script filename. It even does not support several script filenames, as it claim to.
public class Remote
{
public static void main( String args[] ) throws Exception
{
if ( args.length < 2 ) {
System.out.println("usage: Remote URL(http|bsh) file [ file ] ... ");
System.exit(1);
}
String url = args[0];
String text = getFile(args[1]);
int ret = eval( url, text );
System.exit( ret );
}
Also the server-side should be aware about the arguments passed.
The ways out for you:
Create the script template, in which you will substitute the arguments for the script and save the substitute script to temp dir before passing to bsh.Remote
Create a remote file, where the script can read arguments from. You need additional communication with remote site to upload this file before calling bsh.Remote.