In a Script Task, I am trying to retrieve a file from a networked location and FTP that file to an offsite location
In SSIS I created the FTP Connection and tested that it is setup and works
Created three variables
variable 1. FullPathName = \ftpservercsc\\\Filename.txt
variable 2 FTPFilePath = \ftpservercsc\\\
variable 3 FTPFileName = Filename.txt
Created a Script Task and added the vb code as such ...
'Get instance of the connection manager.
Dim cm As ConnectionManager = Dts.Connections("FTP Connection Manager")
Dim remotePath As String = Dts.Variables("FTPFilePath").Value.ToString
'create the FTP object that sends the files and pass it the connection
'created above.
Dim ftp As FtpClientConnection = New FtpClientConnection
(cm.AcquireConnection(Nothing))
'Connect to the ftp server
ftp.Connect()
'Set the path on the FTP server where dropping files
'ftp.SetWorkingDirectory("/Prequalify") 'set the remote directory
Dim files(0) As String
files(0) = Dts.Variables("FTPFileName").Value.ToString 'eg. File1.trg
'Send File
ftp.SendFiles(files, remotePath, True, True)
' Close the ftp connection
ftp.Close()
'Dts.Events.FireInformation(0, context, "File " + fileToGet
' + " retrieved successfully.", Nothing, Nothing, True)
Dts.TaskResult = Dts.Results.Success
Error: The element cannot be found in a collection. This error happens when you try to retrieve an element from a collection on a container during execution of the package and the element is not there.
So I have commented out and found the error is generating on retrieving the variable value but I do not know what is incorrect here
Dim remotePath As String = Dts.Variables("FTPFilePath").Value.ToString
I have tried multiple variable retrievals and all get the same error. Anyone see anything wrong?
Two things:
Make sure you config the Script Task to have Read Access to the variable. To do this right-click on the Script Task and select Edit. Click the ... under ReadOnlyVariables.
Fully qualify your variables such as Dts.Variables["User::RemotePath"].Value
Related
This question already has answers here:
Upload file to FTP site using VB.NET
(5 answers)
Closed 1 year ago.
All,
SSIS - trying to make an ftp connection to upload files to an ftp server using a script task in visual basic programming language, I have run into a problem. I have not found something similar in my searches so I would appreciate your help.
[Connection manager "FTP"] Error: An error occurred in the requested FTP operation. Detailed error description: 220
550 SSL/TLS required on the control channel
CODE:
Public Sub Main()
'
' Add your code here
'
Try
Dim cm As ConnectionManager
cm = Dts.Connections("FTP")
'Set the properties like username & password
cm.Properties("ServerName").SetValue(cm, "ftps.example.com")
cm.Properties("ServerUserName").SetValue(cm, "username")
cm.Properties("ServerPassword").SetValue(cm, "password")
cm.Properties("ServerPort").SetValue(cm, "port")
cm.Properties("Timeout").SetValue(cm, "0") 'The 0 setting will make it not timeout
cm.Properties("ChunkSize").SetValue(cm, "1000") '1000 kb
cm.Properties("Retries").SetValue(cm, "1")
'create the FTP object that sends the files and pass it the connection created above.
Dim ftp As FtpClientConnection = New FtpClientConnection(cm.AcquireConnection(Nothing))
'Connects to the ftp server
ftp.Connect()
'Build a array of all the file names that is going to be FTP'ed (in this case only one file)
Dim files(0) As String
files(0) = "C:\ local path file"
'ftp the file
ftp.SendFiles(files, "/remote path", True, False) ' the True makes it overwrite existing file and False is saying that it is not transferring ASCII
ftp.Close()
Catch ex As Exception
Dts.TaskResult = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
End Try
...... I finally solved it. I changed the language script task to C# (it works in VB you just have to adapt the code) and used the code from this comment.
note: if the server is on a port other than 21 you have to specify it in the remote path. thanks.
public void Main()
{
// TODO: Add your code here
FtpWebRequest request =
(FtpWebRequest)WebRequest.Create("ftp://ftps.example.com:port/remote/path/file.zip");
request.Credentials = new NetworkCredential("username", "password");
request.EnableSsl = true;
request.Method = WebRequestMethods.Ftp.UploadFile;
using (Stream fileStream = File.OpenRead(#"C:\local\path\file.zip"))
using (Stream ftpStream = request.GetRequestStream())
{
fileStream.CopyTo(ftpStream);
}
}
I want to create a SSIS package which need to download a file automatically and place it in our local.
Note: using process executive task and batch script files only.
In a new SSIS project, create a new package. Navigate to the Parameters tab, where we’ll create a handful of runtime values that will make the DownloadSFTP package more reusable.
pFilename: This is the file name to download from the server. Note
that we can also use wildcards (assuming they are supported by the
target server) – in the example above, we’ll be downloading all files
ending in “.TXT”.
pServerHostKey: This is used to satisfy a security mechanism built
into the WinSCP process. By default, WinSCP will prompt the user to
verify and add to local cache the host key when connecting to an SFTP
server for the first time. Because this will be done in an automated,
non-interactive process, getting that prompt would cause an error in
our script. To prevent this, the script is built to supply the server
host key to avoid the error, and also has the added benefit of
ensuring we’re actually connecting to the correct SFTP server. This
brief article on the WinSCP documentation site describes how to
retrieve the server host key for the target server.
pServerUserPassword: This is marked as sensitive to mask the
password. As part of the script logic, this password will be
decrypted before it is sent to the server.
Create a new script task in the control flow, and add all 7 of the parameters shown above to the list of ReadOnlyVariables.
Using the Main() function (which is created automatically in a new script task), create the Process object and configure a few of the runtime options, including the name of the executable and the download directory.
public void Main()
{
// Create a new Process object to execute WinSCP
Process winscp = new Process();
// Set the executable path and download directory
winscp.StartInfo.FileName = Dts.Variables["$Package::pWinSCPLocation"].Value.ToString();
winscp.StartInfo.WorkingDirectory = Dts.Variables["$Package::pDownloadDir"].Value.ToString();
// Set static execution options (these should not need to change)
winscp.StartInfo.UseShellExecute = false;
winscp.StartInfo.RedirectStandardInput = true;
winscp.StartInfo.RedirectStandardOutput = true;
winscp.StartInfo.CreateNoWindow = true;
// Set session options
string sessionOptionString = "option batch abort" + System.Environment.NewLine + "option confirm off";
The next step is to create the input strings that will make the connection and download the file. At the bottom of this snippet, there are 3 variables that will capture output messages, error messages, and the return value, all of which will be used to log runtime information.
// Build the connect string (<user>:<password>#<hostname>)
string connectString = #"open " + Dts.Variables["$Package::pServerUserName"].Value.ToString()
+ ":"
+ Dts.Variables["$Package::pServerUserPassword"].GetSensitiveValue().ToString()
+ "#"
+ Dts.Variables["$Package::pServerName"].Value.ToString();
// Supplying the host key adds an extra level of security, and avoids getting the prompt to trust the server.
string hostKeyString = Dts.Variables["$Package::pServerHostKey"].Value.ToString();
// If hostkey was specified, include it
if (hostKeyString != null && hostKeyString.Length > 0)
connectString += " -hostkey=\"" + hostKeyString + "\"";
// Build the get command string
string getString = "get " + Dts.Variables["$Package::pFilename"].Value.ToString();
// Create output variables to capture execution info
string outStr = "", errStr = "";
int returnVal = 1;
With all of the options configured, it’s time to invoke WinSCP.com. The try/catch block below will attempt to connect and download the specified file from the server.
// This try/catch block will capture catastrophic failures (such as specifying the wrong path to winscp).
try
{
winscp.Start();
winscp.StandardInput.WriteLine(sessionOptionString);
winscp.StandardInput.WriteLine(connectString);
winscp.StandardInput.WriteLine(getString);
winscp.StandardInput.Close();
winscp.WaitForExit();
// Set the outStr to the output value, obfuscating the password
outStr = winscp.StandardOutput.ReadToEnd().Replace(":" + Dts.Variables["$Package::pServerUserPassword"].GetSensitiveValue().ToString() + "#", ":*******#");
returnVal = winscp.ExitCode;
}
catch (Exception ex)
{
errStr = "An error occurred when attempting to execute winscp.com: " + ex.Message.Replace("'", "\"").Replace("--", " - ");
}
The package is ready to be executed. Assuming everything is configured properly, running the package on the system should download exactly two text files (remember, we used the wildcard “*.txt” to get all text files).
Good morning,
As a returning developer (I've been in management for a long time!), I've been tasked with developing a module for our proprietary financial system (Visual Basic/ SQL Server) that will read large .CSV files containing customer information, convert them into JSON format, then submit them to an external party for processing via that company's API.
The conversion part was easy and I'm almost ready to go with it, but I can't establish connectivity to the external API.
There are two parts to the process: -
Submit login details (obfuscated here) {"username": "MadeUpUser","password": "Y66***uYj6%%YY"} and receive a Bearer Token from the API
Submit JSON format customer info to API endpoint, using Bearer Token, receive confirmation
I've submitted both the login creds and my JSON-format data to the API via Postman and it works perfectly, however, when I try to login via my VB app using the HTTPClient class, I'm getting the follow exception (copied from the Exception class properties) :
**HResult = -2146233088
StackTrace = " at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)" & vbCrLf & " at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)"
InnerException = {"The request was aborted: Could not create SSL/TLS secure channel."}**
It's clearly something I'm doing wrong programmatically or something I'm not doing and any help or advice offered will be greatly appreciated.
The code: (AddLogEntry() is a Sub I created to log events to a text file)
Private Async Function PostAsync(ByVal JSONString As String, EndPoint As String) As Task
Dim APIuri As New Uri(EndPoint)
Try
With ZincClient
.BaseAddress = APIuri
.DefaultRequestHeaders.Accept.Clear()
.DefaultRequestHeaders.Accept.Add(New Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"))
End With
Dim content As New Net.Http.StringContent(JSONString, System.Text.Encoding.UTF8, "application/json")
Dim response As Net.Http.HttpResponseMessage = Await ZincClient.PostAsync(APIuri, content)
Dim result As String = Await response.Content.ReadAsStringAsync()
Catch ex As Exception
AddLogEntry("ERROR! - " & ex.Message)
End Try
Return
End Function
I have some ethernet device which collect data and it's possible to download it via data export interface: HTTP-GET query returns the data in [Content-Type: text/plain Charset: utf-8]
I saw this: How to make an HTTP request from SSIS? - it rather doesn't work for me (C# is a little Chinese for me) and it's about how to fetch this data to variable into SSIS
In your SSIS package add a C# Script Task
Edit the Script Task
At the top with the other using statements add using System.Net;
in Main use the following code snippet to make a GET request (Note: Change "https://somewhere.com/contacts/get" to your actual endpoint.)
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://somewhere.com/contacts/get");
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
using(HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using(Stream stream = response.GetResponseStream())
using(StreamReader reader = new StreamReader(stream))
{
return reader.ReadToEnd();
}
I created an SSIS package where i am loading Multiple Flat Files to my database Table.
As you can see, i have used For Each Loop to Loop through Multiple Files, and inside ForEach loop i've used Data Flow Task which is going to take care of Loading Files to Table and Later i'm Storing the Loaded and Failed files name in Log file and Only Renaming those Loaded files.
Everything is working great, but as you can see, i have used 2 script Task to perform a small operation which can be done using a Single Script Task by Setting the Precedence Constraint to "Complete" instead of "Success" on Data Flow Task. But my problem is, how can i fetch the Data Flow Task Result in Script Task i.e., how to check whether Data Flow Task has Failed or Success in Script Task.
Here is my Script Code where i'm Logging the File Name and Renaming based on the Task Result from Data Flow Task.
public void Main()
{
bool DataFlowTaskResult = false;//setting true or False based on Result from Data Flow Task
if (DataFlowTaskResult)
{
Dts.Events.FireError(0, null, Dts.Variables["User::FileName"].Value.ToString() + " " + "Inserted Succefully", string.Empty, 0);
}
else
{
Dts.Events.FireError(0, null, Dts.Variables["User::FileName"].Value.ToString() + " " + "Inserted Succefully", string.Empty, 0);
}
Dts.TaskResult = (DataFlowTaskResult) ? (int)ScriptResults.Success : (int)ScriptResults.Failure;
}