Download SFTP file using SSIS package - ssis

I want to create a SSIS package which need to download a file automatically and place it in our local.
Note: using process executive task and batch script files only.

In a new SSIS project, create a new package. Navigate to the Parameters tab, where we’ll create a handful of runtime values that will make the DownloadSFTP package more reusable.
pFilename: This is the file name to download from the server. Note
that we can also use wildcards (assuming they are supported by the
target server) – in the example above, we’ll be downloading all files
ending in “.TXT”.
pServerHostKey: This is used to satisfy a security mechanism built
into the WinSCP process. By default, WinSCP will prompt the user to
verify and add to local cache the host key when connecting to an SFTP
server for the first time. Because this will be done in an automated,
non-interactive process, getting that prompt would cause an error in
our script. To prevent this, the script is built to supply the server
host key to avoid the error, and also has the added benefit of
ensuring we’re actually connecting to the correct SFTP server. This
brief article on the WinSCP documentation site describes how to
retrieve the server host key for the target server.
pServerUserPassword: This is marked as sensitive to mask the
password. As part of the script logic, this password will be
decrypted before it is sent to the server.
Create a new script task in the control flow, and add all 7 of the parameters shown above to the list of ReadOnlyVariables.
Using the Main() function (which is created automatically in a new script task), create the Process object and configure a few of the runtime options, including the name of the executable and the download directory.
public void Main()
{
// Create a new Process object to execute WinSCP
Process winscp = new Process();
// Set the executable path and download directory
winscp.StartInfo.FileName = Dts.Variables["$Package::pWinSCPLocation"].Value.ToString();
winscp.StartInfo.WorkingDirectory = Dts.Variables["$Package::pDownloadDir"].Value.ToString();
// Set static execution options (these should not need to change)
winscp.StartInfo.UseShellExecute = false;
winscp.StartInfo.RedirectStandardInput = true;
winscp.StartInfo.RedirectStandardOutput = true;
winscp.StartInfo.CreateNoWindow = true;
// Set session options
string sessionOptionString = "option batch abort" + System.Environment.NewLine + "option confirm off";
The next step is to create the input strings that will make the connection and download the file. At the bottom of this snippet, there are 3 variables that will capture output messages, error messages, and the return value, all of which will be used to log runtime information.
// Build the connect string (<user>:<password>#<hostname>)
string connectString = #"open " + Dts.Variables["$Package::pServerUserName"].Value.ToString()
+ ":"
+ Dts.Variables["$Package::pServerUserPassword"].GetSensitiveValue().ToString()
+ "#"
+ Dts.Variables["$Package::pServerName"].Value.ToString();
// Supplying the host key adds an extra level of security, and avoids getting the prompt to trust the server.
string hostKeyString = Dts.Variables["$Package::pServerHostKey"].Value.ToString();
// If hostkey was specified, include it
if (hostKeyString != null && hostKeyString.Length > 0)
connectString += " -hostkey=\"" + hostKeyString + "\"";
// Build the get command string
string getString = "get " + Dts.Variables["$Package::pFilename"].Value.ToString();
// Create output variables to capture execution info
string outStr = "", errStr = "";
int returnVal = 1;
With all of the options configured, it’s time to invoke WinSCP.com. The try/catch block below will attempt to connect and download the specified file from the server.
// This try/catch block will capture catastrophic failures (such as specifying the wrong path to winscp).
try
{
winscp.Start();
winscp.StandardInput.WriteLine(sessionOptionString);
winscp.StandardInput.WriteLine(connectString);
winscp.StandardInput.WriteLine(getString);
winscp.StandardInput.Close();
winscp.WaitForExit();
// Set the outStr to the output value, obfuscating the password
outStr = winscp.StandardOutput.ReadToEnd().Replace(":" + Dts.Variables["$Package::pServerUserPassword"].GetSensitiveValue().ToString() + "#", ":*******#");
returnVal = winscp.ExitCode;
}
catch (Exception ex)
{
errStr = "An error occurred when attempting to execute winscp.com: " + ex.Message.Replace("'", "\"").Replace("--", " - ");
}
The package is ready to be executed. Assuming everything is configured properly, running the package on the system should download exactly two text files (remember, we used the wildcard “*.txt” to get all text files).

Related

SSIS Task - FTP Conecction [duplicate]

This question already has answers here:
Upload file to FTP site using VB.NET
(5 answers)
Closed 1 year ago.
All,
SSIS - trying to make an ftp connection to upload files to an ftp server using a script task in visual basic programming language, I have run into a problem. I have not found something similar in my searches so I would appreciate your help.
[Connection manager "FTP"] Error: An error occurred in the requested FTP operation. Detailed error description: 220
550 SSL/TLS required on the control channel
CODE:
Public Sub Main()
'
' Add your code here
'
Try
Dim cm As ConnectionManager
cm = Dts.Connections("FTP")
'Set the properties like username & password
cm.Properties("ServerName").SetValue(cm, "ftps.example.com")
cm.Properties("ServerUserName").SetValue(cm, "username")
cm.Properties("ServerPassword").SetValue(cm, "password")
cm.Properties("ServerPort").SetValue(cm, "port")
cm.Properties("Timeout").SetValue(cm, "0") 'The 0 setting will make it not timeout
cm.Properties("ChunkSize").SetValue(cm, "1000") '1000 kb
cm.Properties("Retries").SetValue(cm, "1")
'create the FTP object that sends the files and pass it the connection created above.
Dim ftp As FtpClientConnection = New FtpClientConnection(cm.AcquireConnection(Nothing))
'Connects to the ftp server
ftp.Connect()
'Build a array of all the file names that is going to be FTP'ed (in this case only one file)
Dim files(0) As String
files(0) = "C:\ local path file"
'ftp the file
ftp.SendFiles(files, "/remote path", True, False) ' the True makes it overwrite existing file and False is saying that it is not transferring ASCII
ftp.Close()
Catch ex As Exception
Dts.TaskResult = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
End Try
...... I finally solved it. I changed the language script task to C# (it works in VB you just have to adapt the code) and used the code from this comment.
note: if the server is on a port other than 21 you have to specify it in the remote path. thanks.
public void Main()
{
// TODO: Add your code here
FtpWebRequest request =
(FtpWebRequest)WebRequest.Create("ftp://ftps.example.com:port/remote/path/file.zip");
request.Credentials = new NetworkCredential("username", "password");
request.EnableSsl = true;
request.Method = WebRequestMethods.Ftp.UploadFile;
using (Stream fileStream = File.OpenRead(#"C:\local\path\file.zip"))
using (Stream ftpStream = request.GetRequestStream())
{
fileStream.CopyTo(ftpStream);
}
}

Return the Task Result from Data Flow Task to Script Component

I created an SSIS package where i am loading Multiple Flat Files to my database Table.
As you can see, i have used For Each Loop to Loop through Multiple Files, and inside ForEach loop i've used Data Flow Task which is going to take care of Loading Files to Table and Later i'm Storing the Loaded and Failed files name in Log file and Only Renaming those Loaded files.
Everything is working great, but as you can see, i have used 2 script Task to perform a small operation which can be done using a Single Script Task by Setting the Precedence Constraint to "Complete" instead of "Success" on Data Flow Task. But my problem is, how can i fetch the Data Flow Task Result in Script Task i.e., how to check whether Data Flow Task has Failed or Success in Script Task.
Here is my Script Code where i'm Logging the File Name and Renaming based on the Task Result from Data Flow Task.
public void Main()
{
bool DataFlowTaskResult = false;//setting true or False based on Result from Data Flow Task
if (DataFlowTaskResult)
{
Dts.Events.FireError(0, null, Dts.Variables["User::FileName"].Value.ToString() + " " + "Inserted Succefully", string.Empty, 0);
}
else
{
Dts.Events.FireError(0, null, Dts.Variables["User::FileName"].Value.ToString() + " " + "Inserted Succefully", string.Empty, 0);
}
Dts.TaskResult = (DataFlowTaskResult) ? (int)ScriptResults.Success : (int)ScriptResults.Failure;
}

How to Dynamically change sync function in sync gateway couchbase

Is there a way I can dynamically change sync function for eg. Lets ssy my documents have a field ID and I want to get documents belonging to a particular ID so ID is my variable here. eg. below is a sync function for ID=4
"sync":
function (doc) {
if(doc.ID==4){
channel (doc.channels);
}
else{
throw({forbidden: "Missing required properties"});
}
},
Now This will only work for ID=4. How Can I make my sync function dynamic. Is there a way I can supply arguments to my sync function?
EDIT 1 Added Use Case
Ok so my use case is like this.I have an app in which when a user logs in I need to get user specific data from CouchBase Server to CouchBase lite. In My CouchBase Server I have 20000 documents and for each user there are 5 documents so I have (20000/5) 4000 users. So When a user logs in to my app my CouchBase server should send only 5 documents which are related to that user and not all 20000 documents
EDIT 2
This is how I have implemented the replication
private URL createSyncURL(boolean isEncrypted){
URL syncURL = null;
String host = "http://172.16.25.108";
String port = "4986";
String dbName = "sync_gateway";
try {
//syncURL = new URL("http://127.0.0.1 :4986/sync_gateway");
syncURL = new URL(host + ":" + port + "/" + dbName);
} catch (Exception me) {
me.printStackTrace();
}
Log.d(syncURL.toString(),"URL");
return syncURL;
}
private void startReplications() throws CouchbaseLiteException {
Log.d(TAG, "");
Replication pull = database.createPullReplication(this.createSyncURL(false));
Replication push = database.createPushReplication(this.createSyncURL(false));
Authenticator authenticator = AuthenticatorFactory.createBasicAuthenticator("an", "1234");
pull.setAuthenticator(authenticator);
//push.setAuthenticator(authenticator);
List<String> channels1 = new ArrayList<String>();
channels1.add("u1");
pull.setChannels(channels1);
pull.setContinuous(true);
// push.setContinuous(true);
pull.start();
//push.start();
if(!push.isRunning()){
Log.d(TAG, "MyBad");
}
/*if(!push.isRunning()) {
Log.d(TAG, "Replication is not running due to " +push.getLastError().getMessage());
Log.d(TAG, "Replication is not running due to " +push.getLastError().getCause());
Log.d(TAG, "Replication is not running due to " +push.getLastError().getStackTrace());
Log.d(TAG, "Replication is not running due to " +push.getLastError().toString());
}*/
}
The easiest way to achieve this is to qualify each user to one channel, named like the user, and to give to a document the channel (= user) names of all users for whom this document is relevant (maybe just one channel name per document, but that's completely up to you).
So with the standard sync function (without any if condition), if your config.json contains
"users": {
"u1": {
"admin_channels": ["u1"],
"password": "abracadabra"
},
"u2": {
"admin_channels": ["u2"],
"password": "simsalabim"
...
and you have documents having
{"channels": "u1",...
{"channels": "u2",...
{"channels": ["u1", "u2"],...
then the first will be transferred to u1, the second to u2, and the third to both of them. You don't need to make your channel names identical to the user name, but for this scenario it's the easiest way to go.
The programmatic assignment of channels to users can be done via the Sync Gateway Admin REST API, see http://developer.couchbase.com/documentation/mobile/1.2/develop/references/sync-gateway/admin-rest-api/user-admin/post-user/index.html. (Note that the Admin API should run on a port that is opened only to the local server where CB runs, not to the public.)

Store and update JSON Data on a Server

My web-application should be able to store and update (also load) JSON data on a Server.
However, the data may contain some big arrays where every time they are saved only a new entry was appended.
My solution:
send updates to the server with a key-path within the json data.
Currently I'm sending the data with an xmlhttprequest by jquery, like this
/**
* Asynchronously writes a file on the server (via PHP-script).
* #param {String} file complete filename (path/to/file.ext)
* #param content content that should be written. may be a js object.
* #param {Array} updatePath (optional), json only. not the entire file is written,
* but the given path within the object is updated. by default the path is supposed to contain an array and the
* content is appended to it.
* #param {String} key (optional) in combination with updatePath. if a key is provided, then the content is written
* to a field named as this parameters content at the data located at the updatePath from the old content.
*
* #returns {Promise}
*/
io.write = function (file, content, updatePath, key) {
if (utils.isObject(content)) content = JSON.stringify(content, null, "\t");
file = io.parsePath(file);
var data = {f: file, t: content};
if (typeof updatePath !== "undefined") {
if (Array.isArray(updatePath)) updatePath = updatePath.join('.');
data.a = updatePath;
if (typeof key !== "undefined") data.k = key;
}
return new Promise(function (resolve, reject) {
$.ajax({
type: 'POST',
url: io.url.write,
data: data,
success: function (data) {
data = data.split("\n");
if (data[0] == "ok") resolve(data[1]);
else reject(new Error((data[0] == "error" ? "PHP error:\n" : "") + data.slice(1).join("\n")));
},
cache: false,
error: function (j, t, e) {
reject(e);
//throw new Error("Error writing file '" + file + "'\n" + JSON.stringify(j) + " " + e);
}
});
});
};
On the Server, a php script manages the rest like this:
recieves the data and checks if its valid
check if the given file path is writable
if the file exists and is .json
read it and decode the json
return an error on invalid json
if there is no update path given
just write the data
if there is an update path given
return an error if the update path in the JSON data can't be traversed (or file didn't exist)
update the data at update-path
write the pretty-printed json to file
However I'm not perfectly happy and problems kept coming for the last weeks.
My Questions
Generally: How would you approach this problem? alternative suggestions, databases? any libraries that could help?
Note: I would prefer solutions, that just use php or some standart apache stuff.
One problem was, that sometimes, multiple writes on the same file were triggered. To avoid this I used the Promises (wrapped it because I read jquerys deferred stuff isnt Promise/A compliant) client side, but I dont feel 100% sure it is working. Is there a (file) lock in php that works across multiple requests?
Every now and then the JSON files break and its not clear to me how to reproduce the problem. At the time it breaks, I don't have a history of what happened. Any general debugging strategies with a client/server saving/loading process like this?
I wrote a comet enable web server that does diffs on updates of json data structures. For the exactly same reason. The server keeps a few version of a json document and serves client with different version of the json document with the update they need to get to the most reason version of the json data.
Maybe you could reuse some of my code, written in C++ and CoffeeScript: https://github.com/TorstenRobitzki/Sioux
If you have concurrent write accesses to your data structure, are your sure, that who ever writes to the file has the right version of the file in mind when reading the file?

SSIS script task error retrieving variable values

In a Script Task, I am trying to retrieve a file from a networked location and FTP that file to an offsite location
In SSIS I created the FTP Connection and tested that it is setup and works
Created three variables
variable 1. FullPathName = \ftpservercsc\\\Filename.txt
variable 2 FTPFilePath = \ftpservercsc\\\
variable 3 FTPFileName = Filename.txt
Created a Script Task and added the vb code as such ...
'Get instance of the connection manager.
Dim cm As ConnectionManager = Dts.Connections("FTP Connection Manager")
Dim remotePath As String = Dts.Variables("FTPFilePath").Value.ToString
'create the FTP object that sends the files and pass it the connection
'created above.
Dim ftp As FtpClientConnection = New FtpClientConnection
(cm.AcquireConnection(Nothing))
'Connect to the ftp server
ftp.Connect()
'Set the path on the FTP server where dropping files
'ftp.SetWorkingDirectory("/Prequalify") 'set the remote directory
Dim files(0) As String
files(0) = Dts.Variables("FTPFileName").Value.ToString 'eg. File1.trg
'Send File
ftp.SendFiles(files, remotePath, True, True)
' Close the ftp connection
ftp.Close()
'Dts.Events.FireInformation(0, context, "File " + fileToGet
' + " retrieved successfully.", Nothing, Nothing, True)
Dts.TaskResult = Dts.Results.Success
Error: The element cannot be found in a collection. This error happens when you try to retrieve an element from a collection on a container during execution of the package and the element is not there.
So I have commented out and found the error is generating on retrieving the variable value but I do not know what is incorrect here
Dim remotePath As String = Dts.Variables("FTPFilePath").Value.ToString
I have tried multiple variable retrievals and all get the same error. Anyone see anything wrong?
Two things:
Make sure you config the Script Task to have Read Access to the variable. To do this right-click on the Script Task and select Edit. Click the ... under ReadOnlyVariables.
Fully qualify your variables such as Dts.Variables["User::RemotePath"].Value