Copy file found in Script task to folder SSIS - ssis

i am new to coding and know a bit about ssis, i am able to help myself but i am stuck - i wanted to copy the latest .gz file from a folder to my own folder daily with a ssis script task, i was able to identify the lastest file with the script but i am getting stuck where said file needs to copy
here is is my script:
public void Main()
{
// TODO: Add your code here
var directory = new System.IO.DirectoryInfo(Dts.Variables["User::VarFolderPath"].Value.ToString());
System.IO.FileInfo[] files = directory.GetFiles("DailyReport-*.gz");
DateTime lastModified = DateTime.MinValue;
foreach (System.IO.FileInfo file in files)
{
if (file.LastWriteTime > lastModified)
{
lastModified = file.LastWriteTime;
Dts.Variables["User::VarFileName"].Value = file.ToString();
}
}
// MessageBox.Show(Dts.Variables["User::VarFileName"].Value.ToString());
}
}
}
which works in identifying the file which i run within a Foreach loop container, but when i add a file system task, it does not copy the right file
i have variables saved for the copy
User:DestinationFolder as Destination Connection
User:SourceFolder as Source Connection
with copy file as the operation
in my User:SourceFolder variable my value is saved as the exact file name i tested with originally, but it changed the next time the source folder was updated, and now the copy task keeps copying the same file
the file has basically the same name everytime it update, except for the end, for example
DailyReport-20180725050247.gz at 6:00 in the morning
and then
DailyReport-20180725080801.gz at 8:00 the same morning
I want to copy the second one or whichever one updates later in the day
my variables:
SourceFolder - X:\FTP\In\DailyReport-20180725050247.gz
DestinationFolder - X:\MIS_AUTO\Lizl\Test
VarFolderPath - X:\FTP\In
VarFileName -
hope this makes sense

I did a try to re-pro your issue with a Script Task and File System Task.
The only thing that needs to be changed here is what actually passing through User::VarFileName. For a File System Task if you are choosing as variable for the Source Connection -> IsSourcePathVariable then you would need to pass the full file path information.
So, in script task you would need this -
Dts.Variables["User::VarFileName"].Value = file.FullName.ToString();

Related

How to get file name dynamically with time stamp in SSIS

I have a flat file source which has to be loaded daily to a table. I receive the file in the following format "filename_20190509040235.txt"
I used expression to get file name with date, how can I get the time stamp?
The time stamp is different in each date. The file get generated in the afternoon and the package is planning to run every night.
Assuming you want to load files based on a certain time defined by the timestamp on the file name, an overview of this process is below. As noted, files with a timestamp within the 12 hours prior to the package execution are returned, and you may need to adjust this to your specific needs. This also uses the same file name/timestamp format as indicated in your question, i.e. filename_20190509040235.txt.
Create an object and string variable in SSIS. On the Flat File connection manager, add the string variable as the expression for the connection string. This can be done from the Properties window (press F4) on the connection manager, going to the Expressions field, pressing the ellipsis next to it, choosing the ConnectionString property on the next window and selecting the recently created string variable as the expression for this.
Add a Script Task on the Control Flow. Add the object variable in the ReadWriteVariables field. If the directory holding the files is stored in an SSIS variable add this variable in the in the ReadOnlyVariables field.
Example code for this is below. Your post stated the files are generated in the afternoon with the package running nightly. Not being sure of the exact requirements, this just returns files with a timestamp within 12 hours of the current time. You can change this by adjusting the parameter of DateTime.Now.AddHours, which currently subtracts 12 hours from the current time (i.e. adds -12). This will go in the Main method of the Script Task. Be sure to add the references noted below too.
Add a Foreach Loop after the Script Task and for the enumerator type select Foreach From Variable Enumerator. On the Variable field of the Collection tab, choose the object variable that was populated in the Script Task. Next on the Variable Mappings pane select the string variable created earlier (set as the connection string for the Flat File connection manager) at index 0.
Inside the Foreach Loop add a Data Flow Task. Within the Data Flow Task, create a Flat File Source component using the Flat File connection manager and add the appropriate destination component. Connect these two and ensure that the columns are mapped correctly on the destination.
Script Task:
using System.IO;
using System.Collections.Generic;
//get source folder from SSIS string variable (if held there)
string sourceDirectory = Dts.Variables["User::SourceDirectory"].Value.ToString();
DirectoryInfo di = new DirectoryInfo(sourceDirectory);
List<string> recentFiles = new List<string>();
foreach (FileInfo fi in di.EnumerateFiles())
{
//use Name to only get file name, not path
string fileName = fi.Name;
string hour = fileName.Substring(17, 2);
string minute = fileName.Substring(19, 2);
string second = fileName.Substring(21, 2);
string year = fileName.Substring(9, 4);
string month = fileName.Substring(13, 2);
string day = fileName.Substring(15, 2);
string dateOnFile = month + "/" + day + "/" + year + " "
+ hour + ":" + minute + ":" + second;
DateTime fileDate;
//prevent errors in case of bad dates
if (DateTime.TryParse(dateOnFile, out fileDate))
{
//files from last 12 hours
if (fileDate >= DateTime.Now.AddHours(-12))
{
//FullName for file path
recentFiles.Add(fi.FullName);
}
}
}
//populate SSIS object variable with file list
Dts.Variables["User::ObjectVariable"].Value = recentFiles;

Create a txt file with SSIS showing File Movements

I need to create a .txt file every time and SSIS job runs and in that file I need to put the name of the files that I am transmitting. I have a foreach container but I am not sure on how to create the file and then write to it as each file is being moved. Can someone please steer me in the right direction? Thanks in Advance.
You can follow below steps:
Create a dummy empty text file
Copy the dummy file to new name dummyfile_10252016 (using File System Task - need to handle it through variable to make the name dynamic)
Create variable to keep the dummyFilePath
Process your required files in for each loop and store the filename in a variable
Once your file is processed, write the name to dummyfile_10252016
Script task:
string filename = Dts.Variables["User::filename"].Value.ToString();
string path = Dts.Variables["User::dummyFilePath"].Value.ToString();;
System.IO.File.AppendAllText(path, filename );

SSIS Package - Looping through folder to check if files exists

Can anyone help:
Required: SSIS Package to loop through a folder (containing 100 files) and check whether required files (which are 5/6) are present in that folder.
Does anyone already has code for this - where we are checking for multiple files existence in the destination folder
Regards
Add a Foreach loop container to your Control Flow
Double click it and select Collection. On Enumerator, select Foreach
File Enumerator
Select your folder and the type of file
Select the return type when a file is found. The options are the
whole filename including extension and path, the name and extension
or simply the name of the file found
Select the checkbox if you want the subfolders
Click on the variables option on the left and then new variable or
select an existing variable.
At this point you have each file name on the folder. To prove it, add a script component, double click it, and your variable on the read Only Variable and click on Edit Script. Make your Main like this:
public void Main()
{
System.Windows.Forms.MessageBox.Show(Dts.Variables["FileName"].Value.ToString());
Dts.TaskResult = (int)ScriptResults.Success;
}
now, the comparison you can do several ways. I dont know where do you have the "required files" list, but assuming it is on a database, you can add a data flow task and inside of it send the filename to the DB to do the comparisson.

SSIS:Moving .csv files from share folder to oledb destination on daily basis and sending email notification

I am new to SSIS i'm working on a critical deadline it would be great if someone could help me on this.
I have a share folder in server location //share/source/files where on daily basis one file will get loaded and on a monthly basis one more file gets loaded both the files are of same extension .csv
Can anyone help me in moving the files Say A.csv and B.csv to corresponding tables and more important is file name on day1 will be A 2011-09-10.csv and on day2 in source the file will be A 2011-09-11.csv..This files has to be moved to table A and file B.csv has to be moved the corresponding destination table table b ,once after moving the files this file has to be moved to archive folder and also we need to send users that tabl-A got loaded with 1000 rows and succesfull similarly table -b load was sucesfull along with date and time .
Note:Source Files will be automatically updated in the folder everyday at 5am in the morning.
First, create a variable that will hold the file path name.
Secondly, create a script task that checks to see if the file is available.
The script task will be as follows:
public void Main()
{
string FileName = String.Format("{0}.csv", DateTime.Now.ToString("yyyy-MM-dd"));
if (System.IO.File.Exists("\\Shared\\Path\\" + FileName))
{
Dts.Variables["FileName"].Value = FileName;
Dts.TaskResult = (int)ScriptResults.Success;
}
else
{
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
After the script task, create a Data Flow Task.
Create a connection in the Connection Manager at the bottom of the page which will point to the flat file. In the Flat File Connection Manager Editor popup, set the File name to a file you'd like to upload (this will be updated dynamically, so its actual value isn't relevant). In the properties of the new Connection, open the Expressions popup. Select ConnectionString property, and set the expression to point to the path and the FileName variable : "\\Shared\\Path\\" + #[User::FileName].
Create a Flat File Source, and use the connection we just created as the Connection for the flat file.
Create a destination data flow item, and point it to the database you'd like to insert data into.
From here, create a SQL Server job that runs at the time you'd like it to run. This job should execute the package you have just created.

Populate Derived Column with File's Date Modified

I'm a wannabe to .Net and SQL and am working on an SSIS package that is pulling data from flat files and inputting it into a SQL table. The part that I need assistance on is getting the Date Modified of the files and populating a derived column I created in that table with it. I have created the following variables: FileDate of type DateTime, FilePath of String, and SourceFolder of String for the path of the files. I was thinking that the DateModified could be populated in the derived column w/i the DataFlow, using a Script Component? Can someone please advise on if I'm on the right track? I appreciate any help. Thanks.
A Derived Column Transformation can only work with Integration Services Expressions. A script task would allow you to access the .net libraries and you would want to use the method that #wil kindly posted or go with the static methods in System.IO.File
However, I don't believe you would want to do this in a Data Flow Task. SSIS would have to evaluate that code for every row that flows through from the file. On a semi-related note, you cannot write to a variable until the ... event is fired to signal the data flow has completed (I think it's OnPostExecute but don't quote me) so you wouldn't be able to use said variable in a downstream derived column at any rate. You would of course, just modify the data pipeline to inject the file modified date at that point.
What would be preferable and perhaps your intent is to use a Script Task prior to the Data Flow task to assign the value to your FileDate variable. Inside your Data Flow, then use a Derived Column to add the #FileDate variable into the pipeline.
// This code is approximate. It should work but it's only been parsed by my brain
//
// Assumption:
// SourceFolder looks like a path x:\foo\bar
// FilePath looks like a file name blee.txt
// SourceFolder [\] FilePath is a file that the account running the package can access
//
// Assign the last mod date to FileDate variable based on file system datetime
// Original code, minor flaws
// Dts.Variables["FileDate"].Value = File.GetLastWriteTime(System.IO.Path.Combine(Dts.Variables["SourceFolder"].Value,Dts.Variables["FilePath"].Value));
Dts.Variables["FileDate"].Value = System.IO.File.GetLastWriteTime(System.IO.Path.Combine(Dts.Variables["SourceFolder"].Value.ToString(), Dts.Variables["FilePath"].Value.ToString()));
Edit
I believe something is amiss with either your code or your variables. Do your values approximately line up with mine for FilePath and SourceFolder? Variables are case sensitive but I don't believe that to be your issue given the error you report.
This is the full script task and you can see by the screenshot below, the design-time value for FileDate is 2011-10-05 09:06 The run-time value (locals) is 2011-09-23 09:26:59 which is the last mod date for the c:\tmp\witadmin.txt file
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
namespace ST_f74347eb0ac14a048e9ba69c1b1e7513.csproj
{
[System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
public void Main()
{
Dts.Variables["FileDate"].Value = System.IO.File.GetLastWriteTime(System.IO.Path.Combine(Dts.Variables["SourceFolder"].Value.ToString(), Dts.Variables["FilePath"].Value.ToString()));
Dts.TaskResult = (int)ScriptResults.Success;
}
}
}
C:\tmp>dir \tmp\witadmin.txt
Volume in drive C is Local Disk
Volume Serial Number is 3F21-8G22
Directory of C:\tmp
09/23/2011 09:26 AM 670,303 witadmin.txt