The requirement is to execute SSIS package, when a file is arrived at a folder,i do not want to start the package manually .
It is not sure about the file arrival timing ,also the files can arrive multiple times .When ever the files arrived this has to load into a table.I think, some solution like file watcher task ,still expect to start the package
The way I have done this in the past is with an infinite loop package called from SQL Server Agent, for example;
This is my infinite loop package:
Set 3 Variables:
IsFileExists - Boolean - 0
FolderLocation - String - C:\Where the file is to be put in\
IsFileExists Boolean - 0
For the For Loop container:
Set the IsFileExists variables as above.
Setup a C# script task with the ReadOnlyVariable as User::FolderLocation and have the following:
public void Main()
{
int fileCount = 0;
string[] FilesToProcess;
while (fileCount == 0)
{
try
{
System.Threading.Thread.Sleep(10000);
FilesToProcess = System.IO.Directory.GetFiles(Dts.Variables["FolderLocation"].Value.ToString(), "*.txt");
fileCount = FilesToProcess.Length;
if (fileCount != 0)
{
for (int i = 0; i < fileCount; i++)
{
try
{
System.IO.FileStream fs = new System.IO.FileStream(FilesToProcess[i], System.IO.FileMode.Open);
fs.Close();
}
catch (System.IO.IOException ex)
{
fileCount = 0;
continue;
}
}
}
}
catch (Exception ex)
{
throw ex;
}
}
// TODO: Add your code here
Dts.TaskResult = (int)ScriptResults.Success;
}
}
}
What this will do is essentially keep an eye on the folder location for a .txt file, if the file is not there it will sleep for 10 seconds (you can increase this if you want). If the file does exist it will complete and the package will then execute the load package. However it will continue to run, so the next time a file is dropped in it will execute the load package again.
Make sure to run this forever loop package as a sql server agent job so it will run all the time, we have a similar package running and it has never caused any problems.
Also, make sure your input package moves/archives the file away from the drop folder location.
As others have already suggested, using either WMI task or an infinite loop are two options to achieve this, but IMO SSIS is resource intensive. If you let a package constantly run in the background, it could eat up a lot of memory, cpu and cause performance issues with other packages depending on how many other packages you've running. So other option you may want to consider is schedule an Agent job every 5 minutes or 10 minutes or something and call your package in the job. Configure the package to continue only when a file is there or quit otherwise.
You can create a Windows service that uses WMI to detect file arrival and launch packages. Details on how to are located here: http://msbimentalist.wordpress.com/2012/04/27/trigger-ssis-package-when-files-available-in-a-folder-part2/?relatedposts_exclude=330
What about the SSIS File Watcher Task?
Related
I am using Laravel 8 and I want to update billions of Products Prices. I add this code its work fine but it is not efficient it add load on server.
try {
$priceCsvs = PriceCsv::whereStatus(PriceCsv::PENDING)->get();
foreach ($priceCsvs as $price) {
dump($price->name." is started");
$csvData = fopen($price->file_url, 'r');
$firstline = true;
while (($data = fgetcsv($csvData, 555, ',')) !== false) {
if (!$firstline && !empty($data)) {
dump($data);
}
$firstline = false;
}
fclose($csvData);
dump($price->name." is End");
}
} catch (\Exception $ex) {
dump($ex->getMessage());
}
Point is: Is there any way to add sleep after 1000 entry from CSV file or any other efficient way.
If you are looking to schedule jobs then take a look at
Task Scheduling
Basically define some console (artisan) commands and then schedule them in the Console Kernel file. The scheduler is run with artisan and the recommendation is to run the command every minute (define a cron job). In the event that the kernel schedule item definition lines up with the minute that the 'php artisan schedule:work' command runs then your jobs will execute.
I'm trying to iteratively (part of automation):
Create backup of the projects in solution (physical files on the filesystem)
Using Microsoft.Build programmatically load and change projects inside of the solution (refernces, includes, some other properties)
Build it with console call of msbuild
Restore projects (physically overriding patched versions from backups)
This approach works well for first iteration, but for second it appears that it does not load restored projects and trying to work with values that I patched on the first iteration. It looks like projects are cached: inside of the csproj files I see correct values, but on the code I see previously patched values.
My best guess is that Microsoft.Build is caching solution/projects in the context of the current process.
Here is code that is responsible to load project and call method to update project information:
private static void ForEachProject(string slnPath, Func<ProjectRootElement> patchProject)
{
SolutionFile slnFile = SolutionFile.Parse(slnPath);
var filtredProjects = slnFile
.ProjectsInOrder
.Where(prj => prj.ProjectType == SolutionProjectType.KnownToBeMSBuildFormat);
foreach (ProjectInSolution projectInfo in filtredProjects)
{
try
{
ProjectRootElement project = ProjectRootElement.Open(projectInfo.AbsolutePath);
patchProject(project);
project.Save();
}
catch (InvalidProjectFileException ex)
{
Console.WriteLine("Failed to patch project '{0}' with error: {1}", projectInfo.AbsolutePath, ex);
}
}
}
There is Reload method for the ProjectRootElement that migh be called before iteraction with content of the project.
It will enforce Microsoft.Build to read latest information from the file.
Code that is working for me:
private static void ForEachProject(string slnPath, Func<ProjectRootElement> patchProject)
{
SolutionFile slnFile = SolutionFile.Parse(slnPath);
var filtredProjects = slnFile
.ProjectsInOrder
.Where(prj => prj.ProjectType == SolutionProjectType.KnownToBeMSBuildFormat);
foreach (ProjectInSolution projectInfo in filtredProjects)
{
try
{
ProjectRootElement project = ProjectRootElement.Open(projectInfo.AbsolutePath);
project.Reload(false); // Ignore cached state, read actual from the file
patchProject(project);
project.Save();
}
catch (InvalidProjectFileException ex)
{
Console.WriteLine("Failed to patch project '{0}' with error: {1}", projectInfo.AbsolutePath, ex);
}
}
}
Note: It better to use custom properties inside of the project and provide it for each msbuild call instead of physical project patching. Please consider it as better solution and use it if possible.
I have a loop which loops through connections. Each loop connects to a different connection and then loads tables and such. In the 2nd step is where it connects and then loads data into our DWH. Sometimes the connection is down for what ever reason and will fail at this step. I need the package to keep going on this connection fail.
I have read many things on the propogate set to false and this still does not work. As you can see in my screen shot, i have the "Load ..." onError event handler propogate set to false, and the Sequence container onError propogate set to false.
I have also tried setting the sequence container max errors to 0 so that section completes, and on the onError of the "Load ..." to set a flag in a variable to continue going if the connection completes, or stop there if the connection fails.
I have done this in the past and just set the overall package complete status to success on completion, but this will not catch other errors that may occur in this loop that I will need to catch / fail the package on.
Any help here would be appreciated.
Doing more research on connections failing, I found a script by Jamie Thomson: Verify a connection before using it [SSIS].
I modified it a bit to my own usage:
Only used a single connection instead of looping through all of them.
I set the task result to always succeed.
Instead of FireError I did a FireWarning.
I created a variable (connFail) to set to a 0 or a 1 depending on if the connection failed or not.
I places this script task before my table load to catch any failed connections before the task was executed. These modifications allowed me to fire an e-mail alert if the connection failed (connfail = 1), or continue on the package if the connection was successful (`connFail = 0').
Full script I used is below:
bool failure = false;
bool fireAgain = true;
try
{
Dts.Connections["Connection"].AcquireConnection(null);
Dts.Events.FireInformation(1, ""
, String.Format("Connection aquired successfully on '{0}'", Dts.Connections["Connection"].Name)
, "", 0, ref fireAgain);
}
catch (Exception e)
{
Dts.Events.FireWarning(-1, ""
, String.Format("Failed to aquire connection to '{0}'. Error Message='{1}'",Dts.Connections["Connection"].Name, e.Message)
, "", 0);
failure = true;
}
if (failure)
{
Dts.TaskResult = (int)ScriptResults.Success;
Dts.Variables["connFail"].Value = 1;
}
else
{
Dts.TaskResult = (int)ScriptResults.Success;
Dts.Variables["connFail"].Value = 0;
}
In SSIS
In a folder there are many flat files and by using for each loop container we are processing it one by one. If any new file is placed in the folder and it is still in copying mode. Then, We should not take it for continue process. We should process Only fully copied file alone to our next process.
How can we achieve this? Please give your suggestions.
Best way I have done this in the past is to use a C# Script Task and try to open the file - If the file is still being copied you will get an error (which you Catch). Then you can set a boolean variable to conditionally process the file if the Open worked.
EG:
Boolean b = true;
FileStream f;
try
{
f = new FileStream("C:\\Test\\Test.txt", FileMode.Open, FileAccess.ReadWrite, FileShare.None);
}
catch (IOException e)
{
if (e.Message == "hello")
{
b = false;
}
}
What I'm trying to do in SSIS is have a WMI Event Watcher Task which watches a folder for a file to be created, then does something with it. The primary part is the "watching the folder for file creation".
I have a network folder (full path): \\srvblah10\main\child\target\
All the sites I've gone to has this as an example:
SELECT * FROM __InstanceCreationEvent WITHIN 10
WHERE TargetInstance ISA "CIM_DirectoryContainsFile"
AND TargetInstance.GroupComponent = "Win32_Directory.Name=\"d:\\\\NewFiles\""
Since the folder is a network folder, I can't provide the physical disk letter. So is there a way to use a similar WQL query but for network folder paths as opposed to physical folder paths?
You have to map the drive with a dos command:
net use s: \srvblah10\main\child\target\ /user dotnetN00b Pa$$word
then you can the WMI Event Watcher Task to watch it.
I was trying to do this for awhile, and finally gave up on trying to use the SSIS WMI Event Watcher task, and just wrote the equivalent in a Script task. The issue that was the challenge was getting the WMI Event Watcher to make the remote connection with specific user credentials that I wanted to obtain from a configuration section (not hard code into the package).
The second issue that was going to make not using a script difficult was simply translating the network share, into the local path name on the server, which the Event Watcher requires. You'll see from the scrip below, everything is accomplished with a minimal of effort.
Just an additional heads up, make sure to include the DTS.Variables the script uses in the ReadOnlyVariables (as normal). The code below requires three DTS variables, for example if you are trying to watch for files being dropped in the following location \copernicus\dropoff\SAP\Import, then you would set the variables as shown below:
User::ServerName - the hostname of the server where the share lives
(copernicus)
User::ShareName - the name of the network share
(dropoff)
User::ImportPath - the directory path of the directory to
watch for new files in (/SAP/Import)
public void Main()
{
string localPath = "";
try
{
ConnectionOptions connection = new ConnectionOptions();
connection.Username = "<valid username here>";
connection.Password = "<password here>";
connection.Authority = "ntlmdomain:<your domain name here>";
ManagementScope scope = new ManagementScope(#"\\" + Dts.Variables["User::FileServerName"].Value.ToString() + #"\root\CIMV2", connection);
scope.Connect();
/// Retrieve the local path of the network share from the file server
///
string queryStr = string.Format("SELECT Path FROM Win32_Share WHERE Name='{0}'", Dts.Variables["User::ShareName"].Value.ToString());
ManagementObjectSearcher mosLocalPath = new ManagementObjectSearcher(scope, new ObjectQuery(queryStr));
foreach (ManagementObject elements in mosLocalPath.Get())
{
localPath = elements["Path"].ToString();
}
queryStr = string.Format(
"SELECT * FROM __InstanceCreationEvent WITHIN 10 WHERE Targetinstance ISA 'CIM_DirectoryContainsFile' and TargetInstance.GroupComponent=\"Win32_Directory.Name='{0}{1}'\"",
localPath.Replace(#"\", #"\\"),
Dts.Variables["User::ImportPath"].Value.ToString().Replace(#"\", #"\\")); // query requires each seperator to be a double back slash
ManagementEventWatcher watcher = new ManagementEventWatcher(scope, new WqlEventQuery(queryStr));
ManagementBaseObject eventObj = watcher.WaitForNextEvent();
// Cancel the event subscription
watcher.Stop();
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (ManagementException err)
{
Dts.Events.FireError((int)err.ErrorCode, "WMI File Watcher", "An error occurred while trying to receive an event: " + err.Message, String.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
catch (System.UnauthorizedAccessException unauthorizedErr)
{
Dts.Events.FireError((int)ManagementStatus.AccessDenied, "WMI File Watcher", "Connection error (user name or password might be incorrect): " + unauthorizedErr.Message, String.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
}