In SSIS
In a folder there are many flat files and by using for each loop container we are processing it one by one. If any new file is placed in the folder and it is still in copying mode. Then, We should not take it for continue process. We should process Only fully copied file alone to our next process.
How can we achieve this? Please give your suggestions.
Best way I have done this in the past is to use a C# Script Task and try to open the file - If the file is still being copied you will get an error (which you Catch). Then you can set a boolean variable to conditionally process the file if the Open worked.
EG:
Boolean b = true;
FileStream f;
try
{
f = new FileStream("C:\\Test\\Test.txt", FileMode.Open, FileAccess.ReadWrite, FileShare.None);
}
catch (IOException e)
{
if (e.Message == "hello")
{
b = false;
}
}
Related
I'm trying to iteratively (part of automation):
Create backup of the projects in solution (physical files on the filesystem)
Using Microsoft.Build programmatically load and change projects inside of the solution (refernces, includes, some other properties)
Build it with console call of msbuild
Restore projects (physically overriding patched versions from backups)
This approach works well for first iteration, but for second it appears that it does not load restored projects and trying to work with values that I patched on the first iteration. It looks like projects are cached: inside of the csproj files I see correct values, but on the code I see previously patched values.
My best guess is that Microsoft.Build is caching solution/projects in the context of the current process.
Here is code that is responsible to load project and call method to update project information:
private static void ForEachProject(string slnPath, Func<ProjectRootElement> patchProject)
{
SolutionFile slnFile = SolutionFile.Parse(slnPath);
var filtredProjects = slnFile
.ProjectsInOrder
.Where(prj => prj.ProjectType == SolutionProjectType.KnownToBeMSBuildFormat);
foreach (ProjectInSolution projectInfo in filtredProjects)
{
try
{
ProjectRootElement project = ProjectRootElement.Open(projectInfo.AbsolutePath);
patchProject(project);
project.Save();
}
catch (InvalidProjectFileException ex)
{
Console.WriteLine("Failed to patch project '{0}' with error: {1}", projectInfo.AbsolutePath, ex);
}
}
}
There is Reload method for the ProjectRootElement that migh be called before iteraction with content of the project.
It will enforce Microsoft.Build to read latest information from the file.
Code that is working for me:
private static void ForEachProject(string slnPath, Func<ProjectRootElement> patchProject)
{
SolutionFile slnFile = SolutionFile.Parse(slnPath);
var filtredProjects = slnFile
.ProjectsInOrder
.Where(prj => prj.ProjectType == SolutionProjectType.KnownToBeMSBuildFormat);
foreach (ProjectInSolution projectInfo in filtredProjects)
{
try
{
ProjectRootElement project = ProjectRootElement.Open(projectInfo.AbsolutePath);
project.Reload(false); // Ignore cached state, read actual from the file
patchProject(project);
project.Save();
}
catch (InvalidProjectFileException ex)
{
Console.WriteLine("Failed to patch project '{0}' with error: {1}", projectInfo.AbsolutePath, ex);
}
}
}
Note: It better to use custom properties inside of the project and provide it for each msbuild call instead of physical project patching. Please consider it as better solution and use it if possible.
In my project, I have the player info being saved out to a JSON file. I am encrypting the information before I save it but if the player goes into the file and happens to delete one character of the file, when the game loads, because it depends upon that file, the game freezes up. I do have it so that a new player info is created if no file is detected, but if the file is there and they mess with it, is there any way of detecting that and correcting it before the game tries to load it. I am using JSONUtility built into Unity.
Use a try/catch when loading the JSON file
try
{
JsonUtility.FromJSON(...)
}
catch (FileNotFoundException e)
{
Print("The file was not found: '{e}'");
}
catch (DirectoryNotFoundException e)
{
Print("The directory was not found: '{e}'");
}
catch (IOException e)
{
Print("The file could not be opened: '{e}'");
}
However, this shouldn't be that important. If a player is trying to mess with the game files and you are worried about the program crashing, you shouldn't because that player shouldn't have been editing game files.
I think using both hashing and try/catch checking would be the better solution
You can just try to open the file then compare its hash with the hash you saved in the last game session:
private void LoadSave()
{
try
{
JsonUtility.FromJson("filename", ...);
string oldHash = PlayerPrefs.GetString("importantSaveFileHash");
string newHash = CalculateMd5("filename");
if (oldHash == null || oldHash == newHash)
{
//recalculate hash every time you change the save file
//you can also encrypt this hash for better security
PlayerPrefs.SetString("importantSaveFileHash", newHash);
//RESULT: Save file is cool!
}
else
{
//RESULT: Save file was modified!
}
}
catch (Exception e)
{
//RESULT: broken file
}
}
private static string CalculateMd5(string filename)
{
using (var md5 = MD5.Create())
{
using (var stream = File.OpenRead(filename))
{
var hash = md5.ComputeHash(stream);
return BitConverter.ToString(hash).Replace("-", "").ToLowerInvariant();
}
}
}
Btw you can just use PlayerPrefs to store all the game state (I know that sometimes its important to have visible and readable save file) - choose what you need)
UPD: its not good to store game state in PlayerPrefs - use any another way instead (read comments)
The requirement is to execute SSIS package, when a file is arrived at a folder,i do not want to start the package manually .
It is not sure about the file arrival timing ,also the files can arrive multiple times .When ever the files arrived this has to load into a table.I think, some solution like file watcher task ,still expect to start the package
The way I have done this in the past is with an infinite loop package called from SQL Server Agent, for example;
This is my infinite loop package:
Set 3 Variables:
IsFileExists - Boolean - 0
FolderLocation - String - C:\Where the file is to be put in\
IsFileExists Boolean - 0
For the For Loop container:
Set the IsFileExists variables as above.
Setup a C# script task with the ReadOnlyVariable as User::FolderLocation and have the following:
public void Main()
{
int fileCount = 0;
string[] FilesToProcess;
while (fileCount == 0)
{
try
{
System.Threading.Thread.Sleep(10000);
FilesToProcess = System.IO.Directory.GetFiles(Dts.Variables["FolderLocation"].Value.ToString(), "*.txt");
fileCount = FilesToProcess.Length;
if (fileCount != 0)
{
for (int i = 0; i < fileCount; i++)
{
try
{
System.IO.FileStream fs = new System.IO.FileStream(FilesToProcess[i], System.IO.FileMode.Open);
fs.Close();
}
catch (System.IO.IOException ex)
{
fileCount = 0;
continue;
}
}
}
}
catch (Exception ex)
{
throw ex;
}
}
// TODO: Add your code here
Dts.TaskResult = (int)ScriptResults.Success;
}
}
}
What this will do is essentially keep an eye on the folder location for a .txt file, if the file is not there it will sleep for 10 seconds (you can increase this if you want). If the file does exist it will complete and the package will then execute the load package. However it will continue to run, so the next time a file is dropped in it will execute the load package again.
Make sure to run this forever loop package as a sql server agent job so it will run all the time, we have a similar package running and it has never caused any problems.
Also, make sure your input package moves/archives the file away from the drop folder location.
As others have already suggested, using either WMI task or an infinite loop are two options to achieve this, but IMO SSIS is resource intensive. If you let a package constantly run in the background, it could eat up a lot of memory, cpu and cause performance issues with other packages depending on how many other packages you've running. So other option you may want to consider is schedule an Agent job every 5 minutes or 10 minutes or something and call your package in the job. Configure the package to continue only when a file is there or quit otherwise.
You can create a Windows service that uses WMI to detect file arrival and launch packages. Details on how to are located here: http://msbimentalist.wordpress.com/2012/04/27/trigger-ssis-package-when-files-available-in-a-folder-part2/?relatedposts_exclude=330
What about the SSIS File Watcher Task?
What I'm trying to do in SSIS is have a WMI Event Watcher Task which watches a folder for a file to be created, then does something with it. The primary part is the "watching the folder for file creation".
I have a network folder (full path): \\srvblah10\main\child\target\
All the sites I've gone to has this as an example:
SELECT * FROM __InstanceCreationEvent WITHIN 10
WHERE TargetInstance ISA "CIM_DirectoryContainsFile"
AND TargetInstance.GroupComponent = "Win32_Directory.Name=\"d:\\\\NewFiles\""
Since the folder is a network folder, I can't provide the physical disk letter. So is there a way to use a similar WQL query but for network folder paths as opposed to physical folder paths?
You have to map the drive with a dos command:
net use s: \srvblah10\main\child\target\ /user dotnetN00b Pa$$word
then you can the WMI Event Watcher Task to watch it.
I was trying to do this for awhile, and finally gave up on trying to use the SSIS WMI Event Watcher task, and just wrote the equivalent in a Script task. The issue that was the challenge was getting the WMI Event Watcher to make the remote connection with specific user credentials that I wanted to obtain from a configuration section (not hard code into the package).
The second issue that was going to make not using a script difficult was simply translating the network share, into the local path name on the server, which the Event Watcher requires. You'll see from the scrip below, everything is accomplished with a minimal of effort.
Just an additional heads up, make sure to include the DTS.Variables the script uses in the ReadOnlyVariables (as normal). The code below requires three DTS variables, for example if you are trying to watch for files being dropped in the following location \copernicus\dropoff\SAP\Import, then you would set the variables as shown below:
User::ServerName - the hostname of the server where the share lives
(copernicus)
User::ShareName - the name of the network share
(dropoff)
User::ImportPath - the directory path of the directory to
watch for new files in (/SAP/Import)
public void Main()
{
string localPath = "";
try
{
ConnectionOptions connection = new ConnectionOptions();
connection.Username = "<valid username here>";
connection.Password = "<password here>";
connection.Authority = "ntlmdomain:<your domain name here>";
ManagementScope scope = new ManagementScope(#"\\" + Dts.Variables["User::FileServerName"].Value.ToString() + #"\root\CIMV2", connection);
scope.Connect();
/// Retrieve the local path of the network share from the file server
///
string queryStr = string.Format("SELECT Path FROM Win32_Share WHERE Name='{0}'", Dts.Variables["User::ShareName"].Value.ToString());
ManagementObjectSearcher mosLocalPath = new ManagementObjectSearcher(scope, new ObjectQuery(queryStr));
foreach (ManagementObject elements in mosLocalPath.Get())
{
localPath = elements["Path"].ToString();
}
queryStr = string.Format(
"SELECT * FROM __InstanceCreationEvent WITHIN 10 WHERE Targetinstance ISA 'CIM_DirectoryContainsFile' and TargetInstance.GroupComponent=\"Win32_Directory.Name='{0}{1}'\"",
localPath.Replace(#"\", #"\\"),
Dts.Variables["User::ImportPath"].Value.ToString().Replace(#"\", #"\\")); // query requires each seperator to be a double back slash
ManagementEventWatcher watcher = new ManagementEventWatcher(scope, new WqlEventQuery(queryStr));
ManagementBaseObject eventObj = watcher.WaitForNextEvent();
// Cancel the event subscription
watcher.Stop();
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (ManagementException err)
{
Dts.Events.FireError((int)err.ErrorCode, "WMI File Watcher", "An error occurred while trying to receive an event: " + err.Message, String.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
catch (System.UnauthorizedAccessException unauthorizedErr)
{
Dts.Events.FireError((int)ManagementStatus.AccessDenied, "WMI File Watcher", "Connection error (user name or password might be incorrect): " + unauthorizedErr.Message, String.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
Is there any way to get all files and folder in box without knowing their id? Also, how to get all collaboration objects if I don't know collaboration id?
You can get root folders and files by specifying folder id = 0.
And with this result further folders or files can also be fetched.
You can use Get Folder Items on a folder id you do know to retrieve the IDs of the folders and files that it contains. As Shanky says, use 0 to start at the root folder.
Get Collaborations will show the collaborations on a folder. You don't need any information about the collaboration, just the folder id.
Unlike systems that are built exclusively on path-access, Box gives you a durable ID for each folder and file. This has a bunch of advantages. One of the big ones is that you can rename, or move a file around, and getting to it never needs to change. It also means that you can persist the IDs, associate them with some other entity in your own system, and still be able to get back to the same file or folder. Assuming of course that you are still allowed access to it. Permissions can change too of course.
You can get all the collaborations for a user by calling GET /collaborations or all the collaborations on a folder by calling GET /folder/ID/collaborations
Call listFilesInBoxFolders("0") ---> This will parse all files and folders starting from root
public void listFilesInBoxFolders(String folderId) {
try {
// get a list of songs
BoxApiFolder folderApi = new BoxApiFolder(mSession);
BoxListItems items = folderApi.getItemsRequest(folderId).send();
JSONObject object = new JSONObject(items.toJson());
LogUtils.log(TAG, "Object " + object.toString());
JSONArray array = object.getJSONArray("entries");
for (int i = 0; i < array.length(); i++) {
JSONObject object1 = array.getJSONObject(i);
String type = object1.getString("type");
String id = object1.getString("id");
String name = object1.getString("name");
if (type.equals("folder")) {
listFilesInBoxFolders(id);
} else if (type.equals("file")) {
// Supported Media file types
if (name.contains("mp3") || name.contains("m4a") || name.contains("flac")) {
musicItems.add(new BoxMusicItem(id, name));
}
}
}
LogUtils.log(TAG, "array " + array.toString());
} catch (BoxException e) {
e.printStackTrace();
} catch (JSONException e) {
e.printStackTrace();
LogUtils.log(TAG, "Json Error");
}
// For testing to make sure i have all files in box
printFilesInBox();
}