I got an error after running a SSIS package that has worked for a long time.
The error was thrown in a task used to create a directory (like this http://blogs.lessthandot.com/wp-content/uploads/blogs/DataMgmt/ssis_image_05.gif) and says "Cannot create because a file or directory with the same name already exists", but I am sure the directory or a file with the same name didn´t exist.
Before throwing error, the task created a file with no extension named as the expected directory. The file has a modified date more than 8 hours prior to the created date wich is weird.
I checked the date in the server and it is correct. I also tried running the package again and it worked.
What happened?
It sounds like some other process or person made a mistake in that directory and created a file that then blocked your SSIS package's directory create command, not a problem within your package.
Did you look at the security settings of the created file? It might have shown an owner that wasn't the credentials your SSIS package runs under. That won't help if you have many packages or processes that all run under the same credentials, but it might provide useful information.
What was in the file? The contents might provide a clue how it got there.
Did any other packages/processes have errors or warnings within a half day of your package's error? Maybe it was the result of another error. that you could locate through the logs of the other process.
Did your process fail to clean up after itself on the last run?
Does that directory get deleted at the start of your package run, at the end of your package run, or at the end of the run of the downstream consumer of the directory contents? If your package deletes it at the beginning, then something that slows the delete could present a race condition that normally resolves satisfactorily (the delete finishes before the create starts) but once in a while goes the wrong way.
Were you (or anyone) making a copy or scan of the directory in question? Sometimes copy programs (i.e. FTP) or scanning programs (anti virus, PII scans) can make a temporary copy of a large item being processed (i.e. that directory) and maybe it got interrupted and left the temp copy behind.
If it's not repeatable then finding out for sure what happened is tough, but if it happens again try exploring the above. Also, if you can afford to, you might want to increase logging. It takes more CPU and disk space and makes reviewing logs slower, but temporarily increasing log details can help isolate a problem like that.
Good luck!
Related
First time posting a question, please forgive if I don't have enough information.
I have a tiered BIML Script that has the following tiers:
10-Connection – create the connection nodes
20- Model – loop through the connections to build database, table nodes
30-Create/Drop Staging Tables – This is included as the package/s need to be run prior to the remainder of the creation process.
30- Flat File – loop through the table object to create flat file formats and connections
40-Packages – loop through the table objects and create extract and load packages
45-Project Params & Connections– attaches the project params and connections (using named connections and GUIDs from 10-Connections). Project params manually created in SSIS.
The process successfully connects to the source SQL Server database, generates the Create/Drop Staging Tables packages with correct metadata, and will create the extract packages successfully the first time.
Upon a second attempt to process the same BIML scripts with no changes made to the files, the process fails with “Object reference not set to an instance of an object.” & “Unable to Query on Connection” on the OleDBSource Node.
The BIML files generated in preview and output debugging have valid queries and source metadata that indicate a positive connection and proper model. I have used emitted queries in SSMS without error. When I move the BIML files to a new project, the process is successful the first time and fails subsequently.
I have tried the following:
Connection Managers
Delete project connection managers prior to package re-generation
GUIDs annotated and used in PackageProject and Packages Nodes.
Delay Validation/Validate External Metadata – I have tried with both true and false on Package, DFT and OleDBSource
Project
Delete .proj files from directory
Direct PackageProject to new ProjectSubpath
I also tried simply hard coding the BimlScript to simplify and remove any variables with the same result.
The most maddening point is that the metadata and queries all indicate the process can connect and query this exact table and it functions, but only on initial creation. Adding or re-generating during testing fails. Anyone ever come across this before?
Great thanks and shout out to cathrine-wilhelmsen, billinkc, whose posts and tutorials have been very helpful. Any and all help would be greatly appreciated.
I changed the driver from SQLNCLI11 to SQLOLEDB with no changes to code. I tested different drivers after seeing a few example connection strings with different drivers.
I wish I could explain why.
I have an issue writing files to a samba share. We don't seem to receive any failure error during the writing of the files, but a second later when we check from a different process, no files have been written. This problem seems to happen sporadically for about 5 minutes, or 10 minutes, and then go away.
The only clue we have is from the samba's logging. There are STATUS_OBJECT_NAME_COLLISION errors present. My understanding is that this means our software is trying to write a new file over a file that already exists. But what I don't understand is why, then, I see no files in that location at all after the process concludes. Could this error mean something else? Could it be caused by the configuration of the fileshare somehow?
Thank you.
The code STATUS_OBJECT_NAME_COLLISION may indicate an attempt to create a file which already exists while overwrite option was not specified.
Is your soft renaming any file on the destination?
Overall, I am looking for feedback regarding two different design options of running a master package.
I have one package that Agent calls that runs a bunch of packages that process data (I think we are up to about 50 now).
The original design was to group packages into smaller chunks called directorates which call the actual packages. Sample below:
A few perceptions I see (and experienced) with this approach is that:
1. Every package has to open (even if it is unnecessary to run ie no file present)
2. #1 adds so much time for the process to complete
3. Runs in parallel for sure
So I developed a new approach which will only run the packages that have the necessary files and logs the attempt if not. It is so much cleaner and you don't need all the file connections for each package to run since you are iterating through them.
I am not sure it runs in parallel (I actually doubt it).
I am adding the dataflow that populates the ADO Object that is being iterated in foreach to demonstrate the files being processed.
Note: Usually in DEV environment there are not many files to be processed, however, when deploying to TEST and PROD there will be most files present to be processed.
Can I get some feedback on these two different approaches?
Anyone that provides productive feedback will recieve upvotes!!!
I would go with modified first approach ie something like Inside package, use Script task to check if files are present in destination or not.
For instance :
Create a Script task and a variable.
Inside script task, write a code similar to the image below(Logic is, if file is found then flag it as true, else flag is false) :
Now constraint the execution of DFT by using this flag as shown below :
Only con is, you'll have to make changes in 50 packages, but this is a one time activity. Your parallel execution will remain intact.
I will go with 2nd approach as its cleaner and easy to debug.
Here are the suggestions to improve 2nd approach :
Create a Control table with all package Names, Enable/Disable flag, FileAvailable Flag
Create a Poll package which will go through files and sets flag and package flag accordingly
Loop through this Control table and run only those are enabled and having file.
In MVS, I'm looking for a UNIX-like touch command to keep thousands of files 'alive' on a seldomly used system. I have a list of every file name that might exist at any one point in time, but the actual files that exist on catalog can come and go depending on what is running on the system.
HRECALL doesn't work becuase the files are huge, and cannot be allowed to migrate off catalog.
IEBGENER dummy copies don't work because it fails if any of the files are missing.
Is there a 'touch' command that won't fail on missing files?
Thanks!
The LISTDSI function in REXX (or CLIST) fits the bill. It will update the last referenced date, which is what HSM uses for migration.
Short Version:
I have 7 SSISDeploymentManifest files I need to run. Is there a way to alter the SSISDeploymentManifiest file to per-populate Installation value?
Rant Version
At first running 7 deployments did not seem like much of a problem. But the part of the process where you "Select Installation Folder" for package dependencies is horribly designed.
First, you have to enter a network path here if you are not running from the server you will install to. This is because the dialog box makes sure path you enter a valid path... on the local machine you run the manifest from. But when the package is run it will need to also work for the server. (dumb huh?))
The next problem with this screen is that the field is read only. So I cannot just specify the path directly.
Second, the dialog box to "browse" won't let me enter a path.
So... I have to browse my entire network (from home, over a vpn). That takes a long time.
Is there a way to alter the SSISDeploymentManifiest file to pre-populate this value?
No, dtsinstall doesn't accept any command line arguments, pity. My first approach to this was to write a heavy, command line application that made repeated calls to dtutil to get things done. I never finished it but if you want to peek, it's on codeplex
What I do like and prefer is a PowerShell script that handles my SSIS deployments now. Even if PowerShell isn't your cup of tea, the concepts should apply to whatever .NET language you might want to use to handle it.
Attractive features of it are that it will create the folders in SQL Server for you and correctly deploy those packages into said folders. The other nice thing that might be of assistance to you is that if all the 7 deploys are in a common folder structure, the script walks the folder structure looking for manifests and deploys all the packages per manifest so you could conceivably deploy all with a single mouse click.