I have an SSIS package with a Data Flow Task and an FTP Task. I have to use two expression variables like this:
(These create dynamic file names using date parts)
Otherwise if I have just one variable, one task steps on the variable while the other task is trying to use it and gives me the 'cannot lock variable for readonly' error.
Is it possible to have one variable work in two places? Would seem intuitive... This is sloppy. Should someone change one variable without the other to match it would bomb.
I added an Expression Task before the Data Flow... bingo
If you do not require those two tasks to run in parallel, then force one task to complete before the other begins (precedence constraints etc.) - that should prevent race conditions on the single variable.
Related
I have two SSIS projects within the same folder - let's call them Parent.dtsx and Child.dtsx.
Child solution has many SQL task split by different Sequence Containers.
What I need to do is to (in Parent) execute SQL task from (Child). I don't want to execute whole Child solution, only part of it.
I has been searching for a proper solution for a while, but I haven't found a proper answer yet.
Every Parent-Child solution I've seen presents how to execute whole solution (Child) within Parent one.
I tried to execute selected tasks from Child solution by passing the SQL task ID to the Execute Package Task but if failed. Probably, I don't want to pass any variables from Child to Parent - I just need to execute selected SQL tasks from Child.
I'm a beginner when it comes to SSIS.
Thanks,
Karol
Every Parent-Child solution I've seen presents how to execute whole
solution (Child) within Parent one.
That's because that's the only way it works. There is no way to call only some elements of a child package from a parent package; you can only execute the entire child package, unless you want to get into some extremely complicated low-level coding in a script task.
You need to decide where your tipping point is, and do one of the following (whichever is more desirable in your case):
Copy the SQL Task from your Child package and paste it into your parent package, and just have everything in one package.
Modify your child package so that you can pass it a variable, and only execute certain tasks based on the variable that is passed.
Make your solution even more modular: Take the Task you want to execute out of the child package, and put it in its own package all by itself. Then you can call that third package from the child package, and/or you call it directly from the parent package.
Those are your best options.
EDIT: An idea of how to do option 2 - Add a variable to the child package. In the precedence constraints before each task, check the variable, and if it isn't a certain value, then skip that step.
In other words, from your first step, (which may have to be a "dummy" script, because it is going to get executed every time the package starts no matter what), you have multiple constraints coming out. One that says if the first step is complete and the variable equals some value, go to step 2. Another that says if the first step is complete and the variable equals some other value, go to step 3, and so on and so on.
And then from your parent package, you pass whatever variable value will tell the child package to only execute the task you want to execute.
It ends up looking pretty ugly, because you have precedence constraints all over the place, but we have used it in the past and it works. It won't be too bad if you only have two possible paths you want the execution to take.
I have a Custom Source DataFlow Component whose O/P will differ every time, I need to Insert those records in to a Destination table.
Problem:-
Can't Specify the Input columns at Design time for Destination Component.. as in actual for every call to the DataFlow task, The source component gonna return Different O/P Columns based on the Table Schema.
Solution Needed For:-
How to accept whatever inputs are available without any mapping in Destination DataFlow component(either by any Existing Component or by Custom Component)?
The data flow's fixed structure is there for data validation and to optimization purposes. All of it's components are going to have fixed input and output columns. I would suggest the following possibilities:
Write a data flow for every possible schema. There are probably a finite number of possibilities. You could reduce the effort of this task by using BIML which could generate the package structure for you. This may also introduce the possibility of parallel loading.
Use a script task instead of a data flow. In the script task, write the rows for each input into a table.
m
If you need to pass multiple inputs to a single script component, the only way I know to do this is by passing the multiple inputs to a UNION ALL component, and then passing the single output from the UNION ALL to the Script.
You'll have to account for any differences between the two columns structures in the UNION ALL, and maybe use derived columns if you need an easy way to identify which original input a row came from.
I know this is way late but I keep seeing this UNION ALL approach and don't like it.
How about this approach.
Run both data flows into their own recordset destination and save into a variable of type ADO object
Create a new dataflow and use a script source and bring in both ADO objects
Fill datatables using adapter and then do what ever you want with them.
I have a folder with a lot of data files in. I want to be able to loop through the files, look at the headers and sort them into folders if they have the same headers. Is that possible to do in SSIS? If so would anyone be able point me the direction of how to do this?
I am going to try and explain this as best I can without writing a book as this a multi stepped process that isn't too complex but, might be hard to explain with just test. My apologies but I do not have access to ssdt at the moment so I can not provide images to aid here.
I would use the TextFieldParser class in the VisualBasics.dll. in a script task. This will allow you to read the header from file into a string array. You can then build the string array into a delimited column and load an object variable with a datatable that has been populated with two columns. The first column being the filename and the second being the delimiter headers.
Once you have this variable you can load a sql table with this information. (optional to skip if you want to load the columns directly into sql as you read them. your call)
Once you have your sql table you can create an enumerator for that dataset based on the unique headers column.
Then use a foreach loop task with script task to enumerate thru the unique header sets. Use a sql task to assign the file names that belong to the unique header set.
Within the script loop thru the returned file names and apply the necessary logic to move the files to there respective folders.
This is sort of a high level overview as I am assuming you are familiar enough with SSIS to understand the steps necessary to complete each step. If not then I would be able to elaborate later in the day when I am able to get to my SSIS rig.
I have an int variable User::FileLineCount scoped in a for loop container and in a task within the loop, I want to proceed from the task depending in this variable's value.
In the Precedence Constraint Editor I have chosen Evaluation Operation as Expression and the Expression as #FileLineCount!=0. There is another version to some other task as #FileLineCount==0. When I debug, I can see that the User::FileLineCount value is 0 but when I step Over the task I get Unable to step. Not Implemented. error.
Thanks for the help
EDIT: Apparently the debugger could not step over so that was the reason for the error but the conditions still do not work properly.
EDIT2:
The other one is #FileLineCount==0. Doesnt work without OR'in as in the picture.
I had two paths leaving a data flow task one would go to a sql task and the other would go to another task. I was struggling with this until I realized that two paths leaving the same data flow task would be an or if they were different paths. I assume that you would use AND if you had multiple tasks going into one task and you needed them all to be true for it to proceed. I'm not sure if this exactly what you are asking.
This would be or, because I want one or the other.
Where below I want all three to be true to continue to send the email.
I'm looking for some pointers in creating an SSIS based workflow that reads a list of tables at run time from a database and then uses each of these as ADO inputs, selects specific columns from each table and then adds these to a staging area. I've had a quick play with the union task but was looking for some pointers in terms of direction to take ?
I can't seem to find anything on the net that does what I need and am not sure if SSIS can bend to suit my needs.
Many thanks in advance.
You can do this but the only method I can think of is a little convoluted.
You would need to use a "for each loop container" to loop through your list of tables & read each table name into an SSIS variable.
Within the "foreach":
add a script task to build your actual query into another SSIS variable.
add a data flow
within the Data Flow use a source of "SQL Command from variable".
do data flow "stuff"
I hope this makes some kind of sense? :-)