I am bit new to SSIS. I am using SSIS 2012.The input files are excel, csv and txt.
The data has to be dumped from input files to the database. The size of the columns in the input files keep on changing, so i cant stick to a fixed length. Changing the data type of connection managers in the package to ntext would solve this but we have performance constraint too. So customer prefers to truncate the extra data and intimate him than affecting the performance.
Row redirection will give the rows that are truncated. But i want to intimate the customer in each file the columns that are truncated.
Is SSISDB tracks the data that are truncated. If so which table.
I am planning to write the truncated data to a separate files and then use script component to compare the length of each column. Is there a better way.
Write the truncated rows (redirected) to a table. You can add in a Derived Column to add in the name of the source file (if you get the package to hold it in a variable) between the redirecting Input and the Output that writes to a table.
If you truncate the table before each run, a simple ExecuteSQL to get the Count of rows into a variable, and then an email to the customer if Count>0, will work.
Ok, as far as I know, there is nothing in SSIS which automatically tracks which columns are truncated.
There are a couple ways I can think of to handle this and they both require the main logic occurring in a script transformation.
If I had to do this, I would create a script component that "predicts" which columns will be truncated before passing them on to the rest of the dataflow.
As you use LEFT() to truncate each string to the required length, you can check afterwards to see if the old string and the new string are the same.
If not, then you know truncation has occurred and you can populate a variable to use in a send email task.
Or you can truncate the columns with derived columns first and then use a script transformation to compare the old column with the new column. Same logic.
Related
I have created many SSIS packages in the past, though the need for this one is a bit different than the others which I have written.
Here's the quick description of the business need:
We have a small database on our end sourced from a 3rd party vendor, and this needs to be overwritten nightly.
The source of this data is a bunch of flat files (CSV) from the 3rd party vendor.
Current setup: we truncate the tables of this database, and we then insert the new data from the files, all via SSIS.
Problem: There are times when the files fail to come, and what happens is that we truncate the old data, though we don't have the fresh data set. This leaves us without a database where we would prefer to have yesterday's data over no data at all.
Desired Solution: I would like some sort of mechanism to see if the new data truly exists (these files) prior to truncating our current data.
What I have tried: I tried to capture the data from the files and add them to an ADO recordset and only proceeding if this part was successful. This doesn't seem to work for me, as I have all the data capture activities in one data flow and I don't see a way for me to reuse that data. It would seem wasteful of resources for me to do that and let the in-memory tables just sit there.
What have you done in a similar situation?
If files are not present update some flags like IsFile1Found to false and pass these flags to stored procedure which truncates on conditional basis.
If file is empty then Using powershell through Execute Process Task you can extract first two rows if there are two rows (header + data row) then it means data file is not empty. Then you can truncate the table and import the data.
other approach could be
you can load data into some staging table and from these staging table insert data to the destination table using SQL stored procedure and truncate these staging tables after data is moved to all the destination table. In this way before truncating destination table you can check if staging tables are empty or not.
I looked around and found that some others were struggling with the same issue, though none of them had a very elegant solution, nor do I.
What I ended up doing was to create a flat file connection to each file of interest and have a task count records and save to a variable. If a file isn't there, the package fails and you can stop execution at that point. There are some of these files whose actual count is interesting to me, though for the most part, I don't care. If you don't care what the counts are, you can keep recycling the same variable; this will reduce the creation of variables on your end (I needed 31). In order to preserve resources (read: reduce package execution time), I excluded all but one of the columns in each data source; it made a tremendous difference.
I'm trying to figure out how to convert my existing DTS files to DTSX hosted on a SQL 2005 server.
In my first try (first DTS) I'm already stuck.
I don't wanna look how things are done using DTS and wanna focus on the new DTSX techniques.
What has to be done.
Check if input file exists else exit.(not done yet)
Truncate destination table
Import file into DB
Report if everything was alright.
step import file is where I'm stuck. I have a fixed columns sized flat tekst file where housenumber and extension are in a single column. The Database has two columns for it.
I first tried a derived column but could find a check for splittng the (first) numeric part.
When searching for the use of regex i read about the "script component" which i read isn't compatible with SQL 2005.
Is there another possibility?
This brings me to a second question: Is it possible to use SQL Server Data Tools (SSDT) with SQL Server 2005.
You have a job ahead of you. You will need to use Derived Column transformations, so look them up. It will be helpful to add Data Viewers to your dataflows so you can see what data is moving through the flow, and what SSIS thinks is in there.
In your case, you are going to have to manipulate strings. There are two string data types, DT_STR and DT_WSTR (e.g., VARCHAR and NVARCHAR). SSIS is very particular about data types, and you may have to convert one type into the other using cast operations. E.g., (DT_WSTR, 50)Blah converts fifty characters from column Blah into DT_WSTR. The DT_STR type also needs to know the code page,. e.g. something like 1252.
You will likely need the SSIS version of Immediate If, which checks if a condition is true, and returns either one result or the other: Blah==100?1:0 The result of this will be a 1 for those records where column Blah equals 100, and 0 for all other records. You can nest these Immediate If statements, with one inside the next, inside the next, adn so forth.
You will need at least two new columns being created in the Derived Column widget, one for the numbers, and one for what follows. So here is one very painful way to do what you want. Use a string function to check whether the first character is a number, and then either return it, or don't. Do the same for the second character, etc.
Of course, I'm sure there's a better way to do this. Your toolbox consists of the functions in the Derived Column widget, so that's what you've got to work with. (Or, alternatively, you might do better in this case with a SQL UPDATE statement, which you would execute as a subsequent task in the Control Flow above, not in the Data Flow below.)
Fair warning: the SQL Server 2005 version of SSIS has many, many, many bugs and frustrations which were fixed or improved in later versions of SQL Server. Even if you just go to SQL Server 2008 you will save yourself many headaches.
I have a simple DataFlow with two objects the source which is a mdb file and the destination which is an MSSQL database.
The idea is to migrate the data from one to another.
The problem is that the data is extracted from an Access query, and one column has ~1000 characters, and in SSIS in advanced properties the external column has the default 255 length so when i execute the task it tries to truncate it. To disable the throw error on truncate is not an option, and modifying the Length of the external column cannot be done, it throws and error regarding the metadata.
First of all can anyone explain WHY?
Second of all i need a resolution and i need it fast because it's kinda driving me crazy.
This kind of problem occours, because the ssis task "guesses" the length of the column by inspecting the first 100(afaik) rows. So if all rows from 1 to 100 have a length of 10 and the row 101 has the legnth of 11, the task will fail, because the length was "guessed" to 10.
Modifying throws an error, because you have validateExternalMetadata set to true. To solve this problem, go to advanced options of your import task (access) and set the value to false.
This means, the task will accept modified values you entered without checking it.
Did you try to SSIS Import and Export Wizard to import the data, from within the BI development environment? That is the easiest way with MsAccess as this not only imports the data but also saves the package. If you get an error during the import ( using the wizard), please post it, as this helps in further investigation. Also, as #stb suggested, try having the first record over 1000 characters.
Access supports queries which are the equivalent to views in MSSQL.
The column size is defined not by looking at a few results but by the default column length of the column data type.
I created another table with the desired data types and before the data flow i've put in the package 2 sql scripts: one to delete all the data in the table and one to execute the query against the table, as to treat it as a temporary table.
Then the actual data flow is executed against this pseudo-temporary table.
This solved my problem.
I have no idea whether this can be done or not, but basically, I have the following data flow:
Extracts the data from an XML file (works fine)
Simply splits the records based on an enclosed condition (works fine)
Had to add a derived column object due to some character set issues (might be better methods, but it works)
Now "Step 4" is where I'm running into a scenario where I'd only like to insert the values that have a corresponding match in my database, for instance, the XML has about 6000 records, and from those, I have maybe 10 of them that I need to match back against and insert them instead of inserting all 6000 of them and doing the compare after the fact (which I could also do, but was hoping there'd be another method). I was thinking that I might be able to perform a sql insert command within the OLE DB DESTINATION object where the ID value in the file matches, but that's what I'm not 100% clear on or if it's even possible for that matter. Should I simply go the temp table route and scrub the data after the fact, or can I do this directly in the destination piece? Any suggestions would be greatly appreciated.
EDIT
Thanks to the last comment from billinkc, I managed to get bit closer, where I can identify the matches and use that result set, but somehow it seems to be running the data flow twice, which is strange.... I took the lookup object out to see whether it was causing it and somehow it seems to be the case, any reason why it would run this entire flow twice with the addition of the lookup? I should have a total of 8 matches, which I confirmed with the data viewer output, but then it seems to be running it a second time for the same file.
Is there a reason you can't use a Lookup transformation to find existing records. Configure it so that it routes non-match records to the no match output and then only connect the match found connector to the "Navigator Staging Manager Funds"
I believe that answers what you've asked but I wonder if you're expressing the right desire? My assumption is the lookup would go against the existing destination and so the lookup returns the id 10 for a row. All of the out of the box destinations in SSIS only perform inserts, so that row that found a match would now get doubled. As you are looking for existing rows, that usually implies you'd want to perform an update to an existing row. If that's the case, there is a specially designed transformation, the OLE DB Command. It is the component that allows for updates. There is a performance problem with that component, it issues a single update statement per row flowing through it. For 10 rows, I think it'd be fine. Otherwise, the pattern you'd use is to write all the new rows (inserts) into your destination table and then write all of your changed rows (updates) into a second staging-type table. After the data flow is complete, then use an Execute SQL Task to perform a set based update statement.
There are third party options that handle combined upserts. I know Pragmatic Works has an option and there are probably others on the tasks and components site.
I'm fairly new to SSIS,
I'm importing from an XLS spreadsheet into a database table. Along the way I want to select a record from a table, but it is NOT a lookup, ie: a straight SELECT with no join from input source. Then I want to merge this along with the other rows from the XLS.
What is the best way to do this? Variables? OLE DB commands?
Thanks
You could use an OLE DB command but the important thing to remember about this is that it is fired on a per-row basis and could potentially be slow. You can still use a lookup for this purpose, but make sure that you use set the error output to ignore lookup errors for the cases when the lookup transformation does not contain an value for the match you are looking for.
You could also use a merge transformation with an outer join condition rather than an inner join.
If the record that you are retrieving from the database table is not dependent on the data within the row from the spreadsheet then it will probably be the same for each row - is that what you are hoping for?
In this case, I would consider using an Execute SQL Task in the Control Flow to retrieve the record and save it to a variable. You can use a Script Component in the Data Flow to copy the values in the record from the variable to the appropriate fields in each row. This will mean that the lookup data is retrieved only once and not once per row which is slow as jn29098 said above.
If the target for your Data Flow is the same database as the one from which you are extracting the 'lookup' record then you could also consider using an Execute SQL Task (in the Control Flow) to add the lookup values once the spreadsheet data has arrived in the database (once the Data Flow has completed). This would be much more efficient.