I am importing a FoxPro table into SQL Server 2008 using SSIS. The source data is a proprietary database that I have no control over. Let call the table I am importing Customers.
Sometimes, the structure for Customers looks like this:
ID (int)
NAME (char(30))
ADDRESS (char(30))
CITY (char(20))
STATE (char(2))
ZIP (char(10))
CCNUM (char(16))
Other times, it looks like this:
ID (int)
NAME (char(30))
ADDRESS (char(30))
CITY (char(20))
STATE (char(2))
ZIP (char(10))
CCPTR (char(100))
This proprietary database basically has 2 different versions of the database. The older version had a field called CCNUM (credit card #) that was a basic 16 character field. The newer version replaced that field with a field called CCPTR, which was a 100 character field that represented a card pointer (encrypted value for the actual credit card number).
The problem here is everytime I have to switch back and forth between 2 datasets that have these different table structures, SSIS blows up and I have to go in and manually refresh the metadata.
My question is, is there anyway I can have SSIS dynamically look for one of these fields at runtime, and based on which one is there, load the correct data into the correct table structure in SQL?
Forgive me if this has been asked before. I am still fairly new to SSIS and I tried searching for this answer but to no avail.
Thanks,
Mark
The short answer is no. SSIS expects that there are no significant changes to the meta data of its source and destination components. There are ways to programatically influence this with .NET, but that kind of misses the point.
A well-designed solution to this problem is to create 2 separate data flows that copy the data into a shared staging table. Use this staging table as source to transform your data and push it to its final data structure.
if you build your package based on the lenght (100) and tun it on the (16), you should get only a warning. Are you getting an error?
Related
I've been tasked with creating a database to help with my company's annual physical inventory count. We have an ERP system that contains all our part numbers, and all of our part numbers come in boxes that have a carton label containing "part number" and "quantity" which are both barcoded. We are trying to switch to a system in which we scan all the boxes into my access database vs. our previous system of physically counting and writing tags. So far everything works great, the only issue is that some bar codes get damaged or just won't scan properly for some reason, so the scanners read the barcode as a part number that doesn't exist. We are able to export a list of all part numbers from our ERP system into an excel file which i have linked to my database that we use to see which part numbers were entered that don't exist in the system. My question is, is there a way to prevent people from scanning part numbers that don't match any part number in the excel file exported from our ERP system? I'm using ms access 2003
Using a combobox with LimitToList property set to yes is a great solution. Just to let you know that with this solution it is still possible to PASTE values that are NOT on the list. If you want to guarantee that it is not possible that the field contains wrong values, the best is that you create a Relationship (with referential integrity) taking the Table field containing all valid part numbers as the master field, and the one taking the scanned value as the slave field.
I have a file like as seen below: Just Ex:
kwqif h;wehf uhfeqi f ef
fekjfnkenfekfh ijferihfq eiuh qfe iwhuq fbweq
fjqlbflkjqfh iufhquwhfe hued liuwfe
jewbkfb flkeb l jdqj jvfqjwv yjwfvjyvdfe
enjkfne khef kurehf2 kuh fkuwh lwefglu
gjghjgyuhhh jhkvv vytvgyvyv vygvyvv
gldw nbb ouyyu buyuy bjbuy
ID Name Address
1 Andrew UK
2 John US
3 Kate AUS
I want to dynamically skip header information and load flatfile to DB
Like below:
ID Name Address
1 Andrew UK
2 John US
3 Kate AUS
The header information may vary (not fixed no. of rows) from file to file.
Any help..Thanks in advance.
The generic SSIS components cannot meet this requirement. You need to code for this e.g. in an SSIS Script task.
I would code that script to read through the file looking for that header row ID Name Address, and then write that line and the rest of the file out to a new file.
Then I would load that new file using the SSIS Flat File Source component.
You might be able to avoid a script task if you'd prefer not to use one. I'll offer a few ideas here as it's not entirely clear which will be best from your example data. To some extent it's down to personal preference anyway, and also the different ideas might help other people in future:
Convert ID and ignore failures: Set the file source so that it expects however many columns you're forced into having by the header rows, and simply pull everything in as string data. In the data flow - immediately after the source component - add a data conversion component or conditional split component. Try to convert the first column (with the ID) into a number. Add a row count component and set the error output of the data conversion or conditional split to be redirected to that row count rather than causing a failure. Send the rest of the data on its way through the rest of your data flow.
This should mean you only get the rows which have a numeric value in the ID column - but if there's any chance you might get real failures (i.e. the file comes in with invalid ID values on rows you otherwise would want to load), then this might be a bad idea. You could drop your failed rows into a table where you can check for anything unexpected going on.
Check for known header values/header value attributes: If your header rows have other identifying features then you could avoid relying on the error output by simply setting up the conditional split to check for various different things: exact string matches if the header rows always start with certain values, strings over a certain length if you know they're always much longer than the ID column can ever be, etc.
Check for configurable header values: You could also put a list of unacceptable ID values into a table, and then do a lookup onto this table, throwing out the rows which match the lookup - then if you need to update the list of header values, you just have to update the table and not your whole SSIS package.
Check for acceptable ID values: You could set up a table like the above, but populate this with numbers - not great if you have no idea how many rows might be coming in or if the IDs are actually unique each time, but if you're only loading in a few rows each time and they always start at 1, you could chuck the numbers 1 - 100 into a table and throw away and rows you load which don't match when doing a lookup onto this table.
Staging table: This is probably the way I'd deal with it if I didn't want to use a script component, but in part that's because I tend to implement initial staging tables like this anyway, and I'm comfortable working in SQL - so your mileage may vary.
Pick up the file in a data flow and drop it into a staging table as-is. Set your staging table data types to all be large strings which you know will hold the file data - you can always add a derived column which truncates things or set the destination to ignore truncation if you think there's a risk of sometimes getting abnormally large values. In a separate data flow which runs after that, use SQL to pick up the rows where ID is numeric, and carry on with the rest of your processing.
This has the added bonus that you can just pick up the columns which you know will have data you care about in (i.e. columns 1 through 3), you can do any conversions you need to do in SQL rather than in SSIS, and you can make sure your columns have sensible names to be used in SSIS.
I am reworking and expanding a somewhat complex database schema that has a small number of tables and queries but they are closely related. The only problem I had with it was that in one of the tables the 2 fields that were relating to another table were using the field name of the record and not the ID of the record.
I changed the referring fields data type from text to number and entered some data. The queries and the reports work fine with 1 exception:
There is one report that uses both referring fields. One of the fields is ok but the other one shows symbols instead of numbers. ( The IDs in my sample entries were 14 and 20 and the symbols shown were a double barred music note /alt code 14/ and the symbol for an end of a paragraph /alt code 20/ ) Investigating further I have found that if I make a query that contains the query source for the report both fields display fine, but if I add another table to that query the second field once again shows symbols instead of numbers.
I have found a workaround of this by converting those fields back to text and the id fields in the other tables to text as well. This text key will probably haunt me later on, so I'd like to make it right before it is too late.
This is all access 2010 btw. The source file was already in 2010 (couldn't open in 2007 even)
Sounds like a corruption issue for sure. I would try adding a new column and run an update query to populate it with the values from the old column (maybe use cint(indexfield)), then delete the old column.
It might also be a good idea to decompile the database. This often helps resolve corruption issues.
While working with DB, I find useful using some tools, that help me to solve DB problems.
Some of them are:
1) Insert generator
2) A tool that can execute a script on a list of DB's
3) Finding a text in stored procedures and functions.
4) DB Back up scripts
My question is, what are most useful tools, scripts(anything else), that help you to work with SQL Server?
Thanks in advance.
UPDATE
I assume, there are no other tools for SQL Server 2008 or any other version?
Redgate has a collection of quite powerful tools for Sql Server.
Check out the SMSS Tools Pack.
I have stored procedures that do the following:
system utilities:
find and list every occurrence and info about a given column name or partial column name
find and list every occurrence and info about a given object name or partial object name
list out all the information for a given table, all columns, computed columns, column data types, nullability, defaults, identity, check constraints, index, pk, fk, triggers, and column comments.
find every trigger, view, stored procedure, or function that contains a given string
business utilities:
I also make stored procedures that work with business info. When working on an area of our application I'll make a procedure that displays out all the related info of a given thing. I'll usually display all the info using multiple PRINTs and SELECTs for everything that can join to the given PK (not if there are thousands or rows though). For example, one utiltiy would take a DoctorID as a parameter and list out all the doctor's info, offices that they work at, insurance they accept, etc. I like to include the table names in the output so I can remember where the data comes from without looking at the code. I also join in all the codes tables in these displays, so I'm not looking "A" but "Active (A)". After working on a system for a while, I have loads of these utilities, which greatly helps when a support call comes up or you need to debug a problem, etc. I usually build these as I develop, it is difficult to find time to go back and make them.
I have flat file that structured in a hierarchical format that looks something like this:
Area|AreaCode|AreaDescription
Region|RegionCode|RegionDescriptoin
Zone|ZoneCode|ZoneDescription
District|DistrictCode|DistrictDescription
Route|RouteCode|RouateDescription
Record|Name|Address|Ect
RouteFooter
Route|RouteCode|RouateDescription
Record|Name|Address|Ect
RouteFooter
DistrictFooter
District|DistrictCode|DistrictDescription
Route|RouteCode|RouateDescription
Record|Name|Address|Ect
Record|Name|Address|Ect
RouteFooter
Route|RouteCode|RouateDescription
Record|Name|Address|Ect
RouteFooter
DistrictFooter
ZoneFooter
RegionFooter
AreaFooter
I have to bring this into SSIS and consume information about the Record row and also about the header for the current record row. As well as information from several other sources and output a more simple flat file as a result.
I would like to read the flat file above into a structure that each row contains a record with the appropriate header information included.
My question is, what is the best way to do this if it is even possible?
First how do you tell what type of line you are on if you are on say line 3,987,986? How do you tell what is related to what? Is there apossiblity you could get this in a better format? Before spending lots of time (and don't kid yourself, this will take lots of time to set up and test properly) I would kick the file back to the provider and request it in a different format. You won't always get it, but you should at least try.
When I have done this in the past in DTS, the first characters of each line told me which structure the line referred to. I imported all into a staging table with two columns, one for the recordtype data and one for the rest. Then I parsed the rest into the staging tables for the type of record with the correct column structure for that type of record (and any fileds you might need to do the relationships) and then did clean up and then imported to prod tables. AS you also have differnt number of columns I would try that approach (only you may have to manually populate some columns instead of figuring out directly from the file), also give each record an identity filed in the staging tables. this will help you figure out the realtionships I think.