I am trying to import a file with order numbers that have values in excess of 9 billion into Access.
I have already figured out that I needed to switch from Access 2013 to Access 2016 so that I could enable "Support Bigint Data Type for Linked/Imported Tables". This enabled the Large Number data type in the Access Database.
At this point, I can manually enter Large Numbers into Access, and everything works okay, but when I try to import a .csv with Large Numbers, I get data conversion errors for the Large Numbers (and only the Large Numbers).
In the past, I encountered the issue where text columns starting with what appeared to be a number in the first row encountered conversion errors for actual text values, so I tried putting the Large Number order numbers on the top of the file, and that didn't work.
Can anyone confirm that the Access 2016 Import Wizard can't import Large Number values?
If not, has anyone successfully used import wizard to get a Large Number into Access? How did you achieve that?
Actually, you can use a 20 year old version of access and it can handle numbers with 28 digits.
And if you can enter such numbers manually, the the data is messed up, has extra junk, or extra characters. So no, upgrading will not by magic fix bad formatted data.
I would consider perhaps importing the data, and using say a text column for that data, and then perhaps then see if you can remove junk.
but, if you want to try a larger column, then try currency. and if you want REALLY HUGE WHOPPER then you can use this:
As you can see in above, that is 18 digits long!!!!
So, if the import is failing with above, then attempting to use a MUCH smaller big int will not help you at all here - not at all.
So, try above - see if that works.
However, the fact that you can type in the larger values by hand means that the issue here is messy data - and not that the numbers are too large.
You may well have to settle for importing that column as text, and then running some update query, or even some VBA code to clean out or clear out bad junk and characters or even perhaps extra spaces etc.
So, you don't need to change access to get and allow you to work with larger numbers. The "main" goal and idea behind that larger number is really for better SQL server support, and it not really much if at all a new feature designed to allow you to work with larger numbers (that's more of a by-product of the first goal).
I think people are safer importing using a text file. Access' import specification just seems to fail. I can't rely on it for production systems - the comment by James Marshall above is correct. you can tell Access to import as text, but before it does, if it sees a phone number, say, it will try to convert to a long int first and then just not import numbers exceeding memory, like phone numbers with area codes > 212
Related
I have about 300 measurements (each stored in a dat file) that I would like to read using MATLAB or Python. The files can be exported to text or csv using a proprietary program, but this has to be done one by one.
The question is: what would be the best approach to crack the format of the binary file using the known content from the exported file?
Not sure if this makes any difference to make the cracking easier, but the files are just two columns of (900k) numbers, and from the dat files' size (1,800,668 bytes), it appears as if each number is 16 bits (float) and there is some other information (possible the header).
I tried using HEX-Editor, but wasn't able to pick up any trends from there.
Lastly, I want to make sure to specify that these are measurements I made and the data in them belongs to me. I am not trying to obtain data that I am not supposed to.
Thanks for any help.
EDIT: Reading up a little more I realized that there my be some kind of compression going on. When you look at the data in StreamWare, it gives 7 decimal places, leading me to believe that it is a single precision value (4 bytes). However, the size of the files suggests that each value only takes 2 bytes.
After thinking about it a little more, I finally figured it out. This is very specific, but just in case another Dantec StreamWare user runs into the same problem, it could save him/her a little time.
First, the data is actually only a single vector. The time column is calculated from the length of the recorded signal and the sampling frequency. That information is probably in the header (but I wasn't able to crack that portion).
To obtain the values in MATLAB, I skipped the header bytes using fseek(fid, 668, 'bof'), then I read the data as uint16 using fread(fid, 900000, 'uint16'). This gives you integers.
To get the float value, all you have to do is divide by 2^16 (it's a 16 bit resolution system) and multiply by ten. I assume the factor of ten depends on the range of your data acquisition system.
I hope this helps.
I often have to cleanse and import messy CSV and Excel files into my MS SQL Server 2014 (but the question would be the same if I were using Oracle or another database).
I have found a way to do this with Alteryx. Can you help me understand if I can do the same with Pentaho Kettle or SSIS? Alternatively, can you recommend another ETL software which addresses my points below?
I often have tables of, say, 100,000 records where the first 90,000 records may be null. Most ETL tools scan only the first few hundred records to guess data types and therefore fail to guess the types of these fields. Can I force Pentaho or SSIS to scan the WHOLE file before guessing types? I understand this may not be efficient for huge files of many GBs, but for the files I handle scanning the entire file is much better than wasting a lot of time trying to guess each field manually
As above, but with the length of a string. If the first 10,000 records are, say, a 3-character string but the subsequent ones are longer, SSIS and Pentaho tend to guess nvarchar(3) and the import will fail. Can I force them to scan all rows before guessing the length of the strings? Or, alternatively, can I easily force all strings to be nvarchar(x) , where I set x myself?
Alteryx has a multi-field tool, which is particularly convenient when cleansing or converting multiple fields. E.g. I have 10 date columns whose datatype was not guessed automatically. I can use the multi-field formula to get Alteryx to convert all 10 fields to date and create new fields called $oldfield_reformatted. Do Pentajho and SSIS have anything similar?
Thank you!
A silly suggestion. In Excel add a row at the top of the list that has a formula that creates a text string with the same length of the longest value in the column.
This formula entered as an array formula would do it..
=REPT("X",MAX(LEN(A:A)))
You could also use a more advanced VBA function to create other dummy values to force datatypes in SSIS.
I've not used SSIS or anything like it, but in the past I would have loaded a file into a table with columns ALL of varchar 1000 say so that all the data loaded, then processed it across into the main table using SQL that casts or removes the data values as I required.
This gives YOU Ultimate control not a package or driver. I was very surprised to hear how this works!
A customer of mine is looking to mass create some customizing data related the routes. and as such I have a small program which reads in a CSV file with all of the fields as they would be in the customizing transaction.
I'm having a particular problem wrapping my head around a field TVRO-TRAZTD for a couple of reasons.
The user is only filling in a number which represents a number of days.
There is a conversion exit on TRAZTD, except it's obsolete, use CONVERT TIMESTAMP they say
I don't have a timestamp, I have a decimal number representing a part of a day
For example, TRAZTD would be entered as 0,58 from the CSV file, so why is it represented in the table as 135.512?
I tried it the old fashion way and multiplied 0,58 * 24 which gives me 13,92. if I take 13,92 * 10 I get 139.200, which isn't the same but it's the closest I can get, but I don't get it why 10?
Using the conversion exit even though it's obsolete doens't give me a result either, no matter number I give it I always get 0 back. I can't use the convert timestamp either because well, it's not a timestamp or I didn't look up carefully enough how to use it (I didn't see anything other than strings and characters).
The other thing I tried too was just saying "screw it" and placed the data from the CSV directly into the field and hoping the conversion routine will take care of the work, but that doesn't happen either.
Is there anybody out here that can maybe shed some light on where the number after the conversion comes from?
everybody I came to a solution, just incase anybody stumbles upon this same problem.
I took the value from the excel document and multiplied it by 24 to get the amount of hours, and then multipled it 10000 because I don't know, I picked it randomly.
I've been building a custom module for sugarCRM and i'm running into some issue's, when installing the module i'm met with 'Database failure. Please refer to sugarcrm.log for details.'
Upon checking the log file, i can find the error is this:
"MySQL error 1118: Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. You have to change some columns to TEXT or BLOBs
01/03/14"
Whilst my module does have alot of fields, is there anyway i could get around this? Seems like sugar doesn't give me options for varchar/text etc when creating fields.
Thanks!
I ran in to this same problem when implementing SugarCRM as a multi-tenant solution. You have a couple of options.
1) Go in to studio and set the size of your fields to a smaller value. Each character in a varchar field is a few bytes in size on average. Therefore, if you reduce the amount of characters allowed for each of your fields in studio you will allow for more fields in your module. (see http://dev.mysql.com/doc/refman/5.0/en/column-count-limit.html).
2) Divide up those fields into a couple of modules that relate to a parent module. This will spread out your fields over more than 1 table preventing you from hitting the limit.
I would be happy to assist further if you need a more in-depth explanation of either solution.
I have a problem loading the .CSV file as the connection manager editor settings are out of my knowledge.
When i load the .CSV file up to 18 rows i have no problem it is loading in to the table.
However, from the 19th column the data is not partioning correctly.
row delimeter is {CR}{LF}
column delimeter is Comma {,}
How can i partition the data correctly?
any help?
Here are some ideas I have with no details.
What happens when you try to import the same .CSV file into Excel? Anything interesting around row 19?
Does there appear to be anything different about row 19?
If you delete row 19, what happens?
See, I bet you've thought of these things as well, and probably more, since you have the details. If you want anything more than superficial bad guesses, you'll have to provide a little detail.
I've found the CSV Import to be a bit limited with regards to bad data. If you're having trouble with the 19th column, I would suggest figuring out why that column is failing. You can try and tell the import task's error conditions to Ignore Errors with data truncation, etc...but that may not fix the issue.
I have often switched complicated or error-prone CSV imports to simply use a SSIS Script Task, then just write my own code to parse out the CSV and handle bad data.
If it's not partitioning correctly, it might be something as trivial as one of your field values on row 19 containing a comma, thus throwing out the import by making that row seem to have more columns. If this is the case, I hope you can get a revised version of the CSV file - this time with a text qualifier set. If possible, use something like | rather than " as the qualifier so that it's less likely to appear in the field values.
Put the file in a text editor such as notepad++ or textpad and change the view to show control characters. You will probably find your culprit there.
Nothing unusuale. when i paste in excel as one column and converting text to column has no problem. but i can see in the SSIS preview the field value where the problem has started has two square boxs and data of the next row.
if any one want to see the file let me know i will e-mail you the file.