I can see that using maatkit I can export the data as comma or tab separated values. But is there anyway to change the field separator?
http://www.maatkit.org/doc/mk-parallel-dump.html
I have a table that has comma as well as tabs in the data. Besides I need to process the data using awk that does not seem to work with certain fields data. I want to change the separator while dumping data using maatkit. Is it possible?
Commas present in data should not cause any problem: the text fields (and maybe all fields) are within quotes:
"field 1 value", "field 2 value", 3.38823, "field 4 value"
Related
Iam getting the below error when I try to load CSV From my system to Snowflake table:
Unable to copy files into table.
Numeric value '"4' is not recognized File '#EMPP/ui1591621834308/snow.csv', line 2, character 25 Row 1, column "EMPP"["SALARY":5] If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client.
You appear to be loading your CSV with the file format option of FIELD_OPTIONALLY_ENCLOSED_BY='"' specified.
This option will allow reading any fields properly quoted with the " character, and even support such fields carrying the delimiter character as well as the " character if properly escaped. Some examples that could be considered valid:
CSV FORM | ACTUAL DATA
------------------------
abc | abc
"abc" | abc
"a,bc" | a,bc
"a,""bc""" | a,"bc"
In particular, notice that the final example follows the specified rule:
When a field contains this character, escape it using the same character. For example, if the value is the double quote character and a field contains the string A "B" C, escape the double quotes as follows:
A ""B"" C
If your CSV file carries quote marks within the data but is not necessarily quoting the fields (and delimiters and newlines do not appear within data fields), you can remove the FIELD_OPTIONALLY_ENCLOSED_BY option from your file format definition and just read the file at the delimited (,) fields.
If your CSV does use quoting, ensure that whatever is producing the CSV files is using a valid CSV format writer and not simple string munging, and recreate it with the quotes properly escaped. If the above data example is to be considered valid in quoted form, it must instead appear within the file as "4" or 4.
The error message is saying that you have a value in your file that contains a "4 which is being added into a table that has a number field for that value. Since that isn't a number, it fails. This appears to be happening in your very first row of your file, so you could open it up and take a look at the value. If its just one record, you can add the ON_ERROR = 'CONTINUE' to your command, so that it skips it and moves on.
While importing a tab-separated-values (TSV) file, I find that there are two cases where mysql injects double quotes that I would like to disable.
Case 1: When the field value is empty/blank (zero characters) - instead of a NULL, a "" gets inserted. I prefer it be just a NULL.
Case 2: When a double quote is already in the string (i.e. 2" Square), it replaces the occurrence of the double quote with two double quotes and it also wraps the entire string in double quotes. Example: 2" Square gets inserted as "2"" Square". I want it to be just 2" Square, just as it is in the report.tsv file.
The current SQL I am using is (report-columns is a list of columns each with the definition VARCHAR(25) DEFAULT NULL)
LOAD DATA LOCAL INFILE 'report.tsv'
INTO TABLE properties
IGNORE 1 LINES (report-columns)
I tried adding FIELDS ESCAPED BY '' but this didn't change the results.
I prefer a plan to be able to just disable this from happening in the first place in the same query that loads the file, but if that is not possible a backup plan would be to execute a cleanup query for all report-columns to be run immediately after import to address both cases 1 and 2 above.
Ideas on either plan?
Update
After further investigation into this I determined that it was not mysql's loading of the data that was causing case 2, rather it was a clean script just before the load. The answer for using SET and NULLIF in the same query with the LOAD DATA LOCAL INFILE statement has resolved case 1 nicely without needing a second query cleanup.
If you want a NULL to be inserted, you can use a literal \N in your source text file, or the word NULL, depending on some of your options. This is documented in the manual for LOAD DATA INFILE: https://dev.mysql.com/doc/refman/5.7/en/load-data.html
Look for the phrase "Handling of NULL values varies according to the FIELDS and LINES options in use" and read what follows (I'm not going to copy the manual into this Stack Overflow answer).
Another option is to use an expression to modify the value on import if it's an empty string:
LOAD DATA LOCAL INFILE 'report.tsv'
INTO TABLE properties
IGNORE 1 LINES (report-columns)
SET some_col = NULLIF(some_col, '');
I'm not sure what to recommend for the second case, when you have double-quotes in your input text data. I tried this, and it worked fine for me. I was able to import the string 2" square.
It could be complicated if you use LOAD DATA INFILE with the ENCLOSED BY '"' option. You might want to enclose fields by some different character that doesn't appear in the text.
I have an access DB. I exported tables to xlsx. Then I saved as .ods using openOffice
because I found out that phpmyadmin-mysql no longer supports excel files. I have my mySQL database formated exactly as it should to accept the data. I import and everything seems fine except one little detail.
In some fields, the value is NULL instead of the value it should have according to the .ods file. Some rows show the same value for that field correctly, some show NULL.
Also, the "faulty" rows have some fields that show the value 0 for fields that where empty in the imported file (instead of NULL). Default value for those fields in mySQL is NULL. Each row has many fields like that and all of the same data type (tinyint). Some appear correctly NULL and some have the value 0....
I can't see a pattern on all these.
Any help is appreciated.
Check to see that imported strings have ("") quotes and NULL do not and that all are separated appropriately, usually a "," comma with the record/row delimited by ";" semicolon. Best way to check what the MySQL is looking for is to export some existing data to the same format and check it against what you are trying to import. One little missed quote and the deal is off. Be consistent in the use of either double " quotes or single ' quotes. also the ` character is not used as I think. If you are "squishing" your data through an application that applies "smart quotes" like MS word does or "Open Office??' this too can cause issues. Add the word NULL either inside or without quotes in your csv import where values appropriate.
I'm using a SELECT INTO OUTFILE to extract rows from a database into CSV.
I've got a couple issues I am trying to deal with.
The first is that I am formatting currency values in the format "$135,300.00"
When I open the csv in Excel, it shows all the currency fields as "########" until they are click on.
Also, null values are inserted as "\N", I would like them to simply be empty.
Thanks for any help you can provide.
The #### thing is an Excel feature. It shows any values as such when the cell is too small to show the full number. Just increase the column width until the cells are big enough.
As for the \n thing, the reference says
If the FIELDS ESCAPED BY character is empty, no characters are escaped and NULL is output as NULL, not \N.
So you're probably using a FIELDS ESCAPED BY in the query. It'd help to see the full query you're using.
We have a large tab-delimited text file (approximately 120,000 records, 50MB) that we're trying to shove into MySQL using mysqlimport. Some fields are enclosed in double-quotes, some not. We're using the fields-optionally-enclosed-by='\"' switch, but the problem is some of the field values themselves contain double-quotes (indicating inches) so the delimited field value might be something "ABCDEF19"". Make sense?
We have no control over the source of the file, so we can't change the formatting there. I tried removing the fields-optionally-enclosed-by switch, but then the double-quotes that surround the values are imported.
he records with quotes in the values are getting seriously messed up. Is there a way we can tell mysqlimport that some fields are optionally enclosed by quotes, but may still contain quotes? We've thought maybe a global search and replace to escape the double-quotes in field values? Or any other suggestions?
If your data is including quotes inside of the body of the field quote without delimiting that somehow, you have a problem. You can't guarantee that mysqlimport will do this properly.
Massage the data first before trying to insert it in this way.
Luckily, it is tab-delimited, so you can run a regex to replace the quotes with a delimited version and then tell mysqlimport the delimiter.
You could import it with the quotes (fields-optionally-enclosed-by switch removed) and then run a check where if the value has double quotes at the beginning and end (assuming none of the values have inches at the beginning) then truncate by 1 character from the beginning and end to remove the extra quotes you got from importing.
EDIT: after reading kekoav's response I have to agree that if you are able to manipulate the file before importing that would be a much wiser option, but if you are forced to remove quotes afterwards, you could use something like this:
UPDATE table
SET column =
IF(
STRCMP(LEFT(table.column,1),'"'),
MID(table.column,2,(LENGTH(table.column)-2)),
table.column
)
for every 'column' in 'table'