I am attempting to convert some SQL Server data into mySQL, but keep running into hurdle after hurdle.
I assumed the easiest way would be to export the SQL Server data as CSV and import it that way (little did I know it wasn't that easy...!). Once I figured out how to get it to add quotes to the CSV, I attempted an import into mySQL. Now, for some reason the data was all imported WITH the quotes. Here is my mySQL query:
LOAD DATA INFILE 'file.csv' INTO TABLE test
FIELDS ENCLOSED BY '"'
TERMINATED BY ","
LINES TERMINATED BY "\n"
This imports, but it seems to ignore the quotes, and when commas are present in the data it completely screws up the row. The data is also imported including the quotes, see this data directly from the table:
"1" "1" "0" "{08CA6F70-735D-46ED-8EAB-C17A8BED1FCD}" "Reporting_1" "60" "Reporting" "ABC" "2008-04-21 19:25:28.013000000" "False" "True" "False" "True" "3" "164" "2033" "565077" "7929083" "334980" "2013-01-11 15:35:45.970000000" "False" "" "0" "False" "" "0" "0" "" "" "False" "False" "True" "True" "" "" "False"
I need to do some post processing with PHP on this data, so I assumed I could just strip the quotes out with my code. The quotes were not trimmed using the trim() function (it just wouldn't work), so I str_replaced the quotes and this did seem to work, however:
I am remapping the data and inserting it into another mySQL table (with different column names). When inserting the processed data above, not all of it makes it across with the DB query. Take this query for instance:
INSERT INTO tesxf.xf_node (`node_id`,`parent_node_id`,`title`,`description`,`node_name`,`node_type_id`,`display_in_list`)
VALUES
('2123','2281','Container Name','','38064','Forum','1')
The node_id and title etc make it into the new db, but the parent_node_id never does. The value is always inserted as 0. When I copy the query above and run it manually, it inserts the data correctly. Furthermore, if I run the data through mysql_real_escape_string, it comes out like this (Only converted two fields here):
INSERT INTO tesxf.xf_node
(`node_id`,`parent_node_id`,`title`,`description`,`node_name`,`node_type_id`,`display_in_list`)
VALUES
('673','\0\06\08\08\0\0','\0\0S\0t\0o\0r\0y\0 \0W\0r\0i\0t\0i\0n\0g\0 \0-\0 \0F\0o\0r\0u\0m\0\0','','240576','Forum','1')
I have never seen this before. It makes me think that perhaps the data received from the imported table is in some odd format with some hidden characters I can't see? How can I strip these out? Could this be the reason the data didn't import properly in the first place?
I am running SQL server 2012 I think (not too familiar with it). My mySQL instance is running on OSX 10.6 and is usually fine (I have done similar imports using postgres and oracle etc).
Mysql is : mysql Ver 14.12 Distrib 5.0.92, for apple-darwin10.0 (i386) using EditLine wrapper
PHP is: PHP 5.3.26 (cli) (built: Jul 7 2013 18:30:38)
Losing my mind here... I think my next attempt will be piping constructed queries into a text file and then running that manually... I hope there is something simple that I have missed.
CSV files were transferred over FTP in binary mode, and I applied :set ff=unix to them using vim (as I read on here that could be an issue).
I have also attempted to install the SQL Server/freetds wrapper for PHP on a this server + 1 other, but don't get me started on that!!
Have you tried using MySql migration toolkit? If not you may try it. Here is the link MySql Migration Toolkit
Related
I'm trying to import the following file into a MySQL Db:
https://drive.google.com/drive/folders/1WbRdNgqVre3wN4DpJZ-08jtGkJtCDJNQ?usp=sharing
Using the "data import wizard" on MySql Workbench, for some reason I'm getting "218\223 lines imported successfully", whereas the file contains close to 100K.
I tried looking for special chars around lines 210-230, also removing all of them, but still the same happens.
The file is a CSV of Microsoft Bing's geo locations, used in Microsoft Advertising campaigns, downloaded from Microsoft's website (using an ad account there).
I've been googling, reading, StackOverflowing, playing with the file and different import options...
I tried cutting the file into small bits, and the newly created file was completely corrupt somehow...
Encoding seems to be UTF-8, line breaks all "\n". I tried changing them all into "\r\n" using notepad++, but still the same happens.
File opens normally in Excel, looks normal, passes CSVlint.io...
The only weird thing is that the file contains quotes on some of the values but not on the rest (e.g. line 219. Yeah I know it sounds like this would be the problem, but I removed it, and all the rest of the lines with quotes, and it still happens... Also tried loading with ENCLOSED BY ", see below).
I also tried using SQL statements to import:
LOAD DATA LOCAL INFILE 'c:\\Users\\Gilad\\Downloads\\GeoLocations.csv'
INTO TABLE aw_geo_map_bmsl
FIELDS TERMINATED BY ','
(tried also with: ENCLOSED BY '"')
LINES TERMINATED BY '/n'
IGNORE 1 ROWS;
(had to add OPT_LOCAL_INFILE=1 to the connection on Advanced for MySQL Workbench to be allowed access to local files on my computer)
This gives 0 rows affected.
Help?
Epilogue: In the end I just gave up on all these import wizards and did it the old "make your SQL statements from Excel" way.
I imported the CSV data into Excel. Watch out: in this case I found I needed to use a data import wizard from Excel (but that one worked perfectly) to be able to change the encoding to UTF, which Excel 2010 chose as "windows" which was wrong.
After processing the data a bit to my liking, I used the following Excel code:
=CONCATENATE("INSERT INTO aw_geo_map_bmsl (`Location Id`,Name,`Canonical Name`,`Location Type`,Status,`Adwords Location Id`)
VALUES (",
A2,
",""",B2,"""",
",""",C2,"""",
",""",D2,"""",
",""",E2,"""",
",",F2,");")
to generate INSERT statements for every line, then copy-pasted and pasted only values, then pasted into an editor, removed additional quotes that Excel adds, and ran it in MySQL Workbench, which runs it line by line (takes some time), and you can see the progress.
Saved me hours of unsuccessfully playing around with "automatic tools" which fail for unknown reasons and don't give proper logs ootb.
Warning: do NOT do this for unsanitized code as it's vulnerable to SQL injection. In this case it was data from Microsoft so I know it's fine.
I'm running the "LOAD DATA FROM S3" command to load a CSV file from S3 to Aurora MySQL. The command works fine if run it in the Mysql Workbench (it gives me the below exception as warnings though but still inserts the dates fine), but when I run it in Java I get the following exception:
com.mysql.cj.jdbc.exceptions.MysqlDataTruncation:
Data truncation: Incorrect datetime value: '2018-05-16T00:31:14-07:00'
Is there a workaround? Is there something I need to setup on the mysql side or in my app to make this transformation seamless? Should I somehow run a REPLACE() command on the timestamp?
Update 1:
When I use REPLACE to remove the "-07:00" from the time original timestamp (2018-05-16T00:31:14-07:00) it loads the data appropriately. Here's my load statement:
LOAD DATA FROM S3 's3://bucket/object.csv'
REPLACE
INTO TABLE sample
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(#myDate)
SET `created-date` = replace(#myDate, '-07:00', ' ');
For obvious reasons it's not a good solution. Why would the LOAD statement work in the mysql workbench and not in my java code? Can I set some parameter to make it work? Any help is appreciated!!
The way I solved it is by using mysql's SUBSTRING function in the 'SET' part of the LOAD DATA query (instead of the 'replace'):
SUBSTRING(#myDate, 1, 10)
This way the trailing '-07:00' was removed (I actually opted to remove the time as well, since I didn't need it, but you can use it for TIMESTAMPS as well.
When I export the data from excel as a CSV format it encapsulates some data in double quotes.
E.g.
8" becomes "8""". And I believe this operation is trying to get the database to understand the inside quote later on.
but for single quote 8', it keeps the same and this causes problem(see the pic below) while I am importing the csv.
Why not quoting the 8' into "8'" too?
8' becomes ' while importing, while "8'" will result in 8' And not quoting single quote leads some data loss.
question related:
what does quotechar mean in mysql while importing data?
Excel adds extra quotes on CSV export
After doing so many experiments, I finally found a pretty close answer.
Conclusion first:
It is Mysql Workbench's problem. Its import wizard works badly. I test under Navicat for every test data, and Navicat get all things right.
Single quote can caused unexpected behavior.
Test:
By default, Mysql workbench import wizard takes the first row value as column name, while in Navicat, I can configure that.
(All tests files are excel-exported csv utf8 encoding.)
test1:
e.g.: 8'（only 1 record）, without column name.
Mysql: Pop out some unknown error and whatever configuration you change, can't get the original data.
Navicat:works fine.
test2:
e.g.: 8' , only 1 record with column name or have extra records without column name
Mysql: Can handle single quote properly.
Navicat: No problem.
test3:
If single quote exists, for most situation import wizard can't handle double quotes well.
e.g.: Single quote data comes before double quotes data.
Mysql: Fails totally.
Navicat: No problem.
I'm using mysqlimport,which uses LOAD DATA INFILE command. My question is the following: Assume I have --fields-enclosed-by='"', and that I have column with values which have double quoted string, such as "5" object" (which stands for 5 inches). The problem is that when mysql encounter the double quote string after the 5, it treats it as the enclosing character, and things are messed up. How to use mysqlimport with such values? I don't want to just use another character to enclosing, because this other character as well may occur in the data. So what is a general solution for this?
I guess it is will be different this way to import csv.
To solve above issue in another way,
Export or get or convert old data into sql format rather than csv format.
Import the same sql data using mysql command line tool.
mysql -hservername -uusername -p'password' dbname < 'path to you sql imported file.sql'
I have access to a MySQL database hosted on a remote server. I am attempting to migrate this to a local SQLite database. To do this, I am using this script, as suggested by this question. The usage is
./mysql2sqlite mysqldump-opts db-name | sqlite3 database.sqlite
I tried doing exactly that (with no dump options) and sqlite3 returned an error:
Error: near line 4: near "SET": syntax error
So far, I have found that when I only specify one of my tables in the dump options like so
./mysql2sqlite db-name table-B | sqlite3 database.sqlite
It appears to work fine, but when I specify the first table (let's call it table-A) it returns this error. I'm pretty sure it's returning this error because of the output of mysql2sqlite. The 4th line (I guess the 4th logical line, or the command that starts on the 4th actual line) of the dump file looks like this:
CREATE TABLE "Association_data_interaction" (
"id" int(10) DEFAULT NULL,
...
"Comments" text CHARACTER SET latin1,
...
"Experiment" text CHARACTER SET latin1,
"Methods" text CHARACTER SET latin1,
...
);
With many other rows removed. I don't really know SQL that well, but as far as I can tell, the migration script is trying to output a dump file with commands that can create a new database, but the script has to translate between MySQL's output commands and the commands sqlite3 wants to create a database, and is failing to properly handle the text fields. I know that when I run SHOW COLUMNS; in the MySQL database the Comments, Experiment, and Methods columns are of the "text" type. What can I do make sqlite3 accept the database?
Note: I have editing access to the database, but I would much prefer to avoid that if at all possible. I do not believe I have administrative access to the database. Also, if it's relevant, the database has about 1000 tables, most of which have about 10,000 rows and 10-50 columns. I'm not too interested in the performance characteristics of the database; they're currently good enough for me.
That script is buggy; one of the bugs is that it expects a space before the final comma:
gsub( /(CHARACTER SET|character set) [^ ]+ /, "" )
Replace that line with:
gsub( /(CHARACTER SET|character set) [^ ]+/, "" )