Hello! I am new to MYSQL so kindly explain in as simple language as possible!
I have a csv with 400k rows and want to import it into mysql. I am using LOAD DATA LOCAL INFILE command for this purpose:
LOAD DATA LOCAL INFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/Comorbidity Covid-19.csv'
INTO TABLE `comorbidity covid-19`
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
The issue is that only about 200k records are being imported while the csv contains 400k records. Why is this happening? I executed the command both in the command prompt and in MySql Workbench but both give the same output. Also the date column is not being imported correctly. Instead of dates being displayed it is showing 0000-00-00 in each rows.
PS: OPT_LOCAL_INFILE=1 in manage database connections!
PS : Here is some sample data
What I did was first I created an empty table in the database with respective column types by. I created an empty table with only the column headers by right clicking on tables and selecting create new table option where I selected the proper type for each columns.. Date as of and Start Date were given Date type and so on. Then I executed the above query both in command prompt and workbench to import the rows.
show create table comorbidity gives this result:
CREATE TABLE `comorbidity` (
`Date as of` date NOT NULL,
`Start Date` date NOT NULL,
`State` varchar(20) NOT NULL,
`Condition group` varchar(50) NOT NULL,
`Condition` varchar(45) NOT NULL,
`Age group` varchar(15) NOT NULL,
`Covid19 deaths` int NOT NULL,
`Number of mentions` int NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
It may be that the date is not in the correct format that is why it looks wrong. Try to modify the field from excel to a correct format, or perform a DATE() function for the date.
On the subject of importing all the records, check if there is any character that interrupts the execution.
The reason only 200k records were being imported was because I was using:
LINES TERMINATED BY '\n'
When I changed it to:
LINES TERMINATED BY '\r\n'
All 400 k records were imported.
I'm working on a new web project right now, but the data is stored in the excel program, I don't want to add them to the list manually, do you think this is possible?
You have some ways of doing it:
You can use load data.
Let's say you have the table below:
CREATE TABLE `set_of_data` (
`id` int NOT NULL AUTO_INCREMENT,
`x` varchar(10) DEFAULT NULL,
`y` varchar(10) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB ;
Your excel file should be in .csv file format :
The you can use load data.
LOAD DATA INFILE '/var/lib/mysql/your_data.csv' ---path of your file in server, it could be '/var/lib/mysql-files/your_data.csv'
IGNORE INTO TABLE set_of_data
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(id,x,y);
Another way is that you can create an excel formula for your data and insert it.
This is for small tables, with not so much data.
It has been here already, but I did not find the answer.
I am trying to insert data from a csv file into mysql table customers using LOAD DATA INFILE command
I have used many modification of this syntax, but it always returns 0 rows affected. Where do I make mistake? This is my syntax (running MySQL 8 Server on Windows Server 2017 enviroment)
LOAD DATA INFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/customers.csv' INTO TABLE customers
CHARACTER SET UTF8MB4
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES (ID,ClientNo,CustomerNo,LastName,FirstName,PhoneNumber,FaxNumber,PhoneNumberPrivate,MobileNumber,Email);
The CSV file looks like this:
ID,ClientNo,CustomerNo,LastName,FirstName,PhoneNumber,FaxNumber,PhoneNumberPrivate,MobileNumber,Email
4911556,19,540789,Name1,Name2,00403651205,,,+403651205,luboxxxx5#axxxt.com
4911560,19,540790,Name52,Name56,015344492,,,+15344492,alxx.ss#gmail.com
4911561,19,540791,Name222,Name58,0904385448,,,4385448,
4911562,19,540792,Name547,Name555,7314727,,,+07314727,
4911967,5,1311473,Name789,Name562,+444839,,,,semxxxx#cxxxxrum.com
The table I want to insert looks like this:
CREATE TABLE 'customers' (
'ID' int(11) DEFAULT NULL,
'ClientNo' int(11) DEFAULT NULL,
'CustomerNo' int(11) DEFAULT NULL,
'FirstName' varchar(50) DEFAULT NULL,
'LastName' varchar(50) DEFAULT NULL,
'PhoneNumber' varchar(50) DEFAULT NULL,
'FaxNumber' varchar(50) DEFAULT NULL,
'PhoneNumberPrivate' varchar(50) DEFAULT NULL,
'MobileNumber' varchar(50) DEFAULT NULL,
'Email' varchar(50) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8
I really do not know now, where and what. It will be some detail what I missed I suppose, but I am not able to see it now. Could someone shortly look please?
For me the problem was that the column names in the first row were quoted and the data was imported with FIELDS ENCLOSED BY '"' (and ESCAPED BY '"'). Actually, I found that any single quote would lead to the "0 rows affected" result.
Not exactly as described in the question but probably related.
I was stuck on the same issue. I think it has something to do with numbers that start with 0. My column (zipcode) was indeed set to VARCHAR but to no use.
But importing only worked after I opened the CSV in excel which automatically got rid of 0s at the beginning of the number. So now, instead of ["01234"], [1234] showed in the CSV (opened in notepad, used [ ] just to indicate where the value starts and ends).
Luckily for me, I could easily CONCAT("0", zipcode) WHERE zipcode<100000 to restore the true data.
So I suggest maybe finding a way to get rid of the 0 at the beginning or replacing it with a character and adding the 0 afterwards? That's all I got ¯_(ツ)_/¯ (doesn't work, read edit)
Edit: I tried replacing the 0 in front with a $ (did it in excel) and got rid of the " in the CSV file, but it still did not work, 0 rows affected. I'm super confused. I also tried removing special characters like "ã" but it made no difference.
I am trying to import CSV data into MySQL using the LOAD DATA LOCAL INFILE syntax. This is normally a fairly simple task, but in this case the data includes a geometry field that is tripping me up.
When I try to run the import, I'm getting errors like this:
SQLSTATE[HY000]: General error: 4079 Illegal parameter data type longblob for operation 'st_geometryfromwkb'
The records in my CSV file look like this:
'Somewhere', -0.574823, 51.150771, '0x0101000000000000000000F03F000000000000F0BF'
So I have a location name, lat/long coords and a geometry field in binary WKB format. (the example above is a simple geometry that translates to POINT(1,1); the real data has complex polygons, but the content isn't relevant; the issue is the same with this simple example).
My table looks like this:
CREATE TABLE IF NOT EXISTS `mapping` (
`id` int AUTO_INCREMENT PRIMARY KEY,
`location` varchar(80) DEFAULT NULL,
`longitude` double DEFAULT NULL,
`latitude` double DEFAULT NULL,
`geom` geometry NOT NULL,
INDEX mapping_by_location (location),
SPATIAL KEY `mapping_by_geom` (`geom`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
And my import query looks like this:
LOAD DATA LOCAL INFILE '{$file}'
REPLACE INTO TABLE `mapping`
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
(#col1, #col2, #col3, #col4)
SET
`location` = #col1,
`latitude` = #col2,
`longitude` = #col3,
`geom` = GeomFromWKB(#col4);
As stated, with this import query, I am getting the Illegal parameter data type error shown at the top of this question.
However, the query works if I replace the final line with a hard-coded geometry string, like this:
`geom` = GeomFromWKB(0x0101000000000000000000F03F000000000000F0BF);
Obviously this isn't any good, as I need the field to load from the CVS not a hard-coded value in the query, but it does work, whereas loading the same value from the CSV in #col4 does not.
I have tried a bunch of variations on this query - with and without the call to GeomFromWKB(), with both X'...' and 0x... notations for the hex value; nothing seems to work.
Can anyone give me some help please?
Problem: above mentioned load data query inserting 0 instead of null for price field(column)
Mysql Query :
LOAD DATA LOCAL INFILE '/tmp/data.csv'
REPLACE INTO TABLE bug_repeat
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
file data.csv content is as below
name,\N
name,3
bug_repeat table structure:
CREATE TABLE `bug_repeat` (
`name` varchar(10) DEFAULT NULL,
`price` decimal(12,6) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Note: above query is not working on one of MYSQL server but exact same query working fine on two other MYSQL server. I don't know what going wrong. Can some please let me know what is the exact issue.( all MYSQL servers(version 5.7.22) are on Ubuntu 16.xxx OS). I am getting same problem for bigint data type as well.
show warnings result:
1265 (01000): Data truncated for column 'price' at row 1