I'm trying to import csv file to MYSQL, and I have the following schema.
CREATE TABLE `monitor` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`time` time DEFAULT NULL,
`domain_name` text,
`cpu_ns` int(11) DEFAULT NULL,
`cpu_percentage` int(11) DEFAULT NULL,
`mem_bytes` int(11) DEFAULT NULL,
`mem_percentage` int(11) DEFAULT NULL,
`block_rdby` int(11) DEFAULT NULL,
`block_wrby` int(11) DEFAULT NULL,
`net_rxby` int(11) DEFAULT NULL,
`net_wrby` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
I'm having issues importing a file with data presented as follows.
16:48:27,2,s-1-VM,220000000.,0.448204684384,262144,0,0,0,60,0
16:48:30,2,s-1-VM,260000000.,0.528932926209,262144,0,0,16384,300,0
16:48:33,2,s-1-VM,300000000.,0.609786677944,262144,0,0,0,180,0
16:48:37,2,s-1-VM,290000000.,0.59000206364,262144,0,0,16384,120,0
16:48:40,2,s-1-VM,270000000.,0.54985661784,262144,0,0,0,649,425
16:48:43,2,s-1-VM,310000000.,0.631207212346,262144,0,0,0,180,0
16:48:46,2,s-1-VM,220000000.,0.44728232907,262144,0,0,20480,60,0
16:48:49,2,s-1-VM,200000000.,0.407008216196,262144,0,0,0,300,0
16:48:52,2,s-1-VM,250000000.,0.508946559213,262144,0,0,0,240,0
16:48:55,2,s-1-VM,240000000.,0.488674160215,262144,0,0,0,120,0
How can import this to my database?
I have tried the following and I get lots of warnings.
LOAD DATA LOCAL INFILE '/tmp/domain2.csv' INTO TABLE vtop FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n';
your help is highly appreciated.
Thank you
If I understand http://dev.mysql.com/doc/refman/5.1/de/load-data.html correctly, you should set ID to null (that makes it auto_increment) via
"SET id=NULL "
at the end of the statement. Otherwise column counts and column orders have to match perfectly.
But your columns don't match at all (what is the "2" at position 2?). So create a temp-Table with the structure of your CSV and then assign via insert into ... select the matching columns.
MySQL isn't an AI and can't figure out that 16:48:27 should go into the the time field - it'll be trying to stuff that into id instead.
You need to explictly map the columns in your CSV file to the fields they should go into in the table:
LOAD DATA .... (time, domain_name, x, y, z, foo, bar)
Related
I am trying to import data from a text file into table in mysql-5.5. Here is the table create code:
CREATE TABLE `price` (
`symbol` VARCHAR(20) NOT NULL,
`date` DATE NOT NULL,
`open` DOUBLE NULL DEFAULT NULL,
`high` DOUBLE NULL DEFAULT NULL,
`low` DOUBLE NULL DEFAULT NULL,
`close` DOUBLE NULL DEFAULT NULL,
`volume` BIGINT(20) UNSIGNED NULL DEFAULT NULL,
PRIMARY KEY (`date`, `symbol`)
)
ENGINE=InnoDB
;
Here is the import code.
INFILE 'c:/temp/DTB3.csv'
INTO TABLE price
FIELDS TERMINATED BY ','
(symbol, date, #dummy, #dummy, close);
Here are some rows of data.
10,2019-01-30,1.00,1.00,50459.4301
10,2019-01-31,1.00,1.00,50477.9307
10,2019-02-01,1.00,1.00,50496.4382
Here is a sample warning:
Warning 1,265 Data truncated for column 'close' at row 12
The data in the resulting table looks correct.
I have tried both double and float for the price fields.
Why am I getting these warnings and what can I do about them?
My best guess is that you have declared 7 columns in your table but giving data for only 5.
Try adding commas for the missing (null) fields in your import data so the import knows explicitly there's no value to load.
Like this:
10,2019-01-30,1.00,1.00,50459.4301,,
10,2019-01-30,1.00,1.00,50459.4301,,
10,2019-01-30,1.00,1.00,50459.4301,,
In case anyone else stumbles across this problem amending the query to add:
LINES TERMINATED BY '\r\n'
stopped the warnings. Apparently MySQL was trying to append the carriage return to the close field on import.
under following sample im trying to import CSV data into MySQL table,
LOAD DATA INFILE 'file1.csv'
INTO TABLE loc_chg_log
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(#col1,#col2,#col3,#col4,#col5) set failename=#col1,path=#col2,==blank=#col3,!=blank=#col4,++code=#col5;
MySQL table query is like this,
CREATE TABLE `log`.`loc_chg_log` (
`id` INT NOT NULL AUTO_INCREMENT,
`filename` VARCHAR(225) NULL,
`path` VARCHAR(225) NULL,
`==blank` VARCHAR(45) NULL,
`!=blank` VARCHAR(125) NULL,
`++code` VARCHAR(45) NULL
PRIMARY KEY (`id`));
none of these methods work's since the column name includes special character's, im getting error "unknown column'==blank' in field list"
1. '==blank'=#col5
2. ("`!=blank`=#col7")
3. [==blank=#col5]
by '==blank'=#col5 putting backticks in the column name, I was able to write on data in MySQL database. thank you
I am getting a "Failed to read auto-increment value from storage engine" error from MySQL (Percona), after trying to:
LOAD DATA LOCAL INFILE
I have been trying to find an answer on stackoverflow and google, but the answers did not solve the problem/error for my case.
The following shows Auto_increment as 1:
show table status like 'my_table_name';
Setting the auto increment value to the next value or to 1 did not solve it:
ALTER TABLE `my_table_name` AUTO_INCREMENT = 1;
Dropping the auto incrementing id column and recreating it did not solve it.
Dropping the table and recreating it did not solve it.
Dropping the database and recreating it did not solve it.
Destroying all data and rebuilding my docker development environment did not solve it.
Check table extended gave "OK":
CHECK TABLE distributor_users EXTENDED;
And it is the only auto incrementing value in the table.
This is what I use to create the table:
CREATE TABLE `my_table_name` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`x0` varchar(191) DEFAULT NULL,
`x1` varchar(4) DEFAULT NULL,
`x2` varchar(191) DEFAULT NULL,
`x3` varchar(191) DEFAULT NULL,
`x4` varchar(191) DEFAULT NULL,
`x5` varchar(191) DEFAULT NULL,
`x6` DATE DEFAULT NULL,
`x7` int(10) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `x0` (`x0`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
What could it be? Any lead to start looking at is highly welcomed!
I solved it myself by adding to my query:
SET id = NULL
The new query now looks like this:
LOAD DATA LOCAL INFILE '$csv_file_location' INTO TABLE $destination_table
FIELDS TERMINATED BY ','
ENCLOSED BY '\"'
IGNORE 1 LINES
(x0,x1,x2,x3,x4,x5,x6,x7)
SET id = NULL;
Although my query worked previously without it.
MySQL was trying to place the first column in the csv file into the first column of the database. Setting the column that has the auto_increment to NULL solved it.
How can I import data from a text file into a database without giving a primary key in the text file?
So I have a table where I have 3 columns: ID, firstName, lastName.
ID is auto incremented. I would like to read in the names from the text file like that:
John, Smith;
Michael, Jordan;
I don't want to use the primary key, as I don't know what will be the next primary key in the table, that should be done by auto increment.
If I use the text file like this, than I get the error message: Invalid column count...
The settings:
Columns separated with: ,
Columns enclosed with: "
Columns escaped with: \
Lines terminated with: ;
If I use the text file like this:
21, John, Smith; 22, Michael, Jordan;
The file can be imported (with the strange behavior that it tries to read the 3 empty line too, and sends an error, this one I don't understand either, but its a different topic)
This is the dump from the table:
CREATE TABLE IF NOT EXISTS `LoginData2` (
`FirstName` varchar(10) NOT NULL,
`LastName` varchar(10) NOT NULL,
`ID` int(4) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`ID`),
UNIQUE KEY `ID` (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
Your spec says that Columns enclosed with: "
I added quotes around the columns (and took away the spaces):
"John","Smith";"Michael","Jordan";
AND put the auto increment ID column last. It imported fine with these settings.
This is a dump from my test table. Compare it with yours and see what is different. Also, try creating this table and import the data above to see how it works.
CREATE TABLE IF NOT EXISTS `users` (
`firstname` varchar(100) NOT NULL,
`lastname` varchar(100) NOT NULL,
`id` int(11) NOT NULL
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8;
ALTER TABLE `users`
ADD PRIMARY KEY (`id`);
ALTER TABLE `users`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT,AUTO_INCREMENT=7;
UPDATE: When you import with phpMyAdmin, make sure the import setttings are correct. Your settings are not the default. I believe you need to choose csv using LOAD DATA and then fill in all the delimiters as you have stated.
CREATE TABLE `tblspmaster` (
`CSN` bigint(20) NOT NULL AUTO_INCREMENT,
`SP` varchar(50) NOT NULL,
`FileImportedDate` date NOT NULL,
`AMZFileName` varchar(580) NOT NULL,
`CasperBatch` varchar(50) NOT NULL,
`BatchProcessedDate` date NOT NULL,
`ExpiryDate` date NOT NULL,
`Region` varchar(50) NOT NULL,
`FCCity` varchar(50) NOT NULL,
`VendorID` int(11) NOT NULL,
`LocationID` int(11) NOT NULL,
PRIMARY KEY (`CSN`)
) ENGINE=InnoDB AUTO_INCREMENT=20018215 DEFAULT CHARSET=latin1;
What is the meaning of AUTO_INCREMENT=20018215 here in table schema . as i am inserting 500k records my identity is OK from 1 to 500k but when i tried to insert next 500k records, next records identity column value is 524281 instead of 500001.
It means that the first auto-assigned value (to CSN) will be 20018215
The large initial value, 20018215, was probably the previous value of the auto increment when you did a "Send to SQL Editor" -> "Create Statement" menu selection in MySQL Workbench. This is just a safe value to skip over existing data just in case you have to reimport the previous records.
I had the same question, but after generating several "Create" edit templates from known tables, I noticed the AUTO_INCREMENT value corresponded to the quantity of existing records in those tables. I removed the large values from my templates since I want my new tables to begin with a primary key = 1.