I ahve excel sheet with the following columns
Sam/Simon Date Store Customer name Original Order Number
Simon 09/11/2014 Bristol Cr Car 20089691/ 26089697
I need to store these infomration in 2 rows in tables
Simon 09/11/2014 Bristol Cr Car 20089691/
Simon 09/11/2014 Bristol Cr Car 26089697
I need to know the table structure. Split comma separated values from one column to 2 rows in the results and exporting them from excel to mysql.
the actual table structure is as follows.
CREATE TABLE "tblOrderR" (
"intOrderRemedialId" int(10) unsigned NOT NULL AUTO_INCREMENT,
"intOrderId" int(10) unsigned NOT NULL,
"intOrderRemedialGivenPence" int(10) unsigned NOT NULL,
"intRequestedById" smallint(5) unsigned DEFAULT NULL,
"intAuthorizedById" smallint(5) unsigned DEFAULT NULL,
PRIMARY KEY ("intOrderRemedialId"),
KEY "tblOrderRemedial" ("intOrderId"),
KEY "tblOrderRemedial_ibfk_2" ("intRequestedById"),
KEY "tblOrderRemedial_ibfk_3" ("intAuthorizedById"),
CONSTRAINT "tblOrderRemedial_ibfk_1" FOREIGN KEY ("intOrderId") REFERENCES "tblOrder" ("intOrderId"),
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Order Remedial Information';
One way to approach this would be:
1) Split out "Original Order Number" Excel column into two separate columns using TEXT TO COLUMNS
2) Save data in csv format
3) Load data into a staging table that looks something like this
samSimon,date,store,customerName,originalOrder1,originalOrder2
4) Run two inserts into your final table: one with originalOrder1 and a second one with originalOrder2. For example:
insert into tblOrderR (columns)
select samSimon,date,store,customerName,originalOrder1 from stagingTable;
insert into tblOrderR (columns)
select samSimon,date,store,customerName,originalOrder2 from stagingTable;
Very pseudo-code answer but hopefully you get the gist of it!
Just import two times.
First time (example code, you might have to adjust the one or the other thing):
LOAD DATA LOCAL INFILE '/path/to/file/excel.csv'
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\n'
(column1, column2, #my_variable)
SET column3 = SUBSTRING(#my_variable FROM 1 FOR LOCATE('/', #my_variable));
Second time:
LOAD DATA LOCAL INFILE '/path/to/file/excel.csv'
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\n'
(column1, column2, #my_variable)
SET column3 = SUBSTRING(#my_variable FROM LOCATE('/', #my_variable) + 1);
Relevant manual pages:
string functions like substring() and locate()
load data infile
Related
I have a .csv file which look like this:
and that can be downloaded from here.
I create my DB and my table with the following code:
CREATE DATABASE test_schema;
CREATE TABLE test_schema.teams (
teamkey SMALLINT NOT NULL AUTO_INCREMENT,
teamid CHAR(3) NOT NULL,
yearid YEAR(4) NOT NULL,
leagueid CHAR(2) NOT NULL,
teamrank TINYINT(2) NOT NULL,
PRIMARY KEY (teamkey),
UNIQUE KEY teamkey_UNIQUE (teamkey),
KEY teamid_yearid_leagueid_UNIQUE (teamid, yearid, leagueid),
CONSTRAINT check_teamrank CHECK (((teamrank >= 0) and (teamrank <= 12))),
CONSTRAINT check_year CHECK (((yearid >= 1871) and (yearid <=2155))))
ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
Now, when I try to import using:
LOAD DATA LOCAL INFILE "path_to_file_in_my_computer/Teams.csv"
INTO TABLE test_schema.teams
FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(#teamID, #yearID, #lgID, #Rank);
I get 2895 warnings which are all the same:
Warning (Code 3819): Check constraint 'check_year' is violated.
This warning makes no sense since yearid goes from 1871 to 2018 as can be corroborated if you look at the structure of the Teams.csv file. So any advice or suggestion on how to handle this error will be much appreciated. I'm working on MySQL Workbench 8.0.
PS: I posted a similar question (deleted) today morning but it needed more details that are provided here.
You don't have the column names in the correct order in the LOAD DATA query.
LOAD DATA LOCAL INFILE "path_to_file_in_my_computer/Teams.csv"
INTO TABLE test_schema.teams
FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(yearid, leagueid, teamid, #franchid, #divid, teamrank);
You can assign directly to the table column names, you don't need to use #yearID unless you have to do extra processing before storing in the table.
Hello! I am new to MYSQL so kindly explain in as simple language as possible!
I have a csv with 400k rows and want to import it into mysql. I am using LOAD DATA LOCAL INFILE command for this purpose:
LOAD DATA LOCAL INFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/Comorbidity Covid-19.csv'
INTO TABLE `comorbidity covid-19`
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;
The issue is that only about 200k records are being imported while the csv contains 400k records. Why is this happening? I executed the command both in the command prompt and in MySql Workbench but both give the same output. Also the date column is not being imported correctly. Instead of dates being displayed it is showing 0000-00-00 in each rows.
PS: OPT_LOCAL_INFILE=1 in manage database connections!
PS : Here is some sample data
What I did was first I created an empty table in the database with respective column types by. I created an empty table with only the column headers by right clicking on tables and selecting create new table option where I selected the proper type for each columns.. Date as of and Start Date were given Date type and so on. Then I executed the above query both in command prompt and workbench to import the rows.
show create table comorbidity gives this result:
CREATE TABLE `comorbidity` (
`Date as of` date NOT NULL,
`Start Date` date NOT NULL,
`State` varchar(20) NOT NULL,
`Condition group` varchar(50) NOT NULL,
`Condition` varchar(45) NOT NULL,
`Age group` varchar(15) NOT NULL,
`Covid19 deaths` int NOT NULL,
`Number of mentions` int NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
It may be that the date is not in the correct format that is why it looks wrong. Try to modify the field from excel to a correct format, or perform a DATE() function for the date.
On the subject of importing all the records, check if there is any character that interrupts the execution.
The reason only 200k records were being imported was because I was using:
LINES TERMINATED BY '\n'
When I changed it to:
LINES TERMINATED BY '\r\n'
All 400 k records were imported.
I'm working on a new web project right now, but the data is stored in the excel program, I don't want to add them to the list manually, do you think this is possible?
You have some ways of doing it:
You can use load data.
Let's say you have the table below:
CREATE TABLE `set_of_data` (
`id` int NOT NULL AUTO_INCREMENT,
`x` varchar(10) DEFAULT NULL,
`y` varchar(10) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB ;
Your excel file should be in .csv file format :
The you can use load data.
LOAD DATA INFILE '/var/lib/mysql/your_data.csv' ---path of your file in server, it could be '/var/lib/mysql-files/your_data.csv'
IGNORE INTO TABLE set_of_data
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(id,x,y);
Another way is that you can create an excel formula for your data and insert it.
This is for small tables, with not so much data.
I have a CSV file with three columns of "movieId", "imdbId", "tmdbId". The column "tmdbId" contains multiple empty rows.(movieId is a froeign key referring to a primary key in another table)
When I read this data frame into R, the empty rows are treated as NA values. If I import this CSV file into mysql DB using the following command, the rows with NA values don’t get inserted in the table, even though I allow NULL values. I should also mention that, I do not get any errors.
Beside the following command, I also tried importing the dataset using MySQL workbench, but it did not work.
any suggestion?
LOAD DATA LOCAL INFILE 'links.csv' INTO TABLE links
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(movieId, imdbId, tmdbId);
I know NULL and NA values are not the same, but I do not understand why R treats empty rows as NA. I tried to replace NA with NULL, but R does not support this operation.
The TABLE
CREATE TABLE links (
movieId int NOT NULL,
imdbId int DEFAULT NULL,
tmdbId int DEFAULT NULL,
KEY movieId (movieId),
CONSTRAINT links_ibfk_1 FOREIGN KEY (movieId) REFERENCES movieId_title (movieId)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
The CSV file looks like this:
enter image description here
Here is an example of an empty row for the third column:
enter image description here
As #Alec suggested, you can do set foreign_key_checks = 0. Then, you can replace zeros with NULL using the following command:
UPDATE table_name
SET col_name= NULL
WHERE col_name = 0;
I have ASCII files with a static number of characters for each line with no delimiters. I'd like to use LOAD DATA INFILE to import into my table.
Example of file:
USALALABAMA
USARARKANSAS
USFLFLORIDA
The structure for this table:
country Char(2)
state Char(2)
name Varchar(70)
CREATE TABLE `states` (
`country` char(2) COLLATE latin1_general_ci NOT NULL,
`state` char(2) COLLATE latin1_general_ci NOT NULL,
`name` varchar(70) COLLATE latin1_general_ci NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1_general_ci COLLATE=latin1_general_ci;
Is it possible to specify a start and end position for each column?
According to the documentation, you can load a fixed format file without using a temporary table.
If the FIELDS TERMINATED BY and FIELDS ENCLOSED BY values are both empty (''), a fixed-row (nondelimited) format is used. With fixed-row format, no delimiters are used between fields (but you can still have a line terminator). Instead, column values are read and written using a field width wide enough to hold all values in the field. For TINYINT, SMALLINT, MEDIUMINT, INT, and BIGINT, the field widths are 4, 6, 8, 11, and 20, respectively, no matter what the declared display width is.
The positions are derived from the columns definitions, which in your case match the structure of the file. So you just need to do:
LOAD DATA INFILE 'your_file' INTO TABLE your_table
FIELDS TERMINATED BY ''
LINES TERMINATED BY '\r\n'
SET name = trim(name);
First create a temporary table which you will load all lines into it, then you can load the data from the temporary table into the main table and split to fields using substring
Something like this:
CREATE TEMPORARY TABLE tmp_lines
(countrystring TEXT);
LOAD DATA INFILE 'yourfilegoeshere' INTO TABLE tmp_lines
FIELDS TERMINATED BY ''
LINES TERMINATED BY '\r\n';
INSERT INTO main_table SELECT SUBSTRING(countrystring,1,2), SUBSTRING(countrystring,3, 2), SUBSTRING(countrystring,5) from tmp_lines;
Another way to do this is just assigning a variable and splitting it direct in your load.
LOAD DATA INFILE 'yourfilegoeshere' INTO TABLE main_table
LINES TERMINATED BY '\r\n' (#_var)
set
field1=TRIM(SUBSTR(#_var from 1 for 2)),
field2=TRIM(SUBSTR(#_var from 3 for 2)),
field3=TRIM(SUBSTR(#_var from 5 for 70));
Just be sure not to specify any field separator, otherwise you will have to use more variables, note that I'm using TRIM to clean data in the same statement.