MySQL data import auto-increment issue - mysql

I am attempting to import data into a table that has a field as follows:
result_id
This field is set to AUTO_INCREMENT, PRIMARY and UNIQUE.
The data I am importing has information in the result_id field that is the same (in places) as the current data in the table. SQL won't let me import as there are duplicates (which is fair enough).
Is there a way to get SQL to append the data I am importing and not use the duplicate data in result_id, basically to continue the number within the SQL field. The reason I am asking is that I am importing about 25,000 records and I don't want to manually have to remove or alter the result_id information from the data being imported.
Thanks,
H.

How are you importing your data to MySQL?
If you are using SQL queries/script, then there should be something like INSERT INTO.... Open the file in some text editor and replace all INSERT by INSERT IGNORE. This will ignore inserting rows with duplicate primary keys.
Or alternatively if you want to replace older data with same primary keys to that in your import script, then simply use REPLACE query in place of INSERT query.
Hope it helps...
[EDIT]
Since you have Primary key, auto increment. In your table in which you want to import data, add a dummy column say "dummy" and allow it to be NULL. Now, in your import script there will be statement like INSERT INTO () values (). Now in the list of column names replace "result_id" by "dummy" and execute the script.
After executing script simply remove "dummy" column from table. Though it is bit dirty and time consuming but will do your work.

Related

Auto increment column in mysql when importing with phpmyadmin

I have a table in mysql with the following headings:
staff_id,dept_id,role_id,username,firstname,lastname,passwd,backend,email,phone,phone_ext,mobile,signature,lang,timezone,locale,notes,isactive,isadmin,isvisible,onvacation,assigned_only,show_assigned_tickets,change_passwd,max_page_size,auto_refresh_rate,default_signature_type,default_paper_size,extra,permissions,created,lastlogin,passwdreset,updated
staff_id is a primary key value and is set to AUTO_INCREMENT.
I found the solution by Queue in this post really helpful, although when the data is being imported the staff_id column is not being auto-incremented. I am inserting the column names in the Format-Specific Options in phpmysql. I can only get it to populate if the staff_id value exists in the csv file I am trying to import.
So if the data looks like this:
2,1,1,agent,Mista,Busta,NULL,NULL,agent#company.org,,NULL,,,NULL,NULL,NULL,<p>this is an agent; mista busta; agent#company.org</p>,1,0,1,0,0,0,0,0,0,none,Letter,"{""def_assn_role"":true}","{""user.create"":1,""user.delete"":1,""user.edit"":1,""user.manage"":1,""user.dir"":1,""org.create"":1,""org.delete"":1,""org.edit"":1,""faq.manage"":1}",2020-02-04 10:18:42,NULL,NULL,2020-02-04 10:18:42
...note the first '2' is the staff_id. What I would like do is have this in the csv:
,1,1,agent,Mista,Busta,NULL,NULL,agent#company.org,,NULL,,,NULL,NULL,NULL,<p>this is an agent; mista busta; agent#company.org</p>,1,0,1,0,0,0,0,0,0,none,Letter,"{""def_assn_role"":true}","{""user.create"":1,""user.delete"":1,""user.edit"":1,""user.manage"":1,""user.dir"":1,""org.create"":1,""org.delete"":1,""org.edit"":1,""faq.manage"":1}",2020-02-04 10:18:42,NULL,NULL,2020-02-04 10:18:42
...leaving the staff_id column blank and allowing mysql to auto-populate (auto_increment) as it sees fit.
AUTO INCREMENT is set at the server level within MySQL, and can be overwritten if assigned manually within a CSV or other import. If you simply exclude the entire column from your import, you'll allow MySQL to do what is set as default for that column, and automatically assign ID's, since it doesn't think you want to assign them yourself.
Also as a side note, if you import more than once without using TRUNCATE TABLE -- MySQL will pick up on the last inserted ID and move on from there, even if the table is empty. So if you ever want to start over from 1 you'll have to trucate the table.

MySQL import - Can I have INSERT OR UPDATE together

If I do a SQL dump, I only get one option to choose insert, update, or replace for the queries. Can I have it INSERT if not exists otherwise UPDATE?
Probably an easy answer, but I'm just not sure.
I believe REPLACE will work for you (replace mysqldump option), since it behaves like this: If a row exists in the destination table that matches the row data in the REPLACE statement (based on PRIMARY KEY or UNIQUE KEY value(s)), that data in that row is deleted and replaced with the data in the source (dump file) row. If there is no such match, the source row is inserted into the destination table.

mysql: import csv in existing table

Imagine you have a CSV file with only text in it and line ending \r\n.
like this:
abc
def
hij
...
xyz
You want to import this file in an existing multi-column table, where each line of text needs to go in a line of one specific (currently empty) column (lets name it needed) of the table. Like this:
| a | b | c |needed|
|foo|bar|baz|______|<-- abc
|foo|bar|baz|______|<-- def
...
|foo|bar|baz|______|<-- xyz
The data from the CSV file does not need to be inserted in a certain order. It really does not matter which field of needed has which data from the CSV in it, as long as every line gets imported, everything is fine.
I've tried lots of things and its driving me mad, but I can't figure out, how this could be done. Can this be solved somehow with LOAD DATA INFILE and update/replace/insert command? What would you do in a case like this? Is it even possible with mysql? Or do I need a custom php script for this?
I hope the question is clear enough. Please ask, if something is unclear to you.
OK, here you go...
Add a new column to stuff populated with 1-600,000
ALTER TABLE stuff ADD COLUMN newId INTEGER UNIQUE AUTO_INCREMENT;
Create a temp table for your CSV file
CREATE TABLE temp (
id INTEGER PRIMARY KEY AUTO_INCREMENT,
data VARCHAR(32)
);
I'm guessing the required length of the data.
Import into the temporary table
LOAD DATA INFILE <wherever> INTO TABLE temp;
Add the data to stuff
UPDATE stuff AS s
JOIN temp AS t ON t.id=s.newId
SET s.needed=t.data;
Tidy up
DROP TABLE temp;
ALTER TABLE stuff DROP COLUMN newId;
I must admit that I haven't tested the LOAD DATA INFILE or the UPDATE statements, but I have checked all the table fiddling. In particular, MySQL will populate the newId column with consecutive numbers.
I don't think there will be a problem if the number of rows in stuff doesn't match the number of lines in your csv file - obviously, there will be a few rows with needed still NULL if there are fewer lines than rows.
Most easiest way I found was with Libreoffice Calc+Base.
Import/Export Data in Base.
But becarefull,if you have "not null" options in columns and by mistake there is a cell which has no data in it, that row will be skiped.
So first disable the not null options from columns.
Oh and there is Microsoft Office Excel way,but I did not done it since it required a plugin to install and I was lazy.

LOAD DATA INFILE and ON DUPLICATE KEY UPDATE

When using LOAD DATA INFILE, is there a way to get the same functionality provided by ON DUPLICATE KEY UPDATE of regular INSERT statements?
What I want to do is: for each line of my file, if the row doesn't exist, a new row is inserted, otherwise the selected fields are updated.
My table has 5 columns: A, B, C, D and E. A is the primary key. Sometimes, I have to insert new rows with all the values, but sometimes I have to update only B and C, for example. But, the point is that I want to regroup all the INSERT or UPDATE in the same file.
Thanks
If you want to insert/update some of fields, then you should load data into additional table, and then use INSERT, UPDATE or INSERT...SELECT+ON DUPLICATE KEY UPDATE statement to copy/modify data; otherwise other fields will be set to NULL.
The REPLACE option in LOAD DATA INFILE won't help you in this case.
Also, you can use Data Import tool (CSV format) in dbForge Studio for MySQL (free express edition), just choose Append/Update import mode and specify fields mapping in the Data Import wizard.

How to import selected fields in mysql

For example I have a table. With a field Idnumber. And the field has 10 data(ex: 1234, 1235, 1236, etc)
And I have another table, subjects. Is it possible to import the data that is in the table student(field idnumber) and put it on the table subjects(field idnumber)?
So that I won't have to copy and paste the data that is in the source table to the destination table.
I'm talking about phpmyadmin
Perhaps something like this?
INSERT INTO `subjects`(`idnumber`) SELECT `idnumber` FROM `studends`;
The INSERT command can use data provided by a simple SELECT, so you can "copy" data from one table to another using this kind of syntax.