I have a large text file full of INSERT SQL statements that I want to insert into a phpMyAdmin database. The problem I am having is that many of these INSERT statements within this file are identical, resulting in “Duplicate Key” error occuring.
Is there a way to make phpMyAdmin ignore the repeated SQL statements? I have tried running the file through a .vbs script that removes duplicate lines but it failed to deliver.
Logic that I am thinking of so far is the following:
Run the file through a script that removes duplicate lines.
Find a solution in which phpMyAdmin ignores repeated lines.
Has anyone got any other ideas or suggestions on how I could solve this problem?
The easy way is by using INSERT IGNORE statement, but you will not know which record is duplicate.
another way, by create new table like 'table2' with no primary key or unique key, insert all the data into it, then INSERT IGNORE to your main table before, and compare which row are duplicate. Or maybe you can use the COUNT() function to get the duplicate row by.
Related
I'm trying to work with YEAR function on one column in the DB and then add the results to a different table in the DWH.
What am I doing wrong?
INSERT INTO example_dwh1.dim_time (date_year)
SELECT YEAR(time_taken)
FROM exampledb.photos;
When removing the INSERT INTO line, I get the results I want, but I'm not able to insert them into the dwh table.
Thanks for your help!
The following select works, but I don't see the data in the table after the insert:
INSERT INTO example_dwh1.dim_time (date_year)
SELECT YEAR(time_taken)
FROM exampledb.photos;
There is rather broad. Assuming you have no errors in the insert, you might have:
You are incorrectly querying dim_time, so the data is there but your check is wrong.
You are inserting into dim_time in one database but querying it in another.
Assuming you have errors but are missing them, here are some possibilities:
The database does not exist.
The table does not exist.
The column is misnamed.
Other columns are declared NOT NULL.
Triggers defined on the table are preventing the insert.
Unique constraints/indexes on the table are preventing the insert.
Your question does not provide enough information to be more specific. However, it seems highly suspicious to be inserting a bunch of years -- which might include many duplicates -- into a dimension table.
I have two tables:
tableOriginal
tableBackup
They have exactly the same structure.
I want a SQL statement I can run anytime of the day, that will copy all the rows from tableOriginal to tableBackup WITHOUT overwriting items in tableBackup. Basically, this command must synchronize tableBackup with tableOriginal.
How do I do that?
INSERT INTO tableBackup(SELECT * FROM tableOriginal)
As long as there is no issue with primary keys being updated or replaced with new incoming data this should not create an issue for you. However as you already know, backup table will have more data after your command since it did not delete previous data it had
Why don't you delete first all the data in tableBackup, then INSERT the data in tableOriginal to tableBackup
DELETE FROM tableBackup
INSERT INTO tableBackup(SELECT * FROM tableOriginal)
Why do we need to delete first?
Because if we're going to insert unique data into the tableBackup,
next time we insert it will not execute, because we will insert/add some data that is already been there..
Hope you get what I'm trying to say.
I have two INSERT commands, that are useless to me like that because the two sets of rows - the ones that are already in the table, and the ones I have as INSERT commands - are not disjunct. Both commands insert lots of rows, and lots of values.
Therefore I get the duplicate entry error if I want to execute those lines.
Is there any easy way to 'convert' those commands into UPDATE?
I know this sounds stupid, because why do I make INSERT commands, if I want to UPDATE. Just to make it a clear scenario: another developer gave me the script:)
Thanks in advance,
Daniel
EDIT - problem solved
First I created a table and filled it up with my INSERT commands, then I used the following REPLACE command:
REPLACE
INTO table_1
SELECT *
FROM table_2;
This can originally be found at: How can I merge two MySQL tables?
MySQL's REPLACE keyword does this. Simply replace the INSERT keyword in your queries with the word REPLACE and it should update the rows instead of inserting new ones. Please note that it will only work if you're inserting a primary key or unique key column.
You would have to rewrite them to updates by hand. If I encouter such a problem, I query for the count of certain primary key first, if none is found I insert a generic dataset and update it afterwards. By this, new data can be added and already existing data will be updated, and you don't have to differentiate between inserting new data and updating data.
For MySQL, you can use either the INSERT IGNORE or the INSERT ... ON DUPLICATE UPDATE syntaxes. See the MySQL reference manual
You can easily modify your queries to update duplicate rows, see INSERT ... ON DUPLICATE KEY syntax in MySQL
I'm fetching data from a text file or log periodically and it gets inserted in the database every time fetched. Is there a way in MySQL that the insert is only done when the log files are updated or I have to do it using the programming language ? I mean Is there a type of insert that when It sees a duplicate primary key, It doesn't give an error of "Duplicate Entry" .. It just ignore.
Put the fetch in a logrotate postrotate script, and fetch from the just rotated log.
Ignoring duplicates can be done with either INSERT IGNORE OR INSERT .... ON DUPLICATE KEY UPDATE syntax (which will either ignore the lines causing a duplcate unique key, or give you the possibility to alter some values in the existing row.)
I have a Vb form that inserts data into multiple tables and maintains the foreign key using a sope_identity. I am using an insert procedure to deal with the insertion. My problem is that why i insert my values in VB and click the insert button the values in the database are duplicated.
i need to prevent this from happeing. Any ideas please.
You can find your offending code by setting up unique indexes on the tables. This would at the very least help you discover where your code is inserting the duplicates at. Have you considered stored procedures instead of using the code to insert? Not that it will prevent the duplicates from being inserted if you call it twice, but it might help you reduce the possibility of errant data.