Does sql keep a record of composite keys or does it calculate them each time a record is inserted/deleted/updated...?
If it does is there a way to call it without having to get each member field value, something like ...WHERE composite_pk=CONCAT('value1','value2','value3')
That is correct, composite index is updated when field value is changed. But index has to be unique, otherwise MySql won't allow you to save a changed value. (you'll see an error: #1062 - Duplicate entry 'a-b-c' for key 'x')
Index can't be used in WHERE statement.
useful read: http://dev.mysql.com/doc/refman/5.0/en/mysql-indexes.html
Also about index Hinting (use, ignore, force): http://dev.mysql.com/doc/refman/5.1/en/index-hints.html
Related
Background:
I built a scraper in python (not sure if that matters). I scrape the website and update my html table. The main table stores the autogenerated_id, url, raw_html, date_it_was_scrapped, last_date_the_page_was_updated (provided by the website). My table has many duplicate urls which it shouldnt so i am planning on making urls unique in the database.
Desired outcome:
I only want to insert a row if the url doesnt exist and update the html if last_date_the_page_was_updated > date_it_was_scrapped.
Solution:
The following stackoverflow post shows how.
I havent tested it because of the selected answers warning: INSERT ... ON DUPLICATE KEY UPDATE statement against a table having more than one unique or primary key is also marked as unsafe.
What I plan to do based on the stackoverflow question.
INSERT INTO html_table (url, raw_html, date_it_was_scrapped, last_date_the_page_was_updated)
VALUES (the data)
ON DUPLICATE KEY UPDATE
url = VALUES(url),
raw_html = VALUES(raw_html),
date_it_was_scrapped = VALUES(date_it_was_scrapped),
last_date_the_page_was_updated=VALUES(last_date_the_page_was_updated)
WHERE last_date_page_was_update > date_it_was_scrapped
Question:
What is unsafe about it and is there a safe way to do it?
From the description of bug 58637, which is linked in the MySQL documentation page that flags the INSERT ... ON DUPLICATE KEY UPDATE as unsafe :
When the table has more than one unique or primary key, this statement is sensitive to the order in which the storage engines checks the keys. Depending on this order, the storage engine may determine different rows to mysql, and hence mysql can update different rows [...] The order that the storage engine checks keys is not deterministic.
I understand that your table has an autoincremented primary key, and that you are planning to add a unique key on the url column. Because the primary key is autoincremented, you will not pass it as a parameter for INSERT commands, as shown in your SQL command. Hence MySQL will not need to check for duplicate on this column ; it will only check for duplicates on url. As a consequence, this INSERT should be safe.
Other remarks regarding your question.
you don't need to update the url command on duplicate keys (we know it is the same)
The purpose of the WHERE clause in your query is unclear, are you sure that it is needed ?
You will need to remove the duplicates before you enable the unique constraint on URL.
When I add a primary key to a table that already has data, what does each of these database management systems do?
Do they analyze each and every value of the column to confirm it is unique ?
Or do they have some other optimized mechanism ? And if that's the case, what is that mechanism ?
Yes, at least in PostgreSQL and MySQL (probably MonetDB too) the DBMS will first check if all values are unique (like when you use a "unique" parameter in your sql query). You can simulate it by counting all rows and then counting a "unique" select of the same rows. If the row numbers are not equal, you will not be able to create the primary key.
An index really is created, but only to speed things up when you use the primary key after its created.
I have a table that serves as a foreign key lookup from another table. The table is very simple, containing a ID column with is the primary key and a JSON column. I wish to remove abandoned entries from this table.
I tried running this script:
DELETE
FROM `ate`.`test_configuration`
WHERE `ate`.`test_configuration`.`ID` NOT IN (SELECT DISTINCT `ate`.`index`.`TestID` from `ate`.`index`);
But encountered an error stating my I wasn't using a where clause that uses the key column:
Error Code: 1175. You are using safe update mode and you tried to update a table without a WHERE that uses a KEY column To disable safe mode, toggle the option in Preferences -> SQL Editor and reconnect.
This is confusing as my where clause does use the primary key column. I am aware that I can disable safe mode as part of my script as a workaround, but would still like to understand why I'm getting this error. I'd like to avoid unsafe updates if possible.
I believe Optimizer just unable to use index effectively for such query - so it does full table scan.
How many rows are in the test_configuration and how many of them will be deleted?
(You might try to use index hints to force optimizer to use index for the query, just not sure if they are supported in your version of mysql).
is it necessary to check for Unique Value before insert it in to a database? if the unique_col is predefine to be Unique Keys.
for example
SELECT unique_col FROM table WHERE unique_col != unique_val
INSERT INTO table (unique_col) VALUE(:unique_value)
Is it necessary to check? That depends how you are handling the error.
In general, the database is going to do the check anyway, so an additional check on your part is redundant. If you do the check, another thread might insert the same value between your check and the insert, so you can still get an error (this is called a race condition).
So, don't do the check, but do check for the error.
So the table schema looks like:
field - type - index
'id' - INT - PRI
'event_id' - BIGINT(20) - UNI
co1 ... colN (no indexes)
LOAD DATA command:
LOAD DATA LOCAL INFILE 'my-file' REPLACE INTO TABLE mydb.mytable (event_id, col1 .. colN)
and get error:
ERROR 1062 (23000) at line 1: Duplicate entry '2147483647' for key 1
'key 1' refers to the unique key on 'event_id'.
More context:
(1) The table is empty at the time of LOAD DATA.
(2) When I grep for 2147483647 in my source file I get nothing
(3) 2147483647 is the integer max value
(4) I am not actually inserting any value in 'id' -- its just auto incrementing
(5) I am using the 'REPLACE' keyword in the LOAD DATA so even there were dupes, it should know how to deal with them?
This suggest some int overflow issue (ie, I don't think there are any genuine dupes in the source data or in the table), and indeed the values in 'my-file' for 'event_id' are all over the integer max limit. However, the error is odd because 'event_id' column is BIGINT.
As a temporary fix, I dropped the unique index on 'event_id' and the LOAD DATA command worked! The values in 'event_id' were all fine, not truncated or anything. So there is nothing wrong with the table handling the values, but somehow LOAD DATA is checking the uniqueness but as an integer?
Has anyone encountered something similar? Thanks
So what this means is that '2147483647' already exists in that database in a field marked as a key. Just remove the key from that field and you should be fine!
Edit: As stated in your question, event_id is your primary key - you cannot have the same value twice in a primary key.
Best of luck!
the problem is not the data type,
the thing is that you only have 2 fields and one is PK and the other is UNIQUE, so there is no way that you could have repeated a value. When you make an insert or load data it's trying to add twice an entry with this value "2147483647", you have several ways to fixed it.
The first one try to open the file with a text editor and find the repeated value and fixed it, if it's not working try with a mysqldump or phpmyadmin to export the data and edit the file in a text editor.
Make an export again, if the problem persist try export the data by others methods (mysqldum or phpmyadmin) and re import it
Try recreating a table without primary keys, you can use this
create table big_int_test (select * from table1);
and it will make a copy of the table without the PK, INDEXES and FK.
You can delete the index and UNIQUE key constraint, import the data, fix the table (delete the repeated values) and re create the PK and UNIQUE KEY (this is a kind of crazy but It could work)