As an example let's say I want to save the lines of code I've written on specific days of different months in a MySQL database using Hibernate. On the 5th of Janurary, I write 100 lines of code. I have not previously written any lines of code on this day, so it simply inserts the data.
A year passes and it's again the 5th of January. I write 50 lines of code this day, and I try to insert it into the database. The ID of day 5 and month 1 already exists, so it can't insert the data. In this case I would like it to simply add the lines of code the the already existing row, so it now says:
DAY. MONTH. LOC.
5. 1. 150.
What I'm currently doing is that every time I want to insert new data, I'm first making a SELECT query to check if the row already exists. If it does I create an UPDATE statement, otherwise I just create an INSERT statement. I'm working with a lot of data and have noticed that the operations involving my database are starting to take a long time. Therefore, I was wondering if there is a better, more efficient way to do this? Help is much appreciated. Thank you.
You seem to be looking for the update... on duplicate key syntax. This would look like:
insert into mytable(day, month, loc)
values(:day, :month, :loc)
on duplicate key update loc = loc + values(loc)
:day, :month and :loc represent the parameters that you give for insert.
For this to work, you need a unique constraint on (day, month) (or make that tuple of columns the primary key of your table).
When an insert occurs that would violate the unique constraint, MySQL goes to the on duplicate key clause, where we add the new loc value to the one already there in the existing row.
Related
If I have a table that has these rows:
animal (primary)
-------
man
dog
cow
and I want to delete all the rows and insert my new rows (that may contain some of the same data), such as:
animal (primary)
-------
dog
chicken
wolf
I could simply do something like:
delete from animal;
and then insert the new rows.
But when I do that, for a split second, 'dog' won't be accessible through the SELECT statement.
I could simply insert ignore the new data and then delete the rest, one by one, but that doesn't feel like the right solution when I have a lot of rows.
Is there a way to insert the new data and then have MySQL automatically delete the rest afterward?
I have a program that selects data from this table every 5 minutes (and the code I'm writing now will be updating this table once every 30 minutes), so I would like to be as accurate as possible at all times, and I would rather have too many rows for a split second than too few rows for the same time.
Note: I know that this may seem like it is unnecessary but I just feel like if I leave too many of those unlikely possibilities in different places, there will be times where things go wrong.
You may want to use TRUNCATE instead of DELETE here. TRUNCATE is faster than DELETE and resets the table back to its empty state (meaning IDENTITY columns are reset to original values as well).
Not sure why you're having problems with selecting a value that was deleted and re-added, maybe I'm missing some context. But if you're wiping the table clean, you might want to use truncate instead.
You could add another column timestamp and change the select statement to accommodate this scenario where it needs to check for the latest value.
If this is for school, I would argue that you need a timestamp and that is what your professor is looking for. You shouldn't need to truncate a table to get the latest values, you need to adjust the thinking behind the table and how you are querying data. Hope this helps!
Check out these:
How to make a mysql table with date and time columns?
Why not update values instead?
My other questions would be:
How are you loading this into the table?
What does that code look like?
Can you change the way you Select from the table?
What values are being "updated" and change in such a way that you need to truncate the entire table?
If you don't want to add new column, there is an other method.
1. At first step, update table in any way that mark all existing rows for deletion in future. For example:
UPDATE `table_name` SET `animal`=CONCAT('MUST_BE_DELETED_', `animal`)
At second step, insert new rows.
On final step, remove all marked rows:
DELETE FROM `table_name` WHERE `animal` LIKE 'MUST_BE_DELETED_%'
You could implement this by having the updated_on column as timestamp and you may even utilize some default values, but let's go with an example without them.
I presume the table would look something like this:
CREATE TABLE `new_table` (
`animal` varchar(255) NOT NULL,
`updated_on` timestamp,
PRIMARY KEY (`animal`)
) ENGINE=InnoDB
This is just a dummy table example. What's important are the two queries later on.
You would simply perform a query to insert the data, such as:
insert into my_table(animal)
select animal from my_view where animal = 'dogs'
on duplicate key update
updated_on = current_timestamp;
Please notice that my_view is your table/view/query by which you supply the values to insert into your table. Also notice that you need to have primary/unique key constraint on your animal column in this example, in order to work.
Then, you proceed with the following query, to "purge" (delete) the old values:
delete from my_table
where updated_on < (
select *
from (
select max(updated_on) from my_table
) as max_date
);
Please notice that you could make a separate view in order to obtain this max_date value for updated_on entry. This entry should indicate the timestamp for your last updated/inserted values in a previous query, so you could proceed with utilizing it in a where clause in order to issue deletion of old records that you don't want/need anymore.
IMPORTANT NOTE:
Since you are doing multiple queries and it's supposed to be a single operation, I'd advise you to utilize it within a single trancations and to utilize a proper rollback on various potential outcomes (i.e. in case of mysql exceptions). You might wish to utilize a proper stored procedure for that.
I was wondering if there is any way to solve this.
So my row has an column of type date which increments with 1 day daily ( untill the end of the respective month ). At the beggining of a new month a new row has to be generated and the update will start again untill the and of that month, and so on..
Here's a way to think about the problem in MySQL's dialect of SQL.
First, you need a function that changes a datestamp into a value that's unique for each month. That is LAST_DAY(datestamp). It generates DATETIME values like 2017-09-30 00:00:00 from arbitrary inputs.
Next, you can exploit MySQL's INSERT ... ON DUPLICATE KEY UPDATE capability. You will create a table months with, say, these columns
month_ending DATETIME
category VARCHAR(20)
sum_of_input INT
Then you make month_ending, category into a unique compound index.
Then you do something like this
INSERT INTO months /* warning! not debugged! */
(month_ending, category, sum_of_input)
VALUES (LAST_DAY(?date), ?category, ?value)
ON DUPLICATE KEY
UPDATE months
SET sum_of_input = sum_of_input + ?value
WHERE month_ending=LAST_DAY(?date)
AND category=?category
However, this has the hallmarks of a big, hard to debug, pain in the neck. It make make more sense to use features inside your ETL system to do this summarizing work.
I archive last years table and create a new table at the beginning of each year. I'd like to find a way to have one multi-year table so I don't have to manually change anything each year. Columns are: row (unique),date(primary),col1,col2,col3.. Users will type data (every column) into a form. I wanted to have a 'years' column that would be populated from year(date) with composite primary key(row,years). I also need primary key (row,year(date)). So each year we could start with row 1 and a new year. I even looked at and tried insert update triggers but I don't think that's the answer. What do you think? Is this too vague?
I did something similar in one of my Firebird databases (you didn't say which db you are using). First of all, you should have a primary key which is generated automatically - this will make life more simple. Secondly, you need one generator per year - or a table which has one field per year.
In this model, you have a table with two fields: year (primary key) and 'curvalue'. Every year you insert a new tuple such as (2012,0) or (2013, 0). You query this table to find the current value of the 'curvalue' field for the given year, then increment the curvalue field and use the incremented value in the tuple which you are inserting.
Using generators guarantees that you will always receive a unique number; using a table as I wrote above could cause problems if one row starts the process and then another row arrives.
Good evening, I have one table, with a timestamp column, then I have a tool to insert into this table N registries. To avoid duplicate information in this table, I've used INSERT IGNORE, but of course as you can imagine with the timestamp attribute always the new row is different. I cannot make a previous search and check the result set in my code, because I'm adding all the queries into a statement batch, that's why I'm using INSERT IGNORE.
So the question would be, it is possible avoid the timestamp column when I'm making a compare between the new row and all the previous already inserted? the first time that used the tool?
Regards!.
Create a unique key on the fields you don't want duplicated, and exclude the timestamp field from that key.
I don't think this is possible as I couldn't find anything but I thought I would check on here in case I am not searching for the correct thing.
I have a settings table in my database which has two columns. The first column is the setting name and the second column is the value.
I need to update all of these at the same time. I wanted to see if there was a way to update these values at the same time one query like the following
UPDATE table SET col1='setting name' WHERE col2='1 value' AND SET col1='another name' WHERE col2='another value';
I know the above isn't a correct SQL format but this is the sort of thing that I would like to do so was wondering if there was another way that this can be done instead of having to perform separate SQL queries for each setting I want to update.
Thanks for your help.
You can use INSERT INTO .. ON DUPLICATE KEY UPDATE to update multiple rows with different values.
You do need a unique index (like a primary key) to make the "duplicate key"-part work
Example:
INSERT INTO table (a,b,c) VALUES (1,2,3),(4,5,6)
ON DUPLICATE KEY UPDATE b = VALUES(b), c = VALUES(c);
-- VALUES(x) points back to the value you gave for field x
-- so for b it is 2 and 5, for c it is 3 and 6 for rows 1 and 4 respectively (if you assume that a is your unique key field)
If you have a specific case I can give you the exact query.
UPDATE table
SET col2 =
CASE col1
WHEN 'setting1'
THEN 'value'
ELSE col2
END
, SET col1 = ...
...
I decided to use multiple queries all in one go. so the code would go like
UPDATE table SET col2='value1' WHERE col1='setting1';
UPDATE table SET col2='value2' WHERE col1='setting1';
etc
etc
I've just done a test where I insert 1500 records into the database. Do it without starting a DB transaction and it took 35 seconds, blanked the database and did it again but starting a transaction first, then once the 1500th record inserted finish the transaction and the time it took was 1 second, so definetely seems like doing it in a db transaction is the way to go.
You need to run separate SQL queries and make use of Transactions if you want to run as atomic.
UPDATE table SET col1=if(col2='1 value','setting name','another name') WHERE col2='1 value' OR col2='another value'
#Frits Van Campen,
The insert into .. on duplicate works for me.
I am doing this for years when I want to update more than thousand records from an excel import.
Only problem with this trick is, when there is no record to update, instead of ignoring, this method inserts a record and on some instances it is a problem. Then I need to insert another field, then after import I have to delete all the records that has been inserted instead of update.