I have n (source) tables with the same structure that each have few million rows. Each of these table receives new data from different sources on a regular basis.
(Ex: Sales table. Each store have its own sales table. There's 1000 stores selling hundred of thousands items each day. How would you combine those tables?)
I would like to merge them in one summary table. I would like the changes from any of the source tables to be reflected on the summary and changes on the summary to be reflected on the appropriate source table.
(Ex: Sales table. When new sales occurs, the summary table is updated. If a changes to the sale is made in the summary table, it is reflected on the appropriate store table.)
I can see three solutions.
1.Create an event/trigger that would refresh my summary tables at a given time or after an insert/update/delete.
Something like:
#Some event triggers this
DROP TABLE table_summary;
INSERT INTO table_summary
SELECT * FROM table1
UNION ALL
SELECT * FROM table2
UNION ALL
SELECT * FROM tablen...
The downside here, I believe, is performance, I do not think I can afford to run this query every time there is an INSERT/UPDATE/DELETE on one of the table.
2.Create a view.
CREATE VIEW table_summary AS
SELECT * FROM table1
UNION ALL
SELECT * FROM table2;
#This query takes 90s to complete
Performance wise, I have the same kind of problem as with the solution #1
3.Create an INSERT/UPDATE/DELETE trigger for each table. That's a lot of triggers and MySQL limit to one per table. I started that way but the code scaffolding to maintain appears impressive and likely hard to maintain.
I am sure there's a better way I have not think of.
Related
I need to insert values to one final summary table, from a lot of several different summary tables.
I first need to perform various summations on some of these small summary tables, on some I need to calculate averages, on some sum of averages. To achieve this, I created views (around 21 views) and then one view (that is union of select statements for joining 21 views). Then I created stored procedure to call this view and insert values to one final summary table. I able to achieve what I wanted but is this a clean solution ? Can I create temp tables with/without using SP? Will dependency be impacted by having temp tables? I have dependency of views on each other , specially the final view which is union of all other views to calculate final summary of all tables.
I was able to work around this. Simply created SP, that creates temp tables. these temp tables contains select statements to calculate summation, average depending on what I wanted from each temp table. Then in same SP, I inserted data from these temp tables to summary table using insert query and doing UNION of all temp tables
I have a large table that takes around 30 seconds to select the way I want it and it gets larger every day. I'd like to be able to store that selection in a temporary table or cache or something so that this doesn't take so much time.
It doesn't bother me that the table/cache that is being queried over the day may be a few hours old, as the speed is more important.
I suppose someone will wonder why the query is taking so long, so here is the query I'd like to cache every day (or create a temporary table):
select itemcode, warehouse, quantity
from transactions
where (itemcode, warehouse, recordmodified)
in ( select itemcode, warehouse, max(recordmodified)
from transactions
group by warehouse, itemcode )
group by itemcode, warehouse
(I'm asking about the temp table, but if you have suggestions about a faster query, I wouldn't mind hearing about it).
What is making this so slow is the max(recordmodified) that I use to get just the most recent transaction.
I guess I could simply write code in php to drop the table, make the query and create a new table every morning, but I'm hoping there is a way to do this automatically in MySQL.
I have a table with a lot of records of Events. For each row I have two columns “start date” and “end date” . In order to improve performance I was wondering to develop a chron job that create a mini table with only the Events for next 20 hours. But would be even better to have a view that basing on the current time filters only the specific rows. Is that possible in MySql?
Edit: is the view alway worst, in terms of performances, compared to a new small table correct?
Yes, you can create a view that references NOW() or CURDATE() in the where clause. e.g.
create view your_special_v as
select col1, col2, col3
from your table
where datetimecolumn >= now()
MySQL Views do not store data. Its just a representation of table for required data. Like we retrieve selected columns from table by mentioning the names of columns instead of firing "select * from tablename;" query which retrieve us all the columns.
Talking about your question first of all if you want to perform cron job you need to store data. that is not possible with views. So your answer is no we can not perform cron job on views. But you can create events for particular condition on your table...
Regards,
Iroti Mahajan
imahajan#shilpasys.com
Good morning.
I have a table on MySQL DataBase.
In this table there are 5 robots that can write like 10 record each per hour.
Every 3 month a script that I have created, make a copy of the table and then delete all the table entries (In this way I can keep the IDs in a certain order).
My question is.
That are two different statement:
CREATE TABLE omologationResult_{date} AS SELECT * FROM omologationResult
DELETE FROM omologationResult
if the script is going to copy the table at point 0, and a record will be added from the robots, there's no problem, because the SQL statement starts from the lowest ID 'till the end. But if the script is going to delete the table and the robot is writing in it. What will happen? I lose the last robot record?
And if it's true. What can I do to make a copy of the table and then remove only the data that I've copied?
Thank you
Yes, this is not a safe operation because it's not atomic. It's quite possible for another thread to insert values into that table in between your CREATE .. SELECT and the DELETE. One option you have is to use a multi table DELETE
CREATE TABLE omologationResult_{date} AS SELECT * FROM omologationResult;
DELETE omologationResult FROM omologationResult
INNER JOIN omologationResult_{date} ON omologationResult_{date}.id = omologationResult.id
Will ensure that only items that exist in both tables have been deleted from omologationResult
Could someone please explain (or point in right direction) how I would move multiple rows from one table to another and remove the row from the original table based on a set criteria?
I understand
INSERT INTO table2 SELECT * FROM table1
to copy the data from one table to another but I need to then remove the original. The reason being it has been suggested to speed up the querying of the table I should move all redundant data (ended, expired, products older than 3 months) from the main table to another one.
A bit of background, I have a table that holds products, some products have expired but the products still need to be accessible. There are about 50,000 products that have expired and 2,000 which are active. There is a status column (int 1 = active, 2 = expired etc) to determine what to show on the front end.
I guess this post is 2 questions:
Is there a better way to speed up the querying of the product table without removing expired items?
If not, how to move rows from one table to another
Many many thanks!
INSERT INTO table2 (column_name1, column_name2) SELECT column_name1,
column_name2 FROM table 1 WHERE (where clause here)
DELETE FROM table1 WHERE (where clause here)
Source for above: mysql move row between tables
50,000 records in the table really isn't that many. If you're having performance issues, I'd look at your queries and your indexes to help speed up performance. And since those expired records still need to be accessed, then it could be more difficult having multiple tables to maintain.
However, to move data from one table to another as you've asked, you just need to run 2 different statements. Assuming you want to move inactive products:
INSERT INTO ProductsBackup SELECT * FROM Products WHERE Status <> 1
DELETE FROM Products WHERE WHERE Status <> 1
If you have Identities on your columns, you might be better off specifying the column names. But assuming the ProductId is the Identity, then be careful moving those to a different table as you probably don't want to lose that original id as it may point to other tables.
Good luck.