mysql denormalization with triggers, specific advice - mysql

I'm considering adding some denormalized information in my database by adding one denormalized table fed by many (~8) normalized tables, specifically for improving select query times of a core use case on my site.
The problems with the current method of querying are:
Slow query times, there are between 8 and 12 joins (some of the left joins) to access the information for this Use Case this can take ~ 3000ms for some queries.
Table Locking / Blocking, When information is updated during busy times of the day or week, (because I'm using MyIsam tables) queries are locked / blocked and this can cause further issues (connections running out, worse performance)
I'm using Hibernate (3.5.2), Mysql 5.0 (all MyIsam tables) and Java 1.6
I'd like some specific suggestions (preferrably based on concrete experience) about exactly what would be the best way to update the the denormalized table.
The following come to my mind
Create a denormalized table with the InnoDb type so that I get row level locking rather than table locking
Create triggers on the properly normalized tables that update the denormalized table,
I'm looking for:
Gotchas - things that I may not be thinking about that will affect my desired result.
Specific MySql settings that may improve performance, reduce locking / blocking on the denormalized table.
Best approaches to writing the Triggers for this scenario.
?
Let me know if there is any other information needed to help answer this question.
Cheers.

I've now implemented this, so I thought I'd share what I did, I asked a mate who's a dba (Greg) for a few tips and his answers basically drove my implementation:
Anyway like "Catcall" implied using TRIGGERS (in my case at least) probably wasn't the best solution. Greg suggested creating two denormalized tables with the same schema, then creating a VIEW that would alternate between the two denormalised tables one being "active" and the other being "deactive" the active table would be the one that was being actively queried by my web application and the deactive table could be updated with the denormalised information.
My application would run queries against the VIEW whose name would stay the same.
That's the crux of it.
Some implementation details (mysql 5.0.n):
I used stored procedures to update the information and then switch the View from denorm_table_a to denorm_table_b.
Needed to update the permissions for my database user
GRANT CREATE, CREATE VIEW, EXECUTE, CREATE ROUTINE, ALTER ROUTINE, DROP, INSERT, DELETE, UPDATE, ALTER, SELECT, INDEX on dbname.* TO 'dbuser'#'%';
For creating a copy of a table the: CREATE TABLE ... LIKE ....; command is really useful (it also copies the index definitions as well)
Creating the VIEW was simple
CREATE OR REPLACE VIEW denorm_table AS SELECT * FROM denorm_table_a;
CREATE OR REPLACE VIEW denorm_table AS SELECT * FROM denorm_table_b;
I created a special "Denormalised Query" Object in my middle tier which then mapped (through hibernate) to the denormalised table (or View in fact) and allowed easy and flexible querying throught Hibernate Criteria mechanism.
Anyway hope that helps someone if anyone needs any more details let me know,
Cheers
Simon

Here is I solution that I used to denormalize a mysql one-to-many relation using a stored procedure and triggers:
https://github.com/martintaleski/mysql-denormalization
It explains a simple blog article to article image relation, you will need to change the fields the fields and queries to apply it to your scenario.

Related

Comparison between MySQL Federated, Trigger, and Event Schedule?

I have a very specific problem that requires multiple MYSQL DB instances, and I need to "sync" all data from each DB/table into one DB/table.
Basically, [tableA.db1, tableB.db2, tableC.db3] into [TableAll.db4].
Some of the DB instances are on the same machine, and some are on a separate machine.
About 80,000 rows are added to a table per day, and there are 3 tables(DB).
So, about 240,000 would be "synced" to a single table per day.
I've just been using Event Schedule to copy the data from each DB into the "All-For-One" DB every hour.
However, I've been wondering lately if that's the best solution.
I considered using Trigger, but I've been told it puts heavy burden on DB.
Using statement trigger may be better, but it depends too much on how the statement is formed.
Then I heard about Federated (in Oracle term, "DBLink"),
and I thought I could use it to link each table and create a VIEW table on those tables.
But I don't know much about databases, so I don't really know the implication of each method.
So, my question is..
Considering the "All-For-One" DB only needs to be Read-Only,
which method would be better, performance and resource wise, in order to copy data from multiple databases into one database regularly?
Thanks!

SQL Server View Access Speed Versus Writing View to Table

I have a SQL Server 2008 DB that has a set of views that are accessed by a program. Our goal is to optimize the access speed of the program (which pulls in data on a user request), to minimize end-user impact.
Right now we are writing all of our views to tables, and passing those mappings to the application (we found the application performed better reading from tables as opposed to views). We are soon going to implement indexes (still need to discuss with the application vendor what indexes will speed up their import), but for now we're trying to figure out the best way to optimize the import.
The plan currently is to write the views to tables, add the proper indexes and then run a (select *) statement to force them into memory. My question is whether A) writing them to tables is necessary once we have the indices and the select * and B) what are some methods that we are missing?
Edited to clarify question goal.
OK I think I follow
Select into implies you are dropping the table and let select into create
You are probably better off with a truncate
If it is a FK then you need to delete but they tend to be smaller
And then just do an insert into
This way you also don't need to drop and recreate views
If you can take the hit you are better off taking a tablock hit the whole table
If the linked is slow then insert into some local staging tables
From the staging table load the production table
I totally don't get why you would materialize a view into table.
If you have performance issue with a view then first optimize the view.
What is going on in the view that is slow?

How to increase the performance of database schema creation?

For our testing environment, I need to setup und tear down a database multiple times (each test should run independently of any other).
The process is the following:
Create database schema and insert necessary data
Run test 1
Remove all tables in database
Create database schema and insert necessary data
Run test 2
Remove all tables in database
...
The schema and data are the same for each test in the test case.
Basically, this works. The big problem is, that the creation and clearing of the database takes a lot of time. Is there a possibility to improve the performance of mysql for the creation of tables and the insertion of data? Or can you think of a different process for the tests?
Thank for you your help!
Optimize the logical design
The logical level is about the structure of the query and tables themselves. Try to maximize this first. The goal is to access as few data as possible at the logical level.
Have the most efficient SQL queries
Design a logical schema that support the application's need (e.g. type of the columns, etc.)
Design trade-off to support some use case better than other
Relational constraints
Normalization
Optimize the physical design
The physical level deals with non-logical consideration, such as type of indexes, parameters of the tables, etc. Goal is to optimize the IO which is always the bottleneck. Tune each table to fit it's need. Small table can be loaded permanently loaded in the DBMS cache, table with low write rate can have different settings than table with high update rate to take less disk spaces, etc. Depending on the queries, different index can be used, etc. You can denormalized data transparently with materialized views, etc.
Tables paremeters (allocation size, etc.)
Indexes (combined, types, etc.)
System-wide parameters (cache size, etc.)
Partitioning
Denormalization
Try first to improve the logical design, then the physical design. (The boundary between both is however vague, so we can argue about my categorization).
Optimize the maintenance
Database must be operated correctly to stay as efficient as possible. This include a few mainteanance taks that can have impact on the perofrmance, e.g.
Keep statistics up to date
Re-sequence critical tables periodically
Disk maintenance
All the system stuff to have a server that rocks
source from:How to increase the performance of a Database?
I suggest you can write all your need operations into an script using shell、perl or python(init_db).
The first use, you can create、 insert and delete manually,then dump both the schema and data .
You can choose bulk insert and drop table for deleting data to improve the total performance.
Hope this can help you.
Instead of DROP TABLE + CREATE TABLE, just do TRUNCATE TABLE. This may, or may not, be faster; give it a try.
If you are INSERTing multiple rows each time, then either batch them (all rows in one INSERT), or use LOAD DATA. Either of these is much faster than row-by-row INSERTs.
Also fast... If you have the initial data in another table (which you could keep permanently), then do
CREATE TABLE test SELECT * FROM perm_table;
... (run tests using `test`)
DROP TABLE test;

create view on different databases on different hosts

Is it possible to create a view from tables from two different databases? Like:
creative view 'my_view' as
select names as value
from host_a.db_b.locations
union
select description as value
from host_b.db_b.items;
They currently are different database engines (MyISAM and InnoDB).
thx in advance
Yes, you need to access the remote table via the FEDERATED db engine, then create a view using your query.
However this is a rather messy way to solve the problem - particularly as (from your example query) your data is effectively sharded.
This structure won't allow updates/inserts on the view. Even for an updatable/insertable view, my gut feeling is that you'll run into problems if you try to anything other than auto-commit transactions, particularly as you're mixing table types. I'd recommend looking at replication as a better way to solve the problem.

What are the differences between a TABLE and a VIEW in a MySQL database?

I'm joining a number of tables and want to create some tables or views that are easier to query against to do quick analysis of our data. What are the implications of creating a new table or new view with the combined data.
Currently the tables I'm joining are static, but this code may be moved to our live tables in the future.
This is a slight oversimplification, but a view is basically a saved query on a table returning a result (in rows and columns), which you can then query as if it were its own table.
As of MySQL 5.0, views weren't all that great because it executed the underlying query every time it was used, so there really wasn't much point to them (although they could be useful for code reuse). That may have changed since 5.0, though.
A Table stores the data
A View is a stored procedure like select * from table saved in the database for later use
you could have a view joining two tables and then select from that view without a join clause but get a joined result
Be careful with views, as they don't necessarily use the indexes correctly in the underlying tables!
See this article for more information
http://www.mysqlperformanceblog.com/2007/08/12/mysql-view-as-performance-troublemaker/
In addition to Rob's explanation:
You can grant privileges not only on tables, but also on views. With this you can give access only to a compiled subset of a databases data.