Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I'm working on a database that has one table with 21 million records. Data is loaded once when the database is created and there are no more insert, update or delete operations. A web application accesses the database to make select statements.
It currently takes 25 second per request for the server to receive a response. However if multiple clients are making simultaneous requests the response time increases significantly. Is there a way of speeding this process up ?
I'm using MyISAM instead of InnoDB with fixed max rows and have indexed based on the searched field.
If no data is being updated/inserted/deleted, then this might be case where you want to tell the database not to lock the table while you are reading it.
For MYSQL this seems to be something along the lines of:
SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED ;
SELECT * FROM TABLE_NAME ;
SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ ;
(ref: http://itecsoftware.com/with-nolock-table-hint-equivalent-for-mysql)
More reading in the docs, if it helps:
https://dev.mysql.com/doc/refman/5.7/en/innodb-transaction-isolation-levels.html
The TSQL equivalent, which may help if you need to google further, is
SELECT * FROM TABLE WITH (nolock)
This may improve performance. As noted in other comments some good indexing may help, and maybe breaking the table out further (if possible) to spread things around so you aren't accessing all the data if you don't need it.
As a note; locking a table prevents other people changing data while you are using it. Not locking a table that is has a lot of inserts/deletes/updates may cause your selects to return multiple rows of the same data (as it gets moved around on the harddrive), rows with missing columns and so forth.
Since you've got one table you are selecting against your requests are all taking turns locking and unlocking the table. If you aren't doing updates, inserts or deletes then your data shouldn't change, so you should be ok to forgo the locks.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 9 days ago.
Improve this question
I have a wordpress site and when updating the main theme, I saw that mysql was consuming a high percentage of CPU, then I entered phpmyadmin and this appeared in the process list.
"Waiting for table metadata lock" and "copy to tmp table"
what should i do, my site stopped working and my server space is running out
Only the process running "copying to tmp table" is doing any work. The others are waiting.
Many types of ALTER TABLE operations in MySQL work by making a copy of the table and filling it with an altered form of the data. In your case, ALTER TABLE wp_posts ENGINE=InnoDB converts the table to the InnoDB storage engine. If the table was already using that storage engine, it's almost a no-op, but it can serve to defragment a tablespace after you delete a lot of rows.
Because it is incrementally copying rows to a new tablespace, it takes more storage space. Once it is done, it will drop the original tablespace. So it will temporarily need to use up to double the size of that table.
There should be no reason to run that command many times. Did you do that? The one that's doing the work is in progress, but it takes some time, depending on how many rows are stored in the table and also depending on how powerful your database server is. Be patient, and don't try to start the ALTER TABLE again in more than one tab.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
I'm using MySql(innodb storage engine). I want to implement row level implicit locking on update statement. so, that no other transaction can read or update that row concurrently.
Example:
Transaction1 is executing
"UPDATE Customers
SET City='Hamburg'
WHERE CustomerID=1;"
Then, at same time Transaction2 should not able to read or update same row but Transaction2 should be able to access other rows.
Any help would be appreciated.
Thank you for your time.
If there are no other statements supporting that UPDATE, it is atomic.
If, for example, you needed to look at the row before deciding to change it, then it is a little more complex:
BEGIN;
SELECT ... FOR UPDATE;
decide what you need to do
UPDATE ...;
COMMIT;
No other connection can change with the row(s) SELECTed before the COMMIT.
Other connections can see the rows involved, but they may be seeing the values before the BEGIN started. Reads usually don't matter. What usually matters is that everything between BEGIN and COMMIT is "consistent", regardless of what is happening in other connections.
Your connection might be delayed, waiting for another connection to let go of something (such as the SELECT...FOR UPDATE). Some other connection might be delayed. Or there could be a "deadlock" -- when InnoDB decides that waiting will not work.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
So say you have a single SQL table in some relation DB (SQL Server, MySQL, whatever). Say you have 500 tasks that runs every 15mins. Each task will delete a portion of data related to that task, insert new data related to that task then an external source will some selects for data related to that task.
From my experience this inevitably leads to deadlocks, timeouts and all around sub-par performance for selects, even when doing dirty reads.
You can try to stagger the start time of the tasks as best you can but it doesn't really solve the issue. There are so many tasks that there will always be overlap.
You can try upgrading the server with a better CPU to handle the connections but this is very costly for just 500 tasks imo.
So what I've done is duplicated the tables Schema and gave every task it's own distinct table with that schema.
Also, when constructing new data for the table all it does is inserts into a new table then flips the name of current table with that one.
ie
CREATE TABLE Task001_Table_Inactive
Bulk insert the fresh data to Task001_Table_Inactive
DROP TABLE Task001_Table_Active
RENAME Task001_Table_Inactive Task001_Table_Active
The advantages:
-Fast processing. SQL has NEVER been good at constant deletes. The ability to just bulk insert and flips the name has drastically reduced the processing time for the task.
-Scalable. Now that all these inserts and deletes aren't constantly fighting over one table I can run many tasks on one very cheap machine in the cloud.
-Anti-fragmentation. Since the table is recreated every time, any fragmentation issues that plague systems with constant deletes is no longer an issue.
-Selects are as quick as possible WITH OUT the need for dirty reads. Since the insertion is done in separate table the select statements being done by the external source will be as quick as it can be with out the need to do a dirty read. This is the biggest advantage imo!
-Fast migrations. Eventually I'll have too many tasks and run out of processing power, even with this setup. So if I need to migrate a task to a different server it's a simple matter of copying two tables rather than a hacky and extremely slow select statement on a blob table...
-Indexability. When a tables gets too big (300mill rows +) you can not index it. No matter what it'll just chug for a few days and give up because of a transaction buffer limit. This is just how SQL is. By segmenting out the huge blob table into smaller tables you can index successfully. Take this combined with parallelization, and you can index all the data FASTER than if you were doing indexing on a big table.
Disadvantages:
-Makes navigation of tables in a GUI difficult
-Makes select / schema alter statements slightly tricky because now it has to do some sort of cursor over each table like %Table% and apply the SQL command to each.
So why do many SQL enthusiast loathe Schema duplication if it has SO many advantages? Is there a better way to handle this?
There are no enough information to advise.., but below should be considered:
Snapshot isolation
https://msdn.microsoft.com/en-us/library/ms173763.aspx
Except when a database is being recovered, SNAPSHOT transactions do not request locks when reading data. SNAPSHOT transactions reading data do not block other transactions from writing data. Transactions writing data do not block SNAPSHOT transactions from reading data.
Use transactional replication / log shipping / mirroring / Change Data Capture etc. to offload the main table
Instead of deletion, soft deletion (update IsDeleted flag or maintaining xxxx_deleted table) is an option.
If the system (DB, hardware, network architecture) is properly designed, there are no issues even when where are thousands DML requests per second.
In Microsoft SQL Server, if your tasks never touch rows that do not belong to them, you can try to use either optimistic or pessimistic versioning - Snapshot Isolation in SQL Server.
Depending on your situation, either Read Committed Snapshot Isolation (RCSI) or Snapshot isolation level will do the trick. If both seem to suit you, I would recommend the former since it results in much less performance overhead.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
WHEN use attribute DELAY_KEY_WRITE?
How it helped ?
CRETA TABLE(....) ;DELAY_KEY_WRITE =1
Another performance option in MySQL is the DELAY_KEY_WRITE option. According to the MySQL documentation the option makes index updates faster because they are not flushed to disk until the table is closed.
Note that this option applies only to MyISAM tables,
You can enable it on a table by table basis using the following SQL statement:
ALTER TABLE sometable DELAY_KEY_WRITE = 1;
This can also be set in the advanced table options in the MySQL Query Browser.
This performance option could be handy if you have to do a lot of update, because you can delay writing the indexes until tables are closed. So frequent updates to large tables, may want to check out this option.
Ok, so when does MySQL close tables?
That should have been your next question. It looks as though tables are opened when they are needed, but then added to the table cache. This cache can be flushed manually with FLUSH TABLES; but here's how they are closed automatically according to the docs:
1.When the cache is full and a thread tries to open a table that is not in the cache.
2.When the cache contains more than table_cache entries and a thread is no longer using a table.
3.FLUSH TABLES; is called.
"If DELAY_KEY_WRITE is enabled, this means that the key buffer for tables with this option are not flushed on every index update, but only when a table is closed. This speeds up writes on keys a lot, but if you use this feature, you should add automatic checking of all MyISAM tables by starting the server with the --myisam-recover option (for example, --myisam-recover=BACKUP,FORCE)."
So if you do use this option you may want to flush your table cache periodically, and make sure you startup using the myisam-recover option.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions concerning problems with code you've written must describe the specific problem — and include valid code to reproduce it — in the question itself. See SSCCE.org for guidance.
Closed 9 years ago.
Improve this question
Lately i have been tasked to delete and reinsert approximately 15 million rows on a myisam table that has about 150 million rows doing so while the table/db still remains available
for inserts/reads.
In order to do so i have started a process that takes small chunks of data
and reinserts it via insert select statements into a cloned table with the same structure with sleep in between the runs to not overload the server, skips over the data to be deleted and insert the replacement data.
This way while cloned table was in the build process (took 8+ hours) new data was coming in into the source table. At the end i had to just sync the tables with the new data that was
added in the 8+ hours and do a rename of the tables.
Everything was fine with exception of one thing. The cardinality
of the indexes on the cloned table is way off, and execution plans for queries executed against it are awful (went from few seconds to 30+ min for some of them).
I know that this can be fixed by running an Analyze table on it, but this takes also a lot of time (currently i'm running one on a slave server and is been executed for more then 10h now) and i can't afford to have this table offline to write while the analyze is performed. Also this will stress the IO of the server putting pressure on the server and slowing it down.
Can someone explain why building a myisam table via insert select statements results in a table which has such a poor internal statistics for indexes?
Also is there a way to incrementally build the table and have the indexes in good shape at the end?
Thanks in advance.