I have run mysql -u root -p gf < ~/gf_backup.sql to restore my db. However when I see the process list I see that one query has has been idle for a long time. I do not know the reason why.
mysql> show processlist;
+-----+------+-----------+-------------+---------+-------+-----------+------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------+-------------+---------+-------+-----------+------------------------------------------------------------------------------------------------------+
| 662 | root | localhost | gf | Query | 18925 | query end | INSERT INTO `gf_1` VALUES (1767654,'90026','Lddd',3343,34349),(1 |
| 672 | root | localhost | gf | Query | 0 | NULL | show processlist |
+-----+------+-----------+-------------+---------+-------+-----------+------------------------------------------------------------------------------------------------------+
Please check free space with df -h command (if under Linux/Unix) if you're out of space do not kill or restart MySQL until it catch up with changes when you free some space.
you may also want to check max_allowed_packet setting in my.cnf and set it to something like 256M, please refer to http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_max_allowed_packet
Probably your dump is very large and contains much normalized data (records split into a bunch of tables, with a bunch of foreign key constraints, indexes and so on).
If so, you may try to remove all constraints and index definitions from the SQL file, then import the data and re-create the former removed directives. This is a well-known trick to speed up imports, because INSERT commands without validation of any constraints are a lot faster, and creation of an index and so on afterwards can be done in a single transaction.
See also: http://support.tigertech.net/mysql-large-inserts
Of course, you should kill the query first. And remove all fragments it created already.
Related
First of all sorry for repeatating this repeated question as it has been asked numerous times already. I have gone through those questions and answers.
I have a table on production environment of size approx 159 GB and now I want to migrate MySQL to another server. But since total DB size is more than 300 GB I am not able to migrate it easily. So I am trying to reclaim space by deleting records from MySQL. I deleted more than 70 % records from this table and tried OPTIMIZE TABLE but its giving an error as :
mysql> OPTIMIZE TABLE table_name;
+--------------------------------------+----------+----------+-----------------------+
| Table | Op | Msg_type | Msg_text |
+--------------------------------------+----------+----------+-----------------------+
| table_name | optimize | note | Table does not support optimize, doing recreate + analyze instead |
| table_name | optimize | error | The table 'table_name' is full |
| table_name | optimize | status | Operation failed |
+--------------------------------------+----------+----------+-----------------------+
innodb_file_per_table is set to ON
SHOW VARIABLES LIKE '%innodb_file_per_table%';
Variable_name Value
--------------------- --------
innodb_file_per_table ON
Mysql Version: 5.7.28-log
I read somewhere that alter table will help however it slows down all MySQL queries.
In one answer I read that copying data in another table and then renaming it and deleting original table (which I assume OPTIMIZE TABLE does internally) will help but doing so will need downtime.
Is there any other way which I can achieve this.?
I'm running a script with several million update statements like this:
UPDATE data SET value = 0.9234 WHERE fId = 47616 AND modDate = '2018-09-24' AND valueDate = '2007-09-01' AND last_updated < '2018-10-01';
fId, modDate and valueDate are the 3 components of the data table's composite primary key.
I initially ran this with AUTOCOMMIT=1 but I figured it would speed up if I set AUTOCOMMIT=0 and wrapped the transactions into blocks of 25.
In autocommit mode, I used SHOW PROCESSLIST and I'd see the UPDATE statement in the output, so from the fId foreign key, I could tell how far the script had progressed.
However without autocommit, watching it running now, I haven't seen anything with SHOW PROCESSLIST, just this:
610257 schema_owner_2 201.177.12.57:53673 mydb Sleep 0 NULL 0.000
611020 schema_owner_1 201.177.12.57:58904 mydb Query 0 init show processlist 0.000
The Sleep status makes me paranoid that other users on the system are blocking the updates, but if I run SHOW OPEN TABLES I'm not sure whether there's a problem:
MariaDB [mydb]> SHOW OPEN TABLES;
+----------+----------------+--------+-------------+
| Database | Table | In_use | Name_locked |
+----------+----------------+--------+-------------+
| mydb | data | 2 | 0 |
| mydb | forecast | 1 | 0 |
| mydb | modification | 0 | 0 |
| mydb | data3 | 0 | 0 |
+----------+----------------+--------+-------------+
Is my script going to wait forever? Should I go back to using autocommit mode? Is there any way to see how far it's progressed? I guess I can inspect the data for the updates but that would be laborious to put together.
Check for progress by actually checking the data.
I assume you are doing COMMIT?
It is reasonable to see nothing -- each UPDATE will take very few milliseconds; there will be Sleep time between UPDATEs.
Time being 0 is your clue that it is progressing.
There won't necessarily be any clue in the PROCESSLIST of how far it has gotten.
You could add SELECT SLEEP(1), $fid; in the loop, where $fid (or whatever) is the last UPDATEd row id. That would slow down the progress by 1 second per 25 rows, so maybe you should do groups of 100 or 500.
Is there a way to know the bin log files path from mysql console as we can know whether its ON or OFF by using
Select * information_schema.GLOBAL_VARIABLES
where variable_name like '%log_bin%'.
Use this to show on/off:
SHOW VARIABLES LIKE 'log_bin'
Also:
SHOW GLOBAL VARIABLES LIKE '%bin%'
Or
SHOW SESSION VARIABLES LIKE ...
More Information: (notice that some of these values and results changed from 5.5 to 5.6!)
http://dev.mysql.com/doc/refman/5.5/en/show-master-status.html
mysql > SHOW MASTER STATUS;
+---------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+---------------+----------+--------------+------------------+
| mysql-bin.003 | 73 | test | manual,mysql |
+---------------+----------+--------------+------------------+
mysql> SHOW BINARY LOGS;
+---------------+-----------+
| Log_name | File_size |
+---------------+-----------+
| binlog.000015 | 724935 |
| binlog.000016 | 733481 |
+---------------+-----------+
From the docs regarding log-bin"
Setting this option causes the log_bin system variable to be set
to ON (or 1), and not to the base name. This is a known issue;
see Bug #19614 for more information.
There is a workaround using mysqld instead in the Bug 19614 which I modified a bit. If you are scripting, you can use from the mysql client (which I found a bit tedious to do, see the next workaround):
mysql >\! dirname $(mysqld --help --verbose 2> /dev/null | egrep "^log-bin " | grep -o "\/.*")
Looks like there is a patch submitted by Mark Callaghan, and it was never committed. There is a function in WP5465 (which is the work in progress for this patch), however it didn't work properly for me, as the location of the logs can be different across setups.
I am running NDB Cluster and I see that on mysql api nodes, there is a very big binary log table.
+---------------------------------------+--------+-------+-------+------------+---------+
| CONCAT(table_schema, '.', table_name) | rows | DATA | idx | total_size | idxfrac |
+---------------------------------------+--------+-------+-------+------------+---------+
| mysql.ndb_binlog_index | 83.10M | 3.78G | 2.13G | 5.91G | 0.56 |
Is there any recommended way to reduce the size of that without breaking anything? I understand that this will limit the time frame for point-in-time recovery, but the data has is growing out of hand and I need to do a bit of clean up.
It looks like this is possible. I don't see anything here: http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-replication-pitr.html that says you can't based on the last epoch.
Some additional information might be gained by reading this article:
http://www.mysqlab.net/knowledge/kb/detail/topic/backup/id/8309
The mysql.ndb_binlog_index is a MyISAM table. If you are cleaning it,
make sure you don't delete entries of binary logs that you still need.
When I issue SHOW PROCESSLIST query, only the first 100 characters of the running SQL query are returned in the info column.
Is it possible to change MySQL config or issue a different kind of request to see complete query (the queries I'm looking at are longer than 100 characters)
SHOW FULL PROCESSLIST
If you don't use FULL, "only the first 100 characters of each statement are shown in the Info field".
When using phpMyAdmin, you should also click on the "Full texts" option ("← T →" on top left corner of a results table) to see untruncated results.
Show Processlist fetches the information from another table. Here is how you can pull the data and look at 'INFO' column which contains the whole query :
select * from INFORMATION_SCHEMA.PROCESSLIST where db = 'somedb';
You can add any condition or ignore based on your requirement.
The output of the query is resulted as :
+-------+------+-----------------+--------+---------+------+-----------+----------------------------------------------------------+
| ID | USER | HOST | DB | COMMAND | TIME | STATE | INFO |
+-------+------+-----------------+--------+---------+------+-----------+----------------------------------------------------------+
| 5 | ssss | localhost:41060 | somedb | Sleep | 3 | | NULL |
| 58169 | root | localhost | somedb | Query | 0 | executing | select * from sometable where tblColumnName = 'someName' |
See full query from SHOW PROCESSLIST :
SHOW FULL PROCESSLIST;
Or
SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST;
I just read in the MySQL documentation that SHOW FULL PROCESSLIST by default only lists the threads from your current user connection.
Quote from the MySQL SHOW FULL PROCESSLIST documentation:
If you have the PROCESS privilege, you can see all threads.
So you can enable the Process_priv column in your mysql.user table. Remember to execute FLUSH PRIVILEGES afterwards :)
If one want to keep getting updated processes (on the example, 2 seconds) on a shell session without having to manually interact with it use:
watch -n 2 'mysql -h 127.0.0.1 -P 3306 -u some_user -psome_pass some_database -e "show full processlist;"'
The only bad thing about the show [full] processlist is that you can't filter the output result. On the other hand, issuing the SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST open possibilities to remove from the output anything you don't want to see:
SELECT * from INFORMATION_SCHEMA.PROCESSLIST
WHERE DB = 'somedatabase'
AND COMMAND <> 'Sleep'
AND HOST NOT LIKE '10.164.25.133%' \G
SHOW FULL PROCESSLIST
This shows the full processlist with more info.