I have a table which has 38.406.168 rows and according to size in phpmyadmin 4.5GB. I want to see the last row of the table. Unfortunately I couldn't use select * from ... limit 38.406.166,1 or even I couldn't use select count(*) from ... function.
I changed my.ini in wamp server, but still I get mysql server has gone away error while attempting execute one of these queries. BTW; I couldn't even set an index on ID to make these processes much quicker.
My last try was to export the table to look at the last row. However, It just shows me 123MB of the file.
What should I do? Please help me. The features of the computer is 2.93 GHz, 3.50GB
Here is my my.ini file:
# The MySQL server
[wampmysqld]
port = 3306
socket = /tmp/mysql.sock
skip-locking
key_buffer = 384M
max_allowed_packet = 2000M
table_cache = 4096
sort_buffer_size = 2000M
net_buffer_length = 8K
read_buffer_size = 2000M
read_rnd_buffer_size = 2000M
myisam_sort_buffer_size = 2000M
basedir=c:/wamp/bin/mysql/mysql5.1.36
log-error=c:/wamp/logs/mysql.log
datadir=c:/wamp/bin/mysql/mysql5.1.36/data
(.. these parts are deleted, since there is nothing to set as value)
# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = C:\mysql\data/
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = C:\mysql\data/
#innodb_log_arch_dir = C:\mysql\data/
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 384M
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 10M
#innodb_log_buffer_size = 64M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 180
[mysqldump]
quick
max_allowed_packet = 160M
Thank you so much for your help
I tried a lot of stuff and ended up with those 2 working:
Simply mirror the database via mysql's internal master-slave-functions (try google, you ll find good tutorials) onto a simple backup server (most cheap hosting packages will work if they have ssh access)
Try http://www.mysqldumper.net/, the best tool to copy & split huge databases into 100mb-parts. This simple open source tool did everything that "professional" backup scripts couldn't do.
You will want to use the mysqldump command to do this. Here is what I do in linux, but I think it will translate to Windows (I see that you're running WAMP).
mysqldump --opt --force -Q --user=[your_user] -p [database_name] > dump.sql
You may need to change directory to where the mysqldump file is located:
cd c:\path\to\mysql\bin
Related
I need to dump MySQL InnoDB-database consists of several tables. One table that causes problems has nearly 13 Million rows. A fresh install of XAMPP(V.3.2.2) dump process was successful, after that, dump process always failed with the error message "mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table gv_faktur_header_history at row: 2623629". At this point here's the status :
Can not insert any value (Error 2013: Lost connection to MySQL)
Can not issue "CHECK TABLE" command (Error 2013: Lost connection to MySQL)
Can not alter this table (add column)
Can select data from this table
Can select row 2623629 (select * from table limit 2623629 ,1)
Can run "show table status" command
I repeat this process several times like this :
Reinstalling xampp
Importing database using this method
> set global net_buffer_length = 1000000;
> set global max_allowed_packet = 1000000000;
> SET foreign_key_checks = 0;
> SET UNIQUE_CHECKS = 0;
> SET AUTOCOMMIT = 0;
> use db_name;
> source backup-file.sql SET
> foreign_key_checks = 1;
> SET UNIQUE_CHECKS = 1;
> SET AUTOCOMMIT = 1;
Dump the database w/o --skip-extending-insert (success)
Dump the database w/o --skip-extending-insert (failed)
Dump the database w/o --skip-extending-insert (failed)
mysqldump command :
mysqldump -u root -p --skip-extended-insert --max-allowed-packet=1G --net-buffer-length=32704 rent_scaff header_history > D:\dobol
Environment specifications :
i5 8 gen
Ram 8 GB (3 Gb unuse as seen in task manager)
SSD Storage 512G
mysqldump Ver 10.16 Distrib 10.1.10-MariaDB
here's my.ini configuration
[client]
# password = your_password
port = 3306
socket = "C:/xampp/mysql/mysql.sock"
[mysqld]
port= 3306
socket = "C:/xampp/mysql/mysql.sock"
basedir = "C:/xampp/mysql"
tmpdir = "C:/xampp/tmp"
datadir = "C:/xampp/mysql/data"
pid_file = "mysql.pid"
key_buffer = 16M
max_allowed_packet = 1G
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
log_error = "mysql_error.log"
plugin_dir = "C:/xampp/mysql/lib/plugin/"
server-id = 1
innodb_data_home_dir = "C:/xampp/mysql/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp/mysql/data"
innodb_buffer_pool_size = 1G
innodb_additional_mem_pool_size = 2M
innodb_log_file_size = 250M
innodb_log_buffer_size = 250M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
[mysqldump]
quick
max_allowed_packet = 1G
[mysql]
no-auto-rehash
[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
enter code here
Mysql Log :
Server version: 10.1.10-MariaDB
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=2
max_threads=1001
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 787099 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0x3eee2178
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_parameter_handler()
mysqld.exe!my_mb_ctype_mb()
mysqld.exe!??2Geometry##SAPAXIPAX#Z()
mysqld.exe!??2Geometry##SAPAXIPAX#Z()
mysqld.exe!?propagate_equal_fields#Item_func_expr_str_metadata##UAEPAVItem##PAVTHD##ABVContext#Value_source##PAVCOND_EQUAL###Z()
mysqld.exe!??2Geometry##SAPAXIPAX#Z()
mysqld.exe!??2Geometry##SAPAXIPAX#Z()
mysqld.exe!??2Geometry##SAPAXIPAX#Z()
mysqld.exe!??0Alter_table_prelocking_strategy##QAE#XZ()
mysqld.exe!?mysql_alter_table##YA_NPAVTHD##PAD1PAUHA_CREATE_INFO##PAUTABLE_LIST##PAVAlter_info##IPAUst_order##_N#Z()
mysqld.exe!?execute#Sql_cmd_alter_table##UAE_NPAVTHD###Z()
mysqld.exe!?mysql_execute_command##YAHPAVTHD###Z()
mysqld.exe!?mysql_parse##YAXPAVTHD##PADIPAVParser_state###Z()
mysqld.exe!?dispatch_command##YA_NW4enum_server_command##PAVTHD##PADI#Z()
mysqld.exe!?do_command##YA_NPAVTHD###Z()
mysqld.exe!?threadpool_process_request##YAHPAVTHD###Z()
mysqld.exe!?tp_end##YAXXZ()
KERNEL32.DLL!SetUserGeoName()
ntdll.dll!TpCheckTerminateWorker()
ntdll.dll!TpCallbackIndependent()
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlGetAppContainerNamedObjectPath()
ntdll.dll!RtlGetAppContainerNamedObjectPath()
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Please help me how to Dump (backup) a large database, at least how to safely dump the database, so when the process failed, the table remains usable.
What is happening is that either one command is taking forever and it timing out, or that there is too much data to fit in the cache at one time.
I would try using a program, something like HeidiSQL, to do the backup of the database. This would allow the backup to be in stages instead of one giant request.
This would be the example to do this with HeidiSQL. Feel free to replace 127.0.0.1 with a different IP address if the SQL server is on a second computer, or any other setting that you think is different.
You would start by setting up the connection:
HeidiSQL Sesssion Manager Setup
Then, click on OPEN
Right click on your database, located in the top left of the screen
Click EXPORT DATABASE AS SQL
First we'll backup the structure of your database:
Click the checkbox on the left side
Click on SQL export on the right-side tab
Database -> Drop (unchecked), Create (checked)
Table -> Drop (unchecked), Create (checked)
Data -> No Data
Output -> Single .sql file
Filename -> Click the folder icon, pick a location and name the file something like DB_Structure.sql
Finally, click Export
Next, we will backup the data from the database:
-Same steps as before, in SQL Export do:
Database -> Drop (uncheck), Create (uncheck)
Table -> Drop (uncheck), Create (uncheck)
Data -> Change from No Data to Insert
Max INSERT Size -> 1024
Output -> Change from Single .sql file to ZIP compressed .sql file
Filename -> Click the folder icon, Pick a location and name the file something like DB_Data.zip
Finally, click Export
This is how you would Use those backup files:
Click on the tab named â–ºQuery
Press Ctrl + O
In the Open File dialog box, navigate to your backup SQL file for the structure
Then click Open
Next, press F9 to run the SQL commands
When this is finished, you do the same process for the data SQL backup file.
I have MAMP Pro on El Capitan running. It has been fine up until now, but I've run into a problem. I have a mixture of database, some using the MyISAM Engine and others using InnoDB. I don't really know how that works. I guess if some are InnoDB, the Engine is still MyISAM by default. The issue is with databases that I have for Atlassians Confluence and JIRA. In Confluence, all is good, but it says:
You should increase innodb_log_file_size to 256M
I tried playing around with the my.cnf, but ran into issues. I restored things, and these are the relevant sections from the config.
[mysqld]
#port = 9999
socket = /Applications/MAMP/tmp/mysql/mysql.sock
key_buffer_size = 64M
max_allowed_packet = 512M
# table_cache only works for MySQL 5.5.x
#table_cache = 64
# If you are running MySQL 5.6.x, use table_open_cache.
#table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 32M
#Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /Applications/MAMP/db/mysql/
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /Applications/MAMP/db/mysql/
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 128M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 512M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50
When I uncommented the InnoDB section the server crashes and the database got corrupted.
Just wondering how I can turn on InnoDB for MAMP, if that is recommended, and update my existing databases at the same time, the MyISAM ones and the InnoDB ones.
While I'm at it, I might want to upgrade MAMP to the newer MySQL version, maybe later.
How much RAM do you have?
Keep max_allowed_packet under 2% of RAM.
Since you are using both MyISAM and InnoDB, set innodb_buffer_pool_size to about 1/3 of RAM unless; less if you have a tiny system.
Do not change innodb_log_file_size without further instructions. That is, don't set it in my.cnf if it is not already set.
MyISAM and InnoDB can coexist.
Last week after some previous database tuning, I stopped mysql and it failed to restart.
After a long period of troubleshooting I found that the ibdata1 file was not as big as it should have been, it had been deleted and recreated as new.
I retrieved the old 9.5gb file from backup, replaced it and mysql started again, happy days.
I've been having some more server trouble today, had a look in the mysql folder and the file has disappeared again.
I haven't stopped mysql yet so everything is still up and running, I will have to retrieve it from backup and restart with my fingers crossed.
So my question is, why is it disappearing?? My guess us I've made an accidental change in the my.cnf file and then not restarted. Unfortunately I don't have a backup of the file because I didn't know there was a change made.
(Untidy) My.cnf is as follows:
[mysqld]
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
symbolic-links=0
innodb_thread_concurrency= 4
innodb_buffer_pool_size = 2G
thread_concurrency = 3
thread_cache_size = 32
table_cache = 1024
query_cache_size = 64M
query_cache_limit = 2M
join_buffer_size = 8M
tmp_table_size = 256M
key_buffer = 32M
innodb_autoextend_increment=512
max_allowed_packet = 16M
max_heap_table_size = 256M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
innodb_log_file_size = 100M
innodb_additional_mem_pool_size = 20M
innodb_flush_log_at_trx_commit=2
innodb_lock_wait_timeout=1800
innodb_log_buffer_size=500K
log-error=/var/log/mysqld.log
slow_query_log = /var/log/mysql-slow.log
long_query_time = 5
pid-file=/var/run/mysqld/mysqld.pid
sort_buffer_size = 2M
read_buffer_size = 2M
wait_timeout = 120
key_buffer = 384M
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 1M
max_connections=50
query_cache_type = 1
Any help greatly appreciated!
Thanks
Don't stop MySQL! mysqld keeps it open while it's running, so it's still on the file system.
MySQL never deletes ibdata1, so it must be some external command.
To recover the database stop all writes to the database, wait till the main thread is in "waiting for server activity" or "sleeping" state:
mysql> pager grep Main
PAGER set to 'grep Main'
mysql> show engine innodb status\G
Main thread id 4994568192, state: sleeping
1 row in set (0.00 sec)
Take dump of all databases (this step is not necessary, but for extra safety do it);
mysqldump -A > mydb.sql
Find the deleted ibdata1 in /proc filesystem and copy it back to MySQL datadir
# ls -la /proc/`pidof mysqld`/fd/ | grep -e ibdata
lrwx------ 1 root root 64 May 26 02:41 3 -> /var/lib/mysql/ibdata1 (deleted)
Note 3 - it's a file descriptor of ibdata1.
Copy ibdata1 back:
# cp /proc/`pidof mysqld`/fd/3 /var/lib/mysql/ibdata1
Then restart MySQL
#Akuzminsky's answer is what you need to do to get your current ibdata1 back. And he is correct that MySQL never deletes ibdata1, regardless of your my.cnf configuration.
So something else is deleting the file. How can one find out? Try running the Linux Audit Daemon. You won't be able to find out what deleted the file last time (unless you were already running the Audit Daemon), but in case it happens again you're ready.
See this StackExchange answer for details: https://askubuntu.com/questions/48844/how-to-find-the-pid-of-the-process-which-has-deleted-a-file
I am on linux box running only mysql with 8 cores and 16GB ram. All connections come from web server on another machine in the same network running php with codeigniter.
I cannot get more than 150 connections on mysql.
My my.cnf is:
[mysqld]
user=mysql
port = 3306
socket = /tmp/mysql.sock
datadir = /usr/local/mysql/var/
skip-external-locking
max_connections=500
max_user_connections=500
open-files-limit = 500
key_buffer_size = 2048M
max_allowed_packet = 32M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
myisam_sort_buffer_size = 128M
thread_cache_size = 250
table_definition_cache = 1024
query_cache_size = 32M
query_cache_limit = 32M
table_cache=1024
max_heap_table_size=1024M
key_buffer=2048M
wait_timeout=60
thread_concurrency = 16
long_query_time = 1
tmp_table_size=256M
show status returns that max_connections and max_user_connections to be 500.
Since mysql is saying that connection limit is 500, I thought there are other setting in PHP, apache or codeigniter that is limiting the requests to mysql, but I cannot find any. I've searched google for few days trying to find answers without any luck.
Are there limits set on any of the above mentioned software? I will post configs if necessary.
Thank you.
Check the Max number of Apache worker processes (ServerLimit and MaxClients) in httpd.conf. Assuming a fixed number of connections per worker, you might be maxing out your number of workers, so nothing is requesting new MySQL connections.
Sorry, just found this on Google, but your config is foobar
max_connections=500
open-files-limit = 500
table_open_cache = 512
table_definition_cache = 1024
open-files-limit: 500 #; not good!
table_open_cache: 512... * 2+ open-files per table = 1024 open-files used minimum
max_connections: 500 each open connection starts a new file handle to buffer.
Your mysql will open some 130 tables just to start (core tables) which leaves you with a mere 240 file handles to share between data queries and connections. For each connection with a table query, 3+ file handles are consumed (connection, data table, index file(s)). That maxes your open-files long before you get 150 connections. open-files-limit needs to start >2048 for that kind of DB usage.
more help:
show global status like '%open%';
show global status like '%onnect%';
see for yourself how many files/connections are actually in use. some operating systems (Windows XP) hard limit files/network connections. try googling "mysql 'YOUR-OS' 150 connections" and see if that is the limiting factor.
I don't know if this problem is specific to my set up but when I add the line
log = /var/log/mysql.log
to the mysqld section of a copied my-large.cnf file and try to restart the mysql server I get the error
Starting MySQL. ERROR! The server quit without updating PID file (/var/lib/mysql/centos-server.pid).
I've created the file /var/log/mysql.log, set its owner and group to mysql and set the permissions on /var/log to 777 (for the moment)
I'm on centos, with mysql 5.6.5 m8 (the development release).
This is a snippet of the my.cnf file
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-external-locking
key_buffer_size = 256M
max_allowed_packet = 1M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8
log = /var/log/mysql.log
Seems like the log directive is outdated and my.cnf requires the directive
general-log = 1
If specified like this the log file will be created in a default location (which on centos is /var/lib/mysql/centos-server.log)