ERROR 1114 (HY000) at line 4734: The table is full - mysql

I am trying to import a old Vbulletin database but always getting this error
ERROR 1114 (HY000) at line 4734: The table 'session' is full
this database backup size is 2GB and my server have 8GB ram. I tried to add innodb_data_file_path=ibdata1:10M:autoextend and innodb_file_per_table to my.cnf but not solved my problem.
my complete my.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
bind-address = 127.0.0.1
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam-recover = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
query_cache_limit = 1M
query_cache_size = 16M
expire_logs_days = 10
max_binlog_size = 100M
innodb_data_file_path=ibdata1:10M:autoextend
innodb_file_per_table
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 16M
!includedir /etc/mysql/conf.d/

I am very familiar with Vbulletin database schema and tables. I have converted dozens of VB sites to Innodb storage engine.
The reason that you are getting that error is because the session table is a Memory table. The session table(s) must have been quite full when you took this backup that you are trying to restore. That, coupled with your overhead IO during the import, is filling up your RAM. However, for VB to work properly, you do not actually need this table to be a Memory table.
In order to get around this you can convert your session table to InnoDB.
I would open the .sql file up in a text editor (if possible) and change where it says ENGINE = MEMORY for your session table to InnoDB, or use string replace:
sed -i 's/MEMORY/INNODB/g' yourfilename.sql

Related

Yii2 Remote Database Connection Issue

I am trying to get my web server to connect to my remote database (Mysql).
The connection is happening in my web application which is built on yii2.
I have created all user details, GRANTED all privileges and allowed connections from ALL hosts (%) (Progressively in troubleshooting this).
Now I can connect from home using HeidiSQL or using Mysql Workbench without issue.
I have even created a blank yii2 site at home and it works instantly, but uploading the site to the web server instantly gives the same issue.
Database Exception – yii\db\Exception
SQLSTATE[HY000] [2003] Can't connect to MySQL server on '46.105.111.111' (110)
↵
Caused by: PDOException
SQLSTATE[HY000] [2003] Can't connect to MySQL server on '46.105.111.111' (110)
in /home/xxxxxxx/xtest.xxxxxxxx.tv/vendor/yiisoft/yii2/db/Connection.php at line 602
I have run netstat on the db server:
I have been through the etc/mysql/my.cnf
<pre>[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 7999
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
#bind-address = *
key_buffer = 16M
max_allowed_packet = 64M
thread_stack = 192K
thread_cache_size = 8
myisam-recover = BACKUP
query_cache_limit = 1M
query_cache_size = 16M
expire_logs_days = 10
max_binlog_size = 100M
max_connections = 65535
open-files-limit = 65535
max_connect_errors = 65535
innodb_buffer_pool_size = 1G
table_open_cache = 1000
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
tmp_table_size = 256M
max_heap_table_size = 256M
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 16M</pre>
I have looked at the iptables:
I am totally bedazzled as having read about 20 pieces of information about mysql security, i get the feeling its a permissions issue.
I have even tried using my own PDO Wrapper, so i a, 99% sure its not a Yii2 issue, would really appreciate any help.
Hopefully I have provided enough info as I got a downvote previously.
many thanks for the help.
It turns out that the web host was actually blocking the outgoing port for the connection.
The output error from yii being database error, nievly led me to think the issue was at the database end.
Simple logic when you read back the question.
I would say to anyone that finds this question, remember with a connection issue that communication is two way, check both ends of the connection.

MySQL The table is full on query

I can't find the source of the issue on our Master/Slave replication .
Today i was updating the Master and suddenly got the following error from the slave
Error 'The table 'caching_api' is full' on query.
Query: '
ALTER TABLE `caching_api`
ADD UNIQUE INDEX `id` (`id`) USING BTREE ,
ADD INDEX `search` (`component`, `method`) USING BTREE
It's not a disk issue , the Slave is an exact replicate of the Master
And my.cnf config :
[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /data/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
key_buffer = 16M
max_allowed_packet = 128M
thread_stack = 192K
thread_cache_size = 64
table_open_cache = 3000
join_buffer_size = 128k
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
max_connections = 4000
wait_timeout = 150
interactive_timeout = 30
innodb_buffer_pool_size = 25G
innodb_log_file_size = 1G
innodb_buffer_pool_instances = 10
tmp_table_size = 256M
max_heap_table_size = 256M
innodb_flush_log_at_trx_commit = 2
query_cache_limit = 64M
query_cache_size = 256M
relay_log_space_limit = 10G
server-id = 2
relay-log = /var/log/mysql/mysqld-relay-bin
expire_logs_days = 1
max_binlog_size = 100M
slave-skip-errors = 1062,1054
[mysqldump]
quick
quote-names
[mysql]
#no-auto-rehash # faster start of mysql but no tab completition
[isamchk]
key_buffer = 16M
Update to questions from comments :
When i try to run the query directly on the SLAVE :
caching_api table status
Slave Disk Info
ibdata1 about 36GB
ulimit -a
The manual doesn't rule out the possibility of a full disk.
If a table-full error occurs, it may be that the disk is full or that
the table has reached its maximum size. The effective maximum table
size for MySQL databases is usually determined by operating system
constraints on file sizes, not by MySQL internal limits.
But since have checked that already before posting this question. So the only other possiblility is that maximum table size has been reached.

enabling mysql query log causes error

I don't know if this problem is specific to my set up but when I add the line
log = /var/log/mysql.log
to the mysqld section of a copied my-large.cnf file and try to restart the mysql server I get the error
Starting MySQL. ERROR! The server quit without updating PID file (/var/lib/mysql/centos-server.pid).
I've created the file /var/log/mysql.log, set its owner and group to mysql and set the permissions on /var/log to 777 (for the moment)
I'm on centos, with mysql 5.6.5 m8 (the development release).
This is a snippet of the my.cnf file
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-external-locking
key_buffer_size = 256M
max_allowed_packet = 1M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8
log = /var/log/mysql.log
Seems like the log directive is outdated and my.cnf requires the directive
general-log = 1
If specified like this the log file will be created in a default location (which on centos is /var/lib/mysql/centos-server.log)

Optimizing MySQL database for FAST count

i'm currently trying to optimize my database. The problem is the following:
I have a table which currently stores over 83Mio. timedependent values. They are indexed by a highres (ms) timestamp. What i need to do is count how many times a certain value appears in a given interval of time - for example say i want to know how many times value 1.56787 appeared in the interval form timestamp x to timestamp y. Right now this takes almost forever.
Im using InnoDB and i already put a lot of time into optimizing the config files, which increased the speed immensly.
Im thankful for any input, as im pretty much running out of ideas how to pull this off. The only workaround i can think of is to create tables which contain pre counted values for fixed intervals, which would not be really satisfying since the whole thing should also be fully updateable (we are talking about new values arriving every few milliseconds). Would another db system be better suited for my problem?
Here is the explain output:
Field Type Null Key Default Extra
timestamp bigint(20) NO PRI NULL
ask decimal(6,5) NO NULL
bid decimal(6,5) NO NULL
askvolume decimal(6,5) NO NULL
bidvolume decimal(6,5) NO NULL
# The MySQL server
[mysqld]
port= 3306
socket= "C:/xampp/mysql/mysql.sock"
basedir="C:/xampp/mysql"
tmpdir="C:/xampp/tmp"
datadir="C:/xampp/mysql/data"
pid_file="mysql.pid"
skip-external-locking
key_buffer = 16M
max_allowed_packet = 61M
table_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
log_error="mysql_error.log"
bind-address="192.168.1.2"
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
# commented in by lampp security
#skip-networking
skip-federated
# Replication Master Server (default)
# binary logging is required for replication
# log-bin deactivated by default since XAMPP 1.4.11
#log-bin=mysql-bin
# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1
# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
# MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
# where you replace <host>, <user>, <password> by quoted strings and
# <port> by the master's port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
# MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables' values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host = <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user = <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password = <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port = <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin
# Point the following paths to different dedicated disks
#tmpdir = "C:/xampp/tmp"
#log-update = /path-to-dedicated-directory/hostname
# Uncomment the following if you are using BDB tables
#bdb_cache_size = 4M
#bdb_max_lock = 10000
# Comment the following if you are using InnoDB tables
#skip-innodb
innodb_data_home_dir = "C:/xampp/mysql/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp/mysql/data"
#innodb_log_arch_dir = "C:/xampp/mysql/data"
## You can set .._buffer_pool_size up to 50 - 80 %
## of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 1024M
innodb_additional_mem_pool_size = 20M
## Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 16M
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
Oh the machine is an i7-950 with 6GB of RAM and the system+database is on a SSD. So i think that should not be the problem?
Thanks for your help, it will be highly appreciated!
I don't have a feel for the range of values that you have in you indexed timestamp value but it seems to me that partitioning your table could help you out here. Specifically RANGE partitioning or HASH partitioning.
This should give you a significant performance boost.
If the time ranges can be expressed as a series of ranges (months, days, weeks, etc.), you might introduce something like a date-prefix column, that will significatly reduce the number of examined rows using IN() expression.
Here is an article that exposes the idea: http://www.mysqlperformanceblog.com/2010/01/09/getting-around-optimizer-limitations-with-an-in-list/
First step: if you haven't done it, use explain plan to see what exactly is the bottleneck of your query, and if the engine is using the index(es) correctly.
Second step: partition your table by range on the timestamp. I'm not sure if MySQL/InnoDB has that capability, but if it doesn't you'd better change DBMS.
In any case, MySQL is not really a good choice for high performance: depending on your needs you may be better off with Oracle or Postgre, or even an in-memory storage (especially if you don't care too much for safety as opposed to performance).

how can I export 4.5 GB table from mysql?

I have a table which has 38.406.168 rows and according to size in phpmyadmin 4.5GB. I want to see the last row of the table. Unfortunately I couldn't use select * from ... limit 38.406.166,1 or even I couldn't use select count(*) from ... function.
I changed my.ini in wamp server, but still I get mysql server has gone away error while attempting execute one of these queries. BTW; I couldn't even set an index on ID to make these processes much quicker.
My last try was to export the table to look at the last row. However, It just shows me 123MB of the file.
What should I do? Please help me. The features of the computer is 2.93 GHz, 3.50GB
Here is my my.ini file:
# The MySQL server
[wampmysqld]
port = 3306
socket = /tmp/mysql.sock
skip-locking
key_buffer = 384M
max_allowed_packet = 2000M
table_cache = 4096
sort_buffer_size = 2000M
net_buffer_length = 8K
read_buffer_size = 2000M
read_rnd_buffer_size = 2000M
myisam_sort_buffer_size = 2000M
basedir=c:/wamp/bin/mysql/mysql5.1.36
log-error=c:/wamp/logs/mysql.log
datadir=c:/wamp/bin/mysql/mysql5.1.36/data
(.. these parts are deleted, since there is nothing to set as value)
# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = C:\mysql\data/
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = C:\mysql\data/
#innodb_log_arch_dir = C:\mysql\data/
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 384M
#innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 10M
#innodb_log_buffer_size = 64M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 180
[mysqldump]
quick
max_allowed_packet = 160M
Thank you so much for your help
I tried a lot of stuff and ended up with those 2 working:
Simply mirror the database via mysql's internal master-slave-functions (try google, you ll find good tutorials) onto a simple backup server (most cheap hosting packages will work if they have ssh access)
Try http://www.mysqldumper.net/, the best tool to copy & split huge databases into 100mb-parts. This simple open source tool did everything that "professional" backup scripts couldn't do.
You will want to use the mysqldump command to do this. Here is what I do in linux, but I think it will translate to Windows (I see that you're running WAMP).
mysqldump --opt --force -Q --user=[your_user] -p [database_name] > dump.sql
You may need to change directory to where the mysqldump file is located:
cd c:\path\to\mysql\bin