I need to download locally (macOS BigSur) a large dataset (millions of rows) from an AWS RDS database (MySQL 5.7).
Thanks to this great post I am able to connect and download on my machine some data into a csv file:
mysql --host=$HOST --user $USER --password=$PASSWORD --database=$DATABASE --port=$PORT --batch \
--quick -e "$QUERY" \
| sed $'s/\\t/","/g;s/^/"/;s/$/"/;s/\\n//g' > $FILE_PATH
However, if I extend my query to thousands of records, after few seconds the process stops and the csv ends up truncated (literally the last written row is truncated half way), so I assume there is some kind of stream or timeout or buffer issue.
mysql> SHOW VARIABLES LIKE '%timeout';
+-----------------------------+----------+
| Variable_name | Value |
+-----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| rpl_stop_slave_timeout | 31536000 |
| slave_net_timeout | 60 |
| wait_timeout | 28800 |
+-----------------------------+----------+
Since the command does stops after ~10sec I assume it depends on the connect_timeout value. However I tried setting it with SET ##GLOBAL.connect_timeout=7200 but I get a permission error. I tried adding the --connect-timeout=7200 parameter on the command, but it does not work (which strikes me the most).
Running the query (limited to ~250k rows) on my client (SequelAce) it runs fine, so I can exclude issues with the data or the the SQL script itself.
Any ideas or suggestions?
Are there better tools for the job maybe?
Related
I have various Google Cloud functions which are writing and reading to a Cloud SQL database (MySQL). The processes work however when the functions happen to run at the same time I am getting a Broken pipe error. I am using SQLAlchemy with Python, MySQL and the processes are cloud functions and the db is a google cloud database.I have seen suggested solutions that involve setting timeout values to longer. I was wondering if this would be a good approach or if there is a better approach? Thanks for your help in advance.
Heres the SQL broken pipe error:
(pymysql.err.OperationalError) (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")
(Background on this error at: http://sqlalche.me/e/13/e3q8)
Here are the MySQL timeout values:
show variables like '%timeout%';
+-------------------------------------------+----------+
| Variable_name | Value |
+-------------------------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| have_statement_timeout | YES |
| innodb_flush_log_at_timeout | 1 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| rpl_semi_sync_master_async_notify_timeout | 5000000 |
| rpl_semi_sync_master_timeout | 3000 |
| rpl_stop_slave_timeout | 31536000 |
| slave_net_timeout | 30 |
| wait_timeout | 28800 |
+-------------------------------------------+----------+
15 rows in set (0.01 sec)
If you cache your connection, for performance, it's normal to lost the connection after a while. To prevent this, you have to deal with disconnection.
In addition, because you are working with Cloud Functions, only one request can be handle in the same time on one instance (if you have 2 concurrent requests, you will have 2 instances). Thus, set your pool size to 1 to save resource on your database side (in case of huge parallelization)
I searched a lot on the internet but not found any brief explanation on mysql timeouts with examples. I want to know the meaning of mysql diffenernt timeouts as listed below and also want to know why and when we use them.
+----------------------------+----------+
| Variable_name | Value |
+----------------------------+----------+
| connect_timeout | 10 |
| delayed_insert_timeout | 300 |
| innodb_lock_wait_timeout | 50 |
| interactive_timeout | 28800 |
| net_read_timeout | 3 |
| net_write_timeout | 60 |
| slave_net_timeout | 3600 |
| wait_timeout | 28800 |
+----------------------------+----------+
Also in ruby on rails application i can set read_timeout in my database.yml file. if query is not able to read the data within specified read_timeout value mysql will close the connection. so i also want to know what is the differentce between net_read_timeout and read_timeout
Thanks,
From The Ultimate Guide to Ruby Timeouts
connect (or open) - time to open the connection
read (or receive) - time to receive data after connected
write (or send) - time to send data after connected
checkout - time to checkout a connection from the pool
statement - time to execute a database statement
I have this config
mysql> SHOW VARIABLES where Variable_name like '%timeout';
+----------------------------+-------+
| Variable_name | Value |
+----------------------------+-------+
| connect_timeout | 5 |
| delayed_insert_timeout | 300 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| slave_net_timeout | 7200 |
| table_lock_wait_timeout | 50 |
| wait_timeout | 28800 |
+----------------------------+-------+
10 rows in set (0.01 sec)
mysql>
I needs long time connect, want to unlimited timeout.
Look my php source.
<?php
$link = #mysql_connect("localhost","root",$pw);
...
mysql_query($query,$link);
...
// A long time flows (maybe 28,800sec)
mysql_query($query,$link); // error !!
?>
Please advise.
The answer is NO. You can not set the wait_timeout to unlimited.
You can refer MYSQL wait_timeout
However if you want to change it then you can try like this:
SET GLOBAL connect_timeout=....;
There is a limit for wait_timeout. This configuartion value can put in the configuration file my.cnf (for unix)/ my.ini (for windows).
Type Integer
Default Value 28800;
Minimum Value 1;
Maximum Value (Other) 31536000;
Maximum Value (Windows) 2147483
Assign wait_timeout in the configuration file within the above range and restart the mysql server.
My issue is, my website open too many connections in mysql server this is hanged my website and also generate many requests on apache. i have installed the apache module mod-evasive and mod-security i have also empty the iptables rules basically i am using amazon ec2 so it gives security ip i have blocked all outbound and inbound i have opened only 443,80 and ssh 22 port only but still when i take netstat it shows
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
4 --0.0.0.0
4 ---119.159.195.199
25 ---54.69.254.252
374 ---
on above my question is why my server 374 apache connection on ::1:80 how can block this or reduce this
mysql connection stat are
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| Connections | 5208 |
| Threads_cached | 0 |
| Threads_connected | 54 |
| Threads_created | 5207 |
| Threads_running | 54 |
+-------------------+-------+
5 rows in set (0.00 sec)
my second question is why my sql connections is increasing.
Please any one help me i will really appreciated.
I was working with netbean's database explorer (services tab) and accidentally deleted one of the databases.
I was trying to connect to a database and there were 2 options in the context menu for connecting.
1)Connect
2)Delete
I accidentally hit delete. It did not even ask me for confirmation. Is there anyway to recover the database?
I don't have access to mysql server. I can just connect to the database using mysql workbench.
Thanks a ton!
If you have the binary logging enabled, then you might have a chance to use it to restore your database.
Else use an older backup and restore it !
To find binary logs status :
mysql> show variables like '%bin%';
+---------------------------------+----------------------+
| Variable_name | Value |
+---------------------------------+----------------------+
| binlog_cache_size | 32768 |
| innodb_locks_unsafe_for_binlog | OFF |
| log_bin | OFF |
| log_bin_trust_function_creators | OFF |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 1073741824 |
| sync_binlog | 0 |
+---------------------------------+----------------------+
7 rows in set (0.00 sec)