Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 262145 bytes)
I try to Import a csv file 199MB into mysql database.
upload_max_filesize = 300M
post_max_size = 300M
memory_limit = 32M
It's keep pop the error, what should I change in php.ini
33554432 bytes is exactly 32M.
Try to boost up your memory_limit alot.
You shoul try to increase the memory limit. In PHP > 5.2.1 the default value is 128M.
; Maximum amount of memory a script may consume (128MB)
; http://www.php.net/manual/en/ini.core.php#ini.memory-limit
;memory_limit = 32M
memory_limit = 128M
Related
I use Prestashop+ NGINX + PHP-FPM, with more than 30.000 listings + variations, what are synchronized by an external API server.
My prestashop is running under:
AMD Ryzen™ 5 3600 (6 cores) + 64GB RAM + 2xNVME 500GB (RAID 1) + Gigabit ethernet/network
I am not an expert and i would love to get some recommendations
/etc/my.cnf
[mysql]
port = 3306
[mysqld]
# === Required Settings ===
basedir = /usr
bind_address = 127.0.0.1 # Change to 0.0.0.0 to allow remote connections
datadir = /var/lib/mysql
max_allowed_packet = 256M
max_connect_errors = 1000000
port = 3306
skip_external_locking
skip_name_resolve
tmpdir = /tmp
user = mysql
# === SQL Compatibility Mode ===
# Enable for b/c with databases created in older MySQL/MariaDB versions
# (e.g. when using null dates)
#sql_mode = ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES
# === InnoDB Settings ===
default_storage_engine = InnoDB
innodb_buffer_pool_instances = 42 # Use 1 instance per 1GB of InnoDB pool size
innodb_buffer_pool_size = 42G # Use up to 70-80% of RAM
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_log_buffer_size = 16M
innodb_log_file_size = 2G
innodb_stats_on_metadata = 0
performance_schema = ON
#innodb_temp_data_file_path = ibtmp1:64M:autoextend:max:20G # Control the maximum size for the ibtmp1 file
#innodb_thread_concurrency = 4 # Optional: Set to the number of CPUs on your system (minus 1 or 2) to better
# contain CPU usage. E.g. if your system has 8 CPUs, try 6 or 7 and check
# the overall load produced by MySQL/MariaDB.
innodb_read_io_threads = 64
innodb_write_io_threads = 64
#innodb_io_capacity = 1000 # Max is 2000
# === MyISAM Settings ===
# The following 3 options are ONLY supported by MariaDB & up to MySQL 5.7
# Do NOT un-comment on MySQL 8.x+
query_cache_limit = 10M # UPD
query_cache_size = 50000M # UPD
query_cache_type = 1 # Enabled by default
key_buffer_size = 32M # UPD
low_priority_updates = 1
concurrent_insert = 2
# === Connection Settings ===
max_connections = 750 # UPD - Important: high no. of connections = high RAM consumption
back_log = 512
thread_cache_size = 100
thread_stack = 192K
interactive_timeout = 180
wait_timeout = 180
# For MySQL 5.7+ only (disabled by default)
#max_execution_time = 30000 # Set a timeout limit for SELECT statements (value in milliseconds).
# This option may be useful to address aggressive crawling on large sites,
# but it can also cause issues (e.g. with backups). So use with extreme caution and test!
# More info at: https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_execution_time
# For MariaDB 10.1.1+ only (disabled by default)
#max_statement_time = 30 # The equivalent of "max_execution_time" in MySQL 5.7+ (set above)
# The variable is of type double, thus you can use subsecond timeout.
# For example you can use value 0.01 for 10 milliseconds timeout.
# More info at: https://mariadb.com/kb/en/aborting-statements/
# === Buffer Settings ===
innodb_sort_buffer_size = 2M # UPD
join_buffer_size = 8M # UPD
read_buffer_size = 3M # UPD
read_rnd_buffer_size = 4M # UPD
sort_buffer_size = 4M # UPD
# === Table Settings ===
# In systemd managed systems like Ubuntu 16.04+ or CentOS 7+, you need to perform an extra action for table_open_cache & open_files_limit
# to be overriden (also see comment next to open_files_limit).
# E.g. for MySQL 5.7, please check: https://dev.mysql.com/doc/refman/5.7/en/using-systemd.html
# and for MariaDB check: https://mariadb.com/kb/en/library/systemd/
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
open_files_limit = 80000 # UPD - This can be 2x to 3x the table_open_cache value or match the system's
# open files limit usually set in /etc/sysctl.conf or /etc/security/limits.conf
# In systemd managed systems this limit must also be set in:
# /etc/systemd/system/mysqld.service.d/override.conf (for MySQL 5.7+) and
# /etc/systemd/system/mariadb.service.d/override.conf (for MariaDB)
max_heap_table_size = 128M
tmp_table_size = 128M
# === Search Settings ===
ft_min_word_len = 3 # Minimum length of words to be indexed for search results
# === Logging ===
log_error = /var/lib/mysql/mysql_error.log
log_queries_not_using_indexes = 1
long_query_time = 5
slow_query_log = 1 # Disabled for production
slow_query_log_file = /var/lib/mysql/mysql_slow.log
[mysqldump]
# Variable reference
# For MySQL 5.7+: https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html
# For MariaDB: https://mariadb.com/kb/en/library/mysqldump/
quick
quote_names
max_allowed_packet = 512M
tuning-primer.sh
[root#server2 /]# curl -L https://raw.githubusercontent.com/BMDan/tuning-primer.sh/master/tuning-primer.sh | bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 63986 100 63986 0 0 201k 0 --:--:-- --:--:-- --:--:-- 202k
-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -
MySQL Version 10.2.38-MariaDB-log x86_64
Uptime = 3 days 21 hrs 47 min 51 sec
Avg. qps = 18
Total Questions = 6119342
Threads Connected = 11
Server has been running for over 48hrs.
It should be safe to follow these recommendations
To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/10.2/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service
SLOW QUERIES
The slow query log is enabled.
Current long_query_time = 5.000000 sec.
Since startup, 102804 out of 6119356 queries have taken longer than <long_query_time-when-they-were-executed> to complete.
Your long_query_time may be too high, I typically set this under 5 sec.
BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/10.2/en/point-in-time-recovery.html
WORKER THREADS
Current thread_cache_size = 100
Current threads_cached = 42
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine
MAX CONNECTIONS
Current max_connections = 750
Current threads_connected = 11
Historic max_used_connections = 54
The number of used connections is 7% of the configured maximum.
You are using less than 10% of your configured max_connections.
Lowering max_connections could help to avoid an over-allocation of memory
See "MEMORY USAGE" section to make sure you are not over-allocating
INNODB STATUS
Current InnoDB index space = 137 M
Current InnoDB data space = 255 M
Current InnoDB buffer pool free = 99 %
Current innodb_buffer_pool_size = 42.00 G
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory
MEMORY USAGE
Max Memory Ever Allocated : 91.88 G
Configured Max Per-thread Buffers : 14.05 G
Configured Max Global Buffers : 90.87 G
Configured Max Memory Limit : 104.92 G
Physical Memory : 62.75 G
Max memory limit exceeds 90% of physical memory
KEY BUFFER
Current MyISAM index space = 185 K
Current key_buffer_size = 32 M
Key cache miss rate is 1 : 697
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine
QUERY CACHE
Query cache is enabled
Current query_cache_size = 48.82 G
Current query_cache_used = 1.17 G
Current query_cache_limit = 10 M
Current Query cache Memory fill ratio = 2.39 %
Current query_cache_min_res_unit = 4 K
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MySQL won't cache query results that are larger than query_cache_limit in size
SORT OPERATIONS
Current sort_buffer_size = 4 M
Current read_rnd_buffer_size = 4 M
Sort buffer seems to be fine
JOINS
Current join_buffer_size = 8.00 M
You have had 15352 queries where a join could not use an index properly
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
OPEN FILES LIMIT
Current open_files_limit = 32768 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine
TABLE CACHE
Current table_open_cache = 15994 tables
Current table_definition_cache = 40000 tables
You have a total of 483 tables
You have 557 open tables.
The table_cache value seems to be fine
TEMP TABLES
Current max_heap_table_size = 128 M
Current tmp_table_size = 128 M
Of 146965 temp tables, 6% were created on disk
Created disk tmp tables ratio seems fine
TABLE SCANS
Current read_buffer_size = 3 M
Current table scan ratio = 88 : 1
read_buffer_size seems to be fine
TABLE LOCKING
Current Lock Wait ratio = 1 : 37
You may benefit from selective use of InnoDB.
Current query_cache_size = 48.82 G -- NO! Stop at about 50M; it gets terribly inefficient when it is that large.
Use pt-query-digest on the slowlog; then let's discuss the first few queries. That is likely to help performance a lot.
log_queries_not_using_indexes = 1 -- turn off; otherwise it clutters the slowlog.
We will tackle this in a second pass:
max_connections = 750 # UPD
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
If you would like more analysis, see http://mysql.rjweb.org/doc.php/mysql_analysis#tuning
Digest
Query 1
UPDATE ps_stock_available sa
INNER JOIN ps_mmo_connector_product_map pm
ON sa.id_product = pm.id_product_shop
SET sa.quantity = '10'
WHERE sa.quantity > '10'\G
The first query in the digest is an Update that seems to cap quantity. Presumably, you can eliminate that by never changing the value to more than 10? That would eliminate half the burden on the server. If you don't want to do that, please provide these so I can check the indexes:
SHOW CREATE TABLE `presta`.`ps_stock_available`\G
SHOW CREATE TABLE `presta`.`ps_mmo_connector_product_map`\G
But... What is the problem? This implies that there is not much going on:
# Overall: 281.15k total, 799 unique, 0.54 QPS, 0.01x concurrency
And this says that the "worst" query (that Update) is not very slow or frequent:
# Exec time 48 1313s 18ms 1s 85ms 253ms 88ms 53ms
If you expect the server to become much, much, busier, then I will dig deeper. But for now, the only thing that server will do is snore.
While you are getting those Creates, grab these, too, so I can check the next few queries:
SHOW CREATE TABLE `presta`.`ps_category_product`\G
SHOW CREATE TABLE `presta`.`ps_category`\G
SHOW CREATE TABLE `presta`.`ps_category_shop`\G
SHOW CREATE TABLE `presta`.`ps_search_word`\G
SHOW CREATE TABLE `presta`.`ps_search_index`\G
SHOW TABLE STATUS FROM `presta` LIKE 'ps_tag_count'\G
I see a REPLACE that seems to be recomputing tag counts. At first glance, that seems a terribly inefficient way to do it. But looking at how complex the query is, it may not be practical to improve it. Do you the intent of Query 4?
I',m having a huge issue with running MySQL#5.7 on my freshly installed 16" Macbook (with OSX 10.15.1 Catalina) During certain actions I get errors like
PDO::__construct(): MySQL server has gone away.
This is caused by the following error I found in the MySQL log.
2019-11-27T13:24:04.835245Z 0 [Warning] File Descriptor 3226 exceeded FD_SETSIZE=1024
After some research, I tried stuff like sudo launchctl limit maxfiles 65536 200000
When i run launchctl limit i get the follwing data
cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8388608 67104768
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 4096 4096
maxfiles 524288 524288
This looks fine to me. To get the max processes and max files correct I also tried
sudo sysctl -w kern.maxfilesperproc=524288
my.cnf looks like this
[mysqld]
open_files_limit=999999
local_infile=ON
secure_file_priv=""
max_allowed_packet=1073741824
max_connections=100000
key_buffer_size=2G
innodb_buffer_pool_size=12G
query_cache_size=67108864
query_cache_type=1
query_cache_limit=4194304
table_open_cache=4096
innodb_buffer_pool_instances=24
innodb_sort_buffer_size=2G
sort_buffer_size=1G
innodb_flush_log_at_trx_commit=0
innodb_log_file_size=3G
interactive_timeout=3600
max_connect_errors=1000000
thread_cache_size=4096
log_error=/var/log/mysql/error.log
[mysqld_safe]
open_files_limit=999999
There is of course a solution to change my table_open_cache to a lower value but that hurts performance and before i always had this on a higher number.
Anybody any clue where this FD_SETSIZE is coming from and how to change it so its used properly
rebooting has no effect by the way.
Resource explaining issue: https://expressionengine.com/blog/mysql-5.7-server-os-x-has-gone-away
try setting following environment variables in mysql configuration file (my.cnf)
interactive_timeout = 300
wait_timeout = 300
I was having this issue on Big Sur v 11.6. The solution for me was modifying the MySQL config with:
max_allowed_packet=256M
table_open_cache=250
I have a 2GB DB and it is very slow below my configuration file variable , what I shall change to make it faster
read_buffer_size=1M
read_rnd_buffer_size=256K
sort_buffer_size=256K
innodb_additional_mem_pool_size=2M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=1M
innodb_buffer_pool_size=16M
innodb_log_file_size=10M
innodb_thread_concurrency=8
max_connections=151
old_passwords=0
max_user_connections=30
thread_cache_size=4
thread_stack=196608
binlog_cache_size=32768
net_read_timeout=30
net_retry_count=10
net_write_timeout=30
thread_concurrency=10
open_files_limit=4161
table_definition_cache=400
Threads_cached 2
Threads_connected 2
Threads_created 12
Threads_running 1
Usually I'll just leave the rest alone and only adjust:
innodb_buffer_pool_size to around 70~80% or your 2GB, in this case 1.6G
innodb_log_file_size to half of above, in this case 800M
max_connections depends on your application, normally I'll set it to at least 200.
But "slow" is very general and it may be impacted by your logic/structure as well.
I have a copy of my e-shop in my local server (XAMPP). The computer it's an i7 3820 16gb ram with xampp installed in SSD.
I change this values in php.ini file:
memory_limit -1
max_input_time 9999999999
max_execution_time 9999999999
max_file_uploads 1024
post_max_size 2048M
upload_max_filesize 120M
I'm trying to import by the csv importations tool included in Prestashop (v1.6) a file with more or less 9000 rows. The file is like this:
ID: 1
Attribute: AAA:select:1%BBB:select:2%CCC:select:3%DDD:select:4%EEE:select:5%FFF:select:6
Value: UUU:1%VVV:2%WWW:3%XXX:4%YYY:5%ZZZ:6
Price impact: 999
(I used "%" instead of "," to split rows because otherwise I have problems when the strings have "," in text)
Well. I am not able to import more than 1000 rows by file. And still takes about 60-90 minutes by file. If I try with larger files the browser returns me a timeout error.
Therefore, the question is, do you know some trick to optimize this process? another option? more parameters to change in php.ini? I appreciate any advice to help me improve the process.
when i m trying to load the .sql file into wamp using phpmyadmin
i m getting the fatal error as below
Fatal error: Maximum execution time of 300 seconds exceeded in C:\wamp\apps\phpmyadmin4.5.1\libraries\plugins\import\ImportSql.class.php on line 220
Location: C:\xampp\phpmyadmin\config.inc.php
$cfg['ExecTimeLimit'] = 600;
In C:\xampp\php\php.ini
Look for and Change max_execution_time
you already have execution 300 seconds, so go to that file and increase max_execution_time to to what ever you want of seconds.
Add this line
$cfg['ExecTimeLimit'] = 6000;
to phpmyadmin/config.inc.php
And Change php.ini and my.ini
post_max_size = 750M
upload_max_filesize = 750M
max_execution_time = 5000
max_input_time = 5000
memory_limit = 1000M
max_allowed_packet = 200M (in my.ini)
OR
You may also go to xampp\phpMyAdmin\libraries\config.default.php,
and change this line to fix that error.
$cfg['ExecTimeLimit'] = 600;
You are uploading huge sql file. So php cannot executed all queries within a default time limit. You need to change the php execution time limit.
Your Answer is solved in Fatal Error while trying to export a result of a query in mysql
Hope this would help you please refer: http://www.nginxtips.com/configure-max_execution_time-in-php-fpm-using-nginx/