Im working on a browser game and I write 7 mysql events for each player.
5 of events update a table row every 5 secs.
And two of them update 2 other tables every sec.
I have a linux vps that its ram is 512 M and cpu has 1 core.
How many online players can support this VPS.
thanks.
Minimum system requirement for mysql for windows (86 and 64) is 800 mb ram and 500 mb hard disk space
Read this article to get some further knowledge.
dev.mysql.com/doc/mysql-monitor/3.0/en/system-prereqs-reference.html
Related
My DB has around 15 tables, each with 40 columns, with 10.000 rows each.
Most of it with VARCHAR, some indexes and foreign keys.
Sometime I need to reconstruct my database (design flaw, working on it), which takes about 40 seconds locally. Now I'm trying to do the same to a AWS RDS MySQL 5.75 instance, but it takes forever, something like 40-50 minutes. The last time I had to do this same process it took no more than 5 minutes, still way more than the local 40 seconds, but I'm happy with it.
My internet speed is at about 35 Mbps Download / 5 Mbps Upload.
I know it's not fast, but it's consistent, and it hasn't changed since my last rebuilt.
I enabled General Logs, but all I can see are the INSERT queries, occasionally some "SELECT 1".
I do have same space for improvements on my code, but still, from 00:40:00 to 50:00:00, it seems that there's something else going on.
Any ideas on how to diagnose and find the bottleneck?
Thanks
--
Additional relevant information:
It is a Micro instance from AWS, all of the relevant monitoring indicators are basically flat: CPU at 4%, Free Storage Space at 20.000 MB, Freeable Memory at 200 MB, Write IOPS at around 2,5, the server runs a 5.7.25 MySQL, 1vCPU, 1Gb of RAM and 20GB of SSD. This is the same as 3 months ago when I last rebuilt the database.
SHOW GLOBAL STATUS: https://pastebin.com/jSrAzYZP
SHOW GLOBAL VARIABLES: https://pastebin.com/YxD7dVhR
SHOW ENGINE INNODB STATUS: https://pastebin.com/r5wffB5t
SHOW PROCESS LIST: https://pastebin.com/kWwiyGwf
SELECT * FROM information_schema...: https://pastebin.com/eXGBmetP
I haven't made any big changes to the server configuration, except enabling logs, e maxing out max_allowed_packets and saving logs to file.
In my backend I have a Flask app running, when it receives the API call, it takes a bunch of pickled objects and adds them all to the database (appending the Flask SQLAlchemy class to a list) and then running db.session.add_all(entries), trying to run a bulk operation. The code is the same, both for localhost and my remote server.
It does get slower in three specific tables, most of them with VARCHAR columns, but nothing different from my last inserts - it seems odd that the problem would be data, or the way the code is structured, or at least doesn't seem reasonable that this would result in a 20 second (localhost) to 40 minutes (hosted server) time, specially when the rest of the tables work mostly the same.
Enable the slow log, set long_query_time=0, run your code, then put the resulting log through mysqldumpslow.
Establish which queries contribute most to slowness and take it from there.
Compare the config between your old server and your new one.
Also, are they the same version of MySQL? 5.6, 5.7 and 8.0 can produce very different execution plans (with 5.6 usually coming up with the sane one if they differ).
Rate Per Second = RPS
Suggestions to consider for your AWS RDS Parameters group
thread_cache_size=24 # from 8 to reduce threads_created count
innodb_io_capacity=1900 # from 200 to enable more use of SSD IOPS capacity
read_rnd_buffer_size=128K # from 512K to reduce handler_read_rnd_next RPS of 21
query_cache_size=0 # from 1M since you have QC turned off with query_cache_typ=OFF
Determine why com_flush is running 13 times per hour and get it stopped to avoid table open thrashing.
I found that after migrating to RDS all my database Indexes are gone! They weren't migrated along with the schema and data. Make sure you're indexes are there.
Also, MySQL query cache is OFF by default in RDS. This won't help the performance of your initial query, but it may speed things up in general.
You can set query_cache_type to 1 and define a value for query_cache_size. I also changed the thread_cache_size from 8 to 24 and innodb_io_capacity from 200 to 1900 don't know if it helps you.
Also creating AWS DB Parameter Groups helped me a lot with configuring and tuning DB variables. Here you can read more:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html
I am running a flask app with mysql db.
I have 400000 records in a table and there is a query (insert with select) that takes around 3 seconds if run single.
But when I tried to load test it by hitting the api with multiple requests at a time (like 20 hits at a time, 50 and 100 hits at a time), the response for all requests coming at once. For example, if total 100 concurrent requests takes around 3 mins, then all those individual requests are starting immediately but giving response after 3 mins only (instead of 3 or 4 seconds).
Also, I tried with 1 gb ram server, 4 gb ram server and also 32 gb ram server with 16 cpus. Here is the response as below:
# 4GB RAM, 2 CPUS server with only Mysql installed in it
Total time is: 0:05:29.752275 (all 100 requests getting response after 5 mins(total time), not 3 or 4 seconds)
successful: 89
Failed: 11
Tried: 100
# 32 GB RAM, 16 CPUs server with only Mysql installed in it
Total time is: 0:05:17.119773 (all 100 requests getting response after 5 mins(total time), not 3 or 4 seconds)
successful: 86
Failed: 14
Tried: 100
So, if you see, both 4gb and 32gb servers has almost no difference in performance. So it seems like something totally wrong with my setup/configuration/query.
More details:
I ran another script where I directly hit db in the script without any api. This way, even though server has 4gb RAM, mysql server dying(segmentation fault and also mysql server dies) for just 3 concurrent requests.
But then I gave like 0.2, 0.3 and 0.5 milli seconds delay between each hit/thread so the results were slighly meaningful. Each request used to take total time but with 0.5 ms delay between each hit, each request completing in less than 10 seconds.
Can I do anything so my server easily returns response fast for atleast 100 concurrent requests without any gap between requests(and is that necessary?)
Any thoughts on what to do here?
I think the root cause is flask. Flask is not good in multi-process/thread at all.
I meet this problem before, then change to Tornado and use supervisord to keep Tornado as daemon mode.
another solution is Gunicorn => https://intellipaat.com/community/12737/how-to-run-flask-with-gunicorn-in-multithreaded-mode
Simply put, "atleast 100 concurrent requests without any gap" is not realistic. The user goes to the client, which connects to the database, which takes queries rapidly, but not really simultaneously. That is, in real life queries rarely start simultaneously.
Also, if you have the configuration (MySQL's max_connections) and/or the corresponding setting in the client too high, then you are asking for the "thundering herd" syndrome. It's like being in an over-crowded grocery store and you can't move your cart because all the space is taken.
More specifically, 16 CPUs will stumble over each other vying for resources when you throw 100 queries into the mix "concurrently".
As for inserting a lot of rows, there are several techniques.
LOAD DATA is very fast.
"Batched INSERT" is fast. This is where a single INSERT has lots of records. I often see 10x speedup with 100 rows at a time. (versus single-row inserts)
BEGIN...COMMIT around a bunch of single-row inserts. This avoids some of the "transaction" overhead.
Avoid UNIQUE indexes (other than the PRIMARY KEY) on the table you are loading.
Ping-ponging staging tables: http://mysql.rjweb.org/doc.php/staging_table -- this allows multiple clients to rapidly feed data in.
I would like to optimize MySQL for a single user (me). Noone else is connected to the database. MySQL is installed locally on my PC. The data covers around 70 tables with 150 GB of data in total.
The data is static, so I won't do changes to it (no UPDATE or DELETE). I want to analyze the data, so I will run large and complex queries including SELECT and JOIN over large proportion of the data.
Machine:
Windows 7 64-bit
Intel Core i7-4800MQ # 2,70 GHz
32 GB RAM
2x 512 GB SSD
MySQL 5.7, INNODB
What I did so far:
Deactivated HyperThreading (MySQL uses only one virtual Core per
query - CPU usage 12.5% --> 25%)
Declare primary keys and
Indexed all foreign keys
innodb_buffer_pool_size = 25G
max_connections=10
Use of InnoDB
So what can I do to optimize the configuration (not the query itself) for single-user-queries?
database engine?
general configuration?
better cache all the information of the joins?
Note:
Under the current configuration, the CPU usage is the bottleneck because it is on 25% when I run a complex query. As test i tried some huge queries (fetching a lot of data). If I can believe the timing of MYSQL Workbench the duration was only 4 seconds, but after 10 hours of running it couldn't finish fetching the data...
Thanks a lot!
You could try MySiam. It is especially good for read-intensive (select) tables. And you should select a large key_buffer_size. But I am not sure about the JOIN performance... Give it a try
[RESOLVED]
IT SEEMS PHP MYADMIN VARIABLES DID THE TRICK.
I SET wait_timeout to 30 , and Lock_wait_timeout to 50
Took almost 6 hours to get back to stable,including several restarts,may be time needs to read those changes.
PROBLEM :
Site Address http://topyaps.com
Num of queries on homepage 322 in 2 seconds
Nothing in Slow query log file.
I am using BITNAMI LAMP SERVER[amazon ec2] to run my wordpress based heavy site.
Problem is,my server crashes every 10 minutes.
Reason,as i guess,
when i check processlist using putty,it seems to be increasing constantly at very high rate ,all specifies sleep command.
I tried setting variables like :
wait_timeout=30
interactive_timeout=30
connect_timeout=15
max_user_connections=25
max_connections=999
but it doesn't seem to do any help.
LINK TO THE PICTURE OF PHPMYADMIN MONITOR,FOR A SINGLE PAGE LOAD[homepage]: http://postimg.org/image/5qqgb30xb/
HERE IS WHAT I GOT FROM PHPMYADMIN STATUS:
Questions since startup: 50,617 Documentation
ø per hour: 138,361
ø per minute: 2,306
ø per second: 38
Statements # ø per hour %
select 46,128 126.1 k 91.13
set option 2,452 6,702.5 4.84
change db 1,226 3,351.3 2.42
update 516 1,410.5 1.02
insert 135 369 0.27
delete 116 317.1 0.23
show binlogs 13 35.5 0.03
show variables 5 13.7 0.01
show processlist 5 13.7 0.01
show master status 4 10.9 0.01
show slave status 4 10.9 0.01
show databases 4 10.9 0.01
show tables 3 8.2 0.01
show status 3 8.2 0.01
show grants 1 2.7 <0.01
kill 1 2.7 <0.01
show table status 1 2.7 <0.01
select
set option
change db
update
insert
delete
show binlogs
Other
91%5%
I checked out my page queries it shows "328 queries in 2 seconds".
How to actually stop these connections ???
I'd look to see what these queries are first. Why are you having that many. If they are all repetitive queries and they are called on every connection, I would consider caching. You can install memcached and a plugin like W3TotalCache to cache your database queries. You can also cache frontend using W3TotalCache.
Memcached is a key-value store that stores the result of queries in the memory. So your database does not get hits on the same queries every time.
Also if you are using innodb as the storage engine, change the innodb_buffer_pool_size to 60% of your RAM
Although I can barely access yout site, the query to page ratio seems very high for a common wordpress install. My best guess is that you have too many plugins which in turn run too many custom queries on each page load.
You need to identify the source of this queries and find a way to offload them to a cache mechanism such as memcache. There are some caching plugins such as W3TC that offer to do this for you, but in the case of custom tailored queries it might not be too effective.
You say you're running on a bitnami lamp configuration, so your mysql runs in the same machine as your webserver. Have you considered running a separate mysql server, such as Amazon RDS? This won't be a magical solution, but will allow you to better diagnose the bottleneck while at the same time stopping you from tampering with mysql configuration, which is rarely the way to go.
What DNS are you using? I've found out that using Cloudflare's free dns service does provide a basic caching layer for static files, so user might be served a static copy of your html when they hit the frontpage. You'd have to configure cloudflare to perform an aggresive html caching. But as I said, it's free and it helps.
I have a denormalized table product with about 6 million rows (~ 2GB) mainly for lookups. Fields include price, color, unitprice, weight, ...
I have BTREE indexes on color etc. Query conditions are dynamically generated from the Web, such as
select count(*)
from product
where color = 1 and price > 5 and price < 100 and weight > 30 ... etc
and
select *
from product
where color = 2 and price > 35 and unitprice < 110
order by weight
limit 25;
I used to use InnoDB and tried MEMORY tables, and switched to NDB hoping more concurrent queries can be done faster. I have 2 tables with the same schema, indexes, and data. One is InnoDB while the other is NDB. But the results are very disappointing：for the queries mentioned above, InnoDB is like 50 times faster than NDB. It's like 0.8 seocond vs 40 seconds. For this test I was running only a single select query repeatedbly. Both InnoDB and NDB queries are using the same index on color.
I am using mysql-5.1.47 ndb-7.1.5 on a dual Xeon 5506 (8 cores total), 32GB memory running CentOS 5. I set up 2 NDB Data nodes, one MGM node and one MYSQL node on the same box. For each node I allocated like 9GB memory, and also tried MaxNoOfExecutionThreads=8, LockPagesInMainMemory, LockExecuteThreadToCPU and many other config parameters, but no luck. While NDB is running the query, my peak CPU load was only like 200%, i.e., only 2 out of 8 cores were busy. Most of the time it was like 100%. I was using ndbmtd, and verified in the data node log and the LQH threads were indeed spawned.
I also tried explain, profiling -- it just showing that Sending data was consuming most of the time. I also went thru some Mysql Cluster tuning documents available online, not very helpful in my case.
Anybody can shed some light on this? Is there any better way to tune an NDB database? Appreciate it!
You need to pick the right storage engine for your application.
myISAM -- read frequently / write infrequently. Ideal for data lookups in big tables. Does reasonably well with complex indexes and is quite good for batch reloads.
MEMORY -- good for fast access to relatively small and simple tables.
InnoDB -- good for transaction processing. Also good for a mixed read / write workload.
NDB -- relatively less mature. Good for fault tolerance.
The mySQL server is not inherently multiprocessor software. So adding cores isn't necessarily going to jack up performance. A good host for mySQL is a decent two-core system with plenty of RAM and the fastest disk IO channels and disks you can afford. Do NOT put your mySQL data files on a networked or shared file system, unless you don't care about query performance.
If you're running on Linux issue these two commands (on the machine running the mySQL server) to see whether you're burning all your cpu, or burning all your disk IO:
sar -u 1 10
sar -d 1 10
Your application sounds like a candidate for myISAM. It sounds like you have plenty of hardware. In that case you can build a master server and an automatically replicated slave server But you may be fine with just one server. This will be easier to maintain.
Edit It's eight years latar and this answer is now basically obsolete.