RDS uses the same set of parameters for all instance sizes and they don't seem to be that far from defaults (max_packet_size being too small etc.)
Has anyone looked at them and reviewed them for how sensible they are, especially on relatively bigger instances?
The default buffer size for MyISAM is too small in RDS parameter group, I have made changes of that, make significant improvement in read/write performance. However, just beware, not all MySQL parameters are honored by RDS, only those ones shown in parameter group.
you can always change the parameter set to fit your application.
Related
I'm using a MySQL-Server from AWS RDS. I would like to inspect the queries made by the app to optimize them. My problem is, that almost every query is longer than 1024 chars (which is the max-size, as stated here).
So I cannot identify the query by the first 1024 chars, as thats only the SELECT-Part - the interesting parts WHERE, ORDER, and so on are truncated. Since the app uses an ORM-System, I cannot change the queries to shorten them.
Already tried to increase the option performance_schema_max_digest_length in the parameter-group to 4096, but that has no effect (no change can be seen in the options directly on the server).
What can I do?
That is a static parameter, so rebooting your instance may resolve the issue. Please see the following explanation from the AWS docs:
When you change a static parameter and save the DB parameter group,
the parameter change takes effect after you manually reboot the DB
instance. You can reboot a DB instance using the RDS console or by
explicitly calling the RebootDbInstance API action (without failover,
if the DB instance is in a Multi-AZ deployment). The requirement to
reboot the associated DB instance after a static parameter change
helps mitigate the risk of a parameter misconfiguration affecting an
API call, such as calling ModifyDBInstance to change DB instance class
or scale storage
Working with Parameter groups
Now, around three years later, I discovered a page in the documentation about that topic which answers my question:
Accessing more SQL text in the Performance Insights dashboard
So for me it now works as follows:
parameters max_digest_length and performance_schema_max_digest_length are high enough (8192 does it in my case)
in performance insights, tab "Top SQL" only the first 500 chars of the general query are shown. When clicked on the plus on the left, it expands and a query with parameters is visible. Click on the dot in the first column to see the full query in the area below.
I have an AWS RDS MySql Cluster. I'm trying to Auto Scale on Mass Write operations, but unable to do so. But, when I'm Running Read Queries it Scales properly. I'm getting "Too Many Connections" error on write. Can anyone let me know what I'm doing wrong? Thanks in advance.
[Edit: 1]
Screenshot of AWS RDS Cluster Config
I've kept the connection limit to 2 because I was testing.
When I'm sending Multiple read requests to AWS RDS I can see new Instances being launched in my RDS Instances Section:
I've also set Scale In Cool Time to 0 so that it will launch a new Instance Instantly. When I'm reading from the database using read endpoint, Auto Scaling is working properly. But when I'm trying to insert data using write endpoint, Auto-Scaling is not working.
Your question is short on specifics so I will list some possible ways to figure this out and possible solve it.
RDS scaling takes time, so you cannot expect that your DB will increase in capacity instantly when a sudden spike of traffic exceeds its current capacity.
The maximum number of connects to a MySQL instance is set by max_connections in your parameter group. How many connections are happening and what is the max_connections value? This value affects memory usage, so be review any changes. Note: Increasing this value does not always help if there is a bug in your client code that erroneously creates too many connections. If the number of peak connections is exceeding the max_connections value, sometimes you just need to scaled up to a larger instance. Details determine the correct solution.
Use MySQL's Gobal Status History and look into what happens and when. This is useful for detecting locking or memory issues.
Today I have notice that my Amazon RDS instance memory monitor threshold shows me with red line. Here I have attached screen shot for the same.
So, My question is what is that Memory threshold, and why it is crossing limit? Anything wrong with my instance? What is the solution to decrease/control this hike?
The Red line you see is a threshold set by AWS if the RDS is causing that threshold many times then there might be a performance issue that you need to take a look.
MySQL try to use all available memory as needed. However, the limits are defined by RDS' server parameters which you can modify and you may not need to scale up your server.
RDS instances are created with default values for those parameters (the most relevant of them being innodb_buffer_pool_size) to optimize memory usage. In order to see which server variables are applied to your instance, connect to it and execute the "show global variables" command.
It is normal for that number to go up and down as matter of course.
If you are seeing performance issues and you have no more freeable memory, then you should be looking at causes or upgrading to a larger instance.
Those values may not be right for all workloads, but you can adjust them as needed using parameter groups. This document explains how you can use parameter groups:
http://docs.amazonwebservices.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html
I took over a project some time ago, where file binaries were stored as BLOBs. Those were from sizes of 0.5-50 Mb, therefore that table was touched as less as possible (-> eBeans lazyloading etc). The BLOB stuff worked quite fine as long as the whole system was running on one dedicated server, once we switched to AWS EC2 instances + RDS, things were (obviously) slower.
Therefore I switched the data storage from BLOB to S3 (+ reference to the bucket/key stored in the DB), which is much faster for our backend and our clients.
Now my question is, obviously the programmer before set up the mySQL DB to handle bigger chunks of data (max packagesize etc), and I also stumbled over some discussion about connection pool size.
What are critical parameters to check in the mySQL setup, and what are effective ways of evaluating them?
The most likely answer to your question is "change nothing."
MySQL has many, many, many "tunable" parameters, and there is an absolute wealth of bad advice available online about "optimizing" them. But this is a temptation best avoided.
To the extent that system variables have been changed from defaults, if you ever find yourself in a situation where you believe tweaking the configuration is necessary, your first instinct should be to revert settings to their defaults unless you have a specific and justified reason not to.
Settings like max_allowed_packet if set too small, will break some things (like large blobs) but if set larger than necessary will have little or no impact... the "excess" isn't allocated or otherwise harmful. In the case of max_allowed_packet, this does impose a constraint on memory use by limiting the amount of memory the server would ever need to allocate for a single packet, but since it's a brick wall limit, you don't necessarily want to shrink it. If you aren't sending packets that big, it isn't hurting anything.
It is safe to increase the value of this variable because the extra memory is allocated only when needed. For example, mysqld allocates more memory only when you issue a long query or when mysqld must return a large result row. The small default value of the variable is a precaution to catch incorrect packets between the client and server and also to ensure that you do not run out of memory by using large packets accidentally.
http://dev.mysql.com/doc/refman/5.7/en/packet-too-large.html
Other parameters, though, can have dramatically counter-intuitice negative effects because the range of "valid" values is a superset of the range of "optimal" values. The query cache is a prime example of this. "But it's more cache! How can that be bad?!" Well, a bigger house increases the amount of housework you have to do, and the query cache is a big house with only one tiny broom (a global mutex that each thread contends for when entering and leaving).
Still others, like innodb_buffer_pool_size only really have one relatively small optimal range of values for a given server. Too small will increase disk I/O and impair performance because the pool is smaller than the system could support, too large will increase disk I/O due to the server using swap space or crash it entirely by exhausting the system of every last available kilobyte of free RAM.
Perhaps you get the idea.
Unless you have a specific parameter that you believe may be suboptimally configured, leave a working system working. If you change things, change them one at a time, and prove or disprove that each change was a good idea before proceeding. If you are using a non-default value, consider the default as a potentially good candidate value.
And stay away from "tuning scripts" that make suggestions about parameters you should change. Those are interesting to look at, but their advice is often dangerous. I've often thought about writing my own one of these, but all it would do is check for values not set to the default and tell the user to explain themselves or set it back. :) Maybe that would catch on.
I have a server with multiple MySQL databases.
Periodically, I have a problem with one database that may be because of programming errors or data errors. Maybe there's a runaway query. Problems happen... But that problem with the one database can slow down or lock up all the databases on the server, making a smaller problem a huge problem.
Is there anything I can do to contain the effects within one database and let the other databases run smoothly.
One obvious solution would be to run multiple MySQL instances. Although there may be less drastic measures you can take, depending on the exact nature of the specific error conditions you're trying to avoid.
You might have a look at Setting Account Resource Limits. Setting the max per-user connections may go a long way to help some of the problems it sounds like you're facing.
Certainly other database systems can provide additional safety against these types of things, but that doesn't mean MySQL can't be configured for your needs. PostgreSQL, for instance, provides a "statement timeout" variable, which we use at my work, to set a global upper-bound for query run-time, just as a safe-guard against long-running queries. (It can be easily overridden on a per-connection basis if we expect a specific query to take longer than your default of 300 seconds) I don't find a similar option for MySQL.
To fully contain problems from runaway/recursive queries, you need to move that database onto it's own server. The query could potentially use up all allocated memory/disk space that your other databases need.