I have 5 physical servers with low end memory & cpu resources. I want to create 1 cluster using all these servers and want to run mysql db on the same such that mysql db would utilize 5 server's CPU power to execute db queries & same for memory. Could you please help me understanding how to achieve this?
Regards,
You should be aware that MySQL clustering is primarily for availability, not performance. With low end servers, your performance will not be nearly as good as one server with the same aggregate cpu/memory resources.
MySQL doesn't scale up like that. The only effective way to scale up MySQL's performance is more memory.
However, if you have the resources to re-tool your application and your database will partition well at the table level, then it's possible to have your app connect to multiple databases depending on what tables it needs. You basically will want an ORM layer (such as Hibernate) that can be taught to associated an object's table with a particular database.
Related
I have an application that is hosted in AWS ECS and having the database in AWS RDS. I'm using a microservice-based container architecture for my application. The frontend of the application is in Angular and Backends are in Java and Python. Right now, the database size is ~1GB. The database size will increase day by day as the scraped data will be inserted daily.
Right now, some queries are taking 4-6 seconds to execute. We need to host this application to the public and there are a lot of users will be using the application. So when we load tested the application with 50 users, I found that the CPU of RDS reached 100% and some queries had taken more than 60 seconds to execute and then timed-out. Also, the CPU and memory of other microservices (frontend and backend) are normal. I have tried vertically scaling the application up to 64GB RAM and 4 vCPUs but still, this condition remains.
Is this an issue with the query or can I do anything with the database server configuration?
The RDS storage I'm using is 100GB with a general-purpose SSD. So, I guess there will be only 300 IOPS, right? I'm planning to use RDS read replicas but before that, I need to know is there anything that I need to do for improving the performance? Any database configurations etc?
I also not have a good idea about the MySQL connection count. Right now, it is using a total of 24 connections. Do I need to change the connection count also?
Query Optimisation
As Tim pointed out, try and optimise the queries. Since you have more data getting inserted, consider indexing the table and make the queries to use indexed columns if possible. also consider archiving unused old data.
number of connections
If you have control over the code, you can make use of database pools to control the number of connections that your applications can use.
CPU usage
the CPU usage is highly related to the performance of the queries, once you optimise the queries, the CPU usage should come down.
disk usage
Use the cloudwatch metrics to monitor the disk usage, based on that, you can decide on a provisioned IOPS disk.
hope this helps.
Last night I asked a similar question at DB StackExchange platform, but no one replied so putting my question here.
I have huge DB(almost 7-9GiB) over Amazon RDS also CPU Utilization & I/O is also high, I'm not good at optimizing Database queries, enhancing Data Performance and etc. I need help with this DB, in terms of optimization and improve server performance so that users can gain rich experience.
About my research:
As also mentioned there at DBA. Should I upgrade RDS t2.micro to premium db-instance? or create another DB replica on the premium server and balance this with help of EBS(Load Balancers). So, the server becomes smooth for at-least now. Likewise,
RDS Load Balancing Tricks
Then, I need to optimize SQL queries and enhance APIs performance and update DataBase on RDS by removing EBS & Premium Server?
Some general optimizations you could do,
Instead of using t2.micro migrate to a instance class with high IO capacity (As you have mentioned). t2.micro is for general purpose and not recommend for production databses.
Convert from standard storage to Provisioned IOPS storage (If you have high disk utilization, else a general purpose SSD would also be fine)
Depending on the read traffic, you might want to consider using Read replicas.
Other than that, it will narrow down to general DB optimization techniques such as proper indexing etc.
Some of the AWS RDS specific best practices are listed in this doc.
I'm posed with a simple problem of scaling. As college students, we are new to setting up AWS services, droplets, scaling etc. We are stuck at deciding the architecture of our app. I'm unable to decide whether to use a big computing AMAZON EC2 or smaller multiple instances for benchmarking performance.
Even after code optimization, our MySQL queries are not as fast as we want it to be and clearly our hardware will address this problem. We are looking for high performance servers which require mostly searching a lot of MySQL FULL INDEXED search queries over 1.3M records (which clearly is a CPU and Memory intensive task). We intend to switch over to Solr at a later point of time. Both these tasks are very CPU demanding and RAM dependent. As of now, we are running our web app stack entirely on a single CPU with 2 cores and 4 GB RAM. However, we now wish to split the load up into multiple, say 5 instances/droplets of each 2 cores and 4 GB RAM.
Our primary concern is that, If we did create multiple ec2 instances/ droplets, wouldn't there be a considerable overhead for communicating between the instances/droplets for a standard MySQL search. As far as I know, the MySQL connection uses sockets to connect to local/remote host. Being a remote communication between 4 servers, I would expect significant overhead for EACH query.
For instance, let's say I've setup 4 instances and I've been allocated these IP's for each of them.
Server 1- x.x.x.1
Server 2- x.x.x.2
Server 3 - x.x.x.3
Server 4 - x.x.x.4
I setup a MySQL server and dump my databases into each of these instances (sounds like a very bad idea). Now I make a MySQL connection using python as:
conn1 = MySQLdb.connect(host=server1,user=user,passwd=password,db=db)
conn2 = MySQLdb.connect(host=server2,user=user,passwd=password,db=db)
conn3 = MySQLdb.connect(host=server3,user=user,passwd=password,db=db)`
conn4 = MySQLdb.connect(host=server4,user=user,passwd=password,db=db)
Since, each of these databases arn't on the localhost, I would guess that there is a huge overhead involved in contacting the server and getting the data for each query.
My Thoughts:
I'm guessing there must be a solution for integrating different droplets/ instances together. Unfortunately, I haven't found any resources to support that claim.
I've looked into Amazon RDS, which seems like a good fit. But again, I wouldn't be able to benchmark against a 4 instances MySQL search or a single huge AWS RDS server (given, it is quite expensive for new apps.)
We are also unsure of replacement of python with popular languages for scaling such as Scala which will help me tackle this problem of dealing with multiple servers.
Any suggestions will be greatly appreciated by our 3 member team :)
I currently have a cloud server with 2 CPU's and 2GB RAM, Ive noticed that MySQL performs SELECT queries quite slowly on a table size greater than 250,000 rows unless the queries are cached. The table has all the correct indexes and my queries are optimized.
I have run the same MySQL setup on my personal computer and the queries are much faster. So I am thinking its more to do with the server itself rather than the MySQL queries or table setup.
I am considering upgrading the cloud server to 4GB Ram and 4CPU's or switching to a dedicated server with the same spec.
Will changing from a cloud server to a dedicated server improve MySQL performance?
Though Cloud computing utilizes the SAN storage, you will be sharing the resources like CPU, RAM and Storage among number of VMs. This may cause the slowness in I/O operations and hence in your MySQL queries.
In dedicated server, all resources will be allocated to your applications only. Thus, switching to a dedicated server will certainly improve the MySQL server performance. Also, when you have database driven application, SSD drives are highly recommended.
2GB is tiny by today's norms. Ar you using InnoDB or MyISAM? What is the cache size? Does it have SSDs?
It sounds like the table is bigger than can be cached, and that the slow query is scanning the entire table. That says that the query and/or indexes are not well optimized. Let's see the SELECT and SHOW CREATE TABLE; more tuning may be possible.
Does the Cloud limit your IOPs? That mighe explain the difference. And it would further indicate that we should study the query.
Given your query takes 20 seconds to run on Cloud environment, we can rule out network latency problems. To find out the bottleneck. Try using "top" while running the large query on cloud.
If CPU utilization is high, change to a high CPU type, maybe higher memory as well (you can see memory utilization with "top" as well).
If the %CPU of idle or wait is high, it basically suggests underlying disk is the bottleneck here. You can verify this by running some "fio" test, or use "iostat" and monitor the device usage for the query. The solution is adding more disk space or use a SSD device on cloud.
I will prefer cloud always. when it come to dedicated server there you can't find elasticity on server usage. choose a good cloud architecture to enhance the application performance. you have set proper auto scaling, devops methods.
You can also use a remote database to connect your application in order to avoid I/O request to application server.
I want to use a small mysql database in order to store some data that i going to calculate on a VM of GCE (by using Talend).
After store the data on the mysql i want to connect to it by using Excel, and update some registries.
What should be the best approach, install mysql on the VM or use Google Cloud SQL?
Kind Regards
Only you can decide what better fits your needs, but you may consider the following:
Local Mysql Pros:
Faster performances. This could be important if generating a lot of queries you would need a bigger Cloud SQL instance to have similar speed.
Minor costs
Cloud SQL Pros:
High reliability. Data is backed-up without the need of taking snapshots.
Possibility to stop or delete GCE instance and keep database active.
Easier and faster to scale if required
Easily add a read replica.
Less load on the GCE
Sincerely,
Paolo
For better performance you can run your MySQL server on a virtual machine. I have tried that with the same server specifications (1 CPU, 3.75 GB memory) and it runs much better.