I am curious to know the overall effects on performance if i have my Database server separately from Tomcat server(Spin a MySQL server on Amazon). Actually, I am having some performance issues and not sure if it might be the cause of.
Yes, absolutely i have found that separating the DB and application can actually uncover performance issues not evident in a co-location situation, for the network latency reasons mentioned by ck1. In fact, if you capture stack traces during the slow operations, by sampling it will indicate the database/application code sensitive to network latency. The use cases with performance issues ( in non co-located apps) generally make a lot of round trips to the database. Instead try offloading the processing into the DB with a more complex query and reduce the rows returned.
Pros of having database and app servers co-located:
Network latency will be minimized
You only need to maintain a single server
Cons of co-location:
The app and database servers will contend for a common set of CPU, memory, and disk I/O resources. For example, queries causing a spike in CPU usage will affect the app server's performance
You have more than a single server to maintain
It's difficult to scale horizontally
Related
We are facing issue with Django SQL Performance. Even setting up CONN_MAX_AGE setting, Connections are not closing. Following configurations we used,
CONN_MAX_AGE: 5
Is there any place to do the configurations for performance?
Can you suggest me any Solution for multiple application servers with Single Database server architecture?
"Persistent connections" don't close; they stay open for reuse.
Anyway, tuning that will not affect scaling and performance much.
If you have slow queries, high CPU, high I/O, discover what query is the villain; then we can discuss improving it.
I currently have a cloud server with 2 CPU's and 2GB RAM, Ive noticed that MySQL performs SELECT queries quite slowly on a table size greater than 250,000 rows unless the queries are cached. The table has all the correct indexes and my queries are optimized.
I have run the same MySQL setup on my personal computer and the queries are much faster. So I am thinking its more to do with the server itself rather than the MySQL queries or table setup.
I am considering upgrading the cloud server to 4GB Ram and 4CPU's or switching to a dedicated server with the same spec.
Will changing from a cloud server to a dedicated server improve MySQL performance?
Though Cloud computing utilizes the SAN storage, you will be sharing the resources like CPU, RAM and Storage among number of VMs. This may cause the slowness in I/O operations and hence in your MySQL queries.
In dedicated server, all resources will be allocated to your applications only. Thus, switching to a dedicated server will certainly improve the MySQL server performance. Also, when you have database driven application, SSD drives are highly recommended.
2GB is tiny by today's norms. Ar you using InnoDB or MyISAM? What is the cache size? Does it have SSDs?
It sounds like the table is bigger than can be cached, and that the slow query is scanning the entire table. That says that the query and/or indexes are not well optimized. Let's see the SELECT and SHOW CREATE TABLE; more tuning may be possible.
Does the Cloud limit your IOPs? That mighe explain the difference. And it would further indicate that we should study the query.
Given your query takes 20 seconds to run on Cloud environment, we can rule out network latency problems. To find out the bottleneck. Try using "top" while running the large query on cloud.
If CPU utilization is high, change to a high CPU type, maybe higher memory as well (you can see memory utilization with "top" as well).
If the %CPU of idle or wait is high, it basically suggests underlying disk is the bottleneck here. You can verify this by running some "fio" test, or use "iostat" and monitor the device usage for the query. The solution is adding more disk space or use a SSD device on cloud.
I will prefer cloud always. when it come to dedicated server there you can't find elasticity on server usage. choose a good cloud architecture to enhance the application performance. you have set proper auto scaling, devops methods.
You can also use a remote database to connect your application in order to avoid I/O request to application server.
To give you a little background, I currently have a website that allows users to upload photos. The website was initially on a GoDaddy shared server, but recent surges in traffic have forced me to explore other options. During peak hours, the site contains 400+ active visitors, which when combined with user uploads, forces the shared server to shut down.
I have a small amount of experience with setting up servers through AWS and attempted to place the website on a c1.medium instance, Amazon Linux. The website along with the MYSQL Database is on the same instance. While I have read that this is in general frowned upon, I have similarly read that moving the database to another instance would not significantly increase speeds. Unfortunately, the c1.medium instance also was unable to support the traffic and I soon received an error Establishing a Database connection. The site does load on occasion, so the problem stems from the traffic load and not an actual problem with the database.
My question is whether the problem revolves solely around MySQL? The database itself when backed up is around 250MB. Is the issue caused by input / output requests to the database? I read posts with people with similar problems in which they stated that installing MySQL 5.6 solved the problem, but also have read that MySQL 5.6 is slower than MySQL 5.5, which is my current version.
After conducting some preliminary research I started to believe that I could resolve the problem by increasing the IPOS of the EBS. Originally I had it set the IPOS as standard, but changed it to Provisioned IOPS and 30x the size of the EBS (i.e., 60GB – 1800 IOPS). This once again appeared to have little impact. Do I need to upgrade my instance? What measures should I be focused on when deciding on the instance? It appears that the cheapest instance with high network performance and EBS optimized would be c3.xlarge. Suggestions?
Several things to consider:
1)Separate the database server from the web server
Your database should not share resources with your web server. They will both perform poorly as the result.
It is easier to find what the bottle-neck is.
2) Upgrade to MySQL 5.6
In all the benchmarks that I have seen and done 5.6 performs better than 5.5
3) Configure your database to take advantage of your resources
Depending on the storage engine and the memory allocated in your machine configure MySQL for example set innodb_buffer_pool_size to 70% of the (DEDICATED) RAM
4) Monitor MySQL and check slow query log
Slow query log shows the queries that are slow and inefficient
5) Learn to use EXPLAIN
EXPLAIN shows query plan in MySQL run EXPLAIN on slow queries to tune them
6) Use Key-Value Stores to Cache queries
Using Memcached or Redis cache queries so they don't hit your database and return repeated queries from the memory
7) Increasing IOPS and Scaling Out
Increasing IOPS and getting better hardware helps but using efficient queries is much more effective. Queries and application most of the time are a greater contributing factor to performance problems
8) Replication
To help with concurrency consider moving to a MySQL Master/Slave replication , if you still had issues.
Final Note: use EBS because the storage on EC2 is ephemeral and will not persistent.
We recently did extensive research on the performance bottlenecks associated with massive end-user peaks across our global customer base, and the analysis actually indicates the database as - by far - the most frequent cause of slowdowns or even crashes. The report (https://queue-it.com/trend-report) includes best practice advice from our customers on how to improve the situation, which you may find helpful.
I want to import data from a MySQL server into Oracle database, and I found a suggestion to use Oracle database link. The Oracle instance is 10.0.2.1, and the MySQL server instance should be 5.1. The connection between two servers and the hard-disk should not be bottle neck.
I want to ask about the performance of Oracle database link? How fast it is? Is it very slow, slow or fast? Is it capable of transferring 1000 rows/second?
Thank you
1000 rows/sec is definitely acheivable... the question is whether it's acheivable on your database/network infrastructure.
Even if we had a detailed knowledge of your infrastructure it would still be very hard to say... it depends on so many factors like network speed, network latency, the size of the database rows being transfered etc.
The only way to tell for sure is to test it.
I would look on this as a good thing - the process of building the test is bound to teach you a lot about how it could work... it will throw up a number of issues that you're going to have to consider at some point - how do you handle backlogs when they form? What is the max through-put you can acheive? etc. You'll learn what kind of data-transfer works best for you (e.g. single rows at a time or larger batches) You might want to try it with a mechanisms other than SQL (e.g. queues)
You say that you don't think the network / hard disk access will be an issue - again, you need to test this assumption. Every database has a limiting factor on the performance somewhere (or they'd be infinitely fast!) and it's quite often disk access that is the limiting factor. In this case I would speculate that the network may be the limiting factor, but there's no way to know for sure without measuring it.
Generally speaking dblink performance limited by network speed, but there are some pitfalls, leading to performance issues:
unnecessary joins between local and remote tables that leads to transferring large amounts of data;
lack of parallelism built into the query (unions help in this case);
implicit sorting on remote database side;
failure to comply with Oracle recommendations such as using of collocated views and hints (mainly DRIVING_SITE and NO_MERGE).
Did you try amazon-rds? How is it, performance-wise?
I think this is a hard question to answer as it is highly specific to the problem you are trying to solve, but I will try to give you a picture of what we have seen.
We have been benchmarking RDS using CloudWatch metric gathering tools (provided here: http://aws.amazon.com/articles/2934) and have found it does perform nearly as well as our production servers for our data set. We tested both with a single RDS instance and with a Multi-AZ setup (what we plan to use in production) with no back-up retention.
The load we have been able to throw at it so far we are able to get up into the 1000-1100 Write IOPS range (their metric) even on a small database instance (db.m1.small). At least for our load, increasing the instance class did not affect our throughput IOPS or Bytes. We saw about a 10% reduction in performance when
Amazon freely admitted up front that the solution to really scale out is to subdivide your problem such that you can scale/store it across multiple database servers. We in fact have this in our application (very similar to sharding) and therefore will be able to take advantage and very easily move past this IOPS measurement.
We've found RDS to be pretty comparable performance-wise to having our own production servers (either dedicated or virtual or EC2). Note that you will always suffer some IO/performance degradation using a virtualization solution, which is what RDS seems to be using, and this will show up under heavy load (but with heavy load, you should be having a dedicated MySQL/DB box anyway.)
Take note: the biggest performance you will likely see is the network latency - if you are reading/writing from an EC2 box to an RDS box and vice versa, the network latency will probably be the bottlebeck, particularly for a large number of queries. This is likely to be worse if you are connecting from a non-Amazon/non-EC2 box to RDS.
You will probably get more performance from an equivalent spec physical box than a virtual box, but this is true of dedicated vs EC2/RDS, and is not a RDS-specific problem.
Regarding RDS vs EC2, the defaults that Amazon has set up RDS with seem to be pretty good, so if you are simply looking to have database server(s) up and running and connect to it, RDS is more than suitable. Do make sure you have the cost correctly analyzed though - its not the same pricing model as, say, an EC2 instance.