In the Persistent Disk Size Limits documentation here it says:
Standard, high memory, and high CPU machine types can attach up to 16 persistent disks.
Is this a limit that can be raised?
We would like to run many docker containers per machine and give each of them a persistent disk (so they aren't tied specifically to that machine). The current limit would only let us run 16 containers per VM, which is a much lower number than we'd like.
On AWS HVM instances, we can attach up to 73 EBS volumes.
Thanks.
No, it presently cannot. I have just been advised that the number of PDs attached to an instance is fixed.
According to Amazon's documentation, it is inadvisable to attach more than 40 disks per instance.
Related
I am currently using Memory optimized DB class in AWS (8 CPUs) because some push notifications in our app cause the CPU utilization to skyrocket, but 99% of the time CPU utilization is at about 10% so 8 CPUs aren't really needed most of the time.
Would I be able to deploy less CPUs on a Burstable Instance and have CPUs adjusted when there are those heavy traffic push notifications?
How do Burstable Instances work?
I wouldn't choose a burstable instance for any production traffic.
The burstable instance accumulates a number of "performance credits" per hour. These credits can be consumed when traffic increases and you needs a significant amount of resources. When the credits are spent, the instance still runs, but has only "baseline performance" which is, frankly, insufficient to handle production traffic.
I've seen many users try to economize by using the T family of instance types. They are usually quite disappointed, because they underestimate their need for resources. They end up consuming their burst credits too quickly, and then operate at the baseline performance level too often.
I'd use a burstable instance only for CI testing servers, or development. These instances typically run idle most of the time, and accumulate a good level of performance credits. They use these credits for brief periods, and then return to an idle level of activity.
You can also look into Aurora Serverless. This is supposed to auto-scale more replica instances in response to traffic increases, which should give you more CPU capacity. You only pay for the instances you use. That's the theory, but I can't speak from experience because I haven't used or tested Aurora Serverless. How well it works and how economical it is for you depends on your application's workload. All I can suggest is to give it a try.
I believe most RDBMS, particularly MySQL, can only be scaled "vertically", in the sense that you can't dynamically add/remove CPU resources to handle bursts of reads/writes.
Perhaps you can create a "notification/fanout" service which is more easily dynamically scaled up and down, using perhaps DynamoDB or AWS SNS. This way your primary database can avoid all of that traffic and in turn you can use a much less expensive EC2 instance for your RDS.
So I'm looking at the disk types available on the google cloud platform and they have Persistent disks and Local SSD disks (from reading their documentation). After dreading the docs I go to create a disk the disk types are labeled as 'SSD Persistent disk' and 'Standard Persistent disk'. They seems to be named different here to in the docs. I just want a confirmation that:
SSD Persistent disk = Local SSD disk
Standard Persistent disk = Persistent disks
I'm new to running VMs on a cloud platform for hosting your own site and im trying to wrap my head around the different options available for all the different settings and why you would choose one over the other..
I'm trying to learn the different disks available and what they do but when I go to actually create a disk the types they have avaliable aren't even the same as the ones they list in the docs. I figured they could be the same but named different for whatever reason so I wanted to come here and confirm if thats the case.
Your choices are:
local SSD = locally-attached to the VM, SSD
standard persistent = network-attached, persistent, HDD**
SSD persistent = network-attached, persistent, SSD
Type 1 is lower latency than types 2 and 3, because type 1 is physically attached to the VM.
Type 2 and 3 persist beyond instance stop/delete. Type 1 does not.
Type 2 and 3 are durable/redundant (Google replicates them, like Raid 1). Type 1 is not.
Type 2 and 3 can be attached to multiple VMs simultaneously (in read mode). Type 1 cannot.
** nowhere does Google actually indicate afaik that standard persistent is actually HDD, just that it is not SSD, so it may not be guaranteed to be HDD.
You can see more specific data at Storage Options, but in summary:
local SSD is the fastest (by far)
SSD persistent has much higher read/write IOPS than standard persistent
SSD persistent is more expensive (4x) than standard persistent
Basically you have 2 different disk types to choose from when setting up a GCE instance:
Persistent disks can either be regular persistent disks, which could be HDD, or SSD's.
Local disks, which are not persistent are always SSD's.
Like you can read in the docs using local SSD's only allow up to 3TB to be stored on the disks while the Persistent disks allow up to 64 TB.
SSD's are also more expensive per GB used. The upside for SSD's however is that they allow higher throughput and lower latency.
There's several other things to note, everything there is to be found about GCE disks can be found in the aforementioned docs.
Answer to your original question:
Persistent disk = Standard Persistent Disk or SSD persistent disk and non-persistent disk = Local SSD
I'm working on a digital asset management deployment and it's likely we'll need need more than the 10TB storage maximum for a persistent disk. The DAM software does not support multiple storage points and likely won't in the future.
How are people coping with this limitation? Is the 10TB max likely to increase as Google compute engine matures?
As of 1 Feb 2016, you can create a persistent disk of up to 64 TB; see the blog post for more details.
According to the official doc:
Instances with shared-core machine types are limited to a maximum of 16 persistent disks.
For custom machine types or predefined machine types that have a minimum of 1 vCPU, you can attach up to 128 persistent disks.
Each persistent disk can be up to 64 TB in size, so there is no need to manage arrays of disks to create large logical volumes. Each instance can attach only a limited amount of total persistent disk space and a limited number of individual persistent disks. Predefined machine types and custom machine types have the same persistent disk limits.
Most instances can have up to 64 TB of total persistent disk space attached. Shared-core machine types are limited to 3 TB of total persistent disk space. Total persistent disk space for an instance includes the size of the boot persistent disk.
I am looking at setting up a MySQL Cluster with two high end dell servers (duel opteron 4386 cpus, 16gb ram and RAID 10 SAS). I also have a handful of other high end machines that are i7 with 20gb+ ram each on average.
What is the hardware focus for each node in MySQL Cluster?
I know that the management node requires very little as it is simply an interface to control the cluster and can be put on the same machine as mysqld.
What hardware is the most important for the MySQL node and for the data nodes (hard disk IO, ram, cpu etc?
You're correct that the management node needs very little resource - just make sure you run it/them on different machines to the data nodes.
Data Nodes like lots of memory as by default all of the data is held in RAM; they can also make good use of fast cores and in more recent releases lots of them (perhaps upto about 48). Multiple spindles for redo/undo logs,checkpoints, disk table files etc. can speed things up.
MySQL Servers don't need a lot of RAM as they don't store the Cluster data and in most cases you wouldn't use the query cache. If replicating to a second Cluster then you should make sure that they have enough disk space for the binary log files.
When I launch an Amazon MySQL database instance in RDS, I choose the amount of Allocated Storage for it.
When I create a snapshot (either manually or with the automatic backup), it says under "Storage" the same size as the size allocated for the instance, even though my database did not reach that size.
Since the pricing (or the free tier) in Amazon is dependent on the amount of storage used, I would like to know the real storage size I'm using, rather than the size allocated by the original database.
From looking at the Account Activity, and from knowing how mysqldump works, I would guess the snapshot does not really include the empty space allocated.
I was interested in the answer to this question and a google search brought me here. I was surprised to see that although there is an accepted, upvoted answer, it does not actually answer the question that was asked.
The question asked is:
How can I tell the raw size of a MySQL DB snapshots in Amazon RDS?
However, the accepted answer is actually the answer to this question:
Am I charged for the allocated size of the source database when I take an RDS snapshot from it.
As to the the original question, AFAICT, there is no API or console function to determine the storage used by an RDS snapshot. The DBSnapshot resource has allocated_storage (ruby, java), but this returns the storage maximum size requested when the database was created. This mirrors the AWS RDS console:
One might have thought this would be broken out on the AWS bill, but it provides very little details. For RDS:
The S3 part of the bill is even less helpful:
Conclusion, there is no way to tell the raw size of a MySQL DB snapshot in Amazon RDS.
RDS is being stored through EBS according to FAQ:
Amazon RDS uses EBS volumes for database and log storage.
EBS doesn't store empty blocks, according to its pricing page:
Because data is compressed before being saved to Amazon S3, and Amazon EBS does not save empty blocks, it is likely that the snapshot size will be considerably less than your volume size.
And takes space only for changed blocks after initial snapshot was made, according to details page:
If you have a device with 100 GB of data but only 5 GB has changed after your last snapshot, a subsequent snapshot consumes only 5 additional GB and you are billed only for the additional 5 GB of snapshot storage, even though both the earlier and later snapshots appear complete.
RDS backups are block level full virtual machine snapshots; no mysqldump at all is involved. Given this fact, each of your snapshots will use exactly the same ammount of storage as your production instance at the moment the backup took place.