I had my app in the app engine(Flex). But it's costing a lot with no traffic yet!
I decided to move that to Kubernetes Engine which utilizes the compute engine.
Another reason I moved to Kubernetes because I wanted to run docker container services like Memcached that come with an added cost in App Engine Flex.
If you are tempted to ask why am not using App Engine Standard which is economical, that's because I couldn't find any easy way if at all there's any for running services like GDAL & Memcached.
I thought Kubernetes should be a cheaper option, but what I am seeing is the opposite.
I have even had to change the machine type to g1-small from N1...
Am I missing something?
Any ideas on how to reduce cost in Kubernetes / compute engine instances?
Please have a look at the documentation GKE Pricing and App Engine Pricing:
GKE clusters accrue a management fee of $0.10 per cluster per hour,
irrespective of cluster size or topology. One zonal (single-zone or
multi-zonal) cluster per billing account is free.
GKE uses Compute Engine instances for worker nodes in the cluster. You
are billed for each of those instances according to Compute Engine's
pricing, until the nodes are deleted. Compute Engine resources are
billed on a per-second basis with a one-minute minimum usage cost.
and
Apps running in the flexible environment are deployed to virtual
machine types that you specify. These virtual machine resources are
billed on a per-second basis with a 1 minute minimum usage cost.
Billing for the memory resource includes the memory your app uses plus
the memory that the runtime itself needs to run your app. This means
your memory usage and costs can be higher than the maximum memory you
request for your app.
So, both GAE Flex and GKE cluster are "billed on a per-second basis with a 1 minute minimum usage cost".
To estimate usage cost in advance you can use Google Cloud Pricing Calculator, also you can use it to estimate how changing parameters of your cluster can help you to reduce cost and which solution is more cost effective.
In addition, please have a look at the documentation Best practices for running cost-optimized Kubernetes applications on GKE.
Related
I wanted to create a Always Free Eligible VM Instance (VM.Standard.E2.1.Micro) on the Oracle Cloud, but it's not on my list.
And when I check my limit for VM.Standard.E2.1.Micro in
"Governance > Limits, Quotas and Usage", it say 0.
How can I create one? My Home Region is Canada Southeast (Montreal), ca-montreal-1.
My account's trial is not over yet. Should I wait till my trial is over to create it?
As per the Always Free website, at any time you can have up to the following:
Two Oracle Autonomous Databases with powerful tools like Oracle Application Express (APEX) and Oracle SQL Developer
Two Oracle Cloud Infrastructure Compute VMs; Block, Object, and Archive Storage; Load Balancer and data egress; Monitoring and Notifications
If you already are at capacity for this, then you would not be able to add an additional. Further details of Always Free resources can be found here - https://docs.oracle.com/en-us/iaas/Content/FreeTier/resourceref.htm
The always free provide you with the following
2 Compute virtual machines with 1/8 OCPU and 1 GB memory each.
2 Block Volumes Storage
100 GB total.
10 GB Object Storage.
10 GB Archive Storage.
Resource Manager: managed Terraform.
Focus on the specs of the free one
VM.Standard.E2.1.Micro is not available for ca-montreal-1 at this time (January 2021).
I created a new account in the Ashburn region where VM.Standard.E2.1.Micro is available.
I'm starting a project where a CloudSQL instance would be a great fit however I've noticed they are twice the price for the same specification VM on GCP.
I've been told by several devops guys I work with that they are billed by usage only. Which would be perfect for me. However on their pricing page it states "Instance pricing for MySQL is charged for every second that the instance is running".
https://cloud.google.com/sql/pricing#2nd-gen-pricing
I also see several people around the web saying they are usage only.
Cloud SQL or VM Instance to host MySQL Database
Am I interpreting Googles pricing pages incorrectly?
Am I going to be billed for the instance being on or for its usage?
Billed by usage
All depend what you mean by USAGE. When you run a Cloud SQL instance, it's like a server (compute engine). Until you stop it, you will pay for it. It's not a pay-per-request pricing, as you can have with BigQuery.
With Cloud SQL, you will also pay the storage that you use. And the storage can grow automatically according with the usage. Be careful the storage can't be reduce!! even if you delete data in database!
Price is twice a similar Compute engine
True! A compute engine standard1-n1 is about $20 per month and a same config on Cloud SQL is about $45.
BUT, what about the price of the management of your own SQL instance?
You have to update/patch the OS
You have to update/patch the DB engine (MySQL or Postgres)
You have to manage the security/network access
You have to perform snapshots, ensure that the restoration works
You have to ensure the High Availability (people on call in case of server issue)
You have to tune the Database parameters
You have to watch to your storage and to increase it in case of needs
You have to set up manually your replicas
Is it worth twice the price? For me, yes. All depends of your skills and your opinion.
There are a lot of hidden configuration options that when modified can quickly halve your costs per option.
Practically speaking, GCP's SQL product only works by running 24/7, there is no time-based 'by usage' option, short of you manually stopping and restarting the compute engine.
There are a lot of tricks you can follow to lower costs, you can read many of them here: https://medium.com/#the-bumbling-developer/can-you-use-google-cloud-platform-gcp-cheaply-and-safely-86284e04b332
We are using a private GCP account and we would like to process 30 GB of data and do NLP processing using SpaCy. We wanted to use more workers and we decided to start with a maxiumn number of worker of 80 as show below. We submited our job and we got some issue with some of the GCP standard user quotas:
QUOTA_EXCEEDED: Quota 'IN_USE_ADDRESSES' exceeded. Limit: 8.0 in region XXX
So I decided to request some new quotas of 50 for IN_USE_ADDRESSES in some region (it took me few iteration to find a region who could accept this request). We submited a new jobs and we got new quotas issues:
QUOTA_EXCEEDED: Quota 'CPUS' exceeded. Limit: 24.0 in region XXX
QUOTA_EXCEEDED: Quota 'CPUS_ALL_REGIONS' exceeded. Limit: 32.0 globally
My questions is if I want to use 50 workers for example in one region, which quotas do I need to changed ? The doc https://cloud.google.com/dataflow/quotas doesn't seems to be up to date since they only said " To use 10 Compute Engine instances, you'll need 10 in-use IP addresses.". As you can see above this is not enought and other quotas need to be changed as well. Is there some doc, blog or other post where this is documented and explained ? Just for one region there are 49 Compute Engine quotas that can be changed!
I would suggest that you start using Private IP's instead of Public IP addresses. This would help in you in 2 ways:-
You can bypass some of the IP address related quotas as they are related to Public IP addresses.
Reduce costs significantly by eliminating network egress costs as the VM's would not be communicating with each other over public internet. You can find more details in this excellent article [1]
To start using the private IP's please follow the instructions as mentioned here [2]
Apart from this you would need to take care of the following quota's
CPUs
You can increase the quota for a given region by setting the CPUs quota under Compute Engine appropriately.
Persistent Disk
By default each VM needs a storage of 250 GB therefore for 100 instances it would be around 25TB. Please check the disk size of the workers that you are using and set the Persistent Disk quota under Compute Instances appropriately.
The default disk size is 25 GB for Cloud Dataflow Shuffle batch pipelines.
Managed Instance Groups
You would need to take that you have enough quota in the region as Dataflow needs the following quota:-
One Instance Group per Cloud Dataflow job
One Managed Instance Group per Cloud Dataflow job
One Instance Template per Cloud Dataflow job
Once you review these quotas you should be all set for running the job.
1 - https://medium.com/#harshithdwivedi/how-disabling-external-ips-helped-us-cut-down-over-80-of-our-cloud-dataflow-costs-259d25aebe74
2 - https://cloud.google.com/dataflow/docs/guides/specifying-networks
I have been trying to determine what instances I should choose, for Compute engine and Cloud SQL, for when we lunch our product.
Initially I'm working on handling max 500 users per day, with peak traffic likely to occur during the evenings. The users are expected to stay on the site with constant interactions for a lengthily period of time (10min+).
So far my guess's lead me to the following:
Compute engine:
n1-standard-2 ->
2 virtuals cpu's, 3.75GB memory
Cloud SQL:
D2 ->
1GB ram, max 250 concurrent users
Am I in the right ball park, or can I use smaller/larger instances?
I'd say to use appropriate performance testing tools to simulate the traffic that will be hitting your server and estimate the amount of the resources you will require to handle the requests.
For Compute Engine VM instance, you can go with a lighter machine type and take advantage of the GCE Autoscaler to automatically add more resources to your front-end when the traffic goes high.
I recommend watching this video.
There are only 7 performance tiers in GCS (D0, D1, D2, D4, D8, D16, D32), RAM maxes out at 16GB (D32) as they are based on Google Compute Engine(GCE) machine types. See screenshot below (1)
By comparison, Amazon has 13 performance tiers with db.r3.8xlarge's RAM maxes out at 244GB. (2)
So my question is, what is the rough equivalent performance tier in AWS RDS for MySQL for a Google Cloud SQL's D32 tier?
Disclaimer: I am new to Google Cloud SQL. I only start to use Cloud SQL because I started a new job that's 100% Google Cloud. Previously I have been a AWS user since the early days.
The D0-D32 Cloud SQL tiers are not based on GCE VMs so a direct comparison is not straightforward. Note that the storage for D0-D32 is replicated geographically and that makes writes a lot slower. The ASYNC mode improves the performance for small commits. The upside is that the instances can be relocated quickly between location that are farther apart.
The connectivity for Cloud SQL is also different from RDS. RDS can be access using IPs and the latency is comparable with VMs talking over local IPs. Cloud SQL uses only external IPs. That makes the latency from GCE higher (~1.25ms) but it provides a slightly better for experience for connections coming from outside the Google Cloud because the TCP connections are terminated closer to the clients.
That being said, from a memory point of view, the db.m3.xlarge from RDS is the closest match for the D32 from Cloud SQL. If the working set fits in the memory the performance for some queries will be similar.
Currently in Alpha, there is a new feature of Cloud SQL that uses comparable performance to GCE machine types.
A Google Cloud SQL Performance Class instance is a new Google Cloud
SQL instance type that provides better performance and more storage
capacity, and is optimized for heavy workloads running on Google
Compute Engine. The new instance type is an always-on, long-lived
database as compared to a Standard Class instance type. Performance
Class instances are based on tiers that correspond to Google Compute
Engine (GCE) standard or highmem machine types.
Link: https://cloud.google.com/sql/docs/getting-started-performance-class
Anyway, very good question, comparing prices with AWS, I found out that there is a huge difference in resources for the smallest instances and the same price:
GCE, D0 = $0.025 per hour (0,128 GB RAM + "an appropriate amount of CPU")
AWS, db.t2.micro = $0.02 per hour (1 GB RAM + 1 vCPU)
For 1 GB RAM in GCE, one would have to pay $0.19 per hour. Unfortunately Google does not specify anything about SSD storage, something very important for performance comparison.