Python code in my Cloud Function (2nd generation) runs for 45minutes. Cloud Scheduler job triggers this CF. Code completes successfully but Cloud Scheduler job fails as the runtime is beyond 30min. Any solution or workaround for this error?
I tried using eventarc trigger for cloud function but the timeout is 9 minutes and the code stops running after 9 minutes without throwing any error on the Cloud Scheduler job.
HTTPs trigger works well for my code but the scheduler job throws an error in the end.
All the serverless products, including Cloud Task (with HTTP, not App Engine endpoint) have a maximum of 30 minutes time out by waiting a HTTP response. It should be something by default on the Google (Cloud?) infrastructure.
So, you can't use ONLY Cloud Scheduler. My preferred pattern for that is to use Cloud Run Jobs + Cloud Workflows trigger by Cloud Scheduler.
The scheduler trigger the Workflows (the correct invocation of Cloud Workflows ACKs the scheduler)
The workflows start the Cloud Run Jobs async
The workflows poll regularly the Cloud Run jobs status
Handle the jobs status in the workflows (retry on error, alerts, do nothing,...)
Or, you can simply ignore the Cloud Scheduler timeout and rely on monitoring/logging to track the errors
Related
I have a function deployed in GCF which is executed in node js. And it works with Cloud SQL instance of MySQL.
And I receive this warning from time to time.
CloudSQL warning: your action is needed to update your function and avoid potential disruptions. Please see https://cloud.google.com/functions/docs/sql#troubleshooting for additional details: ensure that the account has access to "-mysql-dev7" (and make sure there's no typo in that name). Error during createEphemeral for -mysql-dev7: googleapi: Error 403: Cloud SQL Admin API has not been used in project 319049 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=319049* then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured
And after this warning function continues to execute. What does it mean? Why function is not halted. If it is not important what does it mean and what if I don't enable this admin API?
BTW function only does selects and updates, it doesn't change table structure so theoretically why would it need Admin API?
I understand the confusion that warning messages may bring, especially if they have no noticeable effects on your function’s workflow.
To the best of my knowledge, the warning identifies that through not enabling this API, there will be potential for further issues when in need of performing certain tasks which require the Cloud SQL Admin API. Potential uses for the Cloud SQL Admin API can be found under public documentation. If you would ever need to list the Cloud SQL instances in a project, get the database of a particular instance, create a new Cloud SQL instance, or perform any other operation listed, please enable the API.
To avoid seeing these warnings all together, I would recommend enabling the API.
EDIT:
Cloud Functions utilizes a mechanism to connect to Cloud SQL instances via the Cloud SQL proxy, the Cloud SQL proxy in turn utilizes the Cloud SQL Admin API.
Therefore when connecting to a Cloud SQL instance without the Cloud SQL Admin API enabled, the error message is thrown.
Cloud Functions is still able to connect to the Cloud SQL instance as it utilizes a legacy connection mechanism when it is unable to do so via the Cloud SQL proxy.
This is why the function does not halt and can still perform the operations on the instance as expected.
Enabling Cloud SQL Admin API will solve the warning:
I have problem with Cloud SQL billing.
My Cloud SQL has used all 720 Hours running machine (db-g1-small : changed from db-n1-standard-1 recently)
I've found accordding to Cloud SQL Documentation that
For Second Generation instances, the activation policy is used only to start or stop the instance.
So without ON_DEMAND policy of the First Generation, how can I reduce these costs on my Cloud SQL instance?
PS. Look like my cloud server not automatically down because it stay 4 sleep connections
Indeed for second generation instances of Cloud SQL, the only activation policies available are ALWAYS and NEVER, so it's not possible anymore to leave that kind of instance handling entirely on Cloud SQL's hands.
However you can create a workaround for this by executing a cron job that turns the instances on/off on a fixed schedule. Eg: you can run a cron job that runs on friday night to shutdown the instance and on monday morning to shut it back on.
You can use the following command to do so:
gcloud sql instances patch [INSTANCE_NAME] --activation-policy [ACTIVATION_POLICY_VALUE]
Moreover, you can create a feature request on Google Cloud's Public Issue Tracker System to re-include that functionality on Cloud SQL in the future, but there are no guaratees that this will happen.
I am new to Google Cloud Composer and following the QuickStart instruction, Create the Environment, Load Dag, Check Airflow, and Delete the Environment.
But in (real life) production use case, after we finish load dag files and run them in the environment. Should we delete the Google Cloud Composer Environment? Because there might be several compute instances in that composer and doing nothing now. It is expensive.
But if I delete the environment, then I would lose the access to its airflow web portal, and I could not check the processing logs of my processing on the deleted environment.
So what should I do? In real life production case, should I delete or not delete the environment after the processing is done?
Apache Airflow (and therefore Cloud Composer) is for orchestrating workflows, not for ETL batch jobs that only require transient compute resources. Similarly to how you wouldn't turn a server off just because a scheduled cron task isn't running, Composer environments are meant to be long-running compute resources that are always online, such that you can schedule repeating workflows whenever necessary (whether that be per second, daily, etc)
In a real production case, a Composer environment should always be left running, or no DAGs will be scheduled when it is down. If you have a development environment and wish to save money, then you can resize the Composer environment's attached GKE cluster to 0 nodes so you won't be billed for them. Similarly, if you don't think you're running enough DAGs to justify the cost, consider smaller worker machine sizes.
I'm creating a managed instance group with autoscaling in GCE. When a lot of work is queued up new instances will be created which start doing work.
Let's say each chunk of work takes 10 minutes, could it happen that GCE decides to shut down an instance that still has work in progress?
Autoscaler will immediately terminate instance if the health check condition meets.
However, you can use a shutdown script to control the termination. A shutdown script will run, on a best-effort basis, in the brief period between when the termination request is made and when the instance is actually terminated. During this period, Compute Engine will attempt to run your shutdown script to perform any tasks you provide in the script. You can read more about the autoscaler decision in this document. You can read about using shutdown script and its limitation at this link.
Also if these instances are offering backend service then it is good to enable connection draining. You can enable connection draining on backend services to ensure minimal interruption to your users when an instance is deleted automatically by an autoscaler or manually removed from an instance group. You can find more at this link about enabling connection draining.
I have a perl job that runs for 5 mins at the top of every hour. What is the most cost effective way of running this job on the Google Cloud infrastructure? Running a compute engine VM seems too heavy-weight for this since I'd get charged for the other 55 mins of no use. I don't understand the "Managed VMs" well enough, but it seems like this might be an option, but I'm not sure if pricing is rounded to the hour. Does anyone have any ideas what the best option is so that I only get charged for 120 mins of usage (24 times run * 5 minutes). The script also uses some image processing binaries, so converting to Python won't do the trick.
Managed VMs are linked to Google App Engine. If you have an App in GAE, managed VMs are used to configure the hosting environment for you App using VMs that run on Google Compute Engine and these applications are subject to Java and Python run time. This link can give you an idea on pricing on GAE, however Perl is not a supported language in GAE.
On GCE, you can start up an instance, do the task and then delete the instance without deleting the persistence disk, this will allow you to recreate the instance using this disk, however you will still be charged for the provisioned disk space and you will need to create a script that will spin up the instance and delete it. You can also create a snapshot of your disk and recreate your instance based on the snapshot, this will be little bit less expensive that keeping the disk.
Also, you should look at the type of persistence disks (PD) on GCE, at this link, take a look at the examples provided, since based on your operation, regular PD or SSD PD can make a big difference on price.
You can use the pricing calculator to estimate your charges
When you deploy to App Engine using a managed VM, an compute engine instance (managed by google) is created for you. All request to App Engine will be forwarded to the created compute engine instance.
To run your script in App Engine as a Managed VM, you will have to dockerize your project, as the managed VM runs a docker container.
I don't see a reason to use App Engine managed VM (just for running a script), as the cost will be same as using a compute engine instance.
Probably the most cost effective way is to create a script that:
Launches a compute engine instance
Install perl
Copies your script to the instance
Runs you script in the created instance
To schedule the execution, you can put at home/office a cron job that executes the above script.