I need to execute a bash script/command within a Google compute instance using Google Functions and get a response.
AWS has an agent called SSM that let me do that with no hassle using Lambda, nevertheless I did not find anything like that on Google Cloud.
On AWS using a nodejs lambda I use the following example:
ssm.sendCommand({
DocumentName: documentName,
InstanceIds: [ instanceId ],
TimeoutSeconds: 3600,
Parameters: {
'commands' : commands
}
}
How can I achieve what I want on Google Cloud? Thank you.
The nature of Google Cloud Functions is that it is the most abstract of the Serverless paradigms. It assumes that all you are providing is stateless logic to be executed in one of the supported languages. You aren't supposed to know nor care how that logic is executed. Running bash (for example) makes an assumption that your Cloud Function is running in a Unix like environment where bash is available. While this is likely to be a true statement, it isn't part of the "core" contract that you have with the Cloud Function environment.
An alternative solution I would suggest is to study Google's Cloud Run concept. Just like Cloud Functions, Cloud Run is serverless (you don't provision any servers and it scales to zero) but the distinction is that what is executed is a docker container. Within that container, your "code" within is executed in the container environment when called. Google spins up / spins down these containers as needed to satisfy your incoming load. Since it is running in a container, you have 100% control over what your logic does ... including running bash commands and providing any scripts or other environment needed to be able to run.
This is not possible. You can perform some actions such as start a VM or stop it from a Cloud Function, but you can't get or list the dirs within a VM. In this case, the Compute Engine API is being used, but only the is reached.
The workaround would be to create a request handler in your VM in order to could be rached by the CF. The proper security would be implemented in order to avoid security issues and requests from anonymous callers. You might use the public IP to reach your VM.
You can run a Bash script on Google Cloud Run with a Bash Docker image. As an example - https://github.com/sethvargo/cloud-run-bash-example
Then you can call this Cloud Run service from your other existing Google Cloud Functions, AppEngine etc.
Related
I use the following command in my Compute Engine to run a script that's stored in Cloud Storage:
gsutil cat gs://project/folder/script.sh | sh
I want to create a function that runs this command and eventually schedule to run this function, but I don't know how to do this. Does anyone know how to do this?
Cloud Functions is serverless and you can't manage the runtime environment. You don't know what is installed on the runtime environment of the Cloud Functions and your can't assume that GCLOUD exists.
The solution is to use cloud Run. the behavior is very close to Cloud Functions, simply wrap your function on a webserver (I wrote my first article on that) and, in your container, install what you want, especially GCLOUD SDK (you can also use a base image with GCLOUD SDK already installed). And this time you will be able to call system binaries, because you know that they exist because you installed them!
Anyway, be careful in your script execution: the container is immutable, you can't change file, the binaries, the stored files,... I don't know the content of your script but your aren't on a VM, you are still on serverless environment, with ephemeral runtime.
I am looking to run a script once during VM instantiation. The startup script in the compute engine template runs every time the VM is started. Say for e.g. I have to install gnome desktop on linux host, I don't want to include that in startup script. Rather I am looking for something that runs once whet he host is created. Of course, I want this automated. Is it possible to do this?
Edit: I am trying to achieve this in Linux OS.
As the documentation [1], if we create startup scripts on a compute engine instance then the instances perform automated tasks “every time” the instance boots up.
To run startup script once, the most basic way is to use a file on the filesystem to flag when the script has been run or you could use the instance metadata to store the state.
For example via:
INSTANCE_STATE=$(curl http://metadata.google.internal/computeMetadata/v1/instance/attributes/state -H "Metadata-Flavor: Google")
Then set state = PROVISIONED after running the script etc.
But it is a good idea to have your script check specifically if the actions it is going to do have already been done and handled accordingly.
Another option, in your startup script you can have it removed the startup-script metadata at the end from the host instance
[1] https://cloud.google.com/compute/docs/startupscript
[2] https://cloud.google.com/compute/docs/storing-retrieving-metadata
I want to write a Google Cloud Function that can interact with GCP's Dataproc service to programatically launch Dataproc clusters. We already have a battle-hardened Dataproc infrastructure, we're just looking to extend the ways in which they get launched.
Our Dataproc clusters can only be launched using an appropriate IAM service account that is already a member of the appropriate IAM roles hence the Cloud Function will need to authenticate to the Dataproc service using that service account. What is the most appropriate way for a Cloud Function to authenticate to other GCP services/APIs using a service account?
Options I suspect include:
* running the function as that service account
* providing a JSON key file & setting GOOGLE_APPLICATION_CREDENTIALS environment variable
Is there a recognised way of achieving this?
I have had a look at :
* https://cloud.google.com/docs/authentication/
* https://cloud.google.com/docs/authentication/getting-started
but they are not specific to Cloud Functions.
I've also looked at
* https://cloud.google.com/functions/docs/writing/http
but that seems more concerned with how the caller of the function can authenticate.
I think this is what you're looking for: https://cloud.google.com/functions/docs/concepts/iam
At runtime, Cloud Functions defaults to using the App Engine default service account (PROJECT_ID#appspot.gserviceaccount.com), which has the Editor role on the project. You can change the roles of this service account to limit or extend the permissions for your running functions. You can also change which service account is used by providing a non-default service account on a per-function basis.
tl;dr gcloud functions deploy FUNCTION_NAME --service-account SERVICE_ACCOUNT_EMAIL
By the way, if you ever need more complex scheduling logic, consider looking into Cloud Composer (managed Apache Airflow): https://cloud.google.com/composer/
I came across the article : Bringing the best of serverless to you
where I came to know about upcoming product called Serverless containers on Cloud Functions which is currently in Alpha.
As described in the article:
Today, we’re also introducing serverless containers, which allow you
to run container-based workloads in a fully managed environment and
still only pay for what you use.
and in GCP solutions page
Serverless containers on Cloud Functions enables you to run your own containerized workloads on
GCP with all the benefits of serverless. And you will still pay only
for what you use. If you are interested in learning more about
serverless containers, please sign up for the alpha.
So my question is how this serverless containers different from app engine flexible with custom runtime, which also use a docker file?
And it's my suspicion, since mentioned named is Serverless containers on Cloud Functions, the differentiation may include role of cloud functions. If so what is the role played by cloud functions in the serverless containers?
Please clarify.
What are Cloud Funtions?
From the official documentation:
Google Cloud Functions is a serverless execution environment for building and connecting cloud services. With Cloud Functions you write simple, single-purpose functions that are attached to events emitted from your cloud infrastructure and services. Your function is triggered when an event being watched is fired. Your code executes in a fully managed environment. There is no need to provision any infrastructure or worry about managing any servers.
In simple words, the Cloud Function is triggered by some event (HTTP request, PubSub message, Cloud Storage file insert...), runs the code of the function, returns a result and then the function dies.
Currently there are available four runtime environments:
Node.js 6
Node.js 8 (Beta)
Python (Beta)
Go (Beta)
With the Serverless containers on Cloud Functions product it is intended that you can provide your own custom runtime environment with a Docker Image. But the life cycle of the Cloud Function will be the same:
It is triggered > Runs > Outputs Result > Dies
App Engine Flex applications
Applications running in the App Engine flexible environment are deployed to virtual machines, i.e Google Cloud Compute Engine instances. You can choose the type of machine you want use and the resources (CPU, RAM, disk space). The App Engine flexible environment automatically scales your app up and down while balancing the load.
As well as in the case of the Cloud Functions there runtimes provided by Google but if you would like to use an alternative implementation of Python, Java, Node.js, Go, Ruby, PHP, .NET you can use Custom Runtimes. Or even you can work with another language like C++, Dart..., you just need to provide a Docker Image for your Application.
What are differences between Cloud Functions and App Engine Flex apps?
The main difference between them are its life cycle and the use case.
As commented above a Cloud Function has a defined life cycle and it dies when it task concludes. They should be used to do 1 thing and do it well.
On the other hand an Application running on the GAE Flex environment will always have at least 1 instance running. The typical case for this applications are to serve several endpoints where users can do REST API calls. But they provide more flexibility as you have full control over the Docker Image provided. You can do "almost" whatever you want there.
What is a Serverless Container?
As stated on the official blog post (search for Serverless Containerss), it's basically a Cloud Function running inside a custom environment defined by the Dockerfile.
It is stated on the official blog post:
With serverless containers, we are providing the same underlying
infrastructure that powers Cloud Functions, but you’ll be able to
simply provide a Docker image as input.
So, instead of deploying your code on the CF, you could also just deploy the Docker image with the runtime and the code to execute.
What's the difference between this Cloud Functions with custom runtimes vs App Engine Flexible?
There are 5 basic differences:
Network: On GAE Flexible you can customize the network the instances run. This let's you add firewalls rules to restrict egress and ingress traffic, block specific ports or specify the SSL you wish to run.
Time-Out: Cloud Functions can run for a maximum of 9 minutes, Flexible on the other hand, can run indefinitely.
Ready only environment: Cloud Functions environment is read-only while Flexible could be written (this is only intended to store spontaneous information as once the Flexible instance is restarted or terminated, all the stored data is lost).
Cold Boot: Cloud Functions are fast to deploy and fast to start compared to Flexible. This is because Flexible runs inside a VM, thus this extra time is taken in order for the VM to start.
How they work: Cloud functions are event driven (ex: upload of photo to cloud storage executing a function) on the other hand flexible is request driven.(ex: handling a request coming from a browser)
As you can see, been able to deploy a small amount of code without having to take care of all the things listed above is a feature.
Also, take into account that Serverless Containers are still in Alpha, thus, many things could change in the future and there is still not a lot of documentation explaining in-depth it's behavior.
I want to run the same Python script with different parameters on several instances on google compute engine. Currently I setup all my instances by creating an instance group. Then I ssh into each machine and start the Python script with the correct parameters.
I'm able to automate setup that's common for all the VMs, such as mounting buckets and so on, by using startup scripts. But I still have to ssh into each VM and start the Python script with a different parameter for each VM. Hence, I'm wondering if there's some clever and simple way of running the same Python script with different parameters on each instance.
Any suggestions? Thank you!
One solution is to use metadata: Create your instances separately instead of with an instance group. Make them identical (ie use the same script) except for metadata - use the metadata to give each instance it's unique parameters. In your script, fetch the metadata to determine how to proceed separately.