Is there a way to get/subscribe the/for running instances list in GCE autoscaled group.
Via gcloud tool, we can periodically call for the list, but I would like to subscribe for the list update.
I doubt that there is such API implementation for now in GCE(except project metadata), but I need to have such functionality in my application so I could write a logic on that.
Maybe someone has experience with a similar case or know any "hack" for this?
To the best of my knowledge there is no method to subscribe to a list of instances in an managed instance group.
You will need to poll the managed instance group manually to determine the list of current instances.
gcloud compute instance-groups managed list <NAME>
This is a task that could be done very easily in Cloud Functions. At fixed intervals scan the group and email you the list in Json for example. The possibilities are endless.
You can build this easily using (1) a pubsub topic "instance-group-changes" and (2) pushing events to this in your startup & shutdown scripts.
(1) Create the "instance-group-changes" topic
gcloud init
gcloud pubsub topics create instance-group-changes
(2) Modify the startup script for the instance group to send an addInstance event
note: be sure to add the "cloud pubsub api access scope" in the instance template
Use the meta-data service to obtain the instance-id, hostname, etc.
TOPIC=instance-group-changes
instance_id=`curl -s http://metadata.google.internal/0.1/meta-data/instance-id`
gcloud pubsub topics publish "$TOPIC" \
--attribute 'event=addInstance' \
--message "instance_id=$instance_id"
(3) Modify the shutdown script to send a removeInstance event
TOPIC=instance-group-changes
instance_id=`curl -s http://metadata.google.internal/0.1/meta-data/instance-id`
gcloud pubsub topics publish "$TOPIC" \
--attribute 'event=removeInstance' \
--message "instance_id=$instance_id"
Testing
Create the subscription
gcloud pubsub subscriptions create sub-instance-group-changes --topic=instance-group-changes
Pull from the subscription
gcloud pubsub subscriptions pull --limit 5 sub-instance-group-changes
┌─────────────────────────────────┬─────────────────┬──────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ DATA │ MESSAGE_ID │ ATTRIBUTES │ ACK_ID │
├─────────────────────────────────┼─────────────────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│
│ instance_id=5396233750823583338 │ 407816607936940 │ event=addInstance │ XkASTD4HRElTK0MLKlgRTgQhIT4wPkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUaC1MTUVx1Hk4Qb1gzdQdRDRlze2hxO1kaAFMTUHRdURsfWVx-SgNRChFze2d1bVMQBwtBU1b55f_L9q0zZhs9XBJLLD5-NTJFQQ │
│ instance_id=5396233750823583338 │ 407816742842477 │ event=removeInstance │ XkASTD4HRElTK0MLKlgRTgQhIT4wPkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUaC1MTUVx1Hk4Qb1gzdQdRDRlze2hxO1kaAFMTUHRcURsfWVx-SgNRChFze2ZxaFIXAwZCVFb55f_L9q0zZhs9XBJLLD5-NTJFQQ │
Related
I am wondering if there exists an execution id into Cloud Run as the one into Google Cloud Functions?
An ID that identifies each invocation separately, it's very useful to use the "Show matching entries" in Cloud Logging to get all logs related to an execution.
I understand the execution process is different, Cloud Run allows concurrency, but is there a workaround to assign each log to a certain execution?
My final need is to group at the same line the request and the response. Because, as for now, I am printing them separately and if a few requests arrive at the same time, I can't see what response corresponds to what request...
Thank you for your attention!
Open Telemetry looks like a great solution, but the learning and manipulation time isn't negligible,
I'm going with a custom id created in before_request, stored in Flask g and called at every print().
#app.before_request
def before_request_func():
execution_id = uuid.uuid4()
g.execution_id = execution_id
We are currently trying to debug an issue with a pod and figured out that 6 other pod (not related) was turned off and would want to figure out when that happens and who or what turned it off (to see if it's related or not with the first issue).
Is it possible to get this kind of information with openshift ?
These operations are typically recorded in the audit logs (if you have enabled those): https://docs.openshift.com/container-platform/4.7/security/audit-log-view.html
So you can filter certain actions for example like so (GET actions):
oc adm node-logs node-1.example.com --path=oauth-apiserver/audit.log \
| jq 'select(.verb != "get")'
we are working with PubSub to integrate several systems with each other. Some systems may push data to PubSub as JSON, while others can pull that data and use it. (Note: we have to pull from PubSub instead of push to the app due to other restrictions with the receiving application) Every pulling application gets it's own subscriber to each topic.
I have noticed that the PubSub pull does not get all data currently in the queue if it is triggered too frequently. The problem originally occurred in a Java Spring App with the respective library, but the gcloud command in the cloud console exhibits the same behaviour, so I am just going to use that example. I removed the ack-ids and borders to make it fit this window. Note how I don't use the '--auto-ack' flag, so the queue should stay the same, no other system is pulling from that subscriber.
First pull (complete content):
max_binnewies#cloudshell:~ $ gcloud pubsub subscriptions pull testSubscriber --limit=100
│ DATA │ MESSAGE_ID │
│ 4 - FOUR │ 189640873208084 │
│ 5 - FIVE │ 189636274179799 │
│ 2 - TWO │ 189638666587304 │
│ 3 - THREE │ 189627470480903 │
│ 1 - ONE │ 189639207684195 │
Second pull (only one):
max_binnewies#cloudshell:~ $ gcloud pubsub subscriptions pull testSubscriber --limit=100
│ DATA │ MESSAGE_ID │
│ 1 - ONE │ 189639207684195 │
Third pull (two different ones):
max_binnewies#cloudshell:~ $ gcloud pubsub subscriptions pull testSubscriber --limit=100
│ DATA │ MESSAGE_ID │
│ 4 - FOUR │ 189640873208084 │
│ 5 - FIVE │ 189636274179799 │
Fourth pull (first one again):
max_binnewies#cloudshell:~ $ gcloud pubsub subscriptions pull testSubscriber --limit=100
│ DATA │ MESSAGE_ID │
│ 1 - ONE │ 189639207684195 │
That behaviour is confusing to me. Is that normal PubSub behaviour or am I doing something wrong? The only thing I found is this link where it says that PubSub uses load balancing for the pull method:
https://cloud.google.com/pubsub/docs/subscriber
So I think that the subscriber thinks multiple clients are subscribing to it and spreads out the data if calls come in too quickly. Is that correct? What exactly is happening here?
If I wait a little while, I get more data again, but I never seem to be getting everything even if I wait five minutes... It is very confusing.
Can that cause a problem for the consuming application? How do I make sure all the data arrives at the receiving application even if it pulls very frequently? Is there a way to turn this off?
There are a couple of things the result in you not receiving all messages every time:
With pull requests, there is no guarantee that all messages will be returned in a particular request, even if there are fewer messages available than max messages. This is because Pub/Sub tries to balance returning more messages with minimizing end-to-end latency.
Messages have an ack deadline, which is specified on subscription creation time (and defaults to 10 seconds). What this means is that when you pull messages and don't ack or nack them, they will not be redelivered for the period of the ack deadline, basically giving the process that pulled the messages a lease on them. If you want messages to be redelivered immediately, you'd need to nack them if you are using the Java client library (the preferred way to interact with Cloud Pub/Sub) or you need to send a ModifyAckDeadline request with the ack_deadline_seconds set to 0.
Is it possible to trigger an HTTP cloud function in response to a pubsub message?
When editing a subscription, google makes it possible to push the message to an HTTPS endpoint, but for abuse reasons one has to be able to prove that you own the domain in order to do this, and of course you can't prove that you own google's own *.cloudfunctions.net domain which is where they get deployed.
The particular topic I'm trying to subscribe to is a public one, projects/pubsub-public-data/topics/taxirides-realtime. The answer might be use a background function rather than HTTP triggered, but that doesn't work for different reasons:
gcloud functions deploy echo --trigger-resource projects/pubsub-public-data/topics/taxirides-realtime --trigger-event google.pubsub.topic.publish
ERROR: gcloud crashed (ArgumentTypeError): Invalid value 'projects/pubsub-public-data/topics/taxirides-realtime': Topic must contain only Latin letters (lower- or upper-case), digits and the characters - + . _ ~ %. It must start with a letter and be from 3 to 255 characters long.
This seems to indicate this is only permitted on topics I own, which is a strange limitation.
It is possible to publish from a pub/sub topic to a cloud function. I was looking for a way to publish messages from a topic in project A to a function in project B. This was not possible with a regular topic trigger, but it is possible with http-trigger. Overall steps to follow:
Creata a http-triggered function in project B.
Create a topic in project A.
Create a push subscription on that topic in project A.
Domain verification
Push subscription
Here we have to fill in three things: the endpoint, the audience and the service account under which the function runs.
Push Endpoint: https://REGION-PROJECT_ID.cloudfunctions.net/FUNC_NAME/ (slash at end)
Audience: https://REGION-PROJECT_ID.cloudfunctions.net/FUNC_NAME (no slash at end)
Service Account: Choose a service account under which you want to send the actual message. Be sure the service account has the "roles/cloudfunctions.invoker" role on the cloud function that you are sending the messages to. Since november 2019, http-triggered functions are automatically secured because AllUsers is not set by default. Do not set this property unless you want your http function to be public!
Domain verification
Now you probably can't save your subscription because of an error, that is because the endpoint is not validated by Google. Therefore you need to whitelist the function URL at: https://console.cloud.google.com/apis/credentials/domainverification?project=PROJECT_NAME.
Following this step will also bring you to the Google Search Console, where you would also need to verify you own the endpoint. Sadly, at the time of writing this process cannot be automated.
Next we need to add something in the lines of the following (python example) to your cloud function to allow google to verify the function:
if request.method == 'GET':
return '''
<html>
<head>
<meta name="google-site-verification" content="{token}" />
</head>
<body>
</body>
</html>
'''.format(token=config.SITE_VERIFICATION_CODE)
Et voila! This should be working now.
Sources:
https://github.com/googleapis/nodejs-pubsub/issues/118#issuecomment-379823198,
https://cloud.google.com/functions/docs/calling/http
Currently, Cloud Functions does not allow one to create a function that receives messages for a topic in a different project. Therefore, specifying the full path including "projects/pubsub-public-data" does not work. The gcloud command to deploy a Cloud Function for a topic expects the topic name only (and not the full resource path). Since the full resource path contains the "/" character, it is not a valid specification and results in the error you see.
The error you are getting seems to be that you are misspelling something in the gcloud command you are issuing.
ERROR: gcloud crashed (ArgumentTypeError): Invalid value 'projects/pubsub-public-data/topics/taxirides-realtime': Topic must contain only Latin letters (lower- or upper-case), digits and the characters - + . _ ~ %. It must start with a letter and be from 3 to 255 characters long
Are you putting a newline character in the middle of the command?
I submit a job on SGE with parameter -l like:
qsub -pe orte 4 -l nodes=4 run.sh
However, the system displays that:
Unable to run job: unknown resource "nodes".
Could you tell me why and how to solve it?
Thank you very much!
With Sun Grid Engine, the correct resource parameter is h, not nodes:
echo 'echo `hostname`' | qsub -l h=<some_hostname>
Using this example, you should see the hostname you specified in the standard output file.
There isn't a nodes resource. Instead you request a parallel environment and a number of slots (map to cores usually). The number of nodes you get is determined by the alloaction_rule of the parallel environment. There is usually a simple pe called something like mpi that will pack as many slots(cores) onto each node as will fit. Some people have created configs for grid engine that let it have a more PBS like syntax.