There is a lot of activity on my Google Compute engine API. It's less than 1 request per second which probably keeps me in the free zone but how do I figure out what is running and if I should stop it?
I have some pub/sub topics and a cloud function to copy data into a dataStore database. But even if I am not publishing any data (for days), I still get activity on the compute engine? Can I disable it or will it stop my cloud functions?
Related
I want to stream real time data from Twitter API to Cloud Storage and BigQuery. I have to ingest and transform the data using Cloud Functions but the problem is I have no idea how to pull data from Twitter API and ingest it into the Cloud.
I know I also have to create a scheduler and a Pub/Sub topic to trigger Cloud Functions. I have created a Twitter developer account. The main problem is actually streaming the data into Cloud Storage.
I'm really new to GCP and streaming data so it'll be nice to see a clear explanation on this. Thank you very much :)
You have to design first your solution. What do you want to achieve? Streaming or Microbatches?
If streaming, you have to use the streaming API of Twitter. In short, you initiate a connection and you stay up and running (and connected) receiving the data.
If batches, you have to query an API and to download a set of message. In a Query-response mode.
That being said, how to implement it with Google Cloud. Streaming is problematic because you have to be always connected. And with serverless product you have timeout concern (9 minutes for Cloud Functions V1, 60 minutes for Cloud Run and Cloud Functions V2).
However you can imagine to invoke regularly your serverless product, stay connected for a while (let say 1h) and schedule trigger every hour.
Or use a VM to do that (or a pod on a K8S container)
You can also consider microbatches where you invoke every minute your Cloud Functions and to get all the messages for the past minutes.
At then end, all depends on your use case. What's the real time that you expect? which product do you want to use?
I am in the process of architecting a small application for a G Suite customer. The app will be used by all employees at the customer and during peak times could see as many as 5,000 concurrent users.
They do not have App Maker and do not plan to enable App Maker anytime soon.
Will an Apps Script Web App be able to handle this many concurrent users?
Database:
I was originally thinking of using a Google Sheet as the database. There are no documented Apps Script limits around reading or writing data to a Google Sheet. So as long as I stay within the Google Sheets API quota limits I should be good.
With that said, I am also debating on using Cloud SQL as the database. App Maker forces you to use Cloud SQL so I assume it is a superior option to a Google Sheet. The only way I see to connect to Cloud SQL from Apps Script is via the JDBC service. However there is a hard limit of 50,000 connections per day. https://developers.google.com/apps-script/guides/services/quotas#current_quotas
Does anyone know if this limit is per app script or per user?
If per app script, then 5,000 users would only have 10 calls each per day. That would be unusable for my needs.
Side note, Google Cloud SQL has a maximum of 4,000 connections. I am banking on the fact that reads and writes will be extremely fast so max connections at a single point in time will be less than 4,000.
As TheMaster noted in the above comment you have a max of 30 simultaneous executions which limits you to a max of 30 concurrent users for a GAS Web App.
As an alternative you might be able to leverage Cloud Functions (basically just a Node.js/Express.js module). ̶M̶a̶x̶ ̶n̶u̶m̶b̶e̶r̶ ̶o̶f̶ ̶c̶o̶n̶c̶u̶r̶r̶e̶n̶t̶ ̶u̶s̶e̶r̶s̶ ̶i̶s̶ ̶1̶0̶0̶0̶ ̶t̶h̶o̶u̶g̶h̶. It's not a free platform, but it supports CloudSQL and may be cheaper than getting a Google for Business account (US$11.00 per month per user) as required by App Maker.
Is there any API I can use to get the total cost of a VM instance in Google compute?
The usage scenario is like this:
Server starts
Runs for some Hours / Days
Gets shut down
For reporting purposes, we get the cost of the server and save it in our DB
Thanks
Google has a system for exporting billing information each day in a csv/json file to a storage bucket.
http://googlecloudplatform.blogspot.ca/2013/12/ow-get-programmatic-access-to-your-billing-data-with-new-billing-api.html
Anyone know if there is a way to download historical cpu usage data via API call for Google Compute Engine?
On the Console overview page, graphs are provided for this type of data for at least a month, but I don't see anything obvious on how to download the actual data directly.
The Google Compute Engine usage export feature recently launched - it sounds like what you're looking for. It gives daily detail in a CSV, as well as a month-to-date summary.
I had setup it for more than 5 days, and the error is this:
Error
API rate limit exceeded. Rate limit may take several minutes to update if Google Compute Engine has just been enabled, or if this is the first time you use Google Compute Engine.
the all menu diplay the error,and I can't on API.
I had a same issue as you. After setup new project and activation Compute Engine API I wasn't able to setup anything because of Error API rate limit.
The main issue I guess is in Courtesy Limit for API which I wasn't able to setup from N/A to some value
Steps which helped me to activate the API limit:
Turn OFF the Compute Engine API
Go to https://appengine.google.com -> Application Settings -> Cloud Integration - Create cloud Integration
Turn ON the Compute Engine API, it will automatically setup Courtesy Limit to 250,000 requests/day
After that, you will be able to use Compute Engine instances.
I can't 100% guarantee that will work in your case, but it helped me.
In my case, my trial had ended and I had to upgrade. There was a message at the top to upgrade. Click and refresh. Start her up again.