I'm having issue getting answers from a Google Coud Function going through ESPv2.
Every time I request it, I get a response 15 seconds later with a status code of 504.
My function take between 30 to 45 seconds.
In the logs the functions correctly and answer back after 35 seconds.
Is there a way to increase the timeout in ESPv2 ?
Thanks
For anyone else having this issue, in the openapi-functions.yaml under the x-google-backend you should had the attribut deadline and set it to whatever value in seconds you want.
Here is the hidden documentation https://cloud.google.com/endpoints/docs/openapi/openapi-extensions#deadline
Issue related: https://github.com/GoogleCloudPlatform/esp-v2/issues/4
Depending on the documentation you used to secure the endpoints of your Cloud Function’s with ESPv2, this should be possible. If you are using Cloud Run to host your ESPv2, a 504 error is sent when a request exceeds its request timeout limit. The request timeout limit is a setting that specifies the time within which a response must be returned before sending a 504 response. You can change this value by going in your “Cloud Run” tab, selecting your ESPv2 service, selecting “Edit & Deploy new Revision”, scrolling down to the capacity section and setting the time in milliseconds. This is some documentation that could prove helpful when working with the topics discussed.
Related
I am trying a build a realtime monitoring system for high frequency data. To increase the performance, I used the extendData property of dcc.Graph() and websocket. So that, the brouser does not need to send request to get data.
I found that it still not increasing the performance as expected. The reason I found is, from the browser, I see (by inspecting network from browser) after some miliseconds browser is still sendng request and the initiator is the dash_renderer.
This picture is for a vanilla example just to show even for a textbox example the http request goes on and on. And for my real time websocket dashboard the frequency of requests get very high.
My question is:
What dash_renderer do?
why it is sending http request?
And how to stop that?
If you run Dash in Debug mode, it has a feature called Hot Reloading which regularly (every 3 seconds by default) checks for changes to your codebase and updates your running app if it finds any. That check for updated code is what you're seeing in the network inspection.
To turn it off, either don't run in debug mode or explicitly set dev_tools_hot_reload to False like so:
app.run_server(debug=True, dev_tools_hot_reload=False)
Although it is late, After some experience, my realization is dash is not designed to work with websocket. It uses call-backs which actually sends requests to server and in server, the callback function (which is python) send back some result.
These call-backs are designed to send HTTP request to server.
For high speed data, the websocket should be used with extendTrace method of plotly.js in client side.
I deploy a google cloud function with lazy loading that loads data from google datastore. The last update time of my function is 7/25/18, 11:35 PM. It works well last week.
Normally, if the function is called less than about 30 minutes since last called. The function does not need to load data loaded from google datastore again. But I found that the lazy loading is not working since yesterday. Even the time between two function is less than 1 minute.
Does anyone meet the same problem? Thanks!
The Cloud Functions can fail due to several reasons such as uncaught exception and internal process crashes, therefore, it is required to check the logs files / HTTP responses error messages to verify the issue root cause and determine if the function is being restarted and generating Function execution timeouts that could explain why your function is not working.
I suggest you take a look on the Reporting Errors documentation that explains the process required to return a function error in order to validate the exact error message thrown by the service and return the error at the recommended way. Keep in mind that when the errors are returned correctly, then the function instance that returned the error is labelled as behaving normally, avoiding cold starts that leads higher latency issues, and making the function available to serve future requests if need be.
I have created a URL monitor to check the availability of website.I kept all values as it is in monitor tab and added host( http://abcd.xyz.org ). I scheduled it for every 10 minutes in schedule tab. In Measure tab I put the Lower sever value of HostReachable measure to 0.
I created incident , added HostReachable measure and put its Aggregation to last value. Evaluation time frame is 1 minutes.In Action tab I put my email for notification.
Now after every 10 minutes I started getting email even I browsed the website and it's working fine. Couldn't understand if website loading fine then why incidents occurs.
Email message:
Violations
HostReachable: STG URL Monitor#abcd.xyz.org: Was 0.00 but should be higher than 0.00.
Latest logs given below:
2016-08-17 15:42:33 INFO [UrlMonitor#STG URL Monitor_0] Previous message was repeated 1 times.
2016-08-17 15:42:33 INFO [UrlMonitor#STG URL Monitor_0] Executing method: GET, URI: http://abcd.xyz.org:80/
Thanks
This could have to do with the evaluation timeframe of your Incident. If your monitor is scheduled to run every 10 minutes but your evaluation timeframe is 1 minute it means that you have 9 evaluations where there is "no data".
We typically recommend to align the monitor execution with the Incident. Can you give that a try?
Also - I typically chart these measures to see which values the monitor produces in which timeinterval. this is an easy visual verification check to make sure that e.g: the monitor works correctly and delivers data. Can you do that?
I also wanted to say that we have a very good Dynatrace Online Community and Discussion forum # http://answers.dynatrace.com. You might want to try to post future questions there as there are 100k+ Dynatrace users active on that community
Andi
We are running a Sails.js API on Google Container Engine with a Cloud SQL database and recently we've been finding some of our endpoints have been stalling, never sending a response.
I had a health check monitoring /v1/status and it registered 100% uptime when I had the following simple response;
status: function( req, res ){
res.ok('Welcome to the API');
}
As soon as we added a database query, the endpoint started timing out. It doesn't happen all the time, but seemingly at random intervals, sometimes for hours on end. This is what we have changed the query to;
status: function( req, res ){
Email.findOne({ value: "someone#example.com" }).then(function( email ){
res.ok('Welcome to the API');
}).fail(function(err){
res.serverError(err);
});
}
Rather suspiciously, this all works fine in our staging and development environments, it's only when the code is deployed in production that the timeout occurs and it only occurs some of the time. The only thing that changes between staging and production is the database we are connecting to and the load on the server.
As I mentioned earlier we are using Google Cloud SQL and the Sails-MySQL adapter. We have the following error stacks from the production server;
AdapterError: Invalid connection name specified
at getConnectionObject (/app/node_modules/sails-mysql/lib/adapter.js:1182:35)
at spawnConnection (/app/node_modules/sails-mysql/lib/adapter.js:1097:7)
at Object.module.exports.adapter.find (/app/node_modules/sails-mysql/lib/adapter.js:801:16)
at module.exports.find (/app/node_modules/sails/node_modules/waterline/lib/waterline/adapter/dql.js:120:13)
at module.exports.findOne (/app/node_modules/sails/node_modules/waterline/lib/waterline/adapter/dql.js:163:10)
at _runOperation (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/finders/operations.js:408:29)
at run (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/finders/operations.js:69:8)
at bound.module.exports.findOne (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/finders/basic.js:78:16)
at bound [as findOne] (/app/node_modules/sails/node_modules/lodash/dist/lodash.js:729:21)
at Deferred.exec (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/deferred.js:501:16)
at tryCatcher (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/main/util.js:26:23)
at ret (eval at <anonymous> (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/main/promisify.js:163:12), <anonymous>:13:39)
at Deferred.toPromise (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/deferred.js:510:61)
at Deferred.then (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/deferred.js:521:15)
at Strategy._verify (/app/api/services/passport.js:31:7)
at Strategy.authenticate (/app/node_modules/passport-local/lib/strategy.js:90:12)
at attempt (/app/node_modules/passport/lib/middleware/authenticate.js:341:16)
at authenticate (/app/node_modules/passport/lib/middleware/authenticate.js:342:7)
at Object.AuthController.login (/app/api/controllers/AuthController.js:119:5)
at bound (/app/node_modules/sails/node_modules/lodash/dist/lodash.js:729:21)
at routeTargetFnWrapper (/app/node_modules/sails/lib/router/bind.js:179:5)
at callbacks (/app/node_modules/sails/node_modules/express/lib/router/index.js:164:37)
Error (E_UNKNOWN) :: Encountered an unexpected error :
Could not connect to MySQL: Error: Pool is closed.
at afterwards (/app/node_modules/sails-mysql/lib/connections/spawn.js:72:13)
at /app/node_modules/sails-mysql/lib/connections/spawn.js:40:7
at process._tickDomainCallback (node.js:381:11)
Looking at the errors alone, I'd be tempted to say that we have something misconfigured. But the fact that it works some of the time (and has previously been working fine!) leads me to believe that there's some other black magic at work here. Our Cloud SQL instance is D0 (though we've tried upping the size to D4) and our activation policy is "Always On".
EDIT: I had seen others complain about Google Cloud SQL eg. this SO post and I was suspicious but we have since moved our database to Amazon RDS and we are still seeing the same issues, so it must be a problem with sails and the mysql adapter.
This issue is leading to hours of downtime a day, we need it resolved, any help is much appreciated!
This appears to be a sails issue, and not necessarily related to Cloud SQL.
Is there any way the QPS limit for Google Cloud SQL is being reached? See here: https://cloud.google.com/sql/faq#sizeqps
Why is my database instance sometimes slow to respond?
In order to minimize the amount you are charged for instances on per use billing plans, by default your instance becomes passive if it is not accessed for 15 minutes. The next time it is accessed there will be a short delay while it is activated. You can change this behavior by configuring the activation policy of the instance. For an example, see Editing an Instance Using the Cloud SDK.
It might be related to your policy setting. If you set it to ON_DEMAND, the instance will sleep to save your budget so that the first query to activate the instance is slow. This might cause the timeout.
https://cloud.google.com/sql/faq?hl=en
i'm looking at a stock/index:
http://www.google.com/finance?q=google
I've got firebug open, in firefox and when the price updates on the page, i dont see any GETs or POSTs, just a get maybe 30 to 60 seconds later.
Surely if the page is being updated with a value, wouldnt firebug show this data reaching the page as it happens? Or does firebug collect connections in batches?
Look further back in the GET connection logs to see if there is a connection that was opened but not closed. It's likely that the page is opening an XmlHttpRequest connection to the data server and keeping the connection open indefinitely. This is common for streaming data situations, even when the data stream is fairly low volume. If this is the case, then new data will arrive on the open connection without any new connection activity reported in the log.
There is one request that never finishes responding (or at least not for the time I watched), you can see this in the NET panel, this response periodically outputs more data which is then used to update the application. If you examine the request you will see that it specifies a header Transfer-Encoding: Chunked, which is used for these purposes, see http://en.wikipedia.org/wiki/Chunked_transfer_encoding.