I need to read the status of a storage trigger cloud function, the last line of the log
2021-03-09 15:47:42.908 IST function_name 6cymbohrrv6g Function execution took 11 ms, finished with status: 'ok'
This way I can determine if the function execution was a success or not.
I tried reading the logs using the below script, but it displays all the logs.
logging_client = logging.Client()
for entry in logging_client.list_entries():
timestamp = entry.timestamp.isoformat()
print("* {}: {}".format(timestamp, entry.payload))
Is there some way I can limit the logs to the current execution? Can I read the last log which displays the status of the function?
You can use and advanced filter in your logging_client.list_entries() method. The official documentation of the Python client library for Stackdriver Logging admits this parameter.
filter (str) – a filter expression. See https://cloud.google.com/logging/docs/view/advanced_filters
For example:
from google.cloud import logging
logging_client = logging.Client()
f = 'resource.type="cloud_function" AND resource.labels.function_name="yourFunctionName" AND textPayload:"Function execution took"'
for entry in logging_client.list_entries(filter_=f):
print("{} {}".format(entry.labels['execution_id'], entry.payload))
will output:
...
gssssmirx021 Function execution took 20522 ms, finished with status code: 200
gsssn1aibbc5 Function execution took 20022 ms, finished with status code: 200
Notice the substring operator (:) on textPayload:"Function execution took"
Related
Here is my setting
1.action:sending request to a URL
2.trigger: every 1 min
I checked my action's result
"status":"success",
"success":true
Then, I checked the sysdig's dashboard named [Clound Functions].
The result of [Cloud Functions - status-success] is always 0.
(When the response is 404 error,[Cloud Functions - status-success] is always null. )
I think the right value of [Cloud Functions - status-success] should be 1 when the action is successful.
I wonder if the SYSDIG didn't receive the right metric info from functions?
Clound Functions Monitor on SYSDIG
One Sentence
Got MySQL invalid connection issue when MaxOpenConns are abundant and wait_timeout is 8h.
Detailed
I've a script intending to read all records from table A, make some transformation, and write the resulted records to table B. And the code works this way:
One goroutine scans table A, putting the records into a channel;
Other four goroutine (number configurable) concurrently consume from above channel, accumulating 50 rows (batch size configurable) to insert into table B, then accumulating another 50 rows, and so on so forth.
Scanner goroutine holds one *sql.DB, and inserter goroutines share another *sql.DB
go-sql-driver: either Version 1.4.1 (2018-11-14) or Version 1.5 (2020-01-07)
(problem encountered with 1.4.1, and reproducible demo, see below, uses 1.5)
Go version: go1.13.15 darwin/amd64
The invalid connection issue is almost steadily reproducible.
In a specific running case, table A has 67227 records, channel size is set to 100000, table A scanner (1 goroutine) reads 1000 a time, table B inserter(4 goroutines) write 50 a time. It ends up with 67127 records in table B (2*50 lost), and 2 lines of error output in console:
[mysql] 2020/12/11 21:54:18 packets.go:36: read tcp x.x.x.x:64062->x.x.x.x:3306: read: operation timed out
[mysql] 2020/12/11 21:54:21 packets.go:36: read tcp x.x.x.x:64070->x.x.x.x:3306: read: operation timed out
(The number of error lines varies when I reproduce, it's usually 1, 2 or 3. N error lines coincide with N*50 records insertion failure into table B.)
And from my log file, it prints invalid connection:
2020/12/11 21:54:18 main.go:135: [goroutine 56] BatchExecute: BatchInsertPlace(): SqlDb.ExecContext(): invalid connection
Stats={MaxOpenConnections:0 OpenConnections:4 InUse:3 Idle:1 WaitCount:0 WaitDuration:0s MaxIdleClosed:14 MaxLifetimeClosed:0}
2020/12/11 21:54:21 main.go:135: [goroutine 55] BatchExecute: BatchInsertPlace(): SqlDb.ExecContext(): invalid connection
Stats={MaxOpenConnections:0 OpenConnections:4 InUse:3 Idle:1 WaitCount:0 WaitDuration:0s MaxIdleClosed:14 MaxLifetimeClosed:0}
Trials and observations
By printing each success/ fail write operation with goroutine id in log, it appears that the error always happen when any 1 of all 4 inserting goroutines has an over ~45 seconds interval between 2 consecutive writes. I think it's just taking this long to accumulate 50 records before inserting them to table B.
In contrast, when I happened to make a change so that the 4 inserting goroutines write some averagely, (i.e. no one has a much longer writing interval than others), the error is not seen. Repeated 3 times.
Looks one error only affects one batch write operation, and the following batches work well. So why not retry with the errored batch? I suppose one retry and it will get through. Still, I don't mind keep retrying until success:
var retryExecTillSucc = func(goroutineId int, records []*MyDto) {
err := inserter.BatchInsert(records)
for { // retry until success. This is a workaround for 'invalid connection' issue
if err == nil { break }
logger.Printf("[goroutine %v] BatchExecute: %v \nStats=%+v\n", goroutineId, err, inserter.RdsClient.SqlDb.Stats())
err = inserter.retryBatchInsert(records)
}
logger.Printf("[goroutine %v] BatchExecute: Success \nStats=%+v\n", goroutineId, inserter.RdsClient.SqlDb.Stats())
}
Surprisingly, with this change, retries of the errored batch keep getting error and never succeed...
Summary
It looks obvious that one (idle) connection was broken when the error occur, but my question is:
MySQL wait_timeout is set 8h, so why is the connection timed out so quickly?
Since MaxOpenConns is not set, it shouldn't be a limitation, especially considering the merely 4 OpenConnections in log.
What else to check as potential root cause?
(Too long, but just hope to put it clearly and get some advice~)
Update
Minimal, reproducible example, including:
Code
One sample log file
MySQL error log
Don't you use Context? I suppose the read timeout is caused by Context Timeout, or readTimeout parameter.
MySQL doesn't provide safe and efficient canceling mechanism. When context is cancelled or reached readTimeout, DB.ExecContext returns without terminating using connection. It cause "invalid connection" next time the connection is used.
If you want to limit execution time of long query, you can use MAX_EXECUTION_TIME hint instead of context.
See https://dev.mysql.com/doc/refman/5.7/en/optimizer-hints.html#optimizer-hints-execution-time for reference.
I have a route in my API, as an example lets call it /users/:userId/updateBalance. This route will fetch the users current balance, add whatever comes from the request and then update the balance with the newly calculated balance. A request like this comes into the server for a specific user every 30 minutes, so until recently, I thought a concurrency issue to be impossible.
What ended up happening is that somewhere, a sent request failed and was only sent again 30 minutes later, about within a second of the other request. The result was that, as I can see it in the database, both of these requests fetched the same balance from the DB and both added their respective amounts. Essentially, the second request actually read a stale balance, as normally it should execute after request 1 has executed.
To give a numerical example for some more clarity, lets say request 1 was to add $2 to the balance, and request 2 was to add $5, and the user had a balance of $10. If the requests act in parallel, the users balance would end at either $12 or $15 depending on whether request 1 or request 2 finished first respectively, because both requests fetch a balance of $10 from the DB. However, obviously the expected behaviour is that we want request 1 to execute, update the users balance to $12, and then request 2 to execute and update the balance from $12 to $17.
To give some better perspective of the overall execution of this process: the request is received, a function is called, the function has to wait for the balance from the DB, the function then calculates the new balance and updates the db, after which execution is completed.
So I have a few questions on this. The first being, how does node handle incoming requests when it is waiting for an asynchronous request like a MySQL database read. Given the results I have observed, I assume that when the first request is waiting for the DB, the second request can commence being processed? Otherwise I am uncertain of how such asynchronous behaviour is experienced within a single threaded environment like node.
Secondly, how do I go about controlling this and preventing it. I had wanted to use a MySQL transaction with a forUpdate lock, but it turns out it seems not possible due to the way the code is currently written. Is there a way to tell node that a certain block of code can not be executed "in parallel"? Or any other alternatives?
You are right, while node waits for the database query to return, it will handle any incoming requests and start that requests database call before the first one finishes.
The easiest way to prevent this IMO would be to use queues. Instead of processing the balance update directly in the route handler, that route handler could push an event to a queue (in Redis, in AWS SQS, in RabbitMQ etc) and somewhere else in your app (or even in a completely different service) you would have a consumer that listens to new events in that queue. If an update fails, add it back to the beginning of the queue, add some wait time, and then try again.
This way, no matter how many times your first request fails, your balance will be correct, and pending changes to that balance will be in the correct order. In case of an event in the queue failing repeatedly, you could even send an email or a notification to someone to have a look at it, and while the problem is fixed pending changes to the balance will be added to the queue, and once it's fixed, everything will be processed correctly.
You could even read that queue and display information to your user, for instance tell the user the balance has pending updates so it might not be accurate.
Hope this helps!
The first being, how does node handle incoming requests when it is waiting for an asynchronous request like a MySQL database read
The event loop of nodejs makes this happens, otherwise you'll have a totally sync programm with super-low performances.
Every single async function invocked in a context will be executed after the context itself has been executed.
Between the finish of execution of the context and the execution of the async function, other async functions can be scheduled for been executed (this "insertion" is managed by the event loop).
If an async function is awaited, the remaining code of the context is scheduled somewhere after the execution of the async function.
Is more clear when playing with it. Example 1:
// Expected result: 1, 3, 4, 2
function asyncFunction(x) {
// setTimeout as example of async operation
setTimeout(() => console.log(x), 10)
}
function context() {
console.log(1)
asyncFunction(2)
console.log(3)
}
context()
console.log(4)
Example 2:
// Expected result: 1, 2, 3
function asyncFunction(x) {
// Promise as example of async operation
return new Promise((resolve) => {
console.log(x)
resolve()
})
}
async function context() {
console.log(1)
await asyncFunction(2)
console.log(3)
}
context()
Example 3 (more similar to your situation):
// Expected result: 1, 2, 4, 5, 3, 6
function asyncFunction(x) {
// Promise as example of async operation
return new Promise((resolve) => {
console.log(x)
resolve()
})
}
async function context(a, b, c) {
console.log(a)
await asyncFunction(b)
console.log(c)
}
context(1, 2, 3)
context(4, 5, 6)
In your example:
when the server receive a connection, the execution of the handler is scheduled
when the handler is executed, it schedule the execution of the query, and the remaining portion of the handler context is scheduled after that
In between scheduled executions everything can happen.
I need to elaborate the GUID of every subpackage which is executed by a main package.
In order to do that, I've handled the event Pre-Execute, to record every source of Pre-Execute event. The problem is that in PreExecute event handler, it isnt possible to know if the source of the event, represented by SourceID and SourceName, is a subpackage or a task.
I would code only in the main package, in order to centralize the code.
Other attemps:
I've tried with other events, like PreValidate, and the issue is the same.
Looking at the standard output (see below) I found a specific log row for subpackages, but I dont know how to read standard output log at runtime.
This is the standard output:
Log:
Name: User:PackageStart
Computer: COMPUTERNAME
Operator: username
Source Name: childpackage
Source GUID: {9E629F59-5DD1-44AD-BD64-9704353267E2}
Execution GUID: {4168C4C6-1EC7-41C2-8F2D-98AD3A4F4D82}
Message: Beginning of package execution.
Start Time: 2016-05-16 12:28:16
End Time: 2016-05-16 12:28:16
End Log
Thank you
Question on agents: I specifically want to create a Periodic Task, but only want to run it once every day, say 1am, not every 30 minutes which is the default. In the OnInvoke, do I simply check for the hour, and run it only if current hour matches that desired hour.
But on the next OnInvoke call, it will try to run again in 30 minute, maybe when it's 1:31am.
So I guess I'd use a stored boolean in the app settings to mark as "already run for today" or similar, and then check against that value?
If you specifically want to run a custom action at 1 am, i'm not sure that a single boolean would be enough to make it work.
I guess that you plan to reset your boolean at 1:31 to prepare the execution of the next day, but what if your periodic task is also called at 1h51 (so called more than 2 times between 1am and 2am).
How could this happen? Well maybe this could happen if the device is reboot but i'm not quiet sure about it. In any case, storing the last execution datetime somewhere and comparing it to the current one can be a safer way to ensure that your action is only invoked once per day.
One question remains : Where to store your boolean or datetime (depending which one you'll pick)?
AppSetting does not seem to be a recommanded place according msdn :
Passing information between the foreground app and background agents
can be challenging because it is not possible to predict if the agent
and the app will run simultaneously. The following are recommended
patterns for this.
For Periodic and Resource-intensive Agents: Use LINQ 2 SQL or a file in isolated storage that is guarded with a Mutex. For
one-direction communication where the foreground app writes and the
agent only reads, we recommend using an isolated storage file with a
Mutex. We recommend that you do not use IsolatedStorageSettings to
communicate between processes because it is possible for the data to
become corrupt.
A simple file in isolated storage should get the job done.
If you're going by date (once per day) and it's valid that the task can run at 11pm on a day and 1am the next, then after the agent has run you could store the current date (forgetting about time). Then whenever the agent runs again in 30 minutes, check if the date the task last ran is the same as the current date.
protected override void OnInvoke(ScheduledTask task)
{
var lastRunDate = (DateTime)IsolatedStorageSettings.ApplicationSettings["LastRunDate"];
if(DateTime.Today.Subtract(lastRunDate).Days > 0)
{
// it's a greater date than when the task last ran
// DO STUFF!
// save the date - we only care about the date part
IsolatedStorageSettings.ApplicationSettings["LastRunDate"] = DateTime.Today;
IsolatedStorageSettings.ApplicationSettings.Save();
}
NotifyComplete();
}