my nodejs script is not exiting on its own after successful execution - mysql

I have written a script to update my db table after reading data from db tables and solr. I am using asyn.waterfall module. The problem is that the script is not getting exited after successful completion of all operations. I have used db connection pool also thinking that may be creating the script to wait infinitly.
I want to put this script in crontab and if it will not exit properly it would be creating a hell lot of instances unnecessarily.

I just went through this issue.
The problem with just using process.exit() is that the program I am working on was creating handles, but never destroying them.
It was processing a directory and putting data into orientdb.
so some of the things that I have come to learn is that database connections need to be closed before getting rid of the reference. And that process.exit() does not solve all cases.
When my project processed 2,000 files. It would get down to about 500 left, and the extra handles would have filled up the available working memory. Which means it would not be able to continue. Therefore never reaching the process.exit at the end.
On the other hand, if you close the items that are requesting the app to stay open, you can solve the problem at its source.
The two "Undocumented Functions" that I was able to use, were
process._getActiveHandles();
process._getActiveRequests();
I am not sure what other functions will help with debugging these types of issues, but these ones were amazing.
They return an array, and you can determine a lot about what is going on in your process by using these methods.

You have to tell it when you're done, by calling
process.exit();
More specifically, you'll want to call this in the callback from async.waterfall() (the second argument to that function). At that point, all your asynchronous code has executed, and your script should be ready to exit.
EDIT: As pointed out by #Aaron below, this likely has to do with something like a database connection being active, and not allowing the node process to end.

You can use the node module why-is-node-running:
Run npm install -D why-is-node-running
Add import * as log from 'why-is-node-running'; in your code
When you expect your program to exit, add a log statement:
afterAll(async () => {
await app.close();
log();
})
This will print a list of open handles with a stacktrace to find out where they originated:
There are 5 handle(s) keeping the process running
# Timeout
/home/maf/dev/node_modules/why-is-node-running/example.js:6 - setInterval(function () {}, 1000)
/home/maf/dev/node_modules/why-is-node-running/example.js:10 - createServer()
# TCPSERVERWRAP
/home/maf/dev/node_modules/why-is-node-running/example.js:7 - server.listen(0)
/home/maf/dev/node_modules/why-is-node-running/example.js:10 - createServer()

We can quit the execution by using:
connection.destroy();

If you use Visual Studio code, you can attach to an already running Node script directly from it.
First, run the Debug: Attached to Node Process command:
When you invoke the command, VS Code will prompt you which Node.js process to attach to:
Your terminal should display this message:
Debugger listening on ws://127.0.0.1:9229/<...>
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
Then, inside your debug console, you can use the code from The Lazy Coder’s answer:
process._getActiveHandles();
process._getActiveRequests();

Related

Exception handling with Realbrowserlocusts

In using realbrowserlocusts class it appears that I'm limited in any exception handling.
The only reference that partially works is: self.client.wait.until(EC.visibility_of_element_located ....
In a failed condition where the element is not found the script simply starts over again. With the script I'm working with I need to maintain a solid session state; I need to throw and exception(report an error), log the user out and then let the script start over again. I've been testing out the behavior with the locust.py script that Nick B. created with several approaches to "try, except" and they work running without realbrowserlocusts (selenium only) but with it the execution just stops.
Any examples would be greatly appreciated.
In its current format I've been able to run 3x the amount of a browser-based load per/agent/slave than our commercial tool. My goal is to replace it with a locust/selenium approach.
locust-plugins's WebdriverUser has a little bit better exception handling I think. A failure to find an element will log a failed request and if you use RescheduleTaskOnFail (as in the the example) it will restart the task when that happens.
https://github.com/SvenskaSpel/locust-plugins/blob/master/examples/webdriver_ex.py

Google Cloud Function: lazy loading not working

I deploy a google cloud function with lazy loading that loads data from google datastore. The last update time of my function is 7/25/18, 11:35 PM. It works well last week.
Normally, if the function is called less than about 30 minutes since last called. The function does not need to load data loaded from google datastore again. But I found that the lazy loading is not working since yesterday. Even the time between two function is less than 1 minute.
Does anyone meet the same problem? Thanks!
The Cloud Functions can fail due to several reasons such as uncaught exception and internal process crashes, therefore, it is required to check the logs files / HTTP responses error messages to verify the issue root cause and determine if the function is being restarted and generating Function execution timeouts that could explain why your function is not working.
I suggest you take a look on the Reporting Errors documentation that explains the process required to return a function error in order to validate the exact error message thrown by the service and return the error at the recommended way. Keep in mind that when the errors are returned correctly, then the function instance that returned the error is labelled as behaving normally, avoiding cold starts that leads higher latency issues, and making the function available to serve future requests if need be.

How to know all the resources, connections, sockets, etc. in use by a Node child-process on Linux

I generate a node script dynamically, then fork a child process through node to execute it and pass the output back to the parent process
In parent:
//...build child.js
var child = require('child_process');
childProcess = child.fork('child.js');
childProcess.on('message',
function(dataFromChild) {
//process data
});
In child process
process.send(objectToSend,function() {
//code to send objectToSend
//process.exit(0);
}
The actual code is a lot more extensive. I also use socket.io and mysql. I think I am closing all opened db connections and sockets in the child process when I am done processing. However, the child process does not exit. If I do a process.exit(0) at the point of exit, it does exit. I figure there must be something that it is open that is not allowing it to exit naturally. How do I know what it is?
I have tried a simple child script, basically a one-liner with a console.log statement to test if it will exit without an exit statement. It does.
You can use the linux lsof program to list all open file descriptors owned by the child process. In node, the active-handles npm package should help you track down allocated listeners, timeouts, etc.

Fiware CEP server stops responding

In developing in Fi-Cloud's CEP I've been having an issue that has been happening repeatedly. As I'm trying to develop a definition to perform a task, CEP's server and Authoring Tool stop responding, although ssh is still responsive.
This issue happens as I develop. I'm using the AuthoringTool to alter the definition bit by bit and then I re-upload it to the server through the authoring tool's export feature.
To reinitiate the proton with the new definition each time I alter it, I use Google's Postman with this single operation:
-PUT (url:http://{ip}:8080/ProtonOnWebServerAdmin/resources/instances/ProtonOnWebServer)
header: 'Content-Type' : 'application/json'; body : {"action": "ChangeDefinitions","definitions-url" : "/ProtonOnWebServerAdmin/resources/definitions/Definition_Name"}
At the same time, I'm logged in with three ssh intances, one to monitor the files being created on /opt/tomcat10/sample/ and other things, and the other two to 'tail -f ' log files the definition writes to, as events are processed: one log for events recieved and another log for events detected by the EPAgent.
I'm iterating through these procedures over and over as I'm developing and eventualy CEP server and the Authoring Tool stop responding.
By "tailing" tomcat's log file (# tail -f /opt/tomcat10/logs/catalina.out) I can see that, when under these circumstances, if I attemp a:
-GET (url: http://{ip}:8080/ProtonOnWebServerAdmin/resources/instances/ProtonOnWebServer)
I get no response back and tomcat logs the following response:
11452100 [http-bio-8080-exec-167] ERROR org.apache.wink.server.internal.RequestProcessor - An unhandled exception occurred which will be propagated to the container.
java.lang.OutOfMemoryError: PermGen space
Exception in thread "http-bio-8080-exec-167" java.lang.OutOfMemoryError: PermGen space
Ssh is still responsive and I can look at tomcat's log this way.
To get over this and continue, I exit ssh connections and restart CEP's instance in the Fi-Cloud.
Is the procedure I'm using to re-upload and re-run the definition inapropriate? Should I take a different approach to developing?
When you update a definition that the CEP is already working with, and you want the CEP engine to work with the updated definition, you need to:
Export the definition using the authoring tool export (as you did)
Stop the engine run, using REST PUT
PUT //host:8080/ProtonOnWebServerAdmin/resources/instances/ProtonOnWebServer
{"action":"ChangeState","state":"stop"}
Start the engine, using REST PUT
PUT //host:8080/ProtonOnWebServerAdmin/resources/instances/ProtonOnWebServer
{"action":"ChangeState","state":"start"}
You don't need to activate the "ChangeDefinitions" action, since it is the same definition name that the engine is already working with.
Activating "ChangeDefinitions" action, only influences the next run of the CEP, and has no influence on the current run.
This answer your question about how you should update a CEP definition.
Hope it will solve your issue.

How do BundleActivator, ManagedService, and my application interact on start/stop?

I had a non-OSGi application. To convert it to OSGi, I first bundled it up and gave it a simple BundleActivator. The activator's start() started up a thread of what used to be the main() of my app (and is now a Runnable), and remembered that thread. The activator's stop() interrupted that thread, and waited for it to end (via join()), then returned. This all seemed to be working fine.
As a next step in the OSGiification process, I am now trying to use OSGi configuration management instead of the Properties-based configuration that the application used to use. So I am adding in a ManagedService in addition to the Activator.
But it's no longer clear to me how I am supposed to start and stop my application; examples that I've seen are only serving to confuse me. Specifically, here:
http://felix.apache.org/site/apache-felix-config-admin.html
They no longer seem to do any real starting of the application in BundleActivator.start(). Instead, they just register a ManagedService to receive configuration. So I'm guessing maybe I start up the app's main thread when I receive configuration, in the ManagedService? They don't show it - the ManagedService's updated() just has vague comments saying to "apply configuration from config admin" when it is passed a non-null Dictionary.
So then I look here:
http://blog.osgi.org/2010/06/how-to-use-config-admin.html
In there, it seems like maybe they're doing what I guessed. They seem to have moved the actual app from BundleActivator to ManagedService, and are dealing with starting it when updated() receives non-null configuration, stopping it first if it's already started.
But now what about when the BundleActivator's stop() gets called?
Back on the first example page that I mentioned above, they unregister the ManagedService. On the second example page, they don't show what they do.
So I'm guessing maybe unregistering the ManagedService will cause null configuration to be sent to ManagedService.updated(), at which point I can interrupte the app thread, wait for it to end, and then return?
I suspect that I'm thoroughly incorrect, but I don't know what the "real" way to do this is. Thanks in advance for any help.
BundleActivator (BA) and ManagedService (MS) are callbacks to your bundle. BundleActivator is for the active state of your bundle. BA.start is when you bundle is being started and BA.stop is when it is being stopped. MS is called to provide your bundle a configuration, if there is one, or notify you there is no configuration.
So in BA.start, you register your MS service and return. When MS is called (on some other thread), you will either receive your configuration or be told there is no configuration and you can act accordingly (start app, etc.)
Your MS can also be called at anytime to advice of the modification or deletion of your configuration and you should act accordingly (i.e. adjust your app behavior).
When you are called at BA.stop, you need to stop your app. You can unregister the MS or let the framework do it for you as part of normal bundle stop processing.