Setting up Storm on Google Cloud - google-compute-engine

I was wondering if there's been any work on setting up Apache Storm on Google Compute Engine. I have been using this blog (http://datadventures.ghost.io/2013/12/29/deploying-storm-on-gce/) but I can't seem to get my nimbus instance setup to work properly.
The blog states that to check if my nimbus instance is functioning, I should have a look at the Nimbus Web UI. However, I can't seem to display the UI on my browser. I have made the necessary changes in the storm.yaml file.
My storm.yaml file looks like :
storm.zookeeper.servers:
- "test-1"
ui.port : 8080
nimbus.host: "nimbus"
nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true"
ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
supervisor.childopts: "-Djava.net.preferIPv4Stack=true"
worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
storm.local.dir: "/app/storm"
I checked my nimbus.log file and I see the following:
2016-10-26 18:57:27 b.s.d.nimbus [INFO] Starting Nimbus with conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topolo$
2016-10-26 18:57:27 b.s.d.nimbus [INFO] Using default scheduler
2016-10-26 18:57:27 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2016-10-26 18:57:27 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=test-1:2181 sessionTimeout=20000 watcher=com.netflix.curator.ConnectionSt$
2016-10-26 18:57:27 o.a.z.ClientCnxn [INFO] Opening socket connection to server test-1/10.142.0.9:2181
2016-10-26 18:57:27 o.a.z.ClientCnxn [INFO] Socket connection established to test-1/10.142.0.9:2181, initiating session
2016-10-26 18:57:27 o.a.z.ClientCnxn [INFO] Session establishment complete on server test-1/10.142.0.9:2181, sessionid = 0x157fc2e050f0008, negotiated timeout =$
2016-10-26 18:57:27 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2016-10-26 18:57:27 o.a.z.ClientCnxn [INFO] EventThread shut down
2016-10-26 18:57:27 o.a.z.ZooKeeper [INFO] Session: 0x157fc2e050f0008 closed
2016-10-26 18:57:27 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2016-10-26 18:57:27 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=test-1:2181/storm sessionTimeout=20000 watcher=com.netflix.curator.Connec$
2016-10-26 18:57:27 o.a.z.ClientCnxn [INFO] Opening socket connection to server test-1/10.142.0.9:2181
2016-10-26 18:57:27 o.a.z.ClientCnxn [INFO] Socket connection established to test-1/10.142.0.9:2181, initiating session
2016-10-26 18:57:27 o.a.z.ClientCnxn [INFO] Session establishment complete on server test-1/10.142.0.9:2181, sessionid = 0x157fc2e050f0009, negotiated timeout =$
2016-10-26 18:57:27 b.s.d.nimbus [INFO] Starting Nimbus server...
This shows that my nimbus instance connects with the zookeeper. However, I am stumped as to how to access the UI. I try to access it using:
http://[Public IP of Nimbus GCP Instance]:8080
I would really appreciate any help. I haven't come across anything with respect to building storm over google compute engine.I'm a relative new comer to Storm.
Thanks
Edit:
gcloud compute firewall-rules list | grep 8080
nimbusui default 0.0.0.0/0 udp:8080,tcp:8080
On the nimbus instance, I tried:
netstat -l 8080
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:ssh *:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
udp 0 0 *:42924 *:*
udp 0 0 *:bootpc *:*
udp 0 0 nimbus.c.ecg-sandbo:ntp *:*
udp 0 0 localhost:ntp *:*
udp 0 0 *:ntp *:*
udp6 0 0 [::]:14994 [::]:*
udp6 0 0 fe80::4001:aff:fe8e:ntp [::]:*
udp6 0 0 ip6-localhost:ntp [::]:*
udp6 0 0 [::]:ntp [::]:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 10687 /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 11147 /var/run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 8751 #/com/ubuntu/upstart
unix 2 [ ACC ] SEQPACKET LISTENING 9447 /run/udev/control
I also checked ui.log :
2016-10-27 18:14:52 o.m.log [INFO] Logging to Logger[org.mortbay.log] via org.mortbay.log.Slf4jLog
2016-10-27 18:14:52 o.m.log [INFO] jetty-6.1.26
2016-10-27 18:14:52 o.m.log [INFO] Started SocketConnector#0.0.0.0:8080

Related

DBGp proxy error Handler response error: Error reading command: Don't understand command '[507]'

Remote debugging using Xdebug DBGp proxy and PhpStorm:
Ubuntu 22.10
PHP 8.1
Xdebug v3.1.2
Xdebug DBGp proxy (0.4.2-dev)
./dbgpProxy
Xdebug DBGp proxy (0.4.2-dev)
Copyright 2020 by Derick Rethans
2023-02-13 07:16:48.394 [warn] [SSL] The 'certs/fullchain.pem' file could not be found, not enabling SSL listeners
2023-02-13 07:16:48.394 [info] [dbgpProxy] Proxy started
2023-02-13 07:16:48.395 [info] [server] Started server server on 127.0.0.1:9003
2023-02-13 07:16:48.395 [info] [server] Started client server on 127.0.0.1:9001
Xdebug settings:
zend_extension=xdebug.so
xdebug.mode=debug
xdebug.client_host=127.0.0.1
xdebug.client_port=9003
xdebug.log=/tmp/xdebug.log
SSH settings:
RemoteForward 9001 localhost:9001
LocalForward 9001 localhost:9001
Tools -> DBGp proxy -> Register IDE
2023-02-13 07:17:08.768 [info] [server] Start new client connection from 127.0.0.1:33682
2023-02-13 07:17:08.839 [info] [proxyinit] [POLYANIN] Added connection for IDE Key 'POLYANIN': 127.0.0.1:9001
2023-02-13 07:17:08.840 [info] [server] Closing client connection from 127.0.0.1:33682
The error:
2023-02-13 07:28:18.141 [info] [proxy-client] [POLYANIN] Found connection for IDE Key 'POLYANIN': 127.0.0.1:9001
2023-02-13 07:28:18.141 [info] [proxy-client] [POLYANIN] Connecting to 127.0.0.1:9001
2023-02-13 07:28:18.142 [info] [proxy-client] [POLYANIN] IDE connected
2023-02-13 07:28:18.142 [info] [server] Start new client connection from 127.0.0.1:33694
2023-02-13 07:28:18.142 [info] [proxy-client] [POLYANIN] Init forwarded, start pipe
2023-02-13 07:28:18.142 [warn] [server] Handler response error: Error reading command: Don't understand command '[507]'
2023-02-13 07:28:18.142 [info] [server] Closing client connection from 127.0.0.1:33694
2023-02-13 07:28:18.192 [info] [proxy-client] [POLYANIN] IDE closed connection
I put a breakpoint and refresh the page.
You cannot use the DBGp proxy this way, as it always sends the debug data to the host where the IDE registration with the key came from.
Since you are using a tunnel, it always comes from 127.0.0.1; this is why you get Found connection for IDE Key 'POLYANIN': 127.0.0.1:9001, and this is why you get the error message - the proxy tries to send the debug data to itself instead of PhpStorm.
There is an extremely ugly workaround - you can modify the proxy code so that it uses the IDE key as a port number to connect to:
https://intellij-support.jetbrains.com/hc/en-us/community/posts/360010264920-xdebug-multiple-developers-php-storm-dbgpproxy

Running fastapi app using uvicorn on ubuntu server

I am dealing with the project deposition made on FastAPI to a remote ubuntu server. I'll try to run the project from terminal (using SSH connection) by the command
gunicorn -k uvicorn.workers.UvicornWorker main:app
The output is
gunicorn -k uvicorn.workers.UvicornWorker main:app
[2020-07-14 15:24:28 +0000] [23102] [INFO] Starting gunicorn 20.0.4
[2020-07-14 15:24:28 +0000] [23102] [INFO] Listening at: http://127.0.0.1:8000 (23102)
[2020-07-14 15:24:28 +0000] [23102] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2020-07-14 15:24:28 +0000] [23104] [INFO] Booting worker with pid: 23104
[2020-07-14 15:24:28 +0000] [23104] [INFO] Started server process [23104]
[2020-07-14 15:24:28 +0000] [23104] [INFO] Waiting for application startup.
[2020-07-14 15:24:28 +0000] [23104] [INFO] Application startup complete.
But I need the project to be available at the IP address of the server. If I try smth like
uvicorn main:app --host 66.226.247.55 --port 8000
I get
INFO: Started server process [23308]
INFO: Waiting for application startup.
INFO: Connected to database postgresql://recognition:********#localhost:5432/reco
INFO: Application startup complete.
ERROR: [Errno 99] error while attempting to bind on address ('66.226.247.55', 8000): cannot assign requested address
INFO: Waiting for application shutdown.
INFO: Disconnected from database postgresql://recognition:********#localhost:5432/reco
INFO: Application shutdown complete.
Where 66.226.247.55 - external IP adress from google cloud platform instances
How do I start a project so that it can be accessed via IP?
The --host should be the local address of your GCP server.
uvicorn main:app --host 0.0.0.0 --port 8000
and now access the application by http://66.226.247.55:8000
Note: You should open your 8000 port of GCP server.
If you're using nginx server
create a file in /etc/nginx/sites-enabled/
create file touch fastapi_nginx
copy code into file and adjust accordingly
server{
listen 80;
server_name "your public ip";
location / {
proxy_pass http://127.0.0.1:8000; #localhost
}
}
This should reroute to your public ip
You cannot launch your fast api app on your local to your remote server in GCP.
You must deploy your app to GCP. In other words you need to run that command on a remote server not your localhost.

Ejabberd does not work on MAC with [Failed to open socket at [::]:5222]

I am facing errors at the first cup of Ejabberd.
On my Mac(10.13.6) I installed "ejabberd-18.12.1-osx.app" and I have followed all instruction written in official website. (https://docs.ejabberd.im/admin/installation/#install-on-macos)
After installation was completed I noticed nothing significant and found error logs were generated as below.
2019-01-16 10:02:03.936 [error] <0.316.0>#ejabberd_listener:report_socket_error:417 Failed to open socket at [::]:5222 for ejabberd_c2s: address already in use
2019-01-16 10:02:03.937 [error] <0.315.0> Supervisor ejabberd_listener had child {5222,{0,0,0,0,0,0,0,0},tcp} started with ejabberd_listener:start({5222,{0,0,0,0,0,0,0,0},tcp}, ejabberd_c2s, [{ip,{0,0,0,0,0,0,0,0}},{max_stanza_size,262144},{shaper,c2s_shaper},{access,c2s},{starttls_required,...}]) at undefined exit with reason eaddrinuse in context start_error
2019-01-16 10:02:03.937 [error] <0.274.0> Supervisor ejabberd_sup had child ejabberd_listener started with ejabberd_listener:start_link() at undefined exit with reason {shutdown,{failed_to_start_child,{5222,{0,0,0,0,0,0,0,0},tcp},eaddrinuse}} in context start_error
2019-01-16 10:02:03.942 [critical] <0.81.0>#ejabberd_app:start:66 Failed to start ejabberd application: {error,{shutdown,{failed_to_start_child,ejabberd_listener,{shutdown,{failed_to_start_child,{5222,{0,0,0,0,0,0,0,0},tcp},eaddrinuse}}}}}
I exactly understand what "address already in use" means but netstat does not show any possession on this port. Also I never changed any of the server configuration. I tried to start server manually but same errors repeats.
Does this version of Ejabberd have bugs on Mac installation?
Many thanks in advance.
When ejabberd starts, it uses several ports (some for XMPP, others for additional ejabberd features, others for Erlang). Notice that some ports may be in IPv6:
tcp 0 0 0.0.0.0:42859 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:4560 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:epmd 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:5280 0.0.0.0:* LISTEN
tcp6 0 0 [::]:epmd [::]:* LISTEN
tcp6 0 0 [::]:xmpp-client [::]:* LISTEN
tcp6 0 0 [::]:xmpp-server [::]:* LISTEN
Maybe you have other previous ejabberd installation around there messing? Or other XMPP server?

Mysql Cluster data ndoe can not start to connecto management node

Now, every config is ok , but when I use
./ndbd --initial
on data node , it will ouput log :
ndb_mgm :
Forced node shutdown completed. Occured during startphase 0. Caused by error 2350: 'Invalid configuration received from Management Server(Configuration error). Permanent error, external action needed'.
ndbd :
2018-05-10 13:54:43 [ndbd] INFO -- Angel pid: 14533 started child: 14534
2018-05-10 13:54:43 [ndbd] INFO -- Initial start of data node, ignoring any info on disk
2018-05-10 13:54:43 [ndbd] INFO -- Configuration fetched from '172.19.16.170:1186', generation: 1
2018-05-10 13:54:43 [ndbd] INFO -- Changing directory to '/var/lib/mysql-cluster'
2018-05-10 13:54:43 [ndbd] INFO -- Invalid configuration fetched
2018-05-10 13:54:43 [ndbd] INFO -- ConfigParam: 113 not found
2018-05-10 13:54:43 [ndbd] INFO -- Error handler shutting down system
2018-05-10 13:54:43 [ndbd] INFO -- Error handler shutdown completed - exiting
2018-05-10 13:54:43 [ndbd] ALERT -- Node 2: Forced node shutdown completed. Occured during startphase 0. Caused by error 2350: 'Invalid configuration received from Management Server(Configuration error). Permanent error, external action needed'.
Now, the ndbd log tell me , it fetch configuration from server error, but I dont know where is the error.
Thsi is my management node configuration :
[ndbd default]
NoOfReplicas= 1
[mysqld default]
[ndb_mgmd default]
[tcp default]
[ndb_mgmd]
HostName= 172.19.16.170
[ndbd]
NodeId=2
HostName= 172.19.16.166
DataDir= /var/lib/mysql-cluster
[ndbd]
NodeId=3
HostName= 172.19.16.167
DataDir= /var/lib/mysql-cluster
[mysqld]
[mysqld]
[mysqld]
This is an upgrade issue. The error only occurs in ndbd up to
version 7.5. But it is only the ndb_mgmd from version 7.6 that
can set IndexMemory to 0. ConfigParam: 113 is IndexMemory which
is deprecated in 7.6. So probably you are running a ndb_mgmd from
version 7.6 and ndbd from version 7.5.

Unable to open up SonarQube page

I am installing SonarQube on my Ubuntu Server with the help of this link. When i start/restart Sonar from terminal it shows like this:
But when i try to open sonar page ipaddress:9000/sonar the page is not at all opening.
But i followed same link for installation on my ubuntu VM, it worked as expected.
after netstat -ntl got following output
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:32000 0.0.0.0:* LISTEN
tcp6 0 0 :::36870 :::* LISTEN
tcp6 0 0 :::9000 :::* LISTEN
tcp6 0 0 127.0.0.1:9001 :::* LISTEN
tcp6 0 0 :::8080 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 :::35866 :::* LISTEN
This is my sonarQube property file which i used for installation.
Following logs which i got after restarting SonarQube
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
2017.06.06 23:14:50 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonar/temp
2017.06.06 23:14:50 INFO app[][o.s.a.p.JavaProcessLauncherImpl] Launch process[es]: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Djava.awt.headless=true -Xmx1G -Xms256m -Xss256k -Djna.nosys=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Des.network.host=127.0.0.1 -Djava.io.tmpdir=/opt/sonar/temp -cp ./lib/common/*:./lib/search/* org.sonar.search.SearchServer /opt/sonar/temp/sq-process4114582605652920236properties
2017.06.06 23:15:00 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2017.06.06 23:15:00 INFO app[][o.s.a.p.JavaProcessLauncherImpl] Launch process[web]: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/sonar/temp -cp ./lib/common/*:./lib/server/*:/opt/sonar/lib/jdbc/mysql/mysql-connector-java-5.1.42.jar org.sonar.server.app.WebServer /opt/sonar/temp/sq-process3191822142101588969properties
2017.06.06 23:15:04 INFO app[][o.s.a.SchedulerImpl] Process [web] is stopped
2017.06.06 23:15:04 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped
2017.06.06 23:15:04 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
<-- Wrapper Stopped
After installing Ubuntu 16 Desktop edition the problem got solved. I don't the specific reason why it was giving error, but after installing Desktop version, without changing anything it got solved.