I installed a Cloud Datalab notebook on a Cloud Dataproc cluster following the instructions listed in the official documentation
After creating the cluster, I then created a SSH tunnel to the master node in the Cloud Shell and connected to the cluster interface using the Cloud Shell. Instructions. I could access the Jupyter notebooks after this. I used the -v verbose option to see the SSH connection logs:
gcloud compute ssh cluster-datalab-m --project=abcxyz-123 --zone us-west1-a \
-- -v -4 -N -L 8080:cluster-datalab-m:8080
But after some time, I got a popup saying "A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration." and my cluster stopped responding to any commands.
When I looked at the SSH output on the Cloud Shell, I saw that multiple channels were being requested at this point.
A preview of SSH logs just during the break point:
debug1: channel 1: new [direct-tcpip]
debug1: Connection to port 8080 forwarding to cluster-datalab-m port 8080 requested.
debug1: channel 2: new [direct-tcpip]
debug1: channel 1: free: direct-tcpip: listening port 8080 for cluster-datalab-m port 8080, connect from 127.0.0.1 port 52832 to 127.0.0.1 port 8080, nchannels 3
debug1: channel 2: free: direct-tcpip: listening port 8080 for cluster-datalab-m port 8080, connect from 127.0.0.1 port 52833 to 127.0.0.1 port 8080, nchannels 2
debug1: Connection to port 8080 forwarding to cluster-datalab-m port 8080 requested.
debug1: channel 1: new [direct-tcpip]
debug1: Connection to port 8080 forwarding to cluster-datalab-m port 8080 requested.
debug1: channel 2: new [direct-tcpip]
debug1: channel 2: free: direct-tcpip: listening port 8080 for cluster-datalab-m port 8080, connect from 127.0.0.1 port 52837 to 127.0.0.1 port 8080, nchannels 3
debug1: Connection to port 8080 forwarding to cluster-datalab-m port 8080 requested.
debug1: channel 2: new [direct-tcpip]
debug1: Connection to port 8080 forwarding to cluster-datalab-m port 8080 requested.
debug1: channel 3: new [direct-tcpip]
debug1: Connection to port 8080 forwarding to cluster-datalab-m port 8080 requested.
debug1: channel 4: new [direct-tcpip]
debug1: Connection to port 8080 forwarding to cluster-datalab-m port 8080 requested.
debug1: channel 5: new [direct-tcpip]
debug1: Connection to port 8080 forwarding to cluster-datalab-m port 8080 requested.
debug1: channel 6: new [direct-tcpip]
debug1: Connection to port 8080 forwarding to cluster-datalab-m port 8080 requested.
I closed this SSH connection manually and then tried to SSH into the master node by clicking the SSH button on the Compute Engine console but even that was taking lot of time and didn't complete successfully.
I looked at this stackoverflow question but I couldn't find any /etc/sshguard folder in the master node so I don't think that is the issue for my case. The master node was running Debian 8.10.
Is there any way to ensure that the SSH connection (and the Jupyter notebook) works continuously?
We've updated the documentation at cluster web interfaces. Using cloud shell works for Datalab, but not Jupyter. Cloud Shell Preview only supports HTTP, but Jupyter uses websockets.
Instead, you should follow the instructions for setting up a SOCKS proxy and pointing Chrome at it. There's a handy bash script called launch-jupyter-interface.sh that does that for you. You'll just need to modify it to point to your Chrome installation.
The Jupyter tutorial also mentions using that script.
Related
I have a MongoDB instance on a Google compute engine running that I want to connect to from my Github action (On a windows runner if it makes a difference) to insert test and performance results.
Currently, I am trying to open an SSH tunnel with port forwarding and just test that the port is open.
Here is what my GIthub action step is:
- name: 'Create ssh tunnel'
if: (runner.os == 'Windows')
run: |
gcloud config set auth/impersonate_service_account *****#***.iam.gserviceaccount.com
gcloud compute config-ssh
$sshTunnelJob = Start-Job -Name SshTunnelJob -ScriptBlock { ssh -o "User=*****_iam_gserviceaccount_com" *****.us-east1-b.**** -vvv -fNT -L 27017:0.0.0.0:27017}
Get-Job
Receive-Job -Name SshTunnelJob | Format-List -Force -Expand CoreOnly
netstat -aon
Test-NetConnection localhost -port 27017
gcloud config unset auth/impersonate_service_account
gcloud compute config-ssh --remove
I expect this, Test-NetConnection localhost -port 27017, to succeed, but it fails. Forwarding port 80 is succeeding, though.
Here is the output:
WARNING: TCP connect to (::1 : 27017) failed
WARNING: TCP connect to (127.0.0.1 : 27017) failed
ComputerName: localhost
RemoteAddress: ::1
ResolvedAddresses: {::1, 127.0.0.1}
PingSucceeded: True
PingReplyDetails: System.Net.NetworkInformation.PingReply
TcpClientSocket:
TcpTestSucceeded: False
RemotePort: 27017
TraceRoute:
Detailed: False
InterfaceAlias: Loopback Pseudo-Interface 1
InterfaceIndex: 1
InterfaceDescription:
NetAdapter:
NetRoute: MSFT_NetRoute (InstanceID = "DD;9;?B55;55DD55;")
SourceAddress: ::1
NameResolutionSucceeded: True
BasicNameResolution: {Microsoft.DnsClient.Commands.DnsRecord_AAAA,Microsoft.DnsClient.Commands.DnsRecord_A}
LLMNRNetbiosRecords: {}
DNSOnlyRecords: {Microsoft.DnsClient.Commands.DnsRecord_A}
AllNameResolutionResults: {Microsoft.DnsClient.Commands.DnsRecord_AAAA,Microsoft.DnsClient.Commands.DnsRecord_A}
IsAdmin: True
NetworkIsolationContext: Loopback
MatchingIPsecRules:
What am I missing? Is GitHub limiting ports? I couldn't find any documentation on what ports are blocked or not.
Solution 1 :
The issue might be that the connection from client to server is blocked by a firewall. Can you Please check if the relevant GCP firewall setting is enabled for port 27017.
Also , Please check the target tags and update it accordingly if required . This will allow instances tagged with mongodb-instance to accept connections on port 27017.
Solution 2 :
As per the below output provided by you, it is observed that PingSucceeded was True. Whereas, the response returned as False for the PingSucceeded. In such cases, it is observed that the ICMP requests might be disabled on the remote server/device.
PingSucceeded: True
TcpTestSucceeded: False
As you are expecting Test-NetConnection localhost -port 27017 to succeed,please follow the below steps.
Open PowerShell in the Windows server and type the following command:
tnc <ip_address> -port <PortNumber>
If the device was having issues where it powered off or it got disconnected from the network, a response like below is expected.
PingSucceeded : False
TCPTestSucceeded : False
If the connection is healthy (i.e. MongoDb Server is able to successfully connect) then the following response in PowerShell is expected.
TcpTestSucceeded : True
The above response tells us specifically that the Port 27017 is open and the Test-NetConnection module was able to validate TCP handshake, so the port should be ready to establish a connection.
The above information is derived from the link which was drafted by Rodrigo Restrepo
I ran into a problem exposing a mysql database running inside a kubernetes cluster publicly. The cluster runs with kops on AWS. Im using a helm chart for nginx-ingress: https://github.com/helm/charts/tree/master/stable/nginx-ingress
controller:
config:
use-proxy-protocol: "true"
metrics:
enabled: true
replicaCount: 2
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
stats:
enabled: true
rbac:
create: true
tcp:
5000: default/cbioportal-prod-db-mysql:3306
From within the cluster I can telnet to the db through nginx over port 5000 :
# telnet eating-dingo-nginx-ingress-controller 5000
J
5.7.14
ke_|c&tc"ui%]}mysql_native_passwordConnection closed by foreign host
But i can't seem to connect from outside using the hostname of the aws load balancer.
telnet xxx.us-east-1.elb.amazonaws.com 5000
Trying x.x.x.x...
When i look in aws ec2 dashboard i see the load balancer's security group allows connections from everywhere on port 5000.
UPDATE
I can connect when I use port 3306 instead of 5000:
tcp:
3306: default/cbioportal-prod-db-mysql:3306
However now that the port is open:
$ nmap --verbose -Pn x.x.x.x
PORT STATE SERVICE
21/tcp open ftp
80/tcp open http
443/tcp open https
3306/tcp open mysql
I am getting an authorization issue:
$ mysql -h x.x.x.x -uroot -pabcdef
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 2
I can connect directly to the nginx controller without issues from within the cluster:
kubectl run -it --rm --image=mysql:5.7 --restart=Never mysql-client -- mysql -h eating-dingo-nginx-ingress-controller -uroot -pabcdef
I'm using this mysql helm chart:
https://github.com/helm/charts/tree/master/stable/mysql
I am setting up a reverse tunnel from localhost (win7) to remoteServer (Debian). On localhost I run:
plink.exe -R 12345:localhost:22 username#remoteServer.com.
But when I try to connect back from RemoteServer to localhost I an error:
root#remoteServer:~# ssh -v -p 12345 127.0.0.1
OpenSSH_6.0p1 Debian-4, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 12345.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
ssh_exchange_identification: Connection closed by remote host
root#remoteServer:~#
I have enabled GatewayPorts yes in sshd_config and restarted sshd.
I run cmd.exe with Admin rights.
solved. Needed to install SSH server on localhost (win7). plink.exe is just a client.
I'm using vagrant and I'm using the following config to connect to my database.
SSH Password ist the private_key file in the /.vagrant/machines/default/virtualbox/ folder. This worked without any problems, but since I've updated to El Capitan, I can't connect.
I'm getting those details
Querious was unable to create an SSH connection because the remote host denied permission.
Double-check that the SSH user name and password (or public key, if using key-based authentication) are correct.
OpenSSH_6.9p1, LibreSSL 2.1.7
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 2222.
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tzfrs/work/Server/194.6.226.31/.vagrant/machines/default/virtualbox/private_key type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/tzfrs/work/Server/194.6.226.31/.vagrant/machines/default/virtualbox/private_key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-4+deb7u2
debug1: match: OpenSSH_6.0p1 Debian-4+deb7u2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 127.0.0.1:2222 as 'vagrant'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64#openssh.com none
debug1: kex: client->server aes128-ctr umac-64#openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:kBFNbCLWp1m4X03xMWxWeCjaQUEa426OsJ5IC/PNisM
debug1: Host '[127.0.0.1]:2222' is known and matches the RSA host key.
debug1: Found key in /Users/tzfrs/.ssh/known_hosts:11
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/tzfrs/work/Server/194.6.226.31/.vagrant/machines/default/virtualbox/private_key
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: Device not configured
debug1: permanently_drop_suid: 502
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
As requestes this is the output of vagrant ssh-config
Host default
HostName 127.0.0.1
User vagrant
Port 2222
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentityFile /Users/tzfrs/work/Server/194.6.226.31/.vagrant/machines/default/virtualbox/private_key
IdentitiesOnly yes
LogLevel FATAL
What do I have to do to make it work again?
I had the same problem, connecting with PSequel to the Postgresql database in the Vagrant box. Two variations of the same solution:
Vagrant key
Execute vagrant ssh-config
You'll see a line IdentityFile containing something like /Users/stefbastiaansen/vm/precise-pangolin/.vagrant/machines/default/virtualbox/private_key
Connect to ssh -p 2222 -i /Users/yourusername/vm/precise-pangolin/.vagrant/machines/default/virtualbox/private_key vagrant#127.0.0.1 in a new terminal window
You'll get the notice that the authenticity of the host (127.0.0.1:2222) could not be confirmed, and the question if you're sure you wanted to continue connecting. Answer yes.
Then open PSequel (Postgres) or Sequel Pro (MySQL) and configure it for SSH tunneling.
Host: 127.0.0.1
Port: 2222
User: vagrant
Identity File: /Users/yourusername/vm/precise-pangolin/.vagrant/machines/default/virtualbox/private_key
Connect.
Your own key
Copy your public key to the clipboard: pbcopy < ~/id_rsa.pub
ssh vagrant
Add your public key to ~/.ssh/authorized_keys on the vagrant box
Open up a new terminal window
ssh -p 2222 vagrant#127.0.0.1
You'll get the notice that the authenticity of the host (127.0.0.1:2222) could not be confirmed, and the question if you're sure you wanted to continue connecting. Answer yes.
Then open PSequel (Postgres) or Sequel Pro (MySQL) and configure it for SSH tunneling.
Host: 127.0.0.1
Port: 2222
User: vagrant
Identity File: ~/.ssh/id_rsa
Connect.
It's the confirmation step that needs to happen before you can connect via ssh.
I have been trying to setup Zabbix to monitor my 2 tomcat servers on 2 different Amazon EC2 machines, but in vain.
The Z on the host is green, however te JMX is red with these errors
- ZBX_TCP_READ() failed: [4] Interrupted system call
- Someother error [111] connection refused
and many such errors, one after another, in the sense I resolve an error to see one more new error popping up.
These are some assumptions
All the machines run Ubuntu 12.10 and later
Server's IP address: 66.55.12.120 (Runs Zabbix server v2.2.4 (revision 46772) (23 June 2014) )
Agent's IP address: 87.52.45.198 ( Runs Zabbix agent v2.2.2 (revision 42525) (12 February 2014) )
My local machine's IP address: 76.89.54.111
Here is what I've done so far.
On Server Side:
1) Installed Zabbix_server using sudo apt-get install zabbix-server-mysql.
2) The GUI, mysql database all have been installed and configured.
3) The following are the only 3 changes that I've made in the file /etc/zabbix/zabbix_server.conf
...
JavaGateway=localhost
JavaGatewayPort=10052
StartJavaPollers=5
...
4) The Zabbix Java gateway was installed using sudo apt-get install zabbix-java-gateway.
5) The following are the only 3 changes that I've made in the file
/etc/zabbix/zabbix_java_gateway.conf
...
LISTEN_IP="127.0.0.1"
LISTEN_PORT=10052
START_POLLERS=5
...
On Client Side:
1) Installed Zabbix Client using
sudo apt-get install zabbix-agent
2) The following are the only 3 changes that I've made in the file
/etc/zabbix/zabbix_agentd.conf
...
Server=66.55.12.120
StartAgents=5
ServerActive=66.55.12.120:10051
Hostname=Security-test-JMX-EC2
... <br />
3) The Hostname is the same as the one that is mentioned while creating the Host on the GUI.
I believe that there are some issues with the IP and ports. So, here are the outbound rules for both the machines as obtained from Amazon EC2 Security Groups for the machines
OUTBOUND RULES for SERVER SECURITY GROUP:
Type Protocol Port Source Reasoning
Custom- TCP 8080 0.0.0.0/0
TCP Rule
All ICMP All N/A 0.0.0.0/0
Custom- TCP 10052 27.52.52.128/32 For access from Agent
TCP Rule
Custom- TCP 8081 76.84.120.130/32 To access Zabbix GUI from-
TCP Rule -my local machine's web browser
Custom- TCP 10051 27.52.52.128/32 As the agent responds to-
TCP Rule -the server on Port 10051TCP Rule-
-Must allow inbound communications-
- from the agent.
Custom- TCP 11000 27.52.52.128/32 The agent's JMX reporting-
TCP Rule -happens on port 11000(not on 12345).
OUTBOUND RULES for CLIENT SECURITY GROUP:
Type Protocol Port Source
HTTPS TCP 443 0.0.0.0/0
Custom- TCP 10050 66.55.12.120/32
TCP Rule
Custom- TCP 10052 66.55.12.120/32
TCP Rule
Custom- TCP 11000 66.55.12.120/32
TCP Rule
HTTP TCP 80 76.89.54.111/32
Custom- TCP 8080 76.89.54.111/32
TCP Rule
Custom- TCP 8443 76.89.54.111/32
TCP Rule
What am I missing? Please guide me.
Any help is appreciated.
Thanks
Goutham
If you can, then run VisualVm (probably using a tunneled X session) on the zabbix host, and see if you can connect to the target JVM with that. If you can't connect from that, you won't be able to connect from Zabbix.
Try with the following CATALINA_OPTS, replacing with the IP on the target that you want JMX to listen on:
export CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=falseom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<LOCAL_IP>"
This will disable all JMX security so be aware!
Once you hopefully get it to connect, the "Tomcat JMX" items in Zabbix are also all incorrect! e.g.
Incorrect Zabbix default:
jmx["Catalina:type=GlobalRequestProcessor,name=http-8080",bytesReceived]
Correct entry:
jmx["Catalina:type=ThreadPool,name=\"http-bio-8080\"", bytesReceived]
Note the escaped quotes and incorrect thread name. Add the Mbeans plugin to VisualVM, and use that to browse the MBeans on the target VM, and check the Zabbix names.
It does work eventually, but is a real pain to setup. Zabbix is however one of the few open source monitoring tools that supports JMX at all!
By default, JMX does not work very well with firewalls. You might find related bug reports on Zabbix tracker useful: ZBX-5326 and ZBX-6815. The first one contains a workaround for Tomcat which might work for you.
#gvatreya wrote:
Server: (Runs Zabbix server)
Agent: (Runs Zabbix agent)
It looks like you have to start Zabbix Java gateway as well on host where it is installed (it is a daemon/service).
I configured as follows:
Server: (Runs Zabbix server, Zabbix Java gateway)
Agent: (Runs Zabbix agent)
I think it is possible to install it on a dedicated host.
Have you tried adding -Djava.net.preferIPv4Stack=true to the VM options?
to make it work add next java_opts to your tomcat startup script
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=2345
-Dcom.sun.management.jmxremote.rmi.port=12345
-Djava.rmi.server.hostname=<tomcat_hostname>