Kubernetes service dns resolution returning wrong IP - mysql

I have a simple MYSQL pod sitting behind a MYSQL service.
Additionally I have another pod that is running a python process that is trying to connect to the MYSQL pod.
If I try connecting to the IP address of the MYSQL pod manually from the python pod, everything is A-OK. However if I try connecting to the MYSQL service then I get an error that I can't connect to MYSQL.
grant#grant-Latitude-E7450:~/k8s/objects$ kubectl describe pod mysqlpod
Name: mysqlpod
Namespace: default
Node: minikube/192.168.99.100
Start Time: Fri, 20 Jan 2017 11:10:50 -0600
Labels: <none>
Status: Running
IP: 172.17.0.4
Controllers: <none>
grant#grant-Latitude-E7450:~/k8s/objects$ kubectl describe service mysqlservice
Name: mysqlservice
Namespace: default
Labels: <none>
Selector: db=mysqllike
Type: ClusterIP
IP: None
Port: <unset> 3306/TCP
Endpoints: 172.17.0.5:3306
Session Affinity: None
No events.
grant#grant-Latitude-E7450:~/k8s/objects$ kubectl describe pod basic-python-model
Name: basic-python-model
Namespace: default
Node: minikube/192.168.99.100
Start Time: Fri, 20 Jan 2017 12:01:50 -0600
Labels: db=mysqllike
Status: Running
IP: 172.17.0.5
Controllers: <none>
If I attach to my python container and do an nslookup of the mysqlservice, then I'm actually getting the wrong IP. As you saw above the IP of the mysqlpod is 172.17.0.4 while nslookup mysqlservice is resolving to 172.17.0.5.
grant#grant-Latitude-E7450:~/k8s/objects$ k8s exec -it basic-python-model bash
[root#basic-python-model /]# nslookup mysqlservice
Server: 10.0.0.10
Address: 10.0.0.10#53
Name: mysqlservice.default.svc.cluster.local
Address: 172.17.0.5
I'm fairly new to kubernetes, but I've been banging my head on this issue for a few hours and I can't seem to understand what I'm doing wrong.

So this was the exact correct behavior but I just misconfigured my pods.
For future people who are stuck:
The selector defined in a kubernetes service must match the label of the pod(s) you wish to serve. IE) In my MySqlService.yaml file I have the name selector for "mysqlpod":
apiVersion: v1
kind: Service
metadata:
name: mysqlservice
spec:
clusterIP: None
ports:
- port: 3306
targetPort: 3306
selector:
name: mysqlpod
Thus in my MySqlPod.yaml file I need an exactly matching label.
kind: Pod
apiVersion: v1
metadata:
name: mysqlpod
labels:
name: mysqlpod
spec:
...

For anyone coming again here, please check #gnicholas answer, but also make sure that clusterIP: None is correctly set.
I happened to indent clusterIP: None too much in the .yml file and the command was ignored by Kubernetes therefore clusterIP was mistakenly assigned causing the wrong IP issue.
Be aware that the validation won't throw any error, but will silently ignore it.

Related

mariadb crashes inside kubernetes pod with hostpath volume

I'm trying to move a number of docker containers on a linux server to a test kubernets-based deployment running on a different linux machine where I've installed kubernetes as a k3s instance inside a vagrant virtual machine.
One of these containers is a mariadb container instance, with a bind volume mapped
This is the relevant portion of the docker-compose I'm using:
academy-db:
image: 'docker.io/bitnami/mariadb:10.3-debian-10'
container_name: academy-db
environment:
- ALLOW_EMPTY_PASSWORD=yes
- MARIADB_USER=bn_moodle
- MARIADB_DATABASE=bitnami_moodle
volumes:
- type: bind
source: ./volumes/moodle/mariadb
target: /bitnami/mariadb
ports:
- '3306:3306'
Note that this works correctly. (the container is used by another application container which connects to it and reads data from the db without problems).
I then tried to convert this to a kubernetes configuration, copying the volume folder to the destination machine and using the following kubernetes .yaml deployment files.
This includes a deployment .yaml, a persistent volume claim and a persistent volume, as well as a NodePort service to make the container accessible. For the data volume, I'm using a simple hostPath volume pointing to the contents copied from the docker-compose's bind mounts.
apiVersion: apps/v1
kind: Deployment
metadata:
name: academy-db
spec:
replicas: 1
selector:
matchLabels:
name: academy-db
strategy:
type: Recreate
template:
metadata:
labels:
name: academy-db
spec:
containers:
- env:
- name: ALLOW_EMPTY_PASSWORD
value: "yes"
- name: MARIADB_DATABASE
value: bitnami_moodle
- name: MARIADB_USER
value: bn_moodle
image: docker.io/bitnami/mariadb:10.3-debian-10
name: academy-db
ports:
- containerPort: 3306
resources: {}
volumeMounts:
- mountPath: /bitnami/mariadb
name: academy-db-claim
restartPolicy: Always
volumes:
- name: academy-db-claim
persistentVolumeClaim:
claimName: academy-db-claim
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: academy-db-pv
labels:
type: local
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
hostPath:
path: "<...full path to deployment folder on the server...>/volumes/moodle/mariadb"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: academy-db-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: ""
volumeName: academy-db-pv
---
apiVersion: v1
kind: Service
metadata:
name: academy-db-service
spec:
type: NodePort
ports:
- name: "3306"
port: 3306
targetPort: 3306
selector:
name: academy-db
after applying the deployment, everything seems to work fine, in the sense that with kubectl get ... the pod and the volumes seem to be running correctly
kubectl get pods
NAME READY STATUS RESTARTS AGE
academy-db-5547cdbc5-65k79 1/1 Running 9 15d
.
.
.
kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
academy-db-pv 1Gi RWO Retain Bound default/academy-db-claim 15d
.
.
.
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
academy-db-claim Bound academy-db-pv 1Gi RWO 15d
.
.
.
This is the pod's log:
kubectl logs pod/academy-db-5547cdbc5-65k79
mariadb 10:32:05.66
mariadb 10:32:05.66 Welcome to the Bitnami mariadb container
mariadb 10:32:05.66 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
mariadb 10:32:05.66 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
mariadb 10:32:05.66
mariadb 10:32:05.67 INFO ==> ** Starting MariaDB setup **
mariadb 10:32:05.68 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 10:32:05.68 WARN ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
mariadb 10:32:05.69 INFO ==> Initializing mariadb database
mariadb 10:32:05.69 WARN ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mariadb 10:32:05.70 INFO ==> Using persisted data
mariadb 10:32:05.71 INFO ==> Running mysql_upgrade
mariadb 10:32:05.71 INFO ==> Starting mariadb in background
and the describe pod command:
Name: academy-db-5547cdbc5-65k79
Namespace: default
Priority: 0
Node: zdmp-kube/192.168.33.99
Start Time: Tue, 22 Dec 2020 13:33:43 +0000
Labels: name=academy-db
pod-template-hash=5547cdbc5
Annotations: <none>
Status: Running
IP: 10.42.0.237
IPs:
IP: 10.42.0.237
Controlled By: ReplicaSet/academy-db-5547cdbc5
Containers:
academy-db:
Container ID: containerd://68af105f15a1f503bbae8a83f1b0a38546a84d5e3188029f539b9c50257d2f9a
Image: docker.io/bitnami/mariadb:10.3-debian-10
Image ID: docker.io/bitnami/mariadb#sha256:1d8ca1757baf64758e7f13becc947b9479494128969af5c0abb0ef544bc08815
Port: 3306/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 07 Jan 2021 10:32:05 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 07 Jan 2021 10:22:03 +0000
Finished: Thu, 07 Jan 2021 10:32:05 +0000
Ready: True
Restart Count: 9
Environment:
ALLOW_EMPTY_PASSWORD: yes
MARIADB_DATABASE: bitnami_moodle
MARIADB_USER: bn_moodle
MARIADB_PASSWORD: bitnami
Mounts:
/bitnami/mariadb from academy-db-claim (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-x28jh (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
academy-db-claim:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: academy-db-claim
ReadOnly: false
default-token-x28jh:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-x28jh
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 15d (x8 over 15d) kubelet Container image "docker.io/bitnami/mariadb:10.3-debian-10" already present on machine
Normal Created 15d (x8 over 15d) kubelet Created container academy-db
Normal Started 15d (x8 over 15d) kubelet Started container academy-db
Normal SandboxChanged 18m kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 8m14s (x2 over 18m) kubelet Container image "docker.io/bitnami/mariadb:10.3-debian-10" already present on machine
Normal Created 8m14s (x2 over 18m) kubelet Created container academy-db
Normal Started 8m14s (x2 over 18m) kubelet Started container academy-db
Later, though, I notice that the client application has problems in connecting. After some investigation I've concluded that though the pod is running, the mariadb process running inside it could have crashed just after startup. If I enter the container with kubectl exec and try to run for instance the mysql client I get:
kubectl exec -it pod/academy-db-5547cdbc5-65k79 -- /bin/bash
I have no name!#academy-db-5547cdbc5-65k79:/$ mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)
Any idea of what could cause the problem, or how can I investigate further the issue? (Note: I'm not an expert in Kubernetes, but started only recently to learn it)
Edit: Following #Novo's comment, I tried to delete the volume folder and let mariadb recreate the deployment from scratch.
Now my pod doesn't even start, terminating in CrashLoopBackOff !
By comparing the pod logs I notice that in the previous mariabd log there was a message:
...
mariadb 10:32:05.69 WARN ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mariadb 10:32:05.70 INFO ==> Using persisted data
mariadb 10:32:05.71 INFO ==> Running mysql_upgrade
mariadb 10:32:05.71 INFO ==> Starting mariadb in background
Now replaced with
...
mariadb 14:15:57.32 INFO ==> Updating 'my.cnf' with custom configuration
mariadb 14:15:57.32 INFO ==> Setting user option
mariadb 14:15:57.35 INFO ==> Installing database
Could it be that the issue is related with some access right problem to the volume folders in the host vagrant machine?
By default, hostPath directories are created with permission 755, owned by the user and group of the kubelet. To use the directory, you can try adding the following to your deployment:
spec:
securityContext:
fsGroup: <gid>
Where gid is the group used by the process in your container.
Also, you could fix the issue on the host itself by changing the permissions of the folder you want to mount into the container:
chown-R <uid>:<gid> /path/to/volume
where uid and gid are the userId and groupId from your app.
chmod -R 777 /path/to/volume
This should solve your issue.
But overall, a deployment is not what you want to create in this case, because deployments should not have state. For stateful apps, there are 'StatefulSets' in Kubernetes. Use those together with a 'VolumeClaimTemplate' plus spec.securityContext.fsgroup and k3s will create the persitent volume and the persistent volume claim for you, using it's default storage class, which is local storage (on your node).

Accidentally deleted the glusterfs-dynamic Service for my PVC; how do I recreate it?

Attempting to recreate all my assets from a fresh openshift (except for my PVC), I deleted everything ($ oc delete all --all; oc delete configmap --all; oc delete secret -l namespace=visor). I did this so I could be certain my 'oc process -f template' did a complete job.
This deleted the glusterfs-dynamic Services that I didn't realize were required to mount PVCs (persistent volume claims).
Solution 1: Recreate the Services
I recreated the services so that they look just like similar glusterfs-dynamic services, but that's not enough; even if the IP address matches, the PVC are still not mountable ('endpoints "glusterfs-dynamic-xxx" not found')
Solution 2: Copy data from old PVC to new PVC
To copy, I need to be able to access the PVC from a pod--I can't mount the PVC...
My attempt at recreating the service:
- apiVersion: v1
kind: Service
metadata:
labels:
gluster.kubernetes.io/provisioned-for-pvc: prom-a-pvc
namespace: visor
name: glusterfs-dynamic-615a9bfa-57d9-11e9-b511-001a4a195f6a
namespace: visor
spec:
ports:
- port: 1
protocol: TCP
targetPort: 1
sessionAffinity: None
type: ClusterIP
I want to be able to mount my PVCs.
But instead I get this error:
MountVolume.NewMounter initialization failed for volume "pvc-89647bcb-6df4-11e9-bd79-001a4a195f6a" : endpoints "glusterfs-dynamic-89647bcb-6df4-11e9-bd79-001a4a195f6a" not found
Figured it out. Woot. There was also an "endpoints" asset that got deleted that I had to recreate:
- apiVersion: v1
kind: Service
metadata:
labels:
gluster.kubernetes.io/provisioned-for-pvc: prom-a-pvc
namespace: visor
name: glusterfs-dynamic-615a9bfa-57d9-11e9-b511-001a4a195f6a
namespace: visor
spec:
ports:
- port: 1
protocol: TCP
targetPort: 1
sessionAffinity: None
type: ClusterIP
- apiVersion: v1
kind: Endpoints
metadata:
labels:
gluster.kubernetes.io/provisioned-for-pvc: prom-a-pvc
name: glusterfs-dynamic-615a9bfa-57d9-11e9-b511-001a4a195f6a
namespace: visor
subsets:
- addresses:
- ip: 10.25.6.231
- ip: 10.25.6.232
- ip: 10.27.6.241
- ip: 10.27.6.242
- ip: 10.5.6.221
- ip: 10.5.6.222
ports:
- port: 1
protocol: TCP
I used theses commands to see what I needed to include in my yaml:
$ oc get endpoints
$ oc edit endpoints glusterfs-dynamic-820de1e7-6df6-11e9-bd79-001a4a195f6a
Links:
I'm not alone: https://github.com/heketi/heketi/issues/757
Where I learned the endpoints asset: https://lists.openshift.redhat.com/openshift-archives/users/2019-April/msg00005.html

openshift request fails frequently

I have deployed my webservice in openshift (tomcat) and every time I request my services sometimes it works and sometimes it doesn't work.
it was working perfectly before, number of pod is 1 no logs for failure
Error is
Application is not available
The application is currently not serving requests at this endpoint. It may not have been started or is still starting.
Possible reasons you are seeing this page:
The host doesn't exist. Make sure the hostname was typed correctly and that a route matching this hostname exists.
The host exists, but doesn't have a matching path. Check if the URL path was typed correctly and that the route was created using the desired path.
Route and path matches, but all pods are down. Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
O/P of oc describe routes
Name: mysample
Namespace: enzen
Created: 12 days ago
Labels: app=mysample
Annotations: openshift.io/host.generated=true
Requested Host: mysample-enzen.193b.starter-ca-central-1.openshiftapps.com
exposed on router router (host elb.193b.starter-ca-central-1.openshiftapps.com) 12 days ago
Path: <none>
TLS Termination: <none>
Insecure Policy: <none>
Endpoint Port: 8080-tcp
Service: mysample
Weight: 100 (100%)
Endpoints: 10.128.18.210:8080
O/P of oc describe services
Name: mysample
Namespace: enzen
Labels: app=mysample
Annotations: openshift.io/generated-by=OpenShiftNewApp
Selector: app=mysample,deploymentconfig=mysample
Type: ClusterIP
IP: 172.30.145.245
Port: 8080-tcp 8080/TCP
TargetPort: 8080/TCP
Endpoints: 10.128.18.210:8080
Session Affinity: None
Events: <none>
The initial thought is that route is trying to spread load across multiple services https://docs.openshift.com/container-platform/3.9/architecture/networking/routes.html#alternateBackends, and one of those services is down or not available. Typically in this case I would recreate the service and the route to verify that it's configured as expected. Perhaps you can share the configuration of the route, the service, and the pod?
oc describe routes
oc describe services
oc describe pods
#### EDIT 10-22-18 ####
Adding the output from the google doc with the build pods redacted (as they are not relevant) for the benefit of additional readers. Nothing immediate is jumping out yet as an app/config issue;
oc describe routes
Name: mysample
Namespace: enzen
Created: 12 days ago
Labels: app=mysample
Annotations: openshift.io/host.generated=true
Requested Host: mysample-enzen.193b.starter-ca-central-1.openshiftapps.com
exposed on router router (host elb.193b.starter-ca-central-1.openshiftapps.com) 12 days ago
Path: <none>
TLS Termination: <none>
Insecure Policy: <none>
Endpoint Port: 8080-tcp
Service: mysample
Weight: 100 (100%)
Endpoints: 10.128.18.210:8080
oc describe services
Name: mysample
Namespace: enzen
Labels: app=mysample
Annotations: openshift.io/generated-by=OpenShiftNewApp
Selector: app=mysample,deploymentconfig=mysample
Type: ClusterIP
IP: 172.30.145.245
Port: 8080-tcp 8080/TCP
TargetPort: 8080/TCP
Endpoints: 10.128.18.210:8080
Session Affinity: None
Events: <none>
oc describe pods
Name: mysample-15-z85zt
Namespace: enzen
Priority: 0
PriorityClassName: <none>
Node: ip-172-31-29-189.ca-central-1.compute.internal/172.31.29.189
Start Time: Sun, 21 Oct 2018 20:55:36 +0530
Labels: app=mysample
deployment=mysample-15
deploymentconfig=mysample
Annotations: kubernetes.io/limit-ranger=LimitRanger plugin set: cpu, memory request for container mysample; cpu, memory limit for container mysample
openshift.io/deployment-config.latest-version=15
openshift.io/deployment-config.name=mysample
openshift.io/deployment.name=mysample-15
openshift.io/generated-by=OpenShiftNewApp
openshift.io/scc=restricted
Status: Running
IP: 10.128.18.210
Controlled By: ReplicationController/mysample-15
Containers:
mysample:
Container ID: cri-o://0cd20854571232b310ce22a282c8d5832908533d28d5d720537bbf3618b86c44
Image: docker-registry.default.svc:5000/enzen/mysample#sha256:adadeb7decf82b29699861171c58d7ae5f87ca6eeb1c10e5a1d525e4a0888ebc
Image ID: docker-registry.default.svc:5000/enzen/mysample#sha256:adadeb7decf82b29699861171c58d7ae5f87ca6eeb1c10e5a1d525e4a0888ebc
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 21 Oct 2018 20:55:40 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 512Mi
Requests:
cpu: 20m
memory: 256Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8xjb8 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-8xjb8:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8xjb8
Optional: false
QoS Class: Burstable
Node-Selectors: type=compute
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
Events: <none>

Mysql Communications link failure in kubernetes sample

Step1:finish installing etcd and kubernetes with YUM in CentOS7 and shutdown firewall
Step2:modify related configuration item in /etc/sysconfig/docker
OPTIONS='--selinux-enabled=false --insecure-registry gcr.io'
Step3:modify related configuration item in /etc/kubernetes/apiserver
remove
ServiceAccount
in KUBE_ADMISSION_CONTROL configuration item
Step4:start all the related services of etcd and kubernetes
Step5:start ReplicationController for mysql db
kubectl create -f mysql-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: mysql
spec:
replicas: 1
selector:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- name: mysql
image: hub.c.163.com/library/mysql
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
Step6:start related mysql db service
kubectl create -f mysql-svc.yaml
kind: Service
metadata:
name: mysql
spec:
ports:
- port: 3306
selector:
app: mysql
Step7:start ReplicationController for myweb
kubectl create -f myweb-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: myweb
spec:
replicas: 3
selector:
app: myweb
template:
metadata:
labels:
app: myweb
spec:
containers:
- name: myweb
image: docker.io/kubeguide/tomcat-app:v1
ports:
- containerPort: 8080
env:
- name: MYSQL_SERVICE_HOST
value: "mysql"
- name: MYSQL_SERVICE_PORT
value: "3306"
Step8:start related tomcat service
kubectl create -f myweb-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: myweb
spec:
type: NodePort
ports:
- port: 8080
nodePort: 30001
selector:
app: myweb
When I visit from browser with nodeport(30001),I get the following Exception:
Error:com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure The last packet sent successfully to the
server was 0 milliseconds ago. The driver has not received any packets
from the server.
kubectl get ep
NAME ENDPOINTS AGE
kubernetes 192.168.57.129:6443 1d
mysql 172.17.0.2:3306 1d
myweb 172.17.0.3:8080,172.17.0.4:8080,172.17.0.5:8080 1d
kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.254.0.1 <none> 443/TCP 1d
mysql 10.254.0.5 <none> 3306/TCP 1d
myweb 10.254.220.2 <nodes> 8080:30001/TCP 1d
From the interior of any tomcat container I can see the mysql env and the related mysql link code in JSP is as below:
Class.forName("com.mysql.jdbc.Driver");
String ip=System.getenv("MYSQL_SERVICE_HOST");
String port=System.getenv("MYSQL_SERVICE_PORT");
ip=(ip==null)?"localhost":ip;
port=(port==null)?"3306":port;
System.out.println("Connecting to database...");
conn = java.sql.DriverManager.getConnection("jdbc:mysql://"+ip+":"+port+"?useUnicode=true&characterEncoding=UTF-8", "root","123456");
[root#promote ~]# docker exec -it 1470cfaa1b1c /bin/bash
root#myweb-xswfb:/usr/local/tomcat# env |grep MYSQL_SERVICE
MYSQL_SERVICE_PORT=3306
MYSQL_SERVICE_HOST=mysql
root#myweb-xswfb:/usr/local/tomcat# ping mysql
ping: unknown host
Can someone tell me why I could not ping mysqldb hostname from inner tomcat server?Or how to locate the problem further?
I know the reason, it's the DNS problems. The web server cannot find the IP address of the mysql server. so it failed. Temp solution is change the web server's IP to the mysql db server. Hope can help you. Thank you.
Try to use a Headless Service http://kubernetes.io/v1.0/docs/user-guide/services.html#headless-services
by setting in your mysql Service
clusterIP: None
UPDATE
I have tried your yaml file.
Pods are running:
➜ kb get po
NAME READY STATUS RESTARTS AGE
mysql-ndtxn 1/1 Running 0 7m
myweb-j8xgh 1/1 Running 0 8m
myweb-qc7ws 1/1 Running 0 8m
myweb-zhzll 1/1 Running 0 8m
Services are:
kb get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1h
mysql ClusterIP 10.102.178.190 <none> 3306/TCP 20m
myweb NodePort 10.98.74.113 <none> 8080:30001/TCP 19m
Endpoints are:
kb get ep
NAME ENDPOINTS AGE
kubernetes 10.0.2.15:8443 1h
mysql 172.17.0.7:3306 20m
myweb 172.17.0.2:8080,172.17.0.4:8080,172.17.0.6:8080 19m
I exec bash on a tomcat pod and I can ping my service (it is resolved):
kb exec -ti myweb-zhzll -- bash
root#myweb-zhzll:/usr/local/tomcat# ping mysql
PING mysql.default.svc.cluster.local (10.102.178.190): 56 data bytes
^C--- mysql.default.svc.cluster.local ping statistics ---
I can ping the endpoint:
ping 172.17.0.7
PING 172.17.0.7 (172.17.0.7): 56 data bytes
64 bytes from 172.17.0.7: icmp_seq=0 ttl=64 time=0.181 ms
64 bytes from 172.17.0.7: icmp_seq=1 ttl=64 time=0.105 ms
64 bytes from 172.17.0.7: icmp_seq=2 ttl=64 time=0.119 ms
^C--- 172.17.0.7 ping statistics ---
Connecting to
http://192.168.99.100:30001/
I can see the tomcat page:
UPDATE 2
Here my screenshot... I see data in your database with no error.
I suggest to check your db configuration.
As a beginner, I did the same work with you and got the same problems.
This is my solution,maybe you can have a try:
Delete these configurations in myweb-rc.yaml, because it will override the system default values:
env:
- name: MYSQL_SERVICE_HOST
value: "mysql"
- name: MYSQL_SERVICE_PORT
value: "3306"
Change the mysql image tag in mysql-rc.yaml. use the low version mysql:
image: hub.c.163.com/library/mysql:5.5
Create the service first, then create the pod. refer to the following sequence:
kubectl create -f myweb-svc.yaml
kubectl create -f mysql-svc.yaml
kubectl create -f mysql-rc.yaml
kubectl create -f myweb-rc.yaml
You can refer to this doc:Discovering services
Good luck!

How to setup error reporting in Stackdriver from kubernetes pods?

I'm a bit confused at how to setup error reporting in kubernetes, so errors are visible in Google Cloud Console / Stackdriver "Error Reporting"?
According to documentation
https://cloud.google.com/error-reporting/docs/setting-up-on-compute-engine
we need to enable fluentd' "forward input plugin" and then send exception data from our apps. I think this approach would have worked if we had setup fluentd ourselves, but it's already pre-installed on every node in a pod that just runs gcr.io/google_containers/fluentd-gcp docker image.
How do we enable forward input on those pods and make sure that http port available to every pod on the nodes? We also need to make sure this config is used by default when we add more nodes to our cluster.
Any help would be appreciated, may be I'm looking at all this from a wrong point?
The basic idea is to start a separate pod that receives structured logs over TCP and forwards it to Cloud Logging, similar to a locally-running fluentd agent. See below for the steps I used.
(Unfortunately, the logging support that is built into Docker and Kubernetes cannot be used - it just forwards individual lines of text from stdout/stderr as separate log entries which prevents Error Reporting from seeing complete stack traces.)
Create a docker image for a fluentd forwarder using a Dockerfile as follows:
FROM gcr.io/google_containers/fluentd-gcp:1.18
COPY fluentd-forwarder.conf /etc/google-fluentd/google-fluentd.conf
Where fluentd-forwarder.conf contains the following:
<source>
type forward
port 24224
</source>
<match **>
type google_cloud
buffer_chunk_limit 2M
buffer_queue_limit 24
flush_interval 5s
max_retry_wait 30
disable_retry_limit
</match>
Then build and push the image:
$ docker build -t gcr.io/###your project id###/fluentd-forwarder:v1 .
$ gcloud docker push gcr.io/###your project id###/fluentd-forwarder:v1
You need a replication controller (fluentd-forwarder-controller.yaml):
apiVersion: v1
kind: ReplicationController
metadata:
name: fluentd-forwarder
spec:
replicas: 1
template:
metadata:
name: fluentd-forwarder
labels:
app: fluentd-forwarder
spec:
containers:
- name: fluentd-forwarder
image: gcr.io/###your project id###/fluentd-forwarder:v1
env:
- name: FLUENTD_ARGS
value: -qq
ports:
- containerPort: 24224
You also need a service (fluentd-forwarder-service.yaml):
apiVersion: v1
kind: Service
metadata:
name: fluentd-forwarder
spec:
selector:
app: fluentd-forwarder
ports:
- protocol: TCP
port: 24224
Then create the replication controller and service:
$ kubectl create -f fluentd-forwarder-controller.yaml
$ kubectl create -f fluentd-forwarder-service.yaml
Finally, in your application, instead of using 'localhost' and 24224 to connect to the fluentd agent as described on https://cloud.google.com/error-reporting/docs/setting-up-on-compute-engine, use the values of evironment variables FLUENTD_FORWARDER_SERVICE_HOST and FLUENTD_FORWARDER_SERVICE_PORT.
To add to Boris' answer: As long as errors are logged in the right format (see https://cloud.google.com/error-reporting/docs/troubleshooting) and Cloud Logging is enabled (you can see the errors in https://console.cloud.google.com/logs/viewer) then errors will make it to Error Reporting without any further setup.
Boris' answer was great but was a lot more complicated then it really needed to be (no need to build a docker image). If you have kubectl configured on your local box (or you can use the Google Cloud Shell), copy and paste the following and it will install the forwarder in your cluster (I updated the version of fluent-gcp from the above answer). My solution uses a ConfigMap to store the file so it can be changed easily without rebuilding.
cat << EOF | kubectl create -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-forwarder
data:
google-fluentd.conf: |+
<source>
type forward
port 24224
</source>
<match **>
type google_cloud
buffer_chunk_limit 2M
buffer_queue_limit 24
flush_interval 5s
max_retry_wait 30
disable_retry_limit
</match>
---
apiVersion: v1
kind: ReplicationController
metadata:
name: fluentd-forwarder
spec:
replicas: 1
template:
metadata:
name: fluentd-forwarder
labels:
app: fluentd-forwarder
spec:
containers:
- name: fluentd-forwarder
image: gcr.io/google_containers/fluentd-gcp:2.0.18
env:
- name: FLUENTD_ARGS
value: -qq
ports:
- containerPort: 24224
volumeMounts:
- name: config-vol
mountPath: /etc/google-fluentd
volumes:
- name: config-vol
configMap:
name: fluentd-forwarder
---
apiVersion: v1
kind: Service
metadata:
name: fluentd-forwarder
spec:
selector:
app: fluentd-forwarder
ports:
- protocol: TCP
port: 24224
EOF