Kubernetes php container can't seem to connect to mysql service - mysql

Question

Strange one, i have a php container with symfony, and a mysql pod that is attached to a service called mysql-service. I send the mysql connection details in the env variables on the php deployment.
Weirdly when symfony can't connect to the mysql pod using the service name.When i describe the php pod it says:
An exception occurred in driver: SQLSTATE[HY000] [2002] Connection refused
I can see it's the name resolution:
getaddrinfo failed: Temporary failure in name resolution
If i change the deployment config so the DB_HOST env variable is that of another mysql server on the local network it connects just fine, and i know the mysql user and pass are correct as the mysql deployment and the php deployment both use the mysql secret file; i have also logged into the shell of the mysql pod and connected the database with the same user and pass no problem.
It looks like it's something about the mysql service itself.
Php deployment (some of it's removed as not relevant to my problem):
containers:
- name: php
lifecycle:
postStart:
exec:
command: ["/bin/bash", "-c", "cd /usr/share/nginx/html && php bin/console cache:clear"]
env:
- name: APP_ENV
value: "prod"
- name: DB_NAME
valueFrom:
secretKeyRef:
name: mysql-secret
key: database
- name: DB_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: username
- name: DB_PASS
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
- name: DB_HOST
value: "mysql-service"
- name: DB_PORT
value: "3306"
imagePullPolicy: Always
image: php-image-here
ports:
- containerPort: 9000
the mysql deployment and service:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql-deployment
labels:
deploy: mysql
spec:
replicas: 1
selector:
matchLabels:
deploy: mysql
template:
metadata:
labels:
deploy: mysql
spec:
containers:
- name: mysql
imagePullPolicy: Always
image: mysql-image-here
ports:
- containerPort: 3306
volumeMounts:
- mountPath: /var/lib/mysql
name: mysql-volume
env:
- name: MYSQL_RANDOM_ROOT_PASSWORD
value: "yes"
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: mysql-secret
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
name: mysql-secret
key: database
volumes:
- name: mysql-volume
hostPath:
path: /data/mysql
type: DirectoryOrCreate
---
apiVersion: v1
kind: Service
metadata:
name: mysql-service
spec:
selector:
deploy: mysql
ports:
- protocol: TCP
port: 3306
targetPort: 3306
The service and the mysql pod are up and running:
NAME READY STATUS RESTARTS AGE
pod/backend-deployment-7d585fd8fd-9z5dp 1/2 CrashLoopBackOff 5 (2m54s ago) 7m9s
pod/mysql-deployment-7cb7999d98-blpzr 1/1 Running 0 50m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/backend-service NodePort 10.97.250.92 <none> 80:30002/TCP 38m
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d18h
service/mysql-service ClusterIP 10.111.226.120 <none> 3306/TCP 50m