Apache Mesos,MESOS-DNS, MARATHON and Docker - mysql

In my environment running mesos-slave, mesos-master marathon and mesos-dns in standalone mode.
I deployed mysql app to marathon to run as docker container.
MySql app configurations as follows.
{
"id": "mysql",
"cpus": 0.5,
"mem": 512,
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "mysql:5.6.27",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 3306,
"hostPort": 32000,
"protocol": "tcp"
}
]
}
},
"constraints": [
[
"hostname",
"UNIQUE"
]],
"env": {
"MYSQL_ROOT_PASSWORD": "password"
},
"minimumHealthCapacity" :0,
"maximumOverCapacity" : 0.0
}
Then I deploy app called mysql client. Mysql client app needs to connect to mysql app.
mysql app config as follows.
{
"id": "mysqlclient",
"cpus": 0.3,
"mem": 512.0,
"cmd": "/scripts/create_mysql_dbs.sh",
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "mysqlclient:latest",
"network": "BRIDGE",
"portMappings": [{
"containerPort": 3306,
"hostPort": 0,
"protocol": "tcp"
}]
}
},
"env": {
"MYSQL_ENV_MYSQL_ROOT_PASSWORD": "password",
"MYSQL_PORT_3306_TCP_ADDR": "mysql.marathon.slave.mesos.",
"MYSQL_PORT_3306_TCP_PORT": "32000"
},
"minimumHealthCapacity" :0,
"maximumOverCapacity" : 0.0
}
My mesos-dns config.json. as follows
{
"zk": "zk://127.0.0.1:2181/mesos",
"masters": ["127.0.0.1:5050"],
"refreshSeconds": 60,
"ttl": 60,
"domain": "mesos",
"port": 53,
"resolvers": ["127.0.0.1"],
"timeout": 5,
"httpon": true,
"dnson": true,
"httpport": 8123,
"externalon": true,
"listener": "127.0.0.1",
"SOAMname": "ns1.mesos",
"SOARname": "root.ns1.mesos",
"SOARefresh": 60,
"SOARetry": 600,
"SOAExpire": 86400,
"SOAMinttl": 60,
"IPSources": ["mesos", "host"]
}
I can ping with service name mysql.marathon.slave.mesos. from host machine. But when I try to ping from mysql docker container I get host unreachable. Why docker container cannot resolve hsot name?
I tried with set dns parameter to apps. But its not work.
EDIT:
I can ping mysql.marathon.slave.mesos. from master/slave hosts. But I cannot ping from mysqlclient docker container. It says unreachable. How can I fix this?

Not sure what your actual question is, by guessing I think you want to know how you can resolve a Mesos DNS service name to an actual endpoint the MySQL client.
If so, you can use my mesosdns-resolver bash script to get the endpoint from Mesos DNS:
mesosdns-resolver.sh -sn mysql.marathon.mesos -s <IP_ADDRESS_OF_MESOS_DNS_SERVER>
You can use this in your create_mysql_dbs.sh script (whatever it does) to get the actual IP address and port where your mysql app is running.
You can pass in an environment variable like
"MYSQL_ENV_SERVICE_NAME": "mysql.marathon.mesos"
and then use it like this in the image/script
mesosdns-resolver.sh -sn $MYSQL_ENV_SERVICE_NAME -s <IP_ADDRESS_OF_MESOS_DNS_SERVER>
Also, please note that Marathon is not necessarily the right tool for running one-off operations (I assume you initialize your DBs with the second app). Chronos would be a better choice for this.

Related

Why is my docker networks gateway IP unaccessible from my host OS

I am trying to get the following set up working:
My local machine OS = Linux
I am building a docker mysql container on this local machine
I plan to seed the database within the container, and then run tests locally (on my local Linux machine) against this container (which i will spin up on my linux machine too)
Unfortunately when running my tests and trying to connect to the container, the default bridge networks Gateway IP is inaccessible.
My docker-compose.yaml file is as follows
version: "3.4"
services:
integration-test-mysql:
image: mysql:8.0
container_name: ${MY_SQL_CONTAINER_NAME}
environment:
- MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD
ports:
- "3306:3306"
volumes:
# - ./src/db:/usr/src/db #Mount db folder so we can run seed files etc.
- ./seed.sql:/docker-entrypoint-initdb.d/seed.sql
network_mode: bridge
healthcheck:
test: "mysqladmin -u root -p$MYSQL_ROOT_PASSWORD -h 127.0.0.1 ping --silent 2> /dev/null || exit 1"
interval: 5s
timeout: 30s
retries: 5
start_period: 10s
entrypoint: sh -c "
echo 'CREATE SCHEMA IF NOT EXISTS gigs;' > /docker-entrypoint-initdb.d/init.sql;
/usr/local/bin/docker-entrypoint.sh --default-authentication-plugin=mysql_native_password
"
When running docker network ls i see the following
docker network ls
NETWORK ID NAME DRIVER SCOPE
42a11ef835dd bridge bridge local
c7453acfbc98 host host local
48572c69755a integration_default bridge local
bd470f8620fd none null local
So the integration_default network was created. Then if i inspect this network
docker network inspect integration_default
[
{
"Name": "integration_default",
"Id": "48572c69755ae1bbc1448ab203a01d81be4300da12c97a9c4f1142872b878387",
"Created": "2022-09-28T00:48:20.504251612Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.27.0.0/16",
"Gateway": "172.27.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"79e897decb4f0ae5836c018d82e78997e8ac2f615b399362a307cc7f585c0875": {
"Name": "integration-test-mysql-host",
"EndpointID": "1f7798554029cc2d07f7ba44d057c489b678eac918f7916029798b42585eda41",
"MacAddress": "02:42:ac:1b:00:02",
"IPv4Address": "172.27.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "default",
"com.docker.compose.project": "integration",
"com.docker.compose.version": "2.7.0"
}
}
]
Comparing this to the default bridge
docker inspect bridge
[
{
"Name": "bridge",
"Id": "42a11ef835dd1b2aec3ecea57211bb2753e0ebd4a2a115ace8b7df3075e97d5a",
"Created": "2022-09-27T21:54:44.239215269Z",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.17.0.0/16",
"Gateway": "172.17.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {
"com.docker.network.bridge.default_bridge": "true",
"com.docker.network.bridge.enable_icc": "true",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
"com.docker.network.bridge.name": "docker0",
"com.docker.network.driver.mtu": "1500"
},
"Labels": {}
}
]
Interestingly running ping 172.17.0.1 on my Linux machine works fine but ping 172.27.0.1 fails to return anything
UPDATE
I have got it working now. By specifying network_mode: bridge in my docker compose file i was able to use the default bridge network which was accessible on my local machine as i mentioned.
However, i would like to know why creating my own network didn't work here. Does anyone know why this was the case?
Docker networks are meant to be hidden and you should let docker do its job unless there is a good reason for it.
The correct way to interract with a service is through its open ports. And those ports are mapped on the host so that talking to the host:port is like talking to the app inside the container.
So when you say that you can't ping your container from the host, it is because Docker does its job good. "Fixing" this breaks the isolation of the container and makes it available to other services that shouldn't have acccess to it.

DCOS connection refused on marathon-lb

I have dcos up and running. I created a service and i am able to access it through the ip:port but when i try to do the same with marathon-lb i just cant reach it. I tried curl http://marathon-lb.marathon.mesos:10000/ 10000 being the port number, i still get connection refused.
Here is my json for service:
{
"id": "/nginx-external",
"cmd": null,
"cpus": 0.1,
"mem": 65,
"disk": 0,
"instances": 1,
"acceptedResourceRoles": [],
"container": {
"type": "DOCKER",
"volumes": [],
"docker": {
"image": "nginx:1.7.7",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 80,
"hostPort": 2000,
"servicePort": 10000,
"protocol": "tcp",
"labels": {}
}
],
"privileged": false,
"parameters": [],
"forcePullImage": true
}
},
"healthChecks": [
{
"gracePeriodSeconds": 10,
"intervalSeconds": 2,
"timeoutSeconds": 10,
"maxConsecutiveFailures": 10,
"portIndex": 0,
"path": "/",
"protocol": "HTTP",
"ignoreHttp1xx": false
}
],
"labels": {
"HAPROXY_GROUP": "external"
},
"portDefinitions": [
{
"port": 10000,
"protocol": "tcp",
"name": "default",
"labels": {}
}
]
}
Can anyone help.
Both accessing it from outside the cluster by using public-ip:10000 (see here for finding the public ip) and from inside the cluster using curl http://marathon-lb.marathon.mesos:10000/ worked fine. Note, you need to have marathon-lb installed (dcos package install marathon-lb) and marathon-lb.marathon.mesos can only be resolved from inside the cluster.
In order to debug marathon-lb issues I ususally check the haproxy stats first: https://dcos.io/docs/1.9/networking/marathon-lb/marathon-lb-advanced-tutorial/#deploy-an-external-load-balancer-with-marathon-lb
From outside the cluster
From inside the cluster
core#ip-10-0-4-343 ~ $ curl http://marathon-lb.marathon.mesos:10000/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
nginx.org.<br/>
Commercial support is available at
nginx.com.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

CannotStartContainerError while submitting a AWS Batch Job

In AWS Batch I have a job definition and a job queue and a compute environment where to execute my AWS Batch jobs.
After submitting a job, I find it in the list of the failed ones with this error:
Status reason
Essential container in task exited
Container message
CannotStartContainerError: API error (404): oci runtime error: container_linux.go:247: starting container process caused "exec: \"/var/application/script.sh --file= --key=.
and in the cloudwatch logs I have:
container_linux.go:247: starting container process caused "exec: \"/var/application/script.sh --file=Toulouse.json --key=out\": stat /var/application/script.sh --file=Toulouse.json --key=out: no such file or directory"
I have specified a correct docker image that has all the scripts (we use it already and it works) and I don't know where the error is coming from.
Any suggestions are very appreciated.
The docker file is something like that:
# Pull base image.
FROM account-id.dkr.ecr.region.amazonaws.com/application-image.base-php7-image:latest
VOLUME /tmp
VOLUME /mount-point
RUN chown -R ubuntu:ubuntu /var/application
# Create the source directories
USER ubuntu
COPY application/ /var/application
# Register aws profile
COPY data/aws /home/ubuntu/.aws
WORKDIR /var/application/
ENV COMPOSER_CACHE_DIR /tmp
RUN composer update -o && \
rm -Rf /tmp/*
Here is the Job Definition:
{
"jobDefinitionName": "JobDefinition",
"jobDefinitionArn": "arn:aws:batch:region:accountid:job-definition/JobDefinition:25",
"revision": 21,
"status": "ACTIVE",
"type": "container",
"parameters": {},
"retryStrategy": {
"attempts": 1
},
"containerProperties": {
"image": "account-id.dkr.ecr.region.amazonaws.com/application-dev:latest",
"vcpus": 1,
"memory": 512,
"command": [
"/var/application/script.sh",
"--file=",
"Ref::file",
"--key=",
"Ref::key"
],
"volumes": [
{
"host": {
"sourcePath": "/mount-point"
},
"name": "logs"
},
{
"host": {
"sourcePath": "/var/log/php/errors.log"
},
"name": "php-errors-log"
},
{
"host": {
"sourcePath": "/tmp/"
},
"name": "tmp"
}
],
"environment": [
{
"name": "APP_ENV",
"value": "dev"
}
],
"mountPoints": [
{
"containerPath": "/tmp/",
"readOnly": false,
"sourceVolume": "tmp"
},
{
"containerPath": "/var/log/php/errors.log",
"readOnly": false,
"sourceVolume": "php-errors-log"
},
{
"containerPath": "/mount-point",
"readOnly": false,
"sourceVolume": "logs"
}
],
"ulimits": []
}
}
In Cloudwatch log stream /var/log/docker:
time="2017-06-09T12:23:21.014547063Z" level=error msg="Handler for GET /v1.17/containers/4150933a38d4f162ba402a3edd8b7763c6bbbd417fcce232964e4a79c2286f67/json returned error: No such container: 4150933a38d4f162ba402a3edd8b7763c6bbbd417fcce232964e4a79c2286f67"
This error was because the command was malformed. I was submitting the job by a lambda function (python 2.7) using boto3 and the syntax of the command should be something like this:
'command' : ['sudo','mkdir','directory']
Hope it helps somebody.

Cluster communication and firewalls in Google Container Engine

I'm trying to set up the following environment on Google Cloud and have 3 major problems with it:
Database Cluster
3 nodes
one port open to world, a few ports open to the compute cluster
Compute Cluster
- 5 nodes
- communicated with the database cluster
- two ports open to the world
- runs Docker containers
a) The database cluster runs fine, I have the configuration port open to world, but I don't know how to limit the other ports to only the compute cluster?
I managed to get the first Pod and Replication-Controller running on the compute cluster and created a service to open the container to the world:
controller:
{
"id": "api-controller",
"kind": "ReplicationController",
"apiVersion": "v1beta1",
"desiredState": {
"replicas": 2,
"replicaSelector": {
"name": "api"
},
"podTemplate": {
"desiredState": {
"manifest": {
"version": "v1beta1",
"id": "apiController",
"containers": [{
"name": "api",
"image": "gcr.io/my/api",
"ports": [{
"name": "api",
"containerPort": 3000
}]
}]
}
},
"labels": {
"name": "api"
}
}
}
}
service:
{
"id": "api-service",
"kind": "Service",
"apiVersion": "v1beta1",
"selector": {
"name": "api"
},
"containerPort": "api",
"protocol": "TCP",
"port": 80,
"selector": { "name": "api" },
"createExternalLoadBalancer": true
}
b) The container exposes port 3000, the service port 80. Where's the connection between the two?
The firewall works with labels. I want 4-5 different pods running in my compute cluster with 2 of them having open ports to the world. There can be 2 or more containers running on the same instance. The labels however are specific to the nodes, not the containers.
c) Do I expose all nodes with the same firewall configuration? I can't assign labels to containers, so not sure how to expose the api service for example?
I'll try my best to answer all of your questions as best I can.
First off, you will want to upgrade to using v1 of the Kubernetes API because v1beta1 and v1beta3 will no longer be available after Aug. 5th:
https://cloud.google.com/container-engine/docs/v1-upgrade
Also, Use YAML. It's so much less verbose ;)
--
Now on to the questions you asked:
a) I'm not sure I completely understand what you are asking here but it sounds like running the services in the same cluster (with resource limits) would be way easier than trying to deal with cross cluster networking.
b) You need to specify a targetPort so that the service knows what port to use on the container. This should match port 3000 that you have in your resource controller. See the docs for more info.
{
"kind": "Service",
"apiVersion": "v1",
"metadata: {
"labels": [{
"name": "api-service"
}],
},
"spec": {
"selector": {
"name": "api"
},
"ports": [{
"port": 80,
"targetPort": 3000
}]
"type": "LoadBalancer"
}
}
c) Yes. In Kubernetes the kube-proxy accepts traffic on any node and routes it to the appropriate node or local pod. You don't need to worry about mapping the load balancer to, or writing firewall rules for those specific nodes that happen to be running your pods (it could actually change if you do a rolling update!). kube-proxy will route traffic to the right place even if your service is not running on that node.

Deploying Docker containers with port-mapping on Mesos/Marathon

I am currently working on a team project utilizing Docker with Apache Mesos/Marathon. To deploy MySQL docker containers on Mesos/Marathon, we have to create a JSON file with port mapping. I have searched everywhere on the internet and just can't find any sample JSON file to look on for port mapping. Anyone have done this before?
Here's some example Marathon JSON for using Docker's bridged networking mode:
{
"id": "bridged-webapp",
"cmd": "python3 -m http.server 8080",
"cpus": 0.5,
"mem": 64.0,
"instances": 2,
"container": {
"type": "DOCKER",
"docker": {
"image": "python:3",
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 8080, "hostPort": 0, "servicePort": 9000, "protocol": "tcp" },
{ "containerPort": 161, "hostPort": 0, "protocol": "udp"}
]
}
}
}
See the "Bridged Networking Mode" section in
https://mesosphere.github.io/marathon/docs/native-docker.html for more details.