I have troubles deploying MySQL image on AWS ECS FARGATE.
The cloudformation script that i have is this (dont mind the syntax, i am using python lib Troposphere to manage cloudfromation templates):
TaskDefinition(
'WordpressDatabaseTaskDefinition',
RequiresCompatibilities=['FARGATE'],
Cpu='512',
Memory='2048',
NetworkMode='awsvpc',
ContainerDefinitions=[
ContainerDefinition(
Name='WordpressDatabaseContainer',
Image='mysql:5.7',
Environment=[
Environment(Name='MYSQL_ROOT_PASSWORD', Value='root'),
Environment(Name='MYSQL_DATABASE', Value='wpdb'),
Environment(Name='MYSQL_USER', Value='root'),
Environment(Name='MYSQL_PASSWORD', Value='root'),
],
PortMappings=[
PortMapping(
ContainerPort=3306
)
]
)
]
)
The deployment succeeds. I can even see that the task is running for few seconds until its state changes to STOPPED.
The only thing that i can see is:
Stopped reason Essential container in task exited
Exit Code 1
On localhost it works like a charm. What am i doing here wrong? At least - are there ways to debug this?
With AWS ECS, if it is stopping, it may be failing a health check which is causing the container to restart. What port is the container DB mapped to and can you check the container logs to see what is happening when it starts then stops? Also, check the logs in ECS under the service or task. Post it here so I can take a look at them.
So, I found out a mistake.
THE VERY FIRST THING YOU DO - is you test that docker container on localhost and see if you can reproduce the issue. In my case docker mysql container on a local machine with the exact same environment crashed too. I was able to inspect logs and found out that it fails to create "root" user. Simply changing user and password made everything work, even on ECS.
This is the complete stack to have a mysql docker image running on AWS ECS FARGATE:
self.wordpress_database_task = TaskDefinition(
'WordpressDatabaseTaskDefinition',
RequiresCompatibilities=['FARGATE'],
Cpu='512',
Memory='2048',
NetworkMode='awsvpc',
# If your tasks are using the Fargate launch type, the host and sourcePath parameters are not supported.
Volumes=[
Volume(
Name='MySqlVolume',
DockerVolumeConfiguration=DockerVolumeConfiguration(
Scope='shared',
Autoprovision=True
)
)
],
ContainerDefinitions=[
ContainerDefinition(
Name='WordpressDatabaseContainer',
Image='mysql:5.7',
Environment=[
Environment(Name='MYSQL_ROOT_PASSWORD', Value='root'),
Environment(Name='MYSQL_DATABASE', Value='wpdb'),
Environment(Name='MYSQL_USER', Value='wordpressuser'),
Environment(Name='MYSQL_PASSWORD', Value='wordpressuserpassword'),
],
PortMappings=[
PortMapping(
ContainerPort=3306
)
]
)
]
)
self.wordpress_database_service = Service(
'WordpressDatabaseService',
Cluster=Ref(self.ecs_cluster),
DesiredCount=1,
TaskDefinition=Ref(self.wordpress_database_task),
LaunchType='FARGATE',
NetworkConfiguration=NetworkConfiguration(
AwsvpcConfiguration=AwsvpcConfiguration(
Subnets=[Ref(sub) for sub in VpcFormation().public_subnets],
AssignPublicIp='ENABLED',
SecurityGroups=[Ref(self.security_group)]
)
),
)
Note the AssignPublicIp='ENABLED' option so you would be able to connect to the database remotely.
After the stack completed i was able to successfully connect with a command:
mysql -uwordpressuser -pwordpressuserpassword -h18.202.31.123
Thats it :)
Related
I have a cluster on AWS and it has a task that should run two containers. One of the containers called app has a dependency on my db container which is a mysql image and it should run when my db image is healthy.
I have managed to write a healthcheck in a docker compose with command
["CMD","mysqladmin","ping","-h","localhost"].
However I'm unable to run the same command on my aws container and the container is always unhealthy when I write this command on my task definition.
I have replaced CMD with CMD-Shell and it has the same result. I have also tried the generic healthcheck command advised by AWS [ "CMD-SHELL", "curl -f http://localhost/ || exit 1" ]
but since the mysql image does not have curl installed I get the same result again.
What am I doing wrong?
I'm using exactly the same command (CMD) in my test deployment and it works, this makes me think that maybe you're interpreting the unhealthy state as the health check command not working, but there are a number of other reasons for that state.
For example, during my tests it wasn't able to start because I was using EFS to provide permanent storage and every time I was updating the task, the files were not deleted, so the container was failing because it was Unable to lock ./ibdata1.
There's apparently a second issue anyway, because CMD and CMD-SHELL (which are part of the docker syntax and unrelated the OS), requires a different syntax: CMD need a list, while CMD-SHELL allows a single string as the second item of the array.
Basically, if you try this in compose, it works:
["CMD","mysqladmin","ping","-h","localhost"]
this does not:
["CMD-SHELL","mysqladmin","ping","-h","localhost"]
and it should be instead:
["CMD-SHELL","mysqladmin ping -h localhost"]
So to answer the question, make sure to create a log group on Cloudwatch and set the "logConfiguration" into the container definition and check the logs, more than likely there's something wrong into that container, and make sure to use the right syntax into the container definition, either this example
"healthCheck": {
"retries": 3,
"command": ["CMD","mysqladmin","ping","-h","localhost"],
"timeout": 5,
"interval": 5,
"startPeriod": null
},
or this one - provided a shell has been defined into the container
"healthCheck": {
"retries": 3,
"command": ["CMD-SHELL","mysqladmin ping -h localhost"],
"timeout": 5,
"interval": 5,
"startPeriod": null
},
Into your logs you should find a lot of record like
[Note] Access denied for user 'root'#'localhost' (using password: NO)
Which is the healthcheck running. It fails because we didn't set any password, but as the mysqladmin docs are stating:
The return status from mysqladmin is 0 if the server is running, 1 if it is not. This is 0 even in case of an error such as Access denied, because this means that the server is running but refused the connection, which is different from the server not running.
I am creating an EMR cluster, and using jupyter notebook to run some spark tasks.
My tasks die after approximately 1 hour of execution, and the error is:
An error was encountered:
Invalid status code '400' from https://xxx.xx.x.xxx:18888/sessions/0/statements/20 with error payload: "requirement failed: Session isn't active."
My understanding is that it is related to the Livy config livy.server.session.timeout, but I don't know how I can set it in the bootstrap of the cluster (I need to do it in the bootstrap because the cluster is created with no ssh access)
Thanks a lot in advance
On EMR, livy-conf is the classification for the properties for livy's livy.conf file, so when creating an EMR cluster, choose advanced options with Livy as an application chosen to install, please pass this EMR configuration in the Enter Configuration field.
[{'classification': 'livy-conf','Properties': {'livy.server.session.timeout':'5h'}}]
On EMR, Livy binary is located at /etc/livy/, and so the config file is at /etc/livy/conf/livy.conf
To verify this,
Create an EMR cluster with a known ec2 key-pair, Livy and above config
Using the ec2 key-pair, login to the EC2 Master node associated with the cluster ssh -i some-ec2-key-pair.pem hadoop#ec2-00-00-00-0.ca-region-n.compute.amazonaws.com
Navigate to /etc/livy/conf, vim livy.conf & see the updated value of livy.server.session.timeout
If you don't want the Livy session to go down at all, then set the property livy.server.session.timeout-check to false in /etc/livy/conf/livy.conf.
Another way to do that if you don’t want to recreate the cluster is:
go to /etc/livy/conf/livy.conf and set the livy.server.session.timeout property to the value you would like.
After that, run sudo restart livy-server to make the configuration applied.
I'm trying to setup a private ethereum test network using Puppeth (as Péter Szilágyi demoed in Ethereum devcon three 2017). I'm running it on a macbook pro (macOS Sierra).
When I try to setup the ethstat network component I get an "docker configured incorrectly: bash: docker: command not found" error. I have docker running and I can use it fine in the terminal e.g. docker ps.
Here are the steps I took:
What would you like to do? (default = stats)
1. Show network stats
2. Manage existing genesis
3. Track new remote server
4. Deploy network components
> 4
What would you like to deploy? (recommended order)
1. Ethstats - Network monitoring tool
2. Bootnode - Entry point of the network
3. Sealer - Full node minting new blocks
4. Wallet - Browser wallet for quick sends (todo)
5. Faucet - Crypto faucet to give away funds
6. Dashboard - Website listing above web-services
> 1
Which server do you want to interact with?
1. Connect another server
> 1
Please enter remote server's address:
> localhost
DEBUG[11-15|22:46:49] Attempting to establish SSH connection server=localhost
WARN [11-15|22:46:49] Bad SSH key, falling back to passwords path=/Users/xxx/.ssh/id_rsa err="ssh: cannot decode encrypted private keys"
The authenticity of host 'localhost:22 ([::1]:22)' can't be established.
SSH key fingerprint is xxx [MD5]
Are you sure you want to continue connecting (yes/no)? yes
What's the login password for xxx at localhost:22? (won't be echoed)
>
DEBUG[11-15|22:47:11] Verifying if docker is available server=localhost
ERROR[11-15|22:47:11] Server not ready for puppeth err="docker configured incorrectly: bash: docker: command not found\n"
Here are my questions:
Is there any documentation / tutorial describing how to setup this remote server properly. Or just on puppeth in general?
Can I not use localhost as "remote server address"
Any ideas on why the docker command is not found (it is installed and running and I can use it ok in the terminal).
Here is what I did.
For the docker you have to use the docker-compose binary. You can find it here.
Furthermore, you have to be sure that an ssh server is running on your localhost and that keys have been generated.
I didn't find any documentations for puppeth whatsoever.
I think I found the root cause to this problem. The SSH daemon is compiled with a default path. If you ssh to a machine with a specific command (other than a shell), you get that default path. This does not include /usr/local/bin for example, where docker lives in my case.
I found the solution here: https://serverfault.com/a/585075:
edit /etc/ssh/sshd_config and make sure it contains PermitUserEnvironment yes (you need to edit this with sudo)
create a file ~/.ssh/environment with the path that you want, in my case:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
When you now run ssh localhost env you should see a PATH that matches whatever you put in ~/.ssh/environment.
For testing, I want to be able to run several IPFS nodes on a single machine.
This is the scenario:
I am building small services on top of IPFS core library, following the Making your own IPFS service guide. When I try to put client and server on the same machine (note that each of them will create their own IPFS node), I will get the following:
panic: cannot acquire lock: Lock FcntlFlock of /Users/long/.ipfs/repo.lock failed: resource temporarily unavailable
Usually, when you start with IPFS, you will use ipfs init, which will create a new node. The default data and config stored for that particular node are located at ~/.ipfs. Here is how you can create a new node and config it so it can run besides your default node.
1. Create a new node
For a new node you have to use ipfs init again. Use for instance the following:
IPFS_PATH=~/.ipfs2 ipfs init
This will create a new node at ~/.ipfs2 (not using the default path).
2. Change Address Configs
As both of your nodes now bind to the same ports, you need to change the port configuration, so both nodes can run side by side. For this, open ~/.ipfs2/configand findAddresses`:
"Addresses": {
"API": "/ip4/127.0.0.1/tcp/5001",
"Gateway": "/ip4/127.0.0.1/tcp/8080",
"Swarm": [
"/ip4/0.0.0.0/tcp/4001",
"/ip6/::/tcp/4001"
]
}
To for example the following:
"Addresses": {
"API": "/ip4/127.0.0.1/tcp/5002",
"Gateway": "/ip4/127.0.0.1/tcp/8081",
"Swarm": [
"/ip4/0.0.0.0/tcp/4002",
"/ip6/::/tcp/4002"
]
}
With this, you should be able to run both node .ipfs and .ipfs2 on a single machine.
Notes:
Whenever you use .ipfs2, you need to set the env variable IPFS_PATH=~/.ipfs2
In your example you need to change either your client or server node from ~/.ipfs to ~/.ipfs2
you can also start the daemon on the second node using IPFS_PATH=~/.ipfs2 ipfs daemon &
Hello, I use ipfs2, after running two daemons at the same time, can indeed open localhost:5001 / webui, run the second localhost:5002 / webui has an error, as shown in the attachment
Here are some ways I've used to create multiple nodes/peers ids.
I use windows 10.
1st node go-ipfs (latest version)
2nd node Siderus Orion ifps (connect to Orion node , not local) -- https://orion.siderus.io/
Use VirtualBox to run a minimal ubuntu installation. (You can set up as many as you want)
Repeat the process and you have 4 nodes or as many as you want.
https://discuss.ipfs.io/t/ipfs-manager-download-install-manage-debug-your-ipfs-node/3534 is another gui that installs and lets you manage all ipfs commands without CMD. He just released it a few days ago and it looks well worth lots of reviews.
Disclaimer I am not a coder or computer professional. Just a huge fan of IPFS! I hope we can raise awareness and change the world.
I have a scenario where I need to change several parameters of a hadoop cluster managed by Ambari to document performance of a particular application. The change in the configs entails a restart of the affected components.
I am using the Ambari REST API for achieving this. I figured out how to do this for all service components of hadoop. I' am not sure whether the API provides a way to restart the MySQL server that Hive uses.
I have the following questions:-
Is it the case that a mere stop and start of mysqld on the appropriate machine is enough to ensure that the required configuration changes are recognized by Ambari and the application?
I chose the 'New MySQL database' option while installing Hive via Ambari. Does this mean that restarts are reflected in Ambari only when it is carried out from the Ambari UI?
Your inputs would be highly appreciated.
Thanks!
Found a solution to the problem. I used the following commands using the Ambari REST API for changing configurations and restarting services from the backend.
Login to the host on which the ambari server is running and use the already provided config.sh script as described below.
Modifying configuration files
#!/bin/bash
CLUSTER_NAME=$1
CONFIG_FILE=$2
PROPERTY_NAME=$3
PROPERTY_VALUE=$4
/var/lib/ambari-server/resources/scripts/configs.sh -port <ambari-server-port> set localhost $1 $2 "$3" "$4"
where CONFIG_FILE can take values like tez-site, mapred-site, hadoop-site, hive-site etc. PROPERTY_NAME and PROPERTY_VALUE should be set to values relevant to the specified CONFIG_FILE.
Restarting host components
curl -uadmin:admin -H 'X-Requested-By: ambari' -X POST -d '
{
"RequestInfo":{
"command":"RESTART",
"context":"Restart MySQL server used by Hive Metastore on node3.cluster.com and HDFS client on node1.cluster.com",
"operation_level":{
"level":"HOST",
"cluster_name":"c1"
}
},
"Requests/resource_filters":[
{
"service_name":"HIVE",
"component_name":"MYSQL_SERVER",
"hosts":"node3.cluster.com"
},
{
"service_name":"HDFS",
"component_name":"HDFS_CLIENT",
"hosts":"node1.cluster.com"
}
]
}' http://localhost:<ambari-server-port>/api/v1/clusters/c1/requests
Reference Links:
Restarting components
modifying configurations
Hope this helps!