I have been attempting to deploy my Django project up on AWS ElasticBeanstalk via EB CLI n my local Pycharm project with the built in terminal.
Having setup IAM/group (through aws web console) I migrated and ran server locally (Pycharm terminal) and all works fine, and then created env/instance and requirements.txt and deployed using EB CLI in Pycharm and this also worked fine.
I setup a database connection on RDS (AWS web Console) then changed settings to connect to mysql, pip installed mysqlclient, migrate and ran locally (Pycharm terminal) and again all works fine.
However, when I try to deply the project again with mysqlclient installed I get errors and I just cannot figure out why.
Platform: arn:aws:elasticbeanstalk:us-west-2::platform/Python 3.7 running on 64bit Amazon Linux 2/3.3.7
Requirements.txt
asgiref==3.4.1
cement==2.8.2
certifi==2021.10.8
charset-normalizer==2.0.7
colorama==0.4.3
Django==3.2.9
django-storages==1.12.3
future==0.16.0
idna==3.3
jmespath==0.10.0
mysqlclient
pathspec==0.5.9
python-dateutil==2.8.2
pytz==2021.3
PyYAML==5.4.1
requests==2.26.0
s3transfer==0.5.0
semantic-version==2.8.5
six==1.14.0
sqlparse==0.4.2
termcolor==1.1.0
typing-extensions==3.10.0.2
urllib3==1.26.7
wcwidth==0.1.9
.ebextensions/django.config
option_settings:
aws:elasticbeanstalk:container:python:
WSGIPath: awsdeploy.wsgi:application
aws:elasticbeanstalk:environment:proxy:staticfiles:
/static: static
container_commands:
01_collectstatic:
command: "source /var/app/venv/staging-LQM1lest/bin/activate && python manage.py collectstatic --noinput --clear"
02_migrate:
command: "source /var/app/venv/staging-LQM1lest/bin/activate && python manage.py migrate --noinput"
leader_only: true
.ebextensions/packages.config
packages:
yum:
python3-devel: []
mariadb-devel: []
EB Error Logs
Collecting mysqlclient==2.0.3
Using cached mysqlclient-2.0.3.tar.gz (88 kB)
2021/11/11 02:10:40.712344 [ERROR] An error occurred during execution of command [app-deploy] - [InstallDependency]. Stop running the command. Error: fail to install dependencies with requirements.txt file with error Command /bin/sh -c /var/app/venv/staging-LQM1lest/bin/pip install -r requirements.txt failed wi
th error exit status 1. Stderr: ERROR: Command errored out with exit status 1:
command: /var/app/venv/staging-LQM1lest/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-isj_ea8s/mysqlclient_f221eb2ed3c44d6e868da4539588d2a4/setup.py'"'"'; __file__='"'"'/tmp/pip-install-isj_ea8s/mysqlclient_f221eb2ed3c44d6e868da4539588d2a4/setup.py'"'"';f = ge
tattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-hzv5pj6h
cwd: /tmp/pip-install-isj_ea8s/mysqlclient_f221eb2ed3c44d6e868da4539588d2a4/
Complete output (15 lines):
/bin/sh: mysql_config: command not found
/bin/sh: mariadb_config: command not found
/bin/sh: mysql_config: command not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-isj_ea8s/mysqlclient_f221eb2ed3c44d6e868da4539588d2a4/setup.py", line 15, in <module>
metadata, options = get_config()
File "/tmp/pip-install-isj_ea8s/mysqlclient_f221eb2ed3c44d6e868da4539588d2a4/setup_posix.py", line 70, in get_config
libs = mysql_config("libs")
File "/tmp/pip-install-isj_ea8s/mysqlclient_f221eb2ed3c44d6e868da4539588d2a4/setup_posix.py", line 31, in mysql_config
raise OSError("{} not found".format(_mysql_config_path))
OSError: mysql_config not found
mysql_config --version
mariadb_config --version
mysql_config --libs
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/3c/df/59cd2fa5e48d0804d213bdcb1acb4d08c403b61c7ff7ed4dd4a6a2deb3f7/mysqlclient-2.0.3.tar.gz#sha256=f6ebea7c008f155baeefe16c56cd3ee6239f7a5a9ae42396c2f1860f08a7c432 (from https://pypi.org/simple/mysqlclient/) (requires-python:>=3.5). Command errored out
with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement mysqlclient==2.0.3 (from versions: 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.3.8, 1.3.9, 1.3.10, 1.3.11rc1, 1.3.11, 1.3.12, 1.3.13, 1.3.14, 1.4.0rc1, 1.4.0rc2, 1.4.0rc3, 1.4.0, 1.4.1, 1.4.2, 1.4.2.post1, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 2
.0.0, 2.0.1, 2.0.2, 2.0.3, 2.1.0rc1)
ERROR: No matching distribution found for mysqlclient==2.0.3
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/var/app/venv/staging-LQM1lest/bin/python -m pip install --upgrade pip' command.
2021/11/11 02:10:40.712360 [INFO] Executing cleanup logic
2021/11/11 02:10:40.712489 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[{"msg":"Instance deployment failed to install application dependencies. The deployment failed.","timestamp
":1636596640,"severity":"ERROR"},{"msg":"Instance deployment failed. For details, see 'eb-engine.log'.","timestamp":1636596640,"severity":"ERROR"}]}]}
2021/11/11 02:10:40.712674 [INFO] Platform Engine finished execution on command: app-deploy
I don't understand why prior to installing mysqlclient it's managed to deploy install all my packages from requirements file successfully but fails when mysqlclient was added - what am I missing?
Help is much appreciated, happy to share any other details if needed. Thanks
Edit
I came across the answer on this link: [https://stackoverflow.com/questions/67344766/error-encountered-while-installing-dependencies-for-flask-mysql-application-usin][1]
And created the .ebextensions/packages.config as suggested and removed version from mysqlclient within my requirements.txt file - updated above code.
Now mysqlclient appears to have installed succssfully but now get the following error:
New EB Error Logs
An error occurred during execution of command [app-deploy] - [PostBuildEbExtension]. Stop running the command. Error: container commands build failed. Please refer to /var/log/cfn-init.log for more details.
I'm also not sure how to check this log /var/log/cfn-init.log, when I go to the web console I can't see any eb instances listed or any logs.
$ python setup.py install
running install
Not Installing MySQL C Extension
running build
running build_py
running install_lib
running install_egg_info
running egg_info
writing requirements to lib/mysql_connector_python.egg-info/requires.txt
error: [Errno 13] Permission denied: 'lib/mysql_connector_python.egg-info/requires.txt'
when i try to install mysql connector it shows an error when i execute the command "setup.py install".How can i overcome this error?
Try to run command with sudo.
sudo python setup.py install
I have a brand new django 2.2.1 project I have just installed into a python 3.7 virtualenv on OS X (10.14.4). After some frustrations I got mysqlclient to install but when I run the django dev server it doesn't recognise that it is installed:
Here are the steps I've taken so far:
brew install mysql
pipenv --three
pipenv install django==2.2.1
pipenv install mysqlclient
brew uninstall mysql
brew install mysql-connector-c
pipenv install mysqlclient
brew unlink mysql-connector-c
brew install mysql
django-admin startproject projectname
Now, the only change I have made to the out-of-the-box django installation is to change the default database backend to django.db.backends.mysql and when I run the django server I get the following:
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module.
Did you install mysqlclient?
However, going back to the virtualenv and doing "pip install mysqlclient" gives:
Requirement already satisfied: mysqlclient in /Users/<username>/.local/share/virtualenvs/<projectname>-KrUE_JNo/lib/python3.7/site-packages (1.4.2.post1)
Any ideas why django can't see the mysqlclient installed in the virtualenv? I can confirm that all of the above has been run in the virtualenv. I suspect it has something to do with the faff that OSX makes you go through to install it but I'm not sure how to pick it apart.
I have also tried to replace mysqldb with pymysql as suggested here but that causes the server to give a different error:
raise ImproperlyConfigured('mysqlclient 1.3.13 or newer is required; you have %s.' % Database.__version__)
django.core.exceptions.ImproperlyConfigured: mysqlclient 1.3.13 or newer is required; you have 0.9.3.
Mysql is a given for this project so use of an alternative backend is not an option. Any help very gratefully received, many thanks in advance.
UPDATE: Output of pip freeze:
Django==2.2.1
django-auth-ldap==1.7.0
Markdown==3.1
mysql-connector-python==8.0.16
mysqlclient==1.4.2.post1
protobuf==3.7.1
pyasn1==0.4.5
pyasn1-modules==0.2.5
PyMySQL==0.9.3
pyodbc==4.0.26
python-ldap==3.2.0
pytz==2019.1
simplejson==3.16.0
six==1.12.0
sqlparse==0.3.0
xmltodict==0.12.0
UPDATE 2:
Following conor's post (thanks conor) I started again with a fresh virtualenv and again successfully installed mysqlclient1.4.2.post1. Here's the output from pip freeze now:
Django==2.2.1
mysqlclient==1.4.2.post1
pytz==2019.1
sqlparse==0.3.0
Still getting the same error when I do python manage.py runserver though:
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module.
Did you install mysqlclient?
UPDATE 3:
Thought I would try importing the MySQLdb module on the python command line to see if I can get any more insight and it returns the following:
>>> import MySQLdb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/<username>/.local/share/virtualenvs/<projectname>-KrUE_JNo/lib/python3.7/site-packages/MySQLdb/__init__.py", line 18, in <module>
from . import _mysql
ImportError: dlopen(/Users/<username>/.local/share/virtualenvs/<projectname>-KrUE_JNo/lib/python3.7/site-packages/MySQLdb/_mysql.cpython-37m-darwin.so, 2): Symbol not found: _mysql_affected_rows
Referenced from: /Users/<username>/.local/share/virtualenvs/<projectname>-KrUE_JNo/lib/python3.7/site-packages/MySQLdb/_mysql.cpython-37m-darwin.so
Expected in: flat namespace
in /Users/<username>/.local/share/virtualenvs/<projectname>-KrUE_JNo/lib/python3.7/site-packages/MySQLdb/_mysql.cpython-37m-darwin.so
SOLVED
So it looks like the issue was that on initial installation the mysqlclient library had compiled against the wrong version of mysql (not sure how that happened), so I had to force it to recompile.
Here are the steps:
brew uninstall mysql
brew uninstall myysql-connector-c
pipenv uninstall mysqlclient
brew install mysql-connector-c
At this point we need to update /usr/local/bin/mysql_config as per the instructions that conor linked to (thanks again conor), i.e. change the line that read
libs="$libs -l "
to
libs="$libs -lmysqlclient -lssl -lcrypto "
Then, to fix the resultant "library not found for -lssl" error I used the answer from this question:
export PATH="/usr/local/opt/openssl/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
Then finally force mysqlclient to recompile and reinstall mysql:
pip install --force-reinstall --ignore-installed --no-binary :all: mysqlclient
brew unlink mysql-connector-c
brew install mysql
Thanks to everyone who took time to help out!
If you're on macos do this
$ brew uninstall mysql
$ brew install mysql-connector-c
$ brew unlink mysql-connector-c
$ brew install mysql
$ pip install mysql-python
and follow the instructions here: https://pypi.org/project/mysqlclient/
Downgrading from python 3.7.4 to python 3.6 solved the issue for me on windows 10.
sudo pip install mysqlclient is failing with the famous mysql_config path error
The database is a remote mysql RDS. I tried to install a local mysql using yum (not that i need it) but that didn't help with the error
[ec2-user#ip-172-31-95-95 /]$ sudo pip install mysqlclient
Collecting mysqlclient
Using cached mysqlclient-1.3.12.tar.gz
Complete output from command python setup.py egg_info:
sh: mysql_config: command not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-opVvwg/mysqlclient/setup.py", line 17, in <module>
metadata, options = get_config()
File "setup_posix.py", line 44, in get_config
libs = mysql_config("libs_r")
File "setup_posix.py", line 26, in mysql_config
raise EnvironmentError("%s not found" % (mysql_config.path,))
EnvironmentError: mysql_config not found
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-opVvwg/mysqlclient/
[ec2-user#ip-172-31-95-95 /]$
[ec2-user#ip-172-31-95-95 /]$ cat /etc/*-release
NAME="Amazon Linux AMI"
VERSION="2017.09"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2017.09"
PRETTY_NAME="Amazon Linux AMI 2017.09"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2017.09:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2017.09
[ec2-user#ip-172-31-95-95 /]$
First you have to do some initial setup for the compiler of mysql on the os, at least from what I understand.
Make sure you have compiler collection
sudo yum install gcc
Make sure mysql is configured for development purposes
sudo yum install -y mysql-devel
Try to reinstall
pip install mysqlclient==1.3.12
Hope this helps since it seemed to do the trick for me. I came across this here where I found for the equivalent to ubuntu's sudo apt-get install libmysqlclient-dev
When I try to install MySql Workbench using the yum command :
yum install mysql-workbench-gpl-5.2.45-1fc17.i686.rpm
I get these errors :
Error: Package: mysql-workbench-gpl-5.2.45-1fc17.i686 (/mysql-workbench-gpl-5.2.45-1fc17.i686)
Requires: /bin/python
Error: Package: mysql-workbench-gpl-5.2.45-1fc17.i686 (/mysql-workbench-gpl-5.2.45-1fc17.i686)
Requires: libzip.so.2
Error: Package: mysql-workbench-gpl-5.2.45-1fc17.i686 (/mysql-workbench-gpl-5.2.45-1fc17.i686)
Requires: libctemplate.so.2
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
What are these errors and what do I do to install this MySql GUI tool ?
Because you are installing RedHat 6 OS base RPM. You have to upgrade or install required dependencies first. In your case upgrade/install python libzip ctemplate RPM packages.
Notes: libzip/ctemplate isn't part of RedHat 6 OS. You have to download from third party sources.