Mysql server not running on EC2 instance created from AMI - mysql

Question

I have an EC2 instance in which Mysql server is working fine. I created an AMI from this and launched a new instance from this AMI. Now Mysql server is not able to start on this new AMI at all.
Below is the o/p:
ubuntu#ip-172-31-66-160:~$ sudo service mysql status
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: activating (start-post) (Result: exit-code) since Sun 2019-07-21 20:59:25 IST; 28s ago
Process: 1870 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Process: 1862 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 1870 (code=exited, status=1/FAILURE); : 1871 (mysql-systemd-s)
Tasks: 2
Memory: 3.2M
CPU: 247ms
CGroup: /system.slice/mysql.service
└─control
├─1871 /bin/bash /usr/share/mysql/mysql-systemd-start post
└─2529 sleep 1
Jul 21 20:59:25 ip-172-31-66-160 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Jul 21 20:59:25 ip-172-31-66-160 systemd[1]: Stopped MySQL Community Server.
Jul 21 20:59:25 ip-172-31-66-160 systemd[1]: Starting MySQL Community Server...
Jul 21 20:59:26 ip-172-31-66-160 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
ubuntu#ip-172-31-66-160:~$ sudo service mysql restart
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
I tried everything but not able to get this started.
Closest threads which I came across and followed without any luck:
https://askubuntu.com/questions/916009/mysql-wont-start-because-of-apparmor
https://support.plesk.com/hc/en-us/articles/360004185293-Unable-to-start-MySQL-on-Ubuntu-AVC-apparmor-DENIED-operation-open-
Kindly help, I am just clueless about this.

Answer 1

Finally, I was able to make the server run on my new instance. What I was missing is that mysql also has many config options which gets changed over due course of time and they also depend on the EC2 instance config.
In my case, my new instance has very limited RAM and the mysql.cnf file was trying to allocate more RAM than the machine had hence it was failing. Somehow,
journalctl -xe
is giving a different error which is forcing me to think in wrong direction. I should have looked in to the mysql log file which quickly pointed me to the right issue.
I found below in the error log file which helped me to isolate / fix the issue:
2019-07-21T16:00:55.658985Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12 2019-07-21T16:00:55.658993Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2019-07-21T16:00:55.658998Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2019-07-21T16:00:55.659003Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2019-07-21T16:00:55.659620Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2019-07-21T16:00:55.659628Z 0 [ERROR] Failed to initialize builtin plugins.
2019-07-21T16:00:55.659632Z 0 [ERROR] Aborting
So mysql wasn't able to start since requested amount of memory isn't present in the system. I commented all the memory allocation lines in my.cnf file (relying on the default settings) and then it worked.
innodb_buffer_pool_size = 16G
key_buffer_size = 2G
max_allowed_packet = 128M
group_concat_max_len = 50000
query_cache_size = 2147483648
query_cache_limit =67108864
Hope someone would find this information useful.