mysqld did not start properly and how to troubleshoot
This morning, after rebooting my server, the mysqld service did not listen for incoming connections (port 3306). But I can still see the mysql process:
root@myserver:~ ps aux | grep mysql
mysql 5804 3.8 17.1 2626952 869352 ? Ssl 07:34 4:49 /usr/sbin/mysqld
root 20822 0.0 0.0 9388 932 pts/4 R+ 09:39 0:00 grep --color=auto mysql
First, we should check the mysqld process's status:
root@myserver:~# service mysql status
mysql start/post-start, process 4394
post-start process 4395
Then check the syslog:
root@myserver:~# tail -f /var/log/syslog
May 7 07:33:37 myserver kernel: [ 146.209493] init: mysql post-start process (3786) terminated with status 1
May 7 07:33:41 myserver kernel: [ 150.125783] init: mysql main process (4057) terminated with status 7
May 7 07:33:41 myserver kernel: [ 150.125834] init: mysql main process ended, respawning
May 7 07:33:42 myserver kernel: [ 150.379645] init: mysql post-start process (4059) terminated with status 1
May 7 07:33:45 myserver kernel: [ 154.086347] init: mysql main process (4394) terminated with status 7
May 7 07:33:45 myserver kernel: [ 154.086388] init: mysql main process ended, respawning
May 7 07:33:46 myserver kernel: [ 154.553728] init: mysql post-start process (4395) terminated with status 1
May 7 07:33:49 myserver kernel: [ 158.012980] init: mysql main process (4529) terminated with status 7
May 7 07:33:49 myserver kernel: [ 158.013031] init: mysql main process ended, respawning
As you can see, the mysql process was trying to start but something hold it down. It maked the starting process of mysqld loop forever. But what was it? The log showed us nothing else.
So, try to stop the mysqld service then start it manually with '-v' parameter to get verbose outputs:
root@myserver:~# service mysql stop
root@myserver:~# mysqld -v
140507 7:34:10 InnoDB: The InnoDB memory heap is disabled
140507 7:34:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140507 7:34:10 InnoDB: Compressed tables use zlib 1.2.3.4
140507 7:34:10 InnoDB: Using Linux native AIO
140507 7:34:10 InnoDB: Initializing buffer pool, size = 1.5G
140507 7:34:11 InnoDB: Completed initialization of buffer pool
140507 7:34:11 InnoDB: highest supported file format is Barracuda.
140507 7:34:12 InnoDB: Waiting for the background threads to start
140507 7:34:13 Percona XtraDB (http://www.percona.com) 5.5.37-MariaDB-34.0 started; log sequence number 36793054164
140507 7:34:13 [Note] Plugin 'FEEDBACK' is disabled.
140507 7:34:13 [ERROR] mysqld: unknown variable 'key_buffer_zie=256M'
140507 7:34:13 [ERROR] Aborting
root@myserver:~ ps aux | grep mysql
mysql 5804 3.8 17.1 2626952 869352 ? Ssl 07:34 4:49 /usr/sbin/mysqld
root 20822 0.0 0.0 9388 932 pts/4 R+ 09:39 0:00 grep --color=auto mysql
What happened?
First, we should check the mysqld process's status:
root@myserver:~# service mysql status
mysql start/post-start, process 4394
post-start process 4395
Then check the syslog:
root@myserver:~# tail -f /var/log/syslog
May 7 07:33:37 myserver kernel: [ 146.209493] init: mysql post-start process (3786) terminated with status 1
May 7 07:33:41 myserver kernel: [ 150.125783] init: mysql main process (4057) terminated with status 7
May 7 07:33:41 myserver kernel: [ 150.125834] init: mysql main process ended, respawning
May 7 07:33:42 myserver kernel: [ 150.379645] init: mysql post-start process (4059) terminated with status 1
May 7 07:33:45 myserver kernel: [ 154.086347] init: mysql main process (4394) terminated with status 7
May 7 07:33:45 myserver kernel: [ 154.086388] init: mysql main process ended, respawning
May 7 07:33:46 myserver kernel: [ 154.553728] init: mysql post-start process (4395) terminated with status 1
May 7 07:33:49 myserver kernel: [ 158.012980] init: mysql main process (4529) terminated with status 7
May 7 07:33:49 myserver kernel: [ 158.013031] init: mysql main process ended, respawning
As you can see, the mysql process was trying to start but something hold it down. It maked the starting process of mysqld loop forever. But what was it? The log showed us nothing else.
So, try to stop the mysqld service then start it manually with '-v' parameter to get verbose outputs:
root@myserver:~# service mysql stop
root@myserver:~# mysqld -v
140507 7:34:10 InnoDB: The InnoDB memory heap is disabled
140507 7:34:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140507 7:34:10 InnoDB: Compressed tables use zlib 1.2.3.4
140507 7:34:10 InnoDB: Using Linux native AIO
140507 7:34:10 InnoDB: Initializing buffer pool, size = 1.5G
140507 7:34:11 InnoDB: Completed initialization of buffer pool
140507 7:34:11 InnoDB: highest supported file format is Barracuda.
140507 7:34:12 InnoDB: Waiting for the background threads to start
140507 7:34:13 Percona XtraDB (http://www.percona.com) 5.5.37-MariaDB-34.0 started; log sequence number 36793054164
140507 7:34:13 [Note] Plugin 'FEEDBACK' is disabled.
140507 7:34:13 [ERROR] mysqld: unknown variable 'key_buffer_zie=256M'
140507 7:34:13 [ERROR] Aborting
Here is it. There is one misconfiguration in the /etc/mysql/my.cnf. Change it and the mysqld will start successfully.