納尼？？Rabbitmq竟然被一個逗號給坑了？？

前言

這個問題發生在部署一套新的環境。搭建一個單節點的Rabbitmq，搭建步驟並無按照我在單機版Rabbitmq部署這篇文章的步驟，而是按照同事寫的部署文檔搭建的。其中有一些細微的差異，好比修改配置文件。

部署環境：centos7

問題描述

步驟以下：
一、先安裝erlang；
二、安裝Rabbitmq的rpm包
三、修改配置文件
四、啓動服務

可是在啓動的時候，失敗了：

[root@vm193-168-1-66 ~]# systemctl start rabbitmq-server
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.

根據提示查看日誌systemctl status rabbitmq-server.service：

[root@vm193-168-1-66 ~]# systemctl status rabbitmq-server.service
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; disabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2019-12-17 09:12:50 CST; 8s ago
  Process: 5378 ExecStop=/usr/sbin/rabbitmqctl shutdown (code=exited, status=69)
  Process: 5143 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=64)
 Main PID: 5143 (code=exited, status=64)
Dec 17 09:12:50 vm193-168-1-66 systemd[1]: Failed to start RabbitMQ broker.
Dec 17 09:12:50 vm193-168-1-66 systemd[1]: Unit rabbitmq-server.service entered failed state.
Dec 17 09:12:50 vm193-168-1-66 systemd[1]: rabbitmq-server.service failed.

並無找到很是明顯的緣由

再查看系統日誌，一樣的結果：

[root@vm193-168-1-66 ~]# journalctl -xe
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: * epmd reports: node 'rabbit' not running at all
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: no other nodes on vm193-168-1-66
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: * suggestion: start the node
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: Current node details:
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: * node name: 'rabbitmqcli-13769-rabbit@vm193-168-1-66'
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: * effective user's home directory: /var/lib/rabbitmq
Dec 17 09:15:42 vm193-168-1-66 rabbitmqctl[13769]: * Erlang cookie hash: WQikY88cKw2MYZAMQRGnMA==
Dec 17 09:15:42 vm193-168-1-66 systemd[1]: Failed to start RabbitMQ broker.
-- Subject: Unit rabbitmq-server.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit rabbitmq-server.service has failed.
-- 
-- The result is failed.
Dec 17 09:15:42 vm193-168-1-66 systemd[1]: Unit rabbitmq-server.service entered failed state.
Dec 17 09:15:42 vm193-168-1-66 systemd[1]: rabbitmq-server.service failed.
Dec 17 09:15:43 vm193-168-1-66 sshd[13480]: Received disconnect from 125.133.100.146: 11: Bye Bye [preauth]
Dec 17 09:15:48 vm193-168-1-66 sshd[13949]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=2
Dec 17 09:15:48 vm193-168-1-66 sshd[13949]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Dec 17 09:15:50 vm193-168-1-66 sshd[13949]: Failed password for root from 210.212.237.67 port 35528 ssh2
Dec 17 09:15:50 vm193-168-1-66 sshd[13949]: Received disconnect from 210.212.237.67: 11: Bye Bye [preauth]
lines 1317-1338/1338 (END)

緣由分析

因爲上邊的兩段日誌都沒有找到明顯的緣由，因而嘗試再次啓動rabbitmq，同時tail -f開啓messages日誌：tail -f /var/log/messages，發現有以下日誌內容（比較長，可略過這段日誌直接看後邊）：

Dec 17 09:26:48 vm193-168-1-66 systemd: rabbitmq-server.service holdoff time over, scheduling restart.
Dec 17 09:26:48 vm193-168-1-66 systemd: Cannot add dependency job for unit proc-sys-fs-binfmt_misc.automount, ignoring: Unit is masked.
Dec 17 09:26:48 vm193-168-1-66 systemd: Starting RabbitMQ broker...
    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: 2019-12-17 09:26:48.706 [error] Unable to parse erlang terms from  RABBITMQ_CONFIG_FILE file: /etc/rabbitmq/rabbitmq.config
    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: Error: {512,erl_parse,
    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: ["syntax error before: ","']'"]}
Dec 17 09:26:48 vm193-168-1-66 systemd: rabbitmq-server.service: main process exited, code=exited, status=64/n/a
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: Error: unable to perform an operation on node 'rabbit@vm193-168-1-66'. Please see diagnostics information and suggestions below.
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: Most common reasons for this are:
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * Target node is not running
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: In addition to the diagnostics info below:
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * Consult server logs on node rabbit@vm193-168-1-66
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * If target node is configured to use long node names, don't forget to use --longnames with CLI tools
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: DIAGNOSTICS
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: ========
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: attempted to contact: ['rabbit@vm193-168-1-66']
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: rabbit@vm193-168-1-66:
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * connected to epmd (port 4369) on vm193-168-1-66
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * epmd reports: node 'rabbit' not running at all
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: no other nodes on vm193-168-1-66
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * suggestion: start the node
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: Current node details:
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * node name: 'rabbitmqcli-14476-rabbit@vm193-168-1-66'
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * effective user's home directory: /var/lib/rabbitmq
Dec 17 09:26:49 vm193-168-1-66 rabbitmqctl: * Erlang cookie hash: WQikY88cKw2MYZAMQRGnMA==
Dec 17 09:26:49 vm193-168-1-66 systemd: Failed to start RabbitMQ broker.
Dec 17 09:26:49 vm193-168-1-66 systemd: Unit rabbitmq-server.service entered failed state.
Dec 17 09:26:49 vm193-168-1-66 systemd: rabbitmq-server.service failed.

從上面的/var/log/massages日誌找到了日誌裏下面三行重點提示：

Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: 2019-12-17 09:26:48.706 [error] Unable to parse erlang terms from  RABBITMQ_CONFIG_FILE file: /etc/rabbitmq/rabbitmq.config
    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: Error: {512,erl_parse,
    Dec 17 09:26:48 vm193-168-1-66 rabbitmq-server: ["syntax error before: ","']'"]}

能夠看出，在512行前面存在語法問題。

另外能夠用下面的命令檢測配置文件/etc/rabbitmq/rabbitmq.config是否正確

[root@vm193-168-1-66 lib]# erl -noshell -eval 'io:format("~p~n", [file:consult("/etc/rabbitmq/rabbitmq.config")]).' -eval 'init:stop().'
{error,{512,erl_parse,["syntax error before: ","']'"]}}     #結果也是提示在512行以前有語法錯誤
[root@vm193-168-1-66 lib]#

由於整個部署過程當中只有/etc/rabbitmq/rabbitmq.config這個配置文件改動了惟一的一行內容：第66行，去掉了{loopback_users, []},這一行前面的配置。

仔細檢查文件，發現{loopback_users, []},這一行後邊的內容所有註釋掉了的，所以其實應該是要去掉{loopback_users, []},末尾的逗號的：

解決辦法

嘗試着去掉了行尾的逗號，再次檢測配置文件，發現結果是OK了：

[root@vm193-168-1-66 rabbitmq]# erl -noshell -eval 'io:format("~p~n", [file:consult("/etc/rabbitmq/rabbitmq.config")]).' -eval 'init:stop().'
{ok,[[{rabbit,[{loopback_users,[]}]},
      {kernel,[]},
      {rabbitmq_management,[]},
      {rabbitmq_shovel,[{shovels,[]}]},
      {rabbitmq_stomp,[]},
      {rabbitmq_mqtt,[]},
      {rabbitmq_amqp1_0,[]},
      {rabbitmq_auth_backend_ldap,[]},
      {lager,[]}]]}
[root@vm193-168-1-66 rabbitmq]#

再次啓動rabbitmq，就成功了：

後記：

這是一個教訓：對於這種註釋掉不少內容的配置文件，修改完了以後，最好過濾掉註釋了的內容，查看一下剩下的行有沒有什麼格式、符號的問題！不然，就只有靠爬坑減肥了......