问题描述

线上 mq 集群突然访问异常，以为重新启动节点就能很快解决，结果启动时日志中有下面错误，并不能正常启动：

2021-11-26 09:56:04.474 [error] <0.913.0>
2021-11-26 09:56:04.474 [error] <0.913.0> BOOT FAILED
2021-11-26 09:56:04.474 [error] <0.913.0> ===========
2021-11-26 09:56:04.474 [error] <0.913.0> Exception during startup:
2021-11-26 09:56:04.474 [error] <0.913.0>
2021-11-26 09:56:04.474 [error] <0.913.0>     rabbit:do_run_postlaunch_phase/0 line 932
2021-11-26 09:56:04.475 [error] <0.913.0>     lists:foreach/2 line 1338
2021-11-26 09:56:04.475 [error] <0.913.0>     rabbit:'-do_run_postlaunch_phase/0-fun-12-'/1 line 934
2021-11-26 09:56:04.475 [error] <0.913.0>     rabbit_boot_steps:run_boot_steps/1 line 29
2021-11-26 09:56:04.475 [error] <0.913.0>     rabbit_boot_steps:'-run_boot_steps/1-lc$^0/1-0-'/1 line 28
2021-11-26 09:56:04.475 [error] <0.913.0>     rabbit_boot_steps:run_step/2 line 55
2021-11-26 09:56:04.475 [error] <0.913.0>     rabbit_boot_steps:'-run_step/2-lc$^0/1-0-'/2 line 50
2021-11-26 09:56:04.475 [error] <0.913.0> error:{case_clause,{timeout,['rabbit_delayed_messagerabbit@flowable-mq3',
2021-11-26 09:56:04.475 [error] <0.913.0>                              'rabbit_delayed_messagerabbit@flowable-mq3_index']}}
2021-11-26 09:56:04.476 [error] <0.913.0>

解决过程

通过日志发现，可能跟 rabbitmq-delayed-message-exchange 这个有关。

通过安装记录查看到，rabbitmq-delayed-message-exchange 是一个插件，于是停用了这个插件，果然节点就启动成功了。

[root@flowable-mq3 ~]# rabbitmq-plugins disable  rabbitmq_delayed_message_exchange
Disabling plugins on node rabbit@flowable-mq3:
rabbitmq_delayed_message_exchange

The following plugins have been configured:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit@flowable-mq3...
The following plugins have been disabled:
  rabbitmq_delayed_message_exchange

set 3 plugins.
Offline change; changes will take effect at broker restart.

然后再尝试重新启用这个插件，发现还是有错误。

rabbitmq-plugins enable rabbitmq_delayed_message_exchange
Enabling plugins on node rabbit@flowable-mq3:
rabbitmq_delayed_message_exchange
The following plugins have been configured:
  rabbitmq_delayed_message_exchange
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit@flowable-mq3...
Stack trace:

** (CaseClauseError) no case clause matching: {:badrpc, {:EXIT, {{:case_clause, {:timeout, [:"rabbit_delayed_messagerabbit@flowable-mq3", :"rabbit_delayed_messagerabbit@flowable-mq3_index"]}}, [{:rabbit_boot_steps, :"-run_step/2-lc$^0/1-0-", 2, [file: 'src/rabbit_boot_steps.erl', line: 50]}, {:rabbit_boot_steps, :run_step, 2, [file: 'src/rabbit_boot_steps.erl', line: 55]}, {:rabbit_boot_steps, :"-run_boot_steps/1-lc$^0/1-0-", 1, [file: 'src/rabbit_boot_steps.erl', line: 28]}, {:rabbit_boot_steps, :run_boot_steps, 1, [file: 'src/rabbit_boot_steps.erl', line: 29]}, {:rabbit, :"-start_apps/2-fun-0-", 2, [file: 'src/rabbit.erl', line: 517]}, {:lists, :foreach, 2, [file: 'lists.erl', line: 1338]}, {:rabbit_plugins, :ensure1, 1, [file: 'src/rabbit_plugins.erl', line: 54]}, {:rpc, :"-handle_call_call/6-fun-0-", 5, [file: 'rpc.erl', line: 197]}]}}}
    (rabbitmqctl) lib/rabbitmq/cli/plugins/plugins_helpers.ex:219: RabbitMQ.CLI.Plugins.Helpers.update_enabled_plugins/2
    (rabbitmqctl) lib/rabbitmq/cli/plugins/plugins_helpers.ex:116: RabbitMQ.CLI.Plugins.Helpers.update_enabled_plugins/4
    (rabbitmqctl) lib/rabbitmq/cli/plugins/commands/enable_command.ex:130: anonymous fn/6 in RabbitMQ.CLI.Plugins.Commands.EnableCommand.do_run/2
    (elixir) lib/stream.ex:1311: anonymous fn/2 in Stream.iterate/2
    (elixir) lib/stream.ex:1497: Stream.do_unfold/4
    (elixir) lib/stream.ex:1568: Enumerable.Stream.do_each/4
    (elixir) lib/stream.ex:936: Stream.do_enum_transform/7
    (elixir) lib/stream.ex:1568: Enumerable.Stream.do_each/4

{:case_clause, {:badrpc, {:EXIT, {{:case_clause, {:timeout, [:"rabbit_delayed_messagerabbit@flowable-mq3", :"rabbit_delayed_messagerabbit@flowable-mq3_index"]}}, [{:rabbit_boot_steps, :"-run_step/2-lc$^0/1-0-", 2, [file: 'src/rabbit_boot_steps.erl', line: 50]}, {:rabbit_boot_steps, :run_step, 2, [file: 'src/rabbit_boot_steps.erl', line: 55]}, {:rabbit_boot_steps, :"-run_boot_steps/1-lc$^0/1-0-", 1, [file: 'src/rabbit_boot_steps.erl', line: 28]}, {:rabbit_boot_steps, :run_boot_steps, 1, [file: 'src/rabbit_boot_steps.erl', line: 29]}, {:rabbit, :"-start_apps/2-fun-0-", 2, [file: 'src/rabbit.erl', line: 517]}, {:lists, :foreach, 2, [file: 'lists.erl', line: 1338]}, {:rabbit_plugins, :ensure1, 1, [file: 'src/rabbit_plugins.erl', line: 54]}, {:rpc, :"-handle_call_call/6-fun-0-", 5, [file: 'rpc.erl', line: 197]}]}}}}

查找资料

通过搜索引擎搜索后，在插件的 GitHub 项目 Issus 列表中发现了相关的问题，it works on disk node but can not enable on ram node。

有官方的工程师提到：

RAM nodes are not supported by this plugin, see the README. Most users don't need RAM nodes to begin with.

解决方案

于是根据上述内容，将 RAM 类型的 node 改为 DISK 类型，问题解决。

[root@flowable-mq3 rabbit@flowable-mq3]#  rabbitmqctl stop_app
Stopping rabbit application on node rabbit@flowable-mq3 ...
You have mail in /var/spool/mail/root
[root@flowable-mq3 rabbit@flowable-mq3]#  rabbitmqctl change_cluster_node_type disc
Turning rabbit@flowable-mq3 into a disc node
[root@flowable-mq3 rabbit@flowable-mq3]#  rabbitmqctl start_app
Starting node rabbit@flowable-mq3 ...
[root@flowable-mq3 rabbit@flowable-mq3]# rabbitmq-plugins enable  rabbitmq_delayed_message_exchange
Enabling plugins on node rabbit@flowable-mq3:
rabbitmq_delayed_message_exchange
The following plugins have been configured:
  rabbitmq_delayed_message_exchange
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit@flowable-mq3...
Plugin configuration unchanged.