如何在Kubernetes搭建RabbitMQ集群部署篇

1. 概述

本文档详细介绍了如何在Kubernetes环境中部署高可用的RabbitMQ集群。该部署方案使用StatefulSet确保Pod的有序部署和稳定网络标识，通过Headless Service实现集群节点间的自动发现，并采用emptyDir临时存储用于测试和调试环境。

2. 架构设计

2.1 集群架构图

RabbitMQ集群架构图

如何在Kubernetes搭建RabbitMQ集群部署篇

2.2 网络配置

2.2.1 DNS名称解析

每个Pod有唯一的DNS名称： – 短DNS名称：rabbitmq-{序号}.rabbitmq-headless – 完整DNS名称：rabbitmq-{序号}
.rabbitmq-headless.cluster-playground.svc.cluster.local

2.2.2 端口说明

端口号	协议/服务	用途说明
5672	AMQP协议	消息传输端口，客户端连接使用
15672	HTTP管理界面	Web管理控制台端口
25672	集群通信	节点间数据同步和集群管理
4369	EPMD服务	Erlang端口映射守护进程

2.3 核心组件

组件	功能说明
StatefulSet	管理3个RabbitMQ Pod实例，确保有序部署和稳定网络标识
Headless Service	提供DNS发现机制，每个Pod有唯一的DNS名称
NodePort Service	提供外部访问入口
ConfigMap	包含RabbitMQ配置和健康检查脚本
Secret	存储Erlang Cookie用于集群节点认证
ServiceAccount	提供RBAC权限用于Pod发现

3. YAML文件详解

3.1 ConfigMap配置 (configmap.yaml)

ConfigMap包含两个关键配置：

3.1.1 RabbitMQ主配置文件 (rabbitmq.conf)

## 基本配置
listeners.tcp.default = 5672
management.tcp.port = 15672
## 集群配置
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = rabbitmq-headless

3.1.2 RabbitMQ Kubernetes Peer Discovery 插件介绍

rabbit_peer_discovery_k8s 是RabbitMQ官方提供的Kubernetes对等发现插件，专门用于在Kubernetes环境中实现RabbitMQ集群节点的自动发现和连接。

插件特性：

自动发现：利用Kubernetes API自动发现集群中的其他RabbitMQ节点
StatefulSet集成：专为StatefulSet设计，利用Pod的有序性和稳定网络标识
无需外部依赖：不依赖Consul、etcd等外部服务发现工具
高可用性：支持节点故障自动恢复和重新加入集群

配置参数说明：

cluster_formation.peer_discovery_backend：指定使用Kubernetes发现后端
cluster_formation.k8s.host：Kubernetes API服务器地址
cluster_formation.k8s.address_type：节点地址类型（hostname或ip）
cluster_formation.k8s.service_name：Headless Service名称

3.2 Headless Service (headless.yaml)

提供DNS发现机制，不分配ClusterIP：

spec:
  clusterIP: None
  ports:
    - name: amqp       # AMQP协议端口
      port: 5672
    - name: clustering # 集群通信端口
      port: 25672
    - name: epmd       # Erlang端口映射
      port: 4369

3.3 StatefulSet配置 (statefulsets.yaml)

核心部署配置：

spec:
  replicas: 3
  serviceName: rabbitmq-headless
  template:
    spec:
      # ...
      containers:
        - name: rabbitmq
          image: 'ci.local.com/rabbitmq/rabbitmq:3.9.21-management'
          command:
            - sh
            - '-c'
          args:
            - >
              # 启动节点发现插件

              rabbitmq-plugins enable --offline rabbitmq_management
              rabbitmq_peer_discovery_k8s


              # 启动RabbitMQ

              exec docker-entrypoint.sh rabbitmq-server
        # ....
          env:
            - name: RABBITMQ_DEFAULT_USER
              value: admin
            - name: RABBITMQ_DEFAULT_PASS
              value: Admin#123
            - name: RABBITMQ_ERLANG_COOKIE
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-erlang-cookie
                  key: .erlang.cookie
            # 节点发现使用长域名
            - name: RABBITMQ_USE_LONGNAME
              value: 'true'
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: RABBITMQ_MNESIA_BASE
              value: /var/lib/rabbitmq/mnesia/$(POD_NAME)
            - name: K8S_SERVICE_NAME
              value: rabbitmq-headless
            - name: K8S_HOSTNAME_SUFFIX
              value: .$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
            # 配置节点发现DNS
            - name: RABBITMQ_NODENAME
              value: rabbit@$(POD_NAME)$(K8S_HOSTNAME_SUFFIX)

4. 部署步骤

4.1 步骤1：创建命名空间

确保目标命名空间存在：

kubectl create namespace cluster-playground

4.2 步骤2：按顺序部署资源

创建Secret（集群认证基础）：

kubectl apply -f yaml/secret.yaml

创建ConfigMap（配置和脚本）：

kubectl apply -f yaml/configmap.yaml

创建ServiceAccount和RBAC（权限配置）：

kubectl apply -f yaml/service-account.yaml

创建Headless Service（DNS发现）：

kubectl apply -f yaml/headless.yaml

创建StatefulSet（核心部署）：

kubectl apply -f yaml/statefulsets.yaml

创建NodePort Service（外部访问）：

kubectl apply -f yaml/service.yaml

4.3 步骤3：验证部署

检查Pod状态：

kubectl get pods -n cluster-playground -l app=rabbitmq

检查服务状态：

kubectl get svc -n cluster-playground -l app=rabbitmq

查看Pod日志：

kubectl logs -n cluster-playground rabbitmq-0

5. 集群验证

5.1 健康检查

执行内置的健康检查脚本：

kubectl exec -n cluster-playground rabbitmq-0 -- bash /etc/rabbitmq/cluster-health.sh

5.2 水平扩展

增加副本数量：

kubectl scale statefulset rabbitmq --replicas=5 -n cluster-playground

6. 附录

6.1 完整编排文件内容

6.1.1 ConfigMap配置 (configmap.yaml)

kind: ConfigMap
apiVersion: v1
metadata:
  name: rabbitmq-config
  namespace: cluster-playground
data:
  cluster-health.sh: >-
    #!/bin/bash


    echo "=== RabbitMQ 集群功能验证（容器版本） ==="

    echo ""


    # 从环境变量读取认证参数

    RABBITMQ_USER="${RABBITMQ_USER:-admin}"

    RABBITMQ_PASS="${RABBITMQ_PASS:-Admin#123}"


    # 1. 创建队列

    echo "1. 创建测试队列..."

    rabbitmqadmin -u $RABBITMQ_USER -p $RABBITMQ_PASS declare queue
    name=cluster_test durable=true

    if [ $? -eq 0 ]; then
        echo "队列创建成功"
    else
        echo "队列创建失败"
        exit 1
    fi

    echo ""


    # 2. 发布消息

    echo "2. 发布测试消息..."

    rabbitmqadmin -u $RABBITMQ_USER -p $RABBITMQ_PASS publish
    exchange=amq.default routing_key=cluster_test payload="Hello from RabbitMQ
    Cluster!"

    echo "消息发布成功"

    echo ""


    # 3. 检查队列状态

    echo "3. 队列状态检查："

    rabbitmqadmin -u $RABBITMQ_USER -p $RABBITMQ_PASS list queues name messages
    consumers durable node

    echo ""


    # 4. 设置镜像队列策略

    echo "4. 设置高可用策略..."

    rabbitmqctl set_policy ha-all ".*"
    '{"ha-mode":"all","ha-sync-mode":"automatic"}' --priority 0 --apply-to
    queues

    echo "高可用策略已设置"

    echo ""


    # 5. 检查策略

    echo "5. 当前策略："

    rabbitmqctl list_policies

    echo ""


    # 6. 额外：检查集群状态（可选）

    echo "6. 集群状态："

    rabbitmqctl cluster_status | head -20
  rabbitmq.conf: |
    ## 基本配置
    listeners.tcp.default = 5672
    management.tcp.port = 15672

    ## 集群配置
    cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.k8s.address_type = hostname
    cluster_formation.k8s.service_name = rabbitmq-headless
    cluster_formation.node_cleanup.interval = 30

    ## 磁盘和内存
    disk_free_limit.absolute = 2GB
    vm_memory_high_watermark.absolute = 3GB

6.1.2 Headless Service (headless.yaml)

kind: Service
apiVersion: v1
metadata:
  name: rabbitmq-headless
  namespace: cluster-playground
  labels:
    app: rabbitmq
spec:
  ports:
    - name: amqp
      protocol: TCP
      port: 5672
      targetPort: 5672
    - name: clustering
      protocol: TCP
      port: 25672
      targetPort: 25672
    - name: epmd
      protocol: TCP
      port: 4369
      targetPort: 4369
  selector:
    app: rabbitmq
  clusterIP: None
  clusterIPs:
    - None
  type: ClusterIP

6.1.3 NodePort Service (service.yaml)

kind: Service
apiVersion: v1
metadata:
  name: rabbitmq-client
  namespace: cluster-playground
  labels:
    app: rabbitmq-client
spec:
  ports:
    - name: amqp
      protocol: TCP
      port: 5672
      targetPort: 5672
      nodePort: 30445
    - name: http
      protocol: TCP
      port: 15672
      targetPort: 15672
      nodePort: 31010
  selector:
    app: rabbitmq
  type: NodePort

6.1.4 StatefulSet配置 (statefulsets.yaml)

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: rabbitmq
  namespace: cluster-playground
  labels:
    app: rabbitmq
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      volumes:
        - name: config
          configMap:
            name: rabbitmq-config
            defaultMode: 420
        - name: data
          emptyDir: {}
      containers:
        - name: rabbitmq
          image: 'rabbitmq:3.9.21-management'
          command:
            - sh
            - '-c'
          args:
            - >
              # 启动插件

              rabbitmq-plugins enable --offline rabbitmq_management
              rabbitmq_peer_discovery_k8s


              # 启动RabbitMQ

              exec docker-entrypoint.sh rabbitmq-server
          ports:
            - name: amqp
              containerPort: 5672
              protocol: TCP
            - name: http
              containerPort: 15672
              protocol: TCP
            - name: clustering
              containerPort: 25672
              protocol: TCP
            - name: epmd
              containerPort: 4369
              protocol: TCP
          env:
            - name: RABBITMQ_DEFAULT_USER
              value:  admin
            - name: RABBITMQ_DEFAULT_PASS
              value:  Admin#123
            - name: RABBITMQ_ERLANG_COOKIE
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-erlang-cookie
                  key: .erlang.cookie
            - name: RABBITMQ_USE_LONGNAME
              value: 'true'
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: RABBITMQ_MNESIA_BASE
              value: /var/lib/rabbitmq/mnesia/$(POD_NAME)
            - name: K8S_SERVICE_NAME
              value: rabbitmq-headless
            - name: K8S_HOSTNAME_SUFFIX
              value: .$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
            - name: RABBITMQ_NODENAME
              value: rabbit@$(POD_NAME)$(K8S_HOSTNAME_SUFFIX)
          resources: {}
          volumeMounts:
            - name: config
              mountPath: /etc/rabbitmq/rabbitmq.conf
              subPath: rabbitmq.conf
            - name: config
              mountPath: /etc/rabbitmq/cluster-health.sh
              subPath: cluster-health.sh
            - name: data
              mountPath: /var/lib/rabbitmq
          livenessProbe:
            exec:
              command:
                - rabbitmq-diagnostics
                - ping
            initialDelaySeconds: 60
            timeoutSeconds: 10
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            exec:
              command:
                - rabbitmq-diagnostics
                - status
            initialDelaySeconds: 20
            timeoutSeconds: 5
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 10
      dnsPolicy: ClusterFirst
      serviceAccountName: rabbitmq-sa
      serviceAccount: rabbitmq-sa
      securityContext: {}
      imagePullSecrets:
        - name: harbor
      schedulerName: default-scheduler
  serviceName: rabbitmq-headless
  podManagementPolicy: OrderedReady