K8S下redis哨兵集群使用secret隐藏configmap内明文密码方案详解

#作者：朱雷

文章目录 一、背景环境及方案说明1.1、环境说明1.2、方案一：使用配置文件设置密码1.3、方案二：使用args 的命令行传参设置密码 二、redis secret configmap deployment参考2.1 创建secret-redis.yaml参考2.2 修改configmap配置参考2.2.1 哨兵节点修改（每个节点都修改）2.2.2 主从节点配置修改2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥） 2.3 修改deployment配置参考2.3.1 master&slaves节点的deployment的yaml修改2.3.2 哨兵节点的deployment的yaml修改2.3.4 镜像环境变量参考 三、测试是否生效四、 注意事项

一、背景环境及方案说明

Redis 哨兵configmap里包含明文密码，需要处理不在configmap内显示明文密码。

1.1、环境说明

方案基于Redis-5.0.14 哨兵版本（5.x，6.x版本兼容） 方案基于redis-sentinel-exporter-5.0.8版本 方案基于容器环境变量 使用两种方案的其中任意一种均可实现

1.2、方案一：使用配置文件设置密码

参照 2.2.1 和2.2.2 修改configmap 参照 2.3.1 和2.3.2 修改deployment

1.3、方案二：使用args 的命令行传参设置密码

参照 2.2.3.1 和2.2.3.2 修改configmap 参照 2.3.3.1 和2.3.3.2 修改deployment

二、redis secret configmap deployment参考 2.1 创建secret-redis.yaml参考 ${}内为redis密码的base64的编码，若认证和同步密码不一致分别定义即可 apiVersion: v1 data: password: ${aGFyYm9yMjM0NSM=} kind: Secret metadata: name: redis-auth-secret namespace: paas-middleware 2.2 修改configmap配置参考 2.2.1 哨兵节点修改（每个节点都修改） apiVersion: v1 data: redis-docker-entrypoint.sh: | #!/bin/bash if [ ! -f "/redis-conf/redis.conf" ]; then cp /etc/redis/redis.conf /redis-conf/redis.conf **echo -e "sentinel auth-pass mymaster ${REDIS_PASSWORD}" >> /redis-conf/redis.conf fi** redis-sentinel /redis-conf/redis.conf $@ redis.conf: | port 26379 protected-mode no daemonize no sentinel monitor mymaster 169.169.164.253 6379 2 sentinel down-after-milliseconds mymaster 15000 sentinel failover-timeout mymaster 60000 sentinel deny-scripts-reconfig yes sentinel parallel-syncs mymaster 2 sentinel auth-pass mymaster somepassword # 删除这行配置 kind: ConfigMap metadata: labels: app: redis-base-1 type: redis name: redis-base-1-sentinel-1 namespace: paas-middleware 每个哨兵的configmap 都修改下， 有****不带删除线的为新增行 2.2.2 主从节点配置修改 apiVersion: v1 data: redis-docker-entrypoint.sh: | #!/bin/bash if [ ! -f "/redis-conf/redis.conf" ]; then cp /etc/redis/redis.conf /redis-conf/redis.conf **echo -e "masterauth ${REDIS_MASTER_PASSWORD}" >> /redis-conf/redis.conf echo -e "requirepass ${REDIS_PASSWORD}" >> /redis-conf/redis.conf** fi redis-server /redis-conf/redis.conf $@ redis.conf: | bind 0.0.0.0 :: port 6379 daemonize no protected-mode no timeout 300 tcp-keepalive 300 replica-read-only yes replica-serve-stale-data yes maxclients 20000 maxmemory 0 maxmemory-policy noeviction masterauth somepassword # 删除此行配置 requirepass somepassword # 删除此行配置 rename-command FLUSHALL "" dir "/data/" pidfile "/data/redis.pid" logfile "/data/redis.log" kind: ConfigMap metadata: labels: app: redis-base-1 type: redis name: redis-base-1-master namespace: paas-middleware 所有主从configmap配置文件都修改， 有****不带删除线的为新增行 2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥） 以下为哨兵节点configmap 修改 apiVersion: v1 data: redis-docker-entrypoint.sh: | #!/bin/bash if [ ! -f "/redis-conf/redis.conf" ]; then cp /etc/redis/redis.conf /redis-conf/redis.conf fi redis-sentinel /redis-conf/redis.conf $@ redis.conf: | port 26379 protected-mode no daemonize no sentinel monitor mymaster 169.169.164.253 6379 2 sentinel down-after-milliseconds mymaster 15000 sentinel failover-timeout mymaster 60000 sentinel deny-scripts-reconfig yes sentinel parallel-syncs mymaster 2 sentinel auth-pass mymaster somepassword # 删除这行配置 kind: ConfigMap metadata: labels: app: redis-base-1 type: redis name: redis-base-1-sentinel-1 namespace: paas-middleware 每个哨兵的configmap 都修改下， 有****不带删除线的为新增行 以下为主从节点configmap 修改 下面为主从节点实例configmap修改，有****不带删除线为新增行 apiVersion: v1 data: redis-docker-entrypoint.sh: | #!/bin/bash if [ ! -f "/redis-conf/redis.conf" ]; then cp /etc/redis/redis.conf /redis-conf/redis.conf fi redis-server /redis-conf/redis.conf $@ redis.conf: | bind 0.0.0.0 :: port 6379 daemonize no protected-mode no timeout 300 tcp-keepalive 300 replica-read-only yes replica-serve-stale-data yes maxclients 20000 maxmemory 0 maxmemory-policy noeviction masterauth somepassword # 删除此行配置 requirepass somepassword # 删除此行配置 rename-command FLUSHALL "" dir "/data/" pidfile "/data/redis.pid" logfile "/data/redis.log" kind: ConfigMap metadata: labels: app: redis-base-1 type: redis name: redis-base-1-master namespace: paas-middleware 2.3 修改deployment配置参考 2.3.1 master&slaves节点的deployment的yaml修改 所有主从节点配置文件都修改， 有**xxx**为新增行 apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" labels: app: redis-base-1 type: redis name: redis-base-1-master namespace: paas-middleware spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: redis-base-1 name: redis-base-1-master servicename: redis-base-1 type: redis withexporter: "yes" strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: annotations: prometheus.io/port: "9121" prometheus.io/scrape: "true" labels: app: redis-base-1 name: redis-base-1-master servicename: redis-base-1 type: redis withexporter: "yes" spec: containers: - args: - --replica-announce-ip - 169.169.164.253 - --replica-announce-port - "6379" command: - /etc/redis/redis-docker-entrypoint.sh image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest imagePullPolicy: Always name: redis **env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-auth-secret key: password - name: REDIS_MASTER_PASSWORD valueFrom: secretKeyRef: name: redis-auth-secret key: password** ports: - containerPort: 6379 name: client protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/redis/ name: config - mountPath: /data name: data - mountPath: /redis-conf name: actual-config - args: - --redis.addr - redis://localhost:6379 - --redis.password - somepassword #密码替换成 $(REDIS_PASSWORD) 变量 - **$(REDIS_PASSWORD)** - --web.listen-address - 0.0.0.0:9121 image: harbor.somedomain/paas_middleware/redis-sentinel-exporter-5.0.8:latest imagePullPolicy: Always name: redis-exporter **env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-auth-secret key: password** ports: - containerPort: 9121 name: redis-exporter protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/hostname: 10.179.75.111 restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 509 name: redis-base-1-master name: config - hostPath: path: /data/redis/redis-base-1-master/data type: "" name: data - hostPath: path: /data/redis/redis-base-1-master/redis-conf type: "" name: actual-config 2.3.2 哨兵节点的deployment的yaml修改 所有哨兵节点配置文件都修改， 有****为新增行 apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" labels: app: redis-base-1 type: redis name: redis-base-1-sentinel-1 namespace: paas-middleware spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: redis-base-1 name: redis-base-1-sentinel-1 role: sentinel type: redis withexporter: "no" strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: app: redis-base-1 name: redis-base-1-sentinel-1 role: sentinel type: redis withexporter: "no" spec: containers: - args: - --sentinel - announce-ip - 169.169.196.242 - --replica-announce-port - "26379" command: - /etc/redis/redis-docker-entrypoint.sh image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest imagePullPolicy: Always name: redis **env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-auth-secret key: password - name: REDIS_MASTER_PASSWORD valueFrom: secretKeyRef: name: redis-auth-secret key: password** ports: - containerPort: 26379 name: client protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/redis/ name: config - mountPath: /data name: data - mountPath: /redis-conf name: actual-config dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/hostname: 10.179.75.111 restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 509 name: redis-base-1-sentinel-1 name: config - hostPath: path: /data/redis/redis-base-1-sentinel-1/data type: "" name: data - hostPath: path: /data/redis/redis-base-1-sentinel-1/redis-conf type: "" name: actual-config status: availableReplicas: 1 conditions: - lastTransitionTime: "2023-11-09T03:25:41Z" lastUpdateTime: "2023-11-09T03:25:43Z" message: ReplicaSet "redis-base-1-sentinel-1-668c76f9bc" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing - lastTransitionTime: "2024-07-21T16:48:34Z" lastUpdateTime: "2024-07-21T16:48:34Z" message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available observedGeneration: 3 readyReplicas: 1 replicas: 1 updatedReplicas: 1

2.3.3 使用命令行参数指定密码（本小节与上面两小节互斥）

哨兵节点deployment 修改 所有哨兵节点配置文件都修改， 有****不带删除线的为新增行 apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" labels: app: redis-base-1 type: redis name: redis-base-1-sentinel-1 namespace: paas-middleware spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: redis-base-1 name: redis-base-1-sentinel-1 role: sentinel type: redis withexporter: "no" strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: app: redis-base-1 name: redis-base-1-sentinel-1 role: sentinel type: redis withexporter: "no" spec: containers: - args: - --sentinel - announce-ip - 169.169.196.242 - --replica-announce-port - "26379" - --sentinel - auth-pass - mymaster - $(REDIS_PASSWORD) command: - /etc/redis/redis-docker-entrypoint.sh image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest imagePullPolicy: Always name: redis env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-auth-secret key: password - name: REDIS_MASTER_PASSWORD valueFrom: secretKeyRef: name: redis-auth-secret key: password ports: - containerPort: 26379 name: client protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/redis/ name: config - mountPath: /data name: data - mountPath: /redis-conf name: actual-config dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/hostname: 10.179.75.111 restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 509 name: redis-base-1-sentinel-1 name: config - hostPath: path: /data/redis/redis-base-1-sentinel-1/data type: "" name: data - hostPath: path: /data/redis/redis-base-1-sentinel-1/redis-conf type: "" name: actual-config status: availableReplicas: 1 conditions: - lastTransitionTime: "2023-11-09T03:25:41Z" lastUpdateTime: "2023-11-09T03:25:43Z" message: ReplicaSet "redis-base-1-sentinel-1-668c76f9bc" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing - lastTransitionTime: "2024-07-21T16:48:34Z" lastUpdateTime: "2024-07-21T16:48:34Z" message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available observedGeneration: 3 readyReplicas: 1 replicas: 1 updatedReplicas: 1

2、以下为主从实例deployment 配置修改 所有主从节点配置文件都修改， 有****不带删除线的为新增行

apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" labels: app: redis-base-1 type: redis name: redis-base-1-master namespace: paas-middleware spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: redis-base-1 name: redis-base-1-master servicename: redis-base-1 type: redis withexporter: "yes" strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: annotations: prometheus.io/port: "9121" prometheus.io/scrape: "true" labels: app: redis-base-1 name: redis-base-1-master servicename: redis-base-1 type: redis withexporter: "yes" spec: containers: - args: - --replica-announce-ip - 169.169.164.253 - --replica-announce-port - "6379" - --**requirepass - $(REDIS_PASSWORD) - -- masterauth - $(REDIS_MASTER_PASSWORD)** command: - /etc/redis/redis-docker-entrypoint.sh image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest imagePullPolicy: Always name: redis **env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-auth-secret key: password - name: REDIS_MASTER_PASSWORD valueFrom: secretKeyRef: name: redis-auth-secret key: password** ports: - containerPort: 6379 name: client protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/redis/ name: config - mountPath: /data name: data - mountPath: /redis-conf name: actual-config - args: - --redis.addr - redis://localhost:6379 - --redis.password - somepassword #密码替换成 $(REDIS_PASSWORD) 变量 - **$(REDIS_PASSWORD)** - --web.listen-address - 0.0.0.0:9121 image: harbor.somedomain/paas_middleware/redis-sentinel-exporter-5.0.8:latest imagePullPolicy: Always name: redis-exporter **env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-auth-secret key: password** ports: - containerPort: 9121 name: redis-exporter protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/hostname: 10.179.75.111 restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 509 name: redis-base-1-master name: config - hostPath: path: /data/redis/redis-base-1-master/data type: "" name: data - hostPath: path: /data/redis/redis-base-1-master/redis-conf type: "" name: actual-config 2.3.4 镜像环境变量参考

hub.docker /r/bitnami/redis#configuration github /oliver006/redis_exporter#flags

三、测试是否生效

Master节点 Slave节点 哨兵节点 测试redis-sentinel-exporter 指标抓取

四、 注意事项 所有节点configmap和deployment yaml 配置文件都按照上面修改别遗漏修改完先在测试环境验证没有问题，再连接到连接哨兵集群进行读写测试