由 dawei
Redis Cluster 自动化安装，扩容和缩容
ASP站长网之前写过一篇基于Python的redis集群自动化安装的实现，基于纯命令的集群实现还是相当繁琐的，因此官方提供了redis-trib.rb这个工具
虽然官方的的redis-trib.rb提供了集群创建、 检查、 修复、均衡等命令行工具，之所个人接受不了redis-trib.rb，原因在于redis-trib.rb无法自定义实现集群中节点的主从关系。
比如ABCDEF6个节点，在创建集群的过程中必然要明确指定哪些是主，哪些是从，主从对应关系，可惜通过redis-trib.rb无法自定义控制，参考如下截图。
更多的时候，是需要明确指明哪些机器作为主节点，哪些作为从节点，redis-trib.rb做不到自动控制集群中的哪些机器（实例）作为主，哪些机器（实例）作为从。
如果使用redis-trib.rb，还需要解决ruby的环境依赖，因此个人不太接受使用redis-trib.rb搭建集群。
引用《Redis开发与运维》里面的原话：
如果部署节点使用不同的IP地址， redis-trib.rb会尽可能保证主从节点不分配在同一机器下， 因此会重新排序节点列表顺序。
节点列表顺序用于确定主从角色， 先主节点之后是从节点。
这说明：使用redis-trib.rb是无法人为地完全控制主从节点的分配的。
后面redis 5.0版本的Redis-cli --cluster已经实现了集群的创建，无需依赖redis-trib.rb，包括ruby环境，redis 5.0版本Redis-cli --cluster本身已经实现了集群等相关功能
但是基于纯命令本身还是比较复杂的,尤其是在较为复杂的生产环境，通过手动方式来创建集群，扩容或者缩容，会存在一系列的手工操作，以及一些不安全因素。
所以，自动化的集群创建 ，扩容以及缩容是有必要的。
测试环境
这里基于Python3，以redis-cli --cluster命令为基础，实现redis自动化集群，自动化扩容，自动化缩容
测试环境以单机多实例为示例，一共8个节点，
1，自动化集群的创建，6各节点（10001~10006）创建为3主（10001~10002）3从（10004~10006）的集群
2，集群的自动化扩容，增加新节点10007为主节点，同时添加10008为10007节点的slave节点
3，集群的自动化缩容，与2相反，移除集群中的10007以及其slave的10008节点
Redis集群创建
集群的本质是执行两组命令，一个是将主节点加入到集群中，一个是依次对主节点添加slave节点。
但是期间会涉及到找到各个节点id的逻辑，因此手动实现的话，比较繁琐。
主要命令如下：
################# create cluster #################
redis-cli --cluster create 127.0.0.1:10001 127.0.0.1:10002 127.0.0.1:10003 -a ****** --cluster-yes
################# add slave nodes #################
redis-cli --cluster add-node 127.0.0.1:10004 127.0.0.1:10001 --cluster-slave --cluster-master-id 6164025849a8ff9297664fc835bc851af5004f61 -a ******
redis-cli --cluster add-node 127.0.0.1:10005 127.0.0.1:10002 --cluster-slave --cluster-master-id 64e634307bdc339b503574f5a77f1b156c021358 -a ******
redis-cli --cluster add-node 127.0.0.1:10006 127.0.0.1:10003 --cluster-slave --cluster-master-id 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a -a ******
这里使用python创建的过程中打印出来redis-cli --cluster 命令的日志信息
[root@JD redis_install]# python3 create_redis_cluster.py
################# flush master/slave slots #################
################# create cluster #################
redis-cli --cluster create 127.0.0.1:10001 127.0.0.1:10002 127.0.0.1:10003 -a ****** --cluster-yes
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: 6164025849a8ff9297664fc835bc851af5004f61 127.0.0.1:10001
slots:[0-5460] (5461 slots) master
M: 64e634307bdc339b503574f5a77f1b156c021358 127.0.0.1:10002
slots:[5461-10922] (5462 slots) master
M: 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 127.0.0.1:10003
slots:[10923-16383] (5461 slots) master
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 127.0.0.1:10001)
M: 6164025849a8ff9297664fc835bc851af5004f61 127.0.0.1:10001
slots:[0-5460] (5461 slots) master
M: 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 127.0.0.1:10003
slots:[10923-16383] (5461 slots) master
M: 64e634307bdc339b503574f5a77f1b156c021358 127.0.0.1:10002
slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
0
################# add slave nodes #################
redis-cli --cluster add-node 127.0.0.1:10004 127.0.0.1:10001 --cluster-slave --cluster-master-id 6164025849a8ff9297664fc835bc851af5004f61 -a ******
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 127.0.0.1:10004 to cluster 127.0.0.1:10001
>>> Performing Cluster Check (using node 127.0.0.1:10001)
M: 6164025849a8ff9297664fc835bc851af5004f61 127.0.0.1:10001
slots:[0-5460] (5461 slots) master
M: 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 127.0.0.1:10003
slots:[10923-16383] (5461 slots) master
M: 64e634307bdc339b503574f5a77f1b156c021358 127.0.0.1:10002
slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:10004 to make it join the cluster.
Waiting for the cluster to join
>>> Configure node as replica of 127.0.0.1:10001.
[OK] New node added correctly.
0
redis-cli --cluster add-node 127.0.0.1:10005 127.0.0.1:10002 --cluster-slave --cluster-master-id 64e634307bdc339b503574f5a77f1b156c021358 -a ******
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 127.0.0.1:10005 to cluster 127.0.0.1:10002
>>> Performing Cluster Check (using node 127.0.0.1:10002)
M: 64e634307bdc339b503574f5a77f1b156c021358 127.0.0.1:10002
slots:[5461-10922] (5462 slots) master
S: 026f0179631f50ca858d46c2b2829b3af71af2c8 127.0.0.1:10004
slots: (0 slots) slave
replicates 6164025849a8ff9297664fc835bc851af5004f61
M: 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 127.0.0.1:10003
slots:[10923-16383] (5461 slots) master
M: 6164025849a8ff9297664fc835bc851af5004f61 127.0.0.1:10001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:10005 to make it join the cluster.
Waiting for the cluster to join
>>> Configure node as replica of 127.0.0.1:10002.
[OK] New node added correctly.
0
redis-cli --cluster add-node 127.0.0.1:10006 127.0.0.1:10003 --cluster-slave --cluster-master-id 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a -a ******
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 127.0.0.1:10006 to cluster 127.0.0.1:10003
>>> Performing Cluster Check (using node 127.0.0.1:10003)
M: 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 127.0.0.1:10003
slots:[10923-16383] (5461 slots) master
M: 64e634307bdc339b503574f5a77f1b156c021358 127.0.0.1:10002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 23e1871c4e1dc1047ce567326e74a6194589146c 127.0.0.1:10005
slots: (0 slots) slave
replicates 64e634307bdc339b503574f5a77f1b156c021358
M: 6164025849a8ff9297664fc835bc851af5004f61 127.0.0.1:10001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 026f0179631f50ca858d46c2b2829b3af71af2c8 127.0.0.1:10004
slots: (0 slots) slave
replicates 6164025849a8ff9297664fc835bc851af5004f61
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:10006 to make it join the cluster.
Waiting for the cluster to join
>>> Configure node as replica of 127.0.0.1:10003.
[OK] New node added correctly.
0
################# cluster nodes info: #################
8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 127.0.0.1:10003@20003 myself,master - 0 1575947748000 53 connected 10923-16383
64e634307bdc339b503574f5a77f1b156c021358 127.0.0.1:10002@20002 master - 0 1575947748000 52 connected 5461-10922
23e1871c4e1dc1047ce567326e74a6194589146c 127.0.0.1:10005@20005 slave 64e634307bdc339b503574f5a77f1b156c021358 0 1575947746000 52 connected
6164025849a8ff9297664fc835bc851af5004f61 127.0.0.1:10001@20001 master - 0 1575947748103 51 connected 0-5460
026f0179631f50ca858d46c2b2829b3af71af2c8 127.0.0.1:10004@20004 slave 6164025849a8ff9297664fc835bc851af5004f61 0 1575947749000 51 connected
9f265545ebb799d2773cfc20c71705cff9d733ae 127.0.0.1:10006@20006 slave 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 0 1575947749105 53 connected
[root@JD redis_install]#
Redis集群扩容
redis扩容主要分为两步：
1，增加主节点，同时为主节点增加从节点。
2，重新分配slot到新增加的master节点上。
主要命令如下：
增加主节点到集群中
redis-cli --cluster add-node 127.0.0.1:10007 127.0.0.1:10001 -a ******
为增加的主节点添加从节点
redis-cli --cluster add-node 127.0.0.1:10008 127.0.0.1:10007 --cluster-slave --cluster-master-id 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 -a ******
重新分片slot
############################ execute reshard #########################################
redis-cli -a redis@password --cluster reshard 127.0.0.1:10001 --cluster-from 6164025849a8ff9297664fc835bc851af5004f61 --cluster-to 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 --cluster-slots 1365 --cluster-yes --cluster-timeout 50000 --cluster-pipeline 10000 --cluster-replace >/dev/null 2>&1
############################ execute reshard #########################################
redis-cli -a redis@password --cluster reshard 127.0.0.1:10002 --cluster-from 64e634307bdc339b503574f5a77f1b156c021358 --cluster-to 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 --cluster-slots 1365 --cluster-yes --cluster-timeout 50000 --cluster-pipeline 10000 --cluster-replace >/dev/null 2>&1
############################ execute reshard #########################################
redis-cli -a redis@password --cluster reshard 127.0.0.1:10003 --cluster-from 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a --cluster-to 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 --cluster-slots 1365 --cluster-yes --cluster-timeout 50000 --cluster-pipeline 10000 --cluster-replace >/dev/null 2>&1
################# cluster nodes info: #################
026f0179631f50ca858d46c2b2829b3af71af2c8 127.0.0.1:10004@20004 slave 6164025849a8ff9297664fc835bc851af5004f61 0 1575960493000 64 connected
9f265545ebb799d2773cfc20c71705cff9d733ae 127.0.0.1:10006@20006 slave 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 0 1575960493849 66 connected
64e634307bdc339b503574f5a77f1b156c021358 127.0.0.1:10002@20002 master - 0 1575960494852 65 connected 6826-10922
23e1871c4e1dc1047ce567326e74a6194589146c 127.0.0.1:10005@20005 slave 64e634307bdc339b503574f5a77f1b156c021358 0 1575960492000 65 connected
4854375c501c3dbfb4e2d94d50e62a47520c4f12 127.0.0.1:10008@20008 slave 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 0 1575960493000 67 connected
8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 127.0.0.1:10003@20003 master - 0 1575960493000 66 connected 12288-16383
3645e00a8ec3a902bd6effb4fc20c56a00f2c982 127.0.0.1:10007@20007 myself,master - 0 1575960493000 67 connected 0-1364 5461-6825 10923-12287
6164025849a8ff9297664fc835bc851af5004f61 127.0.0.1:10001@20001 master - 0 1575960492848 64 connected 1365-5460
可见新加的节点成功重新分配了slot，集群扩容成功。
这里有几个需要注意的两个问题，如果是自动化安装的话：
1，add-node之后（不管是柱节点还是从节点），要sleep足够长的时间（这里是20秒），让集群中所有的节点都meet到新节点，否则会扩容失败
2，新节点的reshard之后要sleep足够长的时间（这里是20秒），否则继续reshard其他节点的slot会导致上一个reshared失败
整个过程如下
[root@JD redis_install]# python3 create_redis_cluster.py
#########################cleanup instance#################################
#########################add node into cluster#################################
redis-cli --cluster add-node 127.0.0.1:10007 127.0.0.1:10001 -a redis@password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 127.0.0.1:10007 to cluster 127.0.0.1:10001
>>> Performing Cluster Check (using node 127.0.0.1:10001)
M: 6164025849a8ff9297664fc835bc851af5004f61 127.0.0.1:10001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 9f265545ebb799d2773cfc20c71705cff9d733ae 127.0.0.1:10006
slots: (0 slots) slave
replicates 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a
M: 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 127.0.0.1:10003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 026f0179631f50ca858d46c2b2829b3af71af2c8 127.0.0.1:10004
slots: (0 slots) slave
replicates 6164025849a8ff9297664fc835bc851af5004f61
S: 23e1871c4e1dc1047ce567326e74a6194589146c 127.0.0.1:10005
slots: (0 slots) slave
replicates 64e634307bdc339b503574f5a77f1b156c021358
M: 64e634307bdc339b503574f5a77f1b156c021358 127.0.0.1:10002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:10007 to make it join the cluster.
[OK] New node added correctly.
0
redis-cli --cluster add-node 127.0.0.1:10008 127.0.0.1:10007 --cluster-slave --cluster-master-id 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 -a ******
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 127.0.0.1:10008 to cluster 127.0.0.1:10007
>>> Performing Cluster Check (using node 127.0.0.1:10007)
M: 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 127.0.0.1:10007
slots: (0 slots) master
S: 026f0179631f50ca858d46c2b2829b3af71af2c8 127.0.0.1:10004
slots: (0 slots) slave
replicates 6164025849a8ff9297664fc835bc851af5004f61
S: 9f265545ebb799d2773cfc20c71705cff9d733ae 127.0.0.1:10006
slots: (0 slots) slave
replicates 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a
M: 64e634307bdc339b503574f5a77f1b156c021358 127.0.0.1:10002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 23e1871c4e1dc1047ce567326e74a6194589146c 127.0.0.1:10005
slots: (0 slots) slave
replicates 64e634307bdc339b503574f5a77f1b156c021358
M: 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 127.0.0.1:10003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: 6164025849a8ff9297664fc835bc851af5004f61 127.0.0.1:10001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:10008 to make it join the cluster.
Waiting for the cluster to join
>>> Configure node as replica of 127.0.0.1:10007.
[OK] New node added correctly.
0
#########################reshard slots#################################
############################ execute reshard #########################################
redis-cli -a redis@password --cluster reshard 127.0.0.1:10001 --cluster-from 6164025849a8ff9297664fc835bc851af5004f61 --cluster-to 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 --cluster-slots 1365 --cluster-yes --cluster-timeout 50000 --cluster-pipeline 10000 --cluster-replace >/dev/null 2>&1
############################ execute reshard #########################################
redis-cli -a redis@password --cluster reshard 127.0.0.1:10002 --cluster-from 64e634307bdc339b503574f5a77f1b156c021358 --cluster-to 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 --cluster-slots 1365 --cluster-yes --cluster-timeout 50000 --cluster-pipeline 10000 --cluster-replace >/dev/null 2>&1
############################ execute reshard #########################################
redis-cli -a redis@password --cluster reshard 127.0.0.1:10003 --cluster-from 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a --cluster-to 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 --cluster-slots 1365 --cluster-yes --cluster-timeout 50000 --cluster-pipeline 10000 --cluster-replace >/dev/null 2>&1
################# cluster nodes info: #################
026f0179631f50ca858d46c2b2829b3af71af2c8 127.0.0.1:10004@20004 slave 6164025849a8ff9297664fc835bc851af5004f61 0 1575960493000 64 connected
9f265545ebb799d2773cfc20c71705cff9d733ae 127.0.0.1:10006@20006 slave 8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 0 1575960493849 66 connected
64e634307bdc339b503574f5a77f1b156c021358 127.0.0.1:10002@20002 master - 0 1575960494852 65 connected 6826-10922
23e1871c4e1dc1047ce567326e74a6194589146c 127.0.0.1:10005@20005 slave 64e634307bdc339b503574f5a77f1b156c021358 0 1575960492000 65 connected
4854375c501c3dbfb4e2d94d50e62a47520c4f12 127.0.0.1:10008@20008 slave 3645e00a8ec3a902bd6effb4fc20c56a00f2c982 0 1575960493000 67 connected
8b75325c59a7242344d0ebe5ee1e0068c66ffa2a 127.0.0.1:10003@20003 master - 0 1575960493000 66 connected 12288-16383
3645e00a8ec3a902bd6effb4fc20c56a00f2c982 127.0.0.1:10007@20007 myself,master - 0 1575960493000 67 connected 0-1364 5461-6825 10923-12287
6164025849a8ff9297664fc835bc851af5004f61 127.0.0.1:10001@20001 master - 0 1575960492848 64 connected 1365-5460
[root@JD redis_install]#