设为首页收藏本站
开启辅助访问 软件测试门户软件测试培训软件测试论坛测试解决方案文章资料精选软件测试博客软件测试招聘

51Testing软件测试论坛

 找回密码
 (注-册)加入51Testing

QQ登录

只需一步，快速开始

微信登录，快人一步

手机号码，快捷登录

51Testing软件测试论坛 »软件测试论坛 › [软件测试综合栏目] › [软件测试新手上路] › 高阶测试人员：Redis-Shake工具研究（下）
返回列表 发新帖
查看: 1452|回复: 0
打印 上一主题 下一主题

[原创] 高阶测试人员：Redis-Shake工具研究（下）

[复制链接]
lsekfe
  • TA的每日心情
    无聊
    10 小时前

    • 签到天数: 1052 天

    连续签到: 2 天

    [LV.10]测试总司令
    跳转到指定楼层
    1#
    发表于 2021-7-7 09:35:31 | 只看该作者 回帖奖励 |正序浏览 |阅读模式
    待实例启动后，用同样的方式进行集群的构建。
    1. root@5141467f762d:/data# redis-cli --cluster create 172.25.253.0:8000 172.25.253.0:8001 172.25.253.0:8002 172.25.253.0:8003 172.25.253.0:8004 172.25.253.0:8005 172.25.253.0:8006 172.25.253.0:8007 172.25.253.0:8008 172.25.253.0:8009 --cluster-replicas 1 -a 123456
    2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
    3. >>> Performing hash slots allocation on 10 nodes...
    4. Master[0] -> Slots 0 - 3276
    5. Master[1] -> Slots 3277 - 6553
    6. Master[2] -> Slots 6554 - 9829
    7. Master[3] -> Slots 9830 - 13106
    8. Master[4] -> Slots 13107 - 16383
    9. Adding replica 172.25.253.0:8006 to 172.25.253.0:8000
    10. Adding replica 172.25.253.0:8007 to 172.25.253.0:8001
    11. Adding replica 172.25.253.0:8008 to 172.25.253.0:8002
    12. Adding replica 172.25.253.0:8009 to 172.25.253.0:8003
    13. Adding replica 172.25.253.0:8005 to 172.25.253.0:8004
    14. >>> Trying to optimize slaves allocation for anti-affinity
    15. [WARNING] Some slaves are in the same host as their master
    16. M: e360847d352d53946fe3d8df45eb85d345f5e03b 172.25.253.0:8000
    17.    slots:[0-3276] (3277 slots) master
    18. M: 23a3550fdf50cb86a6d12d56627757318ea9ad67 172.25.253.0:8001
    19.    slots:[3277-6553] (3277 slots) master
    20. M: 054e696ec5639f8bf68293f63a2d1e6507062e47 172.25.253.0:8002
    21.    slots:[6554-9829] (3276 slots) master
    22. M: 4f9cc0b67da6a31ca9a1f596154d1d4e02932086 172.25.253.0:8003
    23.    slots:[9830-13106] (3277 slots) master
    24. M: 708043e6b8b70953cc162b733d15be3fc951efca 172.25.253.0:8004
    25.    slots:[13107-16383] (3277 slots) master
    26. S: c85a6dd73665472fdd76d83b78bf3093669b2c4e 172.25.253.0:8005
    27.    replicates 054e696ec5639f8bf68293f63a2d1e6507062e47
    28. S: 478c2ad79e506106bc5471df3e545f4624c60b43 172.25.253.0:8006
    29.    replicates 23a3550fdf50cb86a6d12d56627757318ea9ad67
    30. S: f9bad9c5b76e9609c16a503dbd9b23f4d62d5c9d 172.25.253.0:8007
    31.    replicates 708043e6b8b70953cc162b733d15be3fc951efca
    32. S: 0c14761377ed09d955e06a8e7bcd8354494bc711 172.25.253.0:8008
    33.    replicates 4f9cc0b67da6a31ca9a1f596154d1d4e02932086
    34. S: 236c13a695d926a2b739b90092de49a8089c30cc 172.25.253.0:8009
    35.    replicates e360847d352d53946fe3d8df45eb85d345f5e03b
    36. Can I set the above configuration? (type 'yes' to accept): yes
    37. >>> Nodes configuration updated
    38. >>> Assign a different config epoch to each node
    39. >>> Sending CLUSTER MEET messages to join the cluster
    40. Waiting for the cluster to join
    41. .
    42. >>> Performing Cluster Check (using node 172.25.253.0:8000)
    43. M: e360847d352d53946fe3d8df45eb85d345f5e03b 172.25.253.0:8000
    44.    slots:[0-3276] (3277 slots) master
    45.    1 additional replica(s)
    46. M: 23a3550fdf50cb86a6d12d56627757318ea9ad67 172.25.253.0:8001
    47.    slots:[3277-6553] (3277 slots) master
    48.    1 additional replica(s)
    49. S: c85a6dd73665472fdd76d83b78bf3093669b2c4e 172.25.253.0:8005
    50.    slots: (0 slots) slave
    51.    replicates 054e696ec5639f8bf68293f63a2d1e6507062e47
    52. S: f9bad9c5b76e9609c16a503dbd9b23f4d62d5c9d 172.25.253.0:8007
    53.    slots: (0 slots) slave
    54.    replicates 708043e6b8b70953cc162b733d15be3fc951efca
    55. S: 478c2ad79e506106bc5471df3e545f4624c60b43 172.25.253.0:8006
    56.    slots: (0 slots) slave
    57.    replicates 23a3550fdf50cb86a6d12d56627757318ea9ad67
    58. S: 0c14761377ed09d955e06a8e7bcd8354494bc711 172.25.253.0:8008
    59.    slots: (0 slots) slave
    60.    replicates 4f9cc0b67da6a31ca9a1f596154d1d4e02932086
    61. S: 236c13a695d926a2b739b90092de49a8089c30cc 172.25.253.0:8009
    62.    slots: (0 slots) slave
    63.    replicates e360847d352d53946fe3d8df45eb85d345f5e03b
    64. M: 708043e6b8b70953cc162b733d15be3fc951efca 172.25.253.0:8004
    65.    slots:[13107-16383] (3277 slots) master
    66.    1 additional replica(s)
    67. M: 054e696ec5639f8bf68293f63a2d1e6507062e47 172.25.253.0:8002
    68.    slots:[6554-9829] (3276 slots) master
    69.    1 additional replica(s)
    70. M: 4f9cc0b67da6a31ca9a1f596154d1d4e02932086 172.25.253.0:8003
    71.    slots:[9830-13106] (3277 slots) master
    72.    1 additional replica(s)
    73. [OK] All nodes agree about slots configuration.
    74. >>> Check for open slots...
    75. >>> Check slots coverage...
    76. [OK] All 16384 slots covered.
    复制代码

    测试数据写入
    　　我们需要给源集群写入测试数据，然后才能进行数据迁移。数据写入脚本如下：
    1. for i in `seq 1000000`;
    2. do
    3. redis-cli -c -h 127.0.0.1 -p 7000 -a 123456 set k:$i v:$i
    4. done
    复制代码

    可以看出，上面的脚本共写入了1000000条数据。Redis集群会根据key的hash进行数据分区，如下：


    　　此外，为了测试数据过滤功能，还额外写入了10W条其他key，如下：


    　　Redis-Shake测试
    　　首先，在github上下载最新的redis-shake-v2.0.3.tar.gz并解压。
    1. /mnt/c/Users/wangy ? tar -xvf redis-shake-v2.0.3.tar.gz
    2. ./._redis-shake-v2.0.3
    3. ./redis-shake-v2.0.3/
    4. ./redis-shake-v2.0.3/._.DS_Store
    5. ./redis-shake-v2.0.3/.DS_Store
    6. ./redis-shake-v2.0.3/ChangeLog
    7. ./redis-shake-v2.0.3/redis-shake.linux
    8. ./redis-shake-v2.0.3/redis-shake.conf
    9. ./redis-shake-v2.0.3/redis-shake.windows
    10. ./redis-shake-v2.0.3/._stop.sh
    11. ./redis-shake-v2.0.3/stop.sh
    12. ./redis-shake-v2.0.3/._start.sh
    13. ./redis-shake-v2.0.3/start.sh
    14. ./redis-shake-v2.0.3/redis-shake.darwin
    复制代码
    打开其中的redis-shake.conf进行修改，下面仅列出与本次测试相关的配置项。
    1. # 源端redis的类型，支持standalone，sentinel，cluster和proxy四种模式，注意：目前proxy只用于rump模式。
    2. source.type = cluster
    3. # 源redis地址。
    4. source.address = 172.25.253.0:7002;172.25.253.0:7000;172.25.253.0:7001;172.25.253.0:7003;172.25.253.0:7040;172.25.253.0:7005
    5. # password of db/proxy. even if type is sentinel.
    6. source.password_raw = 123456
    7. # auth type, don't modify it
    8. source.auth_type = auth
    9. # 目的redis的类型，支持standalone，sentinel，cluster和proxy四种模式。
    10. target.type = cluster
    11. target.address = 172.25.253.0:8000;172.25.253.0:8001;172.25.253.0:8002;172.25.253.0:8003;172.25.253.0:8004
    12. # password of db/proxy. even if type is sentinel.
    13. target.password_raw = 123456
    14. # auth type, don't modify it
    15. target.auth_type = auth
    16. # 支持按前缀过滤key，只让指定前缀的key通过，分号分隔。比如指定abc，将会通过abc, abc1, abcxxx
    17. filter.key.whitelist = k:
    复制代码
    配置好之后，使用下面的命令启动：
    1. /mnt/c/Users/wangy/redis-shake-v2.0.3 ? ./redis-shake.linux -type=sync -conf=redis-shake.conf
    2. 2021/03/20 15:55:10 [WARN] source.auth_type[auth] != auth
    3. 2021/03/20 15:55:10 [WARN] target.auth_type[auth] != auth
    4. 2021/03/20 15:55:10 [INFO] the target redis type is cluster, only pass db0
    5. 2021/03/20 15:55:10 [INFO] source rdb[172.25.253.0:7002] checksum[yes]
    6. 2021/03/20 15:55:10 [INFO] source rdb[172.25.253.0:7000] checksum[yes]
    7. 2021/03/20 15:55:10 [INFO] source rdb[172.25.253.0:7001] checksum[yes]
    8. 2021/03/20 15:55:10 [WARN]
    9. ______________________________
    10. \                             \           _         ______ |
    11. \                             \        /   \___-=O'/|O'/__|
    12.   \   RedisShake, here we go !! \_______\          / | /    )
    13.   /                             /        '/-==__ _/__|/__=-|  -GM
    14. /        Alibaba Cloud        /         *             \ | |
    15. /                             /                        (o)
    16. ------------------------------
    17. if you have any problem, please visit https://github.com/alibaba/RedisShake/wiki/FAQ

    19. 2021/03/20 15:55:10 [INFO] redis-shake configuration: {"ConfVersion":1,"Id":"redis-shake","LogFile":"","LogLevel":"info","SystemProfile":9310,"HttpProfile":9320,"Parallel":32,"SourceType":"cluster","SourceAddress":"172.25.253.0:7002;172.25.253.0:7000;172.25.253.0:7001","SourcePasswordRaw":"***","SourcePasswordEncoding":"***","SourceAuthType":"auth","SourceTLSEnable":false,"SourceRdbInput":["local"],"SourceRdbParallel":3,"SourceRdbSpecialCloud":"","TargetAddress":"172.25.253.0:8000;172.25.253.0:8001;172.25.253.0:8002;172.25.253.0:8003;172.25.253.0:8004","TargetPasswordRaw":"***","TargetPasswordEncoding":"***","TargetDBString":"-1","TargetAuthType":"auth","TargetType":"cluster","TargetTLSEnable":false,"TargetRdbOutput":"local_dump","TargetVersion":"5.0.12","FakeTime":"","KeyExists":"none","FilterDBWhitelist":["0"],"FilterDBBlacklist":[],"FilterKeyWhitelist":[],"FilterKeyBlacklist":[],"FilterSlot":[],"FilterLua":false,"BigKeyThreshold":524288000,"Metric":true,"MetricPrintLog":false,"SenderSize":104857600,"SenderCount":4095,"SenderDelayChannelSize":65535,"KeepAlive":0,"PidPath":"","ScanKeyNumber":50,"ScanSpecialCloud":"","ScanKeyFile":"","Qps":200000,"ResumeFromBreakPoint":false,"Psync":true,"NCpu":0,"HeartbeatUrl":"","HeartbeatInterval":10,"HeartbeatExternal":"","HeartbeatNetworkInterface":"","ReplaceHashTag":false,"ExtraInfo":false,"SockFileName":"","SockFileSize":0,"FilterKey":null,"FilterDB":"","Rewrite":false,"SourceAddressList":["172.25.253.0:7002","172.25.253.0:7000","172.25.253.0:7001"],"TargetAddressList":["172.25.253.0:8000","172.25.253.0:8001","172.25.253.0:8002","172.25.253.0:8003","172.25.253.0:8004"],"SourceVersion":"5.0.12","HeartbeatIp":"127.0.0.1","ShiftTime":0,"TargetReplace":true,"TargetDB":-1,"Version":"develop,d7cb0297f121e2f441d3636ef363871aa2ab5e25,go1.14.4,2020-07-24_17:19:07","Type":"sync"}
    20. 2021/03/20 15:55:10 [INFO] DbSyncer[0] starts syncing data from 172.25.253.0:7002 to [172.25.253.0:8000 172.25.253.0:8001 172.25.253.0:8002 172.25.253.0:8003 172.25.253.0:8004] with http[9323], enableResumeFromBreakPoint[false], slot boundary[-1, -1]
    21. 2021/03/20 15:55:10 [INFO] DbSyncer[2] starts syncing data from 172.25.253.0:7001 to [172.25.253.0:8000 172.25.253.0:8001 172.25.253.0:8002 172.25.253.0:8003 172.25.253.0:8004] with http[9323], enableResumeFromBreakPoint[false], slot boundary[-1, -1]
    22. 2021/03/20 15:55:10 [INFO] DbSyncer[1] starts syncing data from 172.25.253.0:7000 to [172.25.253.0:8000 172.25.253.0:8001 172.25.253.0:8002 172.25.253.0:8003 172.25.253.0:8004] with http[9323], enableResumeFromBreakPoint[false], slot boundary[-1, -1]
    23. 2021/03/20 15:55:10 [INFO] DbSyncer[1] psync connect '172.25.253.0:7000' with auth type[auth] OK!
    24. 2021/03/20 15:55:10 [INFO] DbSyncer[0] psync connect '172.25.253.0:7002' with auth type[auth] OK!
    25. 2021/03/20 15:55:10 [INFO] DbSyncer[2] psync connect '172.25.253.0:7001' with auth type[auth] OK!
    26. 2021/03/20 15:55:10 [INFO] DbSyncer[1] psync send listening port[9320] OK!
    27. 2021/03/20 15:55:10 [INFO] DbSyncer[2] psync send listening port[9320] OK!
    28. 2021/03/20 15:55:10 [INFO] DbSyncer[0] psync send listening port[9320] OK!
    29. 2021/03/20 15:55:10 [INFO] DbSyncer[1] try to send 'psync' command: run-id[?], offset[-1]
    30. 2021/03/20 15:55:10 [INFO] DbSyncer[2] try to send 'psync' command: run-id[?], offset[-1]
    31. 2021/03/20 15:55:10 [INFO] DbSyncer[0] try to send 'psync' command: run-id[?], offset[-1]
    32. 2021/03/20 15:55:10 [INFO] Event:FullSyncStart  Id:redis-shake
    33. 2021/03/20 15:55:10 [INFO] DbSyncer[1] psync runid = 4a2a6e97a05ea7c3ade676209efea6bab61638bc, offset = 444709, fullsync
    34. 2021/03/20 15:55:10 [INFO] Event:FullSyncStart  Id:redis-shake
    35. 2021/03/20 15:55:10 [INFO] DbSyncer[0] psync runid = 3889e6f11bb538d4c21fcd148e9c35715c316a90, offset = 355726, fullsync
    36. 2021/03/20 15:55:10 [INFO] Event:FullSyncStart  Id:redis-shake
    37. 2021/03/20 15:55:10 [INFO] DbSyncer[2] psync runid = 8be6cbd6f67e8b66867e8ca8a3e9cba755c84008, offset = 444493, fullsync
    38. 2021/03/20 15:55:11 [INFO] DbSyncer[0] rdb file size = 200163
    39. 2021/03/20 15:55:11 [INFO] Aux information key:redis-ver value:5.0.12
    40. 2021/03/20 15:55:11 [INFO] Aux information key:redis-bits value:64
    41. 2021/03/20 15:55:11 [INFO] Aux information key:ctime value:1616226910
    42. 2021/03/20 15:55:11 [INFO] Aux information key:used-mem value:3310920
    43. 2021/03/20 15:55:11 [INFO] Aux information key:repl-stream-db value:0
    44. 2021/03/20 15:55:11 [INFO] Aux information key:repl-id value:3889e6f11bb538d4c21fcd148e9c35715c316a90
    45. 2021/03/20 15:55:11 [INFO] Aux information key:repl-offset value:355726
    46. 2021/03/20 15:55:11 [INFO] Aux information key:aof-preamble value:0
    47. 2021/03/20 15:55:11 [INFO] db_size:3344 expire_size:0
    48. 2021/03/20 15:55:11 [INFO] DbSyncer[2] rdb file size = 223563
    49. 2021/03/20 15:55:11 [INFO] Aux information key:redis-ver value:5.0.12
    50. 2021/03/20 15:55:11 [INFO] Aux information key:redis-bits value:64
    51. 2021/03/20 15:55:11 [INFO] Aux information key:ctime value:1616226910
    52. 2021/03/20 15:55:11 [INFO] Aux information key:used-mem value:3380008
    53. 2021/03/20 15:55:11 [INFO] Aux information key:repl-stream-db value:0
    54. 2021/03/20 15:55:11 [INFO] Aux information key:repl-id value:8be6cbd6f67e8b66867e8ca8a3e9cba755c84008
    55. 2021/03/20 15:55:11 [INFO] Aux information key:repl-offset value:444493
    56. 2021/03/20 15:55:11 [INFO] Aux information key:aof-preamble value:0
    57. 2021/03/20 15:55:11 [INFO] db_size:3345 expire_size:0
    58. 2021/03/20 15:55:11 [INFO] DbSyncer[1] rdb file size = 222914
    59. 2021/03/20 15:55:11 [INFO] Aux information key:redis-ver value:5.0.12
    60. 2021/03/20 15:55:11 [INFO] Aux information key:redis-bits value:64
    61. 2021/03/20 15:55:11 [INFO] Aux information key:ctime value:1616226910
    62. 2021/03/20 15:55:11 [INFO] Aux information key:used-mem value:3375512
    63. 2021/03/20 15:55:11 [INFO] Aux information key:repl-stream-db value:0
    64. 2021/03/20 15:55:11 [INFO] Aux information key:repl-id value:4a2a6e97a05ea7c3ade676209efea6bab61638bc
    65. 2021/03/20 15:55:11 [INFO] Aux information key:repl-offset value:444709
    66. 2021/03/20 15:55:11 [INFO] Aux information key:aof-preamble value:0
    67. 2021/03/20 15:55:11 [INFO] db_size:3330 expire_size:0
    68. 2021/03/20 15:55:12 [INFO] DbSyncer[0] total = 195.47KB -     127.74KB [ 65%]  entry=1005
    69. 2021/03/20 15:55:12 [INFO] DbSyncer[2] total = 218.32KB -     124.72KB [ 57%]  entry=949
    70. 2021/03/20 15:55:12 [INFO] DbSyncer[1] total = 217.69KB -     151.26KB [ 69%]  entry=1016
    71. 2021/03/20 15:55:13 [INFO] DbSyncer[0] total = 195.47KB -     184.66KB [ 94%]  entry=2109
    72. 2021/03/20 15:55:13 [INFO] DbSyncer[2] total = 218.32KB -     217.07KB [ 99%]  entry=2296
    73. 2021/03/20 15:55:13 [INFO] DbSyncer[1] total = 217.69KB -     210.68KB [ 96%]  entry=2169
    74. 2021/03/20 15:55:14 [INFO] DbSyncer[0] total = 195.47KB -     195.47KB [100%]  entry=3097
    75. 2021/03/20 15:55:14 [INFO] DbSyncer[2] total = 218.32KB -     218.32KB [100%]  entry=3311
    76. 2021/03/20 15:55:14 [INFO] DbSyncer[2] total = 218.32KB -     218.32KB [100%]  entry=3345
    77. 2021/03/20 15:55:14 [INFO] DbSyncer[2] sync rdb done
    78. 2021/03/20 15:55:14 [INFO] DbSyncer[1] total = 217.69KB -     217.69KB [100%]  entry=3221
    79. 2021/03/20 15:55:14 [INFO] DbSyncer[2] FlushEvent:IncrSyncStart Id:redis-shake
    80. 2021/03/20 15:55:14 [INFO] DbSyncer[0] total = 195.47KB -     195.47KB [100%]  entry=3344
    81. 2021/03/20 15:55:14 [INFO] DbSyncer[0] sync rdb done
    82. 2021/03/20 15:55:14 [INFO] DbSyncer[0] FlushEvent:IncrSyncStart Id:redis-shake
    83. 2021/03/20 15:55:14 [INFO] DbSyncer[1] total = 217.69KB -     217.69KB [100%]  entry=3330
    84. 2021/03/20 15:55:14 [INFO] DbSyncer[1] sync rdb done
    85. 2021/03/20 15:55:14 [INFO] DbSyncer[1] FlushEvent:IncrSyncStart Id:redis-shake
    86. 2021/03/20 15:55:15 [INFO] DbSyncer[2] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
    87. 2021/03/20 15:55:15 [INFO] DbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
    88. 2021/03/20 15:55:15 [INFO] DbSyncer[1] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
    89. 2021/03/20 15:55:16 [INFO] DbSyncer[2] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
    90. 2021/03/20 15:55:16 [INFO] DbSyncer[0] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
    91. 2021/03/20 15:55:16 [INFO] DbSyncer[1] sync:  +forwardCommands=0      +filterCommands=0      +writeBytes=0
    92. 2021/03/20 15:55:17 [INFO] DbSyncer[2] sync:  +forwardCommands=1      +filterCommands=0      +writeBytes=4
    93. ...
    复制代码

    当观察到sync rdb done，表面全量迁移完成，接下来是增量迁移。
    　　经测试，利用工具redis-shake，可以快速对Redis集群的数据进行迁移。

    　　问题
    　　·网络连通性，端口需要确认。
    　　·100W条数据，4分钟左右。
    　　·全量/部分。

    分享到:  QQ好友和群QQ好友和群 QQ空间QQ空间 腾讯微博腾讯微博 腾讯朋友腾讯朋友
    微信
    收藏收藏
    【喜讯】51微信互动平台签到赚积点上线了
    回复

    使用道具 举报

    返回列表 发新帖

    本版积分规则

    关闭

    站长推荐上一条 /1 下一条

    小黑屋|手机版|Archiver|51Testing软件测试网 ( 沪ICP备05003035号 )  关于我们

    GMT+8, 2024-11-27 20:13 , Processed in 0.061674 second(s), 24 queries .

    Powered by Discuz! X3.2

    © 2001-2024 Comsenz Inc.

    快速回复 返回顶部 返回列表