1.Drbd是基于块设备的高可用集群,所以在使用drbd 之前我需要创建一个分区(mysql-a和mysql-b都要去创建分区。分区为/dev/sdb1 当然分区也可以一样,也可以不一样,可以在drbd的配置文件指定。 第一步,节点上分区 [root@mysql-a /]# fdisk -l /dev/sdb Disk /dev/sdb: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xf5c6ec8b
Device Boot Start End Blocks Id System /dev/sdb1 1 2610 20964793+ 83 Linux [root@mysql-b /]# fdisk -l /dev/sdb
Disk /dev/sdb: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x274e7b9c
Device Boot Start End Blocks Id System /dev/sdb1 1 2610 20964793+ 83 Linux 第二步安装drbd 解压后 安装之前的准备工作 参考资料 http://drbd.linbit.com/users-guide-8.4/ap-man-pages.html drbd的手册页 主要包括drbd.confConfigurationfile for DRBD's devices drbdadmAdministrationtool for DRBD drbdsetupSetuptool for DRBD drbdmetaDRBD'smeta data management tool http://drbd.linbit.com/users-guide-8.4/ 版本手册 http://drbd.linbit.com/users-guide-8.4/s-build-from-source.html#s-build-prepare-checkout安装参考资料 阅读完安装手册进行安装 make, gcc, the glibcdevelopment libraries, and the flex scanner generator must be installed [root@mysql-a drbd-8.4.4]# ./configure --prefix=/usr/local/drbd--localstatedir=/usr/local/ --sysconfdir=/etc --with-km [root@mysql-a drbd-8.4.4]# make && make install CompilingDRBD as a kernel module [root@mysql-a drbd-8.4.4]# cd drbd [root@mysql-a drbd]# make clean all Buildingagainst precompiled kernel headers 查看当前我正在用的内核,将机器重启看到我又更新内核 我使用的是2.6.32.-504.8.1 [root@mysql-a drbd]# make clean rm -rf .tmp_versions Module.markers Module.symversmodules.order rm -f *.[oas] *.ko .*NaNd .*.d .*.tmp *.mod.c .*.flags.depend .kernel* rm -f compat/*.[oas] compat/.*NaNd [root@mysql-a drbd]# make KDIR=/lib/modules/2.6.32-504.8.1.el6.x86_64/build/ Buildingagainst a kernel source tree [root@mysql-a drbd]# make clean rm -rf .tmp_versions Module.markers Module.symversmodules.order rm -f *.[oas] *.ko .*NaNd .*.d .*.tmp *.mod.c .*.flags.depend .kernel* rm -f compat/*.[oas] compat/.*NaNd [root@mysql-a drbd]# makeKDIR=/lib/modules/2.6.32-504.8.1.el6.x86_64/source
Using anon-default C compiler [root@mysql-a drbd]# make clean rm -rf .tmp_versions Module.markers Module.symvers modules.order rm -f *.[oas] *.ko .*NaNd .*.d .*.tmp *.mod.c .*.flags.depend .kernel* rm -f compat/*.[oas] compat/.*NaNd [root@mysql-a drbd]# make CC=gcc 检查 If themodule build completes successfully, you should see a kernel module file named drbd.ko in the drbd directory. You mayinterrogate the newly-built module with /sbin/modinfo drbd.ko if you are soinclined. [root@mysql-a kernels]# find / -namedrbd.ko /usr/local/src/drbd-8.4.4/drbd/drbd.ko /lib/modules/2.6.32-504.8.1.el6.x86_64/updates/drbd.ko 查看drbd的配置文件,因为我安装的时候指定drbd配置文件的路径为/etc/可以直接去/etc/修改 [root@mysql-a kernels]# cd /etc/dr dracut.conf dracut.conf.d/ drbd.conf drbd.d/ [root@mysql-a kernels]# cd /etc/drbd.conf [root@mysql-a /]# ls /etc/init.d/drbd 也不需要去cp启动脚本了 发现这样好方便 /etc/init.d/drbd [root@mysql-a /]# 修改drbd的配置文件 http://drbd.linbit.com/users-guide-8.4/re-drbdconf.html drbd.conf 官网说明 配置参考 http://drbd.linbit.com/users-guide/s-configure-resource.html#s-drbdconf-example
[root@mysql-a /]# vim /etc/drbd.conf # You can find an example in /usr/share/doc/drbd.../drbd.conf.example include "drbd.d/global_common.conf"; include "drbd.d/*.res"; The file /etc/drbd.conf is read bydrbdadm. Drbdadm 是读取drbd.conf 这个主配置文件的,但是默认drbd里面又是读取 include "drbd.d/global_common.conf"; include "drbd.d/*.res"; 这两个文件,我们可以去找到drbd.conf.example 模板直接修改drbd.conf 可以分别去修改这个文件 include "drbd.d/global_common.conf"; include "drbd.d/*.res"; 然后在drbd.conf 启用加载这两个文件就可以了。 [root@mysql-a /]#find / -name drbd.conf.example /usr/local/src/drbd-8.4.4/scripts/drbd.conf.example [root@mysql-a /]#cp /usr/local/src/drbd-8.4.4/scripts/drbd.conf.example /etc/drbd.conf cp: overwrite`/etc/drbd.conf'? y drbd不建议使用模板的drbd.conf 去配置,那样的配置比较混乱,容易出错。 drbd 参数介绍 [root@mysql-a ~]# scp /etc/drbd.d/* 20.0.0.36:/etc/drbd.d/ 将修改的drbd.d 下的配置文件拷贝到备节点上启动drbd测试 初始化元数据 两个节点都需要执行 http://drbd.linbit.com/users-guide-8.4/s-first-time-up.html drbd的配置文件参数 [root@mysql-aetc]# cat /etc/drbd.conf # You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf"; include "drbd.d/*.res"; [root@mysql-aetc]# egrep -v "#|^$" /etc/drbd.d/global_common.conf 把带#的都过滤掉 global { usage-countno; } common { protocol C; handlers { pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;reboot -f"; pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;reboot -f"; local-io-error "/usr/lib/drbd/notify-io-error.sh;/usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ;halt -f"; fence-peer "/usr/lib/drbd/crm-fence-peer.sh"; split-brain "/usr/lib/drbd/notify-split-brain.sh root"; out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root"; } startup { wfc-timeout 120; degr-wfc-timeout 120; } options { } disk { on-io-error detach; } net { cram-hmac-alg"sha1"; shared-secret "allendrbd"; } syncer { rate100M; } } [root@mysql-a etc]# cat/etc/drbd.d/drbd_mysql.res resource r0 {
on mysql-a{ device /dev/drbd0; disk /dev/sdb1; meta-disk internal; address 40.0.0.1:7789; } on mysql-b{ device /dev/drbd0; disk /dev/sdb1; meta-disk internal; address 40.0.0.2:7789; } } 在配置drbd的时候心跳是一个网段,数据同步是一个网段,业务是一个网段 Drbd 开始初始化元数据 [root@mysql-a ~]#drbdadm create-md r0 --== Thank you for participating in the globalusage survey ==-- The server's response is:
you are the 3978th user to install this version Writing meta data... initializing activity log NOT initializing bitmap New drbd meta data block successfully created. success [root@mysql-b drbd.d]# drbdadmcreate-md r0 --== Thank you for participating in the globalusage survey ==-- The server's response is:
you are the 3979th user to install this version Writing meta data... initializing activity log NOT initializing bitmap New drbd meta data block successfully created. success 启动drbd 两个节点都启动 ,注意启动a节点的时候再去启动b节点,此时ab两个节点会进行通信 [root@mysql-a ~]# /etc/init.d/drbd start 也可以使用drbdadm 去启动 参数 [root@mysql-b /]#drbdadm –help 去查看参数 Starting DRBD resources: [ create res:r0/usr/local/run/drbd: No such file or directory
prepare disk:r0/usr/local/run/drbd: No such file or directory
adjust disk: r0 adjust net: r0
outdated-wfc-timeout has to be shorter thandegr-wfc-timeout outdated-wfc-timeout implicitly set to degr-wfc-timeout(120s) .......... *************************************************************** DRBD's startupscript waits for the peer node(s) to appear. - In case thisnode was already a degraded cluster before the reboot thetimeout is 120 seconds. [degr-wfc-timeout] - If the peer wasavailable before the reboot the timeout will expire after 120seconds. [wfc-timeout] (These valuesare for resource 'r0'; 0 sec -> wait forever) To abort waitingenter 'yes' [ 23]:yes [root@mysql-a ~]# [root@mysql-b drbd.d]#/etc/init.d/drbd start Starting DRBD resources: [ create res:r0/usr/local/run/drbd: No such file or directory
prepare disk:r0/usr/local/run/drbd: No such file or directory
adjust disk: r0 adjust net: r0
outdated-wfc-timeout has to be shorter thandegr-wfc-timeout outdated-wfc-timeout implicitly set to degr-wfc-timeout(120s) .......... *************************************************************** DRBD's startupscript waits for the peer node(s) to appear. - In case thisnode was already a degraded cluster before the reboot the timeout is 120 seconds.[degr-wfc-timeout] - If the peer wasavailable before the reboot the timeout will expire after 120seconds. [wfc-timeout] (These valuesare for resource 'r0'; 0 sec -> wait forever) To abort waitingenter 'yes' [ 11]: [root@mysql-b drbd.d]# 在任意节点查看节点状态 [root@mysql-a ~]#cat /proc/drbd version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-12 23:44:42
1: cs:Connectedro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20964116 [root@mysql-a ~]# [root@mysql-bdrbd.d]# cat /proc/drbd version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build byroot@mysql-b, 2015-03-13 19:58:56
1: cs:Connectedro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----- ns:0 nr:0 dw:0dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20964116 初始化设备同步,手动设置主备节点 默认是没有主次之分的,需要手动设置 [root@mysql-a ~]# drbdadm primary all 设置主节点 [root@mysql-b drbd.d]# cat /proc/drbd version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-13 19:58:56 0: cs:SyncTargetro:Secondary/Secondary ds:Inconsistent/UpToDate C r----- ns:0 nr:439296dw:439296 dr:0 al:0 bm:26 lo:1 pe:3 ua:0 ap:0 ep:1 wo:f oos:20524820 [>....................] sync'ed: 2.2% (20040/20472)M finish: 0:08:33speed: 39,936 (39,936) want: 78,120 K/sec [root@mysql-a drbd.d]# cat /proc/drbd version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-12 23:44:42 0: cs:SyncSourcero:Secondary/Secondary ds:UpToDate/Inconsistent C r----- ns:831488 nr:0dw:0 dr:831488 al:0 bm:50 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20132628 [>....................] sync'ed: 4.0% (19660/20472)M finish:0:08:28 speed: 39,536 (39,592) K/sec 参数说明我的资源为r0 此时两个节点上都显示的是从,需要手动去配置主 从节点 在主节点上手动设置drbd 主从关系 [root@mysql-a etc]# drbdadm –help 参数
如果有以下提示需要执行: [root@mysql-a ~]# drbdadm primary all 设置主从有提示依次执行以下命令 0: State change failed: (-2) Need access to UpToDate data Command 'drbdsetup primary 0' terminated with exit code17 [root@mysql-a ~]# drbdadm -- --overwrite-data-of-peerprimary all [root@mysql-a ~]# drbdadm primary all [root@mysql-a ~]# cat /proc/drbd version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-17 06:28:45 0: cs:SyncSourcerorimary/Secondary ds:UpToDate/Inconsistent C r----- ns:4277528 nr:0dw:0 dr:4279960 al:0 bm:261 lo:1 pe:1 ua:3 ap:0 ep:1 wo:f oos:16686868 [===>................] sync'ed: 20.5% (16292/20472)M 元数据正在同步 finish:0:06:29 speed: 42,852 (32,400) K/se [root@mysql-b ~]# cat /proc/drbd version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-17 06:29:19 0: cs:SyncTargetro:Secondary/Primary ds:Inconsistent/UpToDate C r----- ns:0 nr:5178368dw:5178368 dr:0 al:0 bm:316 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:15785748 [===>................] sync'ed: 24.8% (15412/20472)M finish:0:06:06 speed: 43,064 (33,624) want: 41,000 K/sec [root@mysql-a /]# mkfs.ext4 /dev/drbd0 只需要在主节点进行格式化分区 挂载drbd0 到mysql的数据存储位置 [root@mysql-a data]# /etc/init.d/mysql status SUCCESS! MySQLrunning (1308) [root@mysql-a data]# /etc/init.d/mysql stop Shutting down MySQL... SUCCESS! [root@mysql-a data]# ls auto.cnf ib_logfile0 mysql mysql-a.pid test ibdata1 ib_logfile1 mysql-a.err performance_schema [root@mysql-a data]# /etc/init.d/mysql stop Shutting down MySQL.. SUCCESS! [root@mysql-a data]# mv * /mnt/ [root@mysql-a /]# mount /dev/drbd0 /data/ [root@mysql-a /]# cd /mnt/ [root@mysql-a mnt]# ls auto.cnf ibdata1 ib_logfile0 ib_logfile1 mysql mysql-a.err performance_schema test [root@mysql-a mnt]# mv * /data/ B节点也需要去清空以下mysql的元数据 [root@mysql-a mnt]# /etc/init.d/mysql start Starting MySQL...... ERROR! The server quit withoutupdating PID file (/data/mysql-a.pid) [root@mysql-a /]# chown -R mysql. /data/ [root@mysql-a mnt]# /etc/init.d/mysql restart Shutting down MySQL.. SUCCESS! Starting MySQL..... SUCCESS! [root@mysql-a /]# mysql -uroot mysql可以正常登陆, 然后在主节点卸载umount /data 然后在从节点上进行挂载,看数据是否ok。 Drbd 测试 [root@mysql-a ~]# drbdadm secondary r0 将主节点设置为从 [root@mysql-a ~]# cat /proc/drbd version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-17 06:28:45 0: cs:Connectedro:Secondary/Secondary ds:UpToDate/UpToDate C r----- ns:0 nr:28672dw:28672 dr:664 al:0 bm:5 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 [root@mysql-b ~]# drbdadm primary r0 [root@mysql-b ~]# cat /proc/drbd version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-17 06:29:19 0: cs:Connectedrorimary/Secondary ds:UpToDate/UpToDate C r----- ns:28672 nr:0dw:0 dr:29336 al:0 bm:5 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 手动切换造成drbd 脑裂 处理方法: 处理方法: 第一步在从节点上: [root@mysql-a /]# drbdadm disconnect r0 r0: Failure: (162) Invalid configuration request additional info from kernel: unknown connection Command 'drbdsetup disconnect ipv4:30.0.0.1:7789ipv4:30.0.0.2:7789' terminated with exit code 10 [root@mysql-a /]# drbdadm secondary r0 [root@mysql-a /]# drbdadm connect --discard-my-data r0 [root@mysql-a /]# cat /proc/drbd version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-17 06:28:45 0: cs:WFConnectionrorimary/Unknown ds:UpToDate/Outdated C r----- ns:0 nr:0 dw:0dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:668436 查看还是报错 在主节点上面的处理 [root@mysql-b log]# drbdadm connect r0 [root@mysql-b log]# drbdadm role r0 Secondary/Primary 再去1节点查看状态 [root@mysql-a /]# drbdadm role r0 Primary/Secondary 故障解决 [root@mysql-a /]# drbdadm dstate r0 查看硬盘状态正常 UpToDate/UpToDate Yum安装heartbeat http://www.linux-ha.org/wiki/Downloads linux-ha官网的提供的yum源 http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/ 但是官网没有heartbeat wget http://mirrors.sohu.com/fedora-e ... ease-6-8.noarch.rpm rpm -ivh epel-release-6-8.noarch.rpm 安装yum源 [root@mysql-a local]# yum install heartbeat* [root@mysql-a local]# yum -y install pacemaker* 集群管理器 配置heartbeat [root@mysql-a /]# cd /usr/share/doc/heartbeat-3.0.4/ [root@mysql-a heartbeat-3.0.4]# ls apphbd.cf authkeys AUTHORS ChangeLog COPYING COPYING.LGPL ha.cf haresources README [root@mysql-a heartbeat-3.0.4]# cp authkeys ha.cfharesources /etc/ha.d/ 拷贝模板文件到/etc/ha.d/ 目录中 [root@mysql-b ~]# egrep -v "#|^$" /etc/ha.d/ha.cf debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 10 warntime 5 initdead 30 udpport 694 baud 19200 ucast eth1 30.0.0.2 auto_failback off node mysql-a node mysql-b ping 30.0.0.10 respawn hacluster /usr/lib64/heartbeat/ipfail [root@mysql-b ~]# egrep -v "#|^$" /etc/ha.d/haresources mysql-a IPaddr::20.0.0.252/8/eth0:1 drbddisk::r0 Filesystem::/dev/drbd0::/data::ext4 mysql [root@mysql-b ~]# egrep -v "#|^$" /etc/ha.d/authkeys auth 2 2 sha1 oldboy! [root@mysql-b ~]# 启动heartbeat 在实际操作中遇到drbd脑裂问题处理 [root@mysql-a ~]# netstat –ltuanp查看心跳ip是否正常 [root@mysql-b ~]# cat /proc/drbd version: 8.4.4 (api:1/proto:86-101) GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-17 06:29:19 0: cs:StandAlonerorimary/Unknown ds:UpToDate/DUnknown r----- ns:0 nr:0dw:256 dr:6889 al:6 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:364 解决思路: 在从节点上 drbdadm secondary r0 drbdadm -- --discard-my-data connect r0 在到主节点上 drbdadm connect r0 解决脑裂 集群测试! 断掉心跳线, 节点关机,vip是否正常切换
本文出自 “小菜鸟” 博客,请务必保留此出处http://xiaocainiaox.blog.51cto.com/4484443/1200323
|