51Testing软件测试论坛

 找回密码
 (注-册)加入51Testing

QQ登录

只需一步,快速开始

微信登录,快人一步

手机号码,快捷登录

查看: 3806|回复: 0
打印 上一主题 下一主题

【转】mysql+drbd+heartbeat

[复制链接]
  • TA的每日心情
    无聊
    昨天 09:05
  • 签到天数: 1050 天

    连续签到: 1 天

    [LV.10]测试总司令

    跳转到指定楼层
    1#
    发表于 2016-5-3 15:50:58 | 只看该作者 回帖奖励 |倒序浏览 |阅读模式

    1.Drbd是基于块设备的高可用集群,所以在使用drbd 之前我需要创建一个分区(mysql-a和mysql-b都要去创建分区。分区为/dev/sdb1

    当然分区也可以一样,也可以不一样,可以在drbd的配置文件指定。

    第一步,节点上分区

    [root@mysql-a /]# fdisk -l /dev/sdb

    Disk /dev/sdb: 21.5 GB, 21474836480 bytes

    255 heads, 63 sectors/track, 2610 cylinders

    Units = cylinders of 16065 * 512 = 8225280 bytes

    Sector size (logical/physical): 512 bytes / 512 bytes

    I/O size (minimum/optimal): 512 bytes / 512 bytes

    Disk identifier: 0xf5c6ec8b


       Device Boot      Start        End      Blocks  Id  System

    /dev/sdb1              1        2610    20964793+ 83  Linux

    [root@mysql-b /]# fdisk -l /dev/sdb


    Disk /dev/sdb: 21.5 GB, 21474836480 bytes

    255 heads, 63 sectors/track, 2610 cylinders

    Units = cylinders of 16065 * 512 = 8225280 bytes

    Sector size (logical/physical): 512 bytes / 512 bytes

    I/O size (minimum/optimal): 512 bytes / 512 bytes

    Disk identifier: 0x274e7b9c


       Device Boot      Start         End      Blocks  Id  System

    /dev/sdb1              1        2610    20964793+ 83  Linux

    第二步安装drbd

    解压后

    安装之前的准备工作

    参考资料

    http://drbd.linbit.com/users-guide-8.4/ap-man-pages.html   drbd的手册页 主要包括drbd.confConfigurationfile for DRBD's devices

    drbdadmAdministrationtool for DRBD

    drbdsetupSetuptool for DRBD

    drbdmetaDRBD'smeta data management tool

    http://drbd.linbit.com/users-guide-8.4/  版本手册

    http://drbd.linbit.com/users-guide-8.4/s-build-from-source.html#s-build-prepare-checkout安装参考资料

    阅读完安装手册进行安装

    make, gcc, the glibcdevelopment libraries, and the flex scanner generator must be installed

    [root@mysql-a drbd-8.4.4]#  ./configure --prefix=/usr/local/drbd--localstatedir=/usr/local/ --sysconfdir=/etc --with-km

    [root@mysql-a drbd-8.4.4]# make && make install

    CompilingDRBD as a kernel module

    [root@mysql-a drbd-8.4.4]# cd drbd

    [root@mysql-a drbd]# make clean all

    Buildingagainst precompiled kernel headers

    查看当前我正在用的内核,将机器重启看到我又更新内核

    我使用的是2.6.32.-504.8.1

    [root@mysql-a drbd]# make clean

    rm -rf .tmp_versions Module.markers Module.symversmodules.order

    rm -f *.[oas] *.ko .*NaNd .*.d .*.tmp *.mod.c .*.flags.depend .kernel*

    rm -f compat/*.[oas] compat/.*NaNd

    [root@mysql-a drbd]# make KDIR=/lib/modules/2.6.32-504.8.1.el6.x86_64/build/

    Buildingagainst a kernel source tree

    [root@mysql-a drbd]# make clean

    rm -rf .tmp_versions Module.markers Module.symversmodules.order

    rm -f *.[oas] *.ko .*NaNd .*.d .*.tmp *.mod.c .*.flags.depend .kernel*

    rm -f compat/*.[oas] compat/.*NaNd

    [root@mysql-a drbd]# makeKDIR=/lib/modules/2.6.32-504.8.1.el6.x86_64/source


    Using anon-default C compiler

    [root@mysql-a drbd]# make clean

    rm -rf .tmp_versions Module.markers Module.symvers modules.order

    rm -f *.[oas] *.ko .*NaNd .*.d .*.tmp *.mod.c .*.flags.depend .kernel*

    rm -f compat/*.[oas] compat/.*NaNd

    [root@mysql-a drbd]# make CC=gcc

    检查

    If themodule build completes successfully, you should see a kernel module file named drbd.ko in the drbd directory. You mayinterrogate the newly-built module with /sbin/modinfo drbd.ko if you are soinclined.

    [root@mysql-a kernels]# find / -namedrbd.ko

    /usr/local/src/drbd-8.4.4/drbd/drbd.ko

    /lib/modules/2.6.32-504.8.1.el6.x86_64/updates/drbd.ko

    查看drbd的配置文件,因为我安装的时候指定drbd配置文件的路径为/etc/可以直接去/etc/修改

    [root@mysql-a kernels]# cd /etc/dr

    dracut.conf   dracut.conf.d/ drbd.conf     drbd.d/      

    [root@mysql-a kernels]# cd /etc/drbd.conf

    [root@mysql-a /]# ls /etc/init.d/drbd   也不需要去cp启动脚本了 发现这样好方便

    /etc/init.d/drbd

    [root@mysql-a /]#

    修改drbd的配置文件

    http://drbd.linbit.com/users-guide-8.4/re-drbdconf.html   drbd.conf 官网说明

    配置参考

    http://drbd.linbit.com/users-guide/s-configure-resource.html#s-drbdconf-example


    [root@mysql-a /]# vim /etc/drbd.conf

    # You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

    include "drbd.d/global_common.conf";

    include "drbd.d/*.res";

    The file /etc/drbd.conf is read bydrbdadm.   Drbdadm 是读取drbd.conf 这个主配置文件的,但是默认drbd里面又是读取

    include "drbd.d/global_common.conf";

    include "drbd.d/*.res";

    这两个文件,我们可以去找到drbd.conf.example 模板直接修改drbd.conf 可以分别去修改这个文件

    include "drbd.d/global_common.conf";

    include "drbd.d/*.res";

    然后在drbd.conf 启用加载这两个文件就可以了。

    [root@mysql-a /]#find / -name drbd.conf.example

    /usr/local/src/drbd-8.4.4/scripts/drbd.conf.example

    [root@mysql-a /]#cp /usr/local/src/drbd-8.4.4/scripts/drbd.conf.example /etc/drbd.conf

    cp: overwrite`/etc/drbd.conf'? y

    drbd不建议使用模板的drbd.conf 去配置,那样的配置比较混乱,容易出错。

    drbd 参数介绍

    [root@mysql-a ~]# scp /etc/drbd.d/*  20.0.0.36:/etc/drbd.d/     将修改的drbd.d 下的配置文件拷贝到备节点上启动drbd测试

    初始化元数据 两个节点都需要执行  http://drbd.linbit.com/users-guide-8.4/s-first-time-up.html

    drbd的配置文件参数

    [root@mysql-aetc]# cat /etc/drbd.conf

    # You can find an example in  /usr/share/doc/drbd.../drbd.conf.example


    include "drbd.d/global_common.conf";

    include "drbd.d/*.res";

    [root@mysql-aetc]# egrep -v "#|^$" /etc/drbd.d/global_common.conf   把带#的都过滤掉

    global {

            usage-countno;

    }

    common {

            protocol C;

            handlers {

                   pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;reboot -f";

                   pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;reboot -f";

                   local-io-error "/usr/lib/drbd/notify-io-error.sh;/usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ;halt -f";

                    fence-peer "/usr/lib/drbd/crm-fence-peer.sh";

                    split-brain "/usr/lib/drbd/notify-split-brain.sh root";

                    out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";

            }

            startup {

                   wfc-timeout 120;

                   degr-wfc-timeout 120;  

            }

            options {

            }

            disk {

                    on-io-error   detach;

                }

            net {

                    cram-hmac-alg"sha1";

                   shared-secret "allendrbd";

            }

            syncer {

              rate100M;

    }

    }

    [root@mysql-a etc]# cat/etc/drbd.d/drbd_mysql.res         

    resource r0 {


            on mysql-a{

                   device    /dev/drbd0;

                    disk      /dev/sdb1;

                   meta-disk internal;

                   address   40.0.0.1:7789;

                    }

            on mysql-b{

                   device    /dev/drbd0;

                   disk      /dev/sdb1;

                   meta-disk internal;

                   address   40.0.0.2:7789;

                   }

    }

    在配置drbd的时候心跳是一个网段,数据同步是一个网段,业务是一个网段

    Drbd 开始初始化元数据

    [root@mysql-a ~]#drbdadm create-md r0                  

      --==  Thank you for participating in the globalusage survey  ==--

    The server's response is:


    you are the 3978th user to install this version

    Writing meta data...

    initializing activity log

    NOT initializing bitmap

    New drbd meta data block successfully created.

    success

    [root@mysql-b drbd.d]# drbdadmcreate-md r0              

      --==  Thank you for participating in the globalusage survey  ==--

    The server's response is:


    you are the 3979th user to install this version

    Writing meta data...

    initializing activity log

    NOT initializing bitmap

    New drbd meta data block successfully created.

    success

    启动drbd 两个节点都启动 ,注意启动a节点的时候再去启动b节点,此时ab两个节点会进行通信

    [root@mysql-a ~]# /etc/init.d/drbd start  也可以使用drbdadm 去启动  参数 [root@mysql-b /]#drbdadm –help 去查看参数

    Starting DRBD resources: [

         create res:r0/usr/local/run/drbd: No such file or directory


       prepare disk:r0/usr/local/run/drbd: No such file or directory


        adjust disk: r0

         adjust net: r0


    outdated-wfc-timeout has to be shorter thandegr-wfc-timeout

    outdated-wfc-timeout implicitly set to degr-wfc-timeout(120s)

    ..........

    ***************************************************************

    DRBD's startupscript waits for the peer node(s) to appear.

    - In case thisnode was already a degraded cluster before the

       reboot thetimeout is 120 seconds. [degr-wfc-timeout]

    - If the peer wasavailable before the reboot the timeout will

       expire after 120seconds. [wfc-timeout]

       (These valuesare for resource 'r0'; 0 sec -> wait forever)

    To abort waitingenter 'yes' [  23]:yes

    [root@mysql-a ~]#

    [root@mysql-b drbd.d]#/etc/init.d/drbd start

    Starting DRBD resources: [

         create res:r0/usr/local/run/drbd: No such file or directory


       prepare disk:r0/usr/local/run/drbd: No such file or directory


        adjust disk: r0

         adjust net: r0


    outdated-wfc-timeout has to be shorter thandegr-wfc-timeout

    outdated-wfc-timeout implicitly set to degr-wfc-timeout(120s)

    ..........

    ***************************************************************

    DRBD's startupscript waits for the peer node(s) to appear.

    - In case thisnode was already a degraded cluster before the

       reboot the timeout is 120 seconds.[degr-wfc-timeout]

    - If the peer wasavailable before the reboot the timeout will

       expire after 120seconds. [wfc-timeout]

       (These valuesare for resource 'r0'; 0 sec -> wait forever)

    To abort waitingenter 'yes' [  11]:

    [root@mysql-b drbd.d]#

    在任意节点查看节点状态

    [root@mysql-a ~]#cat /proc/drbd

    version: 8.4.4 (api:1/proto:86-101)

    GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-12 23:44:42


    1: cs:Connectedro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----

        ns:0 nr:0 dw:0dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20964116

    [root@mysql-a ~]#

    [root@mysql-bdrbd.d]# cat /proc/drbd

    version: 8.4.4 (api:1/proto:86-101)

    GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build byroot@mysql-b, 2015-03-13 19:58:56


    1: cs:Connectedro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----

        ns:0 nr:0 dw:0dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20964116

    初始化设备同步,手动设置主备节点

    默认是没有主次之分的,需要手动设置

    [root@mysql-a ~]# drbdadm primary all  设置主节点

    [root@mysql-b drbd.d]# cat /proc/drbd                    

    version: 8.4.4 (api:1/proto:86-101)

    GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-13 19:58:56

    0: cs:SyncTargetro:Secondary/Secondary ds:Inconsistent/UpToDate C r-----

        ns:0 nr:439296dw:439296 dr:0 al:0 bm:26 lo:1 pe:3 ua:0 ap:0 ep:1 wo:f oos:20524820

           [>....................] sync'ed: 2.2% (20040/20472)M

            finish: 0:08:33speed: 39,936 (39,936) want: 78,120 K/sec

    [root@mysql-a drbd.d]# cat /proc/drbd                    

    version: 8.4.4 (api:1/proto:86-101)

    GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-12 23:44:42

    0: cs:SyncSourcero:Secondary/Secondary ds:UpToDate/Inconsistent C r-----

        ns:831488 nr:0dw:0 dr:831488 al:0 bm:50 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20132628

           [>....................] sync'ed: 4.0% (19660/20472)M

            finish:0:08:28 speed: 39,536 (39,592) K/sec

    参数说明我的资源为r0

    此时两个节点上都显示的是从,需要手动去配置主 从节点

    在主节点上手动设置drbd 主从关系

    [root@mysql-a etc]# drbdadm –help  参数


    如果有以下提示需要执行:

    [root@mysql-a ~]# drbdadm primary all   设置主从有提示依次执行以下命令

    0: State change failed: (-2) Need access to UpToDate data

    Command 'drbdsetup primary 0' terminated with exit code17

    [root@mysql-a ~]# drbdadm -- --overwrite-data-of-peerprimary all

    [root@mysql-a ~]# drbdadm primary all

    [root@mysql-a ~]# cat /proc/drbd

    version: 8.4.4 (api:1/proto:86-101)

    GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-17 06:28:45

    0: cs:SyncSourcerorimary/Secondary ds:UpToDate/Inconsistent C r-----

        ns:4277528 nr:0dw:0 dr:4279960 al:0 bm:261 lo:1 pe:1 ua:3 ap:0 ep:1 wo:f oos:16686868

           [===>................] sync'ed: 20.5% (16292/20472)M   元数据正在同步

            finish:0:06:29 speed: 42,852 (32,400) K/se

    [root@mysql-b ~]# cat /proc/drbd

    version: 8.4.4 (api:1/proto:86-101)

    GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-17 06:29:19

    0: cs:SyncTargetro:Secondary/Primary ds:Inconsistent/UpToDate C r-----

        ns:0 nr:5178368dw:5178368 dr:0 al:0 bm:316 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:15785748

           [===>................] sync'ed: 24.8% (15412/20472)M

            finish:0:06:06 speed: 43,064 (33,624) want: 41,000 K/sec

    [root@mysql-a /]# mkfs.ext4 /dev/drbd0   只需要在主节点进行格式化分区

    挂载drbd0 到mysql的数据存储位置

    [root@mysql-a data]# /etc/init.d/mysql status

    SUCCESS! MySQLrunning (1308)

    [root@mysql-a data]# /etc/init.d/mysql stop

    Shutting down MySQL... SUCCESS!

    [root@mysql-a data]# ls

    auto.cnf ib_logfile0  mysql        mysql-a.pid         test

    ibdata1  ib_logfile1  mysql-a.err  performance_schema

    [root@mysql-a data]# /etc/init.d/mysql stop

    Shutting down MySQL.. SUCCESS!

    [root@mysql-a data]# mv * /mnt/

    [root@mysql-a /]# mount /dev/drbd0 /data/

    [root@mysql-a /]# cd /mnt/

    [root@mysql-a mnt]# ls

    auto.cnf ibdata1  ib_logfile0  ib_logfile1 mysql  mysql-a.err  performance_schema  test

    [root@mysql-a mnt]# mv * /data/

    B节点也需要去清空以下mysql的元数据

    [root@mysql-a mnt]# /etc/init.d/mysql start

    Starting MySQL...... ERROR! The server quit withoutupdating PID file (/data/mysql-a.pid)

    [root@mysql-a /]# chown -R mysql. /data/

    [root@mysql-a mnt]# /etc/init.d/mysql restart

    Shutting down MySQL.. SUCCESS!

    Starting MySQL..... SUCCESS!

    [root@mysql-a /]# mysql -uroot mysql可以正常登陆,

    然后在主节点卸载umount /data

    然后在从节点上进行挂载,看数据是否ok。

    Drbd 测试

    [root@mysql-a ~]# drbdadm secondary r0    将主节点设置为从

    [root@mysql-a ~]# cat /proc/drbd           

    version: 8.4.4 (api:1/proto:86-101)

    GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-17 06:28:45

    0: cs:Connectedro:Secondary/Secondary ds:UpToDate/UpToDate C r-----

        ns:0 nr:28672dw:28672 dr:664 al:0 bm:5 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

    [root@mysql-b ~]# drbdadm primary r0

    [root@mysql-b ~]# cat /proc/drbd   

    version: 8.4.4 (api:1/proto:86-101)

    GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-17 06:29:19

    0: cs:Connectedrorimary/Secondary ds:UpToDate/UpToDate C r-----

        ns:28672 nr:0dw:0 dr:29336 al:0 bm:5 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

    手动切换造成drbd 脑裂 处理方法:

    处理方法:

    第一步在从节点上:

    [root@mysql-a /]# drbdadm disconnect r0

    r0: Failure: (162) Invalid configuration request

    additional info from kernel:

    unknown connection

    Command 'drbdsetup disconnect ipv4:30.0.0.1:7789ipv4:30.0.0.2:7789' terminated with exit code 10

    [root@mysql-a /]# drbdadm secondary r0

    [root@mysql-a /]# drbdadm connect  --discard-my-data r0

    [root@mysql-a /]# cat /proc/drbd

    version: 8.4.4 (api:1/proto:86-101)

    GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-a, 2015-03-17 06:28:45

    0: cs:WFConnectionrorimary/Unknown ds:UpToDate/Outdated C r-----

        ns:0 nr:0 dw:0dr:664 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:668436

    查看还是报错

    在主节点上面的处理

    [root@mysql-b log]# drbdadm connect r0

    [root@mysql-b log]# drbdadm role r0

    Secondary/Primary

    再去1节点查看状态

    [root@mysql-a /]# drbdadm role r0

    Primary/Secondary

    故障解决

    [root@mysql-a /]# drbdadm dstate r0   查看硬盘状态正常

    UpToDate/UpToDate

    Yum安装heartbeat

    http://www.linux-ha.org/wiki/Downloads  linux-ha官网的提供的yum源

    http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/

    但是官网没有heartbeat

    wget http://mirrors.sohu.com/fedora-e ... ease-6-8.noarch.rpm

    rpm -ivh epel-release-6-8.noarch.rpm 安装yum源

    [root@mysql-a local]# yum install heartbeat*

    [root@mysql-a local]# yum -y install pacemaker*  集群管理器

    配置heartbeat

    [root@mysql-a /]# cd /usr/share/doc/heartbeat-3.0.4/

    [root@mysql-a heartbeat-3.0.4]# ls

    apphbd.cf  authkeys AUTHORS  ChangeLog  COPYING COPYING.LGPL  ha.cf haresources  README

    [root@mysql-a heartbeat-3.0.4]# cp authkeys ha.cfharesources  /etc/ha.d/   拷贝模板文件到/etc/ha.d/ 目录中

    [root@mysql-b ~]# egrep -v "#|^$"  /etc/ha.d/ha.cf

    debugfile /var/log/ha-debug

    logfile /var/log/ha-log

    logfacility    local0

    keepalive 2

    deadtime 10

    warntime 5

    initdead 30

    udpport 694

    baud    19200

    ucast eth1 30.0.0.2

    auto_failback off

    node    mysql-a

    node    mysql-b

    ping 30.0.0.10

    respawn hacluster /usr/lib64/heartbeat/ipfail

    [root@mysql-b ~]# egrep -v "#|^$"  /etc/ha.d/haresources

    mysql-a IPaddr::20.0.0.252/8/eth0:1 drbddisk::r0  Filesystem::/dev/drbd0::/data::ext4 mysql

    [root@mysql-b ~]# egrep -v "#|^$"  /etc/ha.d/authkeys

    auth 2

    2 sha1 oldboy!

    [root@mysql-b ~]#

    启动heartbeat

    在实际操作中遇到drbd脑裂问题处理

    [root@mysql-a ~]# netstat –ltuanp查看心跳ip是否正常

    [root@mysql-b ~]# cat /proc/drbd

    version: 8.4.4 (api:1/proto:86-101)

    GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a buildby root@mysql-b, 2015-03-17 06:29:19

    0: cs:StandAlonerorimary/Unknown ds:UpToDate/DUnknown  r-----

        ns:0 nr:0dw:256 dr:6889 al:6 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:364

    解决思路:

    在从节点上

    drbdadm secondary r0

    drbdadm -- --discard-my-data connect r0

    在到主节点上

    drbdadm connect r0

    解决脑裂

    集群测试!

    断掉心跳线, 节点关机,vip是否正常切换


    本文出自 “小菜鸟” 博客,请务必保留此出处http://xiaocainiaox.blog.51cto.com/4484443/1200323


    分享到:  QQ好友和群QQ好友和群 QQ空间QQ空间 腾讯微博腾讯微博 腾讯朋友腾讯朋友
    收藏收藏1
    回复

    使用道具 举报

    本版积分规则

    关闭

    站长推荐上一条 /1 下一条

    小黑屋|手机版|Archiver|51Testing软件测试网 ( 沪ICP备05003035号 关于我们

    GMT+8, 2024-11-22 03:02 , Processed in 0.090420 second(s), 26 queries .

    Powered by Discuz! X3.2

    © 2001-2024 Comsenz Inc.

    快速回复 返回顶部 返回列表