flashcache的使用测试

TimiZheng

参考：

https://github.com/facebookarchive/flashcache/blob/master/README-CentOS6

flashcache和bcache的性能比较

http://www.accelcloud.com/2012/04/18/linux-flashcache-and-bcache-performance-testing/

看起来似乎bcache要更好一些

http://www.fx114.net/qa-171-144648.aspx

适用的场景：离散读写较多的情况

http://blog.csdn.net/wanthelping/article/details/50493352

FlashCache相关文章整理

https://github.com/facebookarchive/flashcache/blob/master/doc/flashcache-sa-guide.txt

http://www.simlinux.com/archives/398.html

安装：

1. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""><span style="line-height: 25.6px;">yum install dkms gcc make
   
2. yum-utils kernel-devel -y</span></p><p>
   
3. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""><span style="line-height: 25.6px;">
   
4. yum -y install dkms
   
5. gcc make yum-utils kernel kernel-devel git make gcc-c++ device-mapper</span></p><p>
   
6. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""><span style="line-height: 25.6px;">
   
7. yum -y install
   
8. unifdef.x86_64</span></p>

复制代码

获取源码

1. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""><span style="line-height: 25.6px;"><wbr>wget "<a href="https://github.com/facebookarchive/flashcache/archive/3.1.2.zip"><font color="#0066cc">https://github.com/facebookarchive/flashcache/archive/3.1.2.zip"</font></a></span></p><p>
   
2. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">unzip flashcache-master.zip</p><p>
   
3. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>cd flashcache-master</p>

复制代码

--编译安装

1. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""><wbr>make
   
2. KERNEL_TREE=/usr/src/kernels/2.6.32-431.el6.x86_64/</p><p>
   
3. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>make install</p>

复制代码

加载模块：

modprobe flashcache

检查：

dmesg | tail

创建设备：

1、原数据盘为/dev/sda5 ,  作为缓存的ssd盘在/dev/sdb1

盘必须不在使用，所以已经挂载的盘需要先卸载

umount /data

2、对ssd 盘进行分区

fdisk /dev/ssd

做一个分区

3、建立带cache的虚拟盘

flashcache_create -p back cachedev /dev/sdb1 /dev/sda5

fdisk -l 可以看到如下的设备：

Disk /dev/mapper/cachedev: 939.2 GB, 939174920192 bytes

255 heads, 63 sectors/track, 114181 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

1. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">[root@localhost ~]# lsblk</p><p>
   
2. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">NAME  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>
   
3. <wbr>MAJ:MIN RM  <wbr> SIZE RO TYPE MOUNTPOINT</p><p>
   
4. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">sdb  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>
   
5. <wbr> 8:16  <wbr> 0 447.1G  <wbr>0 disk <wbr></p><p>
   
6. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">└─sdb1  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>
   
7. <wbr>8:17  <wbr> 0 447.1G  <wbr>0 part <wbr></p><p>
   
8. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr> └─cachedev (dm-0) 253:0  <wbr>  <wbr>0 874.7G  <wbr>0
   
9. dm  <wbr> <wbr></p><p>
   
10. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">sda  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>
    
11. <wbr> 8:0  <wbr>  <wbr>0 931.5G  <wbr>0 disk <wbr></p><p>
    
12. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">├─sda1  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>
    
13. <wbr>8:1  <wbr>  <wbr>0  <wbr> 200M  <wbr>0 part /boot</p><p>
    
14. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">├─sda2  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>
    
15. <wbr>8:2  <wbr>  <wbr>0  <wbr>48.8G  <wbr>0 part /</p><p>
    
16. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">├─sda3  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>
    
17. <wbr>8:3  <wbr>  <wbr>0  <wbr> 7.8G  <wbr>0 part [SWAP]</p><p>
    
18. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">├─sda4  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>
    
19. <wbr>8:4  <wbr>  <wbr>0  <wbr>  <wbr> 1K  <wbr>0 part <wbr></p><p>
    
20. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">└─sda5  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>
    
21. <wbr>8:5  <wbr>  <wbr>0 874.7G  <wbr>0 part <wbr></p><p>
    
22. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr> └─cachedev (dm-0) 253:0  <wbr>  <wbr>0 874.7G  <wbr>0
    
23. dm  <wbr> <wbr></p><p>
    
24. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">sr0  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>  <wbr>
    
25. <wbr>11:0  <wbr>  <wbr>1  <wbr>1024M  <wbr>0 rom  </p>

复制代码

flashcache_create -p back cachedev /dev/sdb1 /dev/sda5

查看信息：

dmsetup status cachedev

dmsetup table cachedev

dmsetup info cachedev

4、参数优化：

sysctl -a |grep flashcache 可以查看到所有的参数

1. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">vi /etc/sysctl.conf</p><p>
   
2. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">dev.flashcache.sdb1+sda5.reclaim_policy = 1</p><p>
   
3. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">dev.flashcache.sdb1+sda5.fast_remove = 0</p><p>
   
4. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">dev.flashcache.sdb1+sda5.new_style_write_merge = 1</p><p>
   
5. </p><p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">dev.flashcache.sdb1+sda5.skip_seq_thresh_kb = 256</p>

复制代码

sysctl -p

文章来源：沧海大声啸的博客           作者：Kervin（博为峰网校讲师）

讲师作品： MySQL实战校园业务数据库：http://www.atstudy.com/course/2001

TimiZheng

5、开启启动项设置

加载模块

vim /etc/sysconfig/modules/flashcache.modules

#! /bin/sh

/sbin/modinfo -F filename flashcache > /dev/null 2>&1

if [ $? -eq 0 ]; then

    /sbin/modprobe flashcache

fi

chmod +x  /etc/sysconfig/modules/flashcache.modules

挂载设备：

mount  /dev/mapper/cachedev /data




修改前面的几项内容：vim /etc/init.d/flashcache

SSD_DISK=/dev/sdb1

BACKEND_DISK=/dev/sda5

CACHEDEV_NAME=cachedev

MOUNTPOINT=/data

FLASHCACHE_NAME=sdb1+sda5

chkconfig flashcache on




去掉原来/dev/sda5的硬盘启动自动加载项

blkid查看（注：会发现/dev/sda5和 /dev/mapper/cachedev id号是一样的）

vim /etc/fstab

将这个id号的加载下注释掉

TimiZheng

6、删除

卸载：umount /data




删除：flashcache_destroy /dev/sdb1 在writeback模式下，由于cache中存在脏数据会有问题

flashcache_destroy: DIRTY BLOCKS EXIST ON /dev/sdb1, ABORTING CACHE DESTROY

flashcache_destroy: Use -f (force) to destroy cache with DIRTY blocks, BUT YOU WILL LOSE DATA GUARANTEED

flashcache_destroy: To clean the DIRTY blocks, flashcache_load, then do_sync until all dirty blocks are cleaned

存在脏块，必须先同步，然后才能删除。如果强制删除将会丢失数据。

重新加载 flashcache_load /dev/sdb1


重新加载后，我们发现所有原来所做的优化都失效了

重新执行 sysctl -p

使用dmsetup remove cachedev

在fast_remove=0时，删除时首先做同步，同步完成后才会被删除。

dev.flashcache.sdb1+sda5.fast_remove = 0  所以这个参数不要轻易修改

脏块越大，同步时间就会越长。没事就别折腾了。





问题：当使用dmsetup remove cachedev 之后，就可以直接再次挂载/dev/sda5使用,如果再卸载，这个时候由于/dev/sdb1还可以通过flashcache_load再次启用，此时由于/dev/sdb1中还有缓存数据和/dev/sda5中数据已经不同，会造成很大问题。所以务必在dmsetup remove之后，那么不要单独使用/dev/sda5，那么就直接通过flashcache_destroy /dev/sdb1 来删除缓存。

否则一旦出现数据问题很难恢复。




性能测试：





异常情况的测试：

dmsetup remove cachedev