KeepAlived+DRDB+MFS安装及配置
好几年前就研究过一些分布式文件系统，如gfs等。但真正让人满意的不多（总有各种各样的问题，如稳定性差，架构复杂，性能损失高等等）。最近工作中有些场景需要用到分布式的存储，这次准备使用MFS（MooseFS），主要是看重它的架构比较简单，使用的人数比较多，可扩展性也比较强，性能损失也相当要小一些。
一. MFS的架构介绍
下面是MFS的架构图（图片来自官网）：
可以看到MFS主要由三部分组成：Master Server，Chunk Servers，Clients，另外还有一个MetaLogger Server.
Master Server主要用于存储分布式系统内文件的Meta信息，以及管理Chunk Servers。Master Server只能有一台，所以会有单点问题。
Chunk Servers用于存储文件，文件将分片存储在多台（可以配置）Chunk Server上，以保证安全性。
MetaLogger Server是对Master Server单点问题的一个解决方案，类似mysql的主从技术，MetaLogger会定时从Master Server上把MFS上的Meta信息拉过来备份，并实时生成Master Server上所以更新的Meta信息。
Clients为使用MFS的应用服务器，Clients通过Fuse的方式挂载MFS，挂载后读写MFS上的文件和读写本地文件没有区别。
由MFS可知，我们需要解决MFS的Master Server的单点问题。
二. KeepAlived+DRBD+MFS架构
为解决MFS的Master Server的单点问题，引入keepalived和Drbd，架构图如下：
原理：
- 在两台MFS Master机服务器安装DRBD做网络磁盘，网络磁盘上存放mfs master的meta文件。
2）在两台机器上都安装keepalived，两台服务器上有一个VIP漂移。keepalived通过检测脚本来检测服务器状态，当一台有问题时，VIP自动切换到另一台上。
3）client,chunk server, metalogger都是连接的VIP，所以当其中一台服务器挂掉后，并不影响服务。
三. 系统环境
操作系统为：
centos linux 6.x
mfs: 1.6.27
keepalived: 1.2.16
drbd: 8.4
服务器 IP地址
mfsmaster 172.21.21.80
mfsbackup 172.21.21.81
mfschunk1 172.21.21.77
mfschunk2 172.21.21.78
mfschunk3 172.21.21.79
VIP：172.21.21.82
在mfsmaster及mfsbackup上：
vim /etc/hosts:
mfsmaster 172.21.21.80
mfsbackup 172.21.21.81
四. DRBD安装及配置
以下操作在无特别说明的情况下都需要在mfsmaster及mfsbackup上执行：
1. 安装DRBD
安装elreo库：
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
安装DRBD:
yum -y install drbd84-utils kmod-drbd84
2. 配置DRBD
编辑DRBD配置文件：
vim /etc/drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
vim /etc/drbd.d/global_common.conf:
# DRBD is the result of over a decade of development by LINBIT.
# In case you need professional services for DRBD or have
# feature requests visit http://www.linbit.com
global {
usage-count no;
# minor-count dialog-refresh disable-ip-verification
}
common {
handlers {
# These are EXAMPLE handlers only.
# They may have severe implications,
# like hard resetting the node under certain circumstances.
# Be careful when chosing your poison.
# pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
# pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
# local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
wfc-timeout 30;
degr-wfc-timeout 30;
outdated-wfc-timeout 30;
}
options {
# cpu-mask on-no-data-accessible
}
disk {
# size on-io-error fencing disk-barrier disk-flushes
# disk-drain md-flushes resync-rate resync-after al-extents
# c-plan-ahead c-delay-target c-fill-target c-max-rate
# c-min-rate disk-timeout
on-io-error detach;
fencing resource-and-stonith;
}
net {
# protocol timeout max-epoch-size max-buffers unplug-watermark
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
# ping-timeout data-integrity-alg tcp-cork on-congestion
# congestion-fill congestion-extents csums-alg verify-alg
# use-rle
protocol C;
cram-hmac-alg sha1;
shared-secret "111111";
}
syncer {
# rate after al-extents use-rle cpu-mask verify-alg csums-alg
rate 40M;
}
}
vim /etc/drbd.d/mfs.res
resource mfs{
device /dev/drbd0;
meta-disk internal;
on mfsmaster{
disk /dev/sdb1;
address 172.21.21.80:9876;
}
on mfsbackup{
disk /dev/sdb1;
address 172.21.21.81:9876;
}
}
给两台机器分区：
[root@mfsmaster drbd.d]# fdisk -l
Disk /dev/sda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00076b23
Device Boot Start End Blocks Id System
/dev/sda1 1 262 2097152 82 Linux swap / Solaris
Partition 1 does not end on cylinder boundary.
/dev/sda2 * 262 3917 29359104 83 Linux
Disk /dev/sdb: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@mfsmaster drbd.d]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x4804513b.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-3916, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-3916, default 3916):
Using default value 3916
Command (m for help): p
Disk /dev/sdb: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4804513b
Device Boot Start End Blocks Id System
/dev/sdb1 1 3916 31455238+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.```
创建DRBD资源：
```[root@mfsmaster drbd.d]# drbdadm create-md mfs
initializing activity log
NOT initializing bitmap
Writing meta data...
New drbd meta data block successfully created.```
启动DRBD服务：
```[root@mfsmaster drbd.d]# service drbd start
Starting DRBD resources: [
create res: mfs
prepare disk: mfs
adjust disk: mfs
adjust net: mfs
]
....
查看初始化状态：
[root@mfsmaster drbd.d]# service drbd status
drbd driver loaded OK; device status:
version: 8.4.5 (api:1/proto:86-101)
GIT-hash: 1d360bde0e095d495786eaeb2a1ac76888e4db96 build by phil@Build64R6, 2014-10-28 10:32:53
m:res cs ro ds p mounted fstype
0:mfs Connected Secondary/Secondary Inconsistent/Inconsistent C
[root@mfsmaster drbd.d]#
[root@mfsbackup drbd.d]# service drbd status
drbd driver loaded OK; device status:
version: 8.4.5 (api:1/proto:86-101)
GIT-hash: 1d360bde0e095d495786eaeb2a1ac76888e4db96 build by phil@Build64R6, 2014-10-28 10:32:53
m:res cs ro ds p mounted fstype
0:mfs Connected Secondary/Secondary Inconsistent/Inconsistent C
[root@mfsbackup drbd.d]#
配置mfsmaster节点为主节点:
[root@mfsmaster drbd.d]# drbdadm primary --force mfs
如果报错，执行：
[root@mfsmaster drbd.d]# drbdadm -- --overwrite-data-of-peer primary all
drbd driver loaded OK; device status:
version: 8.4.5 (api:1/proto:86-101)
GIT-hash: 1d360bde0e095d495786eaeb2a1ac76888e4db96 build by phil@Build64R6, 2014-10-28 10:32:53
m:res cs ro ds p mounted fstype
0:mfs SyncSource Primary/Secondary UpToDate/Inconsistent C
... sync'ed: 0.3% (30648/30716)M
[root@mfsmaster drbd.d]#
可以看到开始同步数据了。
配置mfsmaster节点上格式化drbd设备：
[root@mfsmaster drbd.d]# mkfs -t ext4 /dev/drbd0
mke2fs 1.41.12 (17-May-2010)
文件系统标签=
操作系统:Linux
块大小=4096 (log=2)
分块大小=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1966080 inodes, 7863560 blocks
393178 blocks (5.00%) reserved for the super user
第一个数据块=0
Maximum filesystem blocks=4294967296
240 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
正在写入inode表: 完成
Writing superblocks and filesystem accounting information: 完成
This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
在mfsmaster节点上测试mount设备：
[root@mfsmaster drbd.d]# mkdir -p /data1/drbd/
[root@mfsmaster drbd.d]# mount /dev/drbd0 /data1/drbd/
[root@mfsmaster drbd.d]# ls /data1/drbd/
lost+found
mfsbackup也要建一下挂载目录：
[root@mfsbackup drbd.d]# mkdir -p /data1/drbd/
五. MFS 安装与配置
MFS的安装包在rpmfore库上有，也可以自己下载源码，打包成rpm安装。 在所有5台机器上执行：
[root@mfsmaster mfs]# wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
[root@mfsmaster mfs]# rpm -ivh rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
1. Mfs Master安装
安装mfs-master,mfs-metalogger,mfs-cgiserv,mfs-cgi，在mfsmaster和mfsbackup两台机器上执行：
[root@mfsmaster mfs]# yum -y install mfs-master mfs-metalogger mfs-cgiserv mfs-cgi
2. 配置mfs master
在mfsmaster修改mfs master配置文件：
[root@mfsmaster mfs]# cd /etc/mfs/
[root@mfsmaster mfs]# vim mfsmaster.cfg
WORKING_USER = nobody
WORKING_USER = nobody
WORKING_GROUP = nobody
SYSLOG_IDENT = mfsmaster
LOCK_MEMORY = 0
NICE_LEVEL = -19
EXPORTS_FILENAME = /etc/mfs/mfsexports.cfg
TOPOLOGY_FILENAME = /etc/mfs/mfstopology.cfg
DATA_PATH = /data1/drbd/mfs
BACK_LOGS = 50
BACK_META_KEEP_PREVIOUS = 1
REPLICATIONS_DELAY_INIT = 300
REPLICATIONS_DELAY_DISCONNECT = 3600
MATOML_LISTEN_HOST = *
MATOML_LISTEN_PORT = 9419
MATOML_LOG_PRESERVE_SECONDS = 600
MATOCS_LISTEN_HOST = *
MATOCS_LISTEN_PORT = 9420
MATOCL_LISTEN_HOST = *
MATOCL_LISTEN_PORT = 9421
CHUNKS_LOOP_MAX_CPS = 100000
CHUNKS_LOOP_MIN_TIME = 300
CHUNKS_SOFT_DEL_LIMIT = 10
CHUNKS_HARD_DEL_LIMIT = 25
CHUNKS_WRITE_REP_LIMIT = 2
CHUNKS_READ_REP_LIMIT = 10
ACCEPTABLE_DIFFERENCE = 0.1
SESSION_SUSTAIN_TIME = 86400
REJECT_OLD_CLIENTS = 0
# deprecated:
# CHUNKS_DEL_LIMIT - use CHUNKS_SOFT_DEL_LIMIT instead
[root@mfsmaster mfs]# mv mfsexports.cfg.dist mfsexports.cfg
[root@mfsmaster mfs]# mv mfstopology.cfg.dist mfstopology.cfg
[root@mfsmaster mfs]# mv mfsmetalogger.cfg.dist mfsmetalogger.cfg
[root@mfsmaster mfs]# vim mfsmetalogger.cfg
WORKING_USER = nobody
WORKING_GROUP = nobody
SYSLOG_IDENT = mfsmetalogger
LOCK_MEMORY = 0
NICE_LEVEL = -19
DATA_PATH = /var/lib/mfs
BACK_LOGS = 50
BACK_META_KEEP_PREVIOUS = 3
META_DOWNLOAD_FREQ = 24
MASTER_RECONNECTION_DELAY = 5
MASTER_HOST = 172.21.21.82
MASTER_PORT = 9419
MASTER_TIMEOUT = 60
# deprecated, to be removed in MooseFS 1.7
# LOCK_FILE = /var/run/mfs/mfsmetalogger.lock
把mfsmaster上写好的配置文件拷贝到mfsbackup上去：
[root@mfsmaster mfs]# pwd
/etc/mfs
[root@mfsmaster mfs]# scp * 172.21.21.81:/etc/mfs/
root@172.21.21.81's password:
mfsexports.cfg 100% 4060 4.0KB/s 00:00
mfsmaster.cfg 100% 936 0.9KB/s 00:00
mfsmetalogger.cfg 100% 387 0.4KB/s 00:00
mfstopology.cfg 100% 1123 1.1KB/s 00:00
[root@mfsmaster mfs]#
创建metadata存储目录：
[root@mfsmaster mfs]# mkdir -p /data1/drbd/mfs
[root@mfsmaster mfs]# cp /var/lib/mfs/metadata.mfs.empty /data1/drbd/mfs/metadata.mfs
[root@mfsmaster mfs]# chown -R nobody.nobody /data1/drbd/mfs
[root@mfsmaster mfs]#
在mfsmaster上启动测试：
[root@mfsmaster mfs]# mfsmaster start
working directory: /data1/drbd/mfs
lockfile created and locked
initializing mfsmaster modules ...
loading sessions ... file not found
if it is not fresh installation then you have to restart all active mounts !!!
exports file has been loaded
mfstopology: incomplete definition in line: 7
mfstopology: incomplete definition in line: 7
mfstopology: incomplete definition in line: 22
mfstopology: incomplete definition in line: 22
mfstopology: incomplete definition in line: 28
mfstopology: incomplete definition in line: 28
topology file has been loaded
loading metadata ...
create new empty filesystemmetadata file has been loaded
no charts data file - initializing empty charts
master <-> metaloggers module: listen on *:9419
master <-> chunkservers module: listen on *:9420
main master server module: listen on *:9421
mfsmaster daemon initialized properly
[root@mfsmaster mfs]#
3. 启动MFS监控页服务：
[root@mfsmaster mfs]# cd /usr/share/mfscgi/
[root@mfsmaster mfscgi]# ll
总用量 136
-rw-r--r-- 1 root root 1881 3月 13 10:18 chart.cgi
-rw-r--r-- 1 root root 270 3月 13 10:18 err.gif
-rw-r--r-- 1 root root 562 3月 13 10:18 favicon.ico
-rw-r--r-- 1 root root 510 3月 13 10:18 index.html
-rw-r--r-- 1 root root 3555 3月 13 10:18 logomini.png
-rw-r--r-- 1 root root 107456 3月 13 10:18 mfs.cgi
-rw-r--r-- 1 root root 5845 3月 13 10:18 mfs.css
[root@mfsmaster mfscgi]# chmod 755 *.cgi
[root@mfsmaster mfscgi]# ll
总用量 136
-rwxr-xr-x 1 root root 1881 3月 13 10:18 chart.cgi
-rw-r--r-- 1 root root 270 3月 13 10:18 err.gif
-rw-r--r-- 1 root root 562 3月 13 10:18 favicon.ico
-rw-r--r-- 1 root root 510 3月 13 10:18 index.html
-rw-r--r-- 1 root root 3555 3月 13 10:18 logomini.png
-rwxr-xr-x 1 root root 107456 3月 13 10:18 mfs.cgi
-rw-r--r-- 1 root root 5845 3月 13 10:18 mfs.css
[root@mfsmaster mfscgi]# mfscgiserv start
lockfile created and locked
starting simple cgi server (host: any , port: 9425 , rootpath: /usr/share/mfscgi)
权限mfsbackup上也要执行:
[root@mfsbackup drbd.d]# cd /usr/share/mfscgi/
[root@mfsbackup mfscgi]# chmod 755 *.cgi
用浏览器访问：http://172.21.21.80:9425/
4. 安装mfs chunkservers:
在mfschunk1，mfschunk2，mfschunk3上安装mfs chunkservers:
[root@mfschunk1 ~]# yum -y install mfs-chunkserver
[root@mfschunk1 mfs]# mv mfschunkserver.cfg.dist mfschunkserver.cfg
[root@mfschunk1 mfs]# mv mfshdd.cfg.dist mfshdd.cfg
[root@mfschunk1 mfs]# vim mfschunkserver.cfg
WORKING_USER = nobody
WORKING_GROUP = nobody
SYSLOG_IDENT = mfschunkserver
LOCK_MEMORY = 0
NICE_LEVEL = -19
DATA_PATH = /data1/mfs
MASTER_RECONNECTION_DELAY = 5
BIND_HOST = *
MASTER_HOST = 172.21.21.82
MASTER_PORT = 9420
MASTER_TIMEOUT = 60
CSSERV_LISTEN_HOST = *
CSSERV_LISTEN_PORT = 9422
HDD_CONF_FILENAME = /etc/mfs/mfshdd.cfg
HDD_TEST_FREQ = 10
# deprecated, to be removed in MooseFS 1.7
# LOCK_FILE = /var/run/mfs/mfschunkserver.lock
# BACK_LOGS = 50
# CSSERV_TIMEOUT = 5
/data1/mfs为MFS存储数据的挂载点：
[root@mfschunk1 mfs]# mkdir -p /data1/mfs
[root@mfschunk1 mfs]# chown nobody.nobody /data1/mfs
5. 启动mfs chunkserver:
[root@mfschunk3 ~]# mfschunkserver start
working directory: /data1/mfs
lockfile created and locked
initializing mfschunkserver modules ...
hdd space manager: path to scan: /data1/mfs/
hdd space manager: start background hdd scanning (searching for available chunks)
main server module: listen on *:9422
no charts data file - initializing empty charts
mfschunkserver daemon initialized properly
[root@mfschunk3 ~]#
设置开机启动：
[root@mfschunk1 mfs]# echo "/usr/sbin/mfschunkserver start" >> /etc/rc.local
可通过MFS的监控页面查看是否连接MFS MASTER成功。
6. MFS客户端配置：
[root@fft-vm-new-21-75 ~]# yum -y install fuse mfs-client
[root@fft-vm-new-21-75 ~]# mkdir /data1/mfs
[root@fft-vm-new-21-75 ~]# echo "mfsmount -H 172.21.21.82 /data1/mfs/" >> /etc/rc.local
六. keepalived安装与配置
1. 安装keepalived
keepalived好像centos linux 6.x有自带，但不是最新版本的，我用的是自己打的RPM包，不想打包用自带的也行。 在mfsmaster和mfsbackup机器上：
[root@mfsmaster mfscgi]# yum -y install keepalived
2. 配置keepalived
mfsmaster上的配置文件：
! Configuration File for keepalived
global_defs {
notification_email {
luohui@xxx.com
}
notification_email_from zabbix@xxx.com
smtp_server 172.21.2.51
smtp_connect_timeout 30
router_id mfs_master
}
vrrp_script chk_drbd {
script "/etc/keepalived/keepalived_drbd_mfs.sh check"
interval 15 # check every 30 seconds
weight -40 # if failed, decrease 40 of the priority
fall 2 # require 2 failures for failures
rise 1 # require 1 sucesses for ok
}
# net.ipv4.ip_nonlocal_bind=1
vrrp_instance mfs {
state BACKUP
interface eth1
virtual_router_id 10
# mcast_src_ip 172.21.21.80
nopreempt
priority 100
advert_int 1
debug
authentication {
auth_type PASS
auth_pass xxx_mfs
}
virtual_ipaddress {
172.21.21.82
}
track_script {
chk_drbd
}
notify_master "/etc/keepalived/keepalived_drbd_mfs.sh master"
notify_backup "/etc/keepalived/keepalived_drbd_mfs.sh backup"
notify_fault "/etc/keepalived/keepalived_drbd_mfs.sh fault"
notify_stop "/etc/keepalived/keepalived_drbd_mfs.sh fault"
smtp_alert
}
mfsbackup上的配置文件：
[root@mfsbackup keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
luohui@xxx.com
}
notification_email_from zabbix@xxx.com
smtp_server 172.21.2.51
smtp_connect_timeout 30
router_id mfs_slave
}
vrrp_script chk_drbd {
script "/etc/keepalived/keepalived_drbd_mfs.sh check"
interval 15 # check every 30 seconds
weight -40 # if failed, decrease 40 of the priority
fall 2 # require 2 failures for failures
rise 1 # require 1 sucesses for ok
}
# net.ipv4.ip_nonlocal_bind=1
vrrp_instance mfs {
state BACKUP
interface eth1
virtual_router_id 10
#mcast_src_ip 172.21.21.81
priority 80
advert_int 1
debug
authentication {
auth_type PASS
auth_pass xxx_mfs
}
virtual_ipaddress {
172.21.21.82
}
debug
track_script {
chk_drbd
}
notify_master "/etc/keepalived/keepalived_drbd_mfs.sh master"
notify_backup "/etc/keepalived/keepalived_drbd_mfs.sh backup"
notify_fault "/etc/keepalived/keepalived_drbd_mfs.sh fault"
notify_stop "/etc/keepalived/keepalived_drbd_mfs.sh fault"
smtp_alert
}
3. 编写keepalived检测脚本
检测脚本两台服务器都一样：
[root@mfsbackup keepalived]# vim /etc/keepalived/keepalived_drbd_mfs.sh
#!/bin/bash
# Copyright (c) 2015 Farmer Luo
# Script to handle DRBD & MFS from keepalived.
#
# Usage: keepalived_drbd_mfs.sh action
#
# Where action is :
#
# check : check that the DRBD resource is Primary, Connected and UpToDate
# and that MFS is running
#
# backup: set to backup state. Just checking than DRBD is connected and syncing...
# fault : set to fault state. Stop MFS, unmount partition, set the DRBD resource to Secondary
# master: set to master state. Set the DRBD resource to Primary, mount partition, start MFS
# then invalidate remote DRBD resource
#
# Note: you can use $MAINTENANCE (/etc/keepalived/maintenance) to disable MFS checks
# in case of short MFS maintenance
#
# Usage func :
[ "$1" = "--help" ] && { sed -n -e '/^# Usage:/,/^$/ s/^# \?//p' < $0; exit; }
#
# CONFIG
#
DRBDADM="/sbin/drbdadm"
# DRBD resource
DRBDRESOURCE="mfs"
# local mount point
MOUNTPOINT="/data1/drbd"
# mfsmaster metadata store dir
MFSMETADIR="/data1/drbd/mfs"
# mfsmetalogger metadata store dir
MFSLOGERBAKDIR="/var/lib/mfs"
# mfsmaster user and group
MFSUSER="nobody"
MFSGROUP="nobody"
# how to handle potential split-brain
# 0: manual
# 1: invalidate local data (Recommend)
# 2: invalidate remote data
SPLIT_BRAIN_METHOD=1
# maintenance flag: used to do maintenance on DRBD without switch between nodes
MAINTENANCE="/etc/keepalived/maintenance"
MFSMASTER="/usr/sbin/mfsmaster"
MFSRESTORE="/usr/sbin/mfsmetarestore"
MFSLOGGER="/usr/sbin/mfsmetalogger"
MFSCGI="/usr/sbin/mfscgiserv"
# Finally, to overwrite those defaults :
CONFIG="/etc/keepalived/keepalived_drbd_mfs_config.sh"
#
# CONFIG LOGGER
#
# tail -f /var/log/syslog | grep Keep
LOG="logger -t KeepAlived[$$] -p syslog" # do not use -i
LOGDEBUG="$LOG.debug"
LOGINFO="$LOG.info"
LOGWARN="$LOG.warn"
LOGERR="$LOG.err"
LOGFIFO=$( mktemp -u /var/tmp/$( basename $0 )_fifo.XXXXXX )
LOGPID=0
#
# local vars
#
LOCKACTIONFILE="/tmp/keep_drbd_"
status=
role=
cstate=
dstate=
warmstate="${LOCKACTIONFILE}_warm_state"
init_status() {
# start with NOK status
if [ -z "$status" ]
then
status=1
fi
role=$( $DRBDADM role $DRBDRESOURCE )
cstate=$( $DRBDADM cstate $DRBDRESOURCE )
dstate=$( $DRBDADM dstate $DRBDRESOURCE )
}
set_status() {
status=$1
if [ -n "$role" ]
then
$LOGDEBUG "CheckDRBD: $role $cstate $dstate => $status"
fi
return $status
}
# Check that MFS is responding
# Return:
# 0 if everything is OK (or in maintenance mode)
# 1 if mfsmaster process not running
check_mfs() {
if [ -e $MAINTENANCE ]
then
return 0
fi
if [ $( pidof mfsmaster | wc -w ) -gt 0 ]
then
if [ $( ps auxw | grep 'mfscgiserv' | grep -v grep | awk '{print $2;}' | wc -w ) -eq 0 ]
then
$LOGWARN "mfscgiserv process not running,starting mfscgiserv now..."
start_mfscgi
fi
return 0
else
$LOGWARN "mfsmaster process isn't running!"
return 1
fi
}
check() {
# CHECK DRBD
init_status
status=1
# at least UpToDate
if echo $dstate | grep -q ^UpToDate/
then
# Primary + UpToDate
status=0
if [ "$cstate" = "StandAlone" ]
then
$LOGWARN "$role but not connected"
reconnect_drbd
fi
else
$LOGWARN "DSTATE: $dstate"
# status=1 ?
status=0
fi
# Stop checking if already in fault ...
if [ $status -gt 0 ]
then
set_status $status
return $?
fi
# (check mfs only if Primary)
if [ $status -eq 0 ] && echo "$role" | grep -q ^Primary
then
check_mfs
status=$?
fi
if [ $( pidof mfsmetalogger | wc -w ) -eq 0 ]
then
$LOGWARN "mfsmetalogger process not running,starting mfsmetalogger now..."
$MFSLOGGER start
fi
set_status $status
return $?
}
set_fault() {
# Lock : 1 set_fault at a time
LOCKACTIONFILE="${LOCKACTIONFILE}fault_lock"
lockd=$( mkdir $LOCKACTIONFILE 2>&1 )
if [ "$?" -gt 0 ]
then
$LOGDEBUG "Already setting to fault state...$$ $lockd"
return 1
fi
trap "rm -rf \"$LOCKACTIONFILE\"" 0
set_backup
return $?
}
start_mfsmaster() {
if [ $( pidof mfsmaster | wc -w ) -gt 0 ]
then
$LOGWARN "mfsmaster already started, restart mfsmaster now!"
stop_mfsmaster
fi
if [ ! -f "${MFSMETADIR}/metadata.mfs" ]
then
if [ $( pidof mfsmetalogger | wc -w ) -gt 0 ]
then
$LOGWARN "mfsmaster metadata.mfs file not exist,exec mfsmetarestore -a restore data!"
$MFSRESTORE -a -d ${MFSLOGERBAKDIR}
if [ -f "${MFSLOGERBAKDIR}/metadata.mfs" ]
then
cp "${MFSLOGERBAKDIR}/metadata.mfs" "${MFSMETADIR}/metadata.mfs"
chown ${MFSUSER}.${MFSGROUP} "${MFSMETADIR}/metadata.mfs"
$LOGWARN "mfsmetarestore -a restore data successful!"
else
$LOGWARN "mfsmetarestore -a restore data failure!"
if [ -f "${MFSMETADIR}/metadata.mfs.back" ]
then
$LOGWARN "try use ${MFSMETADIR}/metadata.mfs.back file!"
mv "${MFSMETADIR}/metadata.mfs.back" "${MFSMETADIR}/metadata.mfs"
chown ${MFSUSER}.${MFSGROUP} "${MFSMETADIR}/metadata.mfs"
else
$LOGWARN "mfsmaster metadata.mfs file not exist and can't metaloger data restore or metadata.mfs.back !"
return 1
fi
fi
else
if [ -f "${MFSMETADIR}/metadata.mfs.back" ]
then
$LOGWARN "mfsmaster metadata.mfs file not exist,use ${MFSMETADIR}/metadata.mfs.back file!"
mv "${MFSMETADIR}/metadata.mfs.back" "${MFSMETADIR}/metadata.mfs"
chown ${MFSUSER}.${MFSGROUP} "${MFSMETADIR}/metadata.mfs"
else
$LOGWARN "mfsmaster metadata.mfs file not exist and can't metaloger data restore or metadata.mfs.back !"
return 1
fi
fi
fi
$LOGDEBUG "starting mfsmaster ..."
$MFSMASTER start
sleep 1
if [ $( pidof mfsmaster | wc -w ) -gt 0 ]
then
$LOGDEBUG "mfsmaster start successful."
return 0
else
$LOGWARN "mfsmaster start failure!"
return 1
fi
}
start_mfscgi() {
if [ $( ps auxw | grep 'mfscgiserv' | grep -v grep | awk '{print $2;}' | wc -w ) -gt 0 ]
then
$LOGWARN "mfscgiserv already started, restart mfscgiserv now!"
stop_mfscgi
fi
$LOGDEBUG "starting mfscgiserv ..."
$MFSCGI start
sleep 1
if [ $( ps auxw | grep 'mfscgiserv' | grep -v grep | awk '{print $2;}' | wc -w ) -gt 0 ]
then
$LOGDEBUG "mfscgiserv start successful."
return 0
else
$LOGWARN "mfscgiserv start failure!"
return 1
fi
}
stop_mfsmaster() {
if [ $( pidof mfsmaster | wc -w ) -gt 0 ]
then
$LOGWARN "stop mfsmaster now!"
$MFSMASTER stop
sleep 1
else
return 0
fi
if [ $( pidof mfsmaster | wc -w ) -gt 0 ]
then
mfs_pid=$( ps auxw | grep 'mfsmaster start' | grep -v grep | awk '{print $2;}' )
$LOGWARN "KILLING -9 mfsmaster[$mfs_pid]"
/bin/kill -9 $mfs_pid
else
$LOGDEBUG "mfsmaster has stopped"
fi
return 0
}
stop_mfscgi() {
if [ $( ps auxw | grep 'mfscgiserv' | grep -v grep | awk '{print $2;}' | wc -w ) -gt 0 ]
then
$LOGDEBUG "stop mfscgiserv now"
$MFSCGI stop
sleep 1
else
return 0
fi
if [ $( ps auxw | grep 'mfscgiserv' | grep -v grep | awk '{print $2;}' | wc -w ) -gt 0 ]
then
mfs_pid=$( ps auxw | grep 'mfscgiserv' | grep -v grep | awk '{print $2;}' )
$LOGWARN "KILLING -9 mfscgiserv[$mfs_pid]"
/bin/kill -9 $mfs_pid
else
$LOGDEBUG "mfscgiserv has stopped"
fi
return 0
}
set_backup() {
stop_mfscgi
stop_mfsmaster
# We must be sure to be in replication and secondary state
ensure_drbd_secondary
}
set_drbd_secondary() {
if awk '{print $2}' /etc/mtab | grep -q "^$MOUNTPOINT"
then
$LOGWARN "Unmounting $MOUNTPOINT ..."
sleep 1
fuser -k -9 $MOUNTPOINT/
sleep 1
# -f Force unmount (in case of an unreachable NFS system). (Requires kernel 2.1.116 or later.)
# -l Lazy unmount. Detach the filesystem from the filesystem hierarchy now,
# and cleanup all references to the filesystem as soon as it is not busy anymore. (Requires kernel 2.4.11 or later.)
umount -l $MOUNTPOINT
fi
init_status
# If already Secondary and Connected, do nothing ...
if ! echo $role | grep -q ^Secondary
then
$LOGDEBUG "Set DRBD to secondary"
$DRBDADM disconnect $DRBDRESOURCE
sleep 1
if ! $DRBDADM secondary $DRBDRESOURCE
then
$LOGWARN "Unable to set $DRBDRESOURCE to secondary state"
echo LSOF:
lsof $MOUNTPOINT | while read line
do
$LOGWARN "lsof: $line"
done
return 1
fi
$LOGDEBUG "Sync DRBD"
if [ $SPLIT_BRAIN_METHOD -eq 1 ]
then
#$DRBDADM invalidate $DRBDRESOURCE
$DRBDADM connect --discard-my-data $DRBDRESOURCE
else
$DRBDADM connect $DRBDRESOURCE
fi
sleep 1
init_status
$LOGDEBUG "ROLE=$role CSTATE=$cstate DSTATE=$dstate"
fi
}
ensure_drbd_secondary() {
if ! is_drbd_secondary
then
set_drbd_secondary
return $?
fi
}
is_drbd_secondary() {
init_status
# If already Secondary and Connected, do nothing ...
if echo $role | grep -q ^Secondary
then
if [ "$cstate" != 'StandAlone' ]
then
$LOGDEBUG "Already in BACKUP state..."
return 0
fi
fi
return 1
}
reconnect_drbd() {
init_status
if [ "$cstate" = "StandAlone" ]
then
$DRBDADM connect $DRBDRESOURCE
fi
}
# WARNING set_master is called at keepalived start
# So if already in "good" state we must do nothing :)
set_master() {
init_status
if ! echo "$role" | grep -q ^Primary
then
$LOGDEBUG "Set DRBD to Primary"
$DRBDADM disconnect $DRBDRESOURCE
$DRBDADM primary $DRBDRESOURCE
init_status
if ! echo "$role" | grep -q ^Primary
then
$LOGWARN "Need to force PRIMARY ..."
$DRBDADM -- --overwrite-data-of-peer primary $DRBDRESOURCE
init_status
if ! echo "$role" | grep -q ^Primary
then
$LOGWARN "Unable to set PRIMARY"
return 1
else
$LOGWARN "Forced to PRIMARY : OK"
fi
fi
fi
# We connect to DRBD _after_ MFS has started, to limit IOPS
if [ $SPLIT_BRAIN_METHOD -eq 2 ]
then
$DRBDADM invalidate-remote $DRBDRESOURCE
fi
$LOGDEBUG "SyncTarget..."
# reconnect_drbd
$DRBDADM connect $DRBDRESOURCE
if ! awk '{print $2}' /etc/mtab | grep "^$MOUNTPOINT" >/dev/null
then
device=$( $DRBDADM sh-dev $DRBDRESOURCE )
#$LOGDEBUG "Filesystem check ..."
#FSTYPE="ext4"
#MOUNTOPTS=sync,noauto,noatime,noexec
#FSTYPE=$( grep -F $MOUNTPOINT /etc/fstab | grep -F $( $DRBDADM sh-dev $DRBDRESOURCE ) | awk '{print $3}' )
# Add "-f" to force ?
#if [ "$FSTYPE" != "xfs" ]
#then
# fsck.$FSTYPE -pD $device >&2
#fi
$LOGDEBUG "Mount ..."
#if ! mount -t $FSTYPE $device $MOUNTPOINT
if ! mount $device $MOUNTPOINT
then
$LOGERR "Unable to mount $MOUNTPOINT"
return 1
fi
fi
# Starting MFS
if ! start_mfsmaster
then
$LOGERR "can't start mfsmaster!"
return 1
fi
if ! start_mfscgi
then
$LOGERR "can't start mfscgiserv!"
return 1
fi
}
cleanup() {
kill $LOGPID
#$LOGDEBUG "Cleanup $LOGPID and $LOGFIFO"
# Workarround to log something if there is something ...
# logger will wait for stdin if argument is empty
# and "rm -f $file" will output nothing
yvain=$( mktemp -u /tmp/$( basename $0 )_yvain.XXXXXX )
rm -f $LOGFIFO 2>$yvain || $LOGDEBUG "Remove $LOGFIFO failed: $( cat $yvain )"
test -e $yvain && rm -f $yvain
#wait
}
# Redirect stderr to log
trap "cleanup" INT QUIT TERM TSTP EXIT
mkfifo $LOGFIFO
$LOGWARN < $LOGFIFO &
LOGPID=$!
exec 2>$LOGFIFO
if [ -e $CONFIG ]
then
. $CONFIG
fi
case "$1" in
check)
check
exit $?
;;
backup)
$LOGWARN "=> set to backup state <="
set_backup
exit $?
;;
fault)
$LOGWARN "=> set to fault state <="
set_fault
exit $?
;;
master)
$LOGWARN "=> set to master state <="
set_master
exit $?
;;
esac
给脚本执行权限：
[root@mfsbackup ~]# chmod 755 /etc/keepalived/keepalived_drbd_mfs.sh
好了，最后两台机器都启动测试就OK了。
[root@mfsbackup ~]# service keepalived start
七. 其它
1. 如何手动解决split brain问题
如果系统在某些情况下出现了split brain(脑裂)问题，可用如果方式处理：
- 如果主机上状态不为Primary，在主上执行：
drbdadm primary mfs
如果出错，用下面的命令：
drbdadm -- --overwrite-data-of-peer primary mfs (8.4以前版本)
drbdadm primary --force mfs (8.4版本)
- 在备机上执行 A、在备机上,断开连接
drbdadm disconnect mfs
B、在备机上,将主机状态改为secondary状态
drbdadm secondary mfs
C、在备机上,从远端同步数据
drbdadm -- --discard-my-data connect mfs (8.4以前版本)
drbdadm connect --discard-my-data mfs (8.4版本)
- 登入primary主机，再次发起连接：
drbdadm connect mfs
2. 抓包查看keepalived间的协议通信
tcpdump -i eth1 -n vrrp
