HA集群之三：corosync+pacemaker实现httpd服务的高可用
一、基礎(chǔ)概念
1、集群的組成結(jié)構(gòu)
HA Cluster:
Messaging and Infrastructure Layer|Heartbeat Layer 集群信息事務(wù)層
Membership Layer 集群成員關(guān)系層
CCM 投票系統(tǒng)
Resource Allocation Layer 資源分配層
CRM，
DC：LRM,PE,TE,CIB
Other:LRM,CIB
Resource Layer 資源代理
RA
2、OpenAIS: 開(kāi)放式應(yīng)用接口規(guī)范
提供了一種集群模式，包含集群框架、集群成員管理、通信方式、集群監(jiān)測(cè)，但沒(méi)有集群資源管理功能；
組件包括：AMF, CLM, CPKT, EVT等；分支不同，包含的組件略有區(qū)別；
分支：picacho, whitetank, wilson,
corosync (集群管理引擎)
是openais的一個(gè)子組件；
分裂成為兩個(gè)項(xiàng)目：
corosync, wilson(ais的接口標(biāo)準(zhǔn))
CentOS 5:
cman + rgmanager (RHCS 系統(tǒng)自帶)
CentOS 6:
cman + rgmanager
corosync + pacemaker
3、命令行管理工具：
crmsh: suse, CentOS 6.4-自帶
pcs: RedHat, CentOS 6.5+自帶
二、案例：實(shí)現(xiàn)HA httpd
1、安裝corosync + pacemaker
注意：確定HA集群的前提：時(shí)間同步，集群節(jié)點(diǎn)基于hostname命令顯示的主機(jī)名通信，節(jié)點(diǎn)之間的root用戶能夠基于密鑰認(rèn)證，考慮仲裁設(shè)備是否要使用
yum install corosync pacemaker -y
[root@BAIYU_175 ~]# rpm -ql corosync
/etc/corosync
/etc/corosync/corosync.conf.example
/etc/corosync/corosync.conf.example.udpu
/etc/corosync/service.d
/etc/corosync/uidgid.d
/etc/dbus-1/system.d/corosync-signals.conf
/etc/rc.d/init.d/corosync
/etc/rc.d/init.d/corosync-notifyd
/etc/sysconfig/corosync-notifyd
/usr/bin/corosync-blackbox
/usr/libexec/lcrso
/usr/libexec/lcrso/coroparse.lcrso
/usr/libexec/lcrso/objdb.lcrso
/usr/libexec/lcrso/quorum_testquorum.lcrso
/usr/libexec/lcrso/quorum_votequorum.lcrso
/usr/libexec/lcrso/service_cfg.lcrso
/usr/libexec/lcrso/service_confdb.lcrso
/usr/libexec/lcrso/service_cpg.lcrso
/usr/libexec/lcrso/service_evs.lcrso
/usr/libexec/lcrso/service_pload.lcrso
/usr/libexec/lcrso/vsf_quorum.lcrso
/usr/libexec/lcrso/vsf_ykd.lcrso
/usr/sbin/corosync
/usr/sbin/corosync-cfgtool
/usr/sbin/corosync-cpgtool
/usr/sbin/corosync-fplay
/usr/sbin/corosync-keygen
/usr/sbin/corosync-notifyd
/usr/sbin/corosync-objctl
/usr/sbin/corosync-pload
/usr/sbin/corosync-quorumtool
/usr/share/doc/corosync-1.4.7
/usr/share/doc/corosync-1.4.7/LICENSE
/usr/share/doc/corosync-1.4.7/SECURITY
/usr/share/man/man5/corosync.conf.5.gz
/usr/share/man/man8/confdb_keys.8.gz
/usr/share/man/man8/corosync-blackbox.8.gz
/usr/share/man/man8/corosync-cfgtool.8.gz
/usr/share/man/man8/corosync-cpgtool.8.gz
/usr/share/man/man8/corosync-fplay.8.gz
/usr/share/man/man8/corosync-keygen.8.gz
/usr/share/man/man8/corosync-notifyd.8.gz
/usr/share/man/man8/corosync-objctl.8.gz
/usr/share/man/man8/corosync-pload.8.gz
/usr/share/man/man8/corosync-quorumtool.8.gz
/usr/share/man/man8/corosync.8.gz
/usr/share/man/man8/corosync_overview.8.gz
/usr/share/snmp/mibs/COROSYNC-MIB.txt
/var/lib/corosync
/var/log/cluster
2、配置corosync
[root@BAIYU_173 ~]# cd /etc/corosync/
[root@BAIYU_173 corosync]# ls
corosync.conf.example corosync.conf.example.udpu service.d uidgid.d
[root@BAIYU_173 corosync]# cp corosync.conf.example corosync.conf
1）主配置文件/etc/corosync.conf詳解：
compatibility: whitetank #兼容08.以前的版本
secauth:off 不打開(kāi)集群安全認(rèn)證 推薦打開(kāi) on 使用 corosync-keygen 生成密鑰
threads: 0 定義多線程工作模式 0表示不使用線程而使用進(jìn)程
ringnumber:0 環(huán)數(shù)目，類似ttl 默認(rèn)即可
bindnetaddr: 192.168.1.0 多播地址監(jiān)聽(tīng)哪個(gè)網(wǎng)絡(luò)地址，填上自己的網(wǎng)絡(luò)地址即可：192.168.100.0
mcastaddr: 239.255.1.1 指定多播地址 225.25.25.25
mcastport: 5405 多播使用的端口utp
to_logfile: yes
to_syslog: yes 使用一個(gè)文件記錄地址即可 off
在/etc/crorosync/corosync.conf中添加以下內(nèi)容：
amf {
mode: disabled
}
查看修改后的/etc/corosync/corosync.conf：
[root@BAIYU_173 corosync]# grep -v '#' corosync.conf|grep -v '^$'
compatibility: whitetank
totem {
version: 2
secauth: on
threads: 0
interface {
ringnumber: 0
bindnetaddr: 192.168.100.0
mcastaddr: 225.25.25.25
mcastport: 5405
ttl: 1
}
}
logging {
fileline: off
to_logfile: yes
logfile: /var/log/cluster/corosync.log
to_syslog: off
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
2）在/etc/corosync/service.d中新建pacemaker文件：
[root@BAIYU_173 service.d]# cat pacemaker
service {
name: pacemaker
ver: 1
}
[root@BAIYU_173 corosync]# corosync-keygen #生成用于集群驗(yàn)證時(shí)的密鑰
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /etc/corosync/authkey.
[root@BAIYU_173 corosync]# ls
authkey corosync.conf.example corosync.conf.orig uidgid.d
corosync.conf corosync.conf.example.udpu service.d
3）將密鑰和corosync的配置文件復(fù)制給集群其它節(jié)點(diǎn)：
[root@BAIYU_173 corosync]# scp -p authkey corosync.conf 192.168.100.175:/etc/corosync
3、啟動(dòng)corosync并驗(yàn)證:
[root@BAIYU_173 corosync]# service corosync start #要先啟動(dòng)pacemaker依賴于corosync所以要先啟動(dòng)corosync
Starting Corosync Cluster Engine (corosync): [確定]
[root@BAIYU_173 corosync]# netstat -nlptu
Active Internet connections (only servers)
[root@BAIYU_173 ~]# service pacemaker start #如果啟動(dòng)失敗，先重啟corosync再重啟pacemaker
Starting Pacemaker Cluster Manager[確定]
udp 0 0 0.0.0.0:53243 0.0.0.0:* 1372/rpc.statd
[root@BAIYU_173 ~]# netstat -nlptu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1348/rpcbind
tcp 0 0 0.0.0.0:57269 0.0.0.0:* LISTEN 1372/rpc.statd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1551/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1642/master
udp 0 0 192.168.100.173:5404 0.0.0.0:* 2330/corosync
udp 0 0 192.168.100.173:5405 0.0.0.0:* 2330/corosync
udp 0 0 225.25.25.25:5405 0.0.0.0:* 2330/corosync
udp 0 0 0.0.0.0:671 0.0.0.0:* 1348/rpcbind
udp 0 0 127.0.0.1:703 0.0.0.0:* 1372/rpc.statd
udp 0 0 0.0.0.0:111 0.0.0.0:* 1348/rpcbind
udp 0 0 0.0.0.0:53243 0.0.0.0:* 1372/rpc.statd
1）驗(yàn)證corosync引擎是否正常啟動(dòng)：
[root@BAIYU_173 corosync]# grep -e 'Corosync Cluster Engine' -e 'configuration file' /
var/log/cluster/corosync.log
Oct 24 17:55:00 corosync [MAIN ] Corosync Cluster Engine ('1.4.7'): started and ready to provide service.
Oct 24 17:55:00 corosync [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
2）驗(yàn)證初始化成員節(jié)點(diǎn)通知是否正常發(fā)出：
[root@BAIYU_173 corosync]# grep TOTEM /var/log/cluster/corosync.log
Oct 24 17:55:00 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
Oct 24 17:55:00 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Oct 24 17:55:00 corosync [TOTEM ] The network interface [192.168.100.173] is now up.
Oct 24 17:55:00 corosync [TOTEM ] Process pause detected for 613 ms, flushing membership messages.
Oct 24 17:55:00 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 24 17:55:23 corosync [TOTEM ] A processor failed, forming new configuration.
Oct 24 17:55:23 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Oct 24 17:55:27 corosync [TOTEM ] A processor failed, forming new configuration.
Oct 24 17:55:27 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
3）檢查啟動(dòng)過(guò)程中是否有錯(cuò)誤產(chǎn)生：
下面的錯(cuò)誤信息表示pacemaker不久之后講不再作為corosync的插件運(yùn)行，因此，建議使用cman作為集群基礎(chǔ)架構(gòu)服務(wù)，此處可安全忽略
[root@BAIYU_173 corosync]# grep ERROR: /var/log/cluster/corosync.log | grep -v unpack_
resources
Oct 24 17:55:00 corosync [pcmk ] ERROR: process_ais_conf: You have configured a cluster using the Pacemaker plugin for Corosync. The plugin is not supported in this environment and will be removed very soon.
Oct 24 17:55:00 corosync [pcmk ] ERROR: process_ais_conf: Please see Chapter 8 of 'Clusters from Scratch' (http://www.clusterlabs.org/doc) for details on using Pacemaker with CMAN
Oct 24 17:55:02 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process mgmtd exited (pid=25335, rc=100) # 可以忽略，
3）查看pacemaker是否正常啟動(dòng)：
[root@BAIYU_173 corosync]# grep pcmk_startup /var/log/cluster/corosync.log
Oct 24 17:55:00 corosync [pcmk ] info: pcmk_startup: CRM: Initialized
Oct 24 17:55:00 corosync [pcmk ] Logging: Initialized pcmk_startup
Oct 24 17:55:00 corosync [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
Oct 24 17:55:00 corosync [pcmk ] info: pcmk_startup: Service: 9
Oct 24 17:55:00 corosync [pcmk ] info: pcmk_startup: Local hostname: BAIYU_173
4、安裝crm_sh:
定義資源的命令行
可以只在一個(gè)節(jié)點(diǎn)上安裝crm_sh，配置結(jié)果會(huì)送給DC，然后DC同步給其它節(jié)點(diǎn)，DC在集群系統(tǒng)啟動(dòng)時(shí)集群系統(tǒng)自動(dòng)選舉出
[root@BAIYU_173 ~]# cd /etc/yum.repos.d/
[root@BAIYU_173 ~]# wget http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/network:ha-clustering:Stable.repo
[root@BAIYU_173 ~]# yum install crmsh
[root@BAIYU_173 ~]# crm
crm(live)# status # 查看集群系統(tǒng)狀態(tài)；如果這里報(bào)錯(cuò)很可能是pacemaker停止了或前面配置有誤
Last updated: Sun Oct 25 22:41:42 2015
Last change: Sun Oct 25 22:11:26 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
0 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
crm(live)# configure
crm(live)configure# show # 查看集群系統(tǒng)信息
node BAIYU_173
node BAIYU_175
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2
crm(live)configure#
5、crm使用詳解
crm的常用一級(jí)子命令：
cib manage shadow CIBs #cib沙盒
resource resources management #管理資源 ，所有的資源的狀態(tài)都在這個(gè)子命令后定義
configure CRM cluster configuration #編輯集群配置信息，所有資源的定義
node nodes management #集群節(jié)點(diǎn)管理子命令
options user preferences #用戶優(yōu)先級(jí)
history CRM cluster history#
site Geo-cluster support
ra resource agents information center #資源代理子命令（所有與資源代理相關(guān)的程都在此命令之下）
status show cluster status #顯示當(dāng)前集群的狀態(tài)信息
help,? show help (help topics for list of topics)#查看當(dāng)前區(qū)域可能的命令
end,cd,up go back one level #返回第一級(jí)crm(live)#
quit,bye,exit exit the program #退出crm（live）交互模式
configure常用的子命令：
所有資源的定義都是在此子命令下完成的
node define a cluster node #定義一個(gè)集群節(jié)點(diǎn)
primitive define a resource #定義資源
monitor add monitor operation to a primitive #對(duì)一個(gè)資源添加監(jiān)控選項(xiàng)（如超時(shí)時(shí)間，啟動(dòng)失敗后的操作）
group define a group #定義一個(gè)組類型（將多個(gè)資源整合在一起）
clone define a clone #定義一個(gè)克隆類型（可以設(shè)置總的克隆數(shù)，每一個(gè)節(jié)點(diǎn)上可以運(yùn)行幾個(gè)克隆）
ms define a master-slave resource #定義一個(gè)主從類型（集群內(nèi)的節(jié)點(diǎn)只能有一個(gè)運(yùn)行主資源，其它從的做備用）
rsc_template define a resource template #定義一個(gè)資源模板
location a location preference #定義位置約束優(yōu)先級(jí)（默認(rèn)運(yùn)行于那一個(gè)節(jié)點(diǎn)（如果位置約束的值相同，默認(rèn)傾向性那一個(gè)高，就在那一個(gè)節(jié)點(diǎn)上運(yùn)行））
colocation colocate resources #排列約束資源（多個(gè)資源在一起的可能性）
order order resources #資源的啟動(dòng)的先后順序
rsc_ticket resources ticket dependency
property set a cluster property #設(shè)置集群全局屬性
rsc_defaults set resource defaults #設(shè)置資源默認(rèn)屬性（粘性）
fencing_topology node fencing order #隔離節(jié)點(diǎn)順序
role define role access rights #定義角色的訪問(wèn)權(quán)限
user define user access rights #定義用用戶訪問(wèn)權(quán)限
op_defaults set resource operations defaults #設(shè)置資源默認(rèn)選項(xiàng)
schema set or display current CIB RNG schema
show display CIB objects #顯示集群信息庫(kù)對(duì)
edit edit CIB objects #編輯集群信息庫(kù)對(duì)象（vim模式下編輯）
filter filter CIB objects #過(guò)濾CIB對(duì)象
delete delete CIB objects #刪除CIB對(duì)象
default-timeouts set timeouts for operations to minimums from the meta-data
rename rename a CIB object #重命名CIB對(duì)象
modgroup modify group #改變資源組
refresh refresh from CIB #重新讀取CIB信息
erase erase the CIB #清除CIB信息
ptest show cluster actions if changes were committed
rsctest test resources as currently configured
cib CIB shadow management
cibstatus CIB status management and editing
template edit and import a configuration from a template
commit commit the changes to the CIB #將更改后的信息提交寫入CIB
verify verify the CIB with crm_verify #CIB語(yǔ)法驗(yàn)證
upgrade upgrade the CIB to version 1.0
save save the CIB to a file #將當(dāng)前CIB導(dǎo)出到一個(gè)文件中（導(dǎo)出的文件存于切換crm 之前的目錄）
load import the CIB from a file #從文件內(nèi)容載入CIB
graph generate a directed graph
xml raw xml
help show help (help topics for list of topics) #顯示幫助信息
end go back one level #回到第一級(jí)(crm(live)#)
quit exit the program #退出crm交互模式
常用的全局屬性配置：
configure
property
stonith-eanabled=true|false true 默認(rèn) “沒(méi)有報(bào)頭設(shè)備時(shí)使用”
no-quorum-policy=stopped 停止 默認(rèn) “簡(jiǎn)單的2節(jié)點(diǎn)集群時(shí)使用”
ignore 忽略 仍然運(yùn)行
freeze 凍結(jié) 舊連接可以繼續(xù)訪問(wèn)，新的連接被拒絕
suicide 殉情
default-resource-stickiness=#|-#|inf|-inf “資源對(duì)當(dāng)前節(jié)點(diǎn)的粘性”
symetric-cluster=true|false 是否是對(duì)稱集群 true：rsc可以運(yùn)行于集群中的任意節(jié)點(diǎn)
resource子命令
所有的資源狀態(tài)都此處控制
status show status of resources?#顯示資源狀態(tài)信息
start start a resource?#啟動(dòng)一個(gè)資源
stop stop a resource?#停止一個(gè)資源
restart restart a resource?#重啟一個(gè)資源
promote promote a master-slave resource?#提升一個(gè)主從資源
demote demote a master-slave resource?#降級(jí)一個(gè)主從資源
manage put a resource into managed mode
unmanage put a resource into unmanaged mode
migrate migrate a resource to another node?#將資源遷移到另一個(gè)節(jié)點(diǎn)上
unmigrate unmigrate a resource to another node
param manage a parameter of a resource?#管理資源的參數(shù)
secret manage sensitive parameters?#管理敏感參數(shù)
meta manage a meta attribute?#管理源屬性
utilization manage a utilization attribute
failcount manage failcounts?#管理失效計(jì)數(shù)器
cleanup cleanup resource status?#清理資源狀態(tài)
refresh refresh CIB from the LRM status?#從LRM（LRM本地資源管理）更新CIB（集群信息庫(kù)），在
reprobe probe?for?resources not started by the CRM?#探測(cè)在CRM中沒(méi)有啟動(dòng)的資源
trace start RA tracing?#啟用資源代理（RA）追蹤
untrace stop RA tracing?#禁用資源代理（RA）追蹤
help show help (help topics?for?list of topics)?#顯示幫助
end go back one level?#返回一級(jí)（crm(live)#）
quit?exit?the program?#退出交互式程序
node子命令
節(jié)點(diǎn)管理和狀態(tài)命令
status show nodes status as XML #以xml格式顯示節(jié)點(diǎn)狀態(tài)信息
show show node #命令行格式顯示節(jié)點(diǎn)狀態(tài)信息
standby put node into standby #模擬指定節(jié)點(diǎn)離線（standby在后面必須的FQDN）
online set node online # 節(jié)點(diǎn)重新上線
maintenance put node into maintenance mode
ready put node into ready mode
fence fence node #隔離節(jié)點(diǎn)
clearstate Clear node state #清理節(jié)點(diǎn)狀態(tài)信息
delete delete node #刪除 一個(gè)節(jié)點(diǎn)
attribute manage attributes
utilization manage utilization attributes
status-attr manage status attributes
help show help (help topics for list of topics)
end go back one level
quit exit the program
ra子命令
資源代理類別都在此處
classes list classes and providers #為資源代理分類
list list RA for a class (and provider)#顯示一個(gè)類別中的提供的資源
meta show meta data for a RA #顯示一個(gè)資源代理序的可用參數(shù)（如meta ocf:heartbeat:IPaddr2）
providers show providers for a RA and a class
help show help (help topics for list of topics)
end go back one level
quit exit the program
6、定義資源
webip: 192.168.100.100
那在這里，我們?nèi)绾稳ヅ渲靡粋€(gè)資源呢，雖然它跟heartbeat略有區(qū)別，但是概念基本上是一樣的，下面我們就來(lái)配置一個(gè)web資源吧！
1）配置兩點(diǎn)的corosync+pacemaker集群，設(shè)置兩個(gè)全局屬性：
stonith-enabled=false
no-quorum-policy=ignore
由于我們的corosync默認(rèn)是啟用stonith功能的，但是我們這里沒(méi)有stonith設(shè)備，如果我們直接去配置資源的話，由于沒(méi)有stonith設(shè)備，所以資源的切換并不會(huì)完成，所以要禁用stonith功能，但禁用stonoith需要我們?nèi)ヅ渲眉旱娜謘tonith屬性，全局屬性是對(duì)所有的節(jié)點(diǎn)都生效；
要注意：2節(jié)點(diǎn)的集群，如果一個(gè)節(jié)點(diǎn)掛了，就不擁有法定票數(shù)了，那資源是不會(huì)切換的，資源默認(rèn)會(huì)停止
集群的策略有幾種：
stopped ：停止服務(wù) （默認(rèn)）?
ignore ：忽略，繼續(xù)運(yùn)行?
freeze ：凍結(jié)，已經(jīng)連接的請(qǐng)求繼續(xù)響應(yīng)，新的請(qǐng)求不再響應(yīng)?
suicide ：自殺，將服務(wù)kill掉
a、禁用stonith-enable
[root@BAIYU_173 ~]# crm configure #進(jìn)入crm命令行模式配置資源等
crm(live)configure# property #切換到property目錄下，可以用兩次tab鍵進(jìn)行補(bǔ)全和查看
crm(live)configure# help property #不知道用法就help
Set a cluster property
Set the cluster (crm_config) options.
Usage:
property [$id=<set_id>] [rule ...] <option>=<value> [<option>=<value> ...]
Example:
property stonith-enabled=true
property rule date spec years=2014 stonith-enabled=false
crm(live)configure# property stonith-enabled=false #禁用stonith-enabled
crm(live)configure# verify #檢查設(shè)置的屬性是否正確
crm(live)configure# commit #檢查沒(méi)問(wèn)題就可以提交了
crm(live)configure# show #查看當(dāng)前集群的所有配置信息
node BAIYU_173
node BAIYU_175 #兩個(gè)節(jié)點(diǎn)
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \ #DC的版本號(hào)
cluster-infrastructure="classic openais (with plugin)" \ #集群的基礎(chǔ)架構(gòu)，使用的是OpenAIS，插件式的
expected-quorum-votes=2 \ #期望節(jié)點(diǎn)的票數(shù)
stonith-enabled=false #禁用stonith功能
b、忽略投票規(guī)則
crm(live)configure# property no-quorum-policy=ignore
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
primitive 語(yǔ)法：
primitive <rsc_id> class:provider:ra params param1=value1 param2=value2 op op1 param1=value op op2 parma1=value1
c、添加資源
crm(live)configure# primitive webip ocf:heartbeat:IPaddr params ip=192.168.100.100 nic=eth0 cidr_netmask=24
crm(live)configure# verify # 檢查語(yǔ)法是否有錯(cuò)
crm(live)configure# commit # 提交給內(nèi)核
Online: [ BAIYU_173 BAIYU_175 ]
crm(live)# status
Last updated: Mon Oct 26 16:24:53 2015
Last change: Mon Oct 26 16:24:21 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
1 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
webip (ocf::heartbeat:IPaddr): Started BAIYU_173 #可以看到webip 已經(jīng)在BAIYU_173上運(yùn)行了
crm(live)#
[root@BAIYU_173 ~]# ip addr # 注意configure查看不到
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:6a:dc:8d brd ff:ff:ff:ff:ff:ff
inet 192.168.100.173/24 brd 192.168.100.255 scope global eth0
inet 192.168.100.100/24 brd 192.168.100.255 scope global secondary eth0 #webip
crm(live)configure# primitive webserver lsb:httpd
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# cd
crm(live)# status
Last updated: Mon Oct 26 17:52:46 2015
Last change: Mon Oct 26 17:52:34 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
webip (ocf::heartbeat:IPaddr): Started BAIYU_175
webserver (lsb:httpd): Started BAIYU_173
上面發(fā)現(xiàn)2個(gè)資源默認(rèn)是分散在系統(tǒng)節(jié)點(diǎn)中
把資源定義在同1個(gè)節(jié)點(diǎn)有2種方法：定義組或位置約束：
1）新建組，把資源按啟動(dòng)順序先后加入組
crm(live)configure# group webservice webip webserver #定義組,組中資源在同一個(gè)節(jié)點(diǎn)并 按順序啟動(dòng)
crm(live)configure# cd
crm(live)# status
Last updated: Mon Oct 26 18:09:16 2015
Last change: Mon Oct 26 18:07:19 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Started BAIYU_173
webserver (lsb:httpd): Started BAIYU_173
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
primitive webserver lsb:httpd
group webservice webip webserver
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)configure# cd ../resource
crm(live)resource# status webservice # 資源在運(yùn)行時(shí)不能刪除，要先停止
resource webservice is running on: BAIYU_173
crm(live)resource# stop webservice
crm(live)resource# status webservice
resource webservice is NOT running
crm(live)resource# cd ../configure
crm(live)configure# delete webservice #刪除組
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
primitive webserver lsb:httpd
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)# status
Last updated: Mon Oct 26 18:47:24 2015
Last change: Mon Oct 26 18:46:41 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ] #資源又分散在節(jié)點(diǎn)中
webip (ocf::heartbeat:IPaddr): Started BAIYU_173
webserver (lsb:httpd): Started BAIYU_175
2）用colocation,ored定義資源排列約束，和順序約束：
crm(live)configure# colocation webserver_with_webip inf: webserver webip
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# crm status
ERROR: configure.crm: No such command
crm(live)configure# order webip_before_webserver Mandatory: webip webserver
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
primitive webserver lsb:httpd
colocation webserver_with_webip inf: webserver webip
order webip_before_webserver Mandatory: webip webserver
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)configure# cd
crm(live)# status
Last updated: Mon Oct 26 18:58:09 2015
Last change: Mon Oct 26 18:57:56 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
webip (ocf::heartbeat:IPaddr): Started BAIYU_173
webserver (lsb:httpd): Started BAIYU_173
定義位置約束:
crm(live)configure# location webip_on_BAIYU_175 webip rule 50: #uname eq BA
IYU_175
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.100 nic=eth0 cidr_netmask=24
primitive webserver lsb:httpd
location webip_on_BAIYU_175 webip \
rule 50: #uname eq BAIYU_175
colocation webserver_with_webip inf: webserver webip
order webip_before_webserver Mandatory: webip webserver
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore
crm(live)configure# verify
crm(live)configure# commit
crm(live)# status
Last updated: Mon Oct 26 19:27:28 2015
Last change: Mon Oct 26 19:27:10 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
webip (ocf::heartbeat:IPaddr): Started BAIYU_175
webserver (lsb:httpd): Started BAIYU_175
定義資源對(duì)當(dāng)前節(jié)點(diǎn)的粘性：
crm(live)configure# property default-resource-stickiness=50
crm(live)configure# verify
crm(live)configure# commit
注意： 即使當(dāng)資源在BAIYU_173時(shí)，當(dāng)前有2個(gè)資源，粘性：50+50>location:50 資源也不會(huì)自動(dòng)轉(zhuǎn)到BAIYU_173
因?yàn)橄到y(tǒng)默認(rèn)只監(jiān)控節(jié)點(diǎn)，當(dāng)資源被意外關(guān)閉時(shí)，系統(tǒng)不知道，不會(huì)做資源轉(zhuǎn)移，所以我們要配置系統(tǒng)監(jiān)控資源：
先把之前定義的資源刪除：
crm(live)# cd ../resource
crm(live)resource# stop webip
crm(live)resource# stop webserver
crm(live)resource# cd ../configure
crm(live)configure# edit
crm(live)configure# edit #只留定義的集群全局屬性
1 node BAIYU_173 \
2 attributes standby=off
3 node BAIYU_175 \
4 attributes standby=off
5 property cib-bootstrap-options: \
6 dc-version=1.1.11-97629de \
7 cluster-infrastructure="classic openais (with plugin)" \
8 expected-quorum-votes=2 \
9 stonith-enabled=false \
10 no-quorum-policy=ignore \
11 default-resource-stickiness=50
12 #vim:set syntax=pcmk
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore \
default-resource-stickiness=50
重新定義加入監(jiān)控選項(xiàng)：
crm(live)configure# primitive webip ocf:heartbeat:IPaddr params ip=192.168.
100.110 nic=eth0 cidr_netmask=24 op monitor interval=10s timeout=20s # op表示加入動(dòng)作，monitor表示監(jiān)控 interval 間隔時(shí)間，timeout 超時(shí)時(shí)間
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# primitive webserver lsb:httpd op monitor interval=10s t
imout=20s
crm(live)configure# group webservice webip webserver
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.110 nic=eth0 cidr_netmask=24 \
op monitor interval=10s timeout=20s
primitive webserver lsb:httpd \
op monitor interval=10s timout=20s
group webservice webip webserver
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
stonith-enabled=false \
no-quorum-policy=ignore \
default-resource-stickiness=50
[root@BAIYU_173 ~]# crm
crm(live)# status
Last updated: Mon Oct 26 20:04:41 2015
Last change: Mon Oct 26 20:02:24 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
2 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Started BAIYU_173
webserver (lsb:httpd): Started BAIYU_173
crm(live)#
驗(yàn)證：
[root@BAIYU_173 ~]# netstat -nlpt|grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 7186/httpd
[root@BAIYU_173 ~]# killall httpd
[root@BAIYU_173 ~]# netstat -nlpt|grep 80
[root@BAIYU_173 ~]# netstat -nlpt|grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 8906/httpd
注意：如果資源啟動(dòng)不了，資源會(huì)轉(zhuǎn)移到其它節(jié)點(diǎn)
添加NFS:
[root@BAIYU_173 ~]# crm
crm(live)# resource
crm(live)resource# stop webservice
crm(live)resource# show
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Stopped
webserver (lsb:httpd): Stopped
crm(live)# ra info ocf:heartbeat:Filesystem #查看RA中對(duì)該類型資源的一些默認(rèn)設(shè)置和必須要帶參數(shù)
crm(live)configure# primitive webstore ocf:heartbeat:Filesystem params devi
ce="192.168.100.10:/data/myweb" directory="/var/www/html" fstype="nfs" op m
onitor interval=20s timeout=40s op start timeout=60s op stop timeout=60s
crm(live)configure# edit 手動(dòng)把webstore添加進(jìn)之前定義的組
crm(live)configure# verify
crm(live)configure# commit
crm(live)configure# show
node BAIYU_173 \
attributes standby=off
node BAIYU_175 \
attributes standby=off
primitive webip IPaddr \
params ip=192.168.100.110 nic=eth0 cidr_netmask=24 \
op monitor interval=10s timeout=20s
primitive webserver lsb:httpd \
op monitor interval=10s timout=20s
primitive webstore Filesystem \
params device="192.168.100.10:/data/myweb" directory="/var/www/html" fstype=nfs \
op monitor interval=20s timeout=40s \
op start timeout=60s interval=0 \
op stop timeout=60s interval=0
group webservice webip webserver webstore \
meta target-role=Stopped
property cib-bootstrap-options: \
dc-version=1.1.11-97629de \
cluster-infrastructure="classic openais (with plugin)" \
expected-quorum-votes=2 \
crm(live)# resource start webservice
crm(live)# resource show
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Started
webserver (lsb:httpd): Started
webstore (ocf::heartbeat:Filesystem): Started
crm(live)# status
Last updated: Mon Oct 26 20:42:32 2015
Last change: Mon Oct 26 20:42:00 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
3 Resources configured
Online: [ BAIYU_173 BAIYU_175 ]
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Started BAIYU_173
webserver (lsb:httpd): Started BAIYU_173
webstore (ocf::heartbeat:Filesystem): Started BAIYU_173
Failed actions:
webserver_monitor_10000 on BAIYU_173 'not running' (7): call=40, status=complete, last-rc-change='Mon Oct 26 20:07:46 2015', queued=0ms, exec=0ms
清理之前的狀態(tài)報(bào)錯(cuò)信息：
crm(live)# resource cleanup webservice #清空相關(guān)資源狀態(tài)的報(bào)錯(cuò)信息
Cleaning up webip on BAIYU_173
Cleaning up webip on BAIYU_175
Cleaning up webserver on BAIYU_173
Cleaning up webserver on BAIYU_175
Cleaning up webstore on BAIYU_173
Cleaning up webstore on BAIYU_175
Waiting for 6 replies from the CRMd...... OK
將BAIYU_173備用，資源轉(zhuǎn)移到_175
crm(live)# node standby
crm(live)# status
Last updated: Mon Oct 26 20:53:39 2015
Last change: Mon Oct 26 20:53:35 2015
Stack: classic openais (with plugin)
Current DC: BAIYU_175 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured, 2 expected votes
3 Resources configured
Node BAIYU_173: standby
Online: [ BAIYU_175 ]
Resource Group: webservice
webip (ocf::heartbeat:IPaddr): Started BAIYU_175
webserver (lsb:httpd): Started BAIYU_175
webstore (ocf::heartbeat:Filesystem): Started BAIYU_175
此時(shí)完整的HA httpd配置完成。
轉(zhuǎn)載于:https://blog.51cto.com/hobowizard/1884305
創(chuàng)作挑戰(zhàn)賽新人創(chuàng)作獎(jiǎng)勵(lì)來(lái)咯，堅(jiān)持創(chuàng)作打卡瓜分現(xiàn)金大獎(jiǎng)總結(jié)
以上是生活随笔為你收集整理的HA集群之三：corosync+pacemaker实现httpd服务的高可用的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。
- 上一篇： Laravel 在Centos上的安装
- 下一篇： Thinking In Java 读书笔