CNDBA社区
1、冗余公有IP网卡口
1.1编辑ifcfg-bond0文件
编辑添加ifcfg-bond0文件，绑定第一块、第二块网卡（公有IP）
[root@R930-2 ~]# vi/etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
IPADDR=172.41.189.3 ----**给定的IP地址**
NETMASK=255.255.255.192 ---**子网掩码**
GATEWAY=172.41.189.1 ----**网关**
USERCTL=no
TYPE=Bonding
1.2编辑第一块网卡
[root@R930-2 ~]#vi /etc/sysconfig/network-scripts/ifcfg-em1
DEVICE=em1
HWADDR=D0:94:66:02:20:84
TYPE=Ethernet
UUID=16a24c9f-8c21-4f8b-94c3-eafbec6a85c1
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
----添加如下两句
SLAVE=yes
MASTER=bond0
1.3编辑第二块网卡
[root@R930-2 ~]#cat /etc/sysconfig/network-scripts/ifcfg-em2
DEVICE=em2
HWADDR=D0:94:66:02:20:84
TYPE=Ethernet
UUID=16a24c9f-8c21-4f8b-94c3-eafbec6a85c1
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
----添加如下两句
SLAVE=yes
MASTER=bond0
1.4编辑添加bonding.conf文件
[root@R930-2 ~]#vi /etc/modprobe.d/bonding.conf
---添加如下
alias bond0 bonding miimon=100mode=1 updelay=5000
[root@R930-2~]#modprobe bonding
1.5关闭networkmanager服务
[root@R930-2 ~]#serviceNetworkManager stop
[root@R930-2 ~]#chkconfigNetworkManager off
1.6重启网络服务
[root@R930-2 ~]#servicenetwork restart
2、冗余私有IP及网卡口
绑定第三块、第四块网卡（冗余私有IP）
2.1编辑第三块网卡
[root@R930-2 ~]#vi /etc/sysconfig/network-scripts/ifcfg-em3
DEVICEE=em3
HWADDR=D0:94:66:01:CA:A6
TYPE=Ethernet
UUID=aa6315d8-e09c-4b63-a2ce-73054e5652bf
---**不需要写网关gateway**
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
---**私有IP，心跳IP可自行定**
IPADDR=192.168.1.4
NETMASK=255.255.255.0
2.2编辑第四块网卡（为更好的冗余心跳，将两个互联私有IP设置不同网段）
[root@R930-2 ~]#vi /etc/sysconfig/network-scripts/ifcfg-em4
DEVICEE=em4
HWADDR=D0:94:66:01:CA:A6
TYPE=Ethernet
UUID=aa6315d8-e09c-4b63-a2ce-73054e5652bf
---**不需要写网关gateway**
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
---**私有IP，心跳IP可自行定**
IPADDR=192.168.2.4
NETMASK=255.255.255.0
2.3重启网络服务
ifup 网卡名（比如em3,em4） ----单独启动某个网卡
---假若由于误操作，将网卡配置文件中的HWaddr地址覆盖或重写并保存了，导致无法启动网卡，并报如下错误
[root@R930-2 ~]#ifup em3
Device em3 has differentMAC address than expected, ignoring.
--可通过如下命令找到实际的HWaddr地址
[root@R930-2~]# ifconfig em3 (ifconfig 网卡名)
em3 Link encap:Ethernet HWaddr D0:94:66:02:20:88
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:349 errors:0 dropped:0overruns:0 frame:0
TX packets:3 errors:0 dropped:0overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:22336 (21.8 KiB) TX bytes:270 (270.0 b)
Interrupt:36 Memory:93000000-937fffff
---重新编辑修改网卡中的HWaddr地址，将其修改为上述红色字体标注部分，保存退出，重新启动网卡
如下：
[root@R930-2 ~]#ifup em3
Determining if ip address192.168.1.4 is already in use for device em3...
2.4检查是否都配置好IP
[root@R930-2 ~]# ifconfig
bond0 Link encap:Ethernet HWaddr D0:94:66:02:20:84
inet addr:172.41.189.3 Bcast:172.41.189.63 Mask:255.255.255.192
inet6 addr:fe80::d294:66ff:fe02:2084/64 Scope:Link
UP BROADCAST RUNNING MASTERMULTICAST MTU:1500 Metric:1
RX packets:3484 errors:0 dropped:0overruns:0 frame:0
TX packets:1603 errors:0 dropped:0overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:633493 (618.6 KiB) TX bytes:257749 (251.7 KiB)
em1 Link encap:Ethernet HWaddr D0:94:66:02:20:84
UP BROADCAST RUNNING SLAVEMULTICAST MTU:1500 Metric:1
RX packets:856 errors:0 dropped:0overruns:0 frame:0
TX packets:802 errors:0 dropped:0overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:230353 (224.9 KiB) TX bytes:134397 (131.2 KiB)
Interrupt:32 Memory:95000000-957fffff
em2 Link encap:Ethernet HWaddr D0:94:66:02:20:84
UP BROADCAST RUNNING SLAVEMULTICAST MTU:1500 Metric:1
RX packets:2628 errors:0 dropped:0overruns:0 frame:0
TX packets:801 errors:0 dropped:0overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:403140 (393.6 KiB) TX bytes:123352 (120.4 KiB)
Interrupt:36 Memory:94000000-947fffff
em3 Link encap:Ethernet HWaddr D0:94:66:02:20:88
inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr:fe80::d294:66ff:fe02:2088/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0overruns:0 frame:0
TX packets:10 errors:0 dropped:0overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:676 (676.0 b)
Interrupt:36 Memory:93000000-937fffff
em4 Link encap:Ethernet HWaddr D0:94:66:02:20:8A
inet addr:192.168.1.5 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr:fe80::d294:66ff:fe02:208a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0overruns:0 frame:0
TX packets:8 errors:0 dropped:0overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:528 (528.0 b)
Interrupt:37 Memory:92000000-927fffff
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:252 errors:0 dropped:0overruns:0 frame:0
TX packets:252 errors:0 dropped:0overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:17300 (16.8 KiB) TX bytes:17300 (16.8 KiB)
---最后在RAC节点相互ping私有IP是否联通（输出如下才行）
[root@R930-2 ~]#ping 192.168.1.2
PING 192.168.1.2(192.168.1.2) 56(84) bytes of data.
64 bytes from192.168.1.2: icmp_seq=1 ttl=64 time=1.32 ms
64 bytes from192.168.1.2: icmp_seq=2 ttl=64 time=0.133 ms
^C
3、检查存储
3.1安装multipath包
（安装顺序先Libs再后面的rpm包）
device-mapper-multipath-libs-0.4.9-80.el6.x86_64.rpm
device-mapper-multipath-0.4.9-80.el6.x86_64.rpm
[root@R930-3 ~]#rpm -ivh device-mapper-multipath-libs-0.4.9-80.el6.x86_64.rpm
warning:device-mapper-multipath-libs-0.4.9-80.el6.x86_64.rpm: Header V3 RSA/SHA256Signature, key ID fd431d51: NOKEY
Preparing... ########################################### [100%]
1:device-mapper-multipath###########################################[100%]
[root@R930-3 ~]#rpm -ivh device-mapper-multipath-0.4.9-80.el6.x86_64.rpm
warning:device-mapper-multipath-0.4.9-80.el6.x86_64.rpm: Header V3 RSA/SHA256Signature, key ID fd431d51: NOKEY
Preparing... ########################################### [100%]
1:device-mapper-multipath###########################################[100%]
[root@R930-3 ~]# multipath -ll
Aug 21 17:29:54 |/etc/multipath.conf does not exist, blacklisting all devices.
Aug 21 17:29:54 | A samplemultipath.conf file is located at
Aug 21 17:29:54 |/usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Aug 21 17:29:54 | You can run/sbin/mpathconf to create or modify /etc/multipath.conf
上述报错缺乏下述的文件
3.2编辑添加multipath.conf文件
#查看磁盘的wwid号
/lib/udev/scsi_id--whitelist /dev/盘符
----添加如下文件(磁盘逻辑卷别名配置)
[root@R930-4~]# vi /etc/multipath.conf
multipaths{
multipath {
wwid 36000d3100146c0000000000000000003
如下颜色就是磁盘逻辑卷的别名，可自行命名，当前命名规则为存储名+**逻辑卷号VOL**
alias SC9000-1-VOL1
path_grouping_policy multibus ---组件策略 不用变，照写就可以
}
multipath {
wwid 36000d3100146c0000000000000000004
alias SC9000-1-VOL2
path_grouping_policy multibus
}
multipath {
wwid 36000d3100146c0000000000000000005
alias SC9000-1-VOL3
path_grouping_policy multibus
}
multipath {
wwid 36000d3100146c0000000000000000006
alias SC9000-1-VOL4
path_grouping_policy multibus
}
multipath {
wwid 36000d310014772000000000000000003
alias SC9000-2-VOL1
path_grouping_policy multibus
}
multipath {
wwid 36000d310014772000000000000000004
alias SC9000-2-VOL2
path_grouping_policy multibus
}
multipath {
wwid 36000d310014772000000000000000005
alias SC9000-2-VOL3
path_grouping_policy multibus
}
multipath {
wwid 36000d310014772000000000000000006
alias SC9000-2-VOL4
path_grouping_policy multibus
}
}
所有的搭建的RAC节点都需要这个同样的配置文件
再执行如下操作
[root@R930-3~]# modprobe dm-multipath
[root@R930-3~]# modprobe dm-round-robin
[root@R930-3~]# service multipathd restart
ok
Stoppingmultipathd daemon: [ OK ]
Startingmultipathd daemon: [ OK ]
3.3 验证存储
[root@R930-3 ~]# multipath –ll
------验证存储是否都已加上-出现如下代码则说明存储正常
SC9000-2-VOL1(36000d310014772000000000000000003) dm-8 COMPELNT,Compellent Vol
size=380Gfeatures='1 queue_if_no_path' hwhandler='0' wp=rw
`-+-policy='round-robin 0' prio=1 status=active
|- 1:0:3:1 sdq 65:0 active ready running
|- 2:0:5:1 sdah 66:16 active ready running
|- 0:0:5:1 sdal 66:80 active ready running
|- 3:0:4:1 sdax 67:16 active ready running
|- 1:0:9:1 sdw 65:96 active ready running
|- 2:0:4:1 sdaq 66:160 active ready running
|- 0:0:4:1 sdap 66:144 active ready running
`- 3:0:6:1 sdaz 67:48 active ready running
SC9000-1-VOL4(36000d3100146c0000000000000000006) dm-3 COMPELNT,Compellent Vol
size=380Gfeatures='1 queue_if_no_path' hwhandler='0' wp=rw
`-+-policy='round-robin 0' prio=1 status=active
|- 2:0:12:4 sdc 8:32 active ready running
|- 0:0:10:4 sdd 8:48 active ready running
|- 3:0:11:4 sdbg 67:160 active ready running
|- 1:0:13:4 sdab 65:176 active ready running
|- 2:0:13:4 sdl 8:176 active ready running
|- 0:0:13:4 sdk 8:160 active ready running
|- 3:0:12:4 sdbi 67:192 active ready running
`- 1:0:14:4 sdad 65:208 active ready running
SC9000-1-VOL3(36000d3100146c0000000000000000005) dm-4 COMPELNT,Compellent Vol
size=380Gfeatures='1 queue_if_no_path' hwhandler='0' wp=rw
`-+-policy='round-robin 0' prio=1 status=active
|- 2:0:14:3 sdg 8:96 active ready running
|- 0:0:14:3 sdh 8:112 active ready running
|- 3:0:13:3 sdbk 67:224 active ready running
|- 1:0:12:3 sdz 65:144 active ready running
|- 2:0:15:3 sdp 8:240 active ready running
|- 0:0:15:3 sdn 8:208 active ready running
|- 3:0:14:3 sdbm 68:0 active ready running
`- 1:0:15:3 sdaf 65:240 active ready running
SC9000-2-VOL4 (36000d310014772000000000000000006)dm-10 COMPELNT,Compellent Vol
size=380Gfeatures='1 queue_if_no_path' hwhandler='0' wp=rw
`-+-policy='round-robin 0' prio=1 status=active
|- 1:0:6:4 sdt 65:48 active ready running
|- 2:0:8:4 sdao 66:128 active ready running
|- 0:0:9:4 sdak 66:64 active ready running
|- 3:0:7:4 sdbc 67:96 active ready running
|- 1:0:8:4 sdv 65:80 active ready running
|- 2:0:9:4 sdav 66:240 active ready running
|- 0:0:8:4 sdaw 67:0 active ready running
`- 3:0:8:4 sdbe 67:128 active ready running
SC9000-1-VOL2(36000d3100146c0000000000000000004) dm-6 COMPELNT,Compellent Vol
size=380Gfeatures='1 queue_if_no_path' hwhandler='0' wp=rw
`-+-policy='round-robin 0' prio=1 status=active
|- 0:0:10:2 sdb 8:16 active ready running
|- 2:0:12:2 sda 8:0 active ready running
|- 1:0:13:2 sdaa 65:160 active ready running
|- 3:0:11:2 sdbf 67:144 active ready running
|- 0:0:13:2 sdi 8:128 active ready running
|- 2:0:13:2 sdj 8:144 active ready running
|- 1:0:14:2 sdac 65:192 active ready running
`- 3:0:12:2 sdbh 67:176 active ready running
SC9000-2-VOL3(36000d310014772000000000000000005) dm-7 COMPELNT,Compellent Vol
size=380Gfeatures='1 queue_if_no_path' hwhandler='0' wp=rw
`-+-policy='round-robin 0' prio=1 status=active
|- 1:0:3:3 sdr 65:16 active ready running
|- 2:0:5:3 sdaj 66:48 active ready running
|- 0:0:5:3 sdan 66:112 active ready running
|- 3:0:6:3 sdba 67:64 active ready running
|- 1:0:9:3 sdx 65:112 active ready running
|- 2:0:4:3 sdas 66:192 active ready running
|- 0:0:4:3 sdar 66:176 active ready running
`- 3:0:4:3 sday 67:32 active ready running
SC9000-1-VOL1(36000d3100146c0000000000000000003) dm-5 COMPELNT,Compellent Vol
size=380Gfeatures='1 queue_if_no_path' hwhandler='0' wp=rw
`-+-policy='round-robin 0' prio=1 status=active
|- 2:0:14:1 sde 8:64 active ready running
|- 0:0:14:1 sdf 8:80 active ready running
|- 3:0:13:1 sdbj 67:208 active ready running
|- 1:0:12:1 sdy 65:128 active ready running
|- 2:0:15:1 sdo 8:224 active ready running
|- 0:0:15:1 sdm 8:192 active ready running
|- 3:0:14:1 sdbl 67:240 active ready running
`- 1:0:15:1 sdae 65:224 active ready running
SC9000-2-VOL2 (36000d310014772000000000000000004)dm-9 COMPELNT,Compellent Vol
size=380Gfeatures='1 queue_if_no_path' hwhandler='0' wp=rw
`-+-policy='round-robin 0' prio=1 status=active
|- 1:0:6:2 sds 65:32 active ready running
|- 2:0:8:2 sdam 66:96 active ready running
|- 0:0:9:2 sdai 66:32 active ready running
|- 3:0:7:2 sdbb 67:80 active ready running
|- 1:0:8:2 sdu 65:64 active ready running
|- 2:0:9:2 sdau 66:224 active ready running
|- 0:0:8:2 sdat 66:208 active ready running
`- 3:0:8:2 sdbd 67:112 active ready running
划分共享存储udev规则：
Hba是指多路径配置好后mapper/下面dm对应得设备名，两个节点得mapper下对应块设备dm名会变动，即hba为mapper/下面的设备名
ENV{DM_NAME}=="hba1",OWNER:="grid",GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="hba2",OWNER:="grid",GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="hba*",OWNER:="grid",GROUP:="asmadmin", MODE:="660"
版权声明：本文为博主原创文章，未经博主允许不得转载。
