CNDBA社区
ORACLE 12CR2 传统标准Flex集群RAC增加Leaf操作
当前ORACLE 12CR2 RAC环境使用的是host文件解析，然后我们从安装配置Leaf节点可知：使用Leaf节点需要配置GNS，否则无法使用Leaf功能….故，当前环境需增加GNS并修改Host文件，参考文章链接：www.cndba.cn/Marvinn/article/2766
总结：1、12CR2 RAC Leaf 节点也需要链接存储，否则添加Leaf节点执行root.sh将会无法启动GI资源....我认为可能是无法注册启动GI必需要进程
- 相比hub node , leaf node对于集群来说不是那么核心，与集群耦合度很低，而且leaf node之间不需要互联
- 每个leaf node通过一个hub node连接集群并获取数据。
- 虽然leaf node不要求直接访问共享存储，但最好还是连上共享存储，因为说不准未来哪天就要把这个leaf node转为hub node使用。
(这里译者补充下：虽然leaf node启动集群件是不依赖共享存储的，但当leaf node以只读打开了某数据库时，也就是这个leaf node上运行着数据库实例时，这个leaf node就变成了reader node，官方明确说了reader node是必须连接共享存储的。官方连接：http://docs.oracle.com/database/122/RILIN/running-database-instances-on-leaf-nodes.htm#RILIN-GUID-35BDCD41-375D-4EB2-B0FC-49345AF6C572)
2、12CR2 存在一个BUG，执行root.sh脚本会因为gpnp无法启动，造成执行失败(mos文章:Bug 24411578 - root.sh Execution Fails Starting GPNPD on Non-First Nodes (文档 ID 24411578.8)),需要打个系统补丁(补丁号：27468969)，注意所有节点，都需要打上该补丁，才能执行成功root.sh脚本
3、使用Leaf 节点需要注意的点：
3.1、RAC需要使用GNS，需要搭建DNS,DHCP
3.2、RAC，DBCA建库不能选择使用用admin-managed管理数据库，因为不能扩展admin-managed 到Leaf 节点，也就是说创建数据库需要使用Policy-managed 管理数据库,而非默认的admin-managed管理数据库，好像现在12C默认是Policy-managed
3.3、Leaf可以不需要VIP，但是变只读实例或者转换为Hub时，需要，若是DHCP自动分配，可不用手动添加，会自动分配，否则需手动
3.4、Leaf启动DB只读实例时，必须直连存储，但是不访问，还是通过Hub访问（也就是说增加叶子节点还是直连存储吧）
3.5、Leaf节点ASM实例是不能运行的
一、环境配置
环境 配置
数据库版本 12.2.0.1
系统版本 centos 7
物理内存 8G
CPU 2颗 逻辑4核
二、前期准备：
修改 RAC Hub 所有 节点IP hosts文件配置，增加Leaf1节点IP以及私有IP，
# Public Network - (eth0)
172.16.10.116 rac12chub1.marvin.cn rac12chub1
172.16.10.142 rac12chub2.marvin.cn rac12chub2
172.16.10.146 rac12cleaf1.marvin.cn rac12cleaf1
# Private Interconnect - (eth1)
192.168.122.222 rac12chub1-priv.marvin.cn rac12chub1-priv
192.168.122.238 rac12chub2-priv.marvin.cn rac12chub2-priv
192.168.122.243 rac12cleaf1-priv.marvin.cn rac12cleaf1-priv
修改Leaf1节点的host 文件
[root@rac12cleaf1 ~]# vi /etc/hosts
# Public Network - (eth0)
172.16.10.116 rac12chub1.marvin.cn rac12chub1
172.16.10.142 rac12chub2.marvin.cn rac12chub2
172.16.10.146 rac12cleaf1.marvin.cn rac12cleaf1
# Private Interconnect - (eth1)
192.168.122.222 rac12chub1-priv.marvin.cn rac12chub1-priv
192.168.122.238 rac12chub2-priv.marvin.cn rac12chub2-priv
192.168.122.243 rac12cleaf1-priv.marvin.cn rac12cleaf1-priv
修改网卡eth0配置,DNS1配置成DNS服务器IP
DNS1=172.16.10.138 // DNS服务器IP
停止NetworkManager服务
#systemctl stop NetworkManager
#systemctl disable NetworkManager
修改resolv.conf配置文件
# vi /etc/resolv.conf
添加如下
# Generated by NetworkManager
nameserver 172.16.10.138 // DNS服务器
nameserver 172.16.10.139 // GNS VIP
search marvin.cn
options rotate
options timeout: 2
options attempts: 5
重启网络
#service netork restart
注意：需要注意的是DHCP 服务器网段IP范围，必须足够广....否则添加节点可能会因为分配不到VIP或SCAN IP而报错 又或者超出DHCP服务器指定IP范围，造成不可控局面
三、环境准备：
1、Leaf节点环境配置，从Hub节点1上传送环境配置脚本
[root@rac12chub1 scripts]# scp * 172.16.10.146:/home/scripts/
The authenticity of host '172.16.10.146 (172.16.10.146)' can't be established.
ECDSA key fingerprint is d4:c4:a2:9b:97:e6:2e:19:8d:23:61:7c:76:15:4e:3c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.10.146' (ECDSA) to the list of known hosts.
root@172.16.10.146's password:
check.text 100% 0 0.0KB/s 00:00
grid_env.sh 100% 2439 2.4KB/s 00:00
oracle_env.sh 100% 3616 3.5KB/s 00:00
oracle_OS_12C.sh 100% 6551 6.4KB/s 00:00
rpm_package.txt 100% 742 0.7KB/s 00:00
2、leaf节点1运行配置脚本
[root@rac12cleaf1 scripts]# sh oracle_OS_12C.sh
3、修改ORACLE用户ORACLE_SID
oracle@rac12cleaf1:/home/oracle>vi ~/.bash_profile
ORACLE_SID=marvin3
oracle@rac12cleaf1:/home/oracle>source ~/.bash_profile
4、修改GRID用户ORACLE_SID
grid@rac12cleaf1:/home/grid>vi ~/.bash_profile
ORACLE_SID=+ASM3
grid@rac12cleaf1:/home/grid>source ~/.bash_profile
5、leaf与hub节点互信(grid用户以及oracle用户互信，以下只展示GRID用户，oracle用户操作步骤同样，只是用户名不同而已)
Leaf节点：
grid@rac12cleaf1:/home/grid>ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa): Created directory '/home/grid/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
5e:b3:1e:a0:39:cd:8f:45:44:10:6e:e8:40:c0:20:fa grid@rac12cleaf1
The key's randomart image is:
+--[ RSA 2048]----+
|oo... oo. |
|o .. o . |
|. . . o . |
| . o . . |
| E . S + |
| * + o |
| + + + |
| . = . |
| . o |
+-----------------+
grid@rac12cleaf1:/home/grid>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
增加所有hub节点的公钥
grid@rac12cleaf1:/home/grid>ssh rac12chub1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12cleaf1:/home/grid>ssh rac12chub1-priv cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12cleaf1:/home/grid>ssh rac12chub1-priv.marvin.cn cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12cleaf1:/home/grid>ssh rac12chub1.marvin.cn cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12cleaf1:/home/grid>ssh rac12chub2.marvin.cn cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12cleaf1:/home/grid>ssh rac12chub2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12cleaf1:/home/grid>ssh rac12chub2-priv cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12cleaf1:/home/grid>ssh rac12chub2-priv.marvin.cn cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Hub节点：
grid@rac12chub1:/g01/app/grid/12.2.0>ssh rac12cleaf1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12chub1:/home/grid>ssh rac12cleaf1.marvin.cn cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12chub1:/home/grid>ssh rac12cleaf1-priv cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12chub1:/home/grid>ssh rac12cleaf1-priv.marvin.cn cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12chub2:/home/grid>ssh rac12cleaf1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12chub2:/home/grid>ssh rac12cleaf1.marvin.cn cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12chub1:/home/grid>ssh rac12cleaf1-priv cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
grid@rac12chub1:/home/grid>ssh rac12cleaf1-priv.marvin.cn cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ORACLE 用户互信步骤省略。。。操作见上
6、验证互信
GRID用户
Hub节点:包括原始节点(这是因为我RAC架构已经发生变化，采用的GNS解析SCAN方式),否则执行添加节点会报错：
[FATAL] [INS-06006] Passwordless SSH connectivity not set up between the following node(s): [rac12cleaf1].
CAUSE: Either passwordless SSH connectivity is not setup between specified node(s) or they are not reachable. Refer to the logs for more details.
ACTION: Refer to the logs for more details or contact Oracle Support Services.
grid@rac12chub1:/g01/app/grid/12.2.0>
ssh rac12chub1 date
ssh rac12chub1.marvin.cn date
ssh rac12chub1-priv date
ssh rac12chub1-priv.marvin.cn date
ssh rac12chub2 date
ssh rac12chub2.marvin.cn date
ssh rac12chub2-priv date
ssh rac12chub2-priv.marvin.cn date
ssh rac12cleaf1 date
ssh rac12cleaf1-priv date
ssh rac12cleaf1.marvin.cn date
ssh rac12cleaf1-priv.marvin.cn date
报错:报错忽略...只是因为之前HOSTS解析架构采用GI自动配置主机SSH互信，但仍然需要添加对应主机域名在内
grid@rac12chub1:/home/grid>ssh rac12chub1.marvin.cn date
Warning: the ECDSA host key for 'rac12chub1.marvin.cn' differs from the key for the IP address '172.16.10.116'
Offending key for IP in /home/grid/.ssh/known_hosts:1
Matching host key in /home/grid/.ssh/known_hosts:18
Are you sure you want to continue connecting (yes/no)? yes
Mon May 7 12:17:51 CST 2018
grid@rac12chub1:/home/grid>ssh rac12chub2.marvin.cn date
The authenticity of host 'rac12chub2.marvin.cn (172.16.10.142)' can't be established.
ECDSA key fingerprint is cf:9e:6d:c3:e0:9f:79:23:0c:7c:20:be:70:67:fe:ae.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac12chub2.marvin.cn' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for 'rac12chub2.marvin.cn' differs from the key for the IP address '172.16.10.142'
Offending key for IP in /home/grid/.ssh/known_hosts:5
Are you sure you want to continue connecting (yes/no)? yes
Mon May 7 12:19:40 CST 2018
grid@rac12chub2:/home/grid> ssh rac12chub1 date
ssh rac12chub1.marvin.cn date
ssh rac12chub1-priv date
ssh rac12chub1-priv.marvin.cn date
ssh rac12chub2 date
ssh rac12chub2.marvin.cn date
ssh rac12chub2-priv date
ssh rac12chub2-priv.marvin.cn date
ssh rac12cleaf1 date
ssh rac12cleaf1-priv date
ssh rac12cleaf1.marvin.cn date
ssh rac12cleaf1-priv.marvin.cn date
报错：报错忽略...只是因为之前HOSTS解析架构采用GI自动配置主机SSH互信，但仍然需要添加对应主机域名在内
grid@rac12chub2:/home/grid>ssh rac12chub1.marvin.cn date
The authenticity of host 'rac12chub1.marvin.cn (172.16.10.116)' can't be established.
ECDSA key fingerprint is 79:99:23:37:57:c0:60:d7:f2:e7:d4:b9:14:df:d6:f8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac12chub1.marvin.cn' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for 'rac12chub1.marvin.cn' differs from the key for the IP address '172.16.10.116'
Offending key for IP in /home/grid/.ssh/known_hosts:1
Are you sure you want to continue connecting (yes/no)? yes
Mon May 7 12:21:37 CST 2018
grid@rac12chub2:/home/grid>ssh rac12chub2.marvin.cn date
The authenticity of host 'rac12chub2.marvin.cn (172.16.10.142)' can't be established.
ECDSA key fingerprint is cf:9e:6d:c3:e0:9f:79:23:0c:7c:20:be:70:67:fe:ae.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac12chub2.marvin.cn' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for 'rac12chub2.marvin.cn' differs from the key for the IP address '172.16.10.142'
Offending key for IP in /home/grid/.ssh/known_hosts:5
Are you sure you want to continue connecting (yes/no)? yes
Mon May 7 12:22:25 CST 2018
Leaf节点：
grid@rac12cleaf1:/home/grid> ssh rac12chub1 date
ssh rac12chub1.marvin.cn date
ssh rac12chub1-priv date
ssh rac12chub1-priv.marvin.cn date
ssh rac12chub2 date
ssh rac12chub2.marvin.cn date
ssh rac12chub2-priv date
ssh rac12chub2-priv.marvin.cn date
ssh rac12cleaf1 date
ssh rac12cleaf1-priv date
ssh rac12cleaf1.marvin.cn date
ssh rac12cleaf1-priv.marvin.cn date
ORACLE用户
Hub节点:包括原始节点(这是因为我RAC架构已经发生变化，采用的GNS解析SCAN方式),否则执行添加节点会报错：
[FATAL] [INS-06006] Passwordless SSH connectivity not set up between the following node(s): [rac12cleaf1].
CAUSE: Either passwordless SSH connectivity is not setup between specified node(s) or they are not reachable. Refer to the logs for more details.
ACTION: Refer to the logs for more details or contact Oracle Support Services.
oracle@rac12chub1:/home/oracle> ssh rac12chub1 date
ssh rac12chub1.marvin.cn date
ssh rac12chub1-priv date
ssh rac12chub1-priv.marvin.cn date
ssh rac12chub2 date
ssh rac12chub2.marvin.cn date
ssh rac12chub2-priv date
ssh rac12chub2-priv.marvin.cn date
ssh rac12cleaf1 date
ssh rac12cleaf1-priv date
ssh rac12cleaf1.marvin.cn date
ssh rac12cleaf1-priv.marvin.cn date
报错：报错忽略...只是因为之前HOSTS解析架构采用GI自动配置主机SSH互信，但仍然需要添加对应主机域名在内
oracle@rac12chub1:/home/oracle>ssh rac12chub1.marvin.cn date
The authenticity of host 'rac12chub1.marvin.cn (172.16.10.116)' can't be established.
ECDSA key fingerprint is 79:99:23:37:57:c0:60:d7:f2:e7:d4:b9:14:df:d6:f8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac12chub1.marvin.cn' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for 'rac12chub1.marvin.cn' differs from the key for the IP address '172.16.10.116'
Offending key for IP in /home/oracle/.ssh/known_hosts:1
Are you sure you want to continue connecting (yes/no)? yes
Mon May 7 12:26:57 CST 2018
oracle@rac12chub1:/home/oracle>ssh rac12chub2.marvin.cn date
The authenticity of host 'rac12chub2.marvin.cn (172.16.10.142)' can't be established.
ECDSA key fingerprint is cf:9e:6d:c3:e0:9f:79:23:0c:7c:20:be:70:67:fe:ae.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac12chub2.marvin.cn' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for 'rac12chub2.marvin.cn' differs from the key for the IP address '172.16.10.142'
Offending key for IP in /home/oracle/.ssh/known_hosts:5
Are you sure you want to continue connecting (yes/no)? yes
Mon May 7 12:27:54 CST 2018
oracle@rac12chub2:/home/oracle> ssh rac12chub1 date
ssh rac12chub1.marvin.cn date
ssh rac12chub1-priv date
ssh rac12chub1-priv.marvin.cn date
ssh rac12chub2 date
ssh rac12chub2.marvin.cn date
ssh rac12chub2-priv date
ssh rac12chub2-priv.marvin.cn date
ssh rac12cleaf1 date
ssh rac12cleaf1-priv date
ssh rac12cleaf1.marvin.cn date
ssh rac12cleaf1-priv.marvin.cn date
报错：报错忽略...只是因为之前HOSTS解析架构采用GI自动配置主机SSH互信，但仍然需要添加对应主机域名在内
oracle@rac12chub2:/home/oracle>ssh rac12chub1.marvin.cn date
Warning: the ECDSA host key for 'rac12chub1.marvin.cn' differs from the key for the IP address '172.16.10.116'
Offending key for IP in /home/oracle/.ssh/known_hosts:1
Matching host key in /home/oracle/.ssh/known_hosts:15
Are you sure you want to continue connecting (yes/no)? yes
Mon May 7 12:30:19 CST 2018
oracle@rac12chub2:/home/oracle>ssh rac12chub2.marvin.cn date
Warning: the ECDSA host key for 'rac12chub2.marvin.cn' differs from the key for the IP address '172.16.10.142'
Offending key for IP in /home/oracle/.ssh/known_hosts:5
Matching host key in /home/oracle/.ssh/known_hosts:18
Are you sure you want to continue connecting (yes/no)? yes
Mon May 7 12:31:29 CST 2018
Leaf节点：
oracle@rac12cleaf1:/home/oracle>ssh rac12chub1 date
ssh rac12chub1.marvin.cn date
ssh rac12chub1-priv date
ssh rac12chub1-priv.marvin.cn date
ssh rac12chub2 date
ssh rac12chub2.marvin.cn date
ssh rac12chub2-priv date
ssh rac12chub2-priv.marvin.cn date
ssh rac12cleaf1 date
ssh rac12cleaf1-priv date
ssh rac12cleaf1.marvin.cn date
ssh rac12cleaf1-priv.marvin.cn date
四、运行检查环境
1、Hub节点1：
grid@rac12chub1:/home/grid>cluvfy stage -pre nodeadd -n rac12cleaf1 -fixup -verbose
报错，需要去Leaf 节点以root 身份执行CVU Package包脚本....这个CVU脚本主要适用于访问ASM存储的
[root@rac12cleaf1 CVU_12.2.0.1.0_grid]# sh /tmp/CVU_12.2.0.1.0_grid/runfixup.sh
2、Hub节点1，再次检查运行环境：
grid@rac12chub1:/home/grid>cluvfy stage -pre nodeadd -n rac12cleaf1 -fixup -verbose
报错如下：
Failures were encountered during execution of CVU verification request "stage -pre nodeadd".
忽略.....
Verifying Swap Size ...FAILED
rac12cleaf1: PRVF-7573 : Sufficient swap size is not available on node
"rac12cleaf1" [Required = 7.6404GB (8011500.0KB) ; Found = 3.875GB
(4063228.0KB)]
rac12chub1: PRVF-7573 : Sufficient swap size is not available on node
"rac12chub1" [Required = 7.6393GB (8010420.0KB) ; Found = 2GB
(2097148.0KB)]
忽略.....
Verifying OS Kernel Version ...WARNING
PRVF-7524 : Kernel version is not consistent across all the nodes.
Kernel version = "3.10.0-693.21.1.el7.x86_64" found on nodes: rac12chub1.
Kernel version = "3.10.0-229.el7.x86_64" found on nodes: rac12cleaf1.
安装这两个包....
Verifying Package: nfs-utils-1.2.3-15 ...FAILED
rac12cleaf1: PRVF-7532 : Package "nfs-utils" is missing on node "rac12cleaf1"
Verifying Package: net-tools-2.0-0.17 ...FAILED
rac12cleaf1: PRVF-7532 : Package "net-tools" is missing on node "rac12cleaf1"
这里可以忽略，由于添加的是Leaf节点，所以可以不直连存储，但是也可以直连，方便以后Leaf叶子节点转换为Hub节点,但是添加Hub节点是必须直连存储(其实后续这里验证是被打脸....没直连的话 执行root.sh GI资源都启动不了，还是直连解决了)
Verifying Node Connectivity ...FAILED
Verifying Multicast check ...FAILED
PRVG-11081 : unable to find the matching networks for the interface "eth1" and
the subnet "192.168.122.0" on nodes "rac12cleaf1"
Verifying ASM Integrity ...FAILED
PRVG-11081 : unable to find the matching networks for the interface "eth1" and
the subnet "192.168.122.0" on nodes "rac12cleaf1"
Verifying Device Checks for ASM ...FAILED
Verifying ASM device sharedness check ...WARNING
Verifying Shared Storage
Accessibility:/dev/asm-diskd,/dev/asm-diske,/dev/asm-diskf,/dev/asm-diskg,/d
ev/asm-diska,/dev/asm-diskb,/dev/asm-diskc ...WARNING
PRVG-1615 : Virtual environment detected. Skipping shared storage check for
disks
"/dev/asm-diskd,/dev/asm-diske,/dev/asm-diskf,/dev/asm-diskg,/dev/asm-diska,
/dev/asm-diskb,/dev/asm-diskc".
Verifying Access Control List check ...FAILED
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskg" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskf" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diske" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskd" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskc" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskb" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diska" on node "rac12cleaf1"
忽略....
Verifying Network Time Protocol (NTP) ...FAILED
Verifying '/etc/chrony.conf' ...FAILED
rac12cleaf1: PRVG-1019 : The NTP configuration file "/etc/chrony.conf" does
not exist on nodes "rac12cleaf1"
忽略....
Verifying resolv.conf Integrity ...FAILED
rac12cleaf1: PRVG-10048 : Name "rac12cleaf1" was not resolved to an address of
the specified type by name servers o"172.16.10.254".
rac12cleaf1: Check for integrity of file "/etc/resolv.conf" failed
rac12chub1: PRVG-10048 : Name "rac12chub1" was not resolved to an address of
the specified type by name servers o"172.16.10.254".
rac12chub1: Check for integrity of file "/etc/resolv.conf" failed
忽略....
Verifying (Linux) resolv.conf Integrity ...FAILED
rac12cleaf1: PRVG-10048 : Name "rac12cleaf1" was not resolved to an address
of the specified type by name servers o"172.16.10.254".
rac12chub1: PRVG-10048 : Name "rac12chub1" was not resolved to an address of
the specified type by name servers o"172.16.10.254".
CVU operation performed: stage -pre nodeadd
Date: May 4, 2018 2:12:40 PM
CVU home: /g01/app/grid/12.2.0/
User: grid
3、修理Leaf1系统环境..
[root@rac12cleaf1 ~]# yum install nfs-utils
[root@rac12cleaf1 ~]# yum install net-tools
[root@rac12cleaf1 ~]# vi /etc/fstab
tmpfs /dev/shm tmpfs size=2048m 0 0
[root@rac12cleaf1 ~]# mount –o remount /dev/shm
Hub1节点开始添加节点
grid@rac12chub1:/home/grid>$ORACLE_HOME/addnode/addnode.sh -silent "CLUSTER_NEW_NODES={rac12cleaf1}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac12cleaf1-vip}" "CLUSTER_NEW_NODE_ROLES={leaf}"
[FATAL] [INS-43053] Leaf nodes cannot be added to this Oracle Cluster configuration.
CAUSE: Oracle Grid Infrastructure does not allow LEAF nodes to be configured as part of this Oracle Cluster configuration.
ACTION: Specify only HUB nodes to proceed.
添加节点报错：表明LEAF节点不允许当前的数据库环境....只允许HUb节点...这是因为i，要想使用Hub和Leaf节点新特性，就必须配置GNS以及DNS，否则将不能使用Leaf，这点我们从最开始安装部署，未启动GNS下，直接选Leaf角色直接报错，就可以得知....
Hub节点环境HOSTS解析转换为GNS解析，参考文章链接：http://www.cndba.cn/Marvinn/article/2766
五、若环境已经是GNS解析，即可以直接忽视。。。直接添加
1、将需要加进来的rac12cleaf1 节点和Hub节点rac12chub2对比一下，看看是否有不match的情况：
grid@rac12chub1:/home/grid>cluvfy comp peer -refnode rac12chub2 -n rac12cleaf1 -verbose
中间输出省略...
Verifying Peer Compatibility ...PASSED
Verification of peer compatibility was successful.
CVU operation performed: peer compatibility
Date: May 6, 2018 10:50:31 PM
CVU home: /g01/app/grid/12.2.0/
User: grid
2、RAC架构变更为GNS解析SCAN后，再次运行检查环境
grid@rac12chub1:/home/grid>cluvfy stage -pre nodeadd -n rac12cleaf1 -fixup -verbose
报错如下..
Failures were encountered during execution of CVU verification request "stage -pre nodeadd".
忽略...
Verifying Swap Size ...FAILED
rac12cleaf1: PRVF-7573 : Sufficient swap size is not available on node
"rac12cleaf1" [Required = 7.6404GB (8011500.0KB) ; Found = 3.875GB
(4063228.0KB)]
rac12chub1: PRVF-7573 : Sufficient swap size is not available on node
"rac12chub1" [Required = 7.6393GB (8010420.0KB) ; Found = 2GB
(2097148.0KB)]
忽略...
Verifying OS Kernel Version ...WARNING
PRVF-7524 : Kernel version is not consistent across all the nodes.
Kernel version = "3.10.0-693.21.1.el7.x86_64" found on nodes: rac12chub1.
Kernel version = "3.10.0-229.el7.x86_64" found on nodes: rac12cleaf1.
忽略...(其实后续这里验证是被打脸....没直连的话 执行root.sh GI资源都启动不了，还是不能忽略该问题，直连存储解决)
Verifying Device Checks for ASM ...FAILED
Verifying ASM device sharedness check ...WARNING
Verifying Shared Storage
Accessibility:/dev/asm-diskd,/dev/asm-diske,/dev/asm-diskf,/dev/asm-diskg,/d
ev/asm-diska,/dev/asm-diskb,/dev/asm-diskc ...WARNING
PRVG-1615 : Virtual environment detected. Skipping shared storage check for
disks
"/dev/asm-diskd,/dev/asm-diske,/dev/asm-diskf,/dev/asm-diskg,/dev/asm-diska,
/dev/asm-diskb,/dev/asm-diskc".
Verifying Access Control List check ...FAILED
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskg" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskf" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diske" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskd" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskc" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskb" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diska" on node "rac12cleaf1"
忽略...
Verifying Network Time Protocol (NTP) ...FAILED
Verifying '/etc/chrony.conf' ...FAILED
rac12cleaf1: PRVG-1019 : The NTP configuration file "/etc/chrony.conf" does
not exist on nodes "rac12cleaf1"
这个我们接下来，验证下主机名是否能被DNS解析
Verifying resolv.conf Integrity ...FAILED
rac12cleaf1: PRVG-10048 : Name "rac12cleaf1" was not resolved to an address of
the specified type by name servers o"172.16.10.138".
rac12cleaf1: PRVG-10048 : Name "rac12cleaf1" was not resolved to an address of
the specified type by name servers o"172.16.10.139".
rac12cleaf1: Check for integrity of file "/etc/resolv.conf" failed
rac12chub1: PRVG-10048 : Name "rac12chub1" was not resolved to an address of
the specified type by name servers o"172.16.10.138".
rac12chub1: PRVG-10048 : Name "rac12chub1" was not resolved to an address of
the specified type by name servers o"172.16.10.139".
rac12chub1: PRVG-10048 : Name "rac12chub1" was not resolved to an address of
the specified type by name servers o"172.16.10.254".
rac12chub1: Check for integrity of file "/etc/resolv.conf" failed
Verifying (Linux) resolv.conf Integrity ...FAILED
rac12cleaf1: PRVG-10048 : Name "rac12cleaf1" was not resolved to an address
of the specified type by name servers o"172.16.10.138".
rac12cleaf1: PRVG-10048 : Name "rac12cleaf1" was not resolved to an address
of the specified type by name servers o"172.16.10.139".
rac12chub1: PRVG-10048 : Name "rac12chub1" was not resolved to an address of
the specified type by name servers o"172.16.10.138".
rac12chub1: PRVG-10048 : Name "rac12chub1" was not resolved to an address of
the specified type by name servers o"172.16.10.139".
rac12chub1: PRVG-10048 : Name "rac12chub1" was not resolved to an address of
the specified type by name servers o"172.16.10.254".
CVU operation performed: stage -pre nodeadd
Date: May 6, 2018 5:59:48 PM
CVU home: /g01/app/grid/12.2.0/
User: grid
3、验证主机名是否能被解析
[root@rac12chub1 ~]# nslookup rac12chub1
;; Got SERVFAIL reply from 172.16.10.138, trying next server
;; Got recursion not available from 172.16.10.139, trying next server
Server: 172.16.10.254
Address: 172.16.10.254#53
** server can't find rac12chub1: NXDOMAIN
[root@rac12chub1 ~]# nslookup rac12chub2
;; Got SERVFAIL reply from 172.16.10.138, trying next server
;; Got recursion not available from 172.16.10.139, trying next server
Server: 172.16.10.254
Address: 172.16.10.254#53
** server can't find rac12chub2: NXDOMAIN
[root@rac12chub1 ~]# nslookup rac12cleaf1
;; Got SERVFAIL reply from 172.16.10.138, trying next server
;; Got recursion not available from 172.16.10.139, trying next server
Server: 172.16.10.254
Address: 172.16.10.254#53
** server can't find rac12cleaf1: NXDOMAIN
发现，确实不能备解析..而且DNS服务器为172.16.10.254.显然不对...
4、查看resove.conf配置文件（所有节点)
# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 172.16.10.138
nameserver 172.16.10.139
nameserver 172.16.10.254
options rotate
options timeout: 2
options attempts: 5
发现没有serach 域名配置，导致无法解析主机
5、修改resove.conf配置文件（所有节点)
# vi /etc/resolv.conf
增加一行： search marvin.cn // 域名 marvin.cn
# Generated by NetworkManager
nameserver 172.16.10.138
nameserver 172.16.10.139
nameserver 172.16.10.254
search marvin.cn
options rotate
options timeout: 2
options attempts: 5
6、重启网络
# service network restart
7、再次验证主机是否被解析
[root@rac12chub1 ~]# nslookup rac12chub1
Server: 172.16.10.138
Address: 172.16.10.138#53
Name: rac12chub1.marvin.cn
Address: 172.16.10.116
[root@rac12chub1 ~]# nslookup rac12chub2
Server: 172.16.10.138
Address: 172.16.10.138#53
Name: rac12chub2.marvin.cn
Address: 172.16.10.142
[root@rac12chub1 ~]# nslookup rac12cleaf1
Server: 172.16.10.138
Address: 172.16.10.138#53
Name: rac12cleaf1.marvin.cn
Address: 172.16.10.146
主机名现已可正常解析....
8、再次运行检查环境
grid@rac12chub1:/home/grid>cluvfy stage -pre nodeadd -n rac12cleaf1 -fixup -verbose
报错如下..
Failures were encountered during execution of CVU verification request "stage -pre nodeadd".
Verifying Swap Size ...FAILED
rac12cleaf1: PRVF-7573 : Sufficient swap size is not available on node
"rac12cleaf1" [Required = 7.6404GB (8011500.0KB) ; Found = 3.875GB
(4063228.0KB)]
rac12chub2: PRVF-7573 : Sufficient swap size is not available on node
"rac12chub2" [Required = 7.6403GB (8011444.0KB) ; Found = 3.875GB
(4063228.0KB)]
这里可以忽略，由于添加的是Leaf节点，所以可以不直连存储，但是也可以直连，方便以后Leaf叶子节点转换为Hub节点,但是添加Hub节点是必须直连存储
(其实后续这里验证是被打脸....没直连的话 执行root.sh GI资源都启动不了，还是不能忽略该问题，直连存储解决)
Verifying Device Checks for ASM ...FAILED
Verifying ASM device sharedness check ...WARNING
Verifying Shared Storage
Accessibility:/dev/asm-diskd,/dev/asm-diske,/dev/asm-diskf,/dev/asm-diskg,/d
ev/asm-diska,/dev/asm-diskb,/dev/asm-diskc ...WARNING
PRVG-1615 : Virtual environment detected. Skipping shared storage check for
disks
"/dev/asm-diskd,/dev/asm-diske,/dev/asm-diskf,/dev/asm-diskg,/dev/asm-diska,
/dev/asm-diskb,/dev/asm-diskc".
Verifying Access Control List check ...FAILED
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskg" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskf" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diske" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskd" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskc" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diskb" on node "rac12cleaf1"
rac12cleaf1: PRVF-9994 : Owner, group, permission information could not be
obtained for device(s) "/dev/asm-diska" on node "rac12cleaf1"
发现，现在主机名可以被解析，只不过只能被DNS服务器172.16.10.138解析，其他仍然报错....可直接忽略...
Verifying resolv.conf Integrity ...FAILED
PRVG-12864 : The resolver configuration file "/etc/resolv.conf" on the nodes
"rac12cleaf1" has more than one 'options timeout' entry
rac12cleaf1: PRVG-10048 : Name "rac12cleaf1" was not resolved to an address of
the specified type by name servers o"172.16.10.139".
rac12cleaf1: PRVG-10048 : Name "rac12cleaf1" was not resolved to an address of
the specified type by name servers o"172.16.10.254".
rac12cleaf1: Check for integrity of file "/etc/resolv.conf" failed
rac12chub2: PRVG-10048 : Name "rac12chub2" was not resolved to an address of
the specified type by name servers o"172.16.10.139".
rac12chub2: PRVG-10048 : Name "rac12chub2" was not resolved to an address of
the specified type by name servers o"172.16.10.254".
rac12chub2: Check for integrity of file "/etc/resolv.conf" failed
Verifying (Linux) resolv.conf Integrity ...FAILED
rac12cleaf1: PRVG-10048 : Name "rac12cleaf1" was not resolved to an address
of the specified type by name servers o"172.16.10.139".
rac12cleaf1: PRVG-10048 : Name "rac12cleaf1" was not resolved to an address
of the specified type by name servers o"172.16.10.254".
rac12chub2: PRVG-10048 : Name "rac12chub2" was not resolved to an address of
the specified type by name servers o"172.16.10.139".
rac12chub2: PRVG-10048 : Name "rac12chub2" was not resolved to an address of
the specified type by name servers o"172.16.10.254".
CVU operation performed: stage -pre nodeadd
Date: May 7, 2018 10:25:18 AM
CVU home: /g01/app/grid/12.2.0/
User: grid
9、执行添加节点
grid@rac12chub1:/home/grid>cd $ORACLE_HOME/addnode
grid@rac12chub1:/g01/app/grid/12.2.0/addnode>./addnode.sh -silent "CLUSTER_NEW_NODES={rac12cleaf1}" "CLUSTER_NEW_PRIVATE_NODE_NAMES={rac12cleaf1-priv}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={auto}" "CLUSTER_NEW_NODE_ROLES={leaf}"
[FATAL] [INS-13013] Target environment does not meet some mandatory requirements.
CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /g01/oraInventory/logs/addNodeActions2018-05-07_12-36-46-PM.log
ACTION: Identify the list of failed prerequisite checks from the log: /g01/oraInventory/logs/addNodeActions2018-05-07_12-36-46-PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
解决：增加参数-ignorePrereq 即可
./addnode.sh -silent -ignorePrereq "CLUSTER_NEW_NODES={rac12cleaf1}" "CLUSTER_NEW_PRIVATE_NODE_NAMES={rac12cleaf1-priv}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={auto}" "CLUSTER_NEW_NODE_ROLES={leaf}"
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 7% Done.
Copy Files to Remote Nodes in progress.
.................................................. 12% Done.
.................................................. 17% Done.
..............................
Copy Files to Remote Nodes successful.
You can find the log of this install session at:
/g01/oraInventory/logs/addNodeActions2018-05-07_01-34-36-PM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 49% Done.
Saving cluster inventory in progress.
.................................................. 83% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /g01/app/grid/12.2.0 was successful.
Please check '/g01/app/grid/12.2.0/inventory/silentInstall2018-05-07_1-34-34-PM.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 90% Done.
Update Inventory in progress.
Update Inventory successful.
.................................................. 97% Done.
As a root user, execute the following script(s):
1. /g01/oraInventory/orainstRoot.sh
2. /g01/app/grid/12.2.0/root.sh
Execute /g01/oraInventory/orainstRoot.sh on the following nodes:
[rac12cleaf1]
Execute /g01/app/grid/12.2.0/root.sh on the following nodes:
[rac12cleaf1]
The scripts can be executed in parallel on all the nodes.
.................................................. 100% Done.
Successfully Setup Software.
10、新开窗口查看日志输出：
grid@rac12chub1:/home/grid>cd /g01/oraInventory/logs
grid@rac12chub1:/g01/oraInventory/logs>tail -f addNodeActions2018-05-07_01-34-36-PM.log
11、切换Leaf节点rac12cleaf1执行脚本(root用户)
[root@rac12cleaf1 ~]# /g01/oraInventory/orainstRoot.sh
Changing permissions of /g01/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /g01/oraInventory to oinstall.
The execution of the script is complete.
[root@rac12cleaf1 ~]# /g01/app/grid/12.2.0/root.sh
Check /g01/app/grid/12.2.0/install/root_rac12cleaf1_2018-05-07_13-46-51-600062985.log for the output of root script
查看日志发现报错....
2018/05/07 13:53:51 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.evmd' on 'rac12cleaf1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac12cleaf1'
CRS-2676: Start of 'ora.evmd' on 'rac12cleaf1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac12cleaf1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac12cleaf1'
CRS-2676: Start of 'ora.gpnpd' on 'rac12cleaf1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac12cleaf1'
CRS-2676: Start of 'ora.gipcd' on 'rac12cleaf1' succeeded
CRS-2883: Resource 'ora.gpnpd' failed during Clusterware stack start.
===== Summary of resource auto-start failures follows =====
CRS-2807: Resource 'ora.gpnpd' failed to start automatically.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2018/05/07 14:10:10 CLSRSC-378: Failed to get the configured node role for the local node
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
2018/05/07 14:10:10 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /g01/app/grid/12.2.0/crs/install/crsinstall.pm line 1494.
The command '/g01/app/grid/12.2.0/perl/bin/perl -I/g01/app/grid/12.2.0/perl/lib -I/g01/app/grid/12.2.0/crs/install /g01/app/grid/12.2.0/crs/install/r tcrs.pl ' execution failed
不能开启CRS，怀疑没有直连存储的问题....集群服务都未启动
grid@rac12cleaf1:/home/grid>crsctl status res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
grid@rac12cleaf1:/home/grid>crsctl status res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 OFFLINE OFFLINE STABLE
ora.cluster_interconnect.haip
1 ONLINE OFFLINE STABLE
ora.crf
1 ONLINE OFFLINE STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE OFFLINE STABLE
ora.cssdmonitor
1 ONLINE OFFLINE STABLE
ora.ctssd
1 ONLINE OFFLINE STABLE
ora.diskmon
1 ONLINE OFFLINE STABLE
ora.evmd
1 ONLINE INTERMEDIATE rac12cleaf1 STABLE
ora.gipcd
1 ONLINE ONLINE rac12cleaf1 STABLE
ora.gpnpd
1 ONLINE INTERMEDIATE rac12cleaf1 STABLE
ora.mdnsd
1 ONLINE ONLINE rac12cleaf1 STABLE
ora.storage
1 ONLINE OFFLINE STABLE
------------------------------------------------------------------------
grid@rac12chub1:/g01/app/grid/12.2.0/addnode>cluvfy stage -post nodeadd -n rac12cleaf1
Shared resources check for node addition failed
Post-check for node addition was unsuccessful on all the nodes.
CVU operation performed: stage -post nodeadd
Date: May 7, 2018 2:28:57 PM
CVU home: /g01/app/grid/12.2.0/
User: grid
上面信息也显示，添加节点不成功...
12、尝试将Leaf节点连接存储（注意当前环境是测试环境，所以采用的是ISCSI共享存储方式)
1、安装initiator
[root@rac12cleaf1 ~]# yum -y install iscsi-initiator-utils
报错：
报错截取部分记录....
Trying other mirror.
iscsi-initiator-utils-6.2.0.87 FAILED
http://mirrors.tuna.tsinghua.edu.cn/centos/7.4.1708/os/x86_64/Packages/iscsi-initiator-utils-6.2.0.874-4.el7.x86_64.rpm: [Errno 14] curl#6 - "Could not resolve host: mirrors.tuna.tsinghua.edu.cn; Name or service not known"
Trying other mirror.
Error downloading packages:
iscsi-initiator-utils-6.2.0.874-4.el7.x86_64: [Errno 256] No more mirrors to try.
iscsi-initiator-utils-iscsiuio-6.2.0.874-4.el7.x86_64: [Errno 256] No more mirrors to try.
解决方法：
修改resolve.conf配置文件
注释serach marvin.cn这一行
[root@rac12cleaf1 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 172.16.10.138
nameserver 172.16.10.139
nameserver 172.16.10.254
#search marvin.cn
options rotate
options timeout: 2
options attempts: 5
options rotate
options timeout: 2
重启网络
[root@rac12cleaf1 ~]# service network restart
重新安装即可
[root@rac12cleaf1 ~]# yum -y install iscsi-initiator-utils
2、探测target
#iscsiadm -m discovery --type sendtargets --portal 172.16.10.200
3、写入侦探信息
#ll -R /var/lib/iscsi/nodes/
/var/lib/iscsi/nodes/:
total 0
drw------- 6 root root 129 May 7 14:42 iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c
/var/lib/iscsi/nodes/iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c:
total 0
drw------- 2 root root 20 May 7 14:42 169.254.221.30,3260,1
drw------- 2 root root 20 May 7 14:42 172.16.10.200,3260,1
drw------- 2 root root 20 May 7 14:42 2001:db8:7::1746,3260,1
drw------- 2 root root 20 May 7 14:42 fe80::211:32ff:fe68:5e7b,3260,1
/var/lib/iscsi/nodes/iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c/169.254.221.30,3260,1:
total 4
-rw------- 1 root root 2090 May 7 14:42 default
/var/lib/iscsi/nodes/iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c/172.16.10.200,3260,1:
total 4
-rw------- 1 root root 2089 May 7 14:42 default
/var/lib/iscsi/nodes/iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c/2001:db8:7::1746,3260,1:
total 4
-rw------- 1 root root 2092 May 7 14:42 default
/var/lib/iscsi/nodes/iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c/fe80::211:32ff:fe68:5e7b,3260,1:
total 4
-rw------- 1 root root 2100 May 7 14:42 default
4、手动挂载不用重启，注意下面命令中值Yunqu-DS.Target-1.95b456097c，需要变更成侦探信息中的名字值（保证一致即可)
[root@rac12cleaf1 ~]# iscsiadm -m node -T iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c –p 172.16.10.200:3260 -l
Logging in to [iface: default, target: iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c, portal: 172.16.10.200,3260] (multiple)
Logging in to [iface: default, target: iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c, portal: fe80::211:32ff:fe68:5e7b,3260] (multiple)
Logging in to [iface: default, target: iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c, portal: 169.254.221.30,3260] (multiple)
Logging in to [iface: default, target: iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c, portal: 2001:db8:7::1746,3260] (multiple)
Login to [iface: default, target: iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c, portal: 172.16.10.200,3260] successful.
iscsiadm: Could not login to [iface: default, target: iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c, portal: fe80::211:32ff:fe68:5e7b,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not login to [iface: default, target: iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c, portal: 169.254.221.30,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not login to [iface: default, target: iqn.2000-01.com.synology:Yunqu-DS.Target-1.95b456097c, portal: 2001:db8:7::1746,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not log into all portals
查看磁盘
[root@rac12cleaf1 ~]# ll /dev/sd*
brw-rw---- 1 root disk 8, 0 May 7 14:44 /dev/sda
brw-rw---- 1 root disk 8, 16 May 7 14:44 /dev/sdb
brw-rw---- 1 root disk 8, 32 May 7 14:44 /dev/sdc
brw-rw---- 1 root disk 8, 48 May 7 14:44 /dev/sdd
brw-rw---- 1 root disk 8, 64 May 7 14:44 /dev/sde
brw-rw---- 1 root disk 8, 80 May 7 14:44 /dev/sdf
brw-rw---- 1 root disk 8, 96 May 7 14:44 /dev/sdg
13、传送ASM规则文件，启动UDEV
[root@rac12chub1 rules.d]# scp 99-my-asmdevices.rules 172.16.10.146:/etc/udev/rules.d/
root@172.16.10.146's password:
99-my-asmdevices.rules 100% 1561 1.5KB/s 00:00
[root@rac12chub1 rules.d]#systemctl restart systemd-udev-trigger.service
验证
[root@rac12cleaf1 rules.d]# ll /dev/asm*
lrwxrwxrwx 1 root root 3 May 7 15:02 /dev/asm-diska -> sda
lrwxrwxrwx 1 root root 3 May 7 15:02 /dev/asm-diskb -> sdb
lrwxrwxrwx 1 root root 3 May 7 15:02 /dev/asm-diskc -> sdc
lrwxrwxrwx 1 root root 3 May 7 15:02 /dev/asm-diskd -> sdd
lrwxrwxrwx 1 root root 3 May 7 15:02 /dev/asm-diske -> sde
lrwxrwxrwx 1 root root 3 May 7 15:02 /dev/asm-diskf -> sdf
lrwxrwxrwx 1 root root 3 May 7 15:02 /dev/asm-diskg -> sdg
14、重新执行root.sh脚本
[root@rac12cleaf1 rules.d]# /g01/app/grid/12.2.0/root.sh
Check /g01/app/grid/12.2.0/install/root_rac12cleaf1_2018-05-07_15-05-54-441970221.log for the output of root script
15、查看执行日志：
[root@rac12cleaf1 ~]# tail -f /g01/app/grid/12.2.0/install/root_rac12cleaf1_2018-05-07_15-05-54-441970221.log
报错如下，但之前的gpnpd资源前已经全启动....
CRS-4133: Oracle High Availability Services has been stopped.
2018/05/07 15:07:15 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac12cleaf1'
CRS-2672: Attempting to start 'ora.evmd' on 'rac12cleaf1'
CRS-2676: Start of 'ora.mdnsd' on 'rac12cleaf1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac12cleaf1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac12cleaf1'
CRS-2676: Start of 'ora.gpnpd' on 'rac12cleaf1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac12cleaf1'
CRS-2676: Start of 'ora.gipcd' on 'rac12cleaf1' succeeded
CRS-2883: Resource 'ora.gpnpd' failed during Clusterware stack start.
===== Summary of resource auto-start failures follows =====
CRS-2807: Resource 'ora.gpnpd' failed to start automatically.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2018/05/07 15:23:25 CLSRSC-378: Failed to get the configured node role for the local node
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
2018/05/07 15:23:25 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /g01/app/grid/12.2.0/crs/install/crsinstall.pm line 1494.
The command '/g01/app/grid/12.2.0/perl/bin/perl -I/g01/app/grid/12.2.0/perl/lib -I/g01/app/grid/12.2.0/crs/install /g01/app/grid/12.2.0/crs/install/rootcrs.pl ' execution failed
报错，主要是CRS-2883: Resource 'ora.gpnpd' failed during Clusterware stack start.引起...
解决方法：
1、打补丁...(mos文章:Bug 24411578 - root.sh Execution Fails Starting GPNPD on Non-First Nodes (文档 ID 24411578.8)),需要打个系统补丁(补丁号：27468969)，注意所有RAC节点，都需要打上该补丁
2、打完补丁...还是不行....报错仍然一样....
Died at /g01/app/grid/12.2.0/crs/install/crsinstall.pm line 1494.
/g01/app/grid/12.2.0/perl/bin/perl -I/g01/app/grid/12.2.0/perl/lib -I/g01/app/grid/12.2.0/crs/install /g01/app/grid/12.2.0/crs/install/rootcrs.pl execution failed
在执行1494行，执行rootcrs.pl脚本报错。怀疑是不是perl脚本无法执行，将失败节点执行的sh脚本，换成perl脚本，发现报错
[root@rac12cleaf1 install]# ./rootcrs.pl -deconfig -verbose -force
/usr/bin/perl:bad interpreter :no file or directory
发现perl脚本环境，该台主机根本没有安装
[root@rac12cleaf1 install]# yum install perl*
[root@rac12cleaf1 install]# ./rootcrs.pl -deconfig -verbose -force
3、重新执行root.sh，步骤跳转到第17步骤...[root@rac12cleaf1 install]# /g01/app/grid/12.2.0/root.sh
16、执行失败Leaf节点rac12cleaf1上执行
[root@rac12cleaf1 install]# ./rootcrs.sh -deconfig -verbose -force
17、重新执行
[root@rac12cleaf1 ~]# /g01/app/grid/12.2.0/root.sh
Check /g01/app/grid/12.2.0/install/root_rac12cleaf1_2018-05-11_16-30-57-028496480.log for the output of root script
[root@rac12cleaf1 ~]#
终于执行成功....
六、验证Leaf节点是否添加成功
1、Hub节点验证：
grid@rac12chub1:/home/grid>cluvfy stage -post nodeadd -n rac12cleaf1 -verbose
2、Leaf节点执行：
grid@rac12cleaf1:/home/grid>crsctl check cluster -all
**************************************************************
rac12chub1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac12chub2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac12cleaf1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
grid@rac12cleaf1:/home/grid>olsnodes -n
rac12chub1 1
rac12chub2 2
rac12cleaf1 100
grid@rac12cleaf1:/home/grid>crsctl get node role config -all
Node 'rac12chub1' configured role is 'hub'
Node 'rac12chub2' configured role is 'hub'
Node 'rac12cleaf1' configured role is 'leaf'
grid@rac12cleaf1:/home/grid>
grid@rac12cleaf1:/>crsctl get node role status -all
Node 'rac12chub1' active role is 'hub'
Node 'rac12chub2' active role is 'hub'
Node 'rac12cleaf1' active role is 'leaf'
grid@rac12cleaf1:/home/grid>crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE rac12chub1 STABLE
ONLINE ONLINE rac12chub2 STABLE
ora.DATA.dg
ONLINE ONLINE rac12chub1 STABLE
ONLINE ONLINE rac12chub2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rac12chub1 STABLE
ONLINE ONLINE rac12chub2 STABLE
ora.LISTENER_LEAF.lsnr
OFFLINE OFFLINE rac12cleaf1 STABLE
ora.MGMT.dg
ONLINE ONLINE rac12chub1 STABLE
ONLINE ONLINE rac12chub2 STABLE
ora.OCR.dg
ONLINE ONLINE rac12chub1 STABLE
ONLINE ONLINE rac12chub2 STABLE
ora.net1.network
ONLINE ONLINE rac12chub1 STABLE
ONLINE ONLINE rac12chub2 STABLE
ora.ons
ONLINE ONLINE rac12chub1 STABLE
ONLINE ONLINE rac12chub2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac12chub2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac12chub1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac12chub1 STABLE
ora.MGMTLSNR
1 OFFLINE OFFLINE STABLE
ora.asm
1 ONLINE ONLINE rac12chub1 Started,STABLE
2 ONLINE ONLINE rac12chub2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 OFFLINE OFFLINE STABLE
ora.gns
1 ONLINE ONLINE rac12chub1 STABLE
ora.gns.vip
1 ONLINE ONLINE rac12chub1 STABLE
ora.marvin.db
1 ONLINE ONLINE rac12chub1 Open,HOME=/u01/oracl
e/12.2.0,STABLE
2 ONLINE ONLINE rac12chub2 Open,HOME=/u01/oracl
e/12.2.0,STABLE
ora.qosmserver
1 OFFLINE OFFLINE STABLE
ora.rac12chub1.vip
1 ONLINE ONLINE rac12chub1 STABLE
ora.rac12chub2.vip
1 ONLINE ONLINE rac12chub2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac12chub2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac12chub1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac12chub1 STABLE
--------------------------------------------------------------------------------
至此，clusterware在RAC12CLEAF1复制运行成功，接下来需要复制ORACLE_HOME。
为了确定rac12cleaf1节点连得是那个hub node，我们看下ocssdrim进程的trace文件：
grid@rac12cleaf1:/>cat $ORACLE_BASE/diag/crs/host05/crs/trace/ocssdrim.trc |grep 'Destroying connection' | tail -1
grid@rac12cleaf1:/>cat $ORACLE_BASE/diag/crs/rac12cleaf1/crs/trace/ocssdrim.trc |grep 'Sending a ping msg to' | tail -1
2018-05-11 20:23:57.493 : CSSD:3109656320: clssbnmc_PeriodicPing_CB: Sending a ping msg to host rac12chub2, number 2, using handle (0x234add0) last msg to hub at 626787034, connection timeout at 626817034, current time 626802024
总结：
12c引入的flex cluster有两种节点，hub node和leaf node
当leaf node上的集群件启动时，leaf node自动使用GNS来发现所有hub node，然后通过其中一个hub node连入集群
如果hub node挂了，那么通过这个hub node访问集群的leaf node会自动切换至某存活的hub node继续访问集群()
七、DB ORACLE复制配置
切换到RAC节点1 rac12chub1节点上，执行复制添加ORACLE节点
在rac12cleaf1上以oracle用户运行：
[root@rac12chub1 install]# su - oracle
Last login: Fri May 11 15:53:53 CST 2018 on pts/0
****************************************************************
***You login as oracle,Please ask somebody to double check!******
****************************************************************
oracle@rac12chub1:/home/oracle>$ORACLE_HOME/addnode/addnode.sh -silent "CLUSTER_NEW_NODES={rac12cleaf1} "
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 4883 MB Passed
Checking swap space: must be greater than 150 MB. Actual 1417 MB Passed
[FATAL] [INS-13013] Target environment does not meet some mandatory requirements.
CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /g01/oraInventory/logs/addNodeActions2018-05-11_05-03-01PM.log
ACTION: Identify the list of failed prerequisite checks from the log: /g01/oraInventory/logs/addNodeActions2018-05-11_05-03-01PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
添加参数-ignorePrereq 重新执行
oracle@rac12chub1:/home/oracle>$ORACLE_HOME/addnode/addnode.sh -silent -ignorePrereq "CLUSTER_NEW_NODES={rac12cleaf1} "
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 4881 MB Passed
Checking swap space: must be greater than 150 MB. Actual 1408 MB Passed
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 7% Done.
Copy Files to Remote Nodes in progress.
.................................................. 12% Done.
.................................................. 18% Done.
..............................[FATAL] [INS-10110] Installer encountered errors while copying /u01/oracle/12.2.0 to nodes [rac12cleaf1].
ACTION: Review the log file /g01/oraInventory/logs/addNodeActions2018-05-11_05-05-12PM.log for further details on failure.
*ADDITIONAL INFORMATION:*
Cause
- PRKC-1175 : Failed to copy directories "/u01/oracle/12.2.0" to one or more nodes PRCF-2015 : One or more commands were not executed successfully on one or more nodesrac12cleaf1: PRCF-2014 : Executing command "wrfile" on ".patch_storage/27674384_Apr_9_2018_00_43_55/files/lib/libserver12.a/kxdffr.o" failed on node "rac12cleaf1". No space left on device PRCF-2002 : Connection to node "rac12cleaf1" is lost PRCF-2002 : Connection to node "true" is lost PRCF-2014 : Executing command "wrfile" on "suptools/tfa/release/tfa_home/ext/orachk/.cgrep/exalogic_zfs_checks_el_extensive.aksh" failed on node "rac12cleaf1". No space left on device PRCF-2002 : Connection to node "rac12cleaf1" is lost PRCF-2002 : Connection to node "true" is lost PRCF-2014 : Executing command "wrfile" on "md/admin/sdoinspirexsd.sql" failed on node "rac12cleaf1". No space left on device PRCF-2002 : Connection to node "rac12cleaf1" is lost PRCF-2066 : Error sending file "/u01/oracle/12.2.0/md/admin/sdoepsg.dmp". Details: PRCF-2002 : Connection to node "true" is lost
Summary of the failed nodes
rac12cleaf1
- PRCF-2014 : Executing command "wrfile" on ".patch_storage/27674384_Apr_9_2018_00_43_55/files/lib/libserver12.a/kxdffr.o" failed on node "rac12cleaf1". No space left on device PRCF-2002 : Connection to node "rac12cleaf1" is lost PRCF-2002 : Connection to node "true" is lost PRCF-2014 : Executing command "wrfile" on "suptools/tfa/release/tfa_home/ext/orachk/.cgrep/exalogic_zfs_checks_el_extensive.aksh" failed on node "rac12cleaf1". No space left on device PRCF-2002 : Connection to node "rac12cleaf1" is lost PRCF-2002 : Connection to node "true" is lost PRCF-2014 : Executing command "wrfile" on "md/admin/sdoinspirexsd.sql" failed on node "rac12cleaf1". No space left on device PRCF-2002 : Connection to node "rac12cleaf1" is lost PRCF-2066 : Error sending file "/u01/oracle/12.2.0/md/admin/sdoepsg.dmp". Details: PRCF-2002 : Connection to node "true" is lost
Copy Files to Remote Nodes failed.
看起起来是由于RCA12Cleaf1节点空间不够 - PRCF-2014 : Executing command "wrfile" on ".patch_storage/27674384_Apr_9_2018_00_43_55/files/lib/libserver12.a/kxdffr.o" failed on node "rac12cleaf1". No space left on devic
导致后面多个命令连续报错
[root@rac12cleaf1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 36G 36G 3.3M 100% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 3.9G 57M 3.8G 2% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/vda1 497M 121M 377M 25% /boot
tmpfs 783M 0 783M 0% /run/user/0
tmpfs 783M 0 783M 0% /run/user/1001
确实空间不够..手动清理
[root@rac12cleaf1 home]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 36G 25G 11G 71% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 3.9G 57M 3.8G 2% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/vda1 497M 121M 377M 25% /boot
tmpfs 783M 0 783M 0% /run/user/0
tmpfs 783M 0 783M 0% /run/user/1001
重新执行添加AddNode节点脚本
oracle@rac12chub1:/home/oracle>$ORACLE_HOME/addnode/addnode.sh -silent -ignorePrereq "CLUSTER_NEW_NODES={rac12cleaf1} "
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 4877 MB Passed
Checking swap space: must be greater than 150 MB. Actual 1403 MB Passed
[WARNING] [INS-06009] SSH performance is detected to be slow, which may impact performance during remote node operations like copying the software and executing prerequisite checks.
ACTION: Consider optimizing the ssh configuration.
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 7% Done.
Copy Files to Remote Nodes in progress.
.................................................. 12% Done.
.................................................. 18% Done.
..............................
Copy Files to Remote Nodes successful.
You can find the log of this install session at:
/g01/oraInventory/logs/installActions2018-05-11_05-29-20PM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 52% Done.
Saving cluster inventory in progress.
.................................................. 89% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/oracle/12.2.0 was successful.
Please check '/tmp/silentInstall2018-05-11_5-29-23-PM.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 96% Done.
As a root user, execute the following script(s):
1. /u01/oracle/12.2.0/root.sh
Execute /u01/oracle/12.2.0/root.sh on the following nodes:
[rac12cleaf1]
.................................................. 100% Done.
Successfully Setup Software.
复制成功
切换叶子节点RAC12CLeaf1节点(root用户)
[root@rac12cleaf1 home]# /u01/oracle/12.2.0/root.sh
Check /u01/oracle/12.2.0/install/root_rac12cleaf1_2018-05-11_17-50-15-397701078.log for the output of root script
[root@rac12cleaf1 home]#
[root@rac12cleaf1 bin]# su - grid
Last login: Fri May 11 18:04:50 CST 2018 on pts/3
****************************************************************
***You login as oracle,Please ask somebody to double check!******
****************************************************************
grid@rac12cleaf1:/home/grid>sqlplus / as sysasm
SQL*Plus: Release 12.2.0.1.0 Production on Fri May 11 18:12:39 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to an idle instance.
SQL>
[root@rac12cleaf1 bin]# su - oracle
Last login: Fri May 11 18:12:27 CST 2018 on pts/3
****************************************************************
***You login as oracle,Please ask somebody to double check!******
****************************************************************
oracle@rac12cleaf1:/home/oracle>sqlplus / as sysdba
SQL*Plus: Release 12.2.0.1.0 Production on Fri May 11 18:13:04 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/oracle/12.2.0/dbs/initmarvin3.ora'
SQL> exit
Disconnected
不管是GRID还是ORACLE 都还无法运行....
总结：
1、Leaf节点ASM实例是无法运行的，DB实例默认不运行，只能以只读方式运行DB实例，并且需要另外配置才可以，第八步骤将配置DB实例运行在Leaf节点
2、Leaf节点可以不需要VIP，但是如果Leaf节点想转变成Hub节点则需要VIP，若未使用DHCP服务器，则需要手动添加VIP，手动添加命令如下：
$ srvctl config vip -n rac12cleaf1
PRKO-2310 : VIP does not exist on node rac12cleaf1.
#添加VIP，以root用户运行
# /u01/app/12.1.0/grid/bin/srvctl add vip -node rac12cleaf1 -netnum 1 -address rac12cleaf1-vip.marvin.com/255.255.255.0
# /u01/app/12.1.0/grid/bin/srvctl start vip -n rac12cleaf1
但是,如果服务器是使用带有DHCP自动分配的，则不需要手动添加VIP，在Leaf节点转变成Hub节点时，DHCP会自动分配VIP到转变成Leaf节点
Hub节点转变成Leaf节点：
参考链接：https://asanga-pradeep.blogspot.hk/2015/09/changing-hub-node-to-leaf-node-and-vice.html
八、配置DB实例运行在Leaf节点
参考官方文档：https://docs.oracle.com/en/database/oracle/oracle-database/18/rilin/running-database-instances-on-leaf-nodes.html#GUID-35BDCD41-375D-4EB2-B0FC-49345AF6C572
1、Leaf节点上ASM实例是无法运行的,DB实例默认也不运行,但是可以以只读方式打开DB实例
2、DB实例只读方式打开要求：
2.1、Leaf节点需要直连存储,但是不直接访问，而是通过Hub节点（当前环境是已经直连存储了)
2.2、所有节点安装Oracle Db Home以及至少一个Hub节点
2.3、扩展Public 网络到Leaf 节点（ Leaf节点 root 用户操作）
2.3.1、RAC服务器未使用GNS以及DHCP操作命令如下：
srvctl modify network -netnum 1 -extendtoleaf YES
srvctl add vip -node leaf_node -netnum 1 -address leaf_node_vip
srvctl start vip -node leaf_node
srvctl modify listener -extendtoleaf YES
2.3.2、RAC服务器使用GNS以及DHCP操作命令如下（而不使用上面的命令）：（当前环境是使用了GNS，DHCP）
[root@rac12cleaf1 ~]# cd /g01/app/grid/12.2.0/bin/
[root@rac12cleaf1 bin]# ./srvctl modify network -netnum 1 -extendtoleaf YES
[root@rac12cleaf1 bin]# ./srvctl start nodeapps
PRKO-2420 : VIP rac12chub1 is already started on nodes: rac12chub1
PRKO-2420 : VIP rac12chub2 is already started on nodes: rac12chub2
PRKO-2422 : ONS is already started on node(s): rac12chub1,rac12chub2
等一会再验证DHCP是否分配了VIP到LEAF节点
[root@rac12cleaf1 bin]# ./srvctl status nodeapps
VIP 172.16.10.119 is enabled
VIP 172.16.10.119 is running on node: rac12chub1
VIP 172.16.10.114 is enabled
VIP 172.16.10.114 is running on node: rac12chub2
VIP 172.16.10.135 is enabled
VIP 172.16.10.135 is running on node: rac12cleaf1
Network is enabled
Network is running on node: rac12chub1
Network is running on node: rac12chub2
Network is running on node: rac12cleaf1
ONS is enabled
ONS daemon is running on node: rac12chub1
ONS daemon is running on node: rac12chub2
ONS daemon is not running on node: rac12cleaf1
从上面可以看到，DHCP确实已经分配了VIP到LEAF节点
2.3.3、检查Leaf节点是否在RAC集群中
[root@rac12cleaf1 bin]# ./crsctl get cluster type
CRS-6539: The cluster type is 'flex'.
[root@rac12cleaf1 bin]#
3、使用DBCA创建库，使用policy-managed管理(Hub 节点操作)
对于大规模并行查询和只读节点要求Policy-managed 管理数据库，不能扩展admin-managed 到Leaf 节点，也就是说创建数据库需要使用Policy-managed 管理数据库,而非默认的admin-managed管理数据库
3.1、对于大规模并行查询RAC，需在数据库中创建新的服务器池，需确保有至少可创建一个并行查询服务器池
3.2、对于RAC 只读节点，需在Hub节点创建数据库使用Policy-managed 管理数据库(注意：Hub节点操作)，在Leaf节点上添加数据库实例是动态的，可以从命令行进行管理
由于之前我架构是HOSTS转变GNS而来的，参考链接：http://www.cndba.cn/Marvinn/article/2766，所以当前还是使用的admin-managed
grid@rac12chub1:/home/grid>srvctl config database -d marvin
Database unique name: marvin
Database name: marvin
Oracle home: /u01/oracle/12.2.0
Oracle user: oracle
Spfile: +DATA/MARVIN/PARAMETERFILE/spfile.276.974629993
Password file: +DATA/MARVIN/PASSWORD/pwdmarvin.260.974629243
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oinstall
Database instances: marvin1,marvin2
Configured nodes: rac12chub1,rac12chub2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed
从最后一句查看确实是administrator managed管理
查看当前Server pool状态
grid@rac12chub1:/home/grid>srvctl status srvpool -a
Server pool name: Free
Active servers count: 1
Active server names: rac12cleaf1
NAME=rac12cleaf1 STATE=ONLINE
Server pool name: Generic
Active servers count: 2
Active server names: rac12chub1,rac12chub2
NAME=rac12chub1 STATE=ONLINE
NAME=rac12chub2 STATE=ONLINE
可以看到当前Server pool name 为Generic
停止数据库-为Policy-managed 创建Server Pool（基本上只管理数据库选项 一般用 -l,-u.(-l表示最小数目，最小为0，-u 表示最大数目 可根据节点数调整)）
grid@rac12chub1:/home/grid>srvctl stop database -d marvin
grid@rac12chub1:/home/grid>srvctl add srvpool -g SrvLeaf -l 1 -u 2
变更数据库到新SrvLeaf池
grid@rac12chub1:/home/grid>srvctl modify database -db marvin -serverpool SrvLeaf
PRCD-1130 : Failed to convert administrator-managed database marvin into a policy-managed database to use server pool SrvLeaf
PRCR-1071 : Failed to register or update resource ora.marvin.db
CRS-0245: User doesn't have enough privilege to perform the operation
权限不足，切换到oracle
[root@rac12chub1 bin]# srvctl modify database -db marvin -serverpool SrvLeaf -domain marvin.cn
[root@rac12chub1 bin]#
检查Pool状态
grid@rac12chub1:/home/grid>srvctl status srvpool -a
Server pool name: Free
Active servers count: 1
Active server names: rac12cleaf1
NAME=rac12cleaf1 STATE=ONLINE
Server pool name: Generic
Active servers count: 0
Active server names:
Server pool name: SrvLeaf
Active servers count: 2
Active server names: rac12chub1,rac12chub2
NAME=rac12chub1 STATE=ONLINE
NAME=rac12chub2 STATE=ONLINE
可以看到当前Server pool name 为SrvLeaf
查看是否被反转成了policy managed
grid@rac12chub1:/home/grid>srvctl config database -d marvin
Database unique name: marvin
Database name: marvin
Oracle home: /u01/oracle/12.2.0
Oracle user: oracle
Spfile: +DATA/MARVIN/PARAMETERFILE/spfile.276.974629993
Password file: +DATA/MARVIN/PASSWORD/pwdmarvin.260.974629243
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: SrvLeaf
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oinstall
Database instances:
Configured nodes:
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is policy managed
重启数据库
grid@rac12chub1:/home/grid>srvctl start database -d marvin
grid@rac12chub1:/home/grid>
grid@rac12chub1:/home/grid>crsctl add wallet -type OSUSER -user oracle -passwd
Enter Password:
4、RAC 只读节点，使用 add service 命令另外增加服务器池SrvLeaf1(Leaf 节点操作 oracle用户)
oracle@rac12cleaf1:/home/oracle>srvctl add srvpool -serverpool SrvLeaf1 -category LEAF
oracle@rac12cleaf1:/home/oracle>srvctl add service -database marvin -service Leaf -rfpool SrvLeaf1
oracle@rac12cleaf1:/home/oracle>
Reader farm 服务一直启动不起来，但是叶子节点DB实例又是运行的，而且是可读写模式....很奇怪
oracle@rac12cleaf1:/home/oracle>srvctl start service -database marvin -service Leaf
PRCD-1133 : failed to start services leaf for database marvin
PRCR-1095 : Failed to start resources using filter ((NAME == ora.marvin.leaf.svc) AND (TYPE == ora.service.type))
CRS-5045: Reader farm service 'leaf' does not exist in the database 'ora.marvin.db'.
CRS-2674: Start of 'ora.marvin.leaf.svc' on 'rac12cleaf1' failed
CRS-2632: There are no more servers to try to place resource 'ora.marvin.leaf.svc' on that would satisfy its placement policy
oracle@rac12cleaf1:/home/oracle>srvctl status service -d marvin
Service Leaf is not running.
5、数据库实例名发生变更（Hub节点并非之前的marvin1以及marvin2，跟pocily-managed有关，之前是admin-managed）,所以这里需要修改RAC节点的环境变量，才能进入数据库
oracle@rac12cleaf1:/home/oracle>srvctl status database -d marvin
Instance marvin_2 is running on node rac12chub2
Instance marvin_3 is running on node rac12chub1
Instance marvin_1 is running on node rac12cleaf1
现在就出现了一个比较尴尬的情况（对于我们以前管理 RAC 的常识来说），由于 dbserver1 中的实例是后启动的，因此实例名后缀为 2，而 dbserver2 中的实例名后缀是 1，实际上，在 Policy-Managed 管理的 RAC 环境中，无需关注到底哪个实例启动在哪台机器上，我们需要的就是通过 SCAN IP，通过 Service 名去访问数据库就好，而不需要通过实例名访问数据库
oracle@rac12cleaf1:/home/oracle>sqlplus system/yunq111@rac12c-scanip.marvin.cn:1521/marvin
SQL*Plus: Release 12.2.0.1.0 Production on Tue May 15 13:31:46 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Last Successful login time: Tue May 15 2018 13:27:12 +08:00
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> select open_mode from v$database
2 /
OPEN_MODE
--------------------
READ WRITE ----可读写模式，官方说只读节点.... 可能跟我之前RAC模式有关，推测当时添加Leaf节点是admin-managed模式管理，应该添加前转变为policy-managed，再添加(个人推测）...这点有待验证...所以，之前使用RAC12C架构采用admin-managed管理的，需要使用leaf节点，最好环境搭建一套，数据恢复
SQL>
重新DBCA创建数据库Marvinn,选择已经存在创建Hub跟Leaf的Server Pool,发现marvinn.db叶子节点是只读的
ora.marvin.db
1 ONLINE ONLINE rac12cleaf1 Open,HOME=/u01/oracl
e/12.2.0,STABLE
2 ONLINE ONLINE rac12chub2 Open,HOME=/u01/oracl
e/12.2.0,STABLE
3 ONLINE OFFLINE Instance Shutdown,ST
ABLE
ora.marvin.leaf.svc
1 ONLINE OFFLINE STABLE
ora.marvinn.db
1 ONLINE ONLINE rac12cleaf1 Open,Readonly,HOME=/
u01/oracle/12.2.0,ST
ABLE
2 ONLINE ONLINE rac12chub2 Open,HOME=/u01/oracl
e/12.2.0,STABLE
3 ONLINE ONLINE rac12chub1 Open,HOME=/u01/oracl
e/12.2.0,STABLE
oracle@rac12chub1:/home/oracle>srvctl add service -database marvinn -service Leaff -rfpool SrvLeaf1
PRCD-1026 : failed to create service Leaff for database marvinn
PRCS-1132 : Database marvinn is configured to use parallel query services.
从这可以看出，默认选的是的大规模并行查询服务，而非Reader Farm....
6、Leaf节点ASM 实例没有运行，DB实例实现运行
grid@rac12chub1:/home/grid>srvctl status asm
ASM is running on rac12chub1,rac12chub2
grid@rac12chub1:/home/grid>crsctl get node role config -all
Node 'rac12chub1' configured role is 'hub'
Node 'rac12chub2' configured role is 'hub'
Node 'rac12cleaf1' configured role is 'leaf'
grid@rac12chub1:/home/grid>srvctl status database -d marvin
Instance marvin_1 is running on node rac12cleaf1
Instance marvin_2 is running on node rac12chub2
Instance marvin_3 is running on node rac12chub1
oracle@rac12cleaf1:/home/oracle>srvctl status database -d marvinn
Instance marvinn_1 is running on node rac12cleaf1
Instance marvinn_2 is running on node rac12chub2
Instance marvinn_3 is running on node rac12chub1
5、测试连接数据库(marvinn数据库)
oracle@rac12cleaf1:/home/oracle>export ORACLE_SID=marvinn_1
oracle@rac12cleaf1:/home/oracle>sqlplus / as sysdba
SQL*Plus: Release 12.2.0.1.0 Production on Tue May 15 14:53:28 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ ONLY
版权声明：本文为博主原创文章，未经博主允许不得转载。
