签到成功

知道了

CNDBA社区CNDBA社区

Rootupgrade.sh遇到CLSRSC-400: A system reboot is required to continue installing

2018-11-18 21:32 5241 1 原创 oracle 12c
作者： foxytale璜军
本文链接：http://www.cndba.cn/q195136130/article/3151

Rootupgrade.sh遇到CLSRSC-400: A system reboot is required to continue installing.
运行rootupgrad.sh遇到以下告警
[root@ora12c1 rpm]# /u01/app/12.0.0/grid_1/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.0.0/grid_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The file “oraenv” already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.0.0/grid_1/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/ora12c1/crsconfig/rootcrs_ora12c1_2018-11-14_09-34-27PM.log
2018/11/14 21:34:31 CLSRSC-595: Executing upgrade step 1 of 19: ‘UpgradeTFA’.
2018/11/14 21:34:31 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2018/11/14 21:35:13 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2018/11/14 21:35:13 CLSRSC-595: Executing upgrade step 2 of 19: ‘ValidateEnv’.
2018/11/14 21:35:18 CLSRSC-595: Executing upgrade step 3 of 19: ‘GenSiteGUIDs’.
2018/11/14 21:35:19 CLSRSC-595: Executing upgrade step 4 of 19: ‘GetOldConfig’.
2018/11/14 21:35:19 CLSRSC-464: Starting retrieval of the cluster configuration data
2018/11/14 21:35:24 CLSRSC-515: Starting OCR manual backup.
2018/11/14 21:35:27 CLSRSC-516: OCR manual backup successful.
2018/11/14 21:35:35 CLSRSC-486:
At this stage of upgrade, the OCR has changed.
Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.
2018/11/14 21:35:35 CLSRSC-541:
To downgrade the cluster: http://www.cndba.cn/q195136130/article/3151

  1. All nodes that have been upgraded must be downgraded.
    2018/11/14 21:35:35 CLSRSC-542:
  2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.
    2018/11/14 21:35:35 CLSRSC-615:
  3. The last node to downgrade cannot be a Leaf node.
    2018/11/14 21:35:38 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
    2018/11/14 21:35:39 CLSRSC-595: Executing upgrade step 5 of 19: ‘UpgPrechecks’.
    2018/11/14 21:35:40 CLSRSC-363: User ignored prerequisites during installation
    2018/11/14 21:35:48 CLSRSC-595: Executing upgrade step 6 of 19: ‘SaveParamFile’.
    2018/11/14 21:35:51 CLSRSC-595: Executing upgrade step 7 of 19: ‘SetupOSD’.
    2018/11/14 21:35:59 CLSRSC-595: Executing upgrade step 8 of 19: ‘PreUpgrade’.
    2018/11/14 21:36:02 CLSRSC-470: Starting non-rolling migration of Oracle ASM
    2018/11/14 21:36:02 CLSRSC-482: Running command: ‘/u01/app/12.0.0/grid_1/bin/asmca -silent -upgradeNodeASM -nonRolling true -oldCRSHome /u01/app/12.0.0/grid -oldCRSVersion 12.1.0.2.0 -firstNode true -startRolling false ‘

ASM configuration upgraded in local node successfully.http://www.cndba.cn/q195136130/article/3151

2018/11/14 21:36:05 CLSRSC-471: Successfully initiated non-rolling migration of Oracle ASM
2018/11/14 21:36:14 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2018/11/14 21:36:34 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2018/11/14 21:36:35 CLSRSC-595: Executing upgrade step 9 of 19: ‘CheckCRSConfig’.
2018/11/14 21:36:36 CLSRSC-595: Executing upgrade step 10 of 19: ‘UpgradeOLR’.
2018/11/14 21:36:39 CLSRSC-595: Executing upgrade step 11 of 19: ‘ConfigCHMOS’.
2018/11/14 21:36:39 CLSRSC-595: Executing upgrade step 12 of 19: ‘InstallAFD’.
2018/11/14 21:36:41 CLSRSC-595: Executing upgrade step 13 of 19: ‘createOHASD’.
2018/11/14 21:36:44 CLSRSC-595: Executing upgrade step 14 of 19: ‘ConfigOHASD’.
2018/11/14 21:36:59 CLSRSC-329: Replacing Clusterware entries in file ‘oracle-ohasd.service’
2018/11/14 21:37:40 CLSRSC-595: Executing upgrade step 15 of 19: ‘InstallACFS’.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘ora12c1’
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘ora12c1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2018/11/14 21:38:23 CLSRSC-400: A system reboot is required to continue installing.
The command ‘/u01/app/12.0.0/grid_1/perl/bin/perl -I/u01/app/12.0.0/grid_1/perl/lib -I/u01/app/12.0.0/grid_1/crs/install /u01/app/12.0.0/grid_1/crs/install/rootcrs.pl -upgrade’ execution failed
该状态集群属于升级到一半，12102软件已无法启动，12201的软件可以启动，但是在检查版本时，认为是12102，db升级无法继续
查看日志
路径在/u01/app/grid/crsdata/ora12c1/crsconfig/
关键部分如下http://www.cndba.cn/q195136130/article/3151http://www.cndba.cn/q195136130/article/3151

CRS-4621: Oracle High Availability Services autostart is disabled.
End Command output
2018-11-16 10:08:11: /u01/app/12.0.0/grid_1/bin/crsctl disable crs … CRS disabled, ready for reboot.
2018-11-16 10:08:11: USM driver install status is 3
2018-11-16 10:08:11: ACFS drivers unable to be installed.
2018-11-16 10:08:11: Executing cmd: /u01/app/12.0.0/grid_1/bin/clsecho -p has -f clsrsc -m 400
2018-11-16 10:08:11: Command output:
CLSRSC-400: A system reboot is required to continue installing.
End Command output
2018-11-16 10:08:11: CLSRSC-400: A system reboot is required to continue installing.
发现是ACFS组件的问题
根据mos上的文档27769638.8，发现应该是Bug 27769638
根据要求打补丁后，报错依旧。。。。
根据一篇类似安装情况的文档，按文档说法该组件非必须，然后尝试重启服务器，crsctl enable crs的形式，再次执行
报错还是一样。。。
转换思路，先检查该组件
核心日志有该组件安装失败的日志
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9176: Entering ‘ld usm drvs’
ACFS-9154: Loading ‘oracleoks.ko’ driver.
modprobe: FATAL: Module oracleoks not found.
ACFS-9109: oracleoks.ko driver failed to load.
ACFS-9178: Return code = USM_FAIL
ACFS-9177: Return from ‘ld usm drvs’
ACFS-9428: Failed to load ADVM/ACFS drivers. A system reboot is recommended.
ACFS-9310: ADVM/ACFS installation failed.
ACFS-9178: Return code = USM_REBOOT_RECOMMENDED
ACFS-9177: Return from ‘install’
ACFS-9176: Entering ‘acroot ex’
ACFS-9178: Return code = 3
ACFS-9177: Return from ‘acroot ex’

http://www.cndba.cn/q195136130/article/3151
http://www.cndba.cn/q195136130/article/3151

用以下命令检查ACFS组件的状态
/u01/app/12.0.0/grid_1/bin/acfsdriverstate -orahome /u01/app/12.0.0/grid_1/ supported
/u01/app/12.0.0/grid_1/bin/acfsdriverstate -orahome /u01/app/12.0.0/grid_1/ installed
/u01/app/12.0.0/grid_1/bin/acfsdriverstate -orahome /u01/app/12.0.0/grid_1/ loaded
检查状态是supported，installed，not loaded，说明该组件并没有载入
使用/u01/app/12.0.0/grid_1/bin/acfsload start
反馈不明
使用/u01/app/12.0.0/grid_1/bin/acfsroot install –v
发现这个输出与核心日志一致

modprobe: FATAL: Module oracleoks not found.
判断这个是关键原因
/sbin/modprobe -v oracleoks
发现无法找到lib
使用
find /lib/modules | grep oracle
发现找到了lib
[root@ora12c1 3.10.0-862.el7.x86_64]# ls -l /lib/modules/3.10.0-229.1.2.el7.x86_64/extra/usm/oracle*
-rw-r—r—. 1 root root 55953519 Nov 16 13:02 /lib/modules/3.10.0-229.1.2.el7.x86_64/extra/usm/oracleacfs.ko
-rw-r—r—. 1 root root 15504635 Nov 16 13:02 /lib/modules/3.10.0-229.1.2.el7.x86_64/extra/usm/oracleadvm.ko
-rw-r—r—. 1 root root 10299710 Nov 16 13:02 /lib/modules/3.10.0-229.1.2.el7.x86_64/extra/usm/oracleoks.ko
根据文档说明，这个modprobe工具找的是/lib/modules/uname -r的路径
拷贝这些lib到uname –r的路径下
再执行并检查
[root@ora12c1 3.10.0-862.el7.x86_64]# /u01/app/12.0.0/grid_1/bin/acfsroot install –v
ACFS-9459: ADVM/ACFS is not supported on this OS version: ‘3.10.0-862.el7.x86_64’
[root@ora12c1 3.10.0-862.el7.x86_64]# /u01/app/12.0.0/grid_1/bin/acfsdriverstate -orahome /u01/app/12.0.0/grid_1/ supported
ACFS-9459: ADVM/ACFS is not supported on this OS version: ‘3.10.0-862.el7.x86_64’
ACFS-9201: Not Supported
[root@ora12c1 3.10.0-862.el7.x86_64]# /u01/app/12.0.0/grid_1/bin/acfsdriverstate -orahome /u01/app/12.0.0/grid_1/ installed
ACFS-9203: true
发现已不支持acfs组件
重新跑rootupgrade.sh
[root@ora12c1 3.10.0-862.el7.x86_64]# /u01/app/12.0.0/grid_1/rootupgrade.sh
Performing root user operation.

http://www.cndba.cn/q195136130/article/3151
http://www.cndba.cn/q195136130/article/3151

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.0.0/grid_1

http://www.cndba.cn/q195136130/article/3151

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The file “oraenv” already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The contents of “coraenv” have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option

http://www.cndba.cn/q195136130/article/3151

Using configuration parameter file: /u01/app/12.0.0/grid_1/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/ora12c1/crsconfig/rootcrs_ora12c1_2018-11-16_01-14-58PM.log
2018/11/16 13:15:01 CLSRSC-595: Executing upgrade step 1 of 19: ‘UpgradeTFA’.
2018/11/16 13:15:01 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2018/11/16 13:15:01 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2018/11/16 13:16:40 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2018/11/16 13:16:51 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2018/11/16 13:16:51 CLSRSC-595: Executing upgrade step 2 of 19: ‘ValidateEnv’.
2018/11/16 13:16:54 CLSRSC-595: Executing upgrade step 3 of 19: ‘GenSiteGUIDs’.
2018/11/16 13:16:59 CLSRSC-595: Executing upgrade step 4 of 19: ‘GetOldConfig’.
2018/11/16 13:16:59 CLSRSC-464: Starting retrieval of the cluster configuration data
2018/11/16 13:17:05 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2018/11/16 13:17:05 CLSRSC-595: Executing upgrade step 5 of 19: ‘UpgPrechecks’.
2018/11/16 13:17:06 CLSRSC-595: Executing upgrade step 6 of 19: ‘SaveParamFile’.
2018/11/16 13:17:07 CLSRSC-595: Executing upgrade step 7 of 19: ‘SetupOSD’.
2018/11/16 13:17:07 CLSRSC-595: Executing upgrade step 8 of 19: ‘PreUpgrade’.
2018/11/16 13:17:07 CLSRSC-595: Executing upgrade step 9 of 19: ‘CheckCRSConfig’.
2018/11/16 13:17:08 CLSRSC-595: Executing upgrade step 10 of 19: ‘UpgradeOLR’.
2018/11/16 13:17:09 CLSRSC-595: Executing upgrade step 11 of 19: ‘ConfigCHMOS’.
2018/11/16 13:17:54 CLSRSC-595: Executing upgrade step 12 of 19: ‘InstallAFD’.
2018/11/16 13:17:56 CLSRSC-595: Executing upgrade step 13 of 19: ‘createOHASD’.
2018/11/16 13:17:57 CLSRSC-595: Executing upgrade step 14 of 19: ‘ConfigOHASD’.
2018/11/16 13:18:12 CLSRSC-329: Replacing Clusterware entries in file ‘oracle-ohasd.service’
2018/11/16 13:18:33 CLSRSC-595: Executing upgrade step 15 of 19: ‘InstallACFS’.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘ora12c1’
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘ora12c1’
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘ora12c1’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘ora12c1’
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘ora12c1’
CRS-2677: Stop of ‘ora.gipcd’ on ‘ora12c1’ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘ora12c1’ succeeded
CRS-2677: Stop of ‘ora.gpnpd’ on ‘ora12c1’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘ora12c1’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘ora12c1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2018/11/16 13:19:00 CLSRSC-595: Executing upgrade step 16 of 19: ‘InstallKA’.
2018/11/16 13:19:03 CLSRSC-595: Executing upgrade step 17 of 19: ‘UpgradeCluster’.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘ora12c1’
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘ora12c1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start ‘ora.evmd’ on ‘ora12c1’
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘ora12c1’
CRS-2676: Start of ‘ora.mdnsd’ on ‘ora12c1’ succeeded
CRS-2676: Start of ‘ora.evmd’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘ora12c1’
CRS-2676: Start of ‘ora.gpnpd’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘ora12c1’
CRS-2676: Start of ‘ora.gipcd’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘ora12c1’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘ora12c1’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘ora12c1’
CRS-2676: Start of ‘ora.diskmon’ on ‘ora12c1’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘ora12c1’
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘ora12c1’
CRS-2676: Start of ‘ora.ctssd’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.drivers.acfs’ on ‘ora12c1’
CRS-2674: Start of ‘ora.drivers.acfs’ on ‘ora12c1’ failed
CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘ora12c1’
CRS-2676: Start of ‘ora.asm’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.storage’ on ‘ora12c1’
CRS-2676: Start of ‘ora.storage’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.crf’ on ‘ora12c1’
CRS-2676: Start of ‘ora.crf’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘ora12c1’
CRS-2676: Start of ‘ora.crsd’ on ‘ora12c1’ succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: ora12c1
CRS-2672: Attempting to start ‘ora.ora12c1.vip’ on ‘ora12c1’
CRS-2672: Attempting to start ‘ora.scan1.vip’ on ‘ora12c1’
CRS-2672: Attempting to start ‘ora.ons’ on ‘ora12c1’
CRS-2672: Attempting to start ‘ora.MGMTLSNR’ on ‘ora12c1’
CRS-2676: Start of ‘ora.ora12c1.vip’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER.lsnr’ on ‘ora12c1’
CRS-2676: Start of ‘ora.scan1.vip’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN1.lsnr’ on ‘ora12c1’
CRS-2676: Start of ‘ora.MGMTLSNR’ on ‘ora12c1’ succeeded
CRS-2676: Start of ‘ora.ons’ on ‘ora12c1’ succeeded
CRS-2676: Start of ‘ora.LISTENER.lsnr’ on ‘ora12c1’ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN1.lsnr’ on ‘ora12c1’ succeeded
CRS-2672: Attempting to start ‘ora.orcl.db’ on ‘ora12c1’
CRS-2676: Start of ‘ora.orcl.db’ on ‘ora12c1’ succeeded
CRS-6016: Resource auto-start has completed for server ora12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2018/11/16 13:21:28 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
2018/11/16 13:21:36 CLSRSC-595: Executing upgrade step 18 of 19: ‘UpgradeNode’.
2018/11/16 13:21:39 CLSRSC-474: Initiating upgrade of resource types
2018/11/16 13:22:15 CLSRSC-482: Running command: ‘srvctl upgrade model -s 12.1.0.2.0 -d 12.2.0.1.0 -p first’
2018/11/16 13:22:15 CLSRSC-475: Upgrade of resource types successfully initiated.
Start upgrade invoked..
2018/11/16 13:22:20 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded
2018/11/16 13:22:20 CLSRSC-482: Running command: ‘/u01/app/12.0.0/grid_1/bin/crsctl set crs activeversion’
Started to upgrade the active version of Oracle Clusterware. This operation may take a few minutes.
Started to upgrade CSS.
CSS was successfully upgraded.
Started to upgrade CRS.
CRS was successfully upgraded.
Successfully upgraded the active version of Oracle Clusterware.
Oracle Clusterware active version was successfully set to 12.2.0.1.0.
2018/11/16 13:23:25 CLSRSC-479: Successfully set Oracle Clusterware active version
2018/11/16 13:23:25 CLSRSC-476: Finishing upgrade of resource types
2018/11/16 13:23:42 CLSRSC-482: Running command: ‘srvctl upgrade model -s 12.1.0.2.0 -d 12.2.0.1.0 -p last’
2018/11/16 13:23:42 CLSRSC-477: Successfully completed upgrade of resource types
2018/11/16 13:24:01 CLSRSC-595: Executing upgrade step 19 of 19: ‘PostUpgrade’.
2018/11/16 13:24:06 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster … succeeded
成功

总结：这个问题打了相关的补丁也没有解决，与modeprobe找lib相关性大。我倾向于是系统版本的问题（redhat7.5），可能属于个例情况。

版权声明：本文为博主原创文章，未经博主允许不得转载。

12c升级 12201 12102 rootupgrade.sh acfs

用户评论
* 以下用户言论只代表其个人观点，不代表CNDBA社区的观点或立场
foxytale璜军

foxytale璜军

关注
  • 8
    原创
  • 0
    翻译
  • 0
    转载
  • 5
    评论
  • 访问：38120次
  • 积分：34
  • 等级：注册会员
  • 排名：第58名
精华文章

    达梦数据库运维实战

    Oracle 18c 必须掌握的新特性：管理与实战

    Oracle数据库问题解决方案和故障排除手册
    最新问题
    查看更多+
    热门文章
      热门用户
      推荐用户
        友情链接： CNDBA社区 | openGauss | openEuler | openAnolis | 广告联系: database@ustc.edu
        Copyright © 2016 All Rights Reserved. Powered by CNDBA · 皖ICP备2022006297号-1·

        QQ交流群

        注册联系QQ