CNDBA社区
DM 达梦数据库 Fatal error: desc is full while try alloc new page 错误分析解决
作者：
dave
1 现象说明
机器异常断电后达梦实例无法启动。
在上篇博客也提到这个问题：
DM 达梦数据库 Core Dump 文件 说明
https://www.cndba.cn/dave/article/3732
从目前来看，中标麒麟6.0 这个版本的Linux 真的不好用，一个异常断电，差点把所有的达梦测试环境都铲掉了。
这台异常的主机是运行DEM 监控的，上面只有一个实例dave，查看日志：
[dave@www.cndba.cn log]$ ls
DmAPService.log dm_DAVE_201910.log dm_DMAP_201910.log DmServicedave.log
[dave@www.cndba.cn log]$ tail -30 dm_DAVE_201910.log
……
2019-10-21 12:11:23 [INFO] database P0000005642 main_thread systables desc init success.
2019-10-21 12:11:23 [INFO] database P0000005642 main_thread ndct_db_load_info success.
2019-10-21 12:11:23 [INFO] database P0000005642 main_thread nsvr_process_before_open begin.
2019-10-21 12:11:23 [INFO] database P0000005642 main_thread nsvr_process_before_open success.
2019-10-21 12:11:23 [INFO] database P0000005642 main_thread backup control file /dm/dmdbms/data/dave/dm.ctl to file /dm/dmdbms/data/dave/dm_20191021121123_804696.ctl
2019-10-21 12:11:23 [INFO] database P0000005642 main_thread backup control file /dm/dmdbms/data/dave/dm.ctl to file /dm/dmdbms/data/dave/ctl_bak/dm_20191021121123_805896.ctl succeed
2019-10-21 12:11:23 [INFO] database P0000005642 main_thread SYSTEM IS READY.
2019-10-21 12:11:37 [FATAL] database P0000005642 main_thread Fatal error: desc is full while try alloc new page
2019-10-21 12:11:37 [FATAL] database P0000005642 main_thread dm_sys_halt now!!!
2019-10-21 12:11:37 [INFO] database P0000005642 main_thread total 2 rfil opened!
[dave@www.cndba.cn log]$
从错误提示看：Fatal error: desc is full while try alloc new page 是xx 空间满了之类的信息，我们查看信息：
[dave@www.cndba.cn ~]# drz tsfree
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
| TABLESPACE_NAME | TOTAL_SIZE(M) | FREE_SIZE(M) | USED_SIZE(M) | USED_RATE(%) |
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
| SYSTEM | 23 | 16 | 6 | 26 |
| SYSAUX | 128 | 108 | 19 | 15 |
| dave | 32 | 31 | 0 | 0 |
| dave | 32 | 31 | 0 | 0 |
| MAIN | 1392 | 35 | 1356 | 97 |
| TEMP | 10 | 9 | 0 | 0 |
| ROLL | 2959 | 2923 | 35 | 1 |
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[dave@www.cndba.cn ~]# ps -ef|grep dm.ini
dmdba 2183 1 2 13:45 ? 00:00:01 /dm/dmdbms/bin/dmserver /dm/dmdbms/data/dave/dm.ini -noconsole
root 2490 2103 0 13:47 pts/1 00:00:00 grep dm.ini
[dave@www.cndba.cn ~]#
[dave@www.cndba.cn ~]# df -lh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_dm2-lv_root
36G 24G 11G 69% /
tmpfs 4.8G 88K 4.8G 1% /dev/shm
/dev/sda1 485M 43M 417M 10% /boot
You have mail in /var/spool/mail/root
[dave@www.cndba.cn ~]#
表空间和操作系统磁盘空间没有问题。
2 使用dmdbchk 工具检查
使用dmdbchk 检查db：
DM7 达梦数据库 dmdbchk 工具 说明
https://www.cndba.cn/dave/article/3599
第一次提示时，提示异常关闭，无法使用dmdbchk工具：
[dave@www.cndba.cn dave]$ dmdbchk path=dm.ini
[2019-10-21 13:37:26] dmdbchk V7.6.0.95-Build(2018.09.13-97108)ENT
[2019-10-21 13:37:26] file dm.key not found, use default license!
Can not open ini file dmdcr.ini!
Global parameter value of RT_HEAP_TARGET is illegal, use min value!
ndct db load finished
[2019-10-21 13:37:26] dmserver is running or exited abnormally. can not do check
[dave@www.cndba.cn dave]$ ps -ef|grep dm.ini
dmdba 8891 7712 0 13:37 pts/2 00:00:00 grep dm.ini
[dave@www.cndba.cn dave]$
尝试重启操作系统后，DM 实例启动正常没有问题，趁机正常关闭实例，进行dmdbchk检查：如下。
[dave@www.cndba.cn dave]$ dmdbchk path=dm.ini
[2019-10-21 13:50:21] dmdbchk V7.6.0.95-Build(2018.09.13-97108)ENT
[2019-10-21 13:50:21] file dm.key not found, use default license!
Can not open ini file dmdcr.ini!
Global parameter value of RT_HEAP_TARGET is illegal, use min value!
ndct db load finished
[2019-10-21 13:50:22] dmserver is running or exited abnormally. can not do check
[dave@www.cndba.cn dave]$ /etc/init.d/DmServicedave stop
Stopping DmServicedave: [ OK ]
[dave@www.cndba.cn dave]$ dmdbchk path=dm.ini
[2019-10-21 13:50:46] dmdbchk V7.6.0.95-Build(2018.09.13-97108)ENT
[2019-10-21 13:50:46] file dm.key not found, use default license!
Can not open ini file dmdcr.ini!
Global parameter value of RT_HEAP_TARGET is illegal, use min value!
ndct db load finished
[2019-10-21 13:50:46] DM DB CHECK START......
[2019-10-21 13:50:46] --------check dbf file size start---------
[2019-10-21 13:50:46] FILE=(ts_id=0, fil_id=0, path=/dm/dmdbms/data/dave/SYSTEM.DBF)
[2019-10-21 13:50:46] FILE=(ts_id=1, fil_id=0, path=/dm/dmdbms/data/dave/ROLL.DBF)
[2019-10-21 13:50:46] FILE=(ts_id=4, fil_id=0, path=/dm/dmdbms/data/dave/MAIN.DBF)
[2019-10-21 13:50:46] FILE=(ts_id=5, fil_id=0, path=/dm/dmdbms/data/dave/dave01.dbf)
[2019-10-21 13:50:46] FILE=(ts_id=5, fil_id=1, path=/dm/dmdbms/data/dave/dave02.dbf)
[2019-10-21 13:50:46] FILE=(ts_id=8, fil_id=0, path=/dm/dmdbms/data/dave/SYSAWR.DBF)
[2019-10-21 13:50:46] --------check dbf file size end-----------
[2019-10-21 13:50:46] --------check indexes start---------------
[2019-10-21 13:50:46] INDEX=(id=33554433, name=SYSINDEXCOLUMNS, table_name=SYSCOLUMNS)
[2019-10-21 13:50:46] INDEX=(id=33554434, name=SYSINDEXINDEXES, table_name=SYSINDEXES)
[2019-10-21 13:50:46] INDEX=(id=33554440, name=SYSINDEXTUSERS, table_name=SYSUSER$)
[2019-10-21 13:50:46] INDEX=(id=33554442, name=SYSINDEXSYSGRANTS, table_name=SYSGRANTS)
[2019-10-21 13:50:46] INDEX=(id=33554452, name=SYSINDEXCONSTRAINTS, table_name=SYSCONS)
[2019-10-21 13:50:46] INDEX=(id=33554458, name=SYSINDEXSYSAUDIT, table_name=SYSAUDIT)
[2019-10-21 13:50:46] INDEX=(id=33554459, name=SYSINDEXSYSAUDITSQLSEQ, table_name=SYSAUDITSQLSEQ)
[2019-10-21 13:50:46] INDEX=(id=33554464, name=SYSINDEXCONTEXTINDEXES, table_name=SYSCONTEXTINDEXES)
[2019-10-21 13:50:46] INDEX=(id=33554468, name=SYSINDEXSTATS, table_name=SYSSTATS)
[2019-10-21 13:50:46] INDEX=(id=33554489, name=SYSINDEXSPWDCHGS, table_name=SYSPWDCHGS)
[2019-10-21 13:50:46] INDEX=(id=33554498, name=SYSINDEXSYSAUDITRULES, table_name=SYSAUDITRULES)
[2019-10-21 13:50:46] INDEX=(id=33554510, name=SYSINDEXCONTEXTLIBS, table_name=SYSCONTEXTLIBS)
[2019-10-21 13:50:46] INDEX=(id=33554539, name=SYSINDEXCLASSES, table_name=SYSCLASSES)
[2019-10-21 13:50:46] INDEX=(id=33554540, name=SYSINDEXSYSOBJECTS, table_name=SYSOBJECTS)
[2019-10-21 13:50:46] INDEX=(id=33554541, name=SYSINDEXSYSDUAL, table_name=SYSDUAL)
[2019-10-21 13:50:46] INDEX=(id=33554543, name=SYSINDEXSYSTEXTS, table_name=SYSTEXTS)
[2019-10-21 13:50:46] INDEX=(id=33554544, name=SYSINDEXSYSTYPEINFOS, table_name=SYSTYPEINFOS)
[2019-10-21 13:50:46] INDEX=(id=33554546, name=SYSINDEXSYSACCHISTORIES, table_name=SYSACCHISTORIES)
[2019-10-21 13:50:46] INDEX=(id=33554548, name=SYSINDEXSYSHPARTTABLEINFO, table_name=SYSHPARTTABLEINFO)
[2019-10-21 13:50:46] INDEX=(id=33554549, name=SYSINDEXMACPLYS, table_name=SYSMACPLYS)
[2019-10-21 13:50:46] INDEX=(id=33554550, name=SYSINDEXMACLVLS, table_name=SYSMACLVLS)
[2019-10-21 13:50:46] INDEX=(id=33554551, name=SYSINDEXMACCOMPS, table_name=SYSMACCOMPS)
[2019-10-21 13:50:46] INDEX=(id=33554552, name=SYSINDEXMACGRPS, table_name=SYSMACGRPS)
[2019-10-21 13:50:46] INDEX=(id=33554553, name=SYSINDEXMACLABELS, table_name=SYSMACLABELS)
[2019-10-21 13:50:46] INDEX=(id=33554554, name=SYSINDEXMACTABPLY, table_name=SYSMACTABPLY)
[2019-10-21 13:50:46] INDEX=(id=33554555, name=SYSINDEXMACUSRPLY, table_name=SYSMACUSRPLY)
[2019-10-21 13:50:46] INDEX=(id=33554556, name=SYSINDEXSYSOBJINFOS, table_name=SYSOBJINFOS)
[2019-10-21 13:50:46] INDEX=(id=33554559, name=SYSINDEXCOLCYT, table_name=SYSCOLCYT)
[2019-10-21 13:50:46] INDEX=(id=33554560, name=SYSINDEXMACOBJ, table_name=SYSMACOBJ)
[2019-10-21 13:50:46] INDEX=(id=33554561, name=SYSINDEXSYSRESOURCES, table_name=SYSRESOURCES)
[2019-10-21 13:50:46] INDEX=(id=33554562, name=SYSINDEXTABLECOMMENTS, table_name=SYSTABLECOMMENTS)
[2019-10-21 13:50:46] INDEX=(id=33554563, name=SYSINDEXCOLUMNCOMMENTS, table_name=SYSCOLUMNCOMMENTS)
[2019-10-21 13:50:46] INDEX=(id=33554564, name=SYSINDEXSYSREWRITE, table_name=SYS_REWRITE_EQUIVALENCES)
[2019-10-21 13:50:46] INDEX=(id=33554565, name=SYSINDEXSYSDISTABLEINFO, table_name=SYSDISTABLEINFO)
[2019-10-21 13:50:46] INDEX=(id=33554566, name=RINDEXSYSCONS, table_name=SYSCONS)
[2019-10-21 13:50:46] INDEX=(id=33554567, name=FINDEXSYSCONS, table_name=SYSCONS)
[2019-10-21 13:50:46] INDEX=(id=33554568, name=SYSINDEXCOLINFOS, table_name=SYSCOLINFOS)
[2019-10-21 13:50:46] INDEX=(id=33554569, name=SYSINDEXSYSDEPENDENCIES, table_name=SYSDEPENDENCIES)
[2019-10-21 13:50:46] INDEX=(id=33554570, name=RINDSYSDEPENDENCIES, table_name=SYSDEPENDENCIES)
[2019-10-21 13:50:46] INDEX=(id=33554571, name=SYSINDEXFREQROOTS, table_name=SYSFREQROOTS)
[2019-10-21 13:50:46] INDEX=(id=33554572, name=SYSINDEXTABLECOMMENTSKEYS, table_name=SYSTABLECOMMENTS)
[2019-10-21 13:50:46] INDEX=(id=33554573, name=SYSINDEXCOLUMNCOMMENTSKEYS, table_name=SYSCOLUMNCOMMENTS)
[2019-10-21 13:50:46] INDEX=(id=33554574, name=SYSINDEXSYSUSERINI$, table_name=SYSUSERINI$)
[2019-10-21 13:50:46] INDEX=(id=33554575, name=SYSINDEXSYSINJECTHINT, table_name=SYSINJECTHINT)
[2019-10-21 13:50:46] INDEX=(id=33554576, name=SYSINDEXIDSYSOBJECTS, table_name=SYSOBJECTS)
[2019-10-21 13:50:46] INDEX=(id=33555432, name=INDEX33555432, table_name=SYSMACPLYS)
[2019-10-21 13:50:46] INDEX=(id=33555433, name=SYSINDEXPIDIDSYSOBJECTS, table_name=SYSOBJECTS)
[2019-10-21 13:50:46] INDEX=(id=33555434, name=INDEX33555434, table_name=SYSAUDITSQLSEQ)
[2019-10-21 13:50:46] INDEX=(id=33555435, name=SECOND_INDXE_SYS_REWRITE_EQUIVALENCES_X7Q1, table_name=SYS_REWRITE_EQUIVALENCES)
[2019-10-21 13:50:46] INDEX=(id=33555438, name=SYSINDEXPARTTIDSYSHPARTTABLEINFO, table_name=SYSHPARTTABLEINFO)
[2019-10-21 13:50:46] INDEX=(id=33555441, name=INDEX33555441, table_name=POLICY_GROUPS)
[2019-10-21 13:50:46] INDEX=(id=33555442, name=INDEX33555442, table_name=POLICIES)
[2019-10-21 13:50:46] INDEX=(id=33555443, name=INDEX33555443, table_name=POLICY_CONTEXTS)
[2019-10-21 13:50:46] INDEX=(id=33555444, name=INDEX33555444, table_name=DBMS_LOCK_ALLOCATED)
[2019-10-21 13:50:46] INDEX=(id=33555445, name=INDEX33555445, table_name=DBMS_ALERT_INFO)
[2019-10-21 13:50:46] INDEX=(id=33555446, name=INDEX33555446, table_name=ADMIN_ROLE)
[2019-10-21 13:50:46] INDEX=(id=33555447, name=INDEX33555447, table_name=ADMIN_ROLE)
[2019-10-21 13:50:46] INDEX=(id=33555448, name=INDEX33555448, table_name=ADMIN_SYS_CONFIG)
[2019-10-21 13:50:46] INDEX=(id=33555449, name=INDEX33555449, table_name=ADMIN_SYS_CONFIG)
[2019-10-21 13:50:46] INDEX=(id=33555450, name=INDEX33555450, table_name=ADMIN_USER)
[2019-10-21 13:50:46] INDEX=(id=33555451, name=INDEX33555451, table_name=ADMIN_USER)
[2019-10-21 13:50:46] INDEX=(id=33555452, name=INDEX33555452, table_name=ADMIN_AUDIT)
[2019-10-21 13:50:46] INDEX=(id=33555453, name=INDEX33555453, table_name=ADMIN_AUDIT)
[2019-10-21 13:50:46] INDEX=(id=33555454, name=INDEX33555454, table_name=ADMIN_AUDIT_SQL)
[2019-10-21 13:50:46] INDEX=(id=33555455, name=INDEX33555455, table_name=DBTOOL_PREFERENCE)
[2019-10-21 13:50:46] INDEX=(id=33555456, name=INDEX_DBTOOL_PREFERENCE, table_name=DBTOOL_PREFERENCE)
[2019-10-21 13:50:46] INDEX=(id=33555457, name=INDEX33555457, table_name=DBTOOL_SERVER_CONN)
[2019-10-21 13:50:46] INDEX=(id=33555458, name=INDEX33555458, table_name=DBTOOL_SERVER_CONN)
[2019-10-21 13:50:46] INDEX=(id=33555459, name=INDEX_DBTOOL_SERVER_CONN, table_name=DBTOOL_SERVER_CONN)
[2019-10-21 13:50:46] INDEX=(id=33555460, name=INDEX33555460, table_name=DBTOOL_SERVER_CONN)
[2019-10-21 13:50:46] INDEX=(id=33555461, name=INDEX33555461, table_name=DMA_ALERT)
[2019-10-21 13:50:46] INDEX=(id=33555462, name=INDEX33555462, table_name=DMA_ALERT)
[2019-10-21 13:50:46] INDEX=(id=33555463, name=INDEX33555463, table_name=DMA_ALERT_HIS)
[2019-10-21 13:50:46] INDEX=(id=33555464, name=INDEX33555464, table_name=DMA_ALERT_HIS)
[2019-10-21 13:50:46] INDEX=(id=33555465, name=INDEX_DMA_ALERT_HIS_ALERT_ID, table_name=DMA_ALERT_HIS)
[2019-10-21 13:50:46] INDEX=(id=33555466, name=INDEX_DMA_ALERT_HIS_GMT_CREATE, table_name=DMA_ALERT_HIS)
[2019-10-21 13:50:46] INDEX=(id=33555467, name=INDEX_DMA_ALERT_HIS_RES_ID, table_name=DMA_ALERT_HIS)
[2019-10-21 13:50:46] INDEX=(id=33555468, name=INDEX33555468, table_name=DMA_ALERT_HIS_NOTIFY)
[2019-10-21 13:50:46] INDEX=(id=33555469, name=INDEX33555469, table_name=DMA_ALERT_HIS_NOTIFY)
[2019-10-21 13:50:46] INDEX=(id=33555470, name=INDEX_DMA_ALERT_HIS_NOTIFY_USER_ID, table_name=DMA_ALERT_HIS_NOTIFY)
[2019-10-21 13:50:46] INDEX=(id=33555471, name=INDEX33555471, table_name=DMA_ALERT_RES)
[2019-10-21 13:50:46] INDEX=(id=33555472, name=INDEX_DMA_ALERT_RES_ALERT_ID, table_name=DMA_ALERT_RES)
[2019-10-21 13:50:46] INDEX=(id=33555473, name=INDEX_DMA_ALERT_RES_RES_ID, table_name=DMA_ALERT_RES)
[2019-10-21 13:50:46] INDEX=(id=33555474, name=INDEX33555474, table_name=DMA_ALERT_RULE)
[2019-10-21 13:50:46] INDEX=(id=33555475, name=INDEX_DMA_ALERT_RULE_ALERT_ID, table_name=DMA_ALERT_RULE)
[2019-10-21 13:50:46] INDEX=(id=33555476, name=INDEX33555476, table_name=DMA_ALERT_USER)
[2019-10-21 13:50:46] INDEX=(id=33555477, name=INDEX_DMA_ALERT_USER_ALERT_ID, table_name=DMA_ALERT_USER)
[2019-10-21 13:50:46] INDEX=(id=33555478, name=INDEX33555478, table_name=DMA_DATABASE)
[2019-10-21 13:50:46] INDEX=(id=33555479, name=INDEX33555479, table_name=DMA_DATABASE)
[2019-10-21 13:50:46] INDEX=(id=33555480, name=INDEX_DMA_DB_GMT_CREATE, table_name=DMA_DATABASE)
[2019-10-21 13:50:46] INDEX=(id=33555481, name=INDEX33555481, table_name=DMA_DATABASE_DEADLOCK)
[2019-10-21 13:50:46] INDEX=(id=33555482, name=INDEX_DMA_DB_DEADLOCK, table_name=DMA_DATABASE_DEADLOCK)
[2019-10-21 13:50:46] INDEX=(id=33555483, name=INDEX33555483, table_name=DMA_DATABASE_DF)
[2019-10-21 13:50:46] [FAILED] check index error. index=33555483
[2019-10-21 13:50:46] INDEX=(id=33555484, name=INDEX_DMA_DB_DF, table_name=DMA_DATABASE_DF)
[2019-10-21 13:50:46] [FAILED] check index error. index=33555484
[2019-10-21 13:50:46] INDEX=(id=33555485, name=INDEX33555485, table_name=DMA_DATABASE_EVENT)
[2019-10-21 13:50:53] INDEX=(id=33555486, name=INDEX_DMA_DB_EVENT, table_name=DMA_DATABASE_EVENT)
[2019-10-21 13:51:05] [FAILED] check index error. index=33555486
[2019-10-21 13:51:05] INDEX=(id=33555487, name=INDEX_DMA_DB_EVENT_DBID_EVENT, table_name=DMA_DATABASE_EVENT)
[2019-10-21 13:51:16] [FAILED] check index error. index=33555487
[2019-10-21 13:51:16] INDEX=(id=33555488, name=INDEX_DMA_DB_EVENT_TS, table_name=DMA_DATABASE_EVENT)
[2019-10-21 13:51:19] [FAILED] check index error. index=33555488
[2019-10-21 13:51:19] INDEX=(id=33555489, name=INDEX33555489, table_name=DMA_DATABASE_INFO)
[2019-10-21 13:51:19] INDEX=(id=33555490, name=INDEX33555490, table_name=DMA_DATABASE_INFO)
[2019-10-21 13:51:19] INDEX=(id=33555491, name=INDEX33555491, table_name=DMA_DATABASE_SESSION)
[2019-10-21 13:51:48] [FAILED] check index error. index=33555491
[2019-10-21 13:51:48] INDEX=(id=33555492, name=INDEX_DMA_DB_SESSION, table_name=DMA_DATABASE_SESSION)
[2019-10-21 13:51:55] [FAILED] check index error. index=33555492
[2019-10-21 13:51:55] INDEX=(id=33555493, name=INDEX33555493, table_name=DMA_DATABASE_SQL_COUNT)
[2019-10-21 13:51:55] INDEX=(id=33555494, name=INDEX_DMA_DB_SQL_COUNT, table_name=DMA_DATABASE_SQL_COUNT)
[2019-10-21 13:51:56] INDEX=(id=33555495, name=INDEX33555495, table_name=DMA_DATABASE_SQL_COUNT_SQL_TEXT)
[2019-10-21 13:51:56] INDEX=(id=33555496, name=INDEX33555496, table_name=DMA_DATABASE_SQL_COUNT_SQL_TEXT)
[2019-10-21 13:51:56] INDEX=(id=33555497, name=INDEX_DMA_DB_SQL_COUNT_SQL_TEXT, table_name=DMA_DATABASE_SQL_COUNT_SQL_TEXT)
[2019-10-21 13:51:56] INDEX=(id=33555498, name=INDEX33555498, table_name=DMA_DATABASE_SQL_ERR)
[2019-10-21 13:51:56] INDEX=(id=33555499, name=INDEX_DMA_DB_SQL_ERR, table_name=DMA_DATABASE_SQL_ERR)
[2019-10-21 13:51:56] INDEX=(id=33555500, name=INDEX33555500, table_name=DMA_DATABASE_SQL_SLOW)
[2019-10-21 13:51:56] INDEX=(id=33555501, name=INDEX_DMA_DB_SQL_SLOW, table_name=DMA_DATABASE_SQL_SLOW)
[2019-10-21 13:51:56] INDEX=(id=33555502, name=INDEX33555502, table_name=DMA_DATABASE_STAT)
[2019-10-21 13:51:59] INDEX=(id=33555503, name=INDEX_DMA_DB_STAT, table_name=DMA_DATABASE_STAT)
[2019-10-21 13:52:00] [FAILED] check index error. index=33555503
[2019-10-21 13:52:00] INDEX=(id=33555504, name=INDEX_DMA_DB_STAT_TS, table_name=DMA_DATABASE_STAT)
[2019-10-21 13:52:02] INDEX=(id=33555505, name=INDEX33555505, table_name=DMA_DATABASE_ARCH)
[2019-10-21 13:52:03] [FAILED] check index error. index=33555505
[2019-10-21 13:52:03] INDEX=(id=33555506, name=INDEX_DMA_DB_ARCH, table_name=DMA_DATABASE_ARCH)
[2019-10-21 13:52:05] [FAILED] check index error. index=33555506
[2019-10-21 13:52:05] INDEX=(id=33555507, name=INDEX33555507, table_name=DMA_DATABASE_TABLE)
[2019-10-21 13:52:05] INDEX=(id=33555508, name=INDEX33555508, table_name=DMA_DATABASE_TABLE)
[2019-10-21 13:52:05] INDEX=(id=33555509, name=INDEX33555509, table_name=DMA_DATABASE_TABLE_STAT)
[2019-10-21 13:52:05] INDEX=(id=33555510, name=INDEX_DMA_DATABASE_TABLE_STAT, table_name=DMA_DATABASE_TABLE_STAT)
[2019-10-21 13:52:05] INDEX=(id=33555511, name=INDEX33555511, table_name=DMA_DATABASE_TS)
[2019-10-21 13:52:14] [FAILED] check index error. index=33555511
[2019-10-21 13:52:14] INDEX=(id=33555512, name=INDEX_DMA_DB_TS, table_name=DMA_DATABASE_TS)
[2019-10-21 13:52:15] [FAILED] check index error. index=33555512
[2019-10-21 13:52:15] INDEX=(id=33555513, name=INDEX33555513, table_name=DMA_DATABASE_UD_SQL)
[2019-10-21 13:52:15] INDEX=(id=33555514, name=INDEX33555514, table_name=DMA_DATABASE_UD_SQL_STAT)
[2019-10-21 13:52:15] INDEX=(id=33555515, name=INDEX_DMA_DATABASE_UD_SQL_STAT, table_name=DMA_DATABASE_UD_SQL_STAT)
[2019-10-21 13:52:15] INDEX=(id=33555516, name=INDEX33555516, table_name=DMA_DATABASE_USER)
[2019-10-21 13:52:15] INDEX=(id=33555517, name=INDEX_DMA_DB_USER, table_name=DMA_DATABASE_USER)
[2019-10-21 13:52:15] INDEX=(id=33555518, name=INDEX33555518, table_name=DMA_GROUP)
[2019-10-21 13:52:15] INDEX=(id=33555519, name=INDEX33555519, table_name=DMA_GROUP)
[2019-10-21 13:52:15] INDEX=(id=33555520, name=INDEX33555520, table_name=DMA_GROUP_TYPE)
[2019-10-21 13:52:15] INDEX=(id=33555521, name=INDEX33555521, table_name=DMA_GROUP_TYPE)
[2019-10-21 13:52:15] INDEX=(id=33555522, name=INDEX33555522, table_name=DMA_MAINFRAME)
[2019-10-21 13:52:15] INDEX=(id=33555523, name=INDEX33555523, table_name=DMA_MAINFRAME)
[2019-10-21 13:52:15] INDEX=(id=33555524, name=INDEX_DMA_MF_GMT_CREATE, table_name=DMA_MAINFRAME)
[2019-10-21 13:52:15] INDEX=(id=33555525, name=INDEX33555525, table_name=DMA_MAINFRAME_DISK)
[2019-10-21 13:52:16] INDEX=(id=33555526, name=INDEX_DMA_MF_DISK, table_name=DMA_MAINFRAME_DISK)
[2019-10-21 13:52:18] INDEX=(id=33555527, name=INDEX33555527, table_name=DMA_MAINFRAME_STAT)
[2019-10-21 13:52:36] INDEX=(id=33555528, name=INDEX_DMA_MF_STAT, table_name=DMA_MAINFRAME_STAT)
[2019-10-21 13:52:38] [FAILED] check index error. index=33555528
[2019-10-21 13:52:38] INDEX=(id=33555529, name=INDEX_DMA_MF_STAT_TS, table_name=DMA_MAINFRAME_STAT)
[2019-10-21 13:52:44] INDEX=(id=33555530, name=INDEX33555530, table_name=DMA_MAINFRAME_EXEC)
[2019-10-21 13:52:44] INDEX=(id=33555531, name=IDX_ID, table_name=DMA_MAINFRAME_EXEC)
[2019-10-21 13:52:44] INDEX=(id=33555532, name=IDX_MF_ID, table_name=DMA_MAINFRAME_EXEC)
[2019-10-21 13:52:44] INDEX=(id=33555533, name=INDEX33555533, table_name=DMA_MAINFRAME_EXEC_STAT)
[2019-10-21 13:52:44] INDEX=(id=33555534, name=IDX_TS, table_name=DMA_MAINFRAME_EXEC_STAT)
[2019-10-21 13:52:44] INDEX=(id=33555535, name=IDX_EXEC_ID, table_name=DMA_MAINFRAME_EXEC_STAT)
[2019-10-21 13:52:44] INDEX=(id=33555536, name=INDEX33555536, table_name=DMA_MULTI_CONN_SQL_TEMPLATE)
[2019-10-21 13:52:44] INDEX=(id=33555537, name=INDEX33555537, table_name=DMA_MULTI_CONN_SQL_TEMPLATE)
[2019-10-21 13:52:44] INDEX=(id=33555538, name=INDEX33555538, table_name=DMA_MULTI_CONN_SQL_TEMPLATE_DB)
[2019-10-21 13:52:44] INDEX=(id=33555539, name=INDEX33555539, table_name=DMA_MULTI_CONN_SQL_TEMPLATE_DB)
[2019-10-21 13:52:44] INDEX=(id=33555541, name=INDEX33555541, table_name=DMA_RULE_TEMPL)
[2019-10-21 13:52:44] INDEX=(id=33555542, name=INDEX33555542, table_name=DMA_RULE_TEMPL)
[2019-10-21 13:52:44] INDEX=(id=33555561, name=INDEX33555561, table_name=CNDDBA)
[2019-10-21 13:52:44] INDEX=(id=33555563, name=INDEX33555563, table_name=SYSMAILINFO)
[2019-10-21 13:52:44] INDEX=(id=33555564, name=INDEX33555564, table_name=SYSJOBHISTORIES)
[2019-10-21 13:52:44] INDEX=(id=33555565, name=INDEX33555565, table_name=SYSJOBHISTORIES2)
[2019-10-21 13:52:44] INDEX=(id=33555566, name=INDEX33555566, table_name=SYSJOBHISTORIES2)
[2019-10-21 13:52:44] INDEX=(id=33555567, name=INDEX33555567, table_name=SYSSTEPHISTORIES2)
[2019-10-21 13:52:44] INDEX=(id=33555568, name=INDEX33555568, table_name=SYSALERTHISTORIES)
[2019-10-21 13:52:44] INDEX=(id=33555569, name=INDEX33555569, table_name=SYSOPERATORS)
[2019-10-21 13:52:44] INDEX=(id=33555570, name=INDEX33555570, table_name=SYSJOBS)
[2019-10-21 13:52:44] INDEX=(id=33555571, name=INDEX33555571, table_name=SYSJOBS)
[2019-10-21 13:52:44] INDEX=(id=33555572, name=INDEX33555572, table_name=SYSJOBSTEPS)
[2019-10-21 13:52:44] INDEX=(id=33555573, name=INDEX33555573, table_name=SYSJOBSCHEDULES)
[2019-10-21 13:52:44] INDEX=(id=33555574, name=INDEX33555574, table_name=SYSALERTS)
[2019-10-21 13:52:44] INDEX=(id=33555575, name=INDEX33555575, table_name=SYSALERTNOTIFICATIONS)
[2019-10-21 13:52:44] INDEX=(id=33555576, name=INDEX33555576, table_name=CNDBA)
[2019-10-21 13:52:44] INDEX=(id=33555577, name=INDEX33555577, table_name=SALES)
[2019-10-21 13:52:44] INDEX=(id=33555578, name=INDEX33555578_33555577, table_name=SALES_P1)
[2019-10-21 13:52:44] INDEX=(id=33555579, name=INDEX33555579_33555577, table_name=SALES_P2)
[2019-10-21 13:52:44] INDEX=(id=33555580, name=INDEX33555580_33555577, table_name=SALES_P3)
[2019-10-21 13:52:44] INDEX=(id=33555581, name=INDEX33555581_33555577, table_name=SALES_P_DEFAULT)
[2019-10-21 13:52:44] INDEX=(id=33555626, name=IDX_ID, table_name=CNDBA)
[2019-10-21 13:52:44] INDEX=(id=33555627, name=INDEX33555627, table_name=WRM$_SNAPSHOT)
[2019-10-21 13:52:44] INDEX=(id=33555628, name=INDEX33555628, table_name=WRM$_WR_CONTROL)
[2019-10-21 13:52:44] INDEX=(id=33555629, name=INDEX33555629, table_name=WRH$_SYSSTAT)
[2019-10-21 13:52:44] INDEX=(id=33555630, name=INDEX33555630, table_name=WRH$_SYSTEM_EVENT)
[2019-10-21 13:52:44] INDEX=(id=33555631, name=INDEX33555631, table_name=WRH$_SYSTEMINFO)
[2019-10-21 13:52:44] INDEX=(id=33555632, name=INDEX33555632, table_name=WRH$_BUFFERPOOL)
[2019-10-21 13:52:44] INDEX=(id=33555633, name=INDEX33555633, table_name=WRH$_DATAFILE)
[2019-10-21 13:52:44] INDEX=(id=33555634, name=INDEX33555634, table_name=WRH$_TABLESPACE)
[2019-10-21 13:52:44] INDEX=(id=33555635, name=INDEX33555635, table_name=WRH$_CKPT_HISTORY)
[2019-10-21 13:52:44] INDEX=(id=33555636, name=INDEX33555636, table_name=WRH$_HASH_MERGE_USED_HISTORY)
[2019-10-21 13:52:44] INDEX=(id=33555637, name=INDEX33555637, table_name=WRH$_MTAB_USED_HISTORY)
[2019-10-21 13:52:44] INDEX=(id=33555638, name=INDEX33555638, table_name=WRH$_WAIT_HISTORY)
[2019-10-21 13:52:44] INDEX=(id=33555639, name=INDEX33555639, table_name=WRH$_PSEG_ITEMS)
[2019-10-21 13:52:44] INDEX=(id=33555640, name=INDEX33555640, table_name=WRH$_DEADLOCK_HISTORY)
[2019-10-21 13:52:44] INDEX=(id=33555641, name=INDEX33555641, table_name=WRH$_DB_CACHE)
[2019-10-21 13:52:44] INDEX=(id=33555642, name=INDEX33555642, table_name=WRH$_VPOOL)
[2019-10-21 13:52:44] INDEX=(id=33555643, name=INDEX33555643, table_name=WRH$_DM_INI)
[2019-10-21 13:52:45] INDEX=(id=33555644, name=INDEX33555644, table_name=WRH$_RESOURCE_LIMIT)
[2019-10-21 13:52:45] INDEX=(id=33555645, name=INDEX33555645, table_name=WRH$_SQL_HISTORY)
[2019-10-21 13:52:45] INDEX=(id=33555646, name=INDEX33555646, table_name=WRH$_SQLTEXT)
[2019-10-21 13:52:45] INDEX=(id=33555647, name=INDEX33555647, table_name=WRH$_SESSIONS)
[2019-10-21 13:52:45] INDEX=(id=33555648, name=INDEX33555648, table_name=WRH$_SESSION_WAIT_HISTORY)
[2019-10-21 13:52:45] --------check indexes end-----------------
[2019-10-21 13:52:45] --------check iid start-------------------
[2019-10-21 13:52:45] check cons id ...
[2019-10-21 13:52:45] check index id ...
[2019-10-21 13:52:45] check table id ...
[2019-10-21 13:52:45] check proc id ...
[2019-10-21 13:52:45] check schema id ...
[2019-10-21 13:52:45] check synonym id ...
[2019-10-21 13:52:45] check user id ...
[2019-10-21 13:52:45] --------check iid end---------------------
[2019-10-21 13:52:45] DM DB CHECK END......
[2019-10-21 13:52:45] error count is 13
[dave@www.cndba.cn dave]$
这里check 失败的都是DEM用户的索引：
SQL> select owner,object_name,object_type,status from all_objects where object_id='33555483';
LINEID OWNER OBJECT_NAME OBJECT_TYPE STATUS
---------- ----- ------------- ----------- ------
1 DEM INDEX33555483 INDEX VALID
used time: 17.408(ms). Execute id is 30.
SQL> select owner,object_name,object_type,status from all_objects where object_id='33555486';
LINEID OWNER OBJECT_NAME OBJECT_TYPE STATUS
---------- ----- ------------------ ----------- ------
1 DEM INDEX_DMA_DB_EVENT INDEX VALID
used time: 18.100(ms). Execute id is 31.
SQL>
查看日志：
2019-10-21 13:52:45 [INFO] DB check P0000002655 T0000139702219872032 Database file: /dm/dmdbms/data/dave/SYSTEM.DBF's size verification fails, it may be damaged.
2019-10-21 13:52:45 [INFO] DB check P0000002655 T0000139702219872032 Database file: /dm/dmdbms/data/dave/ROLL.DBF's size verification fails, it may be damaged.
2019-10-21 13:52:45 [INFO] DB check P0000002655 T0000139702219872032 Database file: /dm/dmdbms/data/dave/dave01.log's size verification fails, it may be damaged.
2019-10-21 13:52:45 [INFO] DB check P0000002655 T0000139702219872032 Database file: /dm/dmdbms/data/dave/dave02.log's size verification fails, it may be damaged.
2019-10-21 13:52:45 [INFO] DB check P0000002655 T0000139702219872032 Database file: /dm/dmdbms/data/dave/MAIN.DBF's size verification fails, it may be damaged.
2019-10-21 13:52:45 [INFO] DB check P0000002655 T0000139702219872032 Database file: /dm/dmdbms/data/dave/dave01.dbf's size verification fails, it may be damaged.
2019-10-21 13:52:45 [INFO] DB check P0000002655 T0000139702219872032 Database file: /dm/dmdbms/data/dave/dave02.dbf's size verification fails, it may be damaged.
2019-10-21 13:52:45 [INFO] DB check P0000002655 T0000139702219872032 Database file: /dm/dmdbms/data/dave/SYSAWR.DBF's size verification fails, it may be damaged.
此时DB 还是运行正常的。
3 进行其他非select 操作
对于select 操作，db实例不会崩溃，一旦执行其他操作，立即崩溃：
2019-10-21 14:10:20 [WARNING] database P0000002699 main_thread Failure occurs in data_recv_inet_once, code 6
2019-10-21 14:11:07 [FATAL] database P0000002699 main_thread Fatal error: desc is full while try alloc new page
2019-10-21 14:11:07 [FATAL] database P0000002699 main_thread dm_sys_halt now!!!
2019-10-21 14:11:07 [INFO] database P0000002699 main_thread total 2 rfil opened!
错误信息依旧，此时在重启实例无效：
[dave@www.cndba.cn dave]$ /etc/init.d/DmServicedave start
Starting DmServicedave: [ OK ]
[dave@www.cndba.cn dave]$
必须重启OS后才可以启动，但还是仅限查询操作。 涉及到其他操作就会直接导致实例崩溃。
因为开始dmdbchk 测试的是DEM 模式的对象异常，所以尝试重建该模式：
SQL> drop schema dem cascade;
executed successfully
used time: 156.677(ms). Execute id is 12.
SQL>
SQL> set define off
SQL> set char_code utf8
SQL> start /dm/dmdbms/web/dem_init.sql
……
结果启动库又有其他错误：
2019-10-21 15:59:12 [INFO] database P0000003162 main_thread total redo 37576 ptx, 5363 page, redo percent: 100%, last redo lsn:43032027
2019-10-21 15:59:12 [INFO] database P0000003162 main_thread Apply redo log ok, 37576 ptx, begin_lsn: 43012561, end_lsn: 43032027
2019-10-21 15:59:12 [INFO] database P0000003162 main_thread redo log recover finished
2019-10-21 15:59:12 [INFO] database P0000003162 main_thread total redo 217498 rrec, 36041 rrec opt by pwr
2019-10-21 15:59:12 [INFO] database P0000003162 main_thread redo rrec used 196ms
2019-10-21 15:59:12 [INFO] database P0000003162 main_thread ndct db load finished
2019-10-21 15:59:12 [INFO] database P0000003162 main_thread ndct fill fast pool finished
2019-10-21 15:59:12 [INFO] database P0000003162 main_thread EP[0] pseg_collect_items end, collect 57 trx, 64 committed pages, 0 active pages
2019-10-21 15:59:12 [INFO] database P0000003162 main_thread EP[0] undo over, total 0 page rollbacked
2019-10-21 15:59:12 [FATAL] database P0000003162 main_thread flst corrupt
2019-10-21 15:59:12 [FATAL] database P0000003162 main_thread dm_sys_halt now!!!
2019-10-21 15:59:12 [INFO] database P0000003162 main_thread total 4 rfil opened!
不过这里的提示信息已经变成了：main_thread flst corrupt
4 最终解决方法
因为是一个测试库，没有进行任何备份，从以上的分析来看，应该是数据文件有损坏，最终只能重新初始化了一个库，重建了DEM，恢复了DEM的运行。
DM7 达梦 数据库 DEM(Dameng Enterprise Manager) 安装部署说明
https://www.cndba.cn/dave/article/3707
DM7 达梦数据库 通过dminit 创建 并 注册 数据库实例
https://www.cndba.cn/dave/article/3580
5 小结
中标麒麟在vbox 虚拟机中运行确实不太友好，从目前的使用经验看，只要是异常断电的，必然会导致系统坏块，必须手动修复后才能启动。 本次故障就是在手工修复中标麒麟操作系统出现的故障。 所以从某种角度来说，这里的问题更多的还是操作系统的问题。 当然达梦在这里的错误提示上，也不是非常清晰。 这点也是有可提升的空间。
版权声明：本文为博主原创文章，未经博主允许不得转载。
