CNDBA社区
Greenplum failed segment的恢复
刚刚接到客户消息说服务器环境Greenplum连接不上,远程查看报错疑似防火墙问题，但是防火墙都关闭的，后来检查集群状态发现，数据节点Segment是宕机，存在问题，并且有部分的Mirror与Primary交换了，出现这种情况，一般是由于在使用greenplum过程中有不当的操作时，可能会出现segment节点宕掉的情况（比如在greenplum运行的过程中停掉其中几台segment节点的服务器）
通过下面的方法可以恢复segment。
1、查看Segment节点状态
[gpadmin@server-6 ~]$ gpstate -m
......中间输出省略......
当时发现4个节点Failed，有2个节点的Primary和Mirror交换了。
2、停掉GP
[gpadmin@server-6 ~]$ gpstop -M fast -a 这样会告诉你有几个节点DOWN了
3、启动GP
[gpadmin@server-6 ~]$ gpstart 启动数据库会忽略DOWN的节点
注意：下面输出WARNING信息，可以明确知道跳过忽略了哪些DWON的Segment节点，总共Number of segments not attempted to start: 9
20181030:18:41:46:008069 gpstart:server-6:gpadmin-[INFO]:-Starting gpstart with args:
20181030:18:41:46:008069 gpstart:server-6:gpadmin-[INFO]:-Gathering information and validating the environment...
20181030:18:41:46:008069 gpstart:server-6:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.7.3 build 2'
20181030:18:41:46:008069 gpstart:server-6:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150'
20181030:18:41:46:008069 gpstart:server-6:gpadmin-[INFO]:-Starting Master instance in admin mode
20181030:18:41:48:008069 gpstart:server-6:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20181030:18:41:48:008069 gpstart:server-6:gpadmin-[INFO]:-Obtaining Segment details from master...
20181030:18:41:48:008069 gpstart:server-6:gpadmin-[INFO]:-Setting new master era
20181030:18:41:48:008069 gpstart:server-6:gpadmin-[INFO]:-Master Started...
20181030:18:41:48:008069 gpstart:server-6:gpadmin-[INFO]:-Shutting down master
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on server-6 directory /home/gpadmin/data1/primary/gpseg3 <<<<<
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on server-8 directory /home/gpadmin/data1/mirror/gpseg8 <<<<<
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on server-8 directory /home/gpadmin/data1/mirror/gpseg9 <<<<<
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on server-8 directory /home/gpadmin/data1/mirror/gpseg10 <<<<<
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on server-8 directory /home/gpadmin/data1/mirror/gpseg11 <<<<<
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on server-8 directory /home/gpadmin/data1/primary/gpseg12 <<<<<
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on server-8 directory /home/gpadmin/data1/primary/gpseg13 <<<<<
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on server-8 directory /home/gpadmin/data1/primary/gpseg14 <<<<<
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on server-8 directory /home/gpadmin/data1/primary/gpseg15 <<<<<
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:---------------------------
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:-Master instance parameters
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:---------------------------
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:-Database = template1
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:-Master Port = 5432
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:-Master directory = /home/gpadmin/data1/master/gpseg-1
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:-Timeout = 600 seconds
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:-Master standby start = On
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:---------------------------------------
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:-Segment instances that will be started
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:---------------------------------------
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- Host Datadir Port Role
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/primary/gpseg0 40000 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/mirror/gpseg0 50000 Mirror
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/primary/gpseg1 40001 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/mirror/gpseg1 50001 Mirror
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/primary/gpseg2 40002 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/mirror/gpseg2 50002 Mirror
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/mirror/gpseg3 50003 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/primary/gpseg4 40000 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/mirror/gpseg4 50000 Mirror
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/primary/gpseg5 40001 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/mirror/gpseg5 50001 Mirror
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/primary/gpseg6 40002 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/mirror/gpseg6 50002 Mirror
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/primary/gpseg7 40003 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/mirror/gpseg7 50003 Mirror
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/primary/gpseg8 40000 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/primary/gpseg9 40001 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/primary/gpseg10 40002 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/primary/gpseg11 40003 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/mirror/gpseg12 50000 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/mirror/gpseg13 50001 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/mirror/gpseg14 50002 Primary
20181030:18:41:50:008069 gpstart:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/mirror/gpseg15 50003 Primary
Continue with Greenplum instance startup Yy|Nn (default=N):
> y
20181030:18:41:54:008069 gpstart:server-6:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
.................
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[INFO]:-Process results...
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[INFO]:-----------------------------------------------------
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[INFO]:- Successful segment starts = 23
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[INFO]:- Failed segment starts = 0
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration) = 9 <<<<<<<<
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[INFO]:-----------------------------------------------------
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[INFO]:-
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[INFO]:-Successfully started 23 of 23 segment instances, skipped 9 other segments
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[INFO]:-----------------------------------------------------
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[WARNING]:-****************************************************************************
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[WARNING]:-There are 9 segment(s) marked down in the database
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases.
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[WARNING]:-****************************************************************************
20181030:18:42:11:008069 gpstart:server-6:gpadmin-[INFO]:-Starting Master instance server-6 directory /home/gpadmin/data1/master/gpseg-1
20181030:18:42:12:008069 gpstart:server-6:gpadmin-[INFO]:-Command pg_ctl reports Master server-6 instance active
20181030:18:42:13:008069 gpstart:server-6:gpadmin-[INFO]:-Starting standby master
20181030:18:42:13:008069 gpstart:server-6:gpadmin-[INFO]:-Checking if standby master is running on host: server-7 in directory: /home/gpadmin/data1/master/gpseg-1
20181030:18:42:16:008069 gpstart:server-6:gpadmin-[WARNING]:-Number of segments not attempted to start: 9
20181030:18:42:16:008069 gpstart:server-6:gpadmin-[INFO]:-Check status of database with gpstate utility
4、生成一个恢复配置文件
[gpadmin@server-6 ~]$ gprecoverseg -o ./recov 会在当前目录生成一个recov文件，里面包含了要恢复的节点信息
20181030:18:42:37:008482 gprecoverseg:server-6:gpadmin-[INFO]:-Starting gprecoverseg with args: -o ./recov
20181030:18:42:37:008482 gprecoverseg:server-6:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.7.3 build 2'
20181030:18:42:37:008482 gprecoverseg:server-6:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.7.3 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Feb 25 2016 06:38:30'
20181030:18:42:37:008482 gprecoverseg:server-6:gpadmin-[INFO]:-Checking if segments are ready
20181030:18:42:37:008482 gprecoverseg:server-6:gpadmin-[INFO]:-Obtaining Segment details from master...
20181030:18:42:37:008482 gprecoverseg:server-6:gpadmin-[INFO]:-Obtaining Segment details from master...
20181030:18:42:38:008482 gprecoverseg:server-6:gpadmin-[INFO]:-Configuration file output to ./recov successfully.
recov文件内容如下：（注意：这个文件不是手动创建的，而是通过gprecoverseg -o ./recov命令生成的
[gpadmin@server-6 ~]$ cat ./recov
filespaceOrder=
server-6:40003:/home/gpadmin/data1/primary/gpseg3
server-8:50000:/home/gpadmin/data1/mirror/gpseg8
server-8:50001:/home/gpadmin/data1/mirror/gpseg9
server-8:50002:/home/gpadmin/data1/mirror/gpseg10
server-8:50003:/home/gpadmin/data1/mirror/gpseg11
server-8:40000:/home/gpadmin/data1/primary/gpseg12
server-8:40001:/home/gpadmin/data1/primary/gpseg13
server-8:40002:/home/gpadmin/data1/primary/gpseg14
server-8:40003:/home/gpadmin/data1/primary/gpseg15
5、使用恢复配置文件恢复节点
gpadmin@server-6 ~]$ gprecoverseg -i ./recov
...........中间输出省略...........................
Continue with segment recovery procedure Yy|Nn (default=N):
> y
20181030:18:43:21:008941 gprecoverseg:server-6:gpadmin-[INFO]:-9 segment(s) to recover
20181030:18:43:21:008941 gprecoverseg:server-6:gpadmin-[INFO]:-Ensuring 9 failed segment(s) are stopped
20181030:18:44:24:008941 gprecoverseg:server-6:gpadmin-[INFO]:-Process results...
20181030:18:44:24:008941 gprecoverseg:server-6:gpadmin-[INFO]:-Done updating primaries
20181030:18:44:24:008941 gprecoverseg:server-6:gpadmin-[INFO]:-******************************************************************
20181030:18:44:24:008941 gprecoverseg:server-6:gpadmin-[INFO]:-Updating segments for resynchronization is completed.
20181030:18:44:24:008941 gprecoverseg:server-6:gpadmin-[INFO]:-For segments updated successfully, resynchronization will continue in the background.
20181030:18:44:24:008941 gprecoverseg:server-6:gpadmin-[INFO]:-
20181030:18:44:24:008941 gprecoverseg:server-6:gpadmin-[INFO]:-Use gpstate -s to check the resynchronization progress.
20181030:18:44:24:008941 gprecoverseg:server-6:gpadmin-[INFO]:-******************************************************************
恢复过程中可以用gpstate -m 查看恢复状态：Resynchronizing（表示正在恢复中），Synchronized（表示恢复完毕）
[gpadmin@server-6 ~]$ gpstate -m
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:-Starting gpstate with args: -m
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.7.3 build 2'
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.7.3 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Feb 25 2016 06:38:30'
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:-Obtaining Segment details from master...
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:--------------------------------------------------------------
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:--Current GPDB mirror list and status
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:--Type = Group
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:--------------------------------------------------------------
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/mirror/gpseg0 50000 Passive Synchronized
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/mirror/gpseg1 50001 Passive Synchronized
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/mirror/gpseg2 50002 Passive Synchronized
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-5 /home/gpadmin/data1/mirror/gpseg3 50003 Acting as Primary Synchronized
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/mirror/gpseg4 50000 Passive Synchronized
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/mirror/gpseg5 50001 Passive Synchronized
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/mirror/gpseg6 50002 Passive Synchronized
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-7 /home/gpadmin/data1/mirror/gpseg7 50003 Passive Synchronized
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-8 /home/gpadmin/data1/mirror/gpseg8 50000 Passive Resynchronizing
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-8 /home/gpadmin/data1/mirror/gpseg9 50001 Passive Resynchronizing
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-8 /home/gpadmin/data1/mirror/gpseg10 50002 Passive Resynchronizing
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-8 /home/gpadmin/data1/mirror/gpseg11 50003 Passive Resynchronizing
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/mirror/gpseg12 50000 Acting as Primary Resynchronizing
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/mirror/gpseg13 50001 Acting as Primary Resynchronizing
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/mirror/gpseg14 50002 Acting as Primary Resynchronizing
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:- server-6 /home/gpadmin/data1/mirror/gpseg15 50003 Acting as Primary Resynchronizing
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[INFO]:--------------------------------------------------------------
20181030:18:47:40:010549 gpstate:server-6:gpadmin-[WARNING]:-5 segment(s) configured as mirror(s) are acting as primaries
6、调整Primary和Mirror
上面的情况中有Primary和Mirror兑换的情况，所以需要把他们换回来，可以用下面的命令，但是注意需要等到上面的配置文件恢复节点恢复完成才可执行，否则报错不能交换
[gpadmin@server-6 ~]$ gprecoverseg -r
20181030:18:59:47:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Starting gprecoverseg with args: -r
20181030:18:59:47:011140 gprecoverseg:server-6:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.7.3 build 2'
20181030:18:59:47:011140 gprecoverseg:server-6:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.7.3 build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Feb 25 2016 06:38:30'
20181030:18:59:47:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Checking if segments are ready
20181030:18:59:47:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Obtaining Segment details from master...
20181030:18:59:48:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Obtaining Segment details from master...
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Greenplum instance recovery parameters
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Recovery type = Rebalance
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Unbalanced segment 1 of 10
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance host = server-5
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance address = server-5
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance directory = /home/gpadmin/data1/mirror/gpseg3
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance port = 50003
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance replication port = 51003
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Balanced role = Mirror
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Current role = Primary
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Unbalanced segment 2 of 10
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance host = server-6
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance address = server-6
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance directory = /home/gpadmin/data1/primary/gpseg3
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance port = 40003
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance replication port = 41003
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Balanced role = Primary
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Current role = Mirror
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Unbalanced segment 3 of 10
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance host = server-6
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance address = server-6
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance directory = /home/gpadmin/data1/mirror/gpseg12
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance port = 50000
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance replication port = 51000
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Balanced role = Mirror
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Current role = Primary
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Unbalanced segment 4 of 10
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance host = server-8
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance address = server-8
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance directory = /home/gpadmin/data1/primary/gpseg12
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance port = 40000
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance replication port = 41000
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Balanced role = Primary
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Current role = Mirror
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Unbalanced segment 5 of 10
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance host = server-6
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance address = server-6
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance directory = /home/gpadmin/data1/mirror/gpseg13
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance port = 50001
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance replication port = 51001
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Balanced role = Mirror
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Current role = Primary
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Unbalanced segment 6 of 10
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance host = server-8
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance address = server-8
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance directory = /home/gpadmin/data1/primary/gpseg13
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance port = 40001
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance replication port = 41001
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Balanced role = Primary
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Current role = Mirror
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Unbalanced segment 7 of 10
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance host = server-6
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance address = server-6
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance directory = /home/gpadmin/data1/mirror/gpseg14
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance port = 50002
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance replication port = 51002
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Balanced role = Mirror
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Current role = Primary
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Unbalanced segment 8 of 10
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance host = server-8
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance address = server-8
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance directory = /home/gpadmin/data1/primary/gpseg14
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance port = 40002
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance replication port = 41002
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Balanced role = Primary
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Current role = Mirror
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Unbalanced segment 9 of 10
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance host = server-6
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance address = server-6
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance directory = /home/gpadmin/data1/mirror/gpseg15
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance port = 50003
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance replication port = 51003
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Balanced role = Mirror
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Current role = Primary
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Unbalanced segment 10 of 10
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance host = server-8
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance address = server-8
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance directory = /home/gpadmin/data1/primary/gpseg15
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance port = 40003
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Unbalanced instance replication port = 41003
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Balanced role = Primary
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:- Current role = Mirror
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[INFO]:----------------------------------------------------------
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[WARNING]:-This operation will cancel queries that are currently executing.
20181030:18:59:49:011140 gprecoverseg:server-6:gpadmin-[WARNING]:-Connections to the database however will not be interrupted.
Continue with segment rebalance procedure Yy|Nn (default=N):
> y
20181030:18:59:55:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Getting unbalanced segments
20181030:18:59:55:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Stopping unbalanced primary segments...
...
20181030:18:59:58:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Triggering segment reconfiguration
20181030:19:00:02:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Starting segment synchronization
................
20181030:19:00:18:011140 gprecoverseg:server-6:gpadmin-[INFO]:-******************************************************************
20181030:19:00:18:011140 gprecoverseg:server-6:gpadmin-[INFO]:-The rebalance operation has completed successfully.
20181030:19:00:18:011140 gprecoverseg:server-6:gpadmin-[INFO]:-There is a resynchronization running in the background to bring all
20181030:19:00:18:011140 gprecoverseg:server-6:gpadmin-[INFO]:-segments in sync.
20181030:19:00:18:011140 gprecoverseg:server-6:gpadmin-[INFO]:-
20181030:19:00:18:011140 gprecoverseg:server-6:gpadmin-[INFO]:-Use gpstate -e to check the resynchronization progress.
20181030:19:00:18:011140 gprecoverseg:server-6:gpadmin-[INFO]:-******************************************************************
等待上面的恢复完成后，再进行Primary与Mirror交换
[gpadmin@server-6 ~]$ gprecoverseg -r
最后执行该步骤后，Segment也需要执行同步恢复，具体进程可用gpstate -m查看，等待所有的节点都是Synchronized后，过一段时间一般几分钟即可，根据数据量大小而定，一般很很快同步完毕，segment就恢复好了，
[gpadmin@server-6 ~]$ gpstate -m
版权声明：本文为博主原创文章，未经博主允许不得转载。
