TDSQL 集群赤兔平台手工备份失败解决方法 -- cnDBA.cn

1 错误说明

在赤兔平台中执行手工备份，报如下错误：
http://www.cndba.cn/dave/article/4639http://www.cndba.cn/dave/article/4639

这里提示我们查看mysqlagent 的日志：

Error occurred,see mysqlagent log for detail.

2 分析过程

2.1 mysqlagent 日志说明

这里要注意一点，这里提示要查看的日志，实际上是分片中，备注有：备份节点的 mysqlagent 日志。这个一定不要看错了，否则就不能发现有价值的信息。

这里要查看2个日志：

# cd /data/tdsql_run/4001/mysqlagent/log/
# vim sys_report_4001.log……
# cd /data/tdsql_run/4001/mysqlagent/log/nohup
# vim coldbackupimage_400……

2.2 检查HDFS 运行情况

这里有两点要注意：

确保HDFS 进程正常运行。
确保备份路径是tdsql。如果是root，也会导致备份失败。

TDSQL 集群 HDFS 启动报错 Connection refused / NameNode is not formatted 解决方法
https://www.cndba.cn/dave/article/4619

3 解决问题

3.1 查看日志

我们这里的备份节点是63. 所以查看mysqlagent 日志。http://www.cndba.cn/dave/article/4639

[dave@www.cndba.cn 3 nohup]# tail -100 coldbackupimage_4002_2021-07-31
210731 15:06:13 [02] Streaming ./mysql/time_zone.ibd
210731 15:06:13 [02]        ...done
210731 15:06:13 [04]        ...done
210731 15:06:14 >> log scanned up to (8022294)
210731 15:06:15 >> log scanned up to (8210831)
210731 15:06:16 >> log scanned up to (8297890)
put: Cannot create file/tdsqlbackup/tdsqlzk/autocoldbackup/sets/set_1627394115_18/xtrabackup/xtrabackup+1627715173+20210731+150613+192.168.31.63+4002+524255906+xbstream.lz4._COPYING_. Name node is in safe mode.
innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
……
Can't open shared library '/data1/home/harlylei/txsql/percona-xtrabackup-2.4.15/mysql_install/percona-server-5.7.17/lib/plugin/keyring_file.so' (errno: 2, cannot open shared object file: No such file or directory)
210731 15:07:24 >> log scanned up to (12220842)
xtrabackup: Generating a list of tablespaces
InnoDB: Allocated tablespace ID 23 for sys/sys_config, old maximum was 0
……
put: Cannot create file/tdsqlbackup/tdsqlzk/autocoldbackup/sets/set_1627394115_18/xtrabackup/xtrabackup+1627715243+20210731+150723+192.168.31.63+4002+524255906+xbstream.lz4._COPYING_. Name node is in safe mode.
innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[02] xtrabackup: Error: failed to copy datafile.
innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[03] xtrabackup: Error: failed to copy datafile.
innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[04] xtrabackup: Error: failed to copy datafile.
210731 15:07:27 >> log scanned up to (12224642)

real    0m3.445s
user    0m4.187s
sys     0m0.456s
pipe:1 141 0 1,result:-1
end--- -1  2021-07-31 15:07:27
rm: `/tdsqlbackup/tdsqlzk/autocoldbackup/sets/set_1627394115_18/xtrabackup/xtrabackup+1627715243+20210731+150723+192.168.31.63+4002+524255906+xbstream.lz4': No such file or directory

3.2 离开安全模式（HDFS）

日志提示安全模式，不能操作，所以先设置hdfs离开安全模式：http://www.cndba.cn/dave/article/4639

[dave@www.cndba.cn 1 tdsql_full_install_ansible]$ hdfs dfsadmin -report
Safe mode is ON
……

[dave@www.cndba.cn 1 tdsql_full_install_ansible]$ hadoop dfsadmin -safemode leave
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.

Safe mode is OFF

[dave@www.cndba.cn 1 tdsql_full_install_ansible]$ hdfs dfsadmin -report
……

[dave@www.cndba.cn 1 tdsql_full_install_ansible]$ hadoop fs -ls /
Found 1 items
drwxr-xr-x   - tdsql supergroup          0 2021-07-27 22:26 /tdsqlbackup
[dave@www.cndba.cn 1 tdsql_full_install_ansible]$

然后再次执行备份到HDFS，这次备份成功：

在coldbackupimage的日志中，我们可以看到操作记录：

command: hadoop fs -mv /tdsqlbackup/tdsqlzk/autocoldbackup/sets/set_1627394115_18/xtrabackup/xtrabackup+1627716522+20210731+152842+192.168.31.63+4002+524255906+xbstream.lz4 /tdsqlbackup/tdsqlzk/autocoldbackup/sets/set_1627394115_18/xtrabackup/xtrabackup+1627716522+20210731+152842+192.168.31.63+4002+524255906+20210731+152854+16836735+xbstream.lz4;echo ok | hadoop fs -put - /tdsqlbackup/tdsqlzk/autocoldbackup/sets/set_1627394115_18/xtrabackup/xtrabackup+1627716522+20210731+152842+192.168.31.63+4002+524255906+20210731+152854+16836735+xbstream.lz4.ok
end--- 0  2021-07-31 15:29:00

4 LOCAL 备份说明

在TDSQL 集群中，手动备份支持物理备份和逻辑备份类型，备份可以存储到3种地方：http://www.cndba.cn/dave/article/4639

分布式文件存储系统（HDFS）
DB服务器本地磁盘（Local）
腾讯云分布式存储服务器（COS）

HDFS 这个我们上面已经演示过了。这里重点讲LOCAL 类型。

2点注意事项：

local备份必须通过远程挂载或者独立磁盘挂载到/tdsqlbackup。
简单的直接创建/tdsqlbackup 目录还是会导致手工local 备份失败。如下图：

并且此时mysqlagent log里是没有任何信息的。

我们这里在虚拟机上单独添加了一个硬盘，然后挂在到/tdsqlbackup目录。

不重启OS,直接让新硬盘生效：

[dave@www.cndba.cn 2 nohup]#  ls /sys/class/scsi_host/
host0   host11  host14  host17  host2   host22  host25  host28  host30  host4  host7
host1   host12  host15  host18  host20  host23  host26  host29  host31  host5  host8
host10  host13  host16  host19  host21  host24  host27  host3   host32  host6  host9
[dave@www.cndba.cn 2 nohup]# echo "- - -" > /sys/class/scsi_host/host0/scan
[dave@www.cndba.cn 2 nohup]# echo "- - -" > /sys/class/scsi_host/host1/scan
[dave@www.cndba.cn 2 nohup]# echo "- - -" > /sys/class/scsi_host/host2/scan

挂在/tdsqlbackup 目录，并修改目录权限：

[dave@www.cndba.cn 2 nohup]# mount |grep tdsqlbackup
/dev/sdb1 on /tdsqlbackup type xfs (rw,relatime,attr2,inode64,noquota)
[dave@www.cndba.cn 2 nohup]#
[dave@www.cndba.cn 2 /]# chown tdsql:users /tdsqlbackup

再次执行local 备份成功：
http://www.cndba.cn/dave/article/4639

查看日志：http://www.cndba.cn/dave/article/4639

[dave@www.cndba.cn 2 nohup]# pwd
/data/tdsql_run/4002/mysqlagent/log/nohup
[dave@www.cndba.cn 2 nohup]# ll
total 224
-rw-rw-rw-  1 tdsql users 40184 Jul 31 16:17 alldump_4002_2021-07-31
-rw-rw-rw-. 1 tdsql users     3 Jul 31 16:50 coldbackupbinlog_result_4002
-rw-rw-rw-  1 tdsql users     2 Jul 31 16:50 coldbackupbinlogtolocal_result_4002
-rw-rw-rw-. 1 tdsql users 96726 Jul 30 23:20 coldbackupimage_4002_2021-07-30
-rw-rw-rw-  1 tdsql users  3651 Jul 31 16:34 coldbackupimage_4002_2021-07-31
-rw-rw-rw-  1 tdsql users 38736 Jul 31 16:52 coldbackupimagetolocalfile_4002_2021-07-31
-rw-rw-rw-. 1 tdsql users     3 Jul 31 16:34 coldbackup_result_4002
-rw-r--r--  1 tdsql users     5 Jul 31 16:20 manualBackupResult_4002_0000000000.log
-rw-r--r--  1 tdsql users     5 Jul 31 16:22 manualBackupResult_4002_0000000002.log
-rw-r--r--  1 tdsql users     5 Jul 31 16:25 manualBackupResult_4002_0000000004.log
-rw-r--r--  1 tdsql users     5 Jul 31 16:28 manualBackupResult_4002_0000000006.log
-rw-r--r--  1 tdsql users     5 Jul 31 16:31 manualBackupResult_4002_0000000008.log
-rw-r--r--  1 tdsql users     5 Jul 31 16:40 manualBackupResult_4002_0000000010.log
-rw-r--r--  1 tdsql users     5 Jul 31 16:40 manualBackupResult_4002_0000000012.log
-rw-r--r--  1 tdsql users     2 Jul 31 16:52 manualBackupResult_4002_0000000014.log
[dave@www.cndba.cn 2 nohup]#

我们可以发现，不同类型对应的日志文件是不一样的。

coldbackupimage_4002_2021-07-30：这种是HDFS 的备份。
coldbackupimagetolocalfile_4002_2021-07-31：这种是local 备份。

我们在没有挂在/tdsqlbackup 目录之前，备份根本就没有执行，也就没有生成 coldbackupimagetolocalfile 文件，所以操作提示查看mysqlagent 意义不大。这个细节要留意。 http://www.cndba.cn/dave/article/4639

最后可以查看local 备份的日志：http://www.cndba.cn/dave/article/4639

[dave@www.cndba.cn 2 nohup]# tail -30 coldbackupimagetolocalfile_4002_2021-07-31
210731 16:52:20 [01] Streaming ./cndba/cndba.frm to <STDOUT>
210731 16:52:20 [01]        ...done
210731 16:52:20 [01] Streaming ./cndba/ustc.frm to <STDOUT>
210731 16:52:20 [01]        ...done
210731 16:52:20 Finished backing up non-InnoDB tables and files
210731 16:52:20 [00] Streaming <STDOUT>
210731 16:52:20 [00]        ...done
210731 16:52:20 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
xtrabackup: The latest check point (for incremental): '13244644'
xtrabackup: Stopping log copying thread.
.210731 16:52:20 >> log scanned up to (38532255)

210731 16:52:20 Executing UNLOCK TABLES
210731 16:52:20 All tables unlocked
210731 16:52:20 Backup created in directory '/data1/tdengine/data/4002/dbdata_raw/tmpdir/'
MySQL binlog position: filename 'binlog.000001', position '12238363', GTID of the last change '3422532c-eee2-11eb-8a1b-000c29be8288:1-38146,
859d233e-f1d6-11eb-9ffd-000c29d2361b:1-2'
210731 16:52:20 [00] Streaming <STDOUT>
210731 16:52:20 [00]        ...done
210731 16:52:20 [00] Streaming <STDOUT>
210731 16:52:20 [00]        ...done
xtrabackup: Transaction log of lsn (13244644) to (38532255) was copied.
210731 16:52:20 completed OK!

real    0m9.497s
user    0m2.566s
sys     0m6.874s
pipe:0 0 0,result:0
command: mv /tdsqlbackup/tdsqlzk/autocoldbackup/sets/set_1627394115_18/xtrabackup/xtrabackup+1627721530+20210731+165210+192.168.31.62+4002+524201122+xbstream.lz4 /tdsqlbackup/tdsqlzk/autocoldbackup/sets/set_1627394115_18/xtrabackup/xtrabackup+1627721530+20210731+165210+192.168.31.62+4002+524201122+20210731+165220+13244644+xbstream.lz4;echo ok > /tdsqlbackup/tdsqlzk/autocoldbackup/sets/set_1627394115_18/xtrabackup/xtrabackup+1627721530+20210731+165210+192.168.31.62+4002+524201122+20210731+165220+13244644+xbstream.lz4.ok
end--- 0  2021-07-31 16:52:20
You have new mail in /var/spool/mail/root
[dave@www.cndba.cn 2 nohup]#

签到成功

CNDBA社区