CNDBA社区
dm-worker报错:
[tidb@ip-172-16-30-89 log]$ tail -f dm-worker-stderr.log
2019/02/25 17:22:42.275 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
2019/02/25 17:22:57.525 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
2019/02/25 17:23:12.775 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
2019/02/25 17:23:28.024 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
2019/02/25 17:23:43.277 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
2019/02/25 17:23:58.525 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
2019/02/25 17:24:13.776 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
2019/02/25 17:24:29.025 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
2019/02/25 17:24:44.275 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
2019/02/25 17:24:59.524 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
2019/02/25 17:25:14.775 main.go:38: [error] parse cmd flags err can not decrypt password 123456: illegal base64 data at input byte 4
DM 相关配置文件中b部署dm时inventory.ini 指定source mysql密码需要使用经过 dmctl 加密后的密码，否则会报错。对于同一个原始密码，每次加密后密文不同。
$ ./dmctl -encrypt 123456
VjX8cEeTX+qcvZ3bPaO4h0C80pe/1aU=
将该密码存放于配置文件，相同密码可以放同样存放
task.yaml 任务文件需要使用相对路径
[tidb@ip-172-16-30-86 conf]$ pwd
/data/tidb/wentaojin/dm-ansible-latest/conf
[tidb@ip-172-16-30-86 conf]$ cp task.yaml.example /data/tidb/deploy_dm/conf/task.yaml
/data/tidb/deploy_dm/conf/task.yaml 使用绝对路径无法进行任务开始，即dm-worker无法工作，只能使用相对路径
[tidb@ip-172-16-30-86 dmctl]$ ./dmctl --master-addr 172.16.10.86:8261
Welcome to dmctl
Release Version: v1.0.0-alpha-14-g05c70da
Git Commit Hash: 05c70da45d2439d1d54498f210b868a9ca26a303
Git Branch: master
UTC Build Time: 2019-02-22 08:29:27
Go Version: go version go1.11.2 linux/amd64
»
»
» start-task /data/tidb/deploy_dm/conf/task.yaml
^Cgot signal [interrupt] to exit[tidb@ip-172-16-30-86 dmctl]$
[tidb@ip-172-16-30-86 conf]$ pwd
/data/tidb/deploy_dm/conf
[tidb@ip-172-16-30-86 conf]$ cp task.yaml /data/tidb/wentaojin/dm-ansible-latest/dmctl/
[tidb@ip-172-16-30-86 dmctl]$ ./dmctl -config dmctl.toml
Welcome to dmctl
Release Version: v1.0.0-alpha-14-g05c70da
Git Commit Hash: 05c70da45d2439d1d54498f210b868a9ca26a303
Git Branch: master
UTC Build Time: 2019-02-22 08:29:27
Go Version: go version go1.11.2 linux/amd64
» start-task ./task.yaml
{
"result": true,
"msg": "",
"workers": [
{
"result": true,
"worker": "172.16.30.88:8262",
"msg": ""
},
{
"result": true,
"worker": "172.16.30.89:8262",
"msg": ""
}
]
}
"msg": "invalid mydumper files for there are no `-schema-create.sql` files found/ngithub.com/pingcap/dm/loader.(*Loader).prepareDbFiles/n/t/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:654/ngithub.com/pingcap/dm/loader.(*Loader).prepare/n/t/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:769/ngithub.com/pingcap/dm/loader.(*Loader).Restore/n/t/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:473/ngithub.com/pingcap/dm/loader.(*Loader).Process/n/t/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:411/nruntime.goexit/n/t/usr/local/go/src/runtime/asm_amd64.s:1333"
task文件存在错误，当前模式是all,没有配置mydumper导出哪个库以及黑白名单
重新配置task.yaml，需要清理task子任务，否则报子任务存在冲突
2019/02/25 20:03:33.434 server.go:428: [info] [server] receive query worker config request
2019/02/25 20:03:33.450 server.go:140: [info] [server] receive StartSubTask request task:"is-sharding = true/nonline-ddl-scheme = /"/"/ncase-sensitive = false/nname = /"mysql/"/nmode = /"all/"/nsource-id = /"mysql-replica-01/"/nserver-id = 0/nflavor = /"/"/nmeta-schema = /"dm_meta/"/nremove-meta = false/ndisable-heartbeat = true/nenable-heartbeat = false/ntimezone = /"/"/nbinlog-type = /"local/"/nrelay-dir = /"/"/nmydumper-path = /"./bin/mydumper/"/nthreads = 4/nchunk-filesize = 64/nskip-tz-utc = true/nextra-args = /"-B singlemysql -T t1,t2 --no-locks/"/npool-size = 16/ndir = /"./dumped_data/"/nmeta-file = /"/"/nworker-count = 16/nbatch = 100/nmax-retry = 100/nauto-fix-gtid = false/nenable-gtid = false/ndisable-detect = false/nsafe-mode = false/nenable-ansi-quotes = false/nlog-level = /"/"/nlog-file = /"/"/nlog-rotate = /"/"/npprof-addr = /"/"/nstatus-addr = /"/"/n/n[meta]/n BinLogName = /"mysql-bin.000001/"/n BinLogPos = 4/n/n[from]/n host = /"172.16.30.88/"/n port = 3308/n user = /"root/"/n password = /"123456/"/n/n[to]/n host = /"172.16.30.86/"/n port = 5000/n user = /"root/"/n password = /"/"/n/n[[route-rules]]/n schema-pattern = /"single*/"/n table-pattern = /"/"/n target-schema = /"dmmysql/"/n target-table = /"/"/n/n[[route-rules]]/n schema-pattern = /"single*/"/n table-pattern = /"single/"/n target-schema = /"dmmysql/"/n target-table = /"dmsingle/"/n/n[[filter-rules]]/n schema-pattern = /"test_*/"/n table-pattern = /"t_*/"/n events = [/"truncate table/", /"drop table/"]/n action = /"Ignore/"/n/n[[mapping-rule]]/n schema-pattern = /"test_*/"/n table-pattern = /"t_*/"/n source-column = /"id/"/n target-column = /"id/"/n expression = /"partition id/"/n arguments = [/"1/", /"test_/", /"t_/"]/n create-table-query = /"/"/n/n[black-white-list]/n do-dbs = [/"~^single.*/", /"do/"]/n ignore-dbs = [/"mysql/", /"ignored/"]/n/n [[black-white-list.do-tables]]/n db-name = /"~^single.*/"/n tbl-name = /"do/"/n/n [[black-white-list.do-tables]]/n db-name = /"do/"/n tbl-name = /"do/"/n/n [[black-white-list.do-tables]]/n db-name = /"do/"/n tbl-name = /"do/"/n"
2019/02/25 20:03:33.450 server.go:154: [error] [server] start sub task mysql error sub task with name mysql already started
github.com/pingcap/dm/dm/worker.(*Worker).StartSubTask
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/worker.go:158
github.com/pingcap/dm/dm/worker.(*Server).StartSubTask
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/worker/server.go:152
github.com/pingcap/dm/dm/pb._Worker_StartSubTask_Handler
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/dm/pb/dmworker.pb.go:3504
google.golang.org/grpc.(*Server).processUnaryRPC
/home/jenkins/workspace/build_dm_master/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:966
google.golang.org/grpc.(*Server).handleStream
/home/jenkins/workspace/build_dm_master/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:1245
google.golang.org/grpc.(*Server).serveStreams.func1.1
/home/jenkins/workspace/build_dm_master/go/pkg/mod/google.golang.org/grpc@v1.17.0/server.go:685
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1333
由于第一次启动失败，导致子任务还存在，清理子任务
查看任务状态
» query-status
{
"result": true,
"msg": "",
"workers": [
{
"result": true,
"worker": "172.16.30.88:8262",
"msg": "",
"subTaskStatus": [
{
"name": "mysql",
"stage": "Paused",
"unit": "Dump",
"result": {
"isCanceled": false,
"errors": [
{
"Type": "UnknownError",
"msg": "exit status 1. "
}
],
"detail": null
},
"unresolvedDDLLockID": "",
"dump": {
}
}
],
"relayStatus": {
"masterBinlog": "(mysql-bin.000003, 15806)",
"masterBinlogGtid": "",
"relaySubDir": "70841deb-357b-11e9-b66f-000af7e3703c.000001",
"relayBinlog": "(mysql-bin.000003, 15806)",
"relayBinlogGtid": "",
"relayCatchUpMaster": true,
"stage": "Running",
"result": null
}
},
{
"result": true,
"worker": "172.16.30.89:8262",
"msg": "",
"subTaskStatus": [
{
"name": "mysql",
"stage": "Paused",
"unit": "Load",
"result": {
"isCanceled": false,
"errors": [
{
"Type": "UnknownError",
"msg": "invalid mydumper files for there are no `-schema-create.sql` files found/ngithub.com/pingcap/dm/loader.(*Loader).prepareDbFiles/n/t/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:654/ngithub.com/pingcap/dm/loader.(*Loader).prepare/n/t/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:769/ngithub.com/pingcap/dm/loader.(*Loader).Restore/n/t/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:473/ngithub.com/pingcap/dm/loader.(*Loader).Process/n/t/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:411/nruntime.goexit/n/t/usr/local/go/src/runtime/asm_amd64.s:1333"
}
],
"detail": null
},
"unresolvedDDLLockID": "",
"load": {
"finishedBytes": "0",
"totalBytes": "0",
"progress": "NaN %",
"metaBinlog": "(mysql-bin.000005, 737391979)"
}
}
],
"relayStatus": {
"masterBinlog": "(mysql-bin.000005, 737744635)",
"masterBinlogGtid": "",
"relaySubDir": "acd4a081-34f8-11e9-bcf1-b02628123c70.000001",
"relayBinlog": "(mysql-bin.000005, 737744635)",
"relayBinlogGtid": "",
"relayCatchUpMaster": true,
"stage": "Running",
"result": null
}
}
]
}
清理子任务
» stop-task mysql
{
"op": "Stop",
"result": true,
"msg": "",
"workers": [
{
"op": "Stop",
"result": true,
"worker": "172.16.30.88:8262",
"msg": ""
},
{
"op": "Stop",
"result": true,
"worker": "172.16.30.89:8262",
"msg": ""
}
]
}
查询验证
» query-status
{
"result": true,
"msg": "",
"workers": [
{
"result": true,
"worker": "172.16.30.88:8262",
"msg": "no sub task started",
"subTaskStatus": [
],
"relayStatus": {
"masterBinlog": "(mysql-bin.000003, 15806)",
"masterBinlogGtid": "",
"relaySubDir": "70841deb-357b-11e9-b66f-000af7e3703c.000001",
"relayBinlog": "(mysql-bin.000003, 15806)",
"relayBinlogGtid": "",
"relayCatchUpMaster": true,
"stage": "Running",
"result": null
}
},
{
"result": true,
"worker": "172.16.30.89:8262",
"msg": "no sub task started",
"subTaskStatus": [
],
"relayStatus": {
"masterBinlog": "(mysql-bin.000005, 737756167)",
"masterBinlogGtid": "",
"relaySubDir": "acd4a081-34f8-11e9-bcf1-b02628123c70.000001",
"relayBinlog": "(mysql-bin.000005, 737756167)",
"relayBinlogGtid": "",
"relayCatchUpMaster": true,
"stage": "Running",
"result": null
}
}
]
}
再次开启
» start-task ./task.yaml
{
"result": true,
"msg": "",
"workers": [
{
"result": true,
"worker": "172.16.30.88:8262",
"msg": ""
},
{
"result": true,
"worker": "172.16.30.89:8262",
"msg": ""
}
]
}
重新开启start.yaml，需要删除TiDB中存在得数据库database，尤其是dm_meta,报错元数据得数据库，否则在原库删除某个数据库时再开启同步报错
2019/02/25 22:18:56.717 subtask.go:194: [info] [subtask] mysql dm-unit Sync process returned with stage Paused, status {"masterBinlog":"(mysql-bin.000005, 738434323)","syncerBinlog":"(mysql-bin.000005, 738433951)"}
2019/02/25 22:18:56.717 subtask.go:213: [error] [subtask] mysql dm-unit Sync process error with type UnknownError:
Error 1146: Table 'tidb_binlog.checkpoint' doesn't exist
github.com/pingcap/errors.AddStack
/home/jenkins/workspace/build_dm_master/go/pkg/mod/github.com/pingcap/errors@v0.11.0/errors.go:174
github.com/pingcap/errors.Trace
/home/jenkins/workspace/build_dm_master/go/pkg/mod/github.com/pingcap/errors@v0.11.0/juju_adaptor.go:12
github.com/pingcap/dm/syncer.(*Conn).querySQL
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:94
github.com/pingcap/dm/syncer.getTableColumns
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/db.go:352
github.com/pingcap/dm/syncer.(*Syncer).getTableFromDB
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:518
github.com/pingcap/dm/syncer.(*Syncer).getTable
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:544
github.com/pingcap/dm/syncer.(*Syncer).Run
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/syncer/syncer.go:1144
github.com/pingcap/dm/syncer.(*Syncer).Process
这是因为目标端库TiDB重新DM同步，但是仍然存在原同步失败tidb_binlog.checkpoin得checkpoint元信息，
停止DM同步任务
[tidb@ip-172-16-30-86 dmctl]$ ./dmctl -config dmctl.toml
Welcome to dmctl
Release Version: v1.0.0-alpha-14-g05c70da
Git Commit Hash: 05c70da45d2439d1d54498f210b868a9ca26a303
Git Branch: master
UTC Build Time: 2019-02-22 08:29:27
Go Version: go version go1.11.2 linux/amd64
» stop-task mysql
{
"op": "Stop",
"result": true,
"msg": "",
"workers": [
{
"op": "Stop",
"result": true,
"worker": "172.16.30.88:8262",
"msg": ""
},
{
"op": "Stop",
"result": true,
"worker": "172.16.30.89:8262",
"msg": ""
}
]
}
把TiDB中同步得库以及dm_meta库删除
mysql> drop database dm_meta;
Query OK, 0 rows affected (0.20 sec)
mysql> drop database dmmysql;
Query OK, 0 rows affected (0.21 sec)
mysql> drop database dmtidb;
Query OK, 0 rows affected (0.20 sec)
重新DM同步,报错问题解决
» start-task ./task.yaml
{
"result": true,
"msg": "",
"workers": [
{
"result": true,
"worker": "172.16.30.88:8262",
"msg": ""
},
{
"result": true,
"worker": "172.16.30.89:8262",
"msg": ""
}
]
}
版权声明：本文为博主原创文章，未经博主允许不得转载。
- 上一篇：TiDB DM数据迁移多Source使用
- 下一篇：DM 合库合表问题记录实践
