CNDBA社区
描述一个分库分表合并的业务场景，将上游一个个MySQL 实例的三个库中的同张表同步到下游一个 TiDB 集群中。
注意：当前文章只记录了TiDB DM 将mycat分库分表进行合库合表到TiDB下游的问题记录，仅供参考。其他搭建操作以及配置文件解释见官方文档:https://github.com/pingcap/tidb-tools/tree/docs/docs/dm/zh_CN
mycat 插入测试数据: 分片表主键ID自增长冲突
[root@ip-172-16-30-89 ~]# mysql -utidbuser1 -h172.16.30.89 -pPassword1 -P8066 -D TiDBTEST1
CREATE TABLE `user` (
`id` bigint(4) NOT NULL AUTO_INCREMENT,
`username` varchar(16) NOT NULL,
`userpassword` varchar(32) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=Innodb DEFAULT CHARSET=utf8;
mysql> use TiDBTEST1;
Database changed
mysql> insert into user(username,userpassword) values ('marvin','123456');
Query OK, 1 row affected (0.11 sec)
mysql> insert into user(username,userpassword) values ('marvin','123456');
Query OK, 1 row affected (0.00 sec)
mysql> insert into user(username,userpassword) values ('marvin','123456');
Query OK, 1 row affected (0.00 sec)
分库分表验证:
[root@ip-172-16-30-88 ~]# mysql -uroot -p123456 -h172.16.30.88 -P3308
mysql> use tidb2;
Database changed
mysql> select * from user;
Empty set (0.00 sec)
mysql> select * from user;
+----+----------+--------------+
| id | username | userpassword |
+----+----------+--------------+
| 31 | marvin | 123456 |
+----+----------+--------------+
1 row in set (0.00 sec)
mysql> use tidb3;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from user;
+----+----------+--------------+
| id | username | userpassword |
+----+----------+--------------+
| 32 | marvin | 123456 |
+----+----------+--------------+
1 row in set (0.00 sec)
mysql> use tidb1;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from user;
+----+----------+--------------+
| id | username | userpassword |
+----+----------+--------------+
| 33 | marvin | 123456 |
+----+----------+--------------+
1 row in set (0.00 sec)
上游实例
假设上游只存在一个mysql实例存在下面的 schema
- instance 1
| schma | tables |
|---|---|
| tidb1 | tb1,tb2,tb3,user |
| tidb2 | tb2,tb3,user |
| tidb3 | MYCAT_SEQUENCE,tb2,tb3,user |
同步需求
tidb1、tidb2、tidb3中的表user合并同步到下游TiDB,表存在int型自增主键，合并有冲突,建表语句如下:
CREATE TABLE `user` ( `id` int(4) NOT NULL AUTO_INCREMENT, `username` varchar(16) NOT NULL, `userpassword` varchar(32) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=Innodb DEFAULT CHARSET=utf8;
- tidb1、tidb2、tidb3中的表不进行同步，需要利用黑白名单排除
下游实例
假设同步到下游后的 schema
| schma | tables |
|---|---|
| dminc | user |
同步方案
配置task.yaml，
因为存在自增主键，所以开启任务报错如下，调整字段映射:
instance mysql-replica-01 table tidb1.user of sharding dminc.user have auto-increment key, would conflict with each other to cause data corruption”,”instruction”: “please set column mapping rules for them, or handle it by yourself”, “extra”: “auto-increment key checking”
» start-task ./task.yaml
{
"result": false,
"msg": "fail to check synchronization configuration with type CheckFailed: check was failed, please see detail
detail: {
"results": [
{
"id": 10,
"name": "sharding table consistency check",
"desc": "check consistency of sharding table structures",
"state": "fail",
"errorMsg": "instance mysql-replica-01 table `tidb1`.`user` of sharding `dminc`.`user` have auto-increment key, would conflict with each other to cause data corruption",
"instruction": "please set column mapping rules for them, or handle it by yourself",
"extra": "auto-increment key checking"
},
{
"id": 8,
"name": "source_db_privilege",
"desc": "check privileges of source DB",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.89:3308"
},
{
"id": 3,
"name": "source_db_privilege",
"desc": "check privileges of source DB",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.88:3308"
},
{
"id": 4,
"name": "table structure compatibility check",
"desc": "check compatibility of table structure",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.88:3308"
},
{
"id": 6,
"name": "mysql_binlog_format",
"desc": "check whether mysql binlog_format is ROW",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.89:3308"
},
{
"id": 1,
"name": "mysql_binlog_format",
"desc": "check whether mysql binlog_format is ROW",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.88:3308"
},
{
"id": 9,
"name": "table structure compatibility check",
"desc": "check compatibility of table structure",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.89:3308"
},
{
"id": 0,
"name": "mysql_binlog_enable",
"desc": "check whether mysql binlog is enabled",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.88:3308"
},
{
"id": 5,
"name": "mysql_binlog_enable",
"desc": "check whether mysql binlog is enabled",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.89:3308"
},
{
"id": 7,
"name": "mysql_binlog_row_image",
"desc": "check whether mysql binlog_row_image is FULL",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.89:3308"
},
{
"id": 2,
"name": "mysql_binlog_row_image",
"desc": "check whether mysql binlog_row_image is FULL",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.88:3308"
}
],
"summary": {
"passed": false,
"total": 11,
"successful": 10,
"failed": 1,
"warning": 0
}
}",
"workers": [
]
}
调整字段映射后再次开启再次报错,因为主键类型不是bigint，而是int,需要转换字段类型
“errorMsg”: “instance mysql-replica-01 table tidb3.user of sharding dminc.user have auto-increment key id and column mapping, but type of id should be bigint”, “instruction”: “please set auto-increment key type to bigint”, “extra”: “auto-increment key checking”
» start-task ./task.yaml
{
"result": false,
"msg": "fail to check synchronization configuration with type CheckFailed: check was failed, please see detail
detail: {
"results": [
{
"id": 10,
"name": "sharding table consistency check",
"desc": "check consistency of sharding table structures",
"state": "fail",
"errorMsg": "instance mysql-replica-01 table `tidb3`.`user` of sharding `dminc`.`user` have auto-increment key id and column mapping, but type of id should be bigint",
"instruction": "please set auto-increment key type to bigint",
"extra": "auto-increment key checking"
},
{
"id": 3,
"name": "source_db_privilege",
"desc": "check privileges of source DB",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.88:3308"
},
{
"id": 8,
"name": "source_db_privilege",
"desc": "check privileges of source DB",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.89:3308"
},
{
"id": 4,
"name": "table structure compatibility check",
"desc": "check compatibility of table structure",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.88:3308"
},
{
"id": 6,
"name": "mysql_binlog_format",
"desc": "check whether mysql binlog_format is ROW",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.89:3308"
},
{
"id": 0,
"name": "mysql_binlog_enable",
"desc": "check whether mysql binlog is enabled",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.88:3308"
},
{
"id": 9,
"name": "table structure compatibility check",
"desc": "check compatibility of table structure",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.89:3308"
},
{
"id": 5,
"name": "mysql_binlog_enable",
"desc": "check whether mysql binlog is enabled",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.89:3308"
},
{
"id": 1,
"name": "mysql_binlog_format",
"desc": "check whether mysql binlog_format is ROW",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.88:3308"
},
{
"id": 7,
"name": "mysql_binlog_row_image",
"desc": "check whether mysql binlog_row_image is FULL",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.89:3308"
},
{
"id": 2,
"name": "mysql_binlog_row_image",
"desc": "check whether mysql binlog_row_image is FULL",
"state": "success",
"errorMsg": "",
"instruction": "",
"extra": "address of db instance - 172.16.30.88:3308"
}
],
"summary": {
"passed": false,
"total": 11,
"successful": 10,
"failed": 1,
"warning": 0
}
}",
"workers": [
]
}
目前不支持类型转换，合库合表主键自增冲突解决，必须建立在bigint类型上：
column mapping 提供对表的列值进行修改的功能。可以根据不同的表达式对表的指定列做不同的修改操作，目前只支持 DM 提供的内置表达式。
注意：
- 不支持修改 column 的类型和表结构
- 不支持对同一个表设置多个不同的列值转换规则
mycat端手动drop表再新建
mysql> drop table user;
Query OK, 0 rows affected (0.01 sec)
mysql> CREATE TABLE `user` (
-> `id` bigint(4) NOT NULL AUTO_INCREMENT,
-> `username` varchar(16) NOT NULL,
-> `userpassword` varchar(32) NOT NULL,
-> PRIMARY KEY (`id`)
-> ) ENGINE=Innodb DEFAULT CHARSET=utf8;
Query OK, 0 rows affected (0.01 sec)
mysql> insert into user(username,userpassword) values ('marvin','123456');
Query OK, 1 row affected (0.01 sec)
mysql> insert into user(username,userpassword) values ('marvin','56789');
Query OK, 1 row affected (0.00 sec)
mysql> insert into user(username,userpassword) values ('wentaojin','56789');
Query OK, 1 row affected (0.00 sec)
再次开启task任务，报错:
主要因为字段映射存在问题，参考Issues：https://github.com/pingcap/tidb-tools/issues/183，将arguments table_id 第三个参数置于空，问题解决
cr-instance-01:
schema-pattern: “tidb*”
table-pattern: “user”
expression: “partition id” # handle sharding partition id
source-column: “id”
target-column: “id”
arguments: [“1”, “tidb”, “”]
2019/02/26 16:42:32.558 subtask.go:194: [info] [subtask] mysql dm-unit Load process returned with stage Paused, status {"totalBytes":349,"progress":"0.00 %","metaBinlog":"(mysql-bin.000003, 31025)"}
2019/02/26 16:42:32.558 subtask.go:213: [error] [subtask] mysql dm-unit Load process error with type UnknownError:
user is not the prefix of user not valid
github.com/pingcap/errors.NotValidf
/home/jenkins/workspace/build_dm_master/go/pkg/mod/github.com/pingcap/errors@v0.11.0/juju_adaptor.go:87
github.com/pingcap/tidb-tools/pkg/column-mapping.computeID
/home/jenkins/workspace/build_dm_master/go/pkg/mod/github.com/pingcap/tidb-tools@v2.1.3-0.20190115072802-b674be072353+incompatible/pkg/column-mapping/column.go:510
github.com/pingcap/tidb-tools/pkg/column-mapping.computePartitionID
/home/jenkins/workspace/build_dm_master/go/pkg/mod/github.com/pingcap/tidb-tools@v2.1.3-0.20190115072802-b674be072353+incompatible/pkg/column-mapping/column.go:503
github.com/pingcap/tidb-tools/pkg/column-mapping.(*Mapping).queryColumnInfo
/home/jenkins/workspace/build_dm_master/go/pkg/mod/github.com/pingcap/tidb-tools@v2.1.3-0.20190115072802-b674be072353+incompatible/pkg/column-mapping/column.go:368
github.com/pingcap/tidb-tools/pkg/column-mapping.(*Mapping).HandleRowValue
/home/jenkins/workspace/build_dm_master/go/pkg/mod/github.com/pingcap/tidb-tools@v2.1.3-0.20190115072802-b674be072353+incompatible/pkg/column-mapping/column.go:243
github.com/pingcap/dm/loader.parseRowValues
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/convert_data.go:157
github.com/pingcap/dm/loader.parseInsertStmt
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/convert_data.go:81
github.com/pingcap/dm/loader.reassemble
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/convert_data.go:293
github.com/pingcap/dm/loader.(*Worker).dispatchSQL
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:254
github.com/pingcap/dm/loader.(*Worker).restoreDataFile
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:174
github.com/pingcap/dm/loader.(*Worker).run
/home/jenkins/workspace/build_dm_master/go/src/github.com/pingcap/dm/loader/loader.go:162
runtime.goexit
最终配置task.yaml文件如下:
[tidb@ip-172-16-30-86 dmctl]$ cat task.yaml
---
name: mysql # global unique
task-mode: all # full/incremental/all
is-sharding: true # whether multi dm-worker do one sharding job
meta-schema: "dm_meta" # meta schema in downstreaming database to store meta informaton of dm
remove-meta: false # remove meta from downstreaming database, now we delete checkpoint and online ddl information
enable-heartbeat: false # whether to enable heartbeat for calculating lag between master and syncer
# timezone: "Asia/Shanghai" # target database timezone, all timestamp event in binlog will translate to format time based on this timezone, default use local timezone
target-database:
host: "172.16.30.86"
port: 5000
user: "root"
password: ""
mysql-instances: # one or more source database, config more source database for sharding merge
-
source-id: "mysql-replica-01" # unique in all instances, used as id when save checkpoints, configs, etc.
# binlog pos used to as start pos for syncer, for different task-mode, this maybe used or not
# `full` / `all`:
# never be used
# `incremental`:
# if `remove-meta` is true, this will be used
# else if checkpoints already exists in `meta-schema`, this will not be used
# otherwise, this will be used
meta:
binlog-name: mysql-bin.000001
binlog-pos: 4
route-rules: ["user-route-rules-schema-01","user-route-rules-schema-011","user-route-rules-01"]
#filter-rules: ["user-filter-01"]
column-mapping-rules: ["cr-instance-01"]
black-white-list: "instance-01"
# `mydumper-config-name` and `mydumper` should only set one
mydumper-config-name: "global" # ref `mydumpers` config
# mydumper:
# mydumper-path: "./mydumper"
# threads: 16
# `loader-config-name` and `loader` should only set one
loader-config-name: "global" # ref `loaders` config
#loader: # local loader rule
# pool-size: 32
# `syncer-config-name` and `syncer` should only set one
syncer-config-name: "global" # ref `syncers` config
#syncer:
# worker-count: 32
-
source-id: "mysql-replica-02"
meta:
binlog-name: mysql-bin.000001
binlog-pos: 320
route-rules: ["user-route-rules-schema-02","user-route-rules-02"]
#filter-rules: ["user-filter-02"]
#column-mapping-rules: ["instance-02"]
#black-white-list: "instance-02"
mydumper:
mydumper-path: "./bin/mydumper"
threads: 4
chunk-filesize: 64
skip-tz-utc: true
extra-args: "-B singlemysql --no-locks"
loader: # local loader rule
pool-size: 16
dir: "./dumped_data" # must be unique between tasks for the same instance
syncer:
worker-count: 16
batch: 100
max-retry: 100
# other common configs shared by all instances
routes: # schema/table route mapping
user-route-rules-schema-01:
schema-pattern: "single*"
target-schema: "dmmysql"
user-route-rules-schema-011:
schema-pattern: "tidb*"
target-schema: "dminc"
user-route-rules-01:
schema-pattern: "single*"
table-pattern: "single"
target-schema: "dmmysql"
target-table: "dmsingle"
user-route-rules-schema-02:
schema-pattern: "single*"
target-schema: "dmtidb"
user-route-rules-02:
schema-pattern: "single*"
table-pattern: "single"
target-schema: "dmtidb"
target-table: "dmsingle"
filters: # filter rules, mysql instance can ref rules in it
user-filter-01:
schema-pattern: "test_*"
table-pattern: "t_*"
events: ["truncate table", "drop table"] # ignore truncate/drop table ddl
action: Ignore
user-filter-02:
schema-pattern: "test_*"
events: ["All DML"] # only do all DML events
action: Do
black-white-list:
instance-01:
do-dbs: ["~^tidb.*","singlemysql"]
ignore-dbs: ["mysql"]
do-tables:
- db-name: "~^tidb.*"
tbl-name: "~^user.*"
- db-name: "singlemysql"
tbl-name: "~^.*" # 表示导入singlemysql数据库下的所有表数据到下游TiDB
instance-02:
do-dbs: ["~^single.*"]
ignore-dbs: ["mysql"]
do-tables:
- db-name: "~^single.*"
tbl-name: "do"
column-mappings: # column mapping rules, mysql instance can ref rules in it
cr-instance-01:
schema-pattern: "tidb*"
table-pattern: "user"
expression: "partition id" # handle sharding partition id
source-column: "id"
target-column: "id"
arguments: ["1", "tidb", ""]
instance-02:
schema-pattern: "test_*"
table-pattern: "t_*"
expression: "partition id" # handle sharding partition id
source-column: "id"
target-column: "id"
arguments: ["2", "test_", "t_"]
mydumpers: # mydumper process unit specific configs, mysql instance can ref one config in it
global:
mydumper-path: "./bin/mydumper"
threads: 4
chunk-filesize: 64
skip-tz-utc: true
#extra-args: "-B singlemysql --no-locks" #注释该行表示通过黑白名单配置的库名导出数据，而不是单独指定导出某个库，该命令常用于单库的迁移，对于合库可能需要注释，配置黑白名单
loaders: # loader process unit specific configs, mysql instance can ref one config in it
global:
pool-size: 16
dir: "./dumped_data"
syncers: # syncer process unit specific configs, mysql instance can ref one config in it
global:
worker-count: 16
batch: 100
max-retry: 100
验证合库合表
mycat 插入
[root@ip-172-16-30-89 ~]# mysql -utidbuser1 -h172.16.30.89 -pPassword1 -P8066 -D TiDBTEST1
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or /g.
Your MySQL connection id is 3
Server version: 5.6.29-mycat-1.6.5-release-20180122220033 MyCat Server (OpenCloundDB)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '/h' for help. Type '/c' to clear the current input statement.
mysql>
mysql> use TiDBTEST1;
Database changed
mysql> insert into user(username,userpassword) values ('tidb','00089');
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> insert into user(username,userpassword) values ('tidb','00089');
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 4
Current database: TiDBTEST1
Query OK, 1 row affected (0.01 sec)
TiDB查询
mysql> use dminc;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from suer;
ERROR 1146 (42S02): Table 'dminc.suer' doesn't exist
mysql> select * from uer;
ERROR 1146 (42S02): Table 'dminc.uer' doesn't exist
mysql> select * from user;
+--------------------+-----------+--------------+
| id | username | userpassword |
+--------------------+-----------+--------------+
| 580964351930794020 | wentaojin | 56789 |
| 585467951558164514 | marvin | 123456 |
| 585467951558164517 | tidb | 00089 |
| 589971551185535011 | marvin | 56789 |
+--------------------+-----------+--------------+
4 rows in set (0.00 sec)
附件配置文件解释
---
name: singlemysql
# 任务模式，full / incremental / all
task-mode: all
# 是否为分库分表任务
is-sharding: false
# 下游储存 meta 信息的 database
meta-schema: "dm_meta"
# 是否在任务同步开始前移除上面的 meta(checkpoint, onlineddl)
remove-meta: false
# 是否开启 heartbeat 功能，具体解释见同步功能 heartbeat 介绍
enable-heartbeat: false
# timezone: "Asia/Shanghai"
# target database timezone, all timestamp event in binlog will translate to format time based on this timezone, default use local timezone
# 下游数据库实例配置
target-database:
host: "172.16.30.86"
port: 5000
user: "root"
password: ""
mysql-instances: # one or more source database, config more source database for sharding merge
-
source-id: "mysql-replica-01" # unique in all instances, used as id when save checkpoints, configs, etc.
# binlog pos used to as start pos for syncer, for different task-mode, this maybe used or not
# `full` / `all`:
# never be used
# `incremental`:
# if `remove-meta` is true, this will be used
# else if checkpoints already exists in `meta-schema`, this will not be used
# otherwise, this will be used
meta:
binlog-name: mysql-bin.000001
binlog-pos: 4
# 该上游数据库实例匹配的表到下游数据库的映射规则名称
route-rules: ["user-route-rules-schema", "user-route-rules"]
# 该上游数据库实例匹配的表的 binlog 过滤规则名称
filter-rules: ["user-filter-1", "user-filter-2"]
# 该上游数据库实例匹配的表的列值转换规则名称
column-mapping-rules: ["instance-1"]
# 该上游数据库实例匹配的表的黑白名单过滤规则名称
black-white-list: "instance"
# `mydumper-config-name` and `mydumper` 应该只设置其中一个,引用 `mydumpers` global 配置
mydumper-config-name: "global"
# mydumper:
# mydumper-path: "./mydumper"
# threads: 16
# `loader-config-name` and `loader` 应该只设置其中一个,引用 `loaders` global 配置
loader-config-name: "global"
#loader: # local loader rule
# pool-size: 32
# `syncer-config-name` and `syncer` 应该只设置其中一个，应用 `syncers` global 配置
syncer-config-name: "global"
#syncer:
# worker-count: 32
# other common configs shared by all instances
routes: # 上游和下游表之间的路由映射规则集
user-route-rules-schema:
schema-pattern: "single*"
target-schema: "singlemysql"
user-route-rules:
schema-pattern: "single*"
table-pattern: "single"
target-schema: "singlemysql"
target-table: "singlemysql"
filters: # 上游数据库实例的匹配的表的 binlog 过滤规则集
user-filter-1:
schema-pattern: "test_*"
table-pattern: "t_*"
# 忽略 truncate/drop table ddl
events: ["truncate table", "drop table"]
action: Ignore
user-filter-2:
schema-pattern: "test_*"
# 只执行 schema `test_*` 下面的所有 DML events
events: ["All DML"]
action: Do
black-white-list:
instance:
do-dbs: ["~^test.*", "do"]
ignore-dbs: ["mysql", "ignored"]
do-tables:
- db-name: "~^test.*"
tbl-name: "~^t.*"
- db-name: "do"
tbl-name: "do"
- db-name: "do"
tbl-name: "do"
column-mappings: # 该上游数据库实例的匹配的表的黑白名单过滤规则集
instance-1:
schema-pattern: "test_*"
table-pattern: "t_*"
# 抓取 sharding partition id
expression: "partition id"
source-column: "id"
target-column: "id"
arguments: ["1", "test_", "t_"]
instance-2:
schema-pattern: "test_*"
table-pattern: "t_*"
# 抓取 sharding partition id
expression: "partition id"
source-column: "id"
target-column: "id"
arguments: ["2", "test_", "t_"]
mydumpers: # mydumper 进程单元指定得配置文件, mysql instance 可以引用
global:
# mydumper binary 文件地址，这个无需设置，会由 ansible 部署程序自动生成
mydumper-path: "./bin/mydumper"
# mydumper 从上游数据库实例 dump 数据的线程数量
threads: 4
# mydumper 生成的数据文件大小
chunk-filesize: 64
skip-tz-utc: true
extra-args: "-B test -T t1,t2 --no-locks"
loaders: # loader 进程单元指定得配置文件, mysql instance 可以引用
global:
# loader 并发执行 mydumper 的 SQLs file 的线程数量
pool-size: 16
# loader 读取 mydumper 输出文件的地址，同实例对应的不同任务应该不同 （mydumper 会根据这个地址输出 SQLs 文件）
dir: "./dumped_data"
syncers: # syncer 进程单元指定得配置文件, mysql instance can ref one config in it
global:
# syncer 并发同步 binlog events 的线程数量
worker-count: 16
# syncer 同步到下游数据库的一个事务批次 SQL 数
batch: 100
# syncer 同步到下游数据库出错的事务的重试次数（仅限于 DML）
max-retry: 100
版权声明：本文为博主原创文章，未经博主允许不得转载。
- 上一篇：DM 多source同步问题记录解决
- 下一篇：TiSpark 混合部署以及使用
