CNDBA社区
1 MongoShake 工具说明
MongoShake是基于golang编写的MongoDB数据库高性能同步工具，可以满足用户实时数据同步、迁移、容灾、分析、异构、双活等多场景需求。目前MongoShake已被国内外多个著名厂商使用，具有丰富的使用场景。
Mongoshake 开源地址：
https://github.com/alibaba/MongoShake
MongoShake对接的源数据库支持单个mongod，replica set和sharding三种模式。目的数据库支持mongod和mongos。如果源端数据库为replica set，我们建议对接备库以减少主库的压力；如果为sharding模式，那么每个shard都将对接到MongoShake并进行并行抓取。对于目的库来说，可以对接多个mongos，不同的数据将会哈希后写入不同的mongos。
MongoShake——基于MongoDB的跨数据中心的数据复制平台
https://developer.aliyun.com/article/603329https://blog.csdn.net/xxscj/article/details/115310953
https://developer.aliyun.com/live/45078
2 MongoShake 操作示例
相关说明参考官网手册：
MongoShake最佳实践
https://developer.aliyun.com/article/719704
第一次使用，如何进行配置？
我们这里演示一个Mongodb 副本集同步到集群的操作。
2.1 当前环境说明
副本集环境
rs0:SECONDARY> rs.status().members.map((x) => { return {name: x.name, stateStr: x.stateStr, optimeDate: x.optimeDate, syncingTo: x.syncingTo} })
[
{
"name" : "172.31.185.120:27017",
"stateStr" : "SECONDARY",
"optimeDate" : ISODate("2022-06-07T01:29:14Z"),
"syncingTo" : undefined
},
{
"name" : "172.31.185.131:27017",
"stateStr" : "PRIMARY",
"optimeDate" : ISODate("2022-06-07T01:29:14Z"),
"syncingTo" : undefined
},
{
"name" : "172.31.185.165:27017",
"stateStr" : "SECONDARY",
"optimeDate" : ISODate("2022-06-07T01:29:14Z"),
"syncingTo" : undefined
}
]
rs0:SECONDARY>
集群环境
mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("627147d1763e74c93e6af56a")
}
shards:
{ "_id" : "shard1", "host" : "shard1/172.31.185.120:27018,172.31.185.131:27018,172.31.185.165:27018", "state" : 1 }
{ "_id" : "shard2", "host" : "shard2/172.31.185.120:27019,172.31.185.131:27019,172.31.185.165:27019", "state" : 1 }
{ "_id" : "shard3", "host" : "shard3/172.31.185.120:27020,172.31.185.131:27020,172.31.185.165:27020", "state" : 1 }
active mongoses:
"5.0.8" : 3
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
……
2.2 下载并配置MongoShake
下载MongoShake，目前稳定版是2.6.6,下载地址如下：
https://github.com/alibaba/MongoShake/releases/tag/release-v2.6.6-20220323
[dave@www.cndba.cn_1 data]# tar xzvf mongo-shake-v2.6.6.tgz
mongo-shake-v2.6.6/
mongo-shake-v2.6.6/receiver.windows
mongo-shake-v2.6.6/collector.linux
mongo-shake-v2.6.6/stop.sh
mongo-shake-v2.6.6/collector.darwin
mongo-shake-v2.6.6/hypervisor
mongo-shake-v2.6.6/receiver.darwin
mongo-shake-v2.6.6/collector.windows
mongo-shake-v2.6.6/receiver.linux
mongo-shake-v2.6.6/receiver.conf
mongo-shake-v2.6.6/start.sh
mongo-shake-v2.6.6/mongoshake-stat
mongo-shake-v2.6.6/comparison.py
mongo-shake-v2.6.6/collector.conf
[dave@www.cndba.cn_1 data]#
修改collector.conf 配置文件：
mongo_urls = mongodb://root:root@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017
#源端连接串信息，逗号分隔同一个shard不同的mongod，分号分隔不同的shard。
sync_mode = all # all 表示全量+增量，document表示仅全量，oplog表示仅增量
tunnel.address = mongodb://root:root@172.31.185.120:27000,172.31.185.165:27000,172.31.185.131:27000
#目的端连接串信息，分号分割不同的mongos。也可以只配置部分，配置多个mongos可以做负载均衡写入。
incr_sync.mongo_fetch_method = oplog
# 如果希望以change stream拉取，该值需要配置change_stream，支持>=4.0.1版本。
2.3 启用同步并打印日志信息
[dave@www.cndba.cn_1 mongo-shake-v2.6.6]# ./collector.linux -conf=collector.conf -verbose 1
[2022/06/09 10:12:26 CST] [INFO] log init succ. log.dir[] log.name[collector.log] log.level[info]
[2022/06/09 10:12:26 CST] [INFO] MongoDB Version Source[4.4.14] Target[4.4.14]
[2022/06/09 10:12:26 CST] [WARN]
______________________________
/ / _ ______ |
/ / / /___-=O'/|O'/__|
/ MongoShake, Here we go !! /_______/ / | / )
/ / '/-==__ _/__|/__=-| -GM
/ Alibaba Cloud / * / | |
/ / (o)
------------------------------
if you have any problem, please visit https://github.com/alibaba/MongoShake/wiki/FAQ
[2022/06/09 10:12:26 CST] [INFO] New session to mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin successfully
[2022/06/09 10:12:26 CST] [INFO] Close session with mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin
[2022/06/09 10:12:26 CST] [INFO] New session to mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin successfully
[2022/06/09 10:12:26 CST] [INFO] Close session with mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin
[2022/06/09 10:12:26 CST] [INFO] Collector startup. shard_by[collection] gids[[]]
……
[2022/06/09 10:12:27 CST] [INFO] replication from [replica] to [sharding]
[2022/06/09 10:12:27 CST] [INFO] document syncer-0 do replication for url=mongodb://root:root@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin
[2022/06/09 10:12:27 CST] [INFO] New session to mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin successfully
[2022/06/09 10:12:27 CST] [INFO] Close client with mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin
[2022/06/09 10:12:27 CST] [INFO] DBSyncer id[0] source[mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin] target[mongodb://root:***@172.31.185.120:27000,172.31.185.165:27000,172.31.185.131:27000/admin] startTime[2022-06-09 10:12:27.137923965 +0800 CST m=+0.399184355] collExecutor-5 sync ns {cndba ustc} to {cndba ustc} begin
[2022/06/09 10:12:27 CST] [INFO] DBSyncer id[0] source[mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin] target[mongodb://root:***@172.31.185.120:27000,172.31.185.165:27000,172.31.185.131:27000/admin] startTime[2022-06-09 10:12:27.137923965 +0800 CST m=+0.399184355] collExecutor-0 sync ns {cndba user} to {cndba user} begin
[2022/06/09 10:12:27 CST] [INFO] DBSyncer id[0] source[mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin] target[mongodb://root:***@172.31.185.120:27000,172.31.185.165:27000,172.31.185.131:27000/admin] startTime[2022-06-09 10:12:27.137923965 +0800 CST m=+0.399184355] collExecutor-1 sync ns {m201 ustc} to {m201 ustc} begin
……
[2022/06/09 10:12:47 CST] [INFO] Close session with mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin
全库同步结束后会有提示：
[2022/06/09 10:12:47 CST] [INFO] ------------------------full sync done!------------------------
[2022/06/09 10:12:47 CST] [INFO] oldestTs[7104151747563626680[1654064224, 8376]] fullBeginTs[7107057353068904449[1654740738, 1]] fullFinishTs[7107057477622956034[1654740767, 2]]
[2022/06/09 10:12:47 CST] [INFO] finish full sync, start incr sync with timestamp: fullBeginTs[7107057353068904449[1654740738, 1]], fullFinishTs[7107057477622956034[1654740767, 2]]
然后开始增量同步：
[2022/06/09 10:12:47 CST] [INFO] start incr replication
[2022/06/09 10:12:47 CST] [INFO] RealSourceIncrSync[0]: url[mongodb://root:***@172.31.185.120:27017,172.31.185.165:27017,172.31.185.131:27017/admin], name[rs0], startTimestamp[7107057353068904449]
[2022/06/09 10:12:47 CST] [INFO] New session to mongodb://root:***@172.31.185.120:27000,172.31.185.165:27000,172.31.185.131:27000/admin successfully
……
这里我们是以前台方式进行同步的，生产环境数据量大，需要放到后台执行。
2.4 监控MongoShake状态
增量数据同步开始后，再开启一个命令行窗口，通过如下命令来监控MongoShake。
[dave@www.cndba.cn_1 mongo-shake-v2.6.6]# ./mongoshake-stat --port=9100
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| log_size_avg | log_size_max | logs_get/sec | logs_repl/sec | logs_success/sec | lsn.time | lsn_ack.time | lsn_ckpt.time | now.time | replset | tps/sec |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 109.00B | 339.00B | none | none | none | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-06-09 10:15:38 | 2022-06-09 10:16:47 | rs0 | none |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 106.00B | 339.00B | 1 | 0 | 0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-06-09 10:15:38 | 2022-06-09 10:16:48 | rs0 | 0 |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 106.00B | 339.00B | 0 | 0 | 0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-06-09 10:15:38 | 2022-06-09 10:16:49 | rs0 | 0 |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 106.00B | 339.00B | 0 | 0 | 0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-06-09 10:15:38 | 2022-06-09 10:16:50 | rs0 | 0 |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| 106.00B | 339.00B | 0 | 0 | 0 | 1970-01-01 08:00:00 | 1970-01-01 08:00:00 | 2022-06-09 10:15:38 | 2022-06-09 10:16:51 | rs0 | 0 |
|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
……
2.5 验证同步
2.5.1 源库副本集主库上操作
[dave@www.cndba.cn_2 local]# mongo_conn 27017
MongoDB shell version v4.4.14
connecting to: mongodb://127.0.0.1:27017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("27f87bbd-1fc4-4d27-8b44-8144111bb5d5") }
MongoDB server version: 4.4.14
---
The server generated these startup warnings when booting:
2022-06-09T09:55:16.258+08:00: You are running this process as the root user, which is not recommended
---
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).
The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.
To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
rs0:PRIMARY> show dbs
admin 0.000GB
cndba 0.006GB
config 0.000GB
local 0.645GB
m201 0.147GB
mongoshake 0.000GB
rs0:PRIMARY> use cndba
switched to db cndba
rs0:PRIMARY> show tables
user
ustc
rs0:PRIMARY> db.ustc.find()
{ "_id" : ObjectId("626decb45f444096b30513c5"), "name" : "cndba", "url" : "https://www.cndba.cn" }
{ "_id" : ObjectId("626dfd8c244fe1322d79fc76"), "name" : "cndba", "url" : "https://www.cndba.cn" }
rs0:PRIMARY>
rs0:PRIMARY> db.ustc.insert({name:"dave",url:"https://www.cndba.cn"})
WriteResult({ "nInserted" : 1 })
rs0:PRIMARY> db.ustc.find()
{ "_id" : ObjectId("626decb45f444096b30513c5"), "name" : "cndba", "url" : "https://www.cndba.cn" }
{ "_id" : ObjectId("626dfd8c244fe1322d79fc76"), "name" : "cndba", "url" : "https://www.cndba.cn" }
{ "_id" : ObjectId("62a1592f7d68a2c1a5519192"), "name" : "dave", "url" : "https://www.cndba.cn" }
rs0:PRIMARY>
2.5.2 目标库sharded cluster 验证
在Mongos 上查询，同步成功：
[dave@www.cndba.cn_3 local]# mongo_conn 27000
MongoDB shell version v4.4.14
connecting to: mongodb://127.0.0.1:27000/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("93eb8ae1-81ad-4937-aad8-1dd65a5a2717") }
MongoDB server version: 4.4.14
---
The server generated these startup warnings when booting:
2022-06-09T09:56:07.843+08:00: You are running this process as the root user, which is not recommended
---
mongos> use cndba
switched to db cndba
mongos> db.ustc.find()
{ "_id" : ObjectId("626decb45f444096b30513c5"), "name" : "cndba", "url" : "https://www.cndba.cn" }
{ "_id" : ObjectId("626dfd8c244fe1322d79fc76"), "name" : "cndba", "url" : "https://www.cndba.cn" }
{ "_id" : ObjectId("62a1592f7d68a2c1a5519192"), "name" : "dave", "url" : "https://www.cndba.cn" }
mongos>
版权声明：本文为博主原创文章，未经博主允许不得转载。
