CNDBA社区
1 Oplog 格式说明
在之前的博客中，我们介绍了oplog的相关概念，和维护中的注意事项，如下：
MongoDB 4.4 中 journal 和 oplog 日志 说明
https://www.cndba.cn/dave/article/107979MongoDB 估计Oplog大小 方法
https://www.cndba.cn/dave/article/108008
1.1 查询oplog 数据
oplog 的数据保存在local.oplog.rs 集合中。 在副本集中，可以通过如下命令查看oplog的状态：
shard1:PRIMARY> rs.printReplicationInfo()
configured oplog size: 2500MB
log length start to end: 2011331secs (558.7hrs)
oplog first event time: Sat May 07 2022 14:31:27 GMT+0800 (CST)
oplog last event time: Mon May 30 2022 21:13:38 GMT+0800 (CST)
now: Mon May 30 2022 21:13:39 GMT+0800 (CST)
shard1:PRIMARY>
shard1:PRIMARY> show dbs;
admin 0.000GB
cndba 0.002GB
config 0.001GB
local 0.192GB
test 0.000GB
ycsb 0.267GB
shard1:PRIMARY> use local
switched to db local
shard1:PRIMARY> show collections;
oplog.rs
replset.election
replset.initialSyncId
replset.minvalid
replset.oplogTruncateAfterPoint
startup_log
system.replset
system.rollback.id
system.tenantMigration.oplogView
system.views
shard1:PRIMARY> db.oplog.rs.find().limit(2).pretty()
{
"op" : "n",
"ns" : "",
"o" : {
"msg" : "periodic noop"
},
"ts" : Timestamp(1651905087, 2),
"t" : NumberLong(9),
"v" : NumberLong(2),
"wall" : ISODate("2022-05-07T06:31:27.317Z")
}
{
"op" : "u",
"ns" : "config.system.sessions",
"ui" : UUID("81f9e30f-314d-417b-b1e4-3dc86398fb85"),
"o" : {
"_id" : {
"id" : UUID("ab9fe3fc-70ec-4d8e-ac67-3151f0b2ee7c"),
"uid" : BinData(0,"1Au31c5sDjnDHx/Y8VzkS3IrxbMZ8XX8TX3S1hDQM9o=")
},
"lastUse" : ISODate("2022-05-07T06:31:34.771Z"),
"user" : {
"name" : "pbmbackup@admin"
}
},
"o2" : {
"_id" : {
"id" : UUID("ab9fe3fc-70ec-4d8e-ac67-3151f0b2ee7c"),
"uid" : BinData(0,"1Au31c5sDjnDHx/Y8VzkS3IrxbMZ8XX8TX3S1hDQM9o=")
}
},
"ts" : Timestamp(1651905094, 1),
"t" : NumberLong(9),
"v" : NumberLong(2),
"wall" : ISODate("2022-05-07T06:31:34.771Z")
}
shard1:PRIMARY>
1.2 字段含义
- ts：8字节的时间戳，由4字节unix timestamp + 4字节自增计数表示。这个值很重要，在选举(如master宕机时)新primary时，会选择ts最大的那个secondary作为新primary。
- op：1字节的操作类型，例如i表示insert，d表示delete。
- ns：操作所在的namespace。
- o：操作所对应的document,即当前操作的内容（比如更新操作时要更新的的字段和值）
- o2: 在执行更新操作时的条件，仅限于update时才有该属性。
其中op，可以是如下几种情形之一：
- “i”： insert
- “u”： update
- “d”： delete
- “c”： db cmd
- “db”：声明当前数据库 (其中ns 被设置成为=>数据库名称+ ‘.’)
- “n”: no op,即空操作，其会定期执行以确保时效性。修改配置，会产生 “n” 操作。
2 查询示例
oplog.rs系统集合只用于复制，不能创建索引，查询语句会很慢。
2.1 查询所有的oplog.rs
[dave@www.cndba.cn ~]# mongo_conn 27018
MongoDB shell version v5.0.8
shard1:PRIMARY> db.oplog.rs.find({},{"ts":1}).sort({$natural: -1})
{ "ts" : Timestamp(1653917978, 1) }
{ "ts" : Timestamp(1653917968, 1) }
{ "ts" : Timestamp(1653917958, 1) }
{ "ts" : Timestamp(1653917948, 1) }
{ "ts" : Timestamp(1653917938, 1) }
{ "ts" : Timestamp(1653917922, 1) }
{ "ts" : Timestamp(1653917918, 1) }
{ "ts" : Timestamp(1653917908, 1) }
{ "ts" : Timestamp(1653917898, 1) }
{ "ts" : Timestamp(1653917888, 1) }
{ "ts" : Timestamp(1653917878, 1) }
{ "ts" : Timestamp(1653917868, 1) }
{ "ts" : Timestamp(1653917858, 1) }
{ "ts" : Timestamp(1653917848, 1) }
{ "ts" : Timestamp(1653917838, 1) }
{ "ts" : Timestamp(1653917828, 1) }
{ "ts" : Timestamp(1653917818, 1) }
{ "ts" : Timestamp(1653917808, 1) }
{ "ts" : Timestamp(1653917798, 1) }
{ "ts" : Timestamp(1653917788, 1) }
Type "it" for more
shard1:PRIMARY>
shard1:PRIMARY> db.oplog.rs.find({ "ts" : { "$gte" : Timestamp(1653917958, 1) } },{"ts":1})
{ "ts" : Timestamp(1653917958, 1) }
{ "ts" : Timestamp(1653917968, 1) }
{ "ts" : Timestamp(1653917978, 1) }
{ "ts" : Timestamp(1653917988, 1) }
{ "ts" : Timestamp(1653917998, 1) }
{ "ts" : Timestamp(1653918008, 1) }
……
2.2 查询一小时内的oplog
shard1:PRIMARY> var SECS_PER_HOUR = 3600
shard1:PRIMARY> var now = Math.floor((new Date().getTime()) / 1000) // seconds since epoch right now
shard1:PRIMARY> db.oplog.rs.find({ "ts" : { "$lt" : Timestamp(now, 1), "$gt" : Timestamp(now - SECS_PER_HOUR, 1) } })
{ "op" : "n", "ns" : "", "o" : { "msg" : "periodic noop" }, "ts" : Timestamp(1653914688, 1), "t" : NumberLong(14), "v" : NumberLong(2), "wall" : ISODate("2022-05-30T12:44:48.192Z") }
{ "op" : "n", "ns" : "", "o" : { "msg" : "periodic noop" }, "ts" : Timestamp(1653914698, 1), "t" : NumberLong(14), "v" : NumberLong(2), "wall" : ISODate("2022-05-30T12:44:58.192Z") }
{ "op" : "n", "ns" : "", "o" : { "msg" : "periodic noop" }, "ts" : Timestamp(1653914708, 1), "t" : NumberLong(14), "v" : NumberLong(2), "wall" : ISODate("2022-05-30T12:45:08.193Z") }
{ "op" : "n", "ns" : "", "o" : { "msg" : "periodic noop" }, "ts" : Timestamp(1653914718, 1), "t" : NumberLong(14), "v" : NumberLong(2), "wall" : ISODate("2022-05-30T12:45:18.193Z") }
……
2.3 查询某一时间段内的oplog
注意时间是UTC存储：
shard1:PRIMARY> var since = Math.floor(ISODate("2014-08-12T09:00:00.000Z").getTime() / 1000)
shard1:PRIMARY> var since = Math.floor(ISODate("2022-05-28T09:00:00.000Z").getTime() / 1000)
shard1:PRIMARY> var until = Math.floor(ISODate("2022-05-30T15:00:00.000Z").getTime() / 1000)
shard1:PRIMARY> db.oplog.rs.find({ "ts" : { "$lt" : Timestamp(until, 1), "$gt" : Timestamp(since, 1) } })
{ "op" : "n", "ns" : "", "o" : { "msg" : "periodic noop" }, "ts" : Timestamp(1653728403, 1), "t" : NumberLong(14), "v" : NumberLong(2), "wall" : ISODate("2022-05-28T09:00:03.572Z") }
{ "op" : "n", "ns" : "", "o" : { "msg" : "periodic noop" }, "ts" : Timestamp(1653728413, 1), "t" : NumberLong(14), "v" : NumberLong(2), "wall" : ISODate("2022-05-28T09:00:13.573Z") }
……
2.4 聚合统计各个集合的UPDATE操作量
shard1:PRIMARY> db.oplog.rs.aggregate([
... { $match: { "op":"u" }},
... { $group: { _id:"$ns",count:{$sum:1}}}
... ])
{ "_id" : "config.cache.databases", "count" : 13 }
{ "_id" : "config.system.sessions", "count" : 49157 }
{ "_id" : "config.cache.collections", "count" : 2 }
{ "_id" : "ycsb.dave", "count" : 249724 }
{ "_id" : "admin.system.version", "count" : 2 }
shard1:PRIMARY>
2.5 查询drop 操作的时间
对于利用oplog 进行增量恢复的情况，在应用oplog 之前，都需要先查询操作的时间，这里如果加上ns就是查找特定的数据库，不加就是查找所有数据库。
删除操作数据库、集合的操作也记为”c”,所以要查找操作为”c”的oplog。
shard1:PRIMARY> db.oplog.rs.find({"op": "c"}).sort({"ts": -1}).limit(1)
{ "op" : "c", "ns" : "ustc.$cmd", "ui" : UUID("91bef375-c3b3-4416-94bd-946b57163c8e"), "o" : { "drop" : "dave1" }, "o2" : { "numRecords" : 0 }, "ts" : Timestamp(1652753079, 1), "t" : NumberLong(12), "v" : NumberLong(2), "wall" : ISODate("2022-05-17T02:04:39.269Z") }
shard1:PRIMARY> db.oplog.rs.find({"ns" : "ustc.$cmd", "op": "c"})
{ "op" : "c", "ns" : "ustc.$cmd", "ui" : UUID("233b8dc2-66d4-404a-a8d9-0971af648619"), "o" : { "create" : "dave", "idIndex" : { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_" } }, "ts" : Timestamp(1652753057, 1), "t" : NumberLong(12), "v" : NumberLong(2), "wall" : ISODate("2022-05-17T02:04:17.159Z") }
{ "op" : "c", "ns" : "ustc.$cmd", "ui" : UUID("91bef375-c3b3-4416-94bd-946b57163c8e"), "o" : { "create" : "dave1", "idIndex" : { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_" } }, "ts" : Timestamp(1652753060, 1), "t" : NumberLong(12), "v" : NumberLong(2), "wall" : ISODate("2022-05-17T02:04:20.105Z") }
{ "op" : "c", "ns" : "ustc.$cmd", "ui" : UUID("233b8dc2-66d4-404a-a8d9-0971af648619"), "o" : { "drop" : "dave" }, "o2" : { "numRecords" : 0 }, "ts" : Timestamp(1652753076, 1), "t" : NumberLong(12), "v" : NumberLong(2), "wall" : ISODate("2022-05-17T02:04:36.368Z") }
{ "op" : "c", "ns" : "ustc.$cmd", "ui" : UUID("91bef375-c3b3-4416-94bd-946b57163c8e"), "o" : { "drop" : "dave1" }, "o2" : { "numRecords" : 0 }, "ts" : Timestamp(1652753079, 1), "t" : NumberLong(12), "v" : NumberLong(2), "wall" : ISODate("2022-05-17T02:04:39.269Z") }
shard1:PRIMARY>
版权声明：本文为博主原创文章，未经博主允许不得转载。
