签到成功

知道了

CNDBA社区CNDBA社区

MongoDB 性能测试工具 sysbench-mongodb

2022-05-13 13:43 3403 0 原创 MongoDB
作者： dave
本文链接：http://www.cndba.cn/cndba/dave/article/107997

1 Sysbench-mongodb 说明

可以使用sysbench-mongodb 对MongoDB 进行性能测试：

https://github.com/tmcallaghan/sysbench-mongodb
https://gitee.com/mirrors/sysbench-mongodb

http://www.cndba.cn/cndba/dave/article/107997

默认情况下，benchmark 会创建10个集合，每个集合100万的数据。 可以根据具体的资源情况，修改配置文件：config.bash。http://www.cndba.cn/cndba/dave/article/107997

2 操作示例

2.1 下载sysbench-mongodb

[dave@www.cndba.cn_2 ~]# git clone https://github.com/tmcallaghan/sysbench-mongodb.git
Cloning into 'sysbench-mongodb'...
remote: Enumerating objects: 132, done.
remote: Counting objects: 100% (39/39), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 132 (delta 18), reused 24 (delta 10), pack-reused 93
Receiving objects: 100% (132/132), 72.26 KiB | 0 bytes/s, done.
Resolving deltas: 100% (61/61), done.
[dave@www.cndba.cn_2 ~]# ll
total 8
-rw-------. 1 root root 1431 May 14  2020 anaconda-ks.cfg
-rw-------  1 root root 2707 May  4 16:17 nohup.out
drwxr-xr-x  4 root root  182 May 13 09:26 sysbench-mongodb
[dave@www.cndba.cn_2 ~]# cd sysbench-mongodb/
[dave@www.cndba.cn_2 sysbench-mongodb]# ll
total 40
-rwxr-xr-x 1 root root    49 May 13 09:26 check-results.bash
-rw-r--r-- 1 root root  4805 May 13 09:26 config.bash
-rw-r--r-- 1 root root 11357 May 13 09:26 LICENSE
-rw-r--r-- 1 root root  1892 May 13 09:26 README.md
-rwxr-xr-x 1 root root  3705 May 13 09:26 run.looped.bash
-rwxr-xr-x 1 root root  3022 May 13 09:26 run.simple.bash
drwxr-xr-x 2 root root    71 May 13 09:26 src
-rw-r--r-- 1 root root   124 May 13 09:26 TODO
[dave@www.cndba.cn_2 sysbench-mongodb]# pwd
/root/sysbench-mongodb
[dave@www.cndba.cn_2 sysbench-mongodb]#

2.2 准备工作

使用sysbench-mongodb 有如下要求：

http://www.cndba.cn/cndba/dave/article/107997

  1. Java 版本要求17
  2. 下载mongoDB Java 驱动要求 3.9.1，并放到当前目录（sysbench-mongodb）下。
  3. 修改配置文件：config.bash

[dave@www.cndba.cn_2 sysbench-mongodb]# yum install java-1.7.0-openjdk*

http://www.cndba.cn/cndba/dave/article/107997

2.3 修改配置文件：config.bash

[dave@www.cndba.cn_2 sysbench-mongodb]# cat config.bash
# You will most likely want to change these values

# database in which to run the benchmark
export DB_NAME=admin

# database username on DB_NAME
export USERNAME=root

# database password to use for USERNAME
export PASSWORD=root

# name of the server to connect to
#export MONGO_SERVER="172.31.185.120:27017,172.31.185.165:27017,172.31.185.131"
export MONGO_SERVER="172.31.185.165"

# port of the server to connect to
export MONGO_PORT=27017

……
# total number of simultaneous insertion threads (for loader)
#   valid values : integer > 0
export NUM_LOADER_THREADS=10

# total number of simultaneous benchmark threads
#   valid values : integer > 0
export NUM_WRITER_THREADS=32
……

这里几个重要的参数如下：http://www.cndba.cn/cndba/dave/article/107997

NUM_COLLECTIONS=8      #集合数量
NUM_DOCUMENTS_PER_COLLECTION=1000000      #每集合的文档数量
NUM_DOCUMENTS_PER_INSERT=1000      #每次Insert多少个文档
NUM_LOADER_THREADS=8      #Loaddata时的进程数量
RUN_TIME_MINUTES=10      #负载测试运行多少分钟
NUM_SECONDS_PER_FEEDBACK=10      #每次输出的间隔秒数
MONGO_SERVER=localhost      #mongodb服务器
MONGO_PORT=27017      #mongodb端口号
SYSBENCH_READ_ONLY=N      #是否只读测试

但是在实际测试中发现：http://www.cndba.cn/cndba/dave/article/107997

1.    DB_NAME 只能是admin，否则认证时会报如下错误：
Thread[main,5,main]  connection string = mongodb://cndba:cndba@172.31.185.165:27017/admin?ssl=false&replicaSet=rs0&readpreference=primary
2.    只能测试名称为：‘rs0’ 的副本集，不支持Shard Cluster，否则会报如下错误：
SEVERE: Expecting replica set member, but found a SHARD_ROUTER.
SEVERE: Expecting replica set member from set 'rs0', but found one from set 'shard1'.
3.    MONGO_SERVER 写Primary 的IP 即可。
4.    这里insert 的时间是根据数据量来的，默认是100w，每秒insert 1000条数据，所以insert 就需要10十分钟。 RUN_TIME_MINUTES 只是针对压完数据之后的测试。 如果不想等很长时间，可以修改数据量大小。

2.4 执行压测命令

如果只有一个config.bash配置文件，可以直接执行

./run.simple.bashhttp://www.cndba.cn/cndba/dave/article/107997

如果有多个文件，可以在命令后面指定参数文件：

./run.simple.bash my_custom_config.bash

[dave@www.cndba.cn_2 sysbench-mongodb]# ./run.simple.bash
Loading config from config.bash.....
Note: src/jmongosysbenchload.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: src/jmongosysbenchexecute.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Do query at Fri May 13 11:44:33 CST 2022
Thread[main,5,main]Application Parameters
Thread[main,5,main]-------------------------------------------------------------------------------------------------
Thread[main,5,main]  collections              = 10
Thread[main,5,main]  database name            = admin
Thread[main,5,main]  writer threads           = 4
Thread[main,5,main]  documents per collection = 1,000,000
Thread[main,5,main]  feedback seconds         = 10
Thread[main,5,main]  log file                 = mongoSysbenchExecute-10-1000000-4.txt.tsv
Thread[main,5,main]  auto commit              = Y
Thread[main,5,main]  run seconds              = 600
Thread[main,5,main]  oltp range size          = 100
Thread[main,5,main]  oltp point selects       = 10
Thread[main,5,main]  oltp simple ranges       = 1
Thread[main,5,main]  oltp sum ranges          = 1
Thread[main,5,main]  oltp order ranges        = 1
Thread[main,5,main]  oltp distinct ranges     = 1
Thread[main,5,main]  oltp index updates       = 1
Thread[main,5,main]  oltp non index updates   = 1
Thread[main,5,main]  oltp inserts             = 1
Thread[main,5,main]  write concern            = ACKNOWLEDGED
Thread[main,5,main]  maximum tps (global)     = 999999
Thread[main,5,main]  maximum tps (per thread) = 250000
Thread[main,5,main]  Server:Port = 172.31.185.165:27017
Thread[main,5,main]  seed                     = 1652413470
Thread[main,5,main]  userName                 = root
Thread[main,5,main]  read preference          = secondaryPreferred
Thread[main,5,main]  use SSL                  = false
May 13, 2022 11:44:34 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Cluster created with settings {hosts=[172.31.185.165:27017], mode=MULTIPLE, requiredClusterType=REPLICA_SET, serverSelectionTimeout='30000 ms', maxWaitQueueSize=20480, requiredReplicaSetName='rs0'}
May 13, 2022 11:44:34 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Adding discovered server 172.31.185.165:27017 to client view of cluster
May 13, 2022 11:44:34 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: No server chosen by com.mongodb.client.internal.MongoClientDelegate$1@34704d9 from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=172.31.185.165:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
May 13, 2022 11:44:34 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Opened connection [connectionId{localValue:1, serverValue:16}] to 172.31.185.165:27017
May 13, 2022 11:44:34 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Monitor thread successfully connected to server with description ServerDescription{address=172.31.185.165:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[5, 0, 8]}, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4914819, setName='rs0', canonicalAddress=172.31.185.165:27017, hosts=[172.31.185.120:27017, 172.31.185.131:27017, 172.31.185.165:27017], passives=[], arbiters=[], primary='172.31.185.131:27017', tagSet=TagSet{[]}, electionId=null, setVersion=3, lastWriteDate=Fri May 13 11:44:33 CST 2022, lastUpdateTimeNanos=758032504530000}
May 13, 2022 11:44:34 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Adding discovered server 172.31.185.120:27017 to client view of cluster
May 13, 2022 11:44:34 AM com.mongodb.diagnostics.logging.JULLogger log
INFO: Adding discovered server 172.31.185.131:27017 to client view of cluster
May 13, 2022 11:44:34 AM com.mongodb.diagnostics.logging.JULLogger log
……
Thread[Thread-1,5,main]7,834,000 inserts : 790 seconds : cum ips=9,916.36 : int ips=5,929.65
Thread[Thread-1,5,main]7,885,000 inserts : 800 seconds : cum ips=9,855.61 : int ips=5,077.66
Thread[Thread-1,5,main]7,945,000 inserts : 810 seconds : cum ips=9,807.96 : int ips=5,997.60
Thread[Thread-1,5,main]7,997,000 inserts : 820 seconds : cum ips=9,751.39 : int ips=5,183.41
Thread[Thread-1,5,main]8,053,000 inserts : 830 seconds : cum ips=9,702.12 : int ips=5,635.50
Thread[Thread-1,5,main]8,102,000 inserts : 840 seconds : cum ips=9,644.26 : int ips=4,870.78
Thread[Thread-1,5,main]8,154,000 inserts : 850 seconds : cum ips=9,592.74 : int ips=5,235.08
Thread[Thread-1,5,main]8,207,000 inserts : 860 seconds : cum ips=9,542.41 : int ips=5,280.46
……

************************************************************************
final  interval(s)
************************************************************************
Thread[Thread-2,5,main]Writer thread 1 : started
Thread[Thread-3,5,main]Writer thread 2 : started
Thread[Thread-4,5,main]Writer thread 3 : started
Thread[Thread-5,5,main]10 seconds : cum tps=158.50 : int tps=158.50 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]20 seconds : cum tps=211.44 : int tps=264.19 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]30 seconds : cum tps=226.47 : int tps=256.73 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]40 seconds : cum tps=225.70 : int tps=223.40 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]50 seconds : cum tps=233.56 : int tps=264.83 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]60 seconds : cum tps=237.31 : int tps=256.00 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]70 seconds : cum tps=242.06 : int tps=270.80 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]80 seconds : cum tps=247.39 : int tps=284.78 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]90 seconds : cum tps=249.60 : int tps=267.26 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]100 seconds : cum tps=250.68 : int tps=260.34 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]110 seconds : cum tps=253.63 : int tps=283.29 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]120 seconds : cum tps=249.93 : int tps=209.13 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]130 seconds : cum tps=252.19 : int tps=279.40 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]140 seconds : cum tps=253.58 : int tps=271.61 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]150 seconds : cum tps=255.14 : int tps=276.88 : cum ips=0.00 : int ips=0.00 : writers=4
Thread[Thread-5,5,main]160 seconds : cum tps=253.55 : int tps=229.46 : cum ips=0.00 : int ips=0.00 : writers=4

Fri May 13 11:47:16 CST 2022 | sysbench benchmark duration = 00:00:02:43

参数说明：

  1. cum tps： 是总体的平均每秒事务数量 ，也就是总事务/秒数
  2. int tps ：增量的平均每秒事务数量，也就是本次10秒内的平均每秒事务数量
  3. cum ips： 是总体的平均每秒insert数量 ，也就是总inserts/秒数
  4. int ips ：增量平均每秒insert数量，也就是本次10秒内的平均每秒insert数量

注意，这里只有开启DOLOAD=YES 才会有insert 数据，并且是分开测试的。 如果不想等很久，调小每个集合的数据量：

Thread[Thread-1,5,main]710,000 inserts : 20 seconds : cum ips=35,422.07 : int ips=43,042.04
Thread[Thread-1,5,main]1,057,000 inserts : 30 seconds : cum ips=35,195.79 : int ips=34,741.69
Thread[Thread-1,5,main]1,293,000 inserts : 40 seconds : cum ips=32,308.04 : int ips=23,625.99
Thread[Thread-1,5,main]1,526,000 inserts : 50 seconds : cum ips=30,459.08 : int ips=23,117.37
Thread[Thread-1,5,main]1,736,000 inserts : 60 seconds : cum ips=28,915.98 : int ips=21,135.27

如果要重复测试，需要先删除旧数据

rs0:PRIMARY> show tables
sbtest1
sbtest10
sbtest2
sbtest3
sbtest4
sbtest5
sbtest6
sbtest7
sbtest8
sbtest9
system.keys
system.users
system.version
rs0:PRIMARY> db.sbtest1.drop()
true

其实开始是用32个写进程测试的，但是直接把primary 进程搞挂了，后来改成4个，才稳定运行：

http://www.cndba.cn/cndba/dave/article/107997
http://www.cndba.cn/cndba/dave/article/107997http://www.cndba.cn/cndba/dave/article/107997

Thread[Thread-33,5,main]360 seconds : cum tps=402.66 : int tps=462.67 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]370 seconds : cum tps=404.62 : int tps=475.03 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]380 seconds : cum tps=406.37 : int tps=471.26 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]390 seconds : cum tps=407.20 : int tps=438.74 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]400 seconds : cum tps=408.05 : int tps=441.14 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]410 seconds : cum tps=408.96 : int tps=445.25 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]420 seconds : cum tps=410.46 : int tps=471.97 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]430 seconds : cum tps=411.59 : int tps=459.15 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]440 seconds : cum tps=412.48 : int tps=450.76 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]450 seconds : cum tps=413.52 : int tps=459.08 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]460 seconds : cum tps=414.62 : int tps=463.99 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]470 seconds : cum tps=414.60 : int tps=413.77 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]480 seconds : cum tps=415.53 : int tps=459.32 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]490 seconds : cum tps=416.78 : int tps=476.92 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]500 seconds : cum tps=417.81 : int tps=467.80 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]510 seconds : cum tps=417.50 : int tps=401.90 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]520 seconds : cum tps=418.36 : int tps=462.20 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]530 seconds : cum tps=419.24 : int tps=465.14 : cum ips=0.00 : int ips=0.00 : writers=32
Thread[Thread-33,5,main]540 seconds : cum tps=419.86 : int tps=453.06 : cum ips=0.00 : int ips=0.00 : writers=32

版权声明：本文为博主原创文章，未经博主允许不得转载。

用户评论
* 以下用户言论只代表其个人观点，不代表CNDBA社区的观点或立场
dave

dave

关注

人的一生应该是这样度过的：当他回首往事的时候，他不会因为虚度年华而悔恨，也不会因为碌碌无为而羞耻；这样，在临死的时候，他就能够说：“我的整个生命和全部精力，都已经献给世界上最壮丽的事业....."

  • 2296
    原创
  • 3
    翻译
  • 582
    转载
  • 198
    评论
  • 访问：8508280次
  • 积分：4477
  • 等级：核心会员
  • 排名：第1名
精华文章

    达梦数据库运维实战

    Oracle 18c 必须掌握的新特性：管理与实战

    Oracle数据库问题解决方案和故障排除手册
    最新问题
    查看更多+
    热门文章
      热门用户
      推荐用户
        友情链接： CNDBA社区 | openGauss | openEuler | openAnolis | 广告联系: database@ustc.edu
        Copyright © 2016 All Rights Reserved. Powered by CNDBA · 皖ICP备2022006297号-1·

        QQ交流群

        注册联系QQ