签到成功

知道了

CNDBA社区CNDBA社区

MongoDB 性能测试工具 YCSB 使用说明

2022-05-15 14:30 5480 1 原创 MongoDB
作者： dave
本文链接：http://www.cndba.cn/cndba/dave/article/108004

1 YCSB 工具概述

在之前的博客中我们了解了sysbench-mongodb 工具，如下：

MongoDB 性能测试工具 sysbench-mongodb
https://www.cndba.cn/dave/article/107997

本片我们继续学习MongoDB的压测工作：YCSB。

YCSB(Yahoo! Cloud Serving Benchmark) 是一个Java语言实现的主要用于云端或者服务器端的数据库性能测试工具，其内部涵盖了常见的NoSQL数据库产品。

官方地址:
https://github.com/brianfrankcooper/YCSB

从官网可以看出，目前支持的数据库类型比较多，主流的有如下：couchbase、elasticsearch、hbase、kudu、memcached、mongodb、postgrenosql、redis、rocksdb等等。

工具的具体使用参考官方手册：

https://github.com/brianfrankcooper/YCSB/wiki/Running-a-Workload

完整的数据库类型，在帮助中有显示：

[dave@www.cndba.cn_3 bin]# ./ycsb -help
usage: ./ycsb command database [options]

Commands:
    load           Execute the load phase
    run            Execute the transaction phase
    shell          Interactive mode

Databases:
    accumulo       https://github.com/brianfrankcooper/YCSB/tree/master/accumulo
    accumulo1.6    https://github.com/brianfrankcooper/YCSB/tree/master/accumulo1.6
    accumulo1.7    https://github.com/brianfrankcooper/YCSB/tree/master/accumulo1.7
    accumulo1.8    https://github.com/brianfrankcooper/YCSB/tree/master/accumulo1.8
    aerospike      https://github.com/brianfrankcooper/YCSB/tree/master/aerospike
    arangodb       https://github.com/brianfrankcooper/YCSB/tree/master/arangodb
    arangodb3      https://github.com/brianfrankcooper/YCSB/tree/master/arangodb3
    asynchbase     https://github.com/brianfrankcooper/YCSB/tree/master/asynchbase
    azurecosmos    https://github.com/brianfrankcooper/YCSB/tree/master/azurecosmos
    azuretablestorage https://github.com/brianfrankcooper/YCSB/tree/master/azuretablestorage
    basic          https://github.com/brianfrankcooper/YCSB/tree/master/basic
    basicts        https://github.com/brianfrankcooper/YCSB/tree/master/basicts
    cassandra-cql  https://github.com/brianfrankcooper/YCSB/tree/master/cassandra
    cassandra2-cql https://github.com/brianfrankcooper/YCSB/tree/master/cassandra2
    cloudspanner   https://github.com/brianfrankcooper/YCSB/tree/master/cloudspanner
    couchbase      https://github.com/brianfrankcooper/YCSB/tree/master/couchbase
    couchbase2     https://github.com/brianfrankcooper/YCSB/tree/master/couchbase2
    crail          https://github.com/brianfrankcooper/YCSB/tree/master/crail
    dynamodb       https://github.com/brianfrankcooper/YCSB/tree/master/dynamodb
    elasticsearch  https://github.com/brianfrankcooper/YCSB/tree/master/elasticsearch
    elasticsearch5 https://github.com/brianfrankcooper/YCSB/tree/master/elasticsearch5
    elasticsearch5-rest https://github.com/brianfrankcooper/YCSB/tree/master/elasticsearch5
    foundationdb   https://github.com/brianfrankcooper/YCSB/tree/master/foundationdb
    geode          https://github.com/brianfrankcooper/YCSB/tree/master/geode
    googlebigtable https://github.com/brianfrankcooper/YCSB/tree/master/googlebigtable
    googledatastore https://github.com/brianfrankcooper/YCSB/tree/master/googledatastore
    griddb         https://github.com/brianfrankcooper/YCSB/tree/master/griddb
    hbase098       https://github.com/brianfrankcooper/YCSB/tree/master/hbase098
    hbase10        https://github.com/brianfrankcooper/YCSB/tree/master/hbase10
    hbase12        https://github.com/brianfrankcooper/YCSB/tree/master/hbase12
    hbase14        https://github.com/brianfrankcooper/YCSB/tree/master/hbase14
    hbase20        https://github.com/brianfrankcooper/YCSB/tree/master/hbase20
    hypertable     https://github.com/brianfrankcooper/YCSB/tree/master/hypertable
    ignite         https://github.com/brianfrankcooper/YCSB/tree/master/ignite
    ignite-sql     https://github.com/brianfrankcooper/YCSB/tree/master/ignite
    infinispan     https://github.com/brianfrankcooper/YCSB/tree/master/infinispan
    infinispan-cs  https://github.com/brianfrankcooper/YCSB/tree/master/infinispan
    jdbc           https://github.com/brianfrankcooper/YCSB/tree/master/jdbc
    kudu           https://github.com/brianfrankcooper/YCSB/tree/master/kudu
    maprdb         https://github.com/brianfrankcooper/YCSB/tree/master/maprdb
    maprjsondb     https://github.com/brianfrankcooper/YCSB/tree/master/maprjsondb
    memcached      https://github.com/brianfrankcooper/YCSB/tree/master/memcached
    mongodb        https://github.com/brianfrankcooper/YCSB/tree/master/mongodb
    mongodb-async  https://github.com/brianfrankcooper/YCSB/tree/master/mongodb
    nosqldb        https://github.com/brianfrankcooper/YCSB/tree/master/nosqldb
    orientdb       https://github.com/brianfrankcooper/YCSB/tree/master/orientdb
    postgrenosql   https://github.com/brianfrankcooper/YCSB/tree/master/postgrenosql
    rados          https://github.com/brianfrankcooper/YCSB/tree/master/rados
    redis          https://github.com/brianfrankcooper/YCSB/tree/master/redis
    rest           https://github.com/brianfrankcooper/YCSB/tree/master/rest
    riak           https://github.com/brianfrankcooper/YCSB/tree/master/riak
    rocksdb        https://github.com/brianfrankcooper/YCSB/tree/master/rocksdb
    s3             https://github.com/brianfrankcooper/YCSB/tree/master/s3
    solr           https://github.com/brianfrankcooper/YCSB/tree/master/solr
    solr6          https://github.com/brianfrankcooper/YCSB/tree/master/solr6
    tablestore     https://github.com/brianfrankcooper/YCSB/tree/master/tablestore
    tarantool      https://github.com/brianfrankcooper/YCSB/tree/master/tarantool

Options:
    -P file        Specify workload file
    -cp path       Additional Java classpath entries
    -jvm-args args Additional arguments to the JVM
    -p key=value   Override workload property
    -s             Print status to stderr
    -target n      Target ops/sec (default: unthrottled)
    -threads n     Number of client threads (default: 1)

Workload Files:
    There are various predefined workloads under workloads/ directory.
    See https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties
    for the list of workload properties.
ycsb: error: argument -h/--help: ignored explicit argument 'elp'
[dave@www.cndba.cn_3 bin]#

2 压测示例

我们这里看MongoDB的压测。 官网说明如下：

https://github.com/brianfrankcooper/YCSB/tree/master/mongodb

2.1 准备工作

安装MongoDB 环境，这里参考之前的博客：

MongoDB 4.4 分片集群（3 shard） 搭建手册
https://www.cndba.cn/dave/article/107970

安装Java：要JDK 8 或者 11 的版本：
Java 环境之前用Yum 安装过了。 这里直接忽略。http://www.cndba.cn/cndba/dave/article/108004http://www.cndba.cn/cndba/dave/article/108004

[dave@www.cndba.cn_3 bin]# java -version
openjdk version "11.0.15" 2022-04-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.15+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+9-LTS, mixed mode, sharing)
[dave@www.cndba.cn_3 bin]#

安装Maven：

[dave@www.cndba.cn_3 ~]# wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.tar.gz
[dave@www.cndba.cn_3 ~]# tar xzf apache-maven-3.1.1-bin.tar.gz -C /usr/local
[dave@www.cndba.cn_3 local]# ln -s apache-maven-3.1.1 maven
[dave@www.cndba.cn_3 local]# cat /etc/profile.d/maven.sh
export M2_HOME=/usr/local/maven
export PATH=${M2_HOME}/bin:${PATH}
[dave@www.cndba.cn_3 local]# bash
 [dave@www.cndba.cn_3 local]# mvn -version
Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 23:22:22+0800)
Maven home: /usr/local/maven
Java version: 1.7.0_261, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.261-2.6.22.2.el7_8.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1127.el7.x86_64", arch: "amd64", family: "unix"
[dave@www.cndba.cn_3 local]#

下载YCSB
这里要注意官网文档里的地址比较旧，目前最新的是0.17，文档里用的还是0.5。我们直接从官网下载最新的版本。

https://github.com/brianfrankcooper/YCSB/releases

[dave@www.cndba.cn_3 ~]# wget https://github.com/brianfrankcooper/YCSB/releases/download/0.17.0/ycsb-mongodb-binding-0.17.0.tar.gz
[dave@www.cndba.cn_3 ~]# tar xzvf ycsb-mongodb-binding-0.17.0.tar.gz
[dave@www.cndba.cn_3 ~]# cd ycsb-mongodb-binding-0.17.0/

2.2 修改压测文件

YCSB 的配置文件都在workloads 目录下：http://www.cndba.cn/cndba/dave/article/108004

[dave@www.cndba.cn_3 workloads]# pwd
/root/ycsb-mongodb-binding-0.17.0/workloads
[dave@www.cndba.cn_3 workloads]# ll
total 48
-rw-r--r-- 1 502 games  1464 Sep 26  2019 tsworkloada
-rw-r--r-- 1 502 games 11968 Sep 26  2019 tsworkload_template
-rw-r--r-- 1 502 games  3010 Sep 26  2019 workloada
-rw-r--r-- 1 502 games  3046 Sep 26  2019 workloadb
-rw-r--r-- 1 502 games  3032 Sep 26  2019 workloadc
-rw-r--r-- 1 502 games  3397 Sep 26  2019 workloadd
-rw-r--r-- 1 502 games  3554 Sep 26  2019 workloade
-rw-r--r-- 1 502 games  3147 Sep 26  2019 workloadf
-rw-r--r-- 1 502 games  7370 Sep 26  2019 workload_template
[dave@www.cndba.cn_3 workloads]#

我们这里面主要是修改workloada 文件，从目录看，这里面实际上一直到wordloadf。 他们的区别就是参数中操作的比例不同。Workloada中，read 和 update 各占50%。http://www.cndba.cn/cndba/dave/article/108004

http://www.cndba.cn/cndba/dave/article/108004

  1. workloada：读写均衡型，50%/50%，Reads/Writes
  2. workloadb：读多写少型，95%/5%，Reads/Writes
  3. workloadc：只读型，100%，Reads
  4. workloadd：读最近写入记录型，95%/5%，Reads/insert
  5. workloade：扫描小区间型，95%/5%，scan/insert
  6. workloadf：读写入记录均衡型，50%/50%，Reads/insert
[dave@www.cndba.cn_3 workloads]# cat workloada |grep -v '^#'|grep -v '^$'
recordcount=1000
operationcount=1000
workload=site.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0
requestdistribution=zipfian
[dave@www.cndba.cn_3 workloads]#

完整的参数模板如下：http://www.cndba.cn/cndba/dave/article/108004

[dave@www.cndba.cn_3 workloads]# cat workload_template |grep -v '^#'|grep -v '^$'
workload=site.ycsb.workloads.CoreWorkload
recordcount=1000000
operationcount=3000000
insertstart=0
fieldcount=10
fieldlength=100
readallfields=true
writeallfields=false
fieldlengthdistribution=constant
readproportion=0.95
updateproportion=0.05
insertproportion=0
readmodifywriteproportion=0
scanproportion=0
maxscanlength=1000
scanlengthdistribution=uniform
insertorder=hashed
requestdistribution=zipfian
hotspotdatafraction=0.2
hotspotopnfraction=0.8
table=usertable
measurementtype=histogram
histogram.buckets=1000
timeseries.granularity=1000

相关说明如下：

  1. recordcount：YCSB load阶段加载的记录条数
  2. operationcount：YCSB run阶段执行的操作总数
  3. workload：workload实现类
  4. requestdistribution：请求分布模式，uniform, zipfian, latest三种分布模式
  5. mongodb.url：待测试mongo实例的连接地址 （注：此为添加项）
  6. mongodb.database：测试时使用的数据库名称，默认ycsb（注：此为添加项）
  7. fieldcount: 每条记录字段个数 (default: 10)
  8. fieldlength: 每个字段长度 (default: 100)
  9. readallfields: 是否读取所有字段true或者读取一个字段false (default: true)
  10. readproportion: 读取作业比例 (default: 0.95)
  11. updateproportion: 更新作业比例 (default: 0.05)
  12. insertproportion: 插入作业比例 (default: 0)
  13. scanproportion: 扫描作业比例 (default: 0)
  14. readmodifywriteproportion: 读取一条记录修改它并写回的比例 (default: 0)
  15. requestdistribution: 请求的分布规则 uniform, zipfian or latest (default: uniform)
  16. maxscanlength: 扫描作业最大记录数 (default: 1000)
  17. scanlengthdistribution: 在1和最大扫描记录数的之间的分布规则 (default: uniform)
  18. insertorder: 记录被插入的规则ordered或者hashed (default: hashed)
  19. maxexecutiontime: 执行操作的最长时间，当然如果没有超过这个时间以运行时间为主。
  20. table: 测试表的名称 (default: usertable)

我们这里复制workloada来创建新的测试文件：

[dave@www.cndba.cn_3 workloads]# cp workloada ustc
[dave@www.cndba.cn_3 workloads]# vim ustc
[dave@www.cndba.cn_3 workloads]# cat ustc |grep -v '^#'|grep -v '^$'
mongodb.url=mongodb://root:root@172.31.185.120:27000,172.31.185.165:27000,172.31.185.131:27000
mongodb.database=cndba
table=dave
recordcount=100000
operationcount=500000
workload=site.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0
requestdistribution=zipfian
[dave@www.cndba.cn_3 workloads]#

2.3 执行YCSB 命令

2.3.1 帮助命令

YCSB 命令在软件目录的bin路径下。 

[dave@www.cndba.cn_3 bin]# ./ycsb -h
usage: ./ycsb command database [options]

Commands:
    load           Execute the load phase
    run            Execute the transaction phase
shell          Interactive mode
……
Options:
    -P file        Specify workload file
    -cp path       Additional Java classpath entries
    -jvm-args args Additional arguments to the JVM
    -p key=value   Override workload property
    -s             Print status to stderr
    -target n      Target ops/sec (default: unthrottled)
    -threads n     Number of client threads (default: 1)

Workload Files:
    There are various predefined workloads under workloads/ directory.
    See https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties
    for the list of workload properties.

positional arguments:
  {load,run,shell}      Command to run.
  {accumulo,accumulo1.6,accumulo1.7,accumulo1.8,aerospike,arangodb,arangodb3,asynchbase,azurecosmos,azuretablestorage,basic,basicts,cassandra-cql,cassandra2-cql,cloudspanner,couchbase,couchbase2,crail,dynamodb,elasticsearch,elasticsearch5,elasticsearch5-rest,foundationdb,geode,googlebigtable,googledatastore,griddb,hbase098,hbase10,hbase12,hbase14,hbase20,hypertable,ignite,ignite-sql,infinispan,infinispan-cs,jdbc,kudu,maprdb,maprjsondb,memcached,mongodb,mongodb-async,nosqldb,orientdb,postgrenosql,rados,redis,rest,riak,rocksdb,s3,solr,solr6,tablestore,tarantool}
                        Database to test.
……

实际上，对于mongodb 的测试，YCSB 有两种驱动：

mongodb:site.ycsb.db.MongoDbClient
mongodb-async:site.ycsb.db.AsyncMongoDbClient

从字面看，一个是同步，一个是异步。 但具体的区别并没有描述。

异步：http://www.cndba.cn/cndba/dave/article/108004

./bin/ycsb load mongodb-async -s -P workloads/workloada > outputLoad.txt
./bin/ycsb run mongodb-async -s -P workloads/workloada > outputRun.txt

同步：

./bin/ycsb load mongodb -s -P workloads/workloada > outputLoad.txt
./bin/ycsb run mongodb -s -P workloads/workloada > outputRun.txt

我们在上节是配置了压测文件，如果没有配置，也可以直接使用-p 来指定。-p设置ycsb配置项，会覆盖workload文件的配置项。

比如：http://www.cndba.cn/cndba/dave/article/108004

bin/ycsb.sh load mongodb -s -P workloads/workloada -p recordcount=3000000 –threads 16  mongodb.url="mongodb://user:pwd@server1.example.com:9999,server2.example.com:9999/dbname" -p mongodb.auth="true" > ~/outputLoad.txt
bin/ycsb.sh run mongodb -s -P workloads/workloada -p operationcount=10000000 -threads 2 mongodb.url="mongodb://user:pwd@server1.example.com:9999,server2.example.com:9999/dbname" -p mongodb.auth="true" > ~/outputRun.txt

2.3.2 加载数据

我们这里以同步驱动来进行测试。

在加载之前要确保ycsb的库不存在，否则会报如下错误：

E11000 duplicate key error collection: ycsb.dave index: id dup key:

mongos> show dbs
admin   0.006GB
cndba   0.006GB
config  0.003GB
test    0.000GB
ycsb    0.027GB
mongos> use ycsb
switched to db ycsb
mongos> db.dropDatabase()

加载数据：http://www.cndba.cn/cndba/dave/article/108004

[dave@www.cndba.cn_3 bin]# pwd
/root/ycsb-mongodb-binding-0.17.0/bin
[dave@www.cndba.cn_3 bin]#  ./ycsb load mongodb -s -P ../workloads/ustc -threads 8
java -cp /root/ycsb-mongodb-binding-0.17.0/mongodb-binding/conf:/root/ycsb-mongodb-binding-0.17.0/conf:/root/ycsb-mongodb-binding-0.17.0/lib/core-0.17.0.jar:/root/ycsb-mongodb-binding-0.17.0/lib/htrace-core4-4.1.0-incubating.jar:/root/ycsb-mongodb-binding-0.17.0/lib/jackson-mapper-asl-1.9.4.jar:/root/ycsb-mongodb-binding-0.17.0/lib/jackson-core-asl-1.9.4.jar:/root/ycsb-mongodb-binding-0.17.0/lib/HdrHistogram-2.1.4.jar:/root/ycsb-mongodb-binding-0.17.0/lib/mongo-java-driver-3.8.0.jar:/root/ycsb-mongodb-binding-0.17.0/lib/mongodb-async-driver-2.0.1.jar:/root/ycsb-mongodb-binding-0.17.0/lib/logback-classic-1.1.2.jar:/root/ycsb-mongodb-binding-0.17.0/lib/logback-core-1.1.2.jar:/root/ycsb-mongodb-binding-0.17.0/lib/slf4j-api-1.7.25.jar:/root/ycsb-mongodb-binding-0.17.0/lib/snappy-java-1.1.7.1.jar:/root/ycsb-mongodb-binding-0.17.0/lib/mongodb-binding-0.17.0.jar site.ycsb.Client -db site.ycsb.db.MongoDbClient -s -P ../workloads/ustc -threads 8 -load
Command line: -db site.ycsb.db.MongoDbClient -s -P ../workloads/ustc -threads 8 -load
YCSB Client 0.17.0

Loading workload...
Starting test.
2022-05-15 13:47:30:515 0 sec: 0 operations; est completion in 0 second
mongo client connection created with mongodb://root:root@172.31.185.120:27000,172.31.185.165:27000,172.31.185.131:27000
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
2022-05-15 13:47:40:467 10 sec: 4623 operations; 462.3 current ops/sec; est completion in 3 minutes [INSERT: Count=4632, Max=822783, Min=5428, Avg=15627.85, 90=21215, 99=37087, 99.9=769023, 99.99=822783]
2022-05-15 13:47:50:475 20 sec: 10868 operations; 624.5 current ops/sec; est completion in 2 minutes [INSERT: Count=6243, Max=99967, Min=4816, Avg=12816.97, 90=19023, 99=40351, 99.9=99583, 99.99=99903]
2022-05-15 13:48:00:476 30 sec: 17485 operations; 661.7 current ops/sec; est completion in 2 minutes [INSERT: Count=6627, Max=84607, Min=4740, Avg=12066.67, 90=18287, 99=33791, 99.9=83711, 99.99=84543]
……
2022-05-15 13:50:30:452 180 sec: 88570 operations; 825.4 current ops/sec; est completion in 24 seconds [INSERT: Count=8252, Max=86271, Min=4244, Avg=9688.53, 90=12919, 99=24319, 99.9=79423, 99.99=86143]
2022-05-15 13:50:40:452 190 sec: 95044 operations; 647.4 current ops/sec; est completion in 10 seconds [INSERT: Count=6474, Max=125887, Min=4620, Avg=12329.96, 90=18639, 99=47519, 99.9=93183, 99.99=125823]
2022-05-15 13:50:48:866 198 sec: 100000 operations; 588.95 current ops/sec; [CLEANUP: Count=8, Max=148095, Min=2, Avg=18508.12, 90=19, 99=148095, 99.9=148095, 99.99=148095] [INSERT: Count=4956, Max=101247, Min=4772, Avg=12517.37, 90=21631, 99=44991, 99.9=92927, 99.99=101247]
[OVERALL], RunTime(ms), 198416
[OVERALL], Throughput(ops/sec), 503.99161357955006
[TOTAL_GCS_G1_Young_Generation], Count, 85
[TOTAL_GC_TIME_G1_Young_Generation], Time(ms), 426
[TOTAL_GC_TIME_%_G1_Young_Generation], Time(%), 0.21470042738488834
[TOTAL_GCS_G1_Old_Generation], Count, 0
[TOTAL_GC_TIME_G1_Old_Generation], Time(ms), 0
[TOTAL_GC_TIME_%_G1_Old_Generation], Time(%), 0.0
[TOTAL_GCs], Count, 85
[TOTAL_GC_TIME], Time(ms), 426
[TOTAL_GC_TIME_%], Time(%), 0.21470042738488834
[CLEANUP], Operations, 8
[CLEANUP], AverageLatency(us), 18508.125
[CLEANUP], MinLatency(us), 2
[CLEANUP], MaxLatency(us), 148095
[CLEANUP], 95thPercentileLatency(us), 148095
[CLEANUP], 99thPercentileLatency(us), 148095
[INSERT], Operations, 100000
[INSERT], AverageLatency(us), 14928.95763
[INSERT], MinLatency(us), 4076
[INSERT], MaxLatency(us), 31195135
[INSERT], 95thPercentileLatency(us), 25007
[INSERT], 99thPercentileLatency(us), 46047
[INSERT], Return=OK, 100000
[dave@www.cndba.cn_3 bin]#

2.3.3 执行测试

[dave@www.cndba.cn_3 bin]#  ./ycsb run mongodb -s -P ../workloads/ustc -threads 8
java -cp /root/ycsb-mongodb-binding-0.17.0/mongodb-binding/conf:/root/ycsb-mongodb-binding-0.17.0/conf:/root/ycsb-mongodb-binding-0.17.0/lib/core-0.17.0.jar:/root/ycsb-mongodb-binding-0.17.0/lib/htrace-core4-4.1.0-incubating.jar:/root/ycsb-mongodb-binding-0.17.0/lib/jackson-mapper-asl-1.9.4.jar:/root/ycsb-mongodb-binding-0.17.0/lib/jackson-core-asl-1.9.4.jar:/root/ycsb-mongodb-binding-0.17.0/lib/HdrHistogram-2.1.4.jar:/root/ycsb-mongodb-binding-0.17.0/lib/mongo-java-driver-3.8.0.jar:/root/ycsb-mongodb-binding-0.17.0/lib/mongodb-async-driver-2.0.1.jar:/root/ycsb-mongodb-binding-0.17.0/lib/logback-classic-1.1.2.jar:/root/ycsb-mongodb-binding-0.17.0/lib/logback-core-1.1.2.jar:/root/ycsb-mongodb-binding-0.17.0/lib/slf4j-api-1.7.25.jar:/root/ycsb-mongodb-binding-0.17.0/lib/snappy-java-1.1.7.1.jar:/root/ycsb-mongodb-binding-0.17.0/lib/mongodb-binding-0.17.0.jar site.ycsb.Client -db site.ycsb.db.MongoDbClient -s -P ../workloads/ustc -threads 8 -t
Command line: -db site.ycsb.db.MongoDbClient -s -P ../workloads/ustc -threads 8 -t
YCSB Client 0.17.0

Loading workload...
Starting test.
2022-05-15 13:53:10:310 0 sec: 0 operations; est completion in 0 second
mongo client connection created with mongodb://root:root@172.31.185.120:27000,172.31.185.165:27000,172.31.185.131:27000
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
2022-05-15 13:53:20:235 10 sec: 10103 operations; 1010.3 current ops/sec; est completion in 8 minutes [READ: Count=5043, Max=716799, Min=676, Avg=2927.09, 90=3489, 99=13607, 99.9=116735, 99.99=666111] [UPDATE: Count=5083, Max=676863, Min=4516, Avg=11324.94, 90=16543, 99=32511, 99.9=62879, 99.99=670207]
2022-05-15 13:53:30:232 20 sec: 22901 operations; 1279.8 current ops/sec; est completion in 6 minutes [READ: Count=6416, Max=26879, Min=595, Avg=1810.72, 90=2849, 99=10271, 99.9=20095, 99.99=25807] [UPDATE: Count=6361, Max=123519, Min=3890, Avg=10690.97, 90=16007, 99=35359, 99.9=101183, 99.99=121919]
2022-05-15 13:53:40:239 30 sec: 35461 operations; 1255.12 current ops/sec; est completion in 6 minutes [READ: Count=6296, Max=37471, Min=655, Avg=1627.19, 90=2431, 99=7639, 99.9=18799, 99.99=28447] [UPDATE: Count=6263, Max=124543, Min=4062, Avg=11132.87, 90=15839, 99=61151, 99.9=108415, 99.99=110655]
2022-05-15 13:53:50:232 40 sec: 47986 operations; 1253.38 current ops/sec; est completion in 6 minutes [READ: Count=6324, Max=28191, Min=618, Avg=1846.5, 90=2829, 99=11759, 99.9=20575, 99.99=27887] [UPDATE: Count=6208, Max=133375, Min=3978, Avg=10983.81, 90=16287, 99=58591, 99.9=99519, 99.99=124671]
2022-05-15 13:54:00:245 50 sec: 60638 operations; 1265.2 current ops/sec; est completion in 6 minutes [READ: Count=6266, Max=244479, Min=612, Avg=2092.89, 90=2861, 99=13623, 99.9=32431, 99.99=224383] [UPDATE: Count=6386, Max=266751, Min=4204, Avg=10468.16, 90=14975, 99=51679, 99.9=125631, 99.99=261375]
2022-05-15 13:54:10:259 60 sec: 73494 operations; 1285.6 current ops/sec; est completion in 5 minutes [READ: Count=6562, Max=37119, Min=611, Avg=1896.35, 90=2915, 99=11607, 99.9=23679, 99.99=32895] [UPDATE: Count=6289, Max=155647, Min=4248, Avg=10732.33, 90=14743, 99=47871, 99.9=131711, 99.99=150911]
……
2022-05-15 14:00:20:281 430 sec: 491666 operations; 1073.02 current ops/sec; est completion in 8 second [READ: Count=5371, Max=37375, Min=607, Avg=1967.34, 90=3163, 99=9823, 99.9=28799, 99.99=33375] [UPDATE: Count=5369, Max=227327, Min=3842, Avg=12150.38, 90=17791, 99=65151, 99.9=197631, 99.99=227327]
2022-05-15 14:00:28:824 438 sec: 500000 operations; 971.22 current ops/sec; [READ: Count=4211, Max=49343, Min=551, Avg=1837.75, 90=2977, 99=8655, 99.9=25871, 99.99=49343] [CLEANUP: Count=8, Max=81727, Min=1, Avg=11256.75, 90=8343, 99=81727, 99.9=81727, 99.99=81727] [UPDATE: Count=4123, Max=635903, Min=3536, Avg=12039.51, 90=16223, 99=50751, 99.9=628223, 99.99=635903]
[OVERALL], RunTime(ms), 438592
[OVERALL], Throughput(ops/sec), 1140.011673719539
[TOTAL_GCS_G1_Young_Generation], Count, 578
[TOTAL_GC_TIME_G1_Young_Generation], Time(ms), 1616
[TOTAL_GC_TIME_%_G1_Young_Generation], Time(%), 0.368451772946155
[TOTAL_GCS_G1_Old_Generation], Count, 0
[TOTAL_GC_TIME_G1_Old_Generation], Time(ms), 0
[TOTAL_GC_TIME_%_G1_Old_Generation], Time(%), 0.0
[TOTAL_GCs], Count, 578
[TOTAL_GC_TIME], Time(ms), 1616
[TOTAL_GC_TIME_%], Time(%), 0.368451772946155
[READ], Operations, 250276
[READ], AverageLatency(us), 1915.3477640684685
[READ], MinLatency(us), 534
[READ], MaxLatency(us), 716799
[READ], 95thPercentileLatency(us), 4439
[READ], 99thPercentileLatency(us), 11087
[READ], Return=OK, 250276
[CLEANUP], Operations, 8
[CLEANUP], AverageLatency(us), 11256.75
[CLEANUP], MinLatency(us), 1
[CLEANUP], MaxLatency(us), 81727
[CLEANUP], 95thPercentileLatency(us), 81727
[CLEANUP], 99thPercentileLatency(us), 81727
[UPDATE], Operations, 249724
[UPDATE], AverageLatency(us), 12012.219258060899
[UPDATE], MinLatency(us), 3536
[UPDATE], MaxLatency(us), 676863
[UPDATE], 95thPercentileLatency(us), 25167
[UPDATE], 99thPercentileLatency(us), 58911
[UPDATE], Return=OK, 249724
[dave@www.cndba.cn_3 bin]#

2.4 压测结果数据说明

Load和 RUN返回的数据都类似如下：

[OVERALL], RunTime(ms), 198416
[OVERALL], Throughput(ops/sec), 503.99161357955006
[TOTAL_GCS_G1_Young_Generation], Count, 85
[TOTAL_GC_TIME_G1_Young_Generation], Time(ms), 426
[TOTAL_GC_TIME_%_G1_Young_Generation], Time(%), 0.21470042738488834
[TOTAL_GCS_G1_Old_Generation], Count, 0
[TOTAL_GC_TIME_G1_Old_Generation], Time(ms), 0
[TOTAL_GC_TIME_%_G1_Old_Generation], Time(%), 0.0
[TOTAL_GCs], Count, 85
[TOTAL_GC_TIME], Time(ms), 426
[TOTAL_GC_TIME_%], Time(%), 0.21470042738488834
[CLEANUP], Operations, 8
[CLEANUP], AverageLatency(us), 18508.125
[CLEANUP], MinLatency(us), 2
[CLEANUP], MaxLatency(us), 148095
[CLEANUP], 95thPercentileLatency(us), 148095
[CLEANUP], 99thPercentileLatency(us), 148095
[INSERT], Operations, 100000
[INSERT], AverageLatency(us), 14928.95763
[INSERT], MinLatency(us), 4076
[INSERT], MaxLatency(us), 31195135
[INSERT], 95thPercentileLatency(us), 25007
[INSERT], 99thPercentileLatency(us), 46047
[INSERT], Return=OK, 100000
[dave@www.cndba.cn_3 bin]#

返回字段Return=OK，则表示测试完成，取“[OVERALL],Throughput”值作为测试指标（ops/s）。

测试结果分析：

  1. RunTime(ms): 数据加载所用时间,单位毫秒(ms)
  2. Throughput(ops/sec): 吞吐量,即ops(每秒操作次数)
  3. Operations: 操作的总次数
  4. AverageLatency(us): 平均响应延时,单位是微秒(us)
  5. MinLatency(us): 最小响应时间,单位是微秒(us)
  6. MaxLatency(us): 最大响应时间,单位是微秒(us)
  7. 95thPercentileLatency(us): 95%的操作延时（在该时间内完成）,单位是微秒(us)
  8. 99thPercentileLatency(us): 99%的操作延时（在该时间内完成）,单位是微秒(us)
  9. Return=OK: 成功返回数,这个值不符合测试要求,则证明测试失败.
  10. [READ]开头的代表只读的操作记录,其他还有 [INSERT],[UPDATE]等

3 其他事项

  1. 配置文件中的database 没有生效，实际还是使用的ycsb。

    mongodb.database=cndba

  2. 测试的比例可以根据需要调整，我们之前测试的是5:5开，实际也可以考虑：读、写、更新6：3：1的比例。

    readproportion=0.6
    updateproportion=0.1
    scanproportion=0
    insertproportion=0.3

  3. 两种操作类型：测试结果都是对比“[OVERALL],Throughput”值
    数据装载：顺序写入，单位 ops/s
    读写压测：读写更新比 6:3:1，单位ops/s

  4. 测试线程
    可以按：8、16、32、64 不断增加，找到最大值。

    http://www.cndba.cn/cndba/dave/article/108004

  5. 我们在YCSB 中使用的是Mongos的地址，在sysbench-mongodb中则是不支持的。

    MongoDB 性能测试工具 sysbench-mongodb
    https://www.cndba.cn/dave/article/107997

  6. 在压测过程中，也可以使用mongostat 和 mongotop 工具进行监控：

[dave@www.cndba.cn_3 ~]#  mongostat --port=27000 -u "root" --authenticationDatabase "admin" -p "root"
2022-05-15T13:57:17.878+0800    WARNING: On some systems, a password provided directly using --password may be visible to system status programs such as `ps` that may be invoked by other users. Consider omitting the password to provide it via stdin, or using the --config option to specify a configuration file with the password.
insert query update delete getmore command flushes mapped vsize   res faults qrw arw net_in net_out conn                time
    *0   112    120     *0       0   119|0       0     0B 1.47G 31.0M    127 0|0 0|0  76.6k    198k   12 May 15 13:57:19.147
    *0   233    237     *0       0   237|0       0     0B 1.47G 31.0M     40 0|0 0|0   155k    388k   12 May 15 13:57:20.146
    *0   159    185     *0       0   186|0       0     0B 1.47G 30.0M     49 0|0 0|0   116k    275k   12 May 15 13:57:21.156
    *0   183    169     *0       0   169|0       0     0B 1.47G 30.0M     18 0|0 0|0   115k    304k   12 May 15 13:57:22.160
    *0   154    163     *0       0   167|0       0     0B 1.47G 30.0M     38 0|0 0|0   107k    267k   12 May 15 13:57:23.147
    *0   172    176     *0       0   175|0       0     0B 1.47G 30.0M      5 0|0 0|0   115k    292k   12 May 15 13:57:24.146
    *0   222    223     *0       0   226|0       0     0B 1.47G 29.0M      0 0|0 0|0   148k    370k   12 May 15 13:57:25.177
    *0   188    201     *0       0   202|0       0     0B 1.47G 30.0M     74 0|0 0|0   130k    320k   12 May 15 13:57:26.151
    *0   185    185     *0       0   184|0       0     0B 1.47G 29.0M      0 0|0 0|0   122k    312k   12 May 15 13:57:27.155

MongoDB 监控工具说明
https://www.cndba.cn/dave/article/107971

版权声明：本文为博主原创文章，未经博主允许不得转载。

用户评论
* 以下用户言论只代表其个人观点，不代表CNDBA社区的观点或立场
dave

dave

关注

人的一生应该是这样度过的：当他回首往事的时候，他不会因为虚度年华而悔恨，也不会因为碌碌无为而羞耻；这样，在临死的时候，他就能够说：“我的整个生命和全部精力，都已经献给世界上最壮丽的事业....."

  • 2297
    原创
  • 3
    翻译
  • 630
    转载
  • 198
    评论
  • 访问：9289391次
  • 积分：4528
  • 等级：核心会员
  • 排名：第1名
精华文章

    达梦数据库运维实战

    Oracle 18c 必须掌握的新特性：管理与实战

    Oracle数据库问题解决方案和故障排除手册
    最新问题
    查看更多+
    热门文章
      热门用户
      推荐用户
        友情链接： CNDBA社区 | openGauss | openEuler | openAnolis | 广告联系: database@ustc.edu
        Copyright © 2016 All Rights Reserved. Powered by CNDBA · 皖ICP备2022006297号-1·

        QQ交流群

        注册联系QQ