签到成功

知道了

CNDBA社区CNDBA社区

TiDB PD某节点connect: connection refused报错解决

2019-03-01 20:37 5976 0 原创 TiDB
作者： Marvinn
原文链接：https://www.cndba.cn/Marvinn
本文链接：http://www.cndba.cn/Marvinn/article/3291

[warning] etcdserver: [cannot get the version of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)http://www.cndba.cn/Marvinn/article/3291

pd.log日志：显示pd02节点无法连接
pd.log日志：显示pd02节点无法连接

http://www.cndba.cn/Marvinn/article/3291 

2019/03/01 18:40:42.543 log.go:84: [warning] etcdserver: [cannot get the version of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]
2019/03/01 18:40:46.469 log.go:84: [warning] rafthttp: [health check for peer 581353672014e5e6 could not connect: dial tcp 172.16.30.87:2480: connect: connection refused]
2019/03/01 18:40:46.544 log.go:84: [warning] etcdserver: [failed to reach the peerURL(http://172.16.30.87:2480) of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]
2019/03/01 18:40:46.544 log.go:84: [warning] etcdserver: [cannot get the version of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]
2019/03/01 18:40:50.544 log.go:84: [warning] etcdserver: [failed to reach the peerURL(http://172.16.30.87:2480) of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]
2019/03/01 18:40:50.544 log.go:84: [warning] etcdserver: [cannot get the version of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]
2019/03/01 18:40:51.469 log.go:84: [warning] rafthttp: [health check for peer 581353672014e5e6 could not connect: dial tcp 172.16.30.87:2480: connect: connection refused]
2019/03/01 18:40:54.545 log.go:84: [warning] etcdserver: [failed to reach the peerURL(http://172.16.30.87:2480) of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]
2019/03/01 18:40:54.545 log.go:84: [warning] etcdserver: [cannot get the version of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]
2019/03/01 18:40:56.469 log.go:84: [warning] rafthttp: [health check for peer 581353672014e5e6 could not connect: dial tcp 172.16.30.87:2480: connect: connection refused]
2019/03/01 18:40:58.546 log.go:84: [warning] etcdserver: [failed to reach the peerURL(http://172.16.30.87:2480) of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]
2019/03/01 18:40:58.546 log.go:84: [warning] etcdserver: [cannot get the version of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]
2019/03/01 18:41:01.470 log.go:84: [warning] rafthttp: [health check for peer 581353672014e5e6 could not connect: dial tcp 172.16.30.87:2480: connect: connection refused]
2019/03/01 18:41:02.547 log.go:84: [warning] etcdserver: [failed to reach the peerURL(http://172.16.30.87:2480) of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]
2019/03/01 18:41:02.547 log.go:84: [warning] etcdserver: [cannot get the version of member 581353672014e5e6 (Get http://172.16.30.87:2480/version: dial tcp 172.16.30.87:2480: connect: connection refused)]

解决：删除pd2再添加，经过测试删除再添加的pd name不能重复，否则无法添加会报错
删除不可连接的pd02节点
[tidb@ip-172-16-30-86 bin]$ ./pd-ctl -u http://172.16.30.86:2479
查看当前健康pd节点 pd2 为false

» health
[
  {
    "name": "pd3",
    "member_id": 1122058826700352700,
    "client_urls": [
      "http://172.16.30.88:2479"
    ],
    "health": true
  },
  {
    "name": "pd1",
    "member_id": 1522798235883063874,
    "client_urls": [
      "http://172.16.30.86:2479"
    ],
    "health": true
  },
  {
    "name": "pd2",
    "member_id": 6346508002280138214,
    "client_urls": [
      "http://172.16.30.87:2479"
    ],
    "health": false
  }
]

删除 pd2http://www.cndba.cn/Marvinn/article/3291http://www.cndba.cn/Marvinn/article/3291http://www.cndba.cn/Marvinn/article/3291

» member delete id 6346508002280138214
Success!

再次查看是否删除成功http://www.cndba.cn/Marvinn/article/3291

» health
[
  {
    "name": "pd3",
    "member_id": 1122058826700352700,
    "client_urls": [
      "http://172.16.30.88:2479"
    ],
    "health": true
  },
  {
    "name": "pd1",
    "member_id": 1522798235883063874,
    "client_urls": [
      "http://172.16.30.86:2479"
    ],
    "health": true
  }
]

删除pd节点查看其中一台正常活跃pd.log输出：

http://www.cndba.cn/Marvinn/article/3291 

2019/03/01 18:49:56.495 log.go:84: [warning] rafthttp: [health check for peer 581353672014e5e6 could not connect: dial tcp 172.16.30.87:2480: connect: connection refused]
2019/03/01 18:49:57.116 log.go:88: [info] etcdserver/membership: [removed member 581353672014e5e6 from cluster e62d88fc2a22d675]
2019/03/01 18:49:57.116 log.go:88: [info] rafthttp: [stopping peer 581353672014e5e6...]
2019/03/01 18:49:57.116 log.go:88: [info] rafthttp: [stopped streaming with peer 581353672014e5e6 (writer)]
2019/03/01 18:49:57.116 log.go:88: [info] rafthttp: [stopped streaming with peer 581353672014e5e6 (writer)]
2019/03/01 18:49:57.116 log.go:88: [info] rafthttp: [stopped HTTP pipelining with peer 581353672014e5e6]
2019/03/01 18:49:57.116 log.go:88: [info] rafthttp: [stopped streaming with peer 581353672014e5e6 (stream MsgApp v2 reader)]
2019/03/01 18:49:57.116 log.go:88: [info] rafthttp: [stopped streaming with peer 581353672014e5e6 (stream Message reader)]
2019/03/01 18:49:57.116 log.go:88: [info] rafthttp: [stopped peer 581353672014e5e6]
2019/03/01 18:49:57.116 log.go:88: [info] rafthttp: [removed peer 581353672014e5e6]

添加同名pd name
$ nohup ./bin/pd-server —name=pd2 /
—client-urls=”http://172.16.30.87:2479“ /
—peer-urls=”http://172.16.30.87:2480“ /
—join=”http://172.16.30.88:2479“ >/data/tidb/deploy_tidb/pd/log/pd.log &

添加同名pd节点name查看其中一台正常活跃pd.log输出：日志出现警告

2019/03/01 20:11:47.435 log.go:88: [info] etcdserver/membership: [removed member bbf1420f7c39bcb4 from cluster e62d88fc2a22d675]
2019/03/01 20:11:47.435 log.go:88: [info] rafthttp: [stopping peer bbf1420f7c39bcb4…]
2019/03/01 20:11:47.435 log.go:88: [info] rafthttp: [stopped streaming with peer bbf1420f7c39bcb4 (writer)]
2019/03/01 20:11:47.435 log.go:88: [info] rafthttp: [stopped streaming with peer bbf1420f7c39bcb4 (writer)]
2019/03/01 20:11:47.435 log.go:88: [info] rafthttp: [stopped HTTP pipelining with peer bbf1420f7c39bcb4]
2019/03/01 20:11:47.435 log.go:88: [info] rafthttp: [stopped streaming with peer bbf1420f7c39bcb4 (stream MsgApp v2 reader)]
2019/03/01 20:11:47.435 log.go:88: [info] rafthttp: [stopped streaming with peer bbf1420f7c39bcb4 (stream Message reader)]
2019/03/01 20:11:47.435 log.go:88: [info] rafthttp: [stopped peer bbf1420f7c39bcb4]
2019/03/01 20:11:47.435 log.go:88: [info] rafthttp: [removed peer bbf1420f7c39bcb4]http://www.cndba.cn/Marvinn/article/3291

2019/03/01 20:17:47.841 log.go:84: [warning] rafthttp: [rejected the stream from peer bbf1420f7c39bcb4 since it was removed]
2019/03/01 20:17:47.841 log.go:84: [warning] rafthttp: [rejected the stream from peer bbf1420f7c39bcb4 since it was removed]
2019/03/01 20:17:47.842 log.go:84: [warning] rafthttp: [rejected the stream from peer bbf1420f7c39bcb4 since it was removed]
2019/03/01 20:17:47.842 log.go:84: [warning] rafthttp: [rejected the stream from peer bbf1420f7c39bcb4 since it was removed]

更换pd name（不同名)添加扩容:

nohup ./bin/pd-server —name=pd5 /
—client-urls=”http://172.16.30.87:2479“ /
—peer-urls=”http://172.16.30.87:2480“ /
—join=”http://172.16.30.88:2479“ >/data/tidb/deploy_tidb/pd/log/pd.log &

更换pd name添加扩容,正常pd节点pd.log无警告正常输出:

2019/03/01 20:24:03.445 log.go:88: [info] etcdserver/membership: [added member 4c461b0861cbd30b [http://172.16.30.87:2480] to cluster e62d88fc2a22d675]
2019/03/01 20:24:03.445 log.go:88: [info] rafthttp: [starting peer 4c461b0861cbd30b…]
2019/03/01 20:24:03.445 log.go:88: [info] rafthttp: [started HTTP pipelining with peer 4c461b0861cbd30b]
2019/03/01 20:24:03.446 log.go:88: [info] rafthttp: [started streaming with peer 4c461b0861cbd30b (writer)]
2019/03/01 20:24:03.447 log.go:88: [info] rafthttp: [started streaming with peer 4c461b0861cbd30b (writer)]
2019/03/01 20:24:03.448 log.go:88: [info] rafthttp: [started peer 4c461b0861cbd30b]
2019/03/01 20:24:03.448 log.go:88: [info] rafthttp: [added peer 4c461b0861cbd30b]
2019/03/01 20:24:03.448 log.go:88: [info] rafthttp: [started streaming with peer 4c461b0861cbd30b (stream Message reader)]
2019/03/01 20:24:03.448 log.go:88: [info] rafthttp: [started streaming with peer 4c461b0861cbd30b (stream MsgApp v2 reader)]
2019/03/01 20:24:03.461 log.go:88: [info] rafthttp: [peer 4c461b0861cbd30b became active]
2019/03/01 20:24:03.461 log.go:88: [info] rafthttp: [established a TCP streaming connection with peer 4c461b0861cbd30b (stream Message writer)]
2019/03/01 20:24:03.462 log.go:88: [info] rafthttp: [established a TCP streaming connection with peer 4c461b0861cbd30b (stream MsgApp v2 writer)]
2019/03/01 20:24:03.539 log.go:88: [info] rafthttp: [start to send database snapshot [index: 781664, to 4c461b0861cbd30b]…]
2019/03/01 20:24:03.544 log.go:88: [info] etcdserver: [wrote database snapshot out [total bytes: 3641344]]
2019/03/01 20:24:03.548 log.go:88: [info] rafthttp: [database snapshot [index: 781664, to: 4c461b0861cbd30b] sent out successfully]
2019/03/01 20:24:03.549 log.go:88: [info] rafthttp: [established a TCP streaming connection with peer 4c461b0861cbd30b (stream Message reader)]
2019/03/01 20:24:03.550 log.go:88: [info] rafthttp: [established a TCP streaming connection with peer 4c461b0861cbd30b (stream MsgApp v2 reader)]
2019/03/01 20:24:03.559 log.go:84: [warning] rafthttp: [lost the TCP streaming connection with peer 4c461b0861cbd30b (stream MsgApp v2 reader)]
2019/03/01 20:24:03.559 log.go:84: [warning] rafthttp: [lost the TCP streaming connection with peer 4c461b0861cbd30b (stream Message reader)]
2019/03/01 20:24:03.562 log.go:84: [warning] rafthttp: [closed an existing TCP streaming connection with peer 4c461b0861cbd30b (stream Message writer)]
2019/03/01 20:24:03.562 log.go:88: [info] rafthttp: [established a TCP streaming connection with peer 4c461b0861cbd30b (stream Message writer)]
2019/03/01 20:24:03.562 log.go:84: [warning] rafthttp: [closed an existing TCP streaming connection with peer 4c461b0861cbd30b (stream MsgApp v2 writer)]
2019/03/01 20:24:03.562 log.go:88: [info] rafthttp: [established a TCP streaming connection with peer 4c461b0861cbd30b (stream MsgApp v2 writer)]
2019/03/01 20:24:03.649 log.go:88: [info] rafthttp: [established a TCP streaming connection with peer 4c461b0861cbd30b (stream MsgApp v2 reader)]
2019/03/01 20:24:03.649 log.go:88: [info] rafthttp: [established a TCP streaming connection with peer 4c461b0861cbd30b (stream Message reader)]http://www.cndba.cn/Marvinn/article/3291

查看节点pd5是否添加成信息

$ ./pd-ctl -u http://172.16.30.86:2479
» health
[
{
“name”: “pd3”,
“member_id”: 1122058826700352700,
“client_urls”: [
“http://172.16.30.88:2479“
],
“health”: true
},
{
“name”: “pd1”,
“member_id”: 1522798235883063874,
“client_urls”: [
“http://172.16.30.86:2479“
],
“health”: true
},
{
“name”: “pd5”,
“member_id”: 5496110118066705163,
“client_urls”: [
“http://172.16.30.87:2479“
],
“health”: true
}
]

http://www.cndba.cn/Marvinn/article/3291

```

版权声明：本文为博主原创文章，未经博主允许不得转载。

用户评论
* 以下用户言论只代表其个人观点，不代表CNDBA社区的观点或立场
Marvinn

Marvinn

关注

路漫漫其修远兮、吾将上下而求索

  • 99
    原创
  • 0
    翻译
  • 2
    转载
  • 36
    评论
  • 访问：458380次
  • 积分：449
  • 等级：中级会员
  • 排名：第12名
精华文章

    达梦数据库运维实战

    Oracle 18c 必须掌握的新特性：管理与实战

    Oracle数据库问题解决方案和故障排除手册
    最新问题
    查看更多+
    热门文章
      热门用户
      推荐用户
        友情链接： CNDBA社区 | openGauss | openEuler | openAnolis | 广告联系: database@ustc.edu
        Copyright © 2016 All Rights Reserved. Powered by CNDBA · 皖ICP备2022006297号-1·

        QQ交流群

        注册联系QQ