CNDBA社区
1 Supervisor 工具概述
Supervisor是用Python开发的一个client/server服务，是Linux/Unix系统下的一个进程管理工具，不支持Windows系统。它可以很方便的监听、启动、停止、重启一个或多个进程。用Supervisor管理的进程，当一个进程意外被杀死，supervisort监听到进程死后，会自动将它重新拉起，很方便的做到进程自动恢复的功能，不再需要自己写shell脚本来控制。
官网地址：
Supervisor 不支持Windows 环境，在Python 2.7及Python 3.4+ 的版本都可以正常工作。
supervisor 主要有3个组件：
supervisord
supervisor的服务器端命令，负责在自己的调用中启动子程序，响应来自客户端的命令，重新启动崩溃或退出的子进程，记录子进程的stdout和stderr输出。
supervisord使用配置文件是：/etc/supervisord.conf。 该配置文件是一个“Windows-INI”风格的配置文件。 因为该文件可能记录有未加密的用户名和密码，所以需要通过适当的文件系统权限来保护这个文件的安全。Supervisorctl
supervisor的客户端命令。 它为supervisor提供的特性提供了一个类似shell的接口。 用户可以通过supersuperctl连接到不同的supervisord进程(一次一个)，获取该supervisord进程所控制的子进程的状态，停止子进程和启动子进程，以及获取该supervisord进程的运行进程列表。
Supervisorctl通常使用与服务器相同的配置文件，对应的配置部分是：[supervisorctl]。Web Server
和Supervisorctl 类似功能的一个web 接口，在配置文件中设置：[inet_http_server] 选项后，可以直接通过http://localhost:9001/ 来查看和管理进程状态。
2 Supervisor 工具安装
2.1 安装软件
官网在安装页面介绍了几种不同的安装方法：
在网络可用的情况下，直接pip 进行安装：
pip install supervisor
通过yum 安装：
wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install epel-release-latest-7.noarch.rpm
yum install -y supervisor
[dave@www.cndba.cn_2 backup]# yum install -y supervisor
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
c7.8 | 3.6 kB 00:00:00
iflytekdc-epel | 4.7 kB 00:00:00
iflytekdc-extras | 2.9 kB 00:00:00
iflytekdc-os | 3.6 kB 00:00:00
iflytekdc-updates | 2.9 kB 00:00:00
Resolving Dependencies
--> Running transaction check
---> Package supervisor.noarch 0:3.4.0-1.el7 will be installed
--> Processing Dependency: python-meld3 >= 0.6.5 for package: supervisor-3.4.0-1.el7.noarch
--> Running transaction check
---> Package python-meld3.x86_64 0:0.6.10-1.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
……
Total download size: 571 k
Installed size: 2.9 M
Downloading packages:
(1/2): python-meld3-0.6.10-1.el7.x86_64.rpm | 73 kB 00:00:00
(2/2): supervisor-3.4.0-1.el7.noarch.rpm | 498 kB 00:00:00
---------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.1 MB/s | 571 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : python-meld3-0.6.10-1.el7.x86_64 1/2
Installing : supervisor-3.4.0-1.el7.noarch 2/2
Verifying : python-meld3-0.6.10-1.el7.x86_64 1/2
Verifying : supervisor-3.4.0-1.el7.noarch 2/2
Installed:
supervisor.noarch 0:3.4.0-1.el7
Dependency Installed:
python-meld3.x86_64 0:0.6.10-1.el7
Complete!
[dave@www.cndba.cn_2 backup]#
[dave@www.cndba.cn_1 ~]# find / -name supervisord
/usr/bin/supervisord
[dave@www.cndba.cn_1 ~]# find / -name supervisorctl
/usr/bin/supervisorctl
[dave@www.cndba.cn_1 ~]#
2.2 创建配置文件
执行echo_supervisord_conf命令会显示配置文件的相关示例，也可以直接将示例写入到/etc/supervisord.conf 配置文件。
[dave@www.cndba.cn_1 ~]#echo_supervisord_conf
[dave@www.cndba.cn_1 ~]#echo_supervisord_conf > /etc/supervisord.conf
使用YUM 安装时，会自动创建/etc/supervisord.conf 文件，所以可以不执行该步骤。
[dave@www.cndba.cn_1 ~]# cat /etc/supervisord.conf
; Sample supervisor config file.
[unix_http_server]
file=/var/run/supervisor/supervisor.sock ; (the path to the socket file)
;chmod=0700 ; sockef file mode (default 0700)
;chown=nobody:nogroup ; socket file uid:gid owner
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))
;[inet_http_server] ; inet (TCP) server disabled by default
;port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
;umask=022 ; (process file creation umask;default 022)
;user=chrism ; (default is current user, required if root)
;identifier=supervisor ; (supervisord identifier, default is 'supervisor')
;directory=/tmp ; (default is not to cd during start)
;nocleanup=true ; (don't clean up tempfiles at start;default false)
;childlogdir=/tmp ; ('AUTO' child log dir, default $TEMP)
;environment=KEY=value ; (key value pairs to add to environment)
;strip_ansi=false ; (strip ansi escape codes in logs; def. false)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL for a unix socket
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
;username=chris ; should be same as http_username if set
;password=123 ; should be same as http_password if set
;prompt=mysupervisor ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history ; use readline history if available
; The below sample program section shows all possible program subsection values,
; create one or more 'real' program: sections to be able to control them under
; supervisor.
;[program:theprogramname]
;command=/bin/cat ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;directory=/tmp ; directory to cwd to before exec (def no cwd)
;umask=022 ; umask for process (default None)
;priority=999 ; the relative start priority (default 999)
;autostart=true ; start at supervisord start (default: true)
;autorestart=true ; retstart at unexpected quit (default: true)
;startsecs=10 ; number of secs prog must stay running (def. 1)
;startretries=3 ; max # of serial start failures (default 3)
;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
;user=chrism ; setuid to this UNIX account to run the program
;redirect_stderr=true ; redirect proc stderr to stdout (default false)
;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
;stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
;stdout_events_enabled=false ; emit events on stdout writes (default false)
;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10 ; # of stderr logfile backups (default 10)
;stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
;stderr_events_enabled=false ; emit events on stderr writes (default false)
;environment=A=1,B=2 ; process environment additions (def no adds)
;serverurl=AUTO ; override serverurl computation (childutils)
; The below sample eventlistener section shows all possible
; eventlistener subsection values, create one or more 'real'
; eventlistener: sections to be able to handle event notifications
; sent by supervisor.
;[eventlistener:theeventlistenername]
;command=/bin/eventlistener ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;events=EVENT ; event notif. types to subscribe to (req'd)
;buffer_size=10 ; event buffer queue size (default 10)
;directory=/tmp ; directory to cwd to before exec (def no cwd)
;umask=022 ; umask for process (default None)
;priority=-1 ; the relative start priority (default -1)
;autostart=true ; start at supervisord start (default: true)
;autorestart=unexpected ; restart at unexpected quit (default: unexpected)
;startsecs=10 ; number of secs prog must stay running (def. 1)
;startretries=3 ; max # of serial start failures (default 3)
;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
;user=chrism ; setuid to this UNIX account to run the program
;redirect_stderr=true ; redirect proc stderr to stdout (default false)
;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
;stdout_events_enabled=false ; emit events on stdout writes (default false)
;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups ; # of stderr logfile backups (default 10)
;stderr_events_enabled=false ; emit events on stderr writes (default false)
;environment=A=1,B=2 ; process environment additions
;serverurl=AUTO ; override serverurl computation (childutils)
; The below sample group section shows all possible group values,
; create one or more 'real' group: sections to create "heterogeneous"
; process groups.
;[group:thegroupname]
;programs=progname1,progname2 ; each refers to 'x' in [program:x] definitions
;priority=999 ; the relative start priority (default 999)
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
[include]
files = supervisord.d/*.ini
[dave@www.cndba.cn_1 ~]#
3 Supervisor 配置文件编写
Supervisor 的配置文件内容较多，具体参考官方手册：
我们这里以我们之前 MongoDB shard 分片集群为例，进行配置：
MongoDB 4.4 分片集群（3 shard） 搭建手册
在此环境中，每个节点上都需要启动5个进程，我们这里使用supervisor 进行统一管理。
[dave@www.cndba.cn_1 ~]# cat /etc/supervisord.conf |grep -v '^;'|grep -v '^$'
[unix_http_server]
file=/var/run/supervisor/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)
username=admin ; (default is no username (open server))
password=admin ; (default is no password (open server))
[inet_http_server] ; inet (TCP) server disabled by default
port=0.0.0.0:9001 ; (ip_address:port specifier, *:port for all iface)
username=admin ; (default is no username (open server))
password=admin ; (default is no password (open server))
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL for a unix socket
serverurl=http://0.0.0.0:9001 ; use an http:// url to specify an inet socket
username=admin ; should be same as http_username if set
password=admin ; should be same as http_password if set
history_file=~/.sc_history ; use readline history if available
[program:shard1]
command = /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/shard1.conf
user = root
startsecs = 3
priority=990
autorestart = true
redirect_stderr = true
stdout_logfile_maxbytes = 50MB
stdout_logfile_backups = 10
stdout_logfile = /data/mongodb/data/shard1/super.log
[program:shard2]
command = /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/shard2.conf
user = root
startsecs = 3
priority=990
autorestart = true
redirect_stderr = true
stdout_logfile_maxbytes = 50MB
stdout_logfile_backups = 10
stdout_logfile = /data/mongodb/data/shard2/super.log
killasgroup = true
stopasgroup = true
[program:shard3]
command = /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/shard3.conf
user = root
startsecs = 3
priority=990
autorestart = true
redirect_stderr = true
stdout_logfile_maxbytes = 50MB
stdout_logfile_backups = 10
stdout_logfile = /data/mongodb/data/shard3/super.log
killasgroup = true
stopasgroup = true
[program:configdb]
command = /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/configdb.conf
user = root
startsecs = 3
priority=995
autorestart = true
redirect_stderr = true
stdout_logfile_maxbytes = 50MB
stdout_logfile_backups = 10
stdout_logfile = /data/mongodb/data/configdb/super.log
killasgroup = true
stopasgroup = true
[program:mongos]
command = /usr/local/mongodb/bin/mongos -f /data/mongodb/etc/mongos.conf
user = root
startsecs = 3
priority=999
autorestart = true
redirect_stderr = true
stdout_logfile_maxbytes = 50MB
stdout_logfile_backups = 10
stdout_logfile = /data/mongodb/data/configdb/configdb_super.log
killasgroup = true
stopasgroup = true
[include]
files = supervisord.d/*.ini
[dave@www.cndba.cn_1 ~]#
4 启动supervisor 进程
4.1 启动/关闭服务端supervisord
命令帮助：
[dave@www.cndba.cn_1 ~]# supervisord --help
supervisord -- run a set of applications as daemons.
Usage: /usr/bin/supervisord [options]
启动：supervisord：
[dave@www.cndba.cn_1 ~]# supervisord -c /etc/supervisord.conf
[dave@www.cndba.cn_1 ~]# ps -ef|grep supervisord
root 14781 1 0 15:32 ? 00:00:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
root 14914 10700 0 15:32 pts/0 00:00:00 grep --color=auto supervisord
[dave@www.cndba.cn_1 ~]#
关闭supervisord：
[dave@www.cndba.cn_1 ~]# supervisorctl shutdown
Shut down
[dave@www.cndba.cn_1 ~]# ps -ef|grep supervisor
root 20810 15561 0 16:09 pts/1 00:00:00 grep --color=auto supervisor
[dave@www.cndba.cn_1 ~]#
注意：
关闭supervisord 服务端，也会关闭对应的管理进程。
4.2 客户端supervisorctl 管理进程
命令帮助：
[dave@www.cndba.cn_1 ~]# supervisorctl --help
supervisorctl -- control applications run by supervisord from the cmd line.
Usage: /usr/bin/supervisorctl [options] [action [arguments]]
常用命令：
supervisorctl status 查看所有进程状态
supervisorctl tail shard1 # 查看最后的日志
supervisorctl tail -f shard1 # 持续日志
supervisorctl restart shard1
supervisorctl status shard1
supervisorctl start shard1
supervisorctl stop shard1
修改的配置文件生效，设置autostart=true的程序，会自动启动
supervisorctl reread
supervisorctl update
4.3 注意事项：Exited too quickly
Supervisor只能管理非daemon的进程，也就是说Supervisor不能管理守护进程。否则提示Exited too quickly (process log may have details)异常。我们这里的几个程序都是fork 方式运行的，所以都报错了。 修改mongodb 配置文件，取消fork以前台进程的方式运行。
虽然这里会报错：
[dave@www.cndba.cn_1 ~]# supervisorctl status
configdb FATAL Exited too quickly (process log may have details)
mongos FATAL Exited too quickly (process log may have details)
shard1 FATAL Exited too quickly (process log may have details)
shard2 FATAL Exited too quickly (process log may have details)
shard3 FATAL Exited too quickly (process log may have details)
[dave@www.cndba.cn_1 ~]#
但查看进程是已经启动了，并且可以连接：
[dave@www.cndba.cn_1 ~]# ps -ef|grep mongo
root 21099 1 5 16:11 ? 00:00:01 /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/configdb.conf
root 21101 1 8 16:11 ? 00:00:02 /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/shard3.conf
root 21102 1 8 16:11 ? 00:00:02 /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/shard1.conf
root 21103 1 8 16:11 ? 00:00:02 /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/shard2.conf
root 21718 15561 0 16:11 pts/1 00:00:00 grep --color=auto mongo
root 32149 1 0 May01 ? 00:37:27 /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/mongo.conf
[dave@www.cndba.cn_1 ~]# cat /data/mongodb/data/shard1/super.log
about to fork child process, waiting until server is ready for connections.
forked process: 14805
ERROR: child process failed, exited with 48
To see additional information in this output, start without the "--fork" option.
about to fork child process, waiting until server is ready for connections.
forked process: 14868
[dave@www.cndba.cn_1 ~]# ps -ef|grep mongod
root 17466 1 2 15:47 ? 00:00:03 /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/shard1.conf
root 17897 15561 0 15:49 pts/1 00:00:00 grep --color=auto mongod
root 32149 1 0 May01 ? 00:37:16 /usr/local/mongodb/bin/mongod -f /data/mongodb/etc/mongo.conf
[dave@www.cndba.cn_1 ~]# mongo
mongo mongod mongodump mongoexport mongofiles mongoimport mongorestore mongos mongostat mongotop
[dave@www.cndba.cn_1 ~]# mongo --port 27018
MongoDB shell version v4.4.13
connecting to: mongodb://127.0.0.1:27018/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("af2e965e-040c-4de7-8cec-4ac6b58c4545") }
MongoDB server version: 4.4.13
shard1:SECONDARY>
以前台方式运行：
修改monggodb 参数：
#processManagement:
# fork: true
# pidFilePath: /data/mongodb/run/shard1.pid
手工关闭mongodb实例，并用supervisorctl 启动：
[dave@www.cndba.cn_1 ~]# supervisorctl start shard3
shard3: started
[dave@www.cndba.cn_1 ~]# supervisorctl status
configdb RUNNING pid 5449, uptime 0:00:47
mongos RUNNING pid 5450, uptime 0:00:47
shard1 RUNNING pid 5448, uptime 0:00:47
shard2 RUNNING pid 5446, uptime 0:00:47
shard3 RUNNING pid 5447, uptime 0:00:47
[dave@www.cndba.cn_1 ~]#
4.4 测试自动启动
[dave@www.cndba.cn_1 ~]# supervisorctl status
configdb RUNNING pid 5449, uptime 0:02:34
mongos RUNNING pid 5450, uptime 0:02:34
shard1 RUNNING pid 6278, uptime 0:00:14
shard2 RUNNING pid 5446, uptime 0:02:34
shard3 RUNNING pid 5447, uptime 0:02:34
[dave@www.cndba.cn_1 ~]# kill 6278
[dave@www.cndba.cn_1 ~]# supervisorctl status
configdb RUNNING pid 5449, uptime 0:02:49
mongos RUNNING pid 5450, uptime 0:02:49
shard1 STARTING
shard2 RUNNING pid 5446, uptime 0:02:49
shard3 RUNNING pid 5447, uptime 0:02:49
[dave@www.cndba.cn_1 ~]# supervisorctl status
configdb RUNNING pid 5449, uptime 0:02:52
mongos RUNNING pid 5450, uptime 0:02:52
shard1 RUNNING pid 6618, uptime 0:00:03
shard2 RUNNING pid 5446, uptime 0:02:52
shard3 RUNNING pid 5447, uptime 0:02:52
[dave@www.cndba.cn_1 ~]#
5 配置supervisord 开机自启动
因为我们这里是Redhat 7.8 的OS，所以直接配置systemctl服务。
5.1 创建服务文件
创建文件：/lib/systemd/system/supervisor.service。
添加如下内容：
[dave@www.cndba.cn_2 etc]# cat /lib/systemd/system/supervisor.service
[Unit]
Description=supervisor
After=network.target
[Service]
Type=forking
ExecStart=/usr/bin/supervisord -c /etc/supervisord.conf
ExecStop=/usr/bin/supervisorctl $OPTIONS shutdown
ExecReload=/usr/bin/supervisorctl $OPTIONS reload
KillMode=process
Restart=on-failure
RestartSec=42s
[Install]
WantedBy=multi-user.target
[dave@www.cndba.cn_2 etc]#
5.2 修改文件权限为766
[dave@www.cndba.cn_1 configdb]# chmod 766 /lib/systemd/system/supervisor.service
5.3 设置开机启动
[dave@www.cndba.cn_1 configdb]# systemctl enable supervisor.service
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisor.service to /usr/lib/systemd/system/supervisor.service.
[dave@www.cndba.cn_1 configdb]# systemctl daemon-reload
[dave@www.cndba.cn_1 configdb]#
重启操作系统验证：
[dave@www.cndba.cn_1 ~]# ps -ef|grep super
root 853 1 0 17:09 ? 00:00:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
root 2009 1937 0 17:09 pts/0 00:00:00 grep --color=auto super
[dave@www.cndba.cn_1 ~]# supervisorctl status
configdb RUNNING pid 1210, uptime 0:00:37
mongos RUNNING pid 1211, uptime 0:00:37
shard1 RUNNING pid 1209, uptime 0:00:37
shard2 RUNNING pid 1207, uptime 0:00:37
shard3 RUNNING pid 1208, uptime 0:00:37
[dave@www.cndba.cn_1 ~]#
版权声明：本文为博主原创文章，未经博主允许不得转载。
