为了监控 elasticsearch 死掉后,能自动重启,网上查了下,supervisor 可以实现这个要求,以下是配置: [program=sticsearch] command=/usr/share/elasticsearch/bin/elasticsearch autostart=true autorestart=true stdout_logfile=/home/demoer/log/elasticsearch/elasticsearch-stdout.log stderr_logfile=/home/demoer/log/elasticsearch/elasticsearch-stderr.log startsecs=5
我把这个直接写在 supervisor.conf ,在启动 supervisorctl reload, reread, update,都说没有找到配置( No config updates to processes )。 我用 include_file 单独来引入,也是同样的结果。 请问下配置代码是否正确呢?
1
RIcter 2017-04-15 01:23:11 +08:00 via iPhone
得重启 supervisor 服务…
|
2
MrMike OP supervisord -c /etc/supervisord.conf
[root tmp]# ps aux | grep supervisord root 10926 0.0 0.3 220120 11836 ? Ss Apr14 0:11 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf root 12554 0.0 0.2 217092 10664 ? Ss Apr14 0:10 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf root 12564 0.0 0.2 217128 10716 ? Ss Apr14 0:10 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf root 12575 0.0 0.2 217280 10824 ? Ss Apr14 0:11 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf root 12764 0.0 0.2 219728 11464 ? Ss Apr14 0:11 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf root 13844 0.0 0.2 217588 11164 ? Ss 00:11 0:05 /usr/bin/python /usr/bin/supervisord root 14070 0.0 0.2 216964 9828 ? Ss 01:00 0:04 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf root 14117 0.0 0.2 217144 10644 ? Ss 01:02 0:04 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf root 14128 0.0 0.2 217520 11144 ? Ss 01:04 0:04 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf root 14936 0.0 0.2 216960 9832 ? Ss 08:07 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf root 14938 0.0 0.0 112648 960 pts/0 S+ 08:07 0:00 grep --color=auto supervisord [root tmp]# supervisorctl supervisor> reload Really restart the remote supervisord process y/N? y Restarted supervisord supervisor> reread No config updates to processes supervisor> update supervisor> status supervisor> 已经重启过很多次,还是一样的结果。 下面是 supervisor.conf 和 elasticsearch.conf 的配置内容。 supervisor.conf: [include] files = /etc/supervisord/conf.d/elasticsearch.conf elasticsearch.conf: [program=sticsearch] command=systemctl restart elasticsearch.service ;command=/usr/share/elasticsearch/bin/elasticsearch ;directory=/usr/share/elasticsearch autostart=true autorestart=true startsecs=10 startretries=3 redirect_stderr=true user=root stdout_logfile=/home/public/log/elasticsearch/elasticsearch-stdout.log stderr_logfile=/home/public/log/elasticsearch/elasticsearch-stderr.log |
3
zzlyzq 2017-04-15 08:25:14 +08:00 via iPhone
主配置里面默认会包含所有 conf.d 下面的 conf 后缀文件 回头我看看我的配置
|
4
MrMike OP @zzlyzq 我安装完 supervisor 后,在 etc 下面没有 /etc/supervisord 文件夹的,我是自己创建的 /etc/supervisord/con.d/文件夹,然后把 elasticsearch.conf 放在里面。
|
5
zzlyzq 2017-04-15 09:00:18 +08:00
恩 看起来没啥错,我的 centos6.8 安装后,主文件是 包括如下两行
[include] files = /etc/supervisor.d/*.conf 你的配置看起来也没啥问题的,要不试试 ls /etc/supervisord/conf.d/elasticsearch.conf 看看这个文件是不是真正的存在呢? 不存在的概率可能比较小,不过工作中经常会遇到这种情况,呵呵。 |
7
zzlyzq 2017-04-15 09:24:03 +08:00
@MrMike 既然文件已经加载了,怀疑还是配置的问题,
1. 是不是配置的 program 名字有冲突? 2. 打开界面,可能更一目了然 在 supervisord.conf 开头加上或者修改如下,打开 http ,一目了然 [unix_http_server] file=/tmp/supervisor.sock ; (the path to the socket file) [inet_http_server] ; inet (TCP) server disabled by default port=0.0.0.0:1009 ; (ip_address:port specifier, *:port for all iface) username=ops ; (default is no username (open server)) password=123123 ; (default is no password (open server)) |
8
zzlyzq 2017-04-15 09:27:39 +08:00
@MrMike 我大概知道了, supervisor 在执行命令的时候要求比较苛刻,比如全路径呀,环境变量报错呀啥的, 刚刚的 http 页面可以让你看到
1. elasticsearch 进程是否配置到 supervisord 2. 启动情况如何,是 ok 还是 fail 3. 瞅瞅 log 里面有啥吧 |
9
MrMike OP @zzlyzq supervisord.conf 用的是默认设置,我没修过过。
[unix_http_server] file=/tmp/supervisor.sock ; (the path to the socket file) chmod=0777 ; socket file mode (default 0700) ;chown=nobody:nogroup ; socket file uid:gid owner ;username=user ; (default is no username (open server)) ;password=123 ; (default is no password (open server)) ;[inet_http_server] ; inet (TCP) server disabled by default ;port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface) ;username=user ; (default is no username (open server)) ;password=123 ; (default is no password (open server)) supervisord.log 2017-04-15 08:08:21,706 INFO RPC interface 'supervisor' initialized 2017-04-15 08:08:21,706 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2017-04-15 08:08:21,707 INFO supervisord started with pid 14936 2017-04-15 08:43:04,969 CRIT Supervisor running as root (no user in config file) 2017-04-15 08:43:04,969 INFO Included extra file "/etc/supervisord/conf.d/elasticsearch.conf" during parsing 2017-04-15 08:43:04,980 INFO RPC interface 'supervisor' initialized 2017-04-15 08:43:04,980 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2017-04-15 08:43:04,981 INFO daemonizing the supervisord process 2017-04-15 08:43:04,981 INFO supervisord started with pid 15044 貌似没看出有啥问题呢。也没有报错。 |
10
zzlyzq 2017-04-15 09:42:31 +08:00
Included extra file "/etc/supervisord/conf.d/elasticsearch.conf" during parsing
上面的一行说明你的 elasticsearch.conf 已经被加载了,现在我越来越猜测是 elasticsearch.conf 配置问题导致没有启动 既然没有打开 http ,那就使用如下命令看看 status 吧 supervisorctl status |
11
zsz 2017-04-15 10:06:43 +08:00 via iPhone
1.supervisor 配置中的 command 一定要处于挂起运行,把标准输出,是他的子进程,运行期间不能返回
2.你需要用命令行启动 es 的方式,配置到 command 3.既然用 systemd 和 supervisor 用一个就可以,他们都是守护 有问题可以加群问,效率高些,面向初学者的 python Linux 学习群, 由一群有经验的工程师组成, python qq 群号: 278529278 , PHP qq 群号: 476648701 ,非商业性质,拒绝广告,只接收真正想学这方面技术的朋友,交流学习,申请请说明来自 v2ex |
12
yzongyue 2017-04-15 10:16:29 +08:00
这样写试试 [program:sticsearch]
|
13
ksupertu 2017-04-15 14:52:24 +08:00 via iPhone
新版本先要启动 supervisord 然后在启动 supervisorctl ,不然会一直报错……
|
14
ksupertu 2017-04-15 14:54:34 +08:00 via iPhone
而且你 command 那里最好写成 shell 脚本去执行,错误概率小一点
|
15
MrMike OP @ksupertu 已经启动过了。 command 我用 /usr/share/elasticsearch/bin/elasticsearch 也是不能用的。在日志里面也看不到错误提示呢。
|
16
zsz 2017-04-15 19:11:22 +08:00 via iPhone
@MrMike ,你看的什么错误日志,是 supervisor 的?还是 supervisor 配置文件中 es 的?这个启动不了应该是环境问题,一般是 java 环境, supervisor 自己的错误日志中应该有信息才对
|
18
a128445 2017-04-15 21:48:13 +08:00
[program=sticsearch]
改为 [program:sticsearch] |
19
zsz 2017-04-15 21:48:56 +08:00
@MrMike 这两个 log 呢? 有错误信息
stdout_logfile=/home/public/log/elasticsearch/elasticsearch-stdout.log stderr_logfile=/home/public/log/elasticsearch/elasticsearch-stderr.log |
20
zsz 2017-04-15 21:51:56 +08:00
先按照上面说的把配置写对 [program:sticsearch]
|
21
MrMike OP @zsz
@a128445 elasticsearch.con : [program:elasticsearch] command=/usr/share/elasticsearch/bin/elasticsearch directory=/usr/share/elasticsearch autostart=true autorestart=true startsecs=10 startretries=3 redirect_stderr=true user=root stdout_logfile=/home/demoer/log/elasticsearch/elasticsearch-stdout.log stderr_logfile=/home/demoer/log/elasticsearch/elasticsearch-stderr.log elasticsearch-stdout.log : Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config]; Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.nio.file.Files.readAttributes(Files.java:1737) at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225) at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) at java.nio.file.Files.walkFileTree(Files.java:2662) at org.elasticsearch.common.logging.LogConfigurator.resolveConfig(LogConfigurator.java:150) at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:114) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:236) at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96) at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91) at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91) at org.elasticsearch.cli.Command.main(Command.java:53) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63) Refer to the log for complete error details. Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config]; Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.nio.file.Files.readAttributes(Files.java:1737) at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225) at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) at java.nio.file.Files.walkFileTree(Files.java:2662) at org.elasticsearch.common.logging.LogConfigurator.resolveConfig(LogConfigurator.java:150) at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:114) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:236) at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96) at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91) at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91) at org.elasticsearch.cli.Command.main(Command.java:53) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63) Refer to the log for complete error details. Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config]; Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.nio.file.Files.readAttributes(Files.java:1737) at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225) at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) at java.nio.file.Files.walkFileTree(Files.java:2662) at org.elasticsearch.common.logging.LogConfigurator.resolveConfig(LogConfigurator.java:150) at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:114) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:236) at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96) at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91) at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91) at org.elasticsearch.cli.Command.main(Command.java:53) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63) Refer to the log for complete error details. Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config]; Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144) at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.nio.file.Files.readAttributes(Files.java:1737) at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225) at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) at java.nio.file.Files.walkFileTree(Files.java:2662) at org.elasticsearch.common.logging.LogConfigurator.resolveConfig(LogConfigurator.java:150) at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:114) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:236) at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96) at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91) at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91) at org.elasticsearch.cli.Command.main(Command.java:53) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63) Refer to the log for complete error details. elasticsearch-stderr.log 内容为空。 我在重启 supervisor 后,进入 supervisorctl ,提示“ elasticsearch FATAL Exited too quickly (process log may have details)” |
22
MrMike OP elasticsearch 安装后,在 /usr/share/elasticsearch/就没有 config 这个文件的。
|
23
zsz 2017-04-15 22:35:35 +08:00 via iPhone
1. 你命令行成功启动过 es 么?
2.如果你只是在 systemd 中成功启动过,你的配置文件应该在系统级的目录下,找到他,先用命令行能启动,再把命令配置进去, es 在命令行可以指定配置文件的 |
26
MrMike OP @zsz
@a128445 配置文件改成这样的: [program:elasticsearch] command=/etc/init.d/elasticsearch start directory=/usr/share/elasticsearch autostart=true autorestart=true startsecs=10 startretries=3 redirect_stderr=true user=root stdout_logfile=/home/demoer/log/elasticsearch/elasticsearch-stdout.log stderr_logfile=/home/demoer/log/elasticsearch/elasticsearch-stderr.log 我手动关闭 es 后,等了几秒钟 es 没有启动,就使用 supervisorctl restart elasticsearch 启动, elasticsearch: ERROR (not running) elasticsearch: ERROR (spawn error) 得到这样的错误提示,但是 es 却重新启动了。不能自动启动 es 呢。。 |
27
MrMike OP @zsz stop es 后,我目前的配置,只能手动执行 supervisorctl reload ,才能再让 es 启动。
|
28
zsz 2017-04-15 23:16:06 +08:00 via iPhone
1. 上面我指出过 supervisor 中 command 不能使用 systemctl 或者 init.d 的方式,因为这两种是 daemon 运行,命令执行完就返回了
2. 你加我上面的发群我用 qq 跟你说,这样解释太慢了,非商业化的群,不用担心 |