V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
V2EX 提问指南
sakurazensen
V2EX  ›  问与答

logstash6.5.4 解析 nginx 日志格式报错

  •  
  •   sakurazensen · 2020-05-16 12:59:57 +08:00 · 1115 次点击
    这是一个创建于 1650 天前的主题,其中的信息可能已经有所发展或是发生改变。
    日志格式如下:
    log_format elk '{"time_local":"$time_iso8601",'
    '"remote_addr":"$remote_addr",'
    '"referer":"$http_referer",'
    '"request":"$request",'
    '"status":$status,'
    '"bytes":$body_bytes_sent,'
    '"agent":"$http_user_agent",'
    '"x_forwarded":"$http_x_forwarded_for",'
    '"up_addr":"$upstream_addr",'
    '"up_host":"$upstream_http_host",'
    '"reqeust_time":"$request_time"'

    日志如下:
    {"time_local":"2020-05-16T12:43:48+08:00","remote_addr":"192.168.5.148","referer":"-","request":"GET / HTTP/1.1","status":304,"bytes":0,"agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36","x_forwarded":"-","up_addr":"-","up_host":"-","reqeust_time":"0.000"}

    如果只是单纯解析时间,是没问题
    input {
    file {
    path => "/var/log/nginx/access.elk.log"
    }
    }
    filter {
    grok {
    match => [ "message","%{TIMESTAMP_ISO8601:locals}" ]
    }
    }
    output {
    stdout { codec => rubydebug }
    }
    结果:
    "@version" => "1",
    "host" => "localhost.localdomain",
    "path" => "/var/log/nginx/access.elk.log",
    "message" => "{\"time_local\":\"2020-05-16T12:43:48+08:00\",\"remote_addr\":\"192.168.5.148\",\"referer\":\"-\",\"request\":\"GET / HTTP/1.1\",\"status\":304,\"bytes\":0,\"agent\":\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36\",\"x_forwarded\":\"-\",\"up_addr\":\"-\",\"up_host\":\"-\",\"reqeust_time\":\"0.000\"}",
    "locals" => "2020-05-16T12:43:48+08:00",
    "@timestamp" => 2020-05-16T04:43:49.638Z
    }

    如果和解析 IP 一起使用,就报错:
    input {
    file {
    path => "/var/log/nginx/access.elk.log"
    }
    }
    filter {
    grok {
    match => [ "message","%{IP:client} %{TIMESTAMP_ISO8601:locals}" ]
    }
    }
    output {
    stdout { codec => rubydebug }
    }
    结果:
    {
    "@version" => "1",
    "host" => "localhost.localdomain",
    "path" => "/var/log/nginx/access.elk.log",
    "message" => "{\"time_local\":\"2020-05-16T12:50:00+08:00\",\"remote_addr\":\"192.168.5.148\",\"referer\":\"-\",\"request\":\"GET / HTTP/1.1\",\"status\":304,\"bytes\":0,\"agent\":\"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36\",\"x_forwarded\":\"-\",\"up_addr\":\"-\",\"up_host\":\"-\",\"reqeust_time\":\"0.000\"}",
    "tags" => [
    [0] "_grokparsefailure"
    ],
    "@timestamp" => 2020-05-16T04:50:01.476Z
    }

    有大神了解这是为什么吗,只要是和解析时间的表达式一起用,就报错。%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"一起用就没问题
    2 条回复    2020-05-16 16:00:54 +08:00
    37Y37
        1
    37Y37  
       2020-05-16 13:04:12 +08:00
    直接记录 json 的应该不用 grok 解析,可以参考下这个 https://blog.ops-coffee.cn/s/cyuls7uczvwgzwptzox0dg
    polaa
        2
    polaa  
       2020-05-16 16:00:54 +08:00
    emmm 写 logstash 的时候经常出现奇怪的问题


    这种直接 codec= json 解析就行 不用 grok
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   4700 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 115ms · UTC 04:00 · PVG 12:00 · LAX 20:00 · JFK 23:00
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.