{ "version": "https://jsonfeed.org/version/1", "title": "Logstash", "description": "Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite \u201cstash.\u201d", "home_page_url": "https://www.v2ex.com/go/logstash", "feed_url": "https://www.v2ex.com/feed/logstash.json", "icon": "https://cdn.v2ex.com/navatar/b8c3/7e33/1001_large.png?m=1502861312", "favicon": "https://cdn.v2ex.com/navatar/b8c3/7e33/1001_normal.png?m=1502861312", "items": [ { "author": { "url": "https://www.v2ex.com/member/InDom", "name": "InDom", "avatar": "https://cdn.v2ex.com/avatar/b426/2e9e/530124_large.png?m=1748407942" }, "url": "https://www.v2ex.com/t/1038102", "title": "\u4f7f\u7528 logstash \u6d88\u8d39 avro \u6570\u636e\u51fa\u73b0\u90e8\u5206\u5b57\u8282\u88ab\u66ff\u6362\u4e3a ef bf bd \u5bfc\u81f4\u89e3\u6790\u51fa\u9519\u3002", "id": "https://www.v2ex.com/t/1038102", "date_published": "2024-05-06T07:41:00+00:00", "content_html": "

\u6d88\u8d39\u6e20\u9053 kafka \u4e0e redis \u5747\u51fa\u73b0\u76f8\u540c\u95ee\u9898\uff0c\u76f4\u63a5\u4ee5 plain => { charset => \"BINARY\" } \u4f9d\u7136\u51fa\u73b0\u6b64\u95ee\u9898\u3002

\n

avro \u6570\u636e\u539f\u6587\uff08 16 \u8fdb\u5236\uff09

\n
00000000: 00 80 40 00 cc e1 85 98 0e 00 3a 32 30 32 34 2d  ..@.......:2024-\n00000010: 30 34 2d 31 38 54 30 36 3a 33 32 3a 30 34 2e 30  04-18T06:32:04.0\n00000020: 30 30 2b 30 38 3a 30 30                          00+08:00\n
\n

logstash \u83b7\u53d6\u5230\u7684\u5185\u5bb9

\n
00000000: 00 ef bf bd 40 00 ef bf bd ef bf bd ef bf bd ef  ....@...........\n00000010: bf bd 0e 00 3a 32 30 32 34 2d 30 34 2d 31 38 54  ....:2024-04-18T\n00000020: 30 36 3a 33 32 3a 30 34 2e 30 30 30 2b 30 38 3a  06:32:04.000+08:\n00000030: 30 30 0a                                         00.\n
\n

logstash \u662f\u901a\u8fc7 docker \u8d77\u7684 8.13.0 \uff0c\u4ee5\u4e0b\u662f\u5bb9\u5668\u76f8\u5173\u914d\u7f6e

\n

docker-compose.yml

\n
  logstash:\n    image: logstash:8.13.0\n    volumes:\n      - /root/kafka-2-es/logstash.conf:/usr/share/logstash/pipeline/logstash.conf\n    environment:\n      - \"XPACK_MONITORING_ENABLED=false\"\n      - \"KAFKA_GROUP_ID\"\n
\n

logstash.conf

\n
input {\n  kafka {\n    bootstrap_servers => \"kafka:9092\"\n    topics => [\"urllog-test-01\"]\n    codec => plain {\n      charset => \"BINARY\"\n    }\n    group_id => \"${KAFKA_GROUP_ID}\"\n    auto_offset_reset => \"earliest\"\n  }\n  redis {\n    host => \"192.168.4.101\"\n    port => 6379\n    password => \"Fle7YuG22qIh7ZNPkceopo3oZb1UFZrX\"\n    data_type => \"list\"\n    key => \"urllog-test-01\"\n    codec => plain {\n      charset => \"BINARY\"\n    }\n  }\n}\n\noutput {\n  stdout {\n    codec => rubydebug\n  }\n  elasticsearch {\n    hosts => [\"https://192.168.1.4:9200\"]\n    index => \"urllog-test-01\"\n    user => \"elastic\"\n    password => \"123456\"\n    ssl => true\n    ssl_certificate_verification => false\n    manage_template => true\n  }\n}\n
\n

\u5df2\u7ecf\u786e\u8ba4\u7684\u662f:

\n
    \n
  1. \u4f7f\u7528 go \u4ece kafka \u4e2d\u6d88\u8d39\u83b7\u53d6\u5230\u7684\u4e8c\u8fdb\u5236\u5185\u5bb9\u662f\u6b63\u786e\u7684\u3002
    2. \n
  2. \u4f7f\u7528 logstash \u4ece kafka \u4e0e redis \u6d88\u8d39\u76f8\u540c\u7684\u5185\u5bb9\uff0c\u5747\u51fa\u73b0\u7279\u6b8a\u5b57\u7b26\u88ab\u66ff\u6362\u4e3a ef bf bd \u7684\u60c5\u51b5
    3. \n
  3. \u4f7f\u7528 kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic urllog-test-01 --from-beginning >> /tmp/data-1.txt \u76f4\u63a5\u4ece kafka \u4e2d\u5f97\u5230\u7684\u5185\u5bb9\u662f\u6b63\u786e\u7684
    4. \n
\n

\u6709\u6ca1\u6709\u5927\u4f6c\u77e5\u9053\u662f\u8fd9\u4e48\u56de\u4e8b\uff0c\u7ed9\u6307\u6761\u8def\u5457\uff1f\u4e07\u5206\u611f\u8c22\u3002

\n" }, { "author": { "url": "https://www.v2ex.com/member/watara", "name": "watara", "avatar": "https://cdn.v2ex.com/gravatar/c7722e018256b4b9b842d95d3b468775?s=73&d=retro" }, "url": "https://www.v2ex.com/t/794702", "title": "\u8bf7\u6559\u4e0b\u8fd9\u79cd\u60c5\u51b5\u4e0b\u5982\u4f55 logstash \u5904\u7406\u65f6\u95f4\u95ee\u9898\uff1f", "id": "https://www.v2ex.com/t/794702", "date_published": "2021-08-09T10:35:59+00:00", "content_html": "

\u80cc\u666f\uff1a

\n

\u56e0\u4e3a\u9879\u76ee\u66f4\u65b0\uff0c\u6253\u7b97\u5f03\u7528\u539f\u6765\u7684\u65e5\u5fd7\u6d41\u67b6\u6784\uff0c\u6362\u7528 logstash \u5904\u7406\u65e5\u5fd7\u3002\u4e3a\u4e86\u4fdd\u6301\u540e\u7eed\u6d41\u7a0b\u4e0d\u52a8\uff0c\u6240\u4ee5\u9700\u8981 logstash \u7684\u8f93\u51fa\u8981\u4fdd\u6301\u4e8e\u4e4b\u524d\u67b6\u6784\u7684\u683c\u5f0f\u4e00\u81f4\u3002

\n

\u95ee\u9898\uff1a

\n

\u65e5\u5fd7\u6d41\u9700\u8981\u5904\u7406\u7684\u662f nginx \u7684\u65e5\u5fd7\uff0c\u65e5\u5fd7\u4f53\u4e2d\u5df2\u7ecf\u6709\u4e00\u4e2a\u65f6\u95f4\u5b57\u6bb5\uff08\u5728\u8fd9\u91cc\u8bb0\u4f5c ST1 \u5427\uff0c\u539f\u59cb\u683c\u5f0f\u662f\"23/Jul/2021:14:25:29 +0800\"\uff09\uff0c\u7136\u540e\u8f93\u51fa\u7684\u5185\u5bb9\u4e2d\u4f1a\u6d89\u53ca\u5230\u65f6\u95f4\u7684\u5b57\u6bb5\uff0c\u683c\u5f0f\u4ee5\u53ca\u6765\u6e90\u5206\u522b\u5982\u4e0b\uff1a

\n

\u5b57\u6bb5 1:\"2021/7/23 14:25\",\u8bb0\u5f55 logstash \u5904\u7406\u7684\u65f6\u95f4

\n

\u5b57\u6bb5 2:\"20210723\"\uff0c\u6839\u636e nginx \u539f\u59cb\u65e5\u5fd7\u65f6\u95f4 ST1 \u8f6c\u6362\u51fa\u7684\u65e5\u671f

\n

\u5b57\u6bb5 3:\"2021/7/23 14:25\",\u6839\u636e nginx \u539f\u59cb\u65e5\u5fd7\u65f6\u95f4 ST1 \u8f6c\u6362\u51fa\u7684\u65f6\u95f4

\n

\u5b57\u6bb5 4:\"23/Jul/2021:14:25:29 +0800\"\uff0c\u539f\u59cb\u7684 nginx \u539f\u59cb\u65e5\u5fd7\u65f6\u95f4 ST1

\n

\u6211\u73b0\u5728\u80fd\u6b63\u786e\u586b\u5145\u5b57\u6bb5 4\uff0c\u5176\u4ed6 3 \u4e2a\u5b57\u6bb5\uff08\u7279\u522b\u662f\u5b57\u6bb5 2 \uff09\u6ca1\u6709\u601d\u8def\uff0c\u8bf7\u6559\u6709\u5927\u4f6c\u80fd\u6307\u70b9\u4e00\u4e8c\u4e48\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/duanlian", "name": "duanlian", "avatar": "https://cdn.v2ex.com/gravatar/a702212e42e603c6a9c936109a0cb134?s=73&d=retro" }, "url": "https://www.v2ex.com/t/718792", "date_modified": "2020-10-26T10:56:02+00:00", "content_html": "

1 \u3001\u5927\u5bb6\u6709\u7528\u5230\u54ea\u4e9b\u65b9\u6848\u5462\uff1f\n2 \u3001\u670d\u52a1\u5668\u5206\u5e03\u8de8\u533a\u57df\uff0c\u56fd\u5185\u670d\u52a1\u5668\u6bd4\u8f83\u96c6\u4e2d\uff0c\u6d77\u5916\u662f\u591a\u56fd\u5206\u5e03\uff0c\u9700\u8981\u91c7\u96c6\u65e5\u5fd7\uff0c\u8fd9\u79cd\u60c5\u51b5\uff0c\u6709\u4ec0\u4e48\u597d\u7684\u529e\u6cd5\u4e48\uff1f\n3 \u3001\u6d77\u5916\u548c\u56fd\u5185\u5404\u4e00\u5957\u7cfb\u7edf\uff1f

\n", "date_published": "2020-10-26T10:52:56+00:00", "title": "\u5404\u4f4d\u5927\u5927\uff0c\u65e5\u5fd7\u76d1\u63a7\u544a\u8b66\u7cfb\u7edf\uff0c\u5927\u5bb6\u6709\u4ec0\u4e48\u597d\u7684\u65b9\u6848\u63a8\u8350\u4e48\uff1f", "id": "https://www.v2ex.com/t/718792" }, { "author": { "url": "https://www.v2ex.com/member/wildcat007", "name": "wildcat007", "avatar": "https://cdn.v2ex.com/gravatar/7924a51a63b59135919172b62eb1780a?s=73&d=retro" }, "url": "https://www.v2ex.com/t/329635", "date_modified": "2017-08-08T20:43:58+00:00", "content_html": "\u65e5\u5fd7\u5185\u5bb9\u5982\u4e0b\r
{"id":"549","msg-type":"activity","date":"1482372059180","thread-id":"628913","query-id":"16363060","user":"uuuser","priv_user":"uuuser","ip":"10.7.32.64","cmd":"show_create_table","query":"SHOW CREATE TABLE `uc_logindata`"}\r
\r
\u642d\u5efa elk \uff0c\u4f46\u662f\u60f3\u628a date \u6539\u6210\u5317\u4eac\u65f6\u95f4\uff0c\u4e0d\u77e5\u9053 filter \u5982\u4f55\u53bb\u5199\uff1f\r
\r
\u76ee\u524d\u7684 filter \u683c\u5f0f\u5982\u4e0b\r
input {\r
\t\tfile {\r
\t\t\t\ttype => "mysql-audit"\r
\t\t\t\tpath => ["/opt/audit_logs/mysql-audit/*"]\r
}\r
}\r
\r
filter{\r
\t\tjson {\r
\t\t\t\tsource => "message"\r
\t\t\t\ttarget => "json"\r
\t\t\t\t}\r
}\r
\r
\r
output {\r
\t\tstdout { \r
\t\t\t\tcodec => rubydebug\r
\t\t\t}\r
\t\telasticsearch {\r
\t\t\t\thosts => ["172.19.11.32:9200"]\r
\t\t\t\t}\r
}\r
\r
\r
\u8f93\u51fa\u540e\u7684\u7ed3\u679c\u5982\u4e0b\r
{\r
"path" => "/opt/audit_logs/mysql-audit/test.json",\r
"@timestamp" => 2016-12-23T03:05:49.752Z,\r
"@version" => "1",\r
"host" => "0.0.0.0",\r
"json" => {\r
"date" => "1482372059180",\r
"query-id" => "16363060",\r
"msg-type" => "activity",\r
"ip" => "10.7.32.64",\r
"query" => "SHOW CREATE TABLE `uc_logindata`",\r
"priv_user" => "uuuser",\r
"id" => "7",\r
"cmd" => "show_create_table",\r
"user" => "uuuser",\r
"thread-id" => "628913"\r
},\r
"message" => "{\\"id\\":\\"7\\",\\"msg-type\\":\\"activity\\",\\"date\\":\\"1482372059180\\",\\"thread-id\\":\\"628913\\",\\"query-id\\":\\"16363060\\",\\"user\\":\\"uuuser\\",\\"priv_user\\":\\"uuuser\\",\\"ip\\":\\"10.7.32.64\\",\\"cmd\\":\\"show_create_table\\",\\"query\\":\\"SHOW CREATE TABLE `uc_logindata`\\"}",\r
"type" => "mysql-audit",\r
"tags" => []\r
}", "date_published": "2016-12-23T03:55:00+00:00", "title": "logstash \u5982\u4f55\u5c06\u65f6\u95f4\u6233 UNIX_MS \u8f6c\u5316\u4e3a\u5317\u4eac\u65f6\u95f4\uff1f", "id": "https://www.v2ex.com/t/329635" }, { "author": { "url": "https://www.v2ex.com/member/wsgzao", "name": "wsgzao", "avatar": "https://cdn.v2ex.com/avatar/10c8/844e/57018_large.png?m=1731505795" }, "url": "https://www.v2ex.com/t/217318", "date_modified": "2017-08-08T20:43:53+00:00", "content_html": "

\"\"

\n\n

\u524d\u8a00

\n\n

Elasticsearch + Logstash + Kibana \uff08 ELK \uff09\u662f\u4e00\u5957\u5f00\u6e90\u7684\u65e5\u5fd7\u7ba1\u7406\u65b9\u6848\uff0c\u5206\u6790\u7f51\u7ad9\u7684\u8bbf\u95ee\u60c5\u51b5\u65f6\u6211\u4eec\u4e00\u822c\u4f1a\u501f\u52a9 Google/\u767e\u5ea6 /CNZZ \u7b49\u65b9\u5f0f\u5d4c\u5165 JS \u505a\u6570\u636e\u7edf\u8ba1\uff0c\u4f46\u662f\u5f53\u7f51\u7ad9\u8bbf\u95ee\u5f02\u5e38\u6216\u8005\u88ab\u653b\u51fb\u65f6\u6211\u4eec\u9700\u8981\u5728\u540e\u53f0\u5206\u6790\u5982 Nginx \u7684\u5177\u4f53\u65e5\u5fd7\uff0c\u800c Nginx \u65e5\u5fd7\u5206\u5272 /GoAccess/Awstats \u90fd\u662f\u76f8\u5bf9\u7b80\u5355\u7684\u5355\u8282\u70b9\u89e3\u51b3\u65b9\u6848\uff0c\u9488\u5bf9\u5206\u5e03\u5f0f\u96c6\u7fa4\u6216\u8005\u6570\u636e\u91cf\u7ea7\u8f83\u5927\u65f6\u4f1a\u663e\u5f97\u5fc3\u6709\u4f59\u800c\u529b\u4e0d\u8db3\uff0c\u800c ELK \u7684\u51fa\u73b0\u53ef\u4ee5\u4f7f\u6211\u4eec\u4ece\u5bb9\u9762\u5bf9\u65b0\u7684\u6311\u6218\u3002

\n\n\n\n
\n

ELK (Elasticsearch + Logstash + Kibana )

\n
\n\n
\n\n

\u66f4\u65b0\u8bb0\u5f55

\n\n

2015 \u5e74 08 \u6708 31 \u65e5 - \u521d\u7a3f

\n\n

\u9605\u8bfb\u539f\u6587 - http://wsgzao.github.io/post/elk/

\n\n

\u6269\u5c55\u9605\u8bfb

\n\n

CentOS 7.x \u5b89\u88c5 ELK (Elasticsearch+Logstash+Kibana ) - http://www.chenshake.com/centos-install-7-x-elk-elasticsearchlogstashkibana/
\nCentos 6.5 \u5b89\u88c5 nginx \u65e5\u5fd7\u5206\u6790\u7cfb\u7edf elasticsearch + logstash + redis + kibana - http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=17291169&id=4898582
\nlogstash-forwarder and grok examples - https://www.ulyaoth.net/threads/logstash-forwarder-and-grok-examples.32413/
\n\u4e09\u6597\u5ba4 - http://chenlinux.com/
\nelastic - https://www.elastic.co/guide
\nLTMP \u7d22\u5f15 - http://wsgzao.github.io/index/#LTMP

\n", "date_published": "2015-08-31T06:44:21+00:00", "title": "\u4f7f\u7528 ELK(Elasticsearch + Logstash + Kibana) \u642d\u5efa\u65e5\u5fd7\u96c6\u4e2d\u5206\u6790\u5e73\u53f0\u5b9e\u8df5", "id": "https://www.v2ex.com/t/217318" }, { "author": { "url": "https://www.v2ex.com/member/zjf061315", "name": "zjf061315", "avatar": "https://cdn.v2ex.com/gravatar/756c00b6c0a37d49d9eca7f39b7011da?s=73&d=retro" }, "url": "https://www.v2ex.com/t/214926", "date_modified": "2017-08-08T20:44:19+00:00", "content_html": "

\u5728\u7528\u7740\u4e09\u4e2a\u5de5\u5177\u642d\u4e00\u4e2a\u65e5\u5fd7\u5904\u7406\u7684\u5e73\u53f0\uff0c\u76ee\u524d\u6709\u4e24\u4e2a\u4e0d\u540c\u7684\u7cfb\u7edf\u65e5\u5fd7\u9700\u8981\u5904\u7406\u3002
\n\u5f00\u59cb\u8003\u8651\u5728\u4e00\u53f0\u673a\u5668\u4e0a\u542f\u52a8\u4e24\u4e2a logstash \u5b9e\u4f8b\uff0c\u6536\u96c6\u65e5\u5fd7\u540e\u53d1\u5230 elasticsearch \u4e0a\uff0c\u4e0d\u77e5\u9053\u4e3a\u4ec0\u4e48\u7b2c\u4e8c\u4e2a\u603b\u662f\u8fde\u63a5\u4e0d\u4e0a\uff0c\u4f20\u4e0d\u51fa\u6570\u636e\uff0c\u8fdb\u800c\u8003\u8651\u7528\u4e00\u4e2a logstash \u5b9e\u4f8b\uff0c\u7136\u540e\u5728\u914d\u7f6e\u6587\u4ef6\u91cc\u7ed9\u6587\u4ef6\u52a0 type \uff0c\u5728 output \u4e2d\u6839\u636e type \u5224\u65ad\u5904\u7406\uff0c\u4f46\u662f\u8fd8\u662f\u6709\u4e00\u4e2a\u6761\u4ef6\u91cc\u7684\u4e0d\u6210\u529f\u554a\u3002\u3002\u3002\u6c42\u52a9\u5927\u5bb6\uff0c\u76ee\u524d\u8f93\u51fa\u7684\u914d\u7f6e\u6587\u4ef6\u5982\u4e0b
\noutput {

\n
if [type] == "targeted_coupon" {\n    elasticsearch {\n        host     => ["127.0.0.1:9300"]\n        cluster  => "cluster_spark"\n        protocol => "node"\n        index    => "coupon_%{+YYYYMMdd}"\n        max_retries => 10\n    }\n    #stdout { codec => rubydebug }\n\n} else {\n    elasticsearch {\n        host    => ["127.0.0.1:9300"]\n        cluster => "cluster_spark"\n        protocol => "node"\n        index   => "gift_%{+YYYYMMdd}"\n        document_type  => "%{type}"\n        max_retries => 10\n    }\n    #stdout { codec => rubydebug }\n}\n
\n", "date_published": "2015-08-21T09:21:00+00:00", "title": "logstash+elasticsearch+kibana \u95ee\u9898", "id": "https://www.v2ex.com/t/214926" } ] }