filebeat和ELK全用了6.2.4了，kafka是1.1.0，filebeat写入kafka后，所有信息都保存在message字段中，怎么才能把message里面的字段都单独分离出来呢？ - Elastic 中文社区

我用filebeat收集的是json格式的nginx日志，filebeat收集的信息用output:kafka到达kafka后是这样：

{"@timestamp":"2018-05-07T14:47:43.586Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.4","topic":"elk-nginx"},"source":"/usr/local/nginx/logs/access_json.log","offset":741815,"json":{},"message":"{ \"@timestamp\": \"2018-05-07T19:04:25+08:00\", \"remote_addr\": \"192.168.6.1\", \"remote_user\": \"-\", \"status\": \"200\", \"body_bytes_sent\": \"73\", \"request\": \"POST /jsrpc.php?output=json-rpc HTTP/1.1\", \"request_method\": \"POST\", \"http_referrer\": \"http://192.168.6.71/overview.php?ddreset=1\", \"http_user_agent\": \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331\", \"http_x_forwarded_for\": \"-\", \"request_time\": \"0.076\", \"request_body\": \"\"{\\x22jsonrpc\\x22: \\x222.0\\x22, \\x22method\\x22: \\x22zabbix.status\\x22, \\x22params\\x22: {}, \\x22auth\\x22: \\x228e125896e94285e47e1313be49d5cb55\\x22, \\x22id\\x22: 3}\" }}","prospector":{"type":"log"},"beat":{"name":"master","hostname":"master","version":"6.2.4"}}

logstash用input-kafka后是这样：

{
        "source" => "/usr/local/nginx/logs/access_json.log",
        "offset" => 738665,
       "message" => "{ \"@timestamp\": \"2018-05-07T19:04:08+08:00\", \"remote_addr\": \"192.168.6.1\", \"remote_user\": \"-\", \"status\": \"200\", \"body_bytes_sent\": \"25686\", \"request\": \"GET /overview.php?ddreset=1 HTTP/1.1\", \"request_method\": \"GET\", \"http_referrer\": \"http://192.168.6.71/overview.php?ddreset=1\", \"http_user_agent\": \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.90 Safari/537.36 2345Explorer/9.3.2.17331\", \"http_x_forwarded_for\": \"-\", \"request_time\": \"0.153\", \"request_body\": \"\"-\" }}",
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "type" => "accesslog",
          "beat" => {
            "name" => "master",
         "version" => "6.2.4",
        "hostname" => "master"
    },
          "json" => {},
    "prospector" => {
        "type" => "log"
    },
    "@timestamp" => 2018-05-07T14:47:43.586Z
}

**发现所有需要的信息都在message段里面，现在如何能把message里面的字段都单独分离出来呢？
类似于下面这个（从网上找的），让message中的每个字段都单独列出来：**

{
        "message" => "192.168.154.2 - - [30/Mar/2017:01:27:09 -0700] \"GET /index.html HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\" \"-\"",
       "@version" => "1",
     "@timestamp" => "2017-03-30T08:27:09.539Z",
           "path" => "/var/log/nginx/access.log",
           "host" => "spark4",
           "type" => "nginxlog",
      "remote_ip" => "192.168.154.2",
      "timestamp" => "30/Mar/2017:01:27:09 -0700",
         "method" => "GET",
        "request" => "/index.html",
    "httpversion" => "1.1",
         "status" => "304",
          "bytes" => "0",
        "referer" => "\"-\"",
          "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36\"",
       "xforward" => "\"-\""
}

Original url: Access
Created at: 2018-12-18 18:11:14
Category: default
Tags: none

未标明原创文章均为采集，版权归作者所有，转载无需和我联系，请注明原出处，南摩阿彌陀佛，知识，不只知道，要得到

请先后发表评论
  • 最新评论
  • 总共0条评论