一、介绍

当我们部署集群服务器的时候，日志文件就会散落在多台服务器上。查看日志信息就需要到各个服务器上去取和查看，我们把这些日志文件归集到一个地方统一管理。这个时候ELK系统出现了，ELK是elasticsearch、Logstashh和Kibana三个系统的首字母组合。

二、组件

  • Elasticsearch
    • ELK-ES1(10.74) 192.168.10.74
    • ELK-ES2(10.75) 192.168.10.75
    • ELK-ES3(10.76) 192.168.10.76
  • Logstash
    • ELK-logstash1(10.72) 192.168.10.72
    • ELK-logstash2(10.73) 192.168.10.73
  • Redis
    • ELK-redis(10.71) 192.168.10.71
  • Kibana
    • ELK-kibana(10.71) 192.168.10.71
  • Client(filebeat)
    • ELK-client(10.70) 192.168.10.70

三、架构图

其中filebeat作为agent采集日志，redis作为队列

四、组件安装

1. filebeat安装
参考文档：https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html
2. redis安装
参考脚本：https://github.com/xiangys0134/deploy/blob/master/software_install/redis/redis_install.sh
3. logstash安装
参考文档：https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
4. kibana安装
参考文档：https://www.elastic.co/guide/en/kibana/current/rpm.html
5. elasticsearch安装
参考文档：https://www.elastic.co/guide/en/elasticsearch/reference/6.7/install-elasticsearch.html

五、 日志抓取

1. 抓取nginx日志
nginx格式参数：
    log_format main 'remote_addr -remote_user [time_local] "request" http_host '
                        'status request_lengthbody_bytes_sent "http_referer" '
                        '"http_user_agent" request_timeupstream_response_time';
2.抓取规则
grok匹配正则(含多个)：
(?<ip>[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\s[\d\w-]+\s[\d\w-]+\s+.*\[(?<time>.*)\]\s+\"(?<curl>.*)\"\s+\"(?<http_user_agent>.*)\"\s+(?<request_time>\d+\.{0,}\d+)\s+(?<response_time>\d+\.{0,}\d+)

(?<ip>[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\s[\d\w-]+\s[\d\w-]+\s+.*\[(?<time>.*)\]\s+\"(?<curl>.*)\"\s+\"(?<http_user_agent>.*)\"\s+(?<request_time>\d+\.{0,}\d+)\s+(?<response_time>[-])

备注：此规则需要根据日志结构编写对应变量，logstash如果匹配规则一直匹配不到则很耗费资源，其本身也很耗费资源
3. 配置filebeat将日志传入redis
filebeat端配置：
[root@localhost filebeat]# cd /etc/filebeat
[root@elk-client filebeat]# vi filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/*access.log
  fields:
    web: "192.168.10.70"
output.redis:
  hosts: ["192.168.10.71"]
  port: "6380"
  password: "123"
  key: "web"
  db: 
  timeout: 5
4.测试查看redis中的数据

5. logstash读取redis中的数据
[root@elk-logstash1 ~]# cd /etc/logstash/conf.d/
[root@elk-logstash1 conf.d]# vi 192.168.0.148-access.oms.yml
#nginx access访问日志解析,分别推送至zabbix和ES中
input {
  redis {
    #id => "my_plugin_id"
    host => "192.168.10.71"
    password => "123"
    port => "6380"
    db => 0
    data_type => "list"
    key => "192.168.0.148-access_oms.log"
    type => "nginx-access-192.168.0.148"
  }
}


filter {
    grok {
    match => [
        "message", "(?<ip>[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\s[\d\w-]+\s[\d\w-]+\s+.*\[(?<time>.*)\]\s+\"(?<curl>[(GET)|(POST)]{1,}\s+.*HTTP/1.[01])\"\s+(?<host>(\d+.){3}\d+)\s+(?<status>\d+)\s+(?<length>\d+)\s+(?<bytes_sent>\d+)\s+\"(?<referer>.*)\"\s+\"(?<agent>.*)\"\s+(?<response_time>\d+.{0,1}\d{0,})\s+(?<upstream_response_time>\d+.{0,1}\d{0,})",
        "message", "(?<ip>[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\s[\d\w-]+\s[\d\w-]+\s+.*\[(?<time>.*)\]\s+\"(?<curl>[(GET)|(POST)]{1,} / HTTP/1.1)\"\s+(?<host>(\d+.){3}\d)\s+(?<status>\d+)\s+(?<length>\d+)\s+(?<bytes_sent>\d+)\s+\"(?<referer>[-]{0,}[\w+]{0,})\"\s+\"(?<user_agent>.*)\"\s{1,}(?<response_time>\d+.{0,1}\d+)",
        "message", "(?<ip>[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\s[\d\w-]+\s[\d\w-]+\s+.*\[(?<time>.*)\]\s+\"(?<curl>[(GET)|(POST)]{1,}\s+.*HTTP/1.[01])\"\s+(?<host>(\d+.){3}\d+\:{0,1}\d{0,5})\s+(?<status>\d+)\s+(?<length>\d+)\s+(?<bytes_sent>\d+)\s+\"(?<referer>.*)\"\s+\"(?<agent>.*)\"\s+(?<response_time>\d+.{0,1}\d{0,})\s+(?<upstream_response_time>\d+.{0,1}\d{0,})"
    ]
        remove_field => ["message"]
        timeout_millis => 10000

    }
    date {
      #match => ["mydate", "MMM dd HH:mm:ss"]
      match => ["time", "dd/MMM/yyyy:HH:mm:ss Z"]
      target => "@timestamp"
    }
}

output {
  if [fields][web] == "192.168.0.148-access_oms.log" {
    if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
      elasticsearch {
        hosts => ["http://192.168.10.74:9200","http://192.168.10.75:9200","http://192.168.10.76:9200"]
        index => "192.168.0.148-access_oms.log-%{+YYYY.MM.dd}"
      }
   }
 }

  if [fields][web] == "192.168.0.148-access_pms.log" {
    if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
      elasticsearch {
        hosts => ["http://192.168.10.74:9200","http://192.168.10.75:9200","http://192.168.10.76:9200"]
        index => "192.168.0.148-access_pms.log-%{+YYYY.MM.dd}"
      }
   }
 }


  if [fields][web] == "192.168.0.148-access_rms.log" {
    if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
      elasticsearch {
        hosts => ["http://192.168.10.74:9200","http://192.168.10.75:9200","http://192.168.10.76:9200"]
        index => "192.168.0.148-access_rms.log-%{+YYYY.MM.dd}"
      }
   }
 }


  if [fields][web] == "192.168.0.148-access_est.log" {
    if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
      elasticsearch {
        hosts => ["http://192.168.10.74:9200","http://192.168.10.75:9200","http://192.168.10.76:9200"]
        index => "192.168.0.148-access_est.log-%{+YYYY.MM.dd}"
      }
   }
 }


  if [fields][web] == "192.168.0.148-access_utility.log" {
    if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
      elasticsearch {
        hosts => ["http://192.168.10.74:9200","http://192.168.10.75:9200","http://192.168.10.76:9200"]
        index => "192.168.0.148-access_utility.log-%{+YYYY.MM.dd}"
      }
   }
 }


  if [fields][web] == "192.168.0.148-access_uds.log" {
    if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
      elasticsearch {
        hosts => ["http://192.168.10.74:9200","http://192.168.10.75:9200","http://192.168.10.76:9200"]
        index => "192.168.0.148-access_uds.log-%{+YYYY.MM.dd}"
      }
   }
 }



  if [fields][web] == "192.168.0.148-access_bms.log" {
    if "_grokparsefailure" not in [tags] and "_dateparsefailure" not in [tags] {
      elasticsearch {
        hosts => ["http://192.168.10.74:9200","http://192.168.10.75:9200","http://192.168.10.76:9200"]
        index => "192.168.0.148-access_bms.log-%{+YYYY.MM.dd}"
      }
   }
 }


  stdout {
    codec => rubydebug
  }

}


备注：目前在使用过程中发现无法进行logstash集群的部署，也可能是自身的知识点不够

六、 supervisor进程管理

ELK平时也没有接入过大型日志系统，所以自身的知识面也不够。以下为supervisor管理配置文件示例：

[root@elk-logstash1 supervisord.d]# cat logstash-worker192.168.0.148.ini 
[program:logstash-192.168.0.148web-server]
process_name=%(program_name)s_%(process_num)02d
environment=JAVA_HOME="/usr/local/java/jdk"
environment=PATH="/usr/local/java/jdk/bin:/usr/local/java/jdk/jre/bin:/usr/share/logstash/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin"
command=/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/192.168.0.148-access.oms.yml
autostart=true
autorestart=true
user=root
numprocs=1
redirect_stderr=true
stdout_logfile = /tmp/logstash-192.168.0.148web.log
最后修改日期: 2023年12月13日

作者

留言

撰写回覆或留言

发布留言必须填写的电子邮件地址不会公开。