基於CentOS6.5或Ubuntu14.04下Suricata裏搭配安裝 ELK (elasticsearch, logstash, kibana)（圖文詳解）

時間 2019-11-29

標籤基於 centos6.5 centos ubuntu14.04 ubuntu suricata 搭配安裝 elk elasticsearch logstash kibana 圖文詳解欄目 CentOS 简体版

原文原文鏈接

　　前期博客html

基於CentOS6.5下Suricata（一款高性能的網絡IDS、IPS和網絡安全監控引擎）的搭建（圖文詳解）（博主推薦）

基於Ubuntu14.04下Suricata（一款高性能的網絡IDS、IPS和網絡安全監控引擎）的搭建（圖文詳解）（博主推薦）

　　參考官網java

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output

　　注意，這官網，默認是適合Ubuntu系統的安裝手法。node

　　首先，jquery

　　Make sure your Suricata is compiled/installed with libjansson support enabled:linux

[root@suricata ~]# suricata --build-info
This is Suricata version 3.1 RELEASE
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS HAVE_LIBJANSSON TLS 
SIMD support: SSE_4_2 SSE_4_1 SSE_3 
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.4.7 20120313 (Red Hat 4.4.7-18), C version 199901
compiled with _FORTIFY_SOURCE=0
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.20, linked against LibHTP v0.5.20

Suricata Configuration:
  AF_PACKET support:                       yes
  PF_RING support:                         no
  NFQueue support:                         no
  NFLOG support:                           no
  IPFW support:                            no
  Netmap support:                          no
  DAG enabled:                             no
  Napatech enabled:                        no

  Unix socket enabled:                     yes
  Detection enabled:                       yes

  libnss support:                          yes
  libnspr support:                         yes
  libjansson support:                      yes
  hiredis support:                         no
  Prelude support:                         no
  PCRE jit:                                no
  LUA support:                             no
  libluajit:                               no
  libgeoip:                                no
  Non-bundled htp:                         no
  Old barnyard2 support:                   no
  CUDA enabled:                            no
  Hyperscan support:                       no
  Libnet support:                          yes

  Suricatasc install:                      yes

  Profiling enabled:                       no
  Profiling locks enabled:                 no

Development settings:
  Coccinelle / spatch:                     no
  Unit tests enabled:                      no
  Debug output enabled:                    no
  Debug validation enabled:                no

Generic build parameters:
  Installation prefix:                     /usr/local
  Configuration directory:                 /usr/local/etc/suricata/
  Log directory:                           /usr/local/var/log/suricata/

  --prefix                                 /usr/local
  --sysconfdir                             /usr/local/etc
  --localstatedir                          /usr/local/var

  Host:                                    x86_64-pc-linux-gnu
  Compiler:                                gcc (exec name) / gcc (real)
  GCC Protect enabled:                     no
  GCC march native enabled:                yes
  GCC Profile enabled:                     no
  Position Independent Executable enabled: no
  CFLAGS                                   -g -O2 -march=native
  PCAP_CFLAGS                               
  SECCFLAGS                                
[root@suricata ~]#

　　而後，須要配置git

Configure suricata

[root@suricata suricata-3.1]# pwd
/root/suricata-3.1
[root@suricata suricata-3.1]# ls
aclocal.m4             compile       config.h.in   config.status  configure.ac  depcomp     libhtp   ltmain.sh  Makefile.am  qa                scripts   suricata.yaml
ChangeLog              config.guess  config.log    config.sub     contrib       doc         libtool  m4         Makefile.in  reference.config  src       suricata.yaml.in
classification.config  config.h      config.rpath  configure      COPYING       install-sh  LICENSE  Makefile   missing      rules             stamp-h1  threshold.config
[root@suricata suricata-3.1]# sudo vim suricata.yaml

 # Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve.json
      #prefix: "@cee: " # prefix to prepend to each log entry
      # the following are valid when type: syslog above
      #identity: "suricata"
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      #redis:
      #  server: 127.0.0.1
      #  port: 6379
      #  mode: list ## possible values: list (default), channel
      #  key: suricata ## key or channel to use (default to suricata)
      # Redis pipelining set up. This will enable to only do a query every
      # 'batch-size' events. This should lower the latency induced by network
      # connection at the cost of some memory. There is no flushing implemented
      # so this setting as to be reserved to high traffic suricata.
      #  pipelining:
      #    enabled: yes ## set enable to yes to enable query pipelining
      #    batch-size: 10 ## number of entry to keep in buffer
 types:
        - alert:
            # payload: yes             # enable dumping payload in Base64
            # payload-buffer-size: 4kb # max size of payload buffer to output in eve-log
            # payload-printable: yes   # enable dumping payload in printable (lossy) format
            # packet: yes              # enable dumping of packet (without stream segments)
            http: yes                # enable dumping of http fields
            tls: yes                 # enable dumping of tls fields
            ssh: yes                 # enable dumping of ssh fields
            smtp: yes                # enable dumping of smtp fields

            # HTTP X-Forwarded-For support by adding an extra field or overwriting
            # the source or destination IP address (depending on flow direction)
            # with the one reported in the X-Forwarded-For HTTP header. This is
            # helpful when reviewing alerts for traffic that is being reverse
            # or forward proxied.
            xff:
              enabled: no
              # Two operation modes are available, "extra-data" and "overwrite".
              mode: extra-data
              # Two proxy deployments are supported, "reverse" and "forward". In
              # a "reverse" deployment the IP address used is the last one, in a
              # "forward" deployment the first IP address is used.
              deployment: reverse
              # Header name where the actual IP address will be reported, if more
              # than one IP address is present, the last IP address will be the
              # one taken into consideration.
              header: X-Forwarded-For
 - http:
            extended: yes     # enable this for extended logging information
            # custom allows additional http fields to be included in eve-log
            # the example below adds three additional fields when uncommented
            #custom: [Accept-Encoding, Accept-Language, Authorization]
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: no   # force logging magic on all logged files
            force-md5: no     # force logging of md5 checksums
        #- drop:
        #    alerts: no       # log alerts that caused drops
 - smtp:
            #extended: yes # enable this for extended logging information
            # this includes: bcc, message-id, subject, x_mailer, user-agent
            # custom fields logging from the list:
            #  reply-to, bcc, message-id, subject, x-mailer, user-agent, received,
            #  x-originating-ip, in-reply-to, references, importance, priority,
            #  sensitivity, organization, content-md5, date
            #custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
            # output md5 of fields: body, subject
            # for the body you need to set app-layer.protocols.smtp.mime.body-md5
            # to yes
            #md5: [body, subject]

        - ssh
        - stats:
            totals: yes       # stats for all threads merged together
            threads: no       # per thread stats
            deltas: no        # include delta values
        # bi-directional flows
        - flow
        # uni-directional flows
        #- netflow

　　改成redis

# "United" event log in JSON format
  - eve-log:
      enabled: yes
      type: file #file|syslog|unix_dgram|unix_stream
      filename: eve.json
      # the following are valid when type: syslog above
      #identity: "suricata" 
      #facility: local5
      #level: Info ## possible levels: Emergency, Alert, Critical,
                   ## Error, Warning, Notice, Info, Debug
      types:
        - alert
        - http:
            extended: yes     # enable this for extended logging information
        - dns
        - tls:
            extended: yes     # enable this for extended logging information
        - files:
            force-magic: yes   # force logging magic on all logged files
            force-md5: yes     # force logging of md5 checksums
        #- drop
        - ssh
        - smtp

　　而後，接下是安裝apache

Install ELK (elasticsearch, logstash, kibana)

　　先卸載自帶的openjdkjson

rpm -qa|grep java

rpm -e --nodeps tzdata-java-2013g-1.el6.noarch
rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64



　　注意：這裏必定要是jdk1.8，否則會報以下錯誤

[root@suricata ~]# sudo /etc/init.d/elasticsearch start
Starting elasticsearch: Elasticsearch requires at least Java 8 but your Java version from /usr/local/jdk/jdk1.7.0_79/bin/java does not meet this requirement
                                                           [FAILED]
[root@suricata ~]#

　　官網提示了，建議安裝orcal 的 jdk。ubuntu

　　我這裏安裝jdk18。

[root@suricata suricata-3.1]# cd /usr/local/
[root@suricata local]# ls
bin  etc  games  include  lib  lib64  libexec  sbin  share  src  var
[root@suricata local]# mkdir jdk
[root@suricata local]# cd jdk/
[root@suricata jdk]# pwd
/usr/local/jdk
[root@suricata jdk]# ll
total 0
[root@suricata jdk]# rz

[root@suricata jdk]# pwd
/usr/local/jdk
[root@suricata jdk]# ll
total 176992
-rw-r--r--. 1 root root 181238643 Sep  3  2016 jdk-8u60-linux-x64.tar.gz
[root@suricata jdk]# tar -zxvf jdk-8u60-linux-x64.tar.gz

　　而後，再vim /etc/profile

#java
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

　　文件生效

source /etc/profile

　　而後，接下來就是安裝ELk和配置JDK。（注意如下只是個例子）

wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.tar.gz
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.6.1.deb
wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.3-1_all.deb

tar -C /var/www/ -xzf kibana-3.0.0.tar.gz
dpkg -i elasticsearch-1.6.1.deb
dpkg -i logstash_1.5.3-1_all.deb

　　我這裏啊，Elasticsearch選取5.5.0

　　因此，總的是

[root@suricata ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.deb

[root@suricata ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.deb

[root@suricata ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.0-linux-x86_64.tar.gz

　　最後下載，獲得

[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg  Desktop    Downloads                install.log         kibana-5.5.0-linux-x86_64.tar.gz  Music     Public        suricata-3.1.tar.gz  Videos
data             Documents  elasticsearch-5.5.0.deb  install.log.syslog  logstash-5.5.0.deb                Pictures  suricata-3.1  Templates
[root@suricata ~]#

　　而後，進行對kibana的安裝

[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg  Desktop    Downloads                install.log         kibana-5.5.0-linux-x86_64.tar.gz  Music     Public        suricata-3.1.tar.gz  Videos
data             Documents  elasticsearch-5.5.0.deb  install.log.syslog  logstash-5.5.0.deb                Pictures  suricata-3.1  Templates
[root@suricata ~]# tar -C /var/www/ -xzf kibana-5.5.0-linux-x86_64.tar.gz

　　獲得

[root@suricata www]# pwd
/var/www
[root@suricata www]# ls
cgi-bin  error  html  icons  kibana-5.5.0-linux-x86_64
[root@suricata www]#

　　若你們沒安裝過這個命令，則安裝下便可。

[root@suricata ~]# yum -y install dpkg

　　而後，進行對elasticsearch的安裝

[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg  Desktop    Downloads                install.log         kibana-5.5.0-linux-x86_64.tar.gz  Music     Public        suricata-3.1.tar.gz  Videos
data             Documents  elasticsearch-5.5.0.deb  install.log.syslog  logstash-5.5.0.deb                Pictures  suricata-3.1  Templates
[root@suricata ~]# dpkg -i elasticsearch-5.5.0.deb

　　這裏啊，我建議你們仍是去安裝tar.gz包，這種方式比較好。

　　因此，後面，我仍是下載了，rpm包，以這種方式來進行安裝。

[root@suricata ~]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.rpm

[root@suricata ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.rpm

　　　進行elasticsearch-5.5.0的安裝

[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg  Desktop    Downloads                elasticsearch-5.5.0.rpm  install.log.syslog                logstash-5.5.0.deb  Music     Public        suricata-3.1.tar.gz  Videos
data             Documents  elasticsearch-5.5.0.deb  install.log              kibana-5.5.0-linux-x86_64.tar.gz  logstash-5.5.0.rpm  Pictures  suricata-3.1  Templates

[root@suricata ~]# rpm -ivh elasticsearch-5.5.0.rpm
warning: elasticsearch-5.5.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing... ########################################### [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
1:elasticsearch ########################################### [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig
sudo chkconfig --add elasticsearch
### You can start elasticsearch service by executing
sudo service elasticsearch start

　安裝完，默認即有

　　而後，進行logstash的安裝

[root@suricata ~]# pwd
/root
[root@suricata ~]# ls
anaconda-ks.cfg  Desktop    Downloads                elasticsearch-5.5.0.rpm  install.log.syslog                logstash-5.5.0.deb  Music     Public        suricata-3.1.tar.gz  Videos
data             Documents  elasticsearch-5.5.0.deb  install.log              kibana-5.5.0-linux-x86_64.tar.gz  logstash-5.5.0.rpm  Pictures  suricata-3.1  Templates
[root@suricata ~]# rpm -ivh logstash-5.5.0.rpm
warning: logstash-5.5.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                ########################################### [100%]
   1:logstash               ########################################### [100%]
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/jruby/bin/jruby: line 388: /usr/bin/java: No such file or directory
/usr/share/logstash/vendor/jruby/bin/jruby: line 388: exec: /usr/bin/java: cannot execute: No such file or directory
Unable to install system startup script for Logstash.
[root@suricata ~]#

　　安裝完，默認即有

　　而後，下面是

Logstash configuration

[root@suricata conf.d]# pwd
/etc/logstash/conf.d
[root@suricata conf.d]# ll
total 0
[root@suricata conf.d]# touch /etc/logstash/conf.d/logstash.conf

　　插入以下內容

input {
  file { 
    path => ["/var/log/suricata/eve.json"]
    sincedb_path => ["/var/lib/logstash/"]
    codec =>   json 
    type => "SuricataIDPS" 
  }

}

filter {
  if [type] == "SuricataIDPS" {
    date {
      match => [ "timestamp", "ISO8601" ]
    }
    ruby {
      code => "if event['event_type'] == 'fileinfo'; event['fileinfo']['type']=event['fileinfo']['magic'].to_s.split(',')[0]; end;" 
    }
  }

  if [src_ip]  {
    geoip {
      source => "src_ip" 
      target => "geoip" 
      #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" 
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float" ]
    }
    if ![geoip.ip] {
      if [dest_ip]  {
        geoip {
          source => "dest_ip" 
          target => "geoip" 
          #database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" 
          add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
          add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }
        mutate {
          convert => [ "[geoip][coordinates]", "float" ]
        }
      }
    }
  }
}

output { 
  elasticsearch {
    host => localhost
    #protocol => http
  }
}

　　而後，再

Configure the start-up services

update-rc.d elasticsearch defaults 95 10 （這兩個是ubuntu系統裏的作法，我這裏是centos系統，不須要）
update-rc.d logstash defaults

service httpd start                    啓動apache
service elasticsearch start
service logstash start

或者

啓動服務sudo /etc/init.d/elasticsearch start
中止服務sudo /etc/init.d/elasticsearch stop

啓動服務sudo /etc/init.d/logstash/start
中止服務sudo /etc/init.d/logstashstop

　　若是你們在啓動過程當中，出現以下問題。

[root@suricata ~]# sudo /etc/init.d/elasticsearch start
which: no java in (/sbin:/usr/sbin:/bin:/usr/bin)
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME
[root@suricata ~]#

　　則見

ElasticSearch裏JAVA環境丟失的問題

　　-rwxr-x---. 1 root root 5113 Jul 1 07:20 elasticsearch ，通過

[root@suricata init.d]# chmod 755  elasticsearch
[root@suricata init.d]# pwd
/etc/init.d
[root@suricata init.d]#

　　改成

-rwxr-xr-x. 1 root root 5113 Jul 1 07:20 elasticsearch

　　你們，自行按照本身的jdk位置。

export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60

　　而後，就能夠成功了。

[root@suricata ~]# sudo /etc/init.d/elasticsearch start
Starting elasticsearch:                                    [  OK  ]
[root@suricata ~]#

　　而後，你們也許在啓動，會遇到這樣的問題：

Logstash默認的配置文件位置爲./config和/etc/logstash/，後者已經存在，但直接運行依然會報錯：

　　WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash.

　　見

Logstash使用grok解析IIS日誌

　　簡單起見，直接在Logstash根目錄建立軟連接便可：

cd /usr/share/logstash

[root@suricata logstash]# pwd
/usr/share/logstash
[root@suricata logstash]# ln -s /etc/logstash ./config
[root@suricata logstash]#

　　設置配置文件實時生效，從而不用頻繁地啓停Logstash。

　　修改/etc/logstash/logstash.yml

config.reload.automatic: true

　　Logstash的運行

　　與Elasticsearch、Kibana不一樣，Logstash默認並不做爲系統服務安裝，咱們也不建議做爲服務啓動。主要緣由爲：大多數狀況下，Elasticsearch、Kibana在每臺服務器上只有一個實例，能夠做爲服務啓動；而一個Logstash實例通常只服務於一個數據流，所以一臺服務器上常常運行多個實例。編寫一個配置文件helloworld.conf進行測試。注意，不要把該文件放在/etc/logstash/conf.d下，以免不一樣實例間產生混淆，由於該文件夾是Logstash默認讀取的文件夾。簡單起見，可直接放在Logstash根目錄下。

input { 
    stdin {} 
}
output { 
    stdout {}
}

　　這是一個命令行輸入、命令行輸出的Logstash實例，運行成功則說明已配置正確：

bin/logstash -f helloworld.conf

　　雖然如此，不少時候咱們仍是但願各個Logstash實例能隨系統而啓動並在後臺運行。將其加入計劃任務便可：

 nohup /usr/share/logstash/bin/logstash -f /usr/share/logstash/helloworld.conf > /dev/null &

　　而後，

　　Now make sure Suricata is running and you have logs written in your JSON log files

http://localhost/kibana-5.5.0

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output

　　繼續

相關標籤/搜索

圖文詳解

elasticsearch+logstash+kibana

kibana+logstash+elasticsearch

elasticsearch+kibana+logstash+ik

elasticsearch+kibana+logstash

filebeat+logstash+elasticsearch+kibana

logstash&redis&elasticsearch&kibana

elasticsearch+kibana+logstash+filebeat

elasticsearch+logstash+kibana+redis

elasticsearch+logstash+filebeat+kibana

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。