Ubuntu 下安裝Kibana和logstash

原文地址：http://www.cnblogs.com/saintaxl/p/3946667.html 

  簡單來說他具體的工做流程就是 logstash agent 監控並過濾日誌，將過濾後的日誌內容發給redis(這裏的redis只處理隊列不作存儲)，logstash index將日誌收集在一塊兒交給全文搜索服務ElasticSearch 能夠用ElasticSearch進行自定義搜索 經過Kibana 來結合 自定義搜索進行頁面展現

• ruby 運行Kibana 必須
• rubygems 安裝ruby擴展必須
• bundler 功能相似於yum
• JDK 運行java程序必須 
• redis 用來處理日誌隊列
• logstash 收集、過濾日誌
• ElasticSearch 全文搜索服務(logstash集成了一個)

kibana 頁面展現

首先到 logstash index服務器上面，logstash分爲 index和aget ，agent負責監控、過濾日誌，index負責收集日誌並將日誌交給ElasticSearch 作搜索此外 logstash 的收集方式分爲 standalone 和 centralized。

standalone 是全部功能都在一個服務器上面，自發自收，centralized 就是集中收集，一臺服務器接收全部shipper(我的理解就是logstash agent)的日誌。

其實 logstash自己不分 什麼 shipper 和 collector ，只不過就是配置文件不一樣而已，咱們此次按照集中的方式來測試

這裏有兩臺服務器

192.168.124.128 logstash index,ElasticSearch,kibana,JDK
192.168.124.132 logstash agent,redis,JDK

準備工做

安裝：openssl

卸載舊版本

apt-get remove openssl
apt-get autoremove openssl

下載最新版本

wget http://www.openssl.org/source/openssl-1.0.1i.tar.gz

tar -zxvf openssl-1.0.1i.tar.gz
cd /opt/openssl-1.0.1i
./config --prefix=/usr/local/ssl
make & make install

創建軟鏈接

ln -s /usr/local/ssl/bin/openssl /usr/bin/openssl
ln -s /usr/local/ssl/include/openssl /usr/include/openssl

刷新動態配置

vim /etc/ld.so.conf

在文末插入一行

/usr/local/ssl/lib
ldconfig -v

測試

openssl version -a

 

安裝PCRE庫

wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.33.tar.gz

tar -zxvf pcre-8.33.tar.gz
cd pcre-8.33
./configure --prefix=/usr/local/pcre-8.33
make & make install

 

安裝zlib

wget http://zlib.net/zlib-1.2.8.tar.gz

tar -zxvf zlib-1.2.8.tar.gz
cd zlib-1.2.8
./configure --prefix=/usr/local/zlib-1.2.8
make & make install

 

安裝nginx

wget http://nginx.org/download/nginx-1.6.1.tar.gz

tar -zxvf nginx-1.6.1.tar.gz
cd nginx-1.6.1
./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-openssl=/opt/openssl-1.0.1i --with-pcre=/opt/pcre-8.33 --with-zlib=/opt/zlib-1.2.8

 nginx 命令

啓動：/usr/local/nginx/sbin/nginx
重啓：/usr/local/nginx/sbin/nginx –s reload
中止：/usr/local/nginx/sbin/nginx -s stop
查看主進程：netstat -ntlp
檢查是否啓動成功：netstat -ano|grep 80

 

安裝ruby 運行Kibana 必須

sudo apt-get update  
wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.2.tar.gz
./configure --prefix=/usr/local/ruby
make && make install

 

環境設置

vi /etc/environment

將Ruby的路徑加入環境變量 中並保存/etc/environment，以下面內容：

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/ruby/bin"

修改了環境變量文件後，須要經過source命令讓修改立刻生效，命令以下：

$ source /etc/environment

 爲了檢查安裝是否成功，能夠輸入下面的命令進行測試 ：

$ruby –v

確認安裝成功後經過一下命令添加命令連接，目前我也不清楚建立這些連接的目的是什麼，按照Ruby「約定大於配置」的原則，應該是一種約定。(keyboardota)

$ sudo ln -s /usr/local/ruby/bin/ruby /usr/local/bin/ruby
$ sudo ln -s /usr/local/ruby/bin/gem /usr/bin/gem

或者：

apt-get install ruby-full

 

安裝rubygems ruby擴展必須

wget http://production.cf.rubygems.org/rubygems/rubygems-2.4.1.tgz

tar -zxvf rubygems-2.4.1.tgz
cd rubygems-2.4.1
ruby setup.rb

 

安裝redis 用來處理日誌隊列

wget http://download.redis.io/releases/redis-2.8.13.tar.gz

tar -zxvf redis-2.8.13.tar.gz
cd redis-2.8.13
make
vim redis.conf
設置 "daemonize yes"
啓動：/usr/local/redis-2.8.13/src/redis-server /usr/local/redis-2.8.13/redis.conf 

 

安裝 elasticsearch 全文搜索服務(logstash集成了一個)

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz

tar -zxvf elasticsearch-1.3.2.tar.gz
cd elasticsearch-1.3.2
啓動：
/usr/local/elasticsearch-1.3.2/bin/elasticsearch -d
訪問
http://localhost:9200

 

安裝：logstash 收集、過濾日誌

wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz

tar -zxvf logstash-1.4.2.tar.gz

啓動

nohup /usr/local/logstash-1.4.2/bin/logstash -f /usr/local/logstash-1.4.2/agent.conf &

nohup /usr/local/logstash-1.4.2/bin/logstash -f /usr/local/logstash-1.4.2/indexer.conf &

vim /usr/local/logstash-1.4.2/agent.conf

input {
  file {
    path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog", "/var/log/denyhosts", "/var/log/dmesg", "/var/log/faillog", "/var/log/aptitude" ]
    start_position => beginning
  }
  file {
    type => "nginx-access"
    path => "/var/log/nginx/access.log"
  }
}

output {
  redis{
    host =>"192.168.124.128"
    data_type => "list"
    key => "logstash"
  }
}

 vim /usr/local/logstash-1.4.2/indexer.conf

input {
  redis {
    host => "192.168.124.128"
    data_type => "list"
    key => "logstash"
  }
}

output {
  elasticsearch {
    host => "192.168.124.132" #指定elasticsearch服務位置
  }
}

 

安裝Kibana

wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz

tar -zxvf kibana-3.1.0.tar.gz
vim /usr/local/kibana-3.1.0/config/ 能夠經過whereis kibana找到kibana具體物理地址 在config目錄下有個kibana.yml配置文件

搜索"elasticsearch"參數，並對其進行修改以適應您的環境：

elasticsearch: "http://192.168.124.132:9200",

您還能夠修改default_route參數，默認打開logstash儀表板而不是Kibana歡迎頁面：

default_route     : '/dashboard/file/logstash.json', （這個在哪裏配置，沒看到...）

下載配置模板

wget https://raw.github.com/elasticsearch/kibana/master/sample/nginx.conf

修改Nginx配置

vim /usr/local/nginx/conf/nginx.conf

增長Server節點

    #
    # Nginx proxy for Elasticsearch + Kibana
    #
    # In this setup, we are password protecting the saving of dashboards. You may
    # wish to extend the password protection to all paths.
    #
    # Even though these paths are being called as the result of an ajax request, the
    # browser will prompt for a username/password on the first request
    #
    # If you use this, you'll want to point config.js at http://FQDN:80/ instead of
    # http://FQDN:9200
    #
    server {
      listen                *:80 ;

      server_name           localhost;
      access_log            /usr/local/nginx/logs/kibana.access.log;

      location / {
        root  /usr/local/kibana-3.1.0;
        index  index.html  index.htm;
      }

      location ~ ^/_aliases$ {
        proxy_pass http://127.0.0.1:9200;
        proxy_read_timeout 90;
      }
      location ~ ^/.*/_aliases$ {
        proxy_pass http://127.0.0.1:9200;
        proxy_read_timeout 90;
      }
      location ~ ^/_nodes$ {
        proxy_pass http://127.0.0.1:9200;
        proxy_read_timeout 90;
      }
      location ~ ^/.*/_search$ {
        proxy_pass http://127.0.0.1:9200;
        proxy_read_timeout 90;
      }
      location ~ ^/.*/_mapping {
        proxy_pass http://127.0.0.1:9200;
        proxy_read_timeout 90;
      }

      # Password protected end points
      location ~ ^/kibana-int/dashboard/.*$ {
        proxy_pass http://127.0.0.1:9200;
        proxy_read_timeout 90;
        limit_except GET {
          proxy_pass http://127.0.0.1:9200;
          auth_basic "Restricted";
          auth_basic_user_file /usr/local/nginx/kibana.myhost.org.htpasswd;
        }
      }
      location ~ ^/kibana-int/temp.*$ {
        proxy_pass http://127.0.0.1:9200;
        proxy_read_timeout 90;
        limit_except GET {
          proxy_pass http://127.0.0.1:9200;
          auth_basic "Restricted";
          auth_basic_user_file /usr/local/nginx/kibana.myhost.org.htpasswd;
        }
      }
    }

 

若是有防火牆須要放開這些端口：

• port 80 (for the web interface)
• port 5544 (to receive remote syslog messages)
• port 6379 (for the redis broker)
• port 9200 (so the web interface can access elasticsearch)