遷移到:http://www.bdata-cap.com/newsinfo/1712695.html
本文介紹安裝 logstash 2.2.0 和 elasticsearch 2.2.0,操做系統環境版本是 CentOS/Linux 2.6.32-504.23.4.el6.x86_64。css
安裝 JDK 是必須的,通常操做系統都會有,只是版本的問題,後面會提到。html
而 Kibana 只是一個用純 JavaScript 寫的前端 UI。必定要注意 Kibana 的版本,它會要求 ES 的版本。好比 Kibana 3 要求 Elasticsearch 至少 0.9.9 或更高;Kibana 4.5 要求 ES 至少 2.3.0 或更高。前端
假設,ELK 都位於 10.1.8.166 機器上。java
Elasticsearch主頁node
Elasticsearch(簡稱,ES)提供 ZIP、TAR、DEB 和 RPM 包。但 Github 上提供了一個針對中文環境的 Elasticsearch-RTF,RTF 即 Ready To Fly,它是一個繼承了基本插件(如服務封裝、中文分詞、mapper-attachments、transport-thrift、tools.carrot2 等)的並帶有示例程序的可直接上手的簡易工程版本,換句話說,幫你入門的。本文針對 Elasticsearch-RTF 爲例。基本上,elasticsearch 解壓後就能使用。python
假設你已經從 Github 上下載 elasticsearch-rtf,名爲 elasticsearch-master.zip,並上傳到你的 Linux 服務器 /usr/local/elasticsearch目錄(若是沒有,就用 mkdir 命令建立一個)。ios
[root@vcyber local]# cd /usr/local/elasticsearch
[root@vcyber local]# unzip elasticsearch-master.zip
[root@vcyber elasticsearch]# ls
elasticsearch-master elasticsearch-master.zip
[root@vcyber local]# mv elasticsearch-master elasticsearch
[root@vcyber elasticsearch]# ls
elasticsearch elasticsearch-master.zip
Linux 環境:git
[root@vcyber elasticsearch]# pwd
/usr/local/elasticsearch/elasticsearch
[root@vcyber elasticsearch]# bin/elasticsearch
Windows 環境,執行相應的 .bat 文件,即 elasticsearch.bat。github
但報錯了:
web
[root@vcyber elasticsearch]# bin/elasticsearch
Exception in thread "main" java.lang.RuntimeException: Java version: Oracle Cooration 1.7.0_51 [Java HotSpot(TM) 64-Bit Server VM 24.51-b03] suffers from crical bug https://bugs.openjdk.java.net/browse/JDK-8024830 which can cause dataorruption.
Please upgrade the JVM, see http://www.elastic.co/guide/en/elasticsearch/referce/current/_installation.html for current recommendations.
If you absolutely cannot upgrade, please add -XX:-UseSuperWord to the JAVA_OPT environment variable.
Upgrading is preferred, this workaround will result in degraded performance.
at org.elasticsearch.bootstrap.JVMCheck.check(JVMCheck.java:123)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:283)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:3
Refer to the log for complete error details.
大意是:Java 運行時異常,本機版本 JDK 有 bug……讓升級 JVM。若是實在不能升級,就向 JAVA_OPT 環境變量添加 -XX:-UseSuperWord 選項。
因而,看一下本機的Java 版本:
[root@vcyber elasticsearch]# java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
[root@vcyber elasticsearch]# echo $JAVA_HOME
/usr/java/jdk1.7.0_51
[root@vcyber elasticsearch]#
版本是 1.7.0_51。再在官網查了一下,說:「Elasticsearch requires at least Java 7. Specifically as of this writing, it is recommended that you use the Oracle JDK version 1.8.0_72. Java installation varies from platform to platform so we won’t go into those details here. Oracle’s recommended installation documentation can be found on Oracle’s website. Suffice to say, before you install Elasticsearch, please check your Java version first by running (and then install/upgrade accordingly if needed):」,大意是,ES 至少要求 7,推薦使用 1.8.0_72。
[root@vcyber elasticsearch]# yum list installed | grep java
[root@vcyber elasticsearch]# yum list installed | grep jdk
jdk.x86_64 2000:1.7.0_51-fcs installed
[root@vcyber elasticsearch]# yum -y remove jdk.x86_64
……
[root@vcyber elasticsearch]#yum -y install java-1.8.0-openjdk*
……
注意:java-1.8.0-openjdk*」,後面有個星號,即安裝 java 所有相關的東西~
[root@vcyber elasticsearch]# export JAVA_HOME=/usr/lib/jvm/java-1.8.0
[root@vcyber elasticsearch]# export PATH=$JAVA_HOME/bin:$PATH
[root@vcyber elasticsearch]# export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
[root@vcyber elasticsearch]# java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
[root@vcyber elasticsearch]# echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0
[root@vcyber elasticsearch]#
另外,JDK 安裝在了我機器的 /usr/lib/jvm 目錄下,本身確認一下你的路徑。你能夠把環境變量的設置放到 profile 裏,一勞永逸。
[root@vcyber elasticsearch]# bin/elasticsearch
Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:93)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
[root@vcyber elasticsearch]#
[root@vcyber elasticsearch]# groupadd es
[root@vcyber elasticsearch]# useradd -g es es
[root@vcyber elasticsearch]# passwd es
Changing password for user es.
New password:
BAD PASSWORD: it is WAY too short
BAD PASSWORD: is too simple
Retype new password:
passwd: all authentication tokens updated successfully.
[root@vcyber elasticsearch]#
[root@vcyber elasticsearch]# chown -R root .
[root@vcyber elasticsearch]# chown -R es .
[root@vcyber elasticsearch]# chgrp -R es .
[root@vcyber elasticsearch]# ls -l
total 4
drwxr-xr-x 7 es es 4096 Mar 1 03:07 elasticsearch
[root@vcyber elasticsearch]#
ES 2.* 版本已經不容許用 root 用戶運行了,可是 1.* 版本無所謂。
[root@vcyber ~]$ cd /usr/local/elasticsearch/elasticsearch
[root@vcyber elasticsearch]$ bin/elasticsearch
[2016-03-01 05:11:48,413][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2016-03-01 05:11:48,750][INFO ][node ] [Googam] version[2.1.1], pid[15042], build[40e2c53/2015-12-15T13:05:55Z]
[2016-03-01 05:11:48,750][INFO ][node ] [Googam] initializing ...
[2016-03-01 05:11:49,088][INFO ][plugins ] [Googam] loaded [elasticsearch-analysis-ik, elasticsearch-analysis-mmseg, elasticsearch-analysis-stconvert, elasticsearch-analysis-pinyin], sites []
[2016-03-01 05:11:49,121][INFO ][env ] [Googam] using [1] data paths, mounts [[/ (/dev/mapper/vg_vcyber-lv_root)]], net usable_space [26.1gb], net total_space [34.8gb], spins? [possibly], types [ext4]
[2016-03-01 05:11:51,119][INFO ][mmseg-analyzer ] [Dict Loading] chars loaded time=42ms, line=12638, on file=chars.dic
……
[root@vcyber elasticsearch]# curl -X GET http://localhost:9200
{
"name" : "Captain Savage",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.1.1",
"build_hash" : "40e2c53a6b6c2972b3d13846e450e66f4375bd71",
"build_timestamp" : "2015-12-15T13:05:55Z",
"build_snapshot" : false,
"lucene_version" : "5.3.1"
},
"tagline" : "You Know, for Search"
}
[root@vcyber elasticsearch]#
ES 已經安裝成功。
但此時,ES 不能經過IP訪問,最好修改 config/elasticsearch.yml,配置一下 ES。
首先,找到「network.host」行,添加一行:
network.host: your id address
就能經過IP,或瀏覽器訪問。
再找到「http.port」行,添加一行:
http.port: 9200
不然,ES 每次啓動時,端口可能會變(端口被佔用,ES 本身會改端口)~
若是還不能正常啓動 ES,並提示端口被佔用,就查看一下什麼程序佔用 9200 端口,kill 掉,重啓 ES 就行。
我遇到的,Java 把 9200 端口占用了。
Head 是一個用來監控 ES 狀態的客戶端插件,能夠爲初學用戶提供不少便利,例如,使用 Head 提供的 HTTP 客戶端,經過 HTTP 方式來操做 ES。
先查看你的 ES 都有哪些插件:
[root@vcyber ~]# cd /usr/local/elasticsearch/elasticsearch
[root@cyber elasticsearch]# bin/plugin list
Installed plugins in /usr/local/elasticsearch/elasticsearch/plugins:
- elasticsearch-analysis-mmseg-1.7.0
- elasticsearch-analysis-stconvert-1.6.1
- elasticsearch-analysis-pinyin-1.5.2
- elasticsearch-analysis-ik-1.7.0
[root@vcyber elasticsearch]#
大部分是關於中文分詞的,沒有 Head 插件。
ES 支持在線和本地安裝 Head。本地安裝時,從 Github 上下載 Head 插件,而後上傳到你的 ES 服務器,好比,Elasticsearch/plugins 目錄。
下面是在線安裝:
[root@vcyber ~]# cd /usr/local/elasticsearch/elasticsearch/
[root@vcyber elasticsearch]# bin/plugin install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
Downloading ...................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed head into /usr/local/elasticsearch/elasticsearch/plugins/head
若是命令使用的是「mobz/elasticsearch-head」,那麼 ES 將本身聯網從 Github 下載再安裝。
可是報錯了,說校驗和有問題。加上「-v」選項,此次換本地安裝,並且是 zip 壓縮包(從 Github 上下載的),即「file:plugins/elasticsearch-head-master.zip」,再執行一下:
[root@vcyber elasticsearch]# bin/plugin install -v file:plugins/elasticsearch-head-master.zip
-> Installing from file:plugins/elasticsearch-head-master.zip...
Trying file:plugins/elasticsearch-head-master.zip ...
Downloading .........DONE
Verifying file:plugins/elasticsearch-head-master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
- Plugin information:
Name: head
Description: head - A web front end for an elastic search cluster
Site: true
Version: master
JVM: false
Installed head into /usr/local/elasticsearch/elasticsearch/plugins/head
注意:install 選項,ES 的 2.* 版本,都不帶「-」橫線選項,即「-install」。
注意:如果本地安裝,並且,你把 Head 壓縮包放到了 Elasticsearch/plugins 目錄下,安裝後,必定要將 Elasticsearch-head zip 壓縮包刪掉,不然啓動 ES 時會報「不能初始化插件」錯誤。
[2016-03-02 07:06:16,547][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2016-03-02 07:06:16,866][INFO ][node ] [MODAM] version[2.1.1], pid[19446], build[40e2c53/2015-12-15T13:05:55Z][2016-03-02 07:06:16,866][INFO ][node ] [MODAM] initializing...Exception in thread "main" java.lang.IllegalStateException: Unable to initialize pluginsLikely root cause: java.nio.file.FileSystemException: /usr/local/elasticsearch/elasticsearch/plugins/elasticsearch-head-master.zip/plugin-descriptor.properties: Not a directory
以後用瀏覽器訪問 http://your ip adress:9200/_plugin/head,你就會看到以下界面:
myfirestidx 是我以前經過命令行本身建的。
Elasticsearch Head 就成功安裝了。
[root@vcyber elasticsearch]# bin/plugin install -v elasticsearch/marvel/latest
-> Installing elasticsearch/marvel/latest...
Trying http://download.elasticsearch.org/elasticsearch/marvel/marvel-latest.zip...
Downloading ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Installed elasticsearch/marvel/latest into /usr/local/elasticsearch/elasticsearch-1.7.1/plugins/marvel
[root@vcyber elasticsearch-1.7.1]#
我去啊~此插件收費~
[root@vcyber elasticsearch]# bin/plugin install -v lmenezes/elasticsearch-kopf
-> Installing lmenezes/elasticsearch-kopf...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip...
Downloading ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Installed lmenezes/elasticsearch-kopf into /usr/local/elasticsearch/elasticsearch-1.7.1/plugins/kopf
[root@vcyber elasticsearch]#
雖然安裝成功了,但此版本 kopf 不支持個人 ES 版本。
ES 插件地址。
logstash 提供各類安裝包,包括 tar.gz,ZIP,DEB 和 RPM。另外,又提供了一個包含全部插件的壓縮包——logstash-all-plugins-2.2.0.tar.gz 。本文以它爲例。解壓後,配置 logstash,運行便可。
[root@vcyber local]# cd /usr/local
[root@vcyber local]# pwd
/usr/local
[root@vcyber local]# mkdir logstash
[root@vcyber local]#
[root@vcyber local]# cd logstash
[root@vcyber logstash]# ls
logstash-all-plugins-2.2.0.tar.gz
[root@vcyber logstash]# tar zxf logstash-all-plugins-2.2.0.tar.gz
logstash-2.2.0 logstash-all-plugins-2.2.0.tar.gz
[root@vcyber logstash]# cd logstash-2.2.0
[root@vcyber logstash-2.2.0]# vi logstash.conf
[root@vcyber logstash-2.2.0]#
文件內容以下:
input{
stdin{}
}
output{
stdout{}
}
這個配置是最簡單,輸入是終端命令行,輸出也是終端命令行。
[root@vcyber logstash-2.2.0]# bin/logstash agent -f logstash.conf
hello world
hello world 2
Settings: Default pipeline workers: 2
Logstash startup completed
2016-02-28T22:37:06.130Z vcyber hello world
2016-02-28T22:37:06.132Z vcyber hello world 2
logst 已經成功安裝。其中,agent 表示運行Agent模式,-f 表示指定配置文件,-p 表示端口,命令行參數能夠參考logstash command-lines flags。
[root@vcyber logstash-2.2.0]# bin/plugin list
logstash-codec-avro
logstash-codec-cef
logstash-codec-cloudfront
logstash-codec-cloudtrail
logstash-codec-collectd
logstash-codec-compress_spooler
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-gzip_lines
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-nmap
logstash-codec-oldlogstashjson
logstash-codec-plain
logstash-codec-rubydebug
logstash-codec-s3plain
logstash-codec-spool
logstash-filter-aggregate
logstash-filter-alter
logstash-filter-anonymize
logstash-filter-checksum
logstash-filter-cidr
logstash-filter-cipher
logstash-filter-clone
logstash-filter-collate
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dns
logstash-filter-drop
logstash-filter-elapsed
logstash-filter-elasticsearch
logstash-filter-environment
logstash-filter-extractnumbers
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-i18n
logstash-filter-json
logstash-filter-json_encode
logstash-filter-kv
logstash-filter-metaevent
logstash-filter-metricize
logstash-filter-metrics
logstash-filter-multiline
logstash-filter-mutate
logstash-filter-oui
logstash-filter-prune
logstash-filter-punct
logstash-filter-range
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-tld
logstash-filter-translate
logstash-filter-unique
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-filter-zeromq
logstash-input-beats
logstash-input-cloudwatch
logstash-input-couchdb_changes
logstash-input-elasticsearch
logstash-input-eventlog
logstash-input-exec
logstash-input-file
logstash-input-fluentd
logstash-input-ganglia
logstash-input-gelf
logstash-input-gemfire
logstash-input-generator
logstash-input-github
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-irc
logstash-input-jdbc
logstash-input-jmx
logstash-input-kafka
logstash-input-log4j
logstash-input-lumberjack
logstash-input-meetup
logstash-input-pipe
logstash-input-puppet_facter
logstash-input-rabbitmq
logstash-input-redis
logstash-input-relp
logstash-input-rss
logstash-input-s3
logstash-input-salesforce
logstash-input-snmptrap
logstash-input-sqlite
logstash-input-sqs
logstash-input-stdin
logstash-input-stomp
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-input-varnishlog
logstash-input-websocket
logstash-input-wmi
logstash-input-xmpp
logstash-input-zenoss
logstash-input-zeromq
logstash-output-boundary
logstash-output-circonus
logstash-output-cloudwatch
logstash-output-csv
logstash-output-datadog
logstash-output-datadog_metrics
logstash-output-elasticsearch
logstash-output-elasticsearch-ec2
logstash-output-elasticsearch_http
logstash-output-elasticsearch_java
logstash-output-email
logstash-output-exec
logstash-output-file
logstash-output-ganglia
logstash-output-gelf
logstash-output-google_bigquery
logstash-output-google_cloud_storage
logstash-output-graphite
logstash-output-graphtastic
logstash-output-hipchat
logstash-output-http
logstash-output-influxdb
logstash-output-irc
logstash-output-juggernaut
logstash-output-kafka
logstash-output-librato
logstash-output-loggly
logstash-output-lumberjack
logstash-output-metriccatcher
logstash-output-mongodb
logstash-output-nagios
logstash-output-nagios_nsca
logstash-output-null
logstash-output-opentsdb
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-redmine
logstash-output-riemann
logstash-output-s3
logstash-output-sns
logstash-output-solr_http
logstash-output-sqs
logstash-output-statsd
logstash-output-stdout
logstash-output-stomp
logstash-output-syslog
logstash-output-tcp
logstash-output-udp
logstash-output-websocket
logstash-output-xmpp
logstash-output-zabbix
logstash-output-zeromq
logstash-patterns-core
[root@vcyber logstash-2.2.0]#
你得搞一個 Kibana 3.0 並上傳你服務器的 /usr/local/Kibana 目錄下,沒有創建一個,而後用 unzip 解壓(github 下載的是 zip 文件)。
[root@vcyber /]# cd /usr/local/kibana/
[root@vcyber kibana]# ls
exampledata kibana-3.0 kibana-4.0 kibana-5.0
[root@vcyber kibana]#
你能夠看到,我有三個版本,4.0 和 5.0 沒有安裝成功(總提示 node.js 報錯)~Kibana 這三個主要版本差距很大~
簡單介紹一下,你就知道,Kibana 的每一個版本差距有多大。Logstash 早期曾經自帶了一個特別簡單的 Logstash-Web,用來查看 ES 數據。後來, Rashid Khan 用 PHP 寫了一個更好的 Web,取名 Kibana,發佈於 2011年12月11日。2012年8月19日,Rashid Khan 用 Ruby 重寫了一遍,叫 Kibana 2,。由於,Logstash 也是用 Ruby 寫的,因此就代替了 Logstash-Web。2014年2月,此人,用 AngularJS 又重寫了一遍,(我靠,真是服了他了~),叫 Kibana 3。2014年4月,Kibana 3 中止開發,全面致力於 Kibana 4。到 2015年初,發佈了用 JRuby 作後端的 beta 版,但3月正式推出時卻使用 node.js 作後端。(我靠靠,服了~都被他用遍了~)
[root@vcyber kibana]# cd kibana-3.0/src
[root@vcyber src]# ls
app config.js css favicon.ico font img index.html vendor
[root@vcyber src]#
elasticsearch: http://10.1.8.166:9200,
我 ES 服務器地址是 10.1.8.166,注意,屁股後面的逗號,絕對是有的。若是該文件配置錯了,當你訪問 Kibana 時,界面什麼都看不到~
接下來,把 Kibana 搞成一個 Web 站點。tomcat、Python、Nginx 都行,我暫時用 Python。只要把 Kibana 的 src 目錄發佈出去就行。
[root@vcyber src]# pwd
/usr/local/kibana/kibana-3.0/src
[root@vcyber src]# python –m SimpleHTTPServer 8000
其中,SimpleHTTPServer 是 Python 模塊名,區分大小寫的;8000 是 Web 站點的端口。
這樣,你就能經過 http://10.1.8.166:8000 訪問 Kibana。
提示,Kibana 不能鏈接 ES。至於緣由,從 ES 1.4 版本開始,它加強了安全性。寫得很清楚,照着作就行(過後才知道)。~但問題是,這些具體東西,在官網沒找到(我是沒找到),並且連 Kibana 3.0 的文檔,都已經沒有了~錯誤信息所說的:「http.cors」 相關配置,在 Elasticsearch.yml 中,也沒給出~事實上,還真是寫這兩個配置。
[root@vcyber src]# cd /usr/local/elasticsearch/
[root@vcyber elasticsearch]# ls
elasticsearch-1.7.1 elasticsearch-2.2.0
[root@vcyber elasticsearch]# cd elasticsearch-1.7.1/config/
[root@vcyber config]#
我有兩個 ES 版本,Kibana 訪問的是 1.7 那個版本。
http.cors.allow-origin: http://10.1.8.166:8000
http.cors.enabled: true
其中,http.cors.allow-origin 一行代表,誰能夠訪問 ES。若讓全部域名均可以訪問,寫成 「*」~
本例寫具體的 IP 地址和端口。
這樣,Kibana 就能夠訪問 ES 了。以下圖所示: