1、CentOS7.4下Elastic Stack 6.2.4日誌系統搭建
Elasticsearch是一個高度可擴展的開源全文搜索和分析引擎。它容許您快速，近實時地存儲，搜索和分析大量數據。它一般用做支持具備複雜搜索功能和需求的應用程序的底層引擎/技術。
Logstash是一個開源的用於收集,分析和存儲日誌的工具。
Kibana 也是一個開源和免費的工具，Kibana能夠爲 Logstash 和 ElasticSearch 提供的日誌分析友好的 Web 界面，能夠彙總、分析和搜索重要數據日誌。
Beats是elasticsearch公司開源的一款採集系統監控數據的代理agent，是在被監控服務器上以客戶端形式運行的數據收集器的統稱，能夠直接把數據發送給Elasticsearch或者經過Logstash發送給Elasticsearch，而後進行後續的數據分析活動。Beats由以下組成:
1.Packetbeat：是一個網絡數據包分析器，用於監控、收集網絡流量信息，
Packetbeat嗅探服務器之間的流量，解析應用層協議，並關聯到消息的處理， 其支 持ICMP (v4 and v6)、DNS、HTTP、Mysql、PostgreSQL、Redis、
MongoDB、Memcache等協議；
2. Filebeat：用於監控、收集服務器日誌文件，其已取代 logstash forwarder；
3. Metricbeat：可按期獲取外部系統的監控指標信息，其能夠監控、收集
Apache、HAProxy、MongoDB、MySQL、Nginx、PostgreSQL、
Redis、System、Zookeeper等服務；
4. Winlogbeat：用於監控、收集Windows系統的日誌信息；
5. Create your own Beat：自定義beat ，若是上面的指標不能知足需求，elasticsarch鼓勵開發者 使用go語言，擴展實現自定義的beats，只須要按照模板，實現監控的輸入，日誌，輸出等便可。
Beats 將蒐集到的數據發送到 Logstash，經 Logstash 解析、過濾後，將其發送到 Elasticsearch 存儲，並由 Kibana 呈現給用戶。
Beats 做爲日誌蒐集器沒有Logstash 做爲日誌蒐集器消耗資源，解決了 Logstash 在各服務器節點上佔用系統資源高的問題。
1、環境
# dmidecode|grep "System Information" -A9|egrep "Manufacturer|Product"
Manufacturer: Dell Inc.
Product Name: PowerEdge R630
# uname -a
Linux linux-node2 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
關閉firewalld,selinux
2、下載Elasticsearch軟件包
使用最新Elasticsearch6.2.4包，須要先安裝Java version
1.8.0_131以上版本：
本次基於Centos7 使用RPM安裝
軟件包安裝以下：
2.1使用ZIP包
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.zip wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.zip.sha512 shasum -a 512 -c elasticsearch-6.2.4.zip.sha512 unzip elasticsearch-6.2.4.zip cd elasticsearch-6.2.4/
2.2使用TAR包
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz.sha512 shasum -a 512 -c elasticsearch-6.2.4.tar.gz.sha512 tar -xzf elasticsearch-6.2.4.tar.gz cd elasticsearch-6.2.4/
使用包安裝報錯：
# ./elasticsearch
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
##########/etc/elasticsearch/jvm.options添加參數
[2018-05-23T15:08:06,797][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
▽ at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
... 6 more
2.3使用YUM安裝
#rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch 導入PGP KEY
#vim
/etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
#yum install elasticsearch
[root@linux-node1 yum.repos.d]# yum install elasticsearch
Loaded plugins: fastestmirror
elasticsearch-6.x | 1.3 kB 00:00:00
elasticsearch-6.x/primary | 67 kB 00:00:05
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
elasticsearch-6.x 180/180
Resolving Dependencies
--> Running transaction check
---> Package elasticsearch.noarch 0:6.2.4-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================================================
Installing:
elasticsearch noarch 6.2.4-1 elasticsearch-6.x 28 M
Transaction Summary
====================================================================================================================================================================
Install 1 Package
Total download size: 28 M
Installed size: 31 M
Is this ok [y/d/N]: y
Downloading packages:
elasticsearch-6.2.4.rpm | 28 MB 00:02:10
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Installing : elasticsearch-6.2.4-1.noarch 1/1
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
sudo systemctl start elasticsearch.service
Verifying : elasticsearch-6.2.4-1.noarch 1/1
Installed:
elasticsearch.noarch 0:6.2.4-1
Complete!
elasticsearch 安裝完成後
# systemctl start elasticsearch 默認不記錄LOG須要進行設置
取消/usr/lib/systemd/system/elasticsearch.service文件中--quiet
ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet
# systemctl daemon-reload
# systemctl restart elasticsearch
@@@@@@@@@@@
# journalctl --unit elasticsearch
-- Logs begin at Wed 2018-05-23 14:32:54 CST, end at Wed 2018-05-23 15:53:11 CST. --
May 23 15:34:02 linux-node1 systemd[1]: Started Elasticsearch.
May 23 15:34:02 linux-node1 systemd[1]: Starting Elasticsearch...
May 23 15:34:04 linux-node1 elasticsearch[11511]: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should co
...skipping...
#journalctl --unit elasticsearch --since "2016-10-30 18:17:16"可查看指定時間後的LOG
@@@@@@@@@@@
# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2018-05-23 15:34:02 CST; 11s ago
Docs: http://www.elastic.co
Main PID: 11511 (java)
Tasks: 14
Memory: 1.1G
CGroup: /system.slice/elasticsearch.service
└─11511 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -...
May 23 15:34:02 linux-node1 systemd[1]: Started Elasticsearch.
May 23 15:34:02 linux-node1 systemd[1]: Starting Elasticsearch...
May 23 15:34:04 linux-node1 elasticsearch[11511]: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then y...Threads=N
Hint: Some lines were ellipsized, use -l to show in full.
# ss -tlnp |grep -E '9200|9300'
###ElasticSearch默認的對外服務的HTTP端口是9200，節點間交互的TCP端口是9300。
LISTEN 0 128 ::ffff:127.0.0.1:9200 :::* users:(("java",pid=11511,fd=121))
LISTEN 0 128 ::1:9200 :::* users:(("java",pid=11511,fd=120))
LISTEN 0 128 ::ffff:127.0.0.1:9300 :::* users:(("java",pid=11511,fd=113))
LISTEN 0 128 ::1:9300 :::* users:(("java",pid=11511,fd=111))
# /usr/share/elasticsearch/bin/elasticsearch -V
#ln -s /usr/share/elasticsearch/bin/elasticsearch /bin/elasticsearch
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Version: 6.2.4, Build: ccec39f/2018-04-12T20:37:28.497551Z, JVM: 1.8.0_171
#curl -X GET http://localhost:9200
{
"name" : "IGgk_NL",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "8u-EnhG8QsatgT3r6BDWrA",
"version" : {
"number" : "6.2.4",
"build_hash" : "ccec39f",
"build_date" : "2018-04-12T20:37:28.497551Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
3、配置elasticsearch
YUM安裝默認配置目錄/etc/elasticsearch
# ls -lh
total 16K
-rw-rw----. 1 root elasticsearch 2.9K Apr 13 04:39 elasticsearch.yml
-rw-rw----. 1 root elasticsearch 2.8K Apr 13 04:39 jvm.options
-rw-rw----. 1 root elasticsearch 5.0K Apr 13 04:39 log4j2.properties
# chown -R elasticsearch:elasticsearch /etc/elasticsearch
ZIP和TAR安裝配置目錄在$ES_HOME/config目錄下 （ES_HOME爲解壓的目錄位置）
能夠修改成：
ES_PATH_CONF=/path/to/my/config ./bin/elasticsearch
對於包分發版，配置目錄位置默認爲/etc/elasticsearch。配置目錄的位置也能夠經過ES_PATH_CONF環境變量進行更改，可是請注意，在外殼中設置這個位置是不夠的。相反，這個變量是從/etc/default/elasticsearch（用於Debian軟件包）和/etc/sysconfig/elasticsearch（用於RPM包）的。您將須要在其中一個文件中編輯espathconf=/etc/elasticsearch條目，以更改配置目錄位置。
config/elasticsearch.ymal中配置項說明：
- cluster_name 集羣名稱，默認爲elasticsearch，這裏咱們設置爲es5.2.1Cluster
- node.name配置節點名，用來區分節點
- network.host 是配置能夠訪問本節點的路由地址
- http.port 路由地址端口
- transport.tcp.port TCP協議轉發地址端口
- node.master 是否做爲集羣的主結點 ，值爲true或true
- node.data 是否存儲數據，值爲true或true
- discovery.zen.ping.unicast.hosts 用來配置全部用來組建集羣的機器的IP地址，因爲新版本是不支持多播的，所以這個值須要提早設定好，當集羣須要擴展的時候，該值都要作改變，增長新機器的IP地址，若是是在一個ip上，要把TCP協議轉發端口寫上
- discovery.zen.minimum_master_nodes 用來配置主節點數量的最少值，若是主節點數量低於該值，閉包範圍內的集羣將會中止服務,之因此加粗體，是由於暫時還沒有認證，下面配置爲1方便集羣更容易造成，即便只有一個主節點，也能夠構建集羣
- gateway.* 網關的相關配置
- script.* indices.* 根據需求添加的配置（可選）
# 配置文件中給出了三種配置高性能集羣拓撲結構的模式,以下： # 1. 若是你想讓節點從不選舉爲主節點,只用來存儲數據,可做爲負載器 # node.master: false # node.data: true # 2. 若是想讓節點成爲主節點,且不存儲任何數據,並保有空閒資源,可做爲協調器 # node.master: true # node.data: false # 3. 若是想讓節點既不稱爲主節點,又不成爲數據節點,那麼可將他做爲搜索器,從節點中獲取數據,生成搜索結果等 # node.master: false # node.data: false
配置elasticsearch.yml
path: data: /var/lib/elasticsearch logs: /var/log/elasticsearch
或
path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch
node.name: ${HOSTNAME} network.host: ${ES_NETWORK_HOST}
4、重要配置參數參考：
相關文章
- 1. ELK stack 日誌系統搭建
- 2. 【Elastic Stack】日誌收集系統
- 3. 快速搭建日誌系統——ELK STACK
- 4. Elastic Stack7.2（ELK）日誌系統簡介 及 搭建
- 5. elk日誌系統搭建
- 6. ELK（elasticsearch stack）日誌搭建
- 7. 使用ElasticSearch搭建日誌系統
- 8. Linux 環境 離線 ELK Stack搭建分佈式日誌系統搭建寶典
- 9. Docker搭建Elastic Stack日誌分析平臺
- 10. Elastic Stack 日誌分析平臺搭建筆記
- 更多相關文章...
- • Rust 環境搭建 - RUST 教程
- • 操作系統（OS）平臺 統計 - 瀏覽器信息
- • Docker容器實戰(七) - 容器眼光下的文件系統
- • 適用於PHP初學者的學習線路和建議
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
- 1. eclipse設置粘貼字符串自動轉義
- 2. android客戶端學習-啓動模擬器異常Emulator: failed to initialize HAX: Invalid argument
- 3. android.view.InflateException: class com.jpardogo.listbuddies.lib.views.ListBuddiesLayout問題
- 4. MYSQL8.0數據庫恢復 MYSQL8.0ibd數據恢復 MYSQL8.0恢復數據庫
- 5. 你本是一個肉體，是什麼驅使你前行【1】
- 6. 2018.04.30
- 7. 2018.04.30
- 8. 你本是一個肉體，是什麼驅使你前行【3】
- 9. 你本是一個肉體，是什麼驅使你前行【2】
- 10. 【資訊】LocalBitcoins達到每週交易比特幣的7年低點
歡迎關注本站公眾號,獲取更多信息
