Elastic Search + Logstash + Kibana 初學者日誌
由於我用的是windows操做系統，因此下面的介紹都基於windows環境。功能方面應該和Linux是同樣的。
=======================================================================================
Getting Started with ELK for Apache Logs (這個例子展現了怎樣用logstash把log發送到Elastic Search, 而後用Kibana作圖形化展現）
https://github.com/aagea/elk-example
我修改了上面的例子，用stdin做爲logstash的輸入（輸入了hello world)，用Elastic Search做爲輸出。
那麼爲了驗證，能夠用下面這個命令搜索Elastic Search的內容
固然也能夠直接在Kibana的Discovery界面搜索。具體方法以下。這篇文章寫的很好，尤爲是裏面的圖畫得很清晰，缺點是軟件版本有點老了。
https://my.oschina.net/itblog/blog/547250/
下面介紹一下ELK各個組件安裝後的目錄結構和help命令返回的結果。這個有助於瞭解組件的功能覆蓋和基本使用方法。
Elastic Search 6.6
目錄
bin -- 這裏存放Elastic Search的執行程序. 這裏我用elasticsearch -help /all > rex_output.txt命令獲得全部的可選項
config -- 這裏存放配置文件，主要是一些yml文件。 好比elasticsearch.yml (內容以下)
cluster.name=my-application node.name=node-1 path.data=/path/to/data path.logs=/path/to/logs network.host=127.0.0.1 http.port=9200其他配置文件以下
- log4j2.properties
- role_mapping.yml
- roles.yml
- users
- users_roles
data -- 這裏面好像是跟cluster有關的文件，好比默認的是cluster node-0, 因此這裏面就有一個名字是「0」的文件夾
lib -- 就是elasticsearch依賴的一些JAR包
logs -- 日誌文件
modules -- 這裏面是ElasticSearch用到的主要模塊，包括X-Pack (lets you set up watches (or rules) to detect and alert on changes in your ElasticSearch data) ，也是一些jar包。
plugins -- 插件 sudo bin/elasticsearch-plugin install [plugin_name]
Help命令返回的內容
Option Description
------ -----------
-E <KeyValuePair> Configure a setting
-V, --version Prints elasticsearch version information and exits
-d, --daemonize Starts Elasticsearch in the background
-h, --help show help
-p, --pidfile <Path> Creates a pid file in the specified path on start
-q, --quiet Turns off standard output/error streams logging in console
-s, --silent show minimal output
-v, --verbose show verbose output
------ -----------
-E <KeyValuePair> Configure a setting
-V, --version Prints elasticsearch version information and exits
-d, --daemonize Starts Elasticsearch in the background
-h, --help show help
-p, --pidfile <Path> Creates a pid file in the specified path on start
-q, --quiet Turns off standard output/error streams logging in console
-s, --silent show minimal output
-v, --verbose show verbose output
ElasticSearch的官網上有一些例子（https://github.com/elastic/examples），還沒來得及細看，下一篇文章再細說吧。
Logstash 6.6
目錄 （這裏有詳細介紹 https://www.elastic.co/guide/en/logstash/current/dir-layout.html）
bin -- 可執行程序
config -- 配置文件。能夠參考logstash-sample.conf建立你本身的conf文件
data -- 這裏面的文件有些看不懂。好比一個uuid文件，一個.lock文件，幾個queue文件夾。好像是跟持久化有關的內容。
lib -- 這裏面有bootstrap, pluginmaster, secretstore, systeminstall幾個文件夾。也都是ruby程序。
logstash-core -- 這裏面有一些ruby程序和JAR包，應該是logstash的核心代碼。
logstash-core-plugin-api -- 插件。也是ruby程序。
modules -- 好像是用來存放打包後的ruby module的。ps: Ruby一般用rubygem來打包
tools -- 這裏有一個ingest-converter文件夾，裏面是一個JAR文件。
vendor -- 這裏有bundle和jruby文件夾
x-pack -- 這裏面有不少文件夾。可能和ElasticSearch同樣，做用是lets you set up watches (or rules) to detect and alert on changes in your Logstash data
Help命令返回的內容 (初學者經常使用的命令用粗體表示)
Usage:
bin/logstash [OPTIONS]
Options:
-n, --node.name NAME Specify the name of this logstash instance, if no value is given
it will default to the current hostname.
(default: 這裏指的是主機名)
-f, --path.config 指定配置文件 好比這樣：
bin/logstash [OPTIONS]
Options:
-n, --node.name NAME Specify the name of this logstash instance, if no value is given
it will default to the current hostname.
(default: 這裏指的是主機名)
-f, --path.config 指定配置文件 好比這樣：
config/log4j_to_es.conf
CONFIG_PATH Load the logstash config from a specific file
or directory. If a directory is given, all
files in that directory will be concatenated
in lexicographical order and then parsed as a
single config file. You can also specify
wildcards (globs) and any matched files will
be loaded in the order described above.
-e, --config.string 從命令行指定配置 好比：logstash -e 「input { stdin { } } output { stdout {} }
or directory. If a directory is given, all
files in that directory will be concatenated
in lexicographical order and then parsed as a
single config file. You can also specify
wildcards (globs) and any matched files will
be loaded in the order described above.
-e, --config.string 從命令行指定配置 好比：logstash -e 「input { stdin { } } output { stdout {} }
」CONFIG_STRING Use the given string as the configuration
data. Same syntax as the config file. If no
input is specified, then the following is
used as the default input:
"input { stdin { type => stdin } }"
and if no output is specified, then the
following is used as the default output:
"output { stdout { codec => rubydebug } }"
If you wish to use both defaults, please use
the empty string for the '-e' flag.
(default: nil)
--field-reference-parser MODE Use the given MODE when parsing field
references.
The field reference parser is used to expand
field references in your pipeline configs,
and will be becoming more strict to better
handle illegal and ambbiguous inputs in a
future release of Logstash.
Available MODEs are:
- `LEGACY`: parse with the legacy parser,
which is known to handle ambiguous- and
illegal-syntax in surprising ways;
warnings will not be emitted.
- `COMPAT`: warn once for each distinct
ambiguous- or illegal-syntax input, but
continue to expand field references with
the legacy parser.
- `STRICT`: parse in a strict manner; when
given ambiguous- or illegal-syntax input,
raises a runtime exception that should
be handled by the calling plugin.
The MODE can also be set with
`config.field_reference.parser`
(default: "COMPAT")
--modules MODULES Load Logstash modules.
Modules can be defined using multiple instances
'--modules module1 --modules module2',
or comma-separated syntax
'--modules=module1,module2'
Cannot be used in conjunction with '-e' or '-f'
Use of '--modules' will override modules declared
in the 'logstash.yml' file.
-M, --modules.variable MODULES_VARIABLE Load variables for module template.
Multiple instances of '-M' or
'--modules.variable' are supported.
Ignored if '--modules' flag is not used.
Should be in the format of
'-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"'
as in
'-M "example.var.filter.mutate.fieldname=fieldvalue"'
--setup Load index template into Elasticsearch, and saved searches,
index-pattern, visualizations, and dashboards into Kibana when
running modules.
(default: false)
--cloud.id CLOUD_ID Sets the elasticsearch and kibana host settings for
module connections in Elastic Cloud.
Your Elastic Cloud User interface or the Cloud support
team should provide this.
Add an optional label prefix '<label>:' to help you
identify multiple cloud.ids.
e.g. 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy'
--cloud.auth CLOUD_AUTH Sets the elasticsearch and kibana username and password
for module connections in Elastic Cloud
e.g. 'username:<password>'
--pipeline.id ID Sets the ID of the pipeline.
(default: "main")
-w, --pipeline.workers COUNT Sets the number of pipeline workers to run.
(default: 4)
--java-execution Use Java execution engine.
(default: false)
-b, --pipeline.batch.size SIZE Size of batches the pipeline is to work in.
(default: 125)
-u, --pipeline.batch.delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling
for the next event.
(default: 50)
--pipeline.unsafe_shutdown Force logstash to exit during shutdown even
if there are still inflight events in memory.
By default, logstash will refuse to quit until all
received events have been pushed to the outputs.
(default: false)
--path.data PATH This should point to a writable directory. Logstash
will use this directory whenever it needs to store
data. Plugins will also have access to this path.
(default: "C:/Users/IBM_ADMIN/Downloads/logstash-6.6.0/data")
-p, --path.plugins PATH A path of where to find plugins. This flag
can be given multiple times to include
multiple paths. Plugins are expected to be
in a specific directory hierarchy:
'PATH/logstash/TYPE/NAME.rb' where TYPE is
'inputs' 'filters', 'outputs' or 'codecs'
and NAME is the name of the plugin.
(default: [])
-l, --path.logs PATH Write logstash internal logs to the given
file. Without this flag, logstash will emit
logs to standard output.
(default: "C:/Users/IBM_ADMIN/Downloads/logstash-6.6.0/logs")
--log.level LEVEL Set the log level for logstash. Possible values are:
- fatal
- error
- warn
- info
- debug
- trace
(default: "info")
--config.debug Print the compiled config ruby code out as a debug log (you must also have --log.level=debug enabled).
WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result
in plaintext passwords appearing in your logs!
(default: false)
-i, --interactive SHELL Drop to shell instead of running as normal.
Valid shells are "irb" and "pry"
-V, --version Emit the version of logstash and its friends,
then exit.
-t, --config.test_and_exit Check configuration for valid syntax and then exit.
(default: false)
-r, --config.reload.automatic Monitor configuration changes and reload
whenever it is changed.
NOTE: use SIGHUP to manually reload the config
(default: false)
--config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location
for changes, in seconds.
(default: 3000000000)
--http.host HTTP_HOST Web API binding host (default: "127.0.0.1")
--http.port HTTP_PORT Web API http port (default: 9600..9700)
--log.format FORMAT Specify if Logstash should write its own logs in JSON form (one
event per line) or in plain text (using Ruby's Object#inspect)
(default: "plain")
--path.settings SETTINGS_DIR Directory containing logstash.yml file. This can also be
set through the LS_SETTINGS_DIR environment variable.
(default: config目錄的地址)
--verbose Set the log level to info.
DEPRECATED: use --log.level=info instead.
--debug Set the log level to debug.
DEPRECATED: use --log.level=debug instead.
--quiet Set the log level to info.
DEPRECATED: use --log.level=info instead.
-h, --help print help
data. Same syntax as the config file. If no
input is specified, then the following is
used as the default input:
"input { stdin { type => stdin } }"
and if no output is specified, then the
following is used as the default output:
"output { stdout { codec => rubydebug } }"
If you wish to use both defaults, please use
the empty string for the '-e' flag.
(default: nil)
--field-reference-parser MODE Use the given MODE when parsing field
references.
The field reference parser is used to expand
field references in your pipeline configs,
and will be becoming more strict to better
handle illegal and ambbiguous inputs in a
future release of Logstash.
Available MODEs are:
- `LEGACY`: parse with the legacy parser,
which is known to handle ambiguous- and
illegal-syntax in surprising ways;
warnings will not be emitted.
- `COMPAT`: warn once for each distinct
ambiguous- or illegal-syntax input, but
continue to expand field references with
the legacy parser.
- `STRICT`: parse in a strict manner; when
given ambiguous- or illegal-syntax input,
raises a runtime exception that should
be handled by the calling plugin.
The MODE can also be set with
`config.field_reference.parser`
(default: "COMPAT")
--modules MODULES Load Logstash modules.
Modules can be defined using multiple instances
'--modules module1 --modules module2',
or comma-separated syntax
'--modules=module1,module2'
Cannot be used in conjunction with '-e' or '-f'
Use of '--modules' will override modules declared
in the 'logstash.yml' file.
-M, --modules.variable MODULES_VARIABLE Load variables for module template.
Multiple instances of '-M' or
'--modules.variable' are supported.
Ignored if '--modules' flag is not used.
Should be in the format of
'-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"'
as in
'-M "example.var.filter.mutate.fieldname=fieldvalue"'
--setup Load index template into Elasticsearch, and saved searches,
index-pattern, visualizations, and dashboards into Kibana when
running modules.
(default: false)
--cloud.id CLOUD_ID Sets the elasticsearch and kibana host settings for
module connections in Elastic Cloud.
Your Elastic Cloud User interface or the Cloud support
team should provide this.
Add an optional label prefix '<label>:' to help you
identify multiple cloud.ids.
e.g. 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy'
--cloud.auth CLOUD_AUTH Sets the elasticsearch and kibana username and password
for module connections in Elastic Cloud
e.g. 'username:<password>'
--pipeline.id ID Sets the ID of the pipeline.
(default: "main")
-w, --pipeline.workers COUNT Sets the number of pipeline workers to run.
(default: 4)
--java-execution Use Java execution engine.
(default: false)
-b, --pipeline.batch.size SIZE Size of batches the pipeline is to work in.
(default: 125)
-u, --pipeline.batch.delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling
for the next event.
(default: 50)
--pipeline.unsafe_shutdown Force logstash to exit during shutdown even
if there are still inflight events in memory.
By default, logstash will refuse to quit until all
received events have been pushed to the outputs.
(default: false)
--path.data PATH This should point to a writable directory. Logstash
will use this directory whenever it needs to store
data. Plugins will also have access to this path.
(default: "C:/Users/IBM_ADMIN/Downloads/logstash-6.6.0/data")
-p, --path.plugins PATH A path of where to find plugins. This flag
can be given multiple times to include
multiple paths. Plugins are expected to be
in a specific directory hierarchy:
'PATH/logstash/TYPE/NAME.rb' where TYPE is
'inputs' 'filters', 'outputs' or 'codecs'
and NAME is the name of the plugin.
(default: [])
-l, --path.logs PATH Write logstash internal logs to the given
file. Without this flag, logstash will emit
logs to standard output.
(default: "C:/Users/IBM_ADMIN/Downloads/logstash-6.6.0/logs")
--log.level LEVEL Set the log level for logstash. Possible values are:
- fatal
- error
- warn
- info
- debug
- trace
(default: "info")
--config.debug Print the compiled config ruby code out as a debug log (you must also have --log.level=debug enabled).
WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result
in plaintext passwords appearing in your logs!
(default: false)
-i, --interactive SHELL Drop to shell instead of running as normal.
Valid shells are "irb" and "pry"
-V, --version Emit the version of logstash and its friends,
then exit.
-t, --config.test_and_exit Check configuration for valid syntax and then exit.
(default: false)
-r, --config.reload.automatic Monitor configuration changes and reload
whenever it is changed.
NOTE: use SIGHUP to manually reload the config
(default: false)
--config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location
for changes, in seconds.
(default: 3000000000)
--http.host HTTP_HOST Web API binding host (default: "127.0.0.1")
--http.port HTTP_PORT Web API http port (default: 9600..9700)
--log.format FORMAT Specify if Logstash should write its own logs in JSON form (one
event per line) or in plain text (using Ruby's Object#inspect)
(default: "plain")
--path.settings SETTINGS_DIR Directory containing logstash.yml file. This can also be
set through the LS_SETTINGS_DIR environment variable.
(default: config目錄的地址)
--verbose Set the log level to info.
DEPRECATED: use --log.level=info instead.
--debug Set the log level to debug.
DEPRECATED: use --log.level=debug instead.
--quiet Set the log level to info.
DEPRECATED: use --log.level=info instead.
-h, --help print help
以我初學者的眼光看，Logstash是ELK裏面須要配置最多的組件。經過配置, Logstash能夠蒐集不一樣類型的日誌供咱們分析。下面是網上的一些例子：
Logstash收集RESTful API日誌
Logstash收集log4j日誌
Kibana 6.6
目錄
bin -- 這個目錄存放Kibana主程序
config -- 這個目錄存放kibana.yml配置文件
data -- 好像是存放主程序生成的數據？
dlls (這應該只有windows纔有) -- windows須要的動態連接庫文件
node -- 這裏面存放一個node.exe程序
node_modules -- 這裏存放node插件
optimize -- 這裏存放了一些JS和CSS文件，多是Kibana界面所使用的
plugins -- 插件
src -- 這裏存放的是Kibana的源程序，好比core文件夾，裏面的README描述是這樣的Core is a set of systems (frontend, backend etc.) that Kibana and its plugins are built on top of. 經過修改源程序能夠對Kibana進行定製。
target -- 這裏面好像是一些.ts.map文件，應該是src編譯以後產生的
webpackShims -- 這裏面是一些JS文件，好像是一些common module, 好比ui-bootstrap.js, angular.js, elasticsearch.js
Help命令返回的內容
Usage: bin/kibana [command=serve] [options]
Kibana is an open source (Apache Licensed), browser based analytics and search dashboard for Elasticsearch.
Commands:
serve [options] Run the kibana server
help <command> Get the help for a specific command
"serve" Options:
-h, --help output usage information
-e, --elasticsearch <uri1,uri2> Elasticsearch instances
-c, --config <path> Path to the config file, can be changed with the CONFIG_PATH environment variable as well. Use multiple --config args to include multiple config files.
-p, --port <port> The port to bind to
-q, --quiet Prevent all logging except errors
-Q, --silent Prevent all logging
--verbose Turns on verbose logging
-H, --host <host> The host to bind to
-l, --log-file <path> The file to log to
--plugin-dir <path> A path to scan for plugins, this can be specified multiple times to specify multiple directories
--plugin-path <path> A path to a plugin which should be included by the server, this can be specified multiple times to specify multiple paths
--plugins <path> an alias for --plugin-dir
--optimize Optimize and then stop the server
Kibana is an open source (Apache Licensed), browser based analytics and search dashboard for Elasticsearch.
Commands:
serve [options] Run the kibana server
help <command> Get the help for a specific command
"serve" Options:
-h, --help output usage information
-e, --elasticsearch <uri1,uri2> Elasticsearch instances
-c, --config <path> Path to the config file, can be changed with the CONFIG_PATH environment variable as well. Use multiple --config args to include multiple config files.
-p, --port <port> The port to bind to
-q, --quiet Prevent all logging except errors
-Q, --silent Prevent all logging
--verbose Turns on verbose logging
-H, --host <host> The host to bind to
-l, --log-file <path> The file to log to
--plugin-dir <path> A path to scan for plugins, this can be specified multiple times to specify multiple directories
--plugin-path <path> A path to a plugin which should be included by the server, this can be specified multiple times to specify multiple paths
--plugins <path> an alias for --plugin-dir
--optimize Optimize and then stop the server
在實際使用中，咱們可能須要自定義Kibana界面。 我在網上找到兩篇文章，先收藏着吧。
Kibana界面風格重製
Kibana國際化
https://www.jianshu.com/p/57e0475fda90
相關文章
- 1. 初試 Elastic Search
- 2. logstash+elasticsearch+kibana日誌收集
- 3. LOGSTASH+ELASTICSEARCH+KIBANA處理OSS訪問日誌
- 4. Linux 安裝elastic search 和kibana
- 5. Elasticsearch+Kibana+Logstash 搭建日誌平臺
- 6. elastic search
- 7. logstash+elasticsearch+kibana管理日誌（安裝）
- 8. Elastic Search 初探
- 9. 【Elastic Stack】日誌收集系統
- 10. Elastic Search 安裝與運行
- 更多相關文章...
- • Docker search 命令 - Docker命令大全
- • SQLite 日期 & 時間 - SQLite教程
- • 適用於PHP初學者的學習線路和建議
- • Tomcat學習筆記（史上最全tomcat學習筆記）
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
最新文章
歡迎關注本站公眾號,獲取更多信息
相關文章
- 1. 初試 Elastic Search
- 2. logstash+elasticsearch+kibana日誌收集
- 3. LOGSTASH+ELASTICSEARCH+KIBANA處理OSS訪問日誌
- 4. Linux 安裝elastic search 和kibana
- 5. Elasticsearch+Kibana+Logstash 搭建日誌平臺
- 6. elastic search
- 7. logstash+elasticsearch+kibana管理日誌（安裝）
- 8. Elastic Search 初探
- 9. 【Elastic Stack】日誌收集系統
- 10. Elastic Search 安裝與運行