Self Host 使用 Exceptionless 實時監控程序運行日誌服務
Exceptionless 是一個能夠對 ASP.NET Core, ASP.NET MVC，WebAPI, WebForms, WPF, Console 應用提供系統的日誌，錯誤監控、報表等服務實時日誌收集框架 。具體介紹：http://www.cnblogs.com/savorboard/p/exceptionless.html 。html
日誌分析與程序異常在系統開發與運行中佔很是重要的地位，在技術選型的過程當中須要考慮一些問題前端
- 數據分散在多臺服務器中，難以查找
- 數據量大，查詢速度慢
- 涉及多個系統調用，難以快速定位數據問題
開源日誌項目
https://github.com/Graylog2/graylog2-server
https://github.com/exceptionless/Exceptionless
https://github.com/elastic
https://github.com/getsentry/sentry
https://github.com/dianping/catjava
配置 Elasticsearch 集羣
最終咱們仍是選用個了 Exceptionless ，因而就本身搭建了一套本地環境（Elasticsearch 配置官方推薦：three node with two master nodes on Linux）。node
- .NET 4.6.1
- IIS 7.5+
- ElasticSearch 5.1. We also have a guide located here.
#配置JAVA運行環境
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install software-properties-common htop
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
java -versionpython
# centos
yum install java(openjdk)
tar -zxvf jdk-8u131-linux-x64.tar.gz
#配置環境變量linux
vi /etc/profile JAVA_HOME=/opt/jdk1.8.0_131 CLASSPATH=.:$JAVA_HOME/lib PATH=$JAVA_HOME/bin:$PATH export JAVA_HOME CLASSPATH PATH
#生效環境變量git
source /etc/profilegithub
#安裝elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
tar -xzf elasticsearch-5.5.0.tar.gzweb
#安裝插件
sudo bin/elasticsearch-plugin install mapper-size
#配置elasticsearch-head
Running with docker
https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/
for Elasticsearch 5.x open http://localhost:9100/:
docker run -p 9100:9100 mobz/elasticsearch-head:5docker
#Enable CORS in elasticsearch
http.cors.enabled: true
http.cors.allow-origin: "*"
#建立elastic用戶
useradd elastic (userdel elastic)
#修改全部者
chown -R elastic /opt/elasticsearch-5.5.0
#切換到elastic用戶啓動
su elastic
#之後臺方式啓動
./bin/elasticsearch –d
Elasticsearch 配置
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # multi node configuration cluster.name: exceptionless action.destructive_requires_name: true action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history* #bootstrap.memory_lock: true # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: p2_es_node_03 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # #path.data: /path/to/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0 # # Set a custom port for HTTP: # http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.zen.ping.unicast.hosts: ["host1", "host2"] # # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: 3 # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true # 啓用跨域 #Enable CORS in elasticsearch http.cors.enabled: true http.cors.allow-origin: "*" # 集羣配置 # 這個配置限制了單機上能夠開啓的ES存儲實例的個數,當咱們須要單機多實例，則須要把這個配置賦值2，或者更高。 node.max_local_storage_nodes: 1 #設置這個集羣中節點的數量，默認爲2，一旦這N個節點啓動，就會當即進行數據恢復。 gateway.expected_nodes: 3 #設置這個參數來保證集羣中的節點能夠知道其它N個有master資格的節點。默認爲1，對於大的集羣來講，能夠設置大一點的值（2-4） discovery.zen.minimum_master_nodes: 2 #設置集羣中master節點的初始列表，能夠經過這些節點來自動發現新加入集羣的節點。 discovery.zen.ping.unicast.hosts: ["10.255.131.162","10.255.131.163","10.255.131.164"] #network.bind_host: [ '_site_', '_local_' ] #network.publish_host: '_site_'
cluster.name: exceptionless node.name: exceptionless_node_01 action.destructive_requires_name: true action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history* #bootstrap.memory_lock: true # Set the bind address to a specific IP (IPv4 or IPv6): network.host: 0.0.0.0 http.port: 9200 #Enable CORS in elasticsearch http.cors.enabled: true http.cors.allow-origin: "*" # --------------------------------- Discovery ---------------------------------- # 這個配置限制了單機上能夠開啓的ES存儲實例的個數,當咱們須要單機多實例，則須要把這個配置賦值2，或者更高。 node.max_local_storage_nodes: 1 #設置這個集羣中節點的數量，默認爲2，一旦這N個節點啓動，就會當即進行數據恢復。 gateway.expected_nodes: 3 #設置這個參數來保證集羣中的節點能夠知道其它N個有master資格的節點。默認爲1，對於大的集羣來講，能夠設置大一點的值（2-4）(total number of master-eligible nodes / 2 + 1): discovery.zen.minimum_master_nodes: 2 #設置集羣中master節點的初始列表，能夠經過這些節點來自動發現新加入集羣的節點。 discovery.zen.ping.unicast.hosts: ["10.255.131.162","10.255.131.163","10.255.131.164"]
JVM 配置
## JVM configuration ################################################################ ## IMPORTANT: JVM heap size ################################################################ ## ## You should always set the min and max JVM heap ## size to the same value. For example, to set ## the heap to 4 GB, set: ## ## -Xms4g ## -Xmx4g ## ## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html ## for more information ## ################################################################ # Xms represents the initial size of total heap space # Xmx represents the maximum size of total heap space -Xms4g -Xmx4g ################################################################ ## Expert settings ################################################################ ## ## All settings below this section are considered ## expert settings. Don't tamper with them unless ## you understand what you are doing ## ################################################################ ## GC configuration -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly ## optimizations # disable calls to System#gc -XX:+DisableExplicitGC # pre-touch memory pages used by the JVM during initialization -XX:+AlwaysPreTouch ## basic # force the server VM (remove on 32-bit client JVMs) -server # explicitly set the stack size (reduce to 320k on 32-bit client JVMs) -Xss1m # set to headless, just in case -Djava.awt.headless=true # ensure UTF-8 encoding by default (e.g. filenames) -Dfile.encoding=UTF-8 # use our provided JNA always versus the system one -Djna.nosys=true # use old-style file permissions on JDK9 -Djdk.io.permissionsUseCanonicalPath=true # flags to configure Netty -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 # log4j 2 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true ## heap dumps # generate a heap dump when an allocation from the Java heap fails # heap dumps are created in the working directory of the JVM -XX:+HeapDumpOnOutOfMemoryError # specify an alternative path for heap dumps # ensure the directory exists and has sufficient space #-XX:HeapDumpPath=${heap.dump.path} ## GC logging #-XX:+PrintGCDetails #-XX:+PrintGCTimeStamps #-XX:+PrintGCDateStamps #-XX:+PrintClassHistogram #-XX:+PrintTenuringDistribution #-XX:+PrintGCApplicationStoppedTime # log GC status to a file with time stamps # ensure the directory exists #-Xloggc:${loggc} # By default, the GC log file will not rotate. # By uncommenting the lines below, the GC log file # will be rotated every 128MB at most 32 times. #-XX:+UseGCLogFileRotation #-XX:NumberOfGCLogFiles=32 #-XX:GCLogFileSize=128M # Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON. # If documents were already indexed with unquoted fields in a previous version # of Elasticsearch, some operations may throw errors. # # WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided # only for migration purposes. #-Delasticsearch.json.allow_unquoted_field_names=true
運行日誌
LOG: elastic@iZuf60cj5pna5im3va46nlZ:/opt/elasticsearch-5.5.0$ ls bin config lib LICENSE.txt modules NOTICE.txt plugins README.textile elastic@iZuf60cj5pna5im3va46nlZ:/opt/elasticsearch-5.5.0$ ./bin/elasticsearch [2017-07-12T14:57:36,651][INFO ][o.e.n.Node ] [] initializing ... [2017-07-12T14:57:36,721][INFO ][o.e.e.NodeEnvironment ] [9rZOxk3] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [230.6gb], net total_space [245.9gb], spins? [possibly], types [ext4] [2017-07-12T14:57:36,722][INFO ][o.e.e.NodeEnvironment ] [9rZOxk3] heap size [1.9gb], compressed ordinary object pointers [true] [2017-07-12T14:57:36,723][INFO ][o.e.n.Node ] node name [9rZOxk3] derived from node ID [9rZOxk3hS0Wmj6SPvU0ZSg]; set [node.name] to override [2017-07-12T14:57:36,723][INFO ][o.e.n.Node ] version[5.5.0], pid[27006], build[260387d/2017-06-30T23:16:05.735Z], OS[Linux/4.4.0-63-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11] [2017-07-12T14:57:36,723][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/elasticsearch-5.5.0] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [aggs-matrix-stats] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [ingest-common] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [lang-expression] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [lang-groovy] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [lang-mustache] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [lang-painless] [2017-07-12T14:57:37,513][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [parent-join] [2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [percolator] [2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [reindex] [2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [transport-netty3] [2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded module [transport-netty4] [2017-07-12T14:57:37,514][INFO ][o.e.p.PluginsService ] [9rZOxk3] loaded plugin [mapper-size] [2017-07-12T14:57:38,768][INFO ][o.e.d.DiscoveryModule ] [9rZOxk3] using discovery type [zen] [2017-07-12T14:57:39,323][INFO ][o.e.n.Node ] initialized [2017-07-12T14:57:39,324][INFO ][o.e.n.Node ] [9rZOxk3] starting ... [2017-07-12T14:57:39,463][INFO ][o.e.t.TransportService ] [9rZOxk3] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300} [2017-07-12T14:57:39,475][WARN ][o.e.b.BootstrapChecks ] [9rZOxk3] max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536] [2017-07-12T14:57:39,476][WARN ][o.e.b.BootstrapChecks ] [9rZOxk3] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [2017-07-12T14:57:42,521][INFO ][o.e.c.s.ClusterService ] [9rZOxk3] new_master {9rZOxk3}{9rZOxk3hS0Wmj6SPvU0ZSg}{Khs76coWR6qWwC9DjuhyeQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined) [2017-07-12T14:57:42,543][INFO ][o.e.h.n.Netty4HttpServerTransport] [9rZOxk3] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200} [2017-07-12T14:57:42,543][INFO ][o.e.n.Node ] [9rZOxk3] started [2017-07-12T14:57:42,555][INFO ][o.e.g.GatewayService ] [9rZOxk3] recovered [0] indices into cluster_state
問題一：警告提示
[2016-11-06T16:27:21,712][WARN ][o.e.b.JNANatives ] unable to install syscall filter:
Java.lang.UnsupportedOperationException: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:349) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:630) ~[elasticsearch-5.0.0.jar:5.0.0]
報了一大串錯誤，其實只是一個警告。
解決：使用新的linux版本，就不會出現此類問題了。
問題二：ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
max number of threads [1024] for user [lishang] likely too low, increase to at least [2048]
vi /etc/security/limits.conf
添加以下內容:
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
問題三：max number of threads [1024] for user [lish] likely too low, increase to at least [2048]
vi /etc/security/limits.d/90-nproc.conf
#修改成
* soft nproc 2048
問題四：max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
解決：切換到root用戶修改配置sysctl.conf
vi /etc/sysctl.conf
添加下面配置：
vm.max_map_count=655360
並執行命令：
sysctl -p
而後，從新啓動elasticsearch，便可啓動成功。
#安裝kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-linux-x86_64.tar.gz
tar -xzf kibana-5.4.0-linux-x86_64.tar.gz
cd kibana/
#查看運行進程
ps -ef |grep java
#kill 進程
kill -9 pid
配置 Exceptionless
- 配置項目文件權限爲users
- 修改 app.config.*.js 的 BASE_URL 爲本機訪問IP
- 修改 Web.config 配置
<configuration> <connectionStrings> <add name="RedisConnectionString" connectionString="" /> <add name="ElasticSearchConnectionString" connectionString="http://10.255.131.162:9200" /> <add name="LdapConnectionString" connectionString="" /> </connectionStrings> <appSettings> <!-- Base url for the ui used to build links in emails and other places. --> <add key="BaseURL" value="http://10.255.130.67/#" /> <!-- Controls whether SSL is required. Only enable this if you have SSL configured. --> <add key="EnableSSL" value="false" /> <!-- Dev: Use this mode when debugging. (Outbound emails will not be sent) QA: Use this mode when deployed to staging. (Outbound emails restricted) Production: Use this mode when deployed to production. --> <add key="WebsiteMode" value="Production" /> <!-- Controls whether users can signup. --> <add key="EnableAccountCreation" value="false" /> <!-- Controls whether daily summary emails are sent --> <add key="EnableDailySummary" value="true" /> <!-- Email Client Settings (Uncomment the section below to change the default email settings) There are three valid SmtpEncryption settings: None, SSL and StartTLS --> <!-- Email --> <add key="SmtpHost" value="smtp.exmail.qq.com" /> <add key="SmtpPort" value="465" /> <add key="SmtpEncryption" value="SSL" /> <add key="SmtpFrom" value="test@test.com" /> <add key="SmtpUser" value="test@test.com" /> <add key="SmtpPassword" value="!test." /> <!-- Folder used to store event post data --> <add key="StorageFolder" value="|DataDirectory|\storage" /> <!-- Runs the jobs in the current website process --> <add key="RunJobsInProcess" value="true" /> <add key="aspnet:UseTaskFriendlySynchronizationContext" value="true" /> <!--保留日誌天數 --> <add key="MaximumRetentionDays" value="90" /> <!--分片數量--> <add key="ElasticSearchNumberOfShards" value="3" /> <!--複製分片數量--> <add key="ElasticSearchNumberOfReplicas" value="1" /> </appSettings>
首次訪問註冊用戶後建立項目便可(EnableAccountCreation 須要配置成 true)
檢查服務狀態
http://ip/api/v2/status
開放API
http://api.exceptionless.com/docs/index
客戶端調用示列
https://github.com/exceptionless/Exceptionless.Net/tree/master/samples
QA:
本地部署使用哪一個版本?
使用最新的版本便可
Elasticsearch(v5.5.0)
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.tar.gz
Exceptionless(v4.0.2)
https://github.com/exceptionless/Exceptionless/releases
大量記錄日誌會影響性能嗎？
支持 python 或者其餘語言收集日誌嗎?
默承認以使用相關的擴展來實現寫入到ES
https://github.com/ReactiveMarkets/NLog.Targets.ElasticSearch
https://github.com/jptoto/log4net.ElasticSearch
也可使用 Exceptionless 的擴展
https://github.com/serilog/serilog-sinks-exceptionless
https://github.com/exceptionless/Exceptionless.Net/tree/master/src/Platforms
如NLLog:
<nlog> <extensions> <add assembly="Exceptionless.NLog"/> </extensions> <targets async="true"> <target name="exceptionless" apiKey="API_KEY_HERE" xsi:type="Exceptionless"> <field name="host" layout="${machinename}" /> <field name="identity" layout="${identity}" /> <field name="windows-identity" layout="${windows-identity:userName=True:domain=False}" /> <field name="process" layout="${processname}" /> </target> </targets> <rules> <logger name="*" minlevel="Trace" writeTo="exceptionless" /> </rules> </nlog>
Elasticsearch 主分片與副本數量設定？
修改分片數
curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
"index.number_of_replicas" : "1",
"index.number_of_shards" : "5"
}'
Exceptionless 配置 QQ 郵箱發送不了郵件？
配置 <add key="SmtpFrom" value="test@test.com" /> 節點
怎麼配置保留日誌的天數？
配置 MaximumRetentionDays 屬性便可(https://github.com/exceptionless/Exceptionless/issues/139)
每秒產生大量的日誌請求 ，會有性能問題嗎？
可使用使用內存存儲(UseInMemoryStorage)，相對會穩定不少; Exceptionless 前端作負載（須要配置Redis）
ExceptionlessClient.Default.Configuration.UseInMemoryStorage();
怎麼標識是哪一個用戶操做發生的事件？
https://github.com/exceptionless/Exceptionless/wiki/User-Sessions
用戶標識訪問者歸屬地的 DB 下載失敗？
怎麼樣快速查詢日誌？
怎麼樣監控日誌服務是否正常？
Redis服務是不是必須配置的？
開放了哪些 API 接口 ？
http://api.exceptionless.com/docs/index
怎麼分配 Elasticsearch 中的 JVM 的內存大小？
Exceptionless 日誌沒有實時顯示？
檢查 IIS 是否配置了 websocket 協議 ; 檢查 SignalR 是否鏈接成功
ws://xx.xx.xxx.xxx/api/v2/push/connect?transport=webSockets&clientProtocol=1.5&access_token=hkm9YhSalvakZWkJmI3ek2M7rB8WTeuANmy0kp54&connectionToken=YQdrdiaOnprcRq%2BsaMBKfdpRwLnUtc5NPZ0ZExJVfb3h4j5SrHkUDqQOOq8NAgzGx7YADZ3azpJ3JIFFt217sSlE7pJf%2B7848sxFTfzXg%2FmAHvvcHjipHirL0Xl3FvfqaQaqo50Tyo5m77GvBM1YrQ%3D%3D&tid=8
REFER:
https://github.com/medcl/elasticsearch-analysis-ikhttp://blog.takipi.com/how-to-choose-the-right-log-management-tool/
https://thehftguy.com/2016/09/12/250-gbday-of-logs-with-graylog-lessons-learned/
https://exceptionless.com/sending-log-messages-to-exceptionless/
- 1. Self Host 使用 Exceptionless 實時監控程序運行日誌服務
- 2. Python 實時日誌監控
- 3. 實時日誌監控
- 4. 微服務分佈式日誌-ExceptionLess篇
- 5. 日誌流實時監控
- 6. Exceptionless服務端+kibana部署實時日誌紀要
- 7. php在fpm運行模式下實現服務之間的服務熔斷、服務監控、調用日誌
- 8. .NET Core 下使用 Exceptionless 記錄日誌
- 9. [Asp.net]SignalR實現實時日誌監控
- 10. Linux 運行進程實時監控pidstat
- 更多相關文章...
- • Eclipse 運行程序 - Eclipse 教程
- • SQLite 日期 & 時間 - SQLite教程
- • Spring Cloud 微服務實戰(三) - 服務註冊與發現
- • Git可視化極簡易教程 — Git GUI使用方法
-
每一个你不满意的现在，都有一个你没有努力的曾经。