Elasticsearch之優化

時間 2020-04-19

原文原文鏈接

爲何es須要優化？html

　　答：node

[root@master elasticsearch-2.4.0]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 6661
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 6661
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[root@master elasticsearch-2.4.0]# ulimit -n 32000
[root@master elasticsearch-2.4.0]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 6661
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 6661
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[root@master elasticsearch-2.4.0]#

　　es集羣的3節點，每一個機器都要去設置。master、slave1和slave2都要去操做。express

怎麼來作好es的優化工做？bootstrap

途徑一、解決es啓動的警告信息【或者es中Too many open files的問題】vim

　　max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]tomcat

　　vi /etc/security/limits.conf 添加下面兩行服務器

　　* soft nofile 65536app

　　* hard nofile 131072less

即，意思是把它們調大，重啓es服務進程，就生效了。curl

途徑二、修改配置文件調整ES的JVM內存大小

　　修改bin/elasticsearch.in.sh中ES_MIN_MEM和ES_MAX_MEM的大小，建議設置同樣大，避免頻繁的分配內存，根據服務器內存大小，通常分配60%左右(默認256M)

　　注意：內存最大不要超過32G【詳情請看以下的截圖和文字說明】

　　一旦你越過這個神奇的32 GB邊界，指針會切換回普通對象指針.。每一個指針的大小增長，使用更多的CPU內存帶寬。事實上，你使用40~50G的內存和使用32G的內存效果是同樣的。

連接：https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#compressed_oops

Don’t Cross 32 GB!
There is another reason to not allocate enormous heaps to Elasticsearch. As it turns out, the HotSpot JVM uses a trick to compress object pointers when heaps are less than around 32 GB.
In Java, all objects are allocated on the heap and referenced by a pointer. Ordinary object pointers (OOP) point at these objects, and are traditionally the size of the CPU’s native word: either 32 bits or 64 bits, depending on the processor. The pointer references the exact byte location of the value.
For 32-bit systems, this means the maximum heap size is 4 GB. For 64-bit systems, the heap size can get much larger, but the overhead of 64-bit pointers means there is more wasted space simply because the pointer is larger. And worse than wasted space, the larger pointers eat up more bandwidth when moving values between main memory and various caches (LLC, L1, and so forth).
Java uses a trick called compressed oops to get around this problem. Instead of pointing at exact byte locations in memory, the pointers reference object offsets. This means a 32-bit pointer can reference four billion objects, rather than four billion bytes. Ultimately, this means the heap can grow to around 32 GB of physical size while still using a 32-bit pointer.
Once you cross that magical ~32 GB boundary, the pointers switch back to ordinary object pointers. The size of each pointer grows, more CPU-memory bandwidth is used, and you effectively lose memory. In fact, it takes until around 40–50 GB of allocated heap before you have the same effective memory of a heap just under 32 GB using compressed oops.
The moral of the story is this: even when you have memory to spare, try to avoid crossing the 32 GB heap boundary. It wastes memory, reduces CPU performance, and makes the GC struggle with large heaps.

　　注意：是每一個es實例不要超過32G，而不是全部的。

[hadoop@master bin]$ pwd
/home/hadoop/app/elasticsearch-2.4.0/bin [hadoop@master bin]$ ll total 324 -rwxr-xr-x 1 hadoop hadoop 5551 Aug 24 2016 elasticsearch -rw-rw-r-- 1 hadoop hadoop 909 Aug 24 2016 elasticsearch.bat -rw-rw-r-- 1 hadoop hadoop 3307 Aug 24 2016 elasticsearch.in.bat -rwxr-xr-x 1 hadoop hadoop 2814 Aug 24 2016 elasticsearch.in.sh -rw-rw-r-- 1 hadoop hadoop 104448 Jul 27 2016 elasticsearch-service-mgr.exe -rw-rw-r-- 1 hadoop hadoop 103936 Jul 27 2016 elasticsearch-service-x64.exe -rw-rw-r-- 1 hadoop hadoop 80896 Jul 27 2016 elasticsearch-service-x86.exe -rwxr-xr-x 1 hadoop hadoop 2992 Aug 24 2016 plugin -rw-rw-r-- 1 hadoop hadoop 1303 Aug 24 2016 plugin.bat -rw-rw-r-- 1 hadoop hadoop 6872 Aug 24 2016 service.bat [hadoop@master bin]$ vim elasticsearch.in.sh

　　你們，自行去，根據本身機器內存實情，設置爲其60%。

途徑三、設置memory_lock來鎖定進程的物理內存地址

　　避免交換（swapped）來提升性能

　　修改文件conf/elasticsearch.yml

　　bootstrap.memory_lock: true

　　這裏，我就不贅述了。

[hadoop@master config]$ pwd
/home/hadoop/app/elasticsearch-2.4.0/config [hadoop@master config]$ ll total 12 -rw-rw-r-- 1 hadoop hadoop 3393 Jul 5 22:19 elasticsearch.yml -rw-rw-r-- 1 hadoop hadoop 2571 Aug 24 2016 logging.yml drwxrwxr-x 2 hadoop hadoop 4096 Apr 21 15:43 scripts [hadoop@master config]$ vim elasticsearch.yml

　　　去掉註釋。

　　es的3節點集羣，master、slave1和slave2都要去操做。

途徑四、分片多的話，能夠提高創建索引的能力，5-20個比較合適。

　　若是分片數過少或過多，都會致使檢索比較慢。

　　分片數過多會致使檢索時打開比較多的文件，另外也會致使多臺服務器之間通信。

　　而分片數過少會導至單個分片索引過大，因此檢索速度也會慢。

　　建議單個分片最多存儲20G左右的索引數據，因此，分片數量=數據總量/20G

途徑五、副本多的話，能夠提高搜索的能力，可是若是設置不少副本的話也會對服務器形成額外的壓力，由於須要主分片須要給全部副本同步數據。因此建議最多設置1-2個便可。

途徑六、Elastic 官方文檔建議：一個 es實例中最好不要多於三個 shards，如果 "more shards」，只能增長更多的機器，若是服務器性能好的話能夠在一臺服務器上啓動多個es實例

途徑七、要定時對索引進行合併優化，否則segment越多，佔用的segment memory越多，查詢的性能也越差

　　索引量不是很大的話狀況下能夠將segment設置爲1

　　在es2.1.0之前調用_optimize接口，後期改成_forcemerge接口

　　curl -XPOST 'http://localhost:9200/zhouls/_forcemerge?max_num_segments=1'

　　client.admin().indices().prepareForceMerge("zhouls").setMaxNumSegments(1).get();

途徑八、針對不使用的index，建議close，減小內存佔用。由於只要索引處於open狀態，索引庫中的segement就會佔用內存，close以後就只會佔用磁盤空間了。

curl -XPOST 'localhost:9200/zhouls/_close'

途徑九、刪除文檔：在es中刪除文檔，數據不會立刻在硬盤上除去，而是在es索引中產生一個.del的文件，而在檢索過程當中這部分數據也會參與檢索，es在檢索過程會判斷是否刪除了，若是刪除了在過濾掉。這樣也會下降檢索效率。因此能夠執行清除刪除文檔

curl -XPOST 'http://192.168.80.10:9200/zhouls/_forcemerge?only_expunge_deletes=true'

client.admin().indices().prepareForceMerge("zhouls").setOnlyExpungeDeletes(true).get();

途徑十、若是在項目開始的時候須要批量入庫大量數據的話，建議將副本數設置爲0

　　由於es在索引數據的時候，若是有副本存在，數據也會立刻同步到副本中，這樣會對es增長壓力。能夠等索引完成後將副本按須要改回來。這樣能夠提升索引效率。

途徑十一、去掉mapping中_all字段，Index中默認會有_all這個字段，默認會把全部字段的內容都拷貝到這一個字段裏面，這樣會給查詢帶來方便，可是會增長索引時間和索引尺寸。

　　禁用_all字段 "_all":{"enabled":false}

　　若是隻是某個字段不但願被加到_all中，可使用 "include_in_all":false

途徑十二、log輸出的水平默認爲trace，即查詢超過500ms即爲慢查詢，就要打印日誌，形成cpu和mem，io負載很高。把log輸出水平改成info，能夠減輕服務器的壓力。

　　修改ES_HOME/conf/logging.yaml文件

途徑1:能夠解決es的警告信息

　　其實啊，若咱們在ES_HOME目錄下，不用後臺bin/elasticsearch -d這種方式來啓動的話，用前臺bin/elasticsearch。則會看到，以下：

　　說明，我這裏是由於安裝了tomcat。因此，在前臺直接啓動，會出錯。

　因此，

[hadoop@HadoopMaster bin]$ pwd
/home/hadoop/app/tomcat-7.0.73/bin
[hadoop@HadoopMaster bin]$ ./startup.sh
Using CATALINA_BASE: /home/hadoop/app/tomcat-7.0.73
Using CATALINA_HOME: /home/hadoop/app/tomcat-7.0.73
Using CATALINA_TMPDIR: /home/hadoop/app/tomcat-7.0.73/temp
Using JRE_HOME: /home/hadoop/app/jdk1.7.0_79/jre
Using CLASSPATH: /home/hadoop/app/tomcat-7.0.73/bin/bootstrap.jar:/home/hadoop/app/tomcat-7.0.73/bin/tomcat-juli.jar
Tomcat started.
[hadoop@HadoopMaster bin]$ jps
2916 Jps
2906 Bootstrap
[hadoop@HadoopMaster bin]$ cd ..
[hadoop@HadoopMaster tomcat-7.0.73]$ cd ..
[hadoop@HadoopMaster app]$ cd elasticsearch-2.4.3/
[hadoop@HadoopMaster elasticsearch-2.4.3]$ bin/elasticsearch
[2017-02-28 22:08:49,862][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
[2017-02-28 22:08:51,324][INFO ][node ] [Dragonwing] version[2.4.3], pid[2930], build[d38a34e/2016-12-07T16:28:56Z]
[2017-02-28 22:08:51,324][INFO ][node ] [Dragonwing] initializing ...
[2017-02-28 22:08:55,760][INFO ][plugins ] [Dragonwing] modules [lang-groovy, reindex, lang-expression], plugins [analysis-ik, kopf, head], sites [kopf, head]
[2017-02-28 22:08:55,846][INFO ][env ] [Dragonwing] using [1] data paths, mounts [[/home (/dev/sda5)]], net usable_space [23.4gb], net total_space [26.1gb], spins? [possibly], types [ext4]
[2017-02-28 22:08:55,846][INFO ][env ] [Dragonwing] heap size [1015.6mb], compressed ordinary object pointers [true]
[2017-02-28 22:08:55,848][WARN ][env ] [Dragonwing] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2017-02-28 22:09:00,957][INFO ][ik-analyzer ] try load config from /home/hadoop/app/elasticsearch-2.4.3/config/analysis-ik/IKAnalyzer.cfg.xml
[2017-02-28 22:09:00,959][INFO ][ik-analyzer ] try load config from /home/hadoop/app/elasticsearch-2.4.3/plugins/ik/config/IKAnalyzer.cfg.xml
[2017-02-28 22:09:01,925][INFO ][ik-analyzer ] [Dict Loading] custom/mydict.dic
[2017-02-28 22:09:01,926][INFO ][ik-analyzer ] [Dict Loading] custom/single_word_low_freq.dic
[2017-02-28 22:09:01,932][INFO ][ik-analyzer ] [Dict Loading] custom/zhouls.dic
[2017-02-28 22:09:01,933][INFO ][ik-analyzer ] [Dict Loading] http://192.168.80.10:8081/zhoulshot.dic
[2017-02-28 22:09:09,451][INFO ][ik-analyzer ] 好記性不如爛筆頭感嘆號博客園熱更新詞
[2017-02-28 22:09:09,550][INFO ][ik-analyzer ] 桂林不霧霾
[2017-02-28 22:09:09,615][INFO ][ik-analyzer ] [Dict Loading] custom/ext_stopword.dic
[2017-02-28 22:09:13,620][INFO ][node ] [Dragonwing] initialized
[2017-02-28 22:09:13,621][INFO ][node ] [Dragonwing] starting ...
[2017-02-28 22:09:13,932][INFO ][transport ] [Dragonwing] publish_address {192.168.80.10:9300}, bound_addresses {[::]:9300}
[2017-02-28 22:09:13,960][INFO ][discovery ] [Dragonwing] elasticsearch/eKzsH0g5QoGl6pQlCG4mOQ
[2017-02-28 22:09:17,357][INFO ][cluster.service ] [Dragonwing] detected_master {Carrie Alexander}{98-Mux6mQsu1oE__EJN7yQ}{192.168.80.11}{192.168.80.11:9300}, added {{Carrie Alexander}{98-Mux6mQsu1oE__EJN7yQ}{192.168.80.11}{192.168.80.11:9300},{Shocker}{u_IYMF3ISe6_iki9KwxPCA}{192.168.80.12}{192.168.80.12:9300},}, reason: zen-disco-receive(from master [{Carrie Alexander}{98-Mux6mQsu1oE__EJN7yQ}{192.168.80.11}{192.168.80.11:9300}])
[2017-02-28 22:09:17,637][INFO ][http ] [Dragonwing] publish_address {192.168.80.10:9200}, bound_addresses {[::]:9200}
[2017-02-28 22:09:17,638][INFO ][node ] [Dragonwing] started
[2017-02-28 22:09:19,812][INFO ][ik-analyzer ] 從新加載詞典...
[2017-02-28 22:09:19,816][INFO ][ik-analyzer ] try load config from /home/hadoop/app/elasticsearch-2.4.3/config/analysis-ik/IKAnalyzer.cfg.xml
[2017-02-28 22:09:19,820][INFO ][ik-analyzer ] try load config from /home/hadoop/app/elasticsearch-2.4.3/plugins/ik/config/IKAnalyzer.cfg.xml
[2017-02-28 22:09:23,102][WARN ][monitor.jvm ] [Dragonwing] [gc][young][8][7] duration [1.6s], collections [1]/[1.9s], total [1.6s]/[5.2s], memory [121.7mb]->[79.4mb]/[1015.6mb], all_pools {[young] [59.9mb]->[457kb]/[66.5mb]}{[survivor] [8.2mb]->[8.3mb]/[8.3mb]}{[old] [53.5mb]->[70.6mb]/[940.8mb]}
[2017-02-28 22:09:23,946][INFO ][ik-analyzer ] [Dict Loading] custom/mydict.dic
[2017-02-28 22:09:23,947][INFO ][ik-analyzer ] [Dict Loading] custom/single_word_low_freq.dic
[2017-02-28 22:09:23,953][INFO ][ik-analyzer ] [Dict Loading] custom/zhouls.dic
[2017-02-28 22:09:23,955][INFO ][ik-analyzer ] [Dict Loading] http://192.168.80.10:8081/zhoulshot.dic
[2017-02-28 22:09:23,996][INFO ][ik-analyzer ] 好記性不如爛筆頭感嘆號博客園熱更新詞
[2017-02-28 22:09:23,997][INFO ][ik-analyzer ] 桂林不霧霾
[2017-02-28 22:09:24,000][INFO ][ik-analyzer ] [Dict Loading] custom/ext_stopword.dic
[2017-02-28 22:09:24,002][INFO ][ik-analyzer ] 從新加載詞典完畢...