elasticsearch性能優化筆記（2）—— 單點壓力測試（粗略）

時間 2019-11-10

標籤 elasticsearch 性能優化筆記單點壓力測試粗略欄目日誌分析简体版

原文原文鏈接

本部分主要對ES的一套測試流程有一個熟悉，不斷分析和學習各個指標和統計方法，找到性能瓶頸

1、裝備

Linux環境java

ElasticSearch單結點，無副本單分片linux

Kibana + x-pack 監控集羣web

jmeter壓測centos

nmon監測Linux性能緩存

2、服務器環境：

# cat /proc/version
Linux version 3.10.0-514.2.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Dec 6 23:06:41 UTC 2016服務器

----------------------------------------------------------------------------------------------------------------網絡

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    1
Core(s) per socket:    1
座：                 8
NUMA 節點：         1
廠商 ID：           GenuineIntel
CPU 系列：          6
型號：              42
型號名稱：        Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
步進：              1
CPU MHz：             2394.454
BogoMIPS：            4788.90
超管理器廠商： KVM
虛擬化類型：     徹底
L1d 緩存：          32K
L1i 緩存：          32K
L2 緩存：           4096Kapp

----------------------------------------------------------------------------------------------------------------cors

# free -m
              total        used        free      shared buff/cache   available
Mem:          32013       16583        1027         198       14402       14461
Swap:             0           0           0less

----------------------------------------------------------------------------------------------------------------

# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0   20G 0 disk
└─vda1 253:1    0   20G 0 part /
vdb    253:16   0 200G 0 disk
└─vdb1 253:17   0 200G 0 part

# fdisk -l

磁盤 /dev/vda：21.5 GB, 21474836480 字節，41943040 個扇區
Units = 扇區 of 1 * 512 = 512 bytes
扇區大小(邏輯/物理)：512 字節 / 512 字節
I/O 大小(最小/最佳)：512 字節 / 512 字節
磁盤標籤類型：dos
磁盤標識符：0x000a0566

設備 Boot Start End Blocks Id System
/dev/vda1 * 2048 41943039 20970496 83 Linux

磁盤 /dev/vdb：214.7 GB, 214748364800 字節，419430400 個扇區
Units = 扇區 of 1 * 512 = 512 bytes
扇區大小(邏輯/物理)：512 字節 / 512 字節
I/O 大小(最小/最佳)：512 字節 / 512 字節
磁盤標籤類型：dos
磁盤標識符：0x739398fb

設備 Boot Start End Blocks Id System
/dev/vdb1 2048 419430399 209714176 83 Linux

----------------------------------------------------------------------------------------------------------------

3、準備過程

一、elasticsearch5.4.2，kibana，x-pack

elasticsearch即插即用，下下來改改配置就能用；

kibana相似，在配置文件裏修改成本地的ES地址便可；

本來是打算用marvel監控，但貌似自5.0以後的ES的HEAD啊，marvel啊這些插件全沒了，全集成到x-pack裏面了

elasticsearch中運行 elasticsearch-plugin install x-pack

kibana中運行kibana-plugin install x-pack（時間稍有些慢）

即把ES相關的性能圖添加到kibana中

二、jmeter

jmeter是java開發的一塊集成測試的軟件，經過它能夠構建測試用例，保存jmx測試用例並交到服務器上執行。

jmeter支持遠程啓動，在服務器上啓動jmeter-server(可能須要參數指定ip），在本機jmeter.properites中添加remote_server:遠程ip:1099（1099爲默認端口），便可在jmeter運行腳本時使用遠程啓動。

因爲服務器不在局域網內，所以上傳jmx文件給服務器使用

操做：

右鍵測試計劃，添加線程組，修改線程數，循環次數，運行時間等關鍵參數

在線程組中添加Sampler，選擇HTTP請求

在HTTP請求中添加聚合報告和結果樹

在文件=>保存爲測試計劃中保存jmx文件

三、nmon

nmon是一款linux下的系統性能指標工具，提供CPU，內存，磁盤IO，網絡等多項指標的界面化展現。

安裝：

wget http://sourceforge.net/projects/nmon/files/nmon16d_x86.tar.gz

tar xvzf nmon16d_x86.tar.gz

獲得一系列nmon對應的各類版本的文件

找到相應的版本的nmon文件

chmod +x 賦予可執行權限

運行：

其中每一個參數可控制展現的開關，按下c,m,d,n

便可監控系統性能

4、開始測試

測試樣本對照性

	ES結點數	分片數	副本數	ES配置	線程數	ES數據	索引setting	JVM參數	逐條insert寫入速率範圍(條/s)	5分鐘寫入數據條數／數據量
Sample1	1	1	0	A	1	A	無	A
Sample2	1	1	0	A	200	A	無	A
Sample3	1	1	0	A	1000	B	A	A
Sample4	1	1	0	A	200	B	A	A
Sample5	1	1	0	A	200	C	A	A
Sample6
Sample7

樣本項目描述

——ES配置：

A：

http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: false
indices.memory.index_buffer_size: "50%"
indices.memory.min_index_buffer_size: "1024mb"
http.max_initial_line_length: "64kb"
http.max_header_size: "64kb"
action.search.shard_count.limit: 50000
script.inline: true
script.stored: true
thread_pool.bulk.queue_size: 5000

——JVM參數：

A：

-Xms2g -Xmx2g -XX:+UseConcMarkSweepGC

-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8

-Djna.nosys=true

-Djdk.io.permissionsUseCanonicalPath=true

-Dio.netty.noUnsafe=true

-Dio.netty.noKeySetOptimization=true

-Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false

-Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError

——ES數據：

A：

{
"_MCSJnlNo": "45139477"
}

B：

{
 "_MCSJnlNo": "45139477",
 "_raw": "2017-05-01 04:51:10.723 [[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO c.c.p.s.c.t.TransformTransport - recv:<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<Message>\r\n<Head>\r\n<_MCSJnlNo>45139477</_MCSJnlNo>\r\n<_ResponseSystemId>MCA</_ResponseSystemId>\r\n<_RejCode>000000</_RejCode>\r\n<_RejMsg></_RejMsg>\r\n<_MCHJnlNo>3973735</_MCHJnlNo>\r\n<_TransactionId>per.MCLoanInterestRateQuery</_TransactionId>\r\n<_MCHTimestamp>2017-05-01 04:51:10.61</_MCHTimestamp>\r\n</Head>\r\n<Body>\r\n<InterestRate>10.00</InterestRate>\r\n</Body>\r\n</Message>\r",
 "_context_id": 472,
 "_host": "192.168.0.34",
 "type": "recv",
 "_MCHTimestamp": "2017-05-01 04:51:10.61",
 "_MCHJnlNo": "3973735",
 "_RejCode": "000000",
 "_ResponseSystemId": "MCA",
 "InterestRate": "10.00",
 "_TransactionId": "per.MCLoanInterestRateQuery",
 "_ts": 1493585470000
}

C：

{
 "_raw": "[2017-07-23 22:09:49,435] [elasticsearch] [DEBUG] [< {\"took\":12,\"timed_out\":false,\"_shards\":{\"total\":10,\"successful\":10,\"failed\":0},\"hits\":{\"total\":196699,\"max_score\":null,\"hits\":[{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3QT\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:35 test_longtime1/longtimetxt.0 three line log for one 296963 \",\"_context_id\":1088331,\"_host\":\"192.168.0.33\",\"_ts\":1500818975000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:35 test_longtime1/longtimetxt.0 three line log for one 296963 \"]},\"sort\":[1500818975000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3QQ\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:34 test_longtime1/longtimetxt.0 three line log for one 296962 \",\"_context_id\":1088328,\"_host\":\"192.168.0.33\",\"_ts\":1500818974000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:34 test_longtime1/longtimetxt.0 three line log for one 296962 \"]},\"sort\":[1500818974000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3QN\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:33 test_longtime1/longtimetxt.0 three line log for one 296961 \",\"_context_id\":1088325,\"_host\":\"192.168.0.33\",\"_ts\":1500818973000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:33 test_longtime1/longtimetxt.0 three line log for one 296961 \"]},\"sort\":[1500818973000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3OF\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:32 test_longtime1/longtimetxt.0 three line log for one 296960 \",\"_context_id\":1088322,\"_host\":\"192.168.0.33\",\"_ts\":1500818972000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:32 test_longtime1/longtimetxt.0 three line log for one 296960 \"]},\"sort\":[1500818972000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3OC\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:31 test_longtime1/longtimetxt.0 three line log for one 296959 \",\"_context_id\":1088319,\"_host\":\"192.168.0.33\",\"_ts\":1500818971000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:31 test_longtime1/longtimetxt.0 three line log for one 296959 \"]},\"sort\":[1500818971000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3N_\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:30 test_longtime1/longtimetxt.0 three line log for one 296958 \",\"_context_id\":1088316,\"_host\":\"192.168.0.33\",\"_ts\":1500818970000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:30 test_longtime1/longtimetxt.0 three line log for one 296958 \"]},\"sort\":[1500818970000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Lx\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:29 test_longtime1/longtimetxt.0 three line log for one 296957 \",\"_context_id\":1088313,\"_host\":\"192.168.0.33\",\"_ts\":1500818969000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:29 test_longtime1/longtimetxt.0 three line log for one 296957 \"]},\"sort\":[1500818969000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Lu\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:28 test_longtime1/longtimetxt.0 three line log for one 296956 \",\"_context_id\":1088310,\"_host\":\"192.168.0.33\",\"_ts\":1500818968000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:28 test_longtime1/longtimetxt.0 three line log for one 296956 \"]},\"sort\":[1500818968000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Lr\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:27 test_longtime1/longtimetxt.0 three line log for one 296955 \",\"_context_id\":1088307,\"_host\":\"192.168.0.33\",\"_ts\":1500818967000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:27 test_longtime1/longtimetxt.0 three line log for one 296955 \"]},\"sort\":[1500818967000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Jr\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:26 test_longtime1/longtimetxt.0 three line log for one 296954 \",\"_context_id\":1088304,\"_host\":\"192.168.0.33\",\"_ts\":1500818966000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:26 test_longtime1/longtimetxt.0 three line log for one 296954 \"]},\"sort\":[1500818966000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Jo\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:25 test_longtime1/longtimetxt.0 three line log for one 296953 \",\"_context_id\":1088301,\"_host\":\"192.168.0.33\",\"_ts\":1500818965000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:25 test_longtime1/longtimetxt.0 three line log for one 296953 \"]},\"sort\":[1500818965000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyDSJVct_8CDGz3Jl\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:24 test_longtime1/longtimetxt.0 three line log for one 296952 \",\"_context_id\":1088298,\"_host\":\"192.168.0.33\",\"_ts\":1500818964000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:24 test_longtime1/longtimetxt.0 three line log for one 296952 \"]},\"sort\":[1500818964000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3Hl\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:23 test_longtime1/longtimetxt.0 three line log for one 296951 \",\"_context_id\":1088295,\"_host\":\"192.168.0.33\",\"_ts\":1500818963000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:23 test_longtime1/longtimetxt.0 three line log for one 296951 \"]},\"sort\":[1500818963000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3Hi\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:22 test_longtime1/longtimetxt.0 three line log for one 296950 \",\"_context_id\":1088292,\"_host\":\"192.168.0.33\",\"_ts\":1500818962000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:22 test_longtime1/longtimetxt.0 three line log for one 296950 \"]},\"sort\":[1500818962000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3Hf\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:21 test_longtime1/longtimetxt.0 three line log for one 296949 \",\"_context_id\":1088289,\"_host\":\"192.168.0.33\",\"_ts\":1500818961000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:21 test_longtime1/longtimetxt.0 three line log for one 296949 \"]},\"sort\":[1500818961000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3FU\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:20 test_longtime1/longtimetxt.0 three line log for one 296948 \",\"_context_id\":1088286,\"_host\":\"192.168.0.33\",\"_ts\":1500818960000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:20 test_longtime1/longtimetxt.0 three line log for one 296948 \"]},\"sort\":[1500818960000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3FR\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:19 test_longtime1/longtimetxt.0 three line log for one 296947 \",\"_context_id\":1088283,\"_host\":\"192.168.0.33\",\"_ts\":1500818959000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:19 test_longtime1/longtimetxt.0 three line log for one 296947 \"]},\"sort\":[1500818959000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3FO\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:18 test_longtime1/longtimetxt.0 three line log for one 296946 \",\"_context_id\":1088280,\"_host\":\"192.168.0.33\",\"_ts\":1500818958000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:18 test_longtime1/longtimetxt.0 three line log for one 296946 \"]},\"sort\":[1500818958000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3Cr\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:17 test_longtime1/longtimetxt.0 three line log for one 296945 \",\"_context_id\":1088277,\"_host\":\"192.168.0.33\",\"_ts\":1500818957000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:17 test_longtime1/longtimetxt.0 three line log for one 296945 \"]},\"sort\":[1500818957000]},{\"_index\":\"ls_749bc4e976f24f6f9fee21e4bc6c3e5b\",\"_type\":\"20170723\",\"_id\":\"AV1vyAW0Vct_8CDGz3Co\",\"_score\":null,\"_source\":{\"_raw\":\"2017-07-23 22:09:16 test_longtime1/longtimetxt.0 three line log for one 296944 \",\"_context_id\":1088274,\"_host\":\"192.168.0.33\",\"_ts\":1500818956000},\"highlight\":{\"_raw\":[\"2017-07-23 22:09:16 test_longtime1/longtimetxt.0 three line log for one 296944 \"]},\"sort\":[1500818956000]}]}}]",
 "_context_id": 1247368,
 "_host": "192.168.0.34",
 "_ts": 1500818989000,

"data1": 重複raw,

"data2": 重複raw,

"data3": 重複raw,

"data4": 重複raw,

"data5": 重複raw,

"data6": 重複raw,

"data7": 重複raw,

"data8": 重複raw,

"data9": 重複raw,

"data10": 重複raw,
}

——索引settings：

A：

PUT index1

{
        "template": "*",
        "order": 1,
        "settings": {
            "number_of_shards": 1,
            "number_of_replicas": 0,
            "index.query.default_field": "_raw"
        },
        "mappings": {
            "_default_": {
                "date_detection": false,
                "numeric_detection": false,
                "_all": {
                    "enabled": false
                },
                "properties": {
                    "_host": {
                        "type": "keyword",
                        "norms": false
                    },
                    "_raw": {
                        "type": "text"
                    },
                    "_ts": {
                        "type": "long"
                    },
                    "_context_id": {
                        "type": "long"
                    }
                },
                "dynamic_templates": [
                    {
                        "strings_as_keywords": {
                            "match_mapping_type": "string",
                            "mapping": {
                                "type": "keyword"
                            }
                        }
                    }
                ]
            }
        }
    }