yum -y install openssl-devel zlib-devel bzip2-devel sqlite-devel readline-devel libffi-devel systemtap-sdt-devel
三、下載安裝包node

tar zvxf Python-3.7.0.tgz
cd Python-3.7.0
./configure --prefix=/usr/local/python3.7 --enable-optimizations
make && make installgit

# 編譯完成後，建立軟連接文件到執行文件路徑：
ln -s /usr/local/python3/bin/python3 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3
# 咱們能夠清除以前編譯的可執行文件及配置文件 && 清除全部生成的文件：
make clean && make distcleangithub

if [ -z ${PYTHON37_HOME} ]; then
export PYTHON37_HOME=/usr/local/python3.7
export PATH=${PYTHON37_HOME}/bin:${PATH}
fi
六、加載環境變量shell

2，git2 安裝

centos7系統默認的git安裝版本是1.8，可是在項目構建中發現git版本太低，因而用源碼編譯的方式進行升級.

做者：一介布衣q
連接：https://www.imooc.com/article/275738
來源：慕課網
本文原創發佈於慕課網，轉載請註明出處，謝謝合做

——————————————————————————————————————————

4，使用ESRally壓測ElasticSearch性能

------------------------------------------------------
 _______ __ _____
 / ____(_)___ ____ _/ / / ___/_________ ________
 / /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/
/_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/
------------------------------------------------------

| Metric | Task | Value | Unit |
|-------------------------------:|---------------------:|----------:|-------:|
| Total indexing time | | 28.0997 | min |
| Total merge time | | 6.84378 | min |
| Total refresh time | | 3.06045 | min |
| Total flush time | | 0.106517 | min |
| Total merge throttle time | | 1.28193 | min |
| Median CPU usage | | 471.6 | % |
| Total Young Gen GC | | 16.237 | s |
| Total Old Gen GC | | 1.796 | s |
| Index size | | 2.60124 | GB |
| Total written | | 11.8144 | GB |
| Heap used for segments | | 14.7326 | MB |
| Heap used for doc values | | 0.115917 | MB |
| Heap used for terms | | 13.3203 | MB |
| Heap used for norms | | 0.0734253 | MB |
| Heap used for points | | 0.5793 | MB |
| Heap used for stored fields | | 0.643608 | MB |
| Segment count | | 97 | |
| Min Throughput | index-append | 31925.2 | docs/s |
| Median Throughput | index-append | 39137.5 | docs/s |
| Max Throughput | index-append | 39633.6 | docs/s |
| 50.0th percentile latency | index-append | 872.513 | ms |
| 90.0th percentile latency | index-append | 1457.13 | ms |
| 99.0th percentile latency | index-append | 1874.89 | ms |
| 100th percentile latency | index-append | 2711.71 | ms |
| 50.0th percentile service time | index-append | 872.513 | ms |
| 90.0th percentile service time | index-append | 1457.13 | ms |
| 99.0th percentile service time | index-append | 1874.89 | ms |
| 100th percentile service time | index-append | 2711.71 | ms |
| ... | ... | ... | ... |
| ... | ... | ... | ... |
| Min Throughput | painless_dynamic | 2.53292 | ops/s |
| Median Throughput | painless_dynamic | 2.53813 | ops/s |
| Max Throughput | painless_dynamic | 2.54401 | ops/s |
| 50.0th percentile latency | painless_dynamic | 172208 | ms |
| 90.0th percentile latency | painless_dynamic | 310401 | ms |
| 99.0th percentile latency | painless_dynamic | 341341 | ms |
| 99.9th percentile latency | painless_dynamic | 344404 | ms |
| 100th percentile latency | painless_dynamic | 344754 | ms |
| 50.0th percentile service time | painless_dynamic | 393.02 | ms |
| 90.0th percentile service time | painless_dynamic | 407.579 | ms |
| 99.0th percentile service time | painless_dynamic | 430.806 | ms |
| 99.9th percentile service time | painless_dynamic | 457.352 | ms |
| 100th percentile service time | painless_dynamic | 459.474 | ms |

----------------------------------
[INFO] SUCCESS (took 2634 seconds)
----------------------------------

在部署完一套ES集羣以後，咱們確定想知道這套集羣性能如何？是否能夠支撐將來業務發展？存不存在性能上的瓶頸？要想有依據的回答這些問題，咱們須要經過壓力測試結果中找答案。

介紹

Rally是Elasticsearch的基準測試框架，由官方提供維護。

安裝

安裝Python3.5及以上版本，系統默承認能是2.x版本，若是須要升級請參考《在CentOS7上安裝Python3》。
安裝git1.9及以上版本
安裝esrally pip3 install esrally
配置esrally esrally configure，執行此命令後會在當前用戶根目錄下生成 .rally 目錄，能夠 ll ~/.rally這樣來確認。

使用

快速開始

若是想測試當前機器上某個版本單點ES性能，能夠像下面這樣：

esrally --distribution-version=6.5.3

# 一樣的若是你想測試其它版本
esrally --distribution-version=6.8.1   --car="4gheap"

當執行上面的命令以後會自動下載對應es版本軟件，並在本地啓動，接着執行測試。這個過程在rally中被稱爲比賽，而賽道是用默認的，即geonames。

測試遠程集羣

上面的示例不能測試存在的es集羣，下面介紹使用方法：

指定跑道和ES集羣地址後就能夠執行測試。

esrally --pipeline=benchmark-only \
 --track=http_logs \
 --target-hosts=192.168.1.100:9200,192.168.1.101:9200,192.168.1.102:9200 \
 --report-file=/tmp/report_http_logs.md
--track-params="bulk_indexing_clients:96"
  --include-tasks="index-append"
--challenge=append-fast-with-no-conflicts (只測試寫）

--report-format=csv

備註：  命令能夠查看可用跑道（–track）

可使用  將此報告也保存到文件中，並使用 將其另存爲CSV。

esrally list tracks--report-file=/path/to/your/report.md--report-format=csv

修改默認跑道參數

若是直接在默認跑道上修改，會被還原，因此只能經過增長跑道的方式。

在 .rally/benchmarks/tracks 下面建立新的賽道，好比 custom。
在 custom/http_logs/challenges/default.json 文件中調整賽道的配置並保存，例以下面修改default對應的操做，由10個客戶端發起，每一個用戶端發出100次操做。
1
2
3
4
5
6
7

{
"operation": "default",
"clients": 10,
"warmup-iterations": 500,
"iterations": 100,
"target-throughput": 100
}

在啓動時指定跑道：esrally --track=custom ....，例如：

esrally --pipeline=benchmark-only \
 --track=custom \
 --track=http_logs \
 --target-hosts=192.168.1.100:9200,192.168.1.101:9200,192.168.1.102:9200 \
 --report-file=/tmp/report_http_logs.md

參考文獻：

https://esrally.readthedocs.io/en/stable/index.html
https://github.com/elastic/rally
比較全的壓測介紹：https://www.jianshu.com/p/c89975b50447
使用docker 運行 esrally（包含離線數據集）：https://www.jianshu.com/p/3a019c135e2a
測試參數解釋：https://www.jianshu.com/p/979f548c233e
討論爲何elasticsearch沒有被壓滿：https://discuss.elastic.co/t/es-benchmark-using-rally-to-stress-a-2-node-setup/150020/6
有壓測結果對比：https://www.jianshu.com/p/e7de3b24f505

數據集：
國內下載慢，能夠先執行一遍

esrally --distribution-version=6.5.3 --track=geonames
這樣即便下載測試數據失敗，可是目錄結構都生成好了。能夠自行下載bz文件，在

http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents.json.bz2
存成默認定義的：
documents-2.json.bz2
documents-2.json.offset
而後直接拷貝過去便可：~/.rally/benchmarks/data/geonames


esrally list tracks

This will show the following list:

 
     Name        Description Documents Compressed Size Uncompressed Size Default Challenge All Challenges ---------- ------------------------------------------------- ----------- ----------------- ------------------- ----------------------- --------------------------- geonames POIs from Geonames 11396505 252.4 MB 3.3 GB append-no-conflicts append-no-conflicts,appe... geopoint Point coordinates from PlanetOSM 60844404 481.9 MB 2.3 GB append-no-conflicts append-no-conflicts,appe... http_logs HTTP server log data 247249096 1.2 GB 31.1 GB append-no-conflicts append-no-conflicts,appe... nested StackOverflow Q&A stored as nested docs 11203029 663.1 MB 3.4 GB nested-search-challenge nested-search-challenge,... noaa Global daily weather measurements from NOAA 33659481 947.3 MB 9.0 GB append-no-conflicts append-no-conflicts,appe... nyc_taxis Taxi rides in New York in 2015 165346692 4.5 GB 74.3 GB append-no-conflicts append-no-conflicts,appe... percolator Percolator benchmark based on AOL queries 2000000 102.7 kB 104.9 MB append-no-conflicts append-no-conflicts,appe... pmc Full text benchmark with academic papers from PMC 574199 5.5 GB 21.7 GB  
    

這個地址裏面 https://github.com/elastic/rally-tracks/tree/master/，進入子目錄有各個數據集可配置的參數：

如http_logs：

Parameters （--track-params="bulk_indexing_clients:96"）

This track allows to overwrite the following parameters with Rally 0.8.0+ using --track-params:

bulk_size (default: 5000)
bulk_indexing_clients (default: 8): Number of clients that issue bulk indexing requests.

5,本身定義壓測：

參考：https://esrally.readthedocs.io/en/latest/adding_tracks.html

主要步驟：

1，建立目錄：~/rally-tracks/tutorial

2，手動下載 geonames 壓測數據：　http://download.geonames.org/export/dump/allCountries.zip　。它裏面是用ｔａｂ分開的文本文件。須要轉成ｊｓｏｎ格式。

用這段ｐｙｔｈｏｎ代碼轉：

import json

cols = (("geonameid", "int", True),
        ("name", "string", True),
        ("asciiname", "string", False),
        ("alternatenames", "string", False),
        ("latitude", "double", True),
        ("longitude", "double", True),
        ("feature_class", "string", False),
        ("feature_code", "string", False),
        ("country_code", "string", True),
        ("cc2", "string", False),
        ("admin1_code", "string", False),
        ("admin2_code", "string", False),
        ("admin3_code", "string", False),
        ("admin4_code", "string", False),
        ("population", "long", True),
        ("elevation", "int", False),
        ("dem", "string", False),
        ("timezone", "string", False))


def main():
    with open("allCountries.txt", "rt", encoding="UTF-8") as f:
        for line in f:
            tup = line.strip().split("\t")
            record = {}
            for i in range(len(cols)):
                name, type, include = cols[i]
                if tup[i] != "" and include:
                    if type in ("int", "long"):
                        record[name] = int(tup[i])
                    elif type == "double":
                        record[name] = float(tup[i])
                    elif type == "string":
                        record[name] = tup[i]
            print(json.dumps(record, ensure_ascii=False))


if __name__ == "__main__":
    main()

存到剛纔~/rally-tracks/tutorial目錄下，python3 toJSON.py > documents.json

３，對於７.０如下的ｅｓ，保持這個成ｉｎｄｅｘ．ｊｓｏｎ

{
  "settings": {
    "index.number_of_replicas": 0
  },
  "mappings": {
    "docs": {
      "dynamic": "strict",
      "properties": {
        "geonameid": {
          "type": "long"
        },
        "name": {
          "type": "text"
        },
        "latitude": {
          "type": "double"
        },
        "longitude": {
          "type": "double"
        },
        "country_code": {
          "type": "text"
        },
        "population": {
          "type": "long"
        }
      }
    }
  }
}

５，再保存一個ｔｒａｃｋ．ｊｓｏｎ

{
  "version": 2,
  "description": "Tutorial benchmark for Rally",
  "indices": [
    {
      "name": "geonames",
      "body": "index.json",
      "types": [ "docs" ]
    }
  ],
  "corpora": [
    {
      "name": "rally-tutorial",
      "documents": [
        {
          "source-file": "documents.json",
          "document-count": 11658903,
          "uncompressed-bytes": 1544799789
        }
      ]
    }
  ],
  "schedule": [
    {
      "operation": {
        "operation-type": "delete-index"
      }
    },
    {
      "operation": {
        "operation-type": "create-index"
      }
    },
    {
      "operation": {
        "operation-type": "cluster-health",
        "request-params": {
          "wait_for_status": "green"
        }
      }
    },
    {
      "operation": {
        "operation-type": "bulk",
        "bulk-size": 5000
      },
      "warmup-time-period": 120,
      "clients": 8
    },
    {
      "operation": {
        "operation-type": "force-merge"
      }
    },
    {
      "operation": {
        "name": "query-match-all",
        "operation-type": "search",
        "body": {
          "query": {
            "match_all": {}
          }
        }
      },
      "clients": 8,
      "warmup-iterations": 1000,
      "iterations": 1000,
      "target-throughput": 100
    }
  ]
}

其中documents 屬性裏面的字段值是這麼來的：

wc -l documents.json　ｊｓｏｎ個數

stat -f "%z" documents.json　文件大小

７.０之後版本要去掉types 。

６，檢查創建成功沒有：esrally list tracks --track-path=~/rally-tracks/tutorial

７，執行本身的ｔｒａｃｋ：esrally --distribution-version=6.0.0 --track-path=~/rally-tracks/tutorial

--test-mode　來檢測配置文件對否。

這個來生成１０００條數據：head -n 1000 documents.json > documents-1k.json

ESRally壓測ElasticSearch性能 CentOS 7.5 安裝 Python3.7

1，CentOS 7.5 安裝 Python3.7