運維相關指標數據採集並ES入倉 - 運維筆記

時間 2019-11-21

原文原文鏈接

爲了進行數字化IT治理，須要對一些應用進程相關指標進行採集併入庫。收集到的應用指標數據最好要進行ES入倉，入到Kafka裏面，並經過Kibana可視化展現。python

須要進行採集的應用進程相關指標以下：ios

ES入庫的規範以下：redis

中文名shell	英文名json	類型數組	是否必填tomcat	說明安全
指標名稱bash	indexNameapp	CHAR	是
指標值	indexValue	CHAR	是	支持批量
指標類別	indexType	CHAR	是	安全測試運行應用環境
指標描述	indexDesc	VARCHAR	是	指標說明，指標採集數據源（zbx、dota、legao……）
採集時間	collectTime	TIMESTAMP	是	支持批量
應用名稱	appName	CHAR	是	以AIOPS的3位編碼爲準
主機名	hostName	CHAR	否
發送數據源主機	dataSource	CHAR	是	腳本路徑@主機IP

下面是應用指標數據進行ES入倉的請求說明

測試區接口說明：
訪問連接：http://192.168.10.10:10222/haha/heiheiAPI/test_1_kafka
請求header：Content-Type:application/json
請求方法：POST
請求參數，token中填入示例中的128位token便可，data的值是一個數組，數組中的每一個元素是一個json對象（對應會插入ES中的1個document），給出具體信息：
{
    "token": "bd1df2b11111111112222222222222222333333333333333333yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy",
    "data": [
        {
            "type": "英文字符，最長不超過10位",
            "desc": "指標的中文描述, 可爲空",
            "@timestamp": "時間戳，以標準unix時間戳爲準，下面會給出shell和python的生成示例",
            "appname": "應用系統編碼，以aiops上的信息爲準",
            "hostname": "主機名",
            "value": "採集的指標值, 因爲各系統需求不一致, 此處統一使用string類型, 不然可能會形成mapping衝突, 但某些功能須要int/float,再討論",
            "checker": "發送數據的腳本路徑或IP, 可爲空"
        }
    ]
}


==============================================================================================
# python生成時間戳示例
import datetime
timestamp = datetime.datetime.strftime(datetime.datetime.now(), '%s')
 
==============================================================================================
# python請求示例
import urllib2   # 這裏是python2，根據各人使用有不一樣，總之是構建http request
import hashlib
import json
url = 'http://192.168.10.10:10222/haha/heiheiAPI/test_1_kafka'
data = {
    'token': 'bd1df2b11111111112222222222222222333333333333333333yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy',
    'data': [
        {}
    ]
}
headers = {'Content-Type': 'application/json'}
req = urllib2.Request(url, json.dumps(data), headers = headers)
res = urllib2.urlopen(req).read()
=============================================================================================

# bash shell生成時間戳示例
date +'%s'
# bash shell請求示例
curl -s -XPOST -H "Content-Type:application/json" -d 請求數據 http://192.168.10.10:10222/haha/heiheiAPI/test_1_kafka

這裏使用了兩個腳本：
1）shell腳本進行指標數據採集（最終使用腳本：20_app_process_index.sh）
2）python腳本進行ES入倉請求提交，請求方法:POST。在python腳本里調用shell腳本，調取shell腳本執行結果。（最終使用腳本：20_app_process_index_es.py）

[root@kevin_test app_process_index]# pwd
/root/100_app_index/app_process_index
[root@kevin_test app_process_index]# ll
total 120
-rwxr-xr-x 1 root root   783 Jun 20 18:16 20_app_process_index_es.py
-rwxr-xr-x 1 root root   290 Jun 20 18:16 20_app_process_index.py2
-rwxr-xr-x 1 root root 15398 Jun 20 18:19 20_app_process_index.sh
-rwxr-xr-x 1 root root 10414 Jun 20 18:16 20_app_process_index_shell.sh

[root@kevin_test ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  4.8G   14G  26% /
tmpfs           2.9G   12K  2.9G   1% /dev/shm
/dev/vdb2       9.8G  458M  8.8G   5% /software
/dev/vdb3        30G   21G  7.2G  75% /opt/bobo
/dev/vdb4        38G  6.9G   29G  20% /var/log/bobo

其中，/opt/cbd爲應用程序所在空間，/var/log/bobo是日誌所在空間

1、shell腳本（執行結果要時JSON格式的，這裏在shell腳本里拼出來的JSON樣式）

[root@kevin_test app_process_index]# cat 20_app_process_index_shell.sh 
#!/bin/bash

source /etc/profile
hostName=$(hostname)
IP=$(ifconfig eth0| sed -nr '2s/(^[^0-9]+)([0-9.]+)+( .*$)/\2/gp')
appName=$(echo ${hostName:3:3})

collectTime=`date +'%s'`
dataSource=$(echo "`pwd`/$0@${IP}")

indexType="應用進程相關指標"

echo "指標類型: ${indexType}"
echo "指標採集時間: ${collectTime}"
echo "主機名: ${hostName}"
echo "應用名稱: ${appName}"
echo "發送數據源主機: ${dataSource}"
echo -e "\n這次採集應用進程相關指標狀況以下:"

echo -e "---------------------------------------------------"
###########應用進程佔內存百分比###########
indexName1="應用進程佔內存百分比"
ps auxw|grep bxapp|awk '{if (NR>1){print $4}}' > /tmp/mem_list
NUM1=`cat /tmp/mem_list|wc -l`

if [ $NUM1 -eq 0 ];then
   echo "${indexName}爲0"
else
   MEM_PER_SUM=$(awk '{MEM_PER+=$1}END{print MEM_PER}' /tmp/mem_list)
   echo "${indexName1}:" ${MEM_PER_SUM}%
fi

echo -e "---------------------------------------------------"
###########應用進程佔CPU百分比###########
indexName2="應用進程佔CPU百分比"
ps auxw|grep bxapp|awk '{if (NR>1){print $3}}' > /tmp/cpu_list
NUM2=`cat /tmp/cpu_list|wc -l`

if [ $NUM2 -eq 0 ];then
   echo "${indexName2}爲0"
else
   CPU_PER_SUM=$(awk '{CPU_PER+=$1}END{print CPU_PER}' /tmp/cpu_list)
   echo "${indexName2}:" ${CPU_PER_SUM}%
fi

echo -e "---------------------------------------------------"
###########應用進程run最長時間及其進程###########
indexName3="應用進程run最長時間"
PID=$(ps -eo user,pid,etime|grep bxapp|sort -k 3 -rn|head -1|awk '{print $2}')
RUN_TIME=$(ps -eo user,pid,etime|grep bxapp|sort -k 3 -rn|head -1|awk '{print $3}')
PROCESS=$(ps -ef|grep ${PID}|grep -v grep|awk '{print $8}')

echo -e "${indexName3}: ${RUN_TIME} \n進程: ${PROCESS}"


echo -e "---------------------------------------------------"
###########應用進程TIME-WAIT鏈接數###########
indexName4="應用進程TIME-WAIT鏈接數"
NUM3=$(ss -a|grep -w "TIME_WAIT" |wc -l)
echo "${indexName4}: ${NUM3}"

echo -e "---------------------------------------------------"
###########應用進程ESTABLISHED鏈接數###########
indexName5="應用進程ESTABLISHED鏈接數"
NUM4=$(ss -a|grep -w "ESTAB" |wc -l)
echo "${indexName5}: ${NUM4}"

echo -e "---------------------------------------------------"
###########應用進程FIN_WAIT2鏈接數###########
indexName6="應用進程FIN_WAIT2鏈接數"
NUM5=$(ss -a|grep -w "FIN_WAIT2" |wc -l)
echo "${indexName6}: ${NUM5}"

echo -e "---------------------------------------------------"
###########日誌空間使用率###########
indexName7="日誌空間使用率"
LOG_SPACE_USER=$(df -h|grep "/var/log"|awk '{print $5}')
echo "${indexName7}: ${LOG_SPACE_USER}"

echo -e "---------------------------------------------------"
###########應用程序空間使用率###########
indexName8="應用程序空間使用率"
APP_SPACE_USER=$(df -h|grep "/opt"|awk '{print $5}')
echo "${indexName8}: ${APP_SPACE_USER}"

echo -e "---------------------------------------------------"
###########應用所在磁盤讀速率（Blk/s）###########
indexName9="應用所在磁盤讀速率（Blk/s）"
DISK=$(df -h|grep "/opt/${appName}"|awk '{print $1}')
DEVICE=$(df -h|grep "/opt/${appName}"|awk '{print $1}'|awk -F"/" '{print $3}')
READ=$(iostat ${DISK}|grep -w ${DEVICE}|awk '{print $3}')

echo "${indexName9}: ${READ}"

echo -e "---------------------------------------------------"
###########應用所在磁盤寫速率（Blk/s）###########
indexName10="應用所在磁盤寫速率（Blk/s）"
DISK=$(df -h|grep "/opt/${appName}"|awk '{print $1}')
DEVICE=$(df -h|grep "/opt/${appName}"|awk '{print $1}'|awk -F"/" '{print $3}')
WRITE=$(iostat ${DISK}|grep -w ${DEVICE}|awk '{print $4}')

echo "${indexName10}: ${WRITE}"

echo -e "---------------------------------------------------"
###########應用所在磁盤TPS###########
indexName11="應用所在磁盤TPS"
DISK=$(df -h|grep "/opt/${appName}"|awk '{print $1}')
DEVICE=$(df -h|grep "/opt/${appName}"|awk '{print $1}'|awk -F"/" '{print $3}')
TPS=$(iostat ${DISK}|grep -w ${DEVICE}|awk '{print $2}')

echo "${indexName11}: ${TPS}"

echo -e "---------------------------------------------------"
###########TOMCAT分配最小內存 ###########
indexName12="TOMCAT分配最小內存"
NUM6=$(ps -ef|grep -w tomcat|grep -v grep|wc -l)
if [ ${NUM6} -eq 0 ];then
   echo "當前沒有Tomcat進程，${indexName12}: 0"
else
    for TOMCAT in $(find /opt/${appName}/ -name *tomcat* -type d)
    do
      NUM7=$(ps -ef|grep ${TOMCAT}|grep -v grep|wc -l)
      if [ ${NUM7} -eq 0 ];then
        echo "not ${TOMCAT}" >> /dev/null 2>&1
      else
        XMS=$(ps -ef|grep ${TOMCAT}/|grep "\-Xms"|awk -F"-Xms" '{print $2}'|cut -d" " -f1)
        echo -e "${indexName12}: ${XMS}  (實例名:${TOMCAT})"
      fi
    done
fi

echo -e "---------------------------------------------------"
###########TOMCAT分配最大內存 ###########
indexName13="TOMCAT分配最大內存"
NUM8=$(ps -ef|grep -w tomcat|grep -v grep|wc -l)
if [ ${NUM8} -eq 0 ];then
   echo "當前沒有Tomcat進程，${indexName13}: 0"
else
    for TOMCAT in $(find /opt/${appName}/ -name *tomcat* -type d)
    do
      NUM9=$(ps -ef|grep ${TOMCAT}|grep -v grep|wc -l)
      if [ ${NUM9} -eq 0 ];then
        echo "not ${TOMCAT}" >> /dev/null 2>&1
      else
        XMX=$(ps -ef|grep ${TOMCAT}/|grep "\-Xmx"|awk -F"-Xmx" '{print $2}'|cut -d" " -f1)
        echo -e "${indexName13}: ${XMX}  (實例名:${TOMCAT})"
      fi
    done
fi

echo -e "---------------------------------------------------"
###########TOMCAT分配年輕代空間 ###########
indexName14="TOMCAT分配年輕代空間"
NUM10=$(ps -ef|grep -w tomcat|grep -v grep|wc -l)
if [ ${NUM10} -eq 0 ];then
   echo "當前沒有Tomcat進程，${indexName14}: 0"
else
    for TOMCAT in $(find /opt/${appName}/ -name *tomcat* -type d)
    do
      NUM11=$(ps -ef|grep ${TOMCAT}|grep -v grep|wc -l)
      if [ ${NUM11} -eq 0 ];then
        echo "not ${TOMCAT}" >> /dev/null 2>&1
      else
        XMN=$(ps -ef|grep ${TOMCAT}/|grep "\-Xmn"|awk -F"-Xmn" '{print $2}'|cut -d" " -f1)
        echo -e "${indexName14}: ${XMN}  (實例名:${TOMCAT})"
      fi
    done
fi

echo -e "---------------------------------------------------"
###########TOMCAT每一個線程的堆棧大小 ###########
indexName15="TOMCAT每一個線程的堆棧大小"
NUM12=$(ps -ef|grep -w tomcat|grep -v grep|wc -l)
if [ ${NUM12} -eq 0 ];then
   echo "當前沒有Tomcat進程，${indexName15}: 0"
else
    for TOMCAT in $(find /opt/${appName}/ -name *tomcat* -type d)
    do
      NUM13=$(ps -ef|grep ${TOMCAT}|grep -v grep|wc -l)
      if [ ${NUM13} -eq 0 ];then
        echo "not ${TOMCAT}" >> /dev/null 2>&1
      else
        XSS=$(ps -ef|grep ${TOMCAT}/|grep "\-Xss"|awk -F"-Xss" '{print $2}'|cut -d" " -f1)
        echo -e "${indexName15}: ${XSS}  (實例名:${TOMCAT})"
      fi
    done
fi

echo -e "---------------------------------------------------"
###########TOMCAT-GC時長佔比 ###########
indexName16="TOMCAT-GC時長佔比"
NUM14=$(ps -ef|grep -w tomcat|grep -v grep|wc -l)
if [ ${NUM14} -eq 0 ];then
   echo "當前沒有Tomcat進程，${indexName16}: 0"
else
    for TOMCAT in $(find /opt/${appName}/ -name *tomcat* -type d)
    do
      NUM15=$(ps -ef|grep ${TOMCAT}|grep -v grep|wc -l)
      if [ ${NUM15} -eq 0 ];then
        echo "not ${TOMCAT}" >> /dev/null 2>&1
      else
        GC=$(ps -ef|grep ${TOMCAT}/|grep "\GCTimeRatio"|awk -F"GCTimeRatio" '{print $2}'|cut -d" " -f1|awk -F"=" '{print $2}')
        echo -e "${indexName16}: ${GC}  (實例名:${TOMCAT})"
      fi
    done
fi

echo -e "---------------------------------------------------"
###########Zookeeper分配最小內存 ###########
indexName17="Zookeeper分配最小內存"
#zookeeper安裝時沒有手動定義內存配置，默認爲系統內存的四分之一
NUM16=$(ps -ef|grep zookeeper|grep -v grep|wc -l)
if [ ${NUM16} -eq 0 ];then
   echo "當前沒有zookeeper進程，${indexName17}: 0"
else
   MEM_TOTAL=$(free -m|grep "Mem"|awk '{print $2}')
   ZK_MEM_MAX=`expr ${MEM_TOTAL} / 4`

   echo "${indexName17}: ${ZK_MEM_MAX}M" 
fi 


echo -e "---------------------------------------------------"
###########Zookeeper分配最大內存 ###########
indexName18="Zookeeper分配最大內存"
#zookeeper安裝時沒有手動定義內存配置，默認爲系統內存的四分之一
NUM17=$(ps -ef|grep zookeeper|grep -v grep|wc -l)
if [ ${NUM17} -eq 0 ];then
   echo "當前沒有zookeeper進程，${indexName18}: 0"
else
   MEM_TOTAL=$(free -m|grep "Mem"|awk '{print $2}')
   ZK_MEM_MAX=`expr ${MEM_TOTAL} / 4`

   echo "${indexName18}: ${ZK_MEM_MAX}M" 
fi 

echo -e "---------------------------------------------------"
###########KAFKA分配最小內存 ###########
indexName19="KAFKA分配最小內存"
NUM18=$(ps -ef|grep kafka|grep -v grep|wc -l)
if [ ${NUM18} -eq 0 ];then
    echo "當前沒有KAFKA進程，${indexName19}: 0"
else
    cd /opt/${appName}/
    for KAFKA in $(find /opt/${appName}/ -name *kafka* -type d)
    do
    NUM19=$(ps -ef|grep ${KAFKA}|grep -v grep|wc -l)
       if [ ${NUM19} -eq 0 ];then
          echo "not this ${KAFKA}" >/dev/null 2>&1
       else
          FILE=${KAFKA}/bin/kafka-server-start.sh 
          KAFKA_MEM_MIN=$(cat ${FILE}|grep "Xmx"|awk -F'"' '{print $2}'|awk '{print $2}'|awk -F"Xms" '{print $2}')
          echo "${indexName19}: ${KAFKA_MEM_MIN}"
       fi
    done
fi


echo -e "---------------------------------------------------"
###########KAFKA分配最大內存 ###########
indexName20="KAFKA分配最大內存"
NUM20=$(ps -ef|grep kafka|grep -v grep|wc -l)
if [ ${NUM20} -eq 0 ];then
    echo "當前沒有KAFKA進程，${indexName20}: 0"
else
    cd /opt/${appName}/
    for KAFKA in $(find /opt/${appName}/ -name *kafka* -type d)
    do
    NUM21=$(ps -ef|grep ${KAFKA}|grep -v grep|wc -l)
       if [ ${NUM21} -eq 0 ];then
          echo "not this ${KAFKA}" >/dev/null 2>&1
       else
          FILE=${KAFKA}/bin/kafka-server-start.sh 
          KAFKA_MEM_MAX=$(cat ${FILE}|grep "Xmx"|awk -F'"' '{print $2}'|awk '{print $1}'|awk -F"Xmx" '{print $2}')
          echo "${indexName20}: ${KAFKA_MEM_MAX}"
       fi
    done
fi

echo -e "---------------------------------------------------"
###########應用系統進程數量 ###########
indexName21="應用系統進程數量"
NUM22=$(ps -eo user,pid,cmd|awk '{if($1=="bxapp")print}'|wc -l)
echo "${indexName21}:"${NUM22}

執行這個腳本：

[root@kevin_test app_process_index]# sh 20_app_process_index_shell.sh
指標類型: 應用進程相關指標
指標採集時間: 1561029337
主機名: kevin_test
應用名稱: bobo
發送數據源主機: /root/100_app_index/app_process_index/20_app_process_index_shell.sh@192.168.10.99

這次採集應用進程相關指標狀況以下:
---------------------------------------------------
應用進程佔內存百分比: 69.7%
---------------------------------------------------
應用進程佔CPU百分比: 13.8%
---------------------------------------------------
應用進程run最長時間: 609-08:33:26 
進程: ../src/redis-sentinel
---------------------------------------------------
應用進程TIME-WAIT鏈接數: 0
---------------------------------------------------
應用進程ESTABLISHED鏈接數: 455
---------------------------------------------------
應用進程FIN_WAIT2鏈接數: 0
---------------------------------------------------
日誌空間使用率: 20%
---------------------------------------------------
應用程序空間使用率: 75%
---------------------------------------------------
應用所在磁盤讀速率（Blk/s）: 2.30
---------------------------------------------------
應用所在磁盤寫速率（Blk/s）: 16.41
---------------------------------------------------
應用所在磁盤TPS: 1.31
---------------------------------------------------
當前沒有Tomcat進程，TOMCAT分配最小內存: 0
---------------------------------------------------
當前沒有Tomcat進程，TOMCAT分配最大內存: 0
---------------------------------------------------
當前沒有Tomcat進程，TOMCAT分配年輕代空間: 0
---------------------------------------------------
當前沒有Tomcat進程，TOMCAT每一個線程的堆棧大小: 0
---------------------------------------------------
當前沒有Tomcat進程，TOMCAT-GC時長佔比: 0
---------------------------------------------------
Zookeeper分配最小內存: 1459M
---------------------------------------------------
Zookeeper分配最大內存: 1459M
---------------------------------------------------
當前沒有KAFKA進程，KAFKA分配最小內存: 0
---------------------------------------------------
當前沒有KAFKA進程，KAFKA分配最大內存: 0
---------------------------------------------------
應用系統進程數量:10

注意：上面是尚未轉化JSON格式的腳本。下面是轉化爲JSON格式後的腳本（實際用的就是下面這個腳本）

[root@kevin_test app_process_index]# cat 20_app_process_index.sh 
#!/bin/bash

source /etc/profile
hostName=$(hostname)
IP=$(ifconfig eth0| sed -nr '2s/(^[^0-9]+)([0-9.]+)+( .*$)/\2/gp')
appName=$(echo ${hostName:3:3})

collectTime=$(date +'%s')
dataSource=$(echo "$0@${IP}")
indexType="app_process"


###########應用進程佔內存百分比###########
ps auxw|grep bxapp|awk '{if (NR>1){print $4}}' > /tmp/mem_list
NUM1=`cat /tmp/mem_list|wc -l`

if [ $NUM1 -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用進程佔內存百分比\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有進程\",
   \"checker\":\"${dataSource}\"
   }+"
else
   MEM_PER_SUM=$(awk '{MEM_PER+=$1}END{print MEM_PER}' /tmp/mem_list)
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用進程佔內存百分比\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${MEM_PER_SUM}%\",
   \"checker\":\"${dataSource}\"
   }+"
fi

###########應用進程佔CPU百分比###########
ps auxw|grep bxapp|awk '{if (NR>1){print $3}}' > /tmp/cpu_list
NUM2=`cat /tmp/cpu_list|wc -l`

if [ $NUM2 -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用進程佔CPU百分比\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有進程\",
   \"checker\":\"${dataSource}\"
   }+"
else
   CPU_PER_SUM=$(awk '{CPU_PER+=$1}END{print CPU_PER}' /tmp/cpu_list)
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用進程佔CPU百分比\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${CPU_PER_SUM}%\",
   \"checker\":\"${dataSource}\"
   }+"
fi

###########應用進程run最長時間及其進程###########
PID=$(ps -eo user,pid,etime|grep bxapp|sort -k 3 -rn|head -1|awk '{print $2}')
RUN_TIME=$(ps -eo user,pid,etime|grep bxapp|sort -k 3 -rn|head -1|awk '{print $3}')
PROCESS=$(ps -ef|grep ${PID}|grep -v grep|awk '{print $8}')

   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用進程run最長時間及其進程\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${RUN_TIME}, ${PROCESS}\",
   \"checker\":\"${dataSource}\"
   }+"

###########應用進程TIME-WAIT鏈接數###########
NUM3=$(ss -a|grep -w "TIME_WAIT" |wc -l)
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用進程TIME-WAIT鏈接數\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${NUM3}\",
   \"checker\":\"${dataSource}\"
   }+"

###########應用進程ESTABLISHED鏈接數###########
NUM4=$(ss -a|grep -w "ESTAB" |wc -l)
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用進程ESTABLISHED鏈接數\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${NUM4}\",
   \"checker\":\"${dataSource}\"
   }+"

###########應用進程FIN_WAIT2鏈接數###########
NUM5=$(ss -a|grep -w "FIN_WAIT2" |wc -l)
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用進程FIN_WAIT2鏈接數\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${NUM5}\",
   \"checker\":\"${dataSource}\"
   }+"

###########日誌空間使用率###########
LOG_SPACE_USER=$(df -h|grep "/var/log"|awk '{print $5}')
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"日誌空間使用率\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${LOG_SPACE_USER}\",
   \"checker\":\"${dataSource}\"
   }+"
###########應用程序空間使用率###########
APP_SPACE_USER=$(df -h|grep "/opt"|awk '{print $5}')
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用程序空間使用率\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${APP_SPACE_USER}\",
   \"checker\":\"${dataSource}\"
   }+"

###########應用所在磁盤讀速率（Blk/s）###########
DISK=$(df -h|grep "/opt/${appName}"|awk '{print $1}')
DEVICE=$(df -h|grep "/opt/${appName}"|awk '{print $1}'|awk -F"/" '{print $3}')
READ=$(iostat ${DISK}|grep -w ${DEVICE}|awk '{print $3}')
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用所在磁盤讀速率（Blk/s）\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${READ}\",
   \"checker\":\"${dataSource}\"
   }+"

###########應用所在磁盤寫速率（Blk/s）###########
DISK=$(df -h|grep "/opt/${appName}"|awk '{print $1}')
DEVICE=$(df -h|grep "/opt/${appName}"|awk '{print $1}'|awk -F"/" '{print $3}')
WRITE=$(iostat ${DISK}|grep -w ${DEVICE}|awk '{print $4}')
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用所在磁盤寫速率（Blk/s）\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${WRITE}\",
   \"checker\":\"${dataSource}\"
   }+"

###########應用所在磁盤TPS###########
DISK=$(df -h|grep "/opt/${appName}"|awk '{print $1}')
DEVICE=$(df -h|grep "/opt/${appName}"|awk '{print $1}'|awk -F"/" '{print $3}')
TPS=$(iostat ${DISK}|grep -w ${DEVICE}|awk '{print $2}')
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用所在磁盤TPS\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${TPS}\",
   \"checker\":\"${dataSource}\"
   }+"

###########TOMCAT分配最小內存 ###########
NUM6=$(ps -ef|grep -w tomcat|grep -v grep|wc -l)
if [ ${NUM6} -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"TOMCAT分配最小內存\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有Tomcat進程\",
   \"checker\":\"${dataSource}\"
   }+"
else
    for TOMCAT in $(find /opt/${appName}/ -name *tomcat* -type d)
    do
      NUM7=$(ps -ef|grep ${TOMCAT}|grep -v grep|wc -l)
      if [ ${NUM7} -eq 0 ];then
        echo "not ${TOMCAT}" >> /dev/null 2>&1
      else
        XMS=$(ps -ef|grep ${TOMCAT}/|grep "\-Xms"|awk -F"-Xms" '{print $2}'|cut -d" " -f1)
          echo -e "{
          \"type\":\"${indexType}\",
          \"desc\":\"TOMCAT分配最小內存\",
          \"@timestamp\":\"${collectTime}\",
          \"appname\":\"${appName}\",
          \"hostname\":\"${hostName}\",
          \"value\":\"${XMS}  (實例名:${TOMCAT})\",
          \"checker\":\"${dataSource}\"
          }+"
      fi
    done
fi

###########TOMCAT分配最大內存 ###########
NUM8=$(ps -ef|grep -w tomcat|grep -v grep|wc -l)
if [ ${NUM8} -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"TOMCAT分配最大內存\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有Tomcat進程\",
   \"checker\":\"${dataSource}\"
   }+"
else
    for TOMCAT in $(find /opt/${appName}/ -name *tomcat* -type d)
    do
      NUM9=$(ps -ef|grep ${TOMCAT}|grep -v grep|wc -l)
      if [ ${NUM9} -eq 0 ];then
        echo "not ${TOMCAT}" >> /dev/null 2>&1
      else
        XMX=$(ps -ef|grep ${TOMCAT}/|grep "\-Xmx"|awk -F"-Xmx" '{print $2}'|cut -d" " -f1)
          echo -e "{
          \"type\":\"${indexType}\",
          \"desc\":\"TOMCAT分配最大內存\",
          \"@timestamp\":\"${collectTime}\",
          \"appname\":\"${appName}\",
          \"hostname\":\"${hostName}\",
          \"value\":\"${XMX}  (實例名:${TOMCAT})\",
          \"checker\":\"${dataSource}\"
          }+"
      fi
    done
fi

###########TOMCAT分配年輕代空間 ###########
NUM10=$(ps -ef|grep -w tomcat|grep -v grep|wc -l)
if [ ${NUM10} -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"TOMCAT分配年輕代空間\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有Tomcat進程\",
   \"checker\":\"${dataSource}\"
   }+"

else
    for TOMCAT in $(find /opt/${appName}/ -name *tomcat* -type d)
    do
      NUM11=$(ps -ef|grep ${TOMCAT}|grep -v grep|wc -l)
      if [ ${NUM11} -eq 0 ];then
        echo "not ${TOMCAT}" >> /dev/null 2>&1
      else
        XMN=$(ps -ef|grep ${TOMCAT}/|grep "\-Xmn"|awk -F"-Xmn" '{print $2}'|cut -d" " -f1)
          echo -e "{
          \"type\":\"${indexType}\",
          \"desc\":\"TOMCAT分配年輕代空間\",
          \"@timestamp\":\"${collectTime}\",
          \"appname\":\"${appName}\",
          \"hostname\":\"${hostName}\",
          \"value\":\"${XMN}  (實例名:${TOMCAT})\",
          \"checker\":\"${dataSource}\"
          }+"
      fi
    done
fi

###########TOMCAT每一個線程的堆棧大小 ###########
NUM12=$(ps -ef|grep -w tomcat|grep -v grep|wc -l)
if [ ${NUM12} -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"TOMCAT每一個線程的堆棧大小\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有Tomcat進程\",
   \"checker\":\"${dataSource}\"
   }+"
else
    for TOMCAT in $(find /opt/${appName}/ -name *tomcat* -type d)
    do
      NUM13=$(ps -ef|grep ${TOMCAT}|grep -v grep|wc -l)
      if [ ${NUM13} -eq 0 ];then
        echo "not ${TOMCAT}" >> /dev/null 2>&1
      else
        XSS=$(ps -ef|grep ${TOMCAT}/|grep "\-Xss"|awk -F"-Xss" '{print $2}'|cut -d" " -f1)
          echo -e "{
          \"type\":\"${indexType}\",
          \"desc\":\"TOMCAT每一個線程的堆棧大小\",
          \"@timestamp\":\"${collectTime}\",
          \"appname\":\"${appName}\",
          \"hostname\":\"${hostName}\",
          \"value\":\"${XSS}  (實例名:${TOMCAT})\",
          \"checker\":\"${dataSource}\"
          }+"
      fi
    done
fi

###########TOMCAT-GC時長佔比 ###########
NUM14=$(ps -ef|grep -w tomcat|grep -v grep|wc -l)
if [ ${NUM14} -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"TOMCAT-GC時長佔比\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有Tomcat進程\",
   \"checker\":\"${dataSource}\"
   }+"
else
    for TOMCAT in $(find /opt/${appName}/ -name *tomcat* -type d)
    do
      NUM15=$(ps -ef|grep ${TOMCAT}|grep -v grep|wc -l)
      if [ ${NUM15} -eq 0 ];then
        echo "not ${TOMCAT}" >> /dev/null 2>&1
      else
        GC=$(ps -ef|grep ${TOMCAT}/|grep "\GCTimeRatio"|awk -F"GCTimeRatio" '{print $2}'|cut -d" " -f1|awk -F"=" '{print $2}')
          echo -e "{
          \"type\":\"${indexType}\",
          \"desc\":\"TOMCAT-GC時長佔比\",
          \"@timestamp\":\"${collectTime}\",
          \"appname\":\"${appName}\",
          \"hostname\":\"${hostName}\",
          \"value\":\"${GC}  (實例名:${TOMCAT})\",
          \"checker\":\"${dataSource}\"
          }+"
      fi
    done
fi

###########Zookeeper分配最小內存###########
#zookeeper安裝時沒有手動定義內存配置，默認爲系統內存的四分之一
NUM16=$(ps -ef|grep zookeeper|grep -v grep|wc -l)
if [ ${NUM16} -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"Zookeeper分配最小內存\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有Zookeeper進程\",
   \"checker\":\"${dataSource}\"
   }+"
else
   MEM_TOTAL=$(free -m|grep "Mem"|awk '{print $2}')
   ZK_MEM_MIN=`expr ${MEM_TOTAL} / 4`
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"Zookeeper分配最小內存\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${ZK_MEM_MIN}M\",
   \"checker\":\"${dataSource}\"
   }+"
fi

###########Zookeeper分配最大內存###########
#zookeeper安裝時沒有手動定義內存配置，默認爲系統內存的四分之一
NUM17=$(ps -ef|grep zookeeper|grep -v grep|wc -l)
if [ ${NUM17} -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"Zookeeper分配最大內存\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有Zookeeper進程\",
   \"checker\":\"${dataSource}\"
   }+"
else
   MEM_TOTAL=$(free -m|grep "Mem"|awk '{print $2}')
   ZK_MEM_MAX=`expr ${MEM_TOTAL} / 4`
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"Zookeeper分配最大內存\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${ZK_MEM_MAX}M\",
   \"checker\":\"${dataSource}\"
   }+"
fi

###########KAFKA分配最小內存 ###########
NUM18=$(ps -ef|grep kafka|grep -v grep|wc -l)
if [ ${NUM18} -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"KAFKA分配最小內存\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有KAFKA進程\",
   \"checker\":\"${dataSource}\"
   }+"
else
    cd /opt/${appName}/
    for KAFKA in $(find /opt/${appName}/ -name *kafka* -type d)
    do
    NUM19=$(ps -ef|grep ${KAFKA}|grep -v grep|wc -l)
       if [ ${NUM19} -eq 0 ];then
          echo "not this ${KAFKA}" >/dev/null 2>&1
       else
          FILE=${KAFKA}/bin/kafka-server-start.sh 
          KAFKA_MEM_MIN=$(cat ${FILE}|grep "Xmx"|awk -F'"' '{print $2}'|awk '{print $2}'|awk -F"Xms" '{print $2}')
          echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"KAFKA分配最小內存\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${KAFKA_MEM_MIN}\",
   \"checker\":\"${dataSource}\"
   }+"
       fi
    done
fi

###########KAFKA分配最大內存 ###########
NUM20=$(ps -ef|grep kafka|grep -v grep|wc -l)
if [ ${NUM20} -eq 0 ];then
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"KAFKA分配最大內存\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"當前沒有KAFKA進程\",
   \"checker\":\"${dataSource}\"
   }+"
else
    cd /opt/${appName}/
    for KAFKA in $(find /opt/${appName}/ -name *kafka* -type d)
    do
    NUM21=$(ps -ef|grep ${KAFKA}|grep -v grep|wc -l)
       if [ ${NUM21} -eq 0 ];then
          echo "not this ${KAFKA}" >/dev/null 2>&1
       else
          FILE=${KAFKA}/bin/kafka-server-start.sh 
          KAFKA_MEM_MAX=$(cat ${FILE}|grep "Xmx"|awk -F'"' '{print $2}'|awk '{print $1}'|awk -F"Xmx" '{print $2}')
          echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"KAFKA分配最大內存\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${KAFKA_MEM_MAX}\",
   \"checker\":\"${dataSource}\"
   }+"
       fi
    done
fi

###########應用系統進程數量 ###########
NUM22=$(ps -eo user,pid,cmd|awk '{if($1=="bxapp")print}'|wc -l)
   echo -e "{
   \"type\":\"${indexType}\",
   \"desc\":\"應用系統進程數量\",
   \"@timestamp\":\"${collectTime}\",
   \"appname\":\"${appName}\",
   \"hostname\":\"${hostName}\",
   \"value\":\"${NUM22}\",
   \"checker\":\"${dataSource}\"
   }"

執行這個腳本（JSON字段跟上面ES入倉要求的字段匹配）：

[root@kevin_test app_process_index]# sh 20_app_process_index.sh
{
   "type":"app_process",
   "desc":"應用進程佔內存百分比",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"69.7%",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"應用進程佔CPU百分比",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"13.8%",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"應用進程run最長時間及其進程",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"609-08:36:42, ../src/redis-sentinel",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"應用進程TIME-WAIT鏈接數",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"0",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"應用進程ESTABLISHED鏈接數",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"454",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"應用進程FIN_WAIT2鏈接數",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"0",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"日誌空間使用率",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"20%",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"應用程序空間使用率",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"75%",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"應用所在磁盤讀速率（Blk/s）",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"2.30",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"應用所在磁盤寫速率（Blk/s）",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"16.41",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"應用所在磁盤TPS",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"1.31",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"TOMCAT分配最小內存",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"TOMCAT分配最大內存",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"TOMCAT分配年輕代空間",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"TOMCAT每一個線程的堆棧大小",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"TOMCAT-GC時長佔比",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"Zookeeper分配最小內存",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"1459M",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"Zookeeper分配最大內存",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"1459M",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"KAFKA分配最小內存",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有KAFKA進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"KAFKA分配最大內存",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有KAFKA進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }+
{
   "type":"app_process",
   "desc":"應用系統進程數量",
   "@timestamp":"1561029533",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"10",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

2、而後使用python調用shell腳本，調取shell腳本的執行結果
下面三個腳本均可以使用，區別在於：
第一個腳本：默認就是在當前路徑下調用shell腳本，shell和python兩個腳本是放在統一目錄下的
第二個腳本和第三個腳本：先獲取當前腳本所在目錄路徑，而後在python腳本里調用shell腳本時，使用的是shell腳本全路徑

這裏推薦使用第三個腳本，可是注意：
使用第二或第三個腳本時，shell腳本里的dataSource要寫成：dataSource=$(echo "$0@${IP}")
而使用第一個腳本時，shell腳本里的dataSource要寫成：dataSource=$(echo "`pwd`/$0@${IP}")

區別就是：
第一或第二個腳本，在輸出的JSON格式的結果中，checker不會帶上`pwd`
第三個腳本，在輸出的JSON格式的結果中，checker會帶上`pwd`，因此就不須要在shell腳本里再加上`pwd`，不然路徑就多了~

第一個腳本==================================================================
[root@kevin_test app_process_index]# cat 20_app_process_index.py
#!/usr/bin/python
# -*- coding: UTF-8 -*-
import os
import json

#默認就在當前路徑下執行20_app_process_index.sh
json_set = os.popen("sh 20_app_process_index.sh").read().split('+')
for json_str in json_set:
    print json_str

第二個腳本==================================================================
[root@kevin_test app_process_index]# cat 20_app_process_index.py1 
#!/usr/bin/python
# -*- coding: UTF-8 -*-
import os
import json

#獲取當前路徑，調用shell腳本, 執行時跟全路徑
now_path=os.path.abspath('.')
index_script="sh %s/20_app_process_index.sh" % (now_path)

json_set = os.popen(index_script).read().split('+')
for json_str in json_set:
    print json_str

第三個腳本================================================================
[root@kevin_test app_process_index]# cat 20_app_process_index.py2
#!/usr/bin/python
# -*- coding: UTF-8 -*-
import os
import json

#獲取當前路徑，調用shell腳本, 執行時跟全路徑
now_path=os.path.abspath(__file__)
now_path=os.path.dirname(now_path)
index_script="sh %s/20_app_process_index.sh" % (now_path)

json_set = os.popen(index_script).read().split('+')
for json_str in json_set:
    print json_str

來看看幾個腳本結果：
以下第一個腳本的"checker" 字段，因爲shell腳本里的"dataSource"沒有帶"pwd"，因此"checker"字段中的腳本路徑不全

[root@kevin_test app_process_index]# python 20_app_process_index.py
{
   "type":"app_process",
   "desc":"應用進程佔內存百分比",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"69.7%",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用進程佔CPU百分比",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"13.8%",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用進程run最長時間及其進程",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"609-08:46:50, ../src/redis-sentinel",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用進程TIME-WAIT鏈接數",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"0",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用進程ESTABLISHED鏈接數",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"455",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用進程FIN_WAIT2鏈接數",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"0",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"日誌空間使用率",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"20%",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用程序空間使用率",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"75%",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用所在磁盤讀速率（Blk/s）",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"2.30",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用所在磁盤寫速率（Blk/s）",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"16.41",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用所在磁盤TPS",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"1.31",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"TOMCAT分配最小內存",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"TOMCAT分配最大內存",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"TOMCAT分配年輕代空間",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"TOMCAT每一個線程的堆棧大小",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"TOMCAT-GC時長佔比",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"Zookeeper分配最小內存",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"1459M",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"Zookeeper分配最大內存",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"1459M",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"KAFKA分配最小內存",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有KAFKA進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"KAFKA分配最大內存",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有KAFKA進程",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用系統進程數量",
   "@timestamp":"1561030141",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"11",
   "checker":"20_app_process_index.sh@192.168.10.99"
   }

第二或第三個腳本執行結果同樣，"checker"字段中的腳本路徑是完整路徑

[root@kevin_test app_process_index]# python 20_app_process_index.py 
{
   "type":"app_process",
   "desc":"應用進程佔內存百分比",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"70.8%",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用進程佔CPU百分比",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"144.8%",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用進程run最長時間及其進程",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"609-08:51:53, ../src/redis-sentinel",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用進程TIME-WAIT鏈接數",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"0",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用進程ESTABLISHED鏈接數",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"456",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用進程FIN_WAIT2鏈接數",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"0",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"日誌空間使用率",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"20%",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用程序空間使用率",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"75%",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用所在磁盤讀速率（Blk/s）",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"2.30",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用所在磁盤寫速率（Blk/s）",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"16.41",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用所在磁盤TPS",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"1.31",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"TOMCAT分配最小內存",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"TOMCAT分配最大內存",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"TOMCAT分配年輕代空間",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"TOMCAT每一個線程的堆棧大小",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"TOMCAT-GC時長佔比",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有Tomcat進程",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"Zookeeper分配最小內存",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"1459M",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"Zookeeper分配最大內存",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"1459M",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"KAFKA分配最小內存",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有KAFKA進程",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"KAFKA分配最大內存",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"當前沒有KAFKA進程",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

{
   "type":"app_process",
   "desc":"應用系統進程數量",
   "@timestamp":"1561030444",
   "appname":"bobo",
   "hostname":"kevin_test",
   "value":"10",
   "checker":"/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99"
   }

最後將上面的第三個腳本整合到"20_app_process_index_es.py"腳本里，做爲ES入倉請求的腳本

[root@kevin_test app_process_index]# cat 20_app_process_index_es.py 
#!/usr/bin/python
# -*- coding: UTF-8 -*-

import urllib2  
import hashlib
import json
import os

now_path=os.path.abspath(__file__)
now_path=os.path.dirname(now_path)
index_script="sh %s/20_app_process_index.sh" % (now_path)

json_set = os.popen(index_script).read().split('+')

app_index = []
for json_str in json_set:
    app_index.append(json.loads(json_str))

url = 'http://192.168.10.10:10222/haha/heiheiAPI/test_1_kafka'
data = {
    'token': 'bd1df2b11111111112222222222222222333333333333333333yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy',
    'data': app_index
}

headers = {'Content-Type': 'application/json'}
req = urllib2.Request(url, json.dumps(data), headers = headers)
res = urllib2.urlopen(req).read()

print res
#print app_index

執行這個請求腳本

[root@kevin_test app_process_index]# python 20_app_process_index_es.py
{"retCode": 200}

將腳本中的最後一行註釋打開（即"#print app_index"），測試下入倉的數據：
[root@kevin_test app_process_index]# python 20_app_process_index_es.py
{"retCode": 200}
[{u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': u'kevin_test', u'value': u'69.7%', u'checker': 
u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': u'app_process', u'desc': u'
\u5e94\u7528\u8fdb\u7a0b\u5360\u5185\u5b58\u767e\u5206\u6bd4'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', 
u'hostname': u'kevin_test', u'value': u'13.8%', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', 
u'type': u'app_process', u'desc': u'\u5e94\u7528\u8fdb\u7a0b\u5360CPU\u767e\u5206\u6bd4'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', 
u'hostname': u'kevin_test', u'value': u'609-08:54:24, ../src/redis-sentinel', u'checker': 
u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': u'app_process', u'desc': 
u'\u5e94\u7528\u8fdb\u7a0brun\u6700\u957f\u65f6\u95f4\u53ca\u5176\u8fdb\u7a0b'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': 
u'kevin_test', u'value': u'0', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': u'app_process', 
u'desc': u'\u5e94\u7528\u8fdb\u7a0bTIME-WAIT\u8fde\u63a5\u6570'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': u'kevin_test', 
u'value': u'458', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': u'app_process', u'desc': 
u'\u5e94\u7528\u8fdb\u7a0bESTABLISHED\u8fde\u63a5\u6570'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': u'kevin_test', 
u'value': u'0', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': u'app_process', u'desc': 
u'\u5e94\u7528\u8fdb\u7a0bFIN_WAIT2\u8fde\u63a5\u6570'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': u'kevin_test', 
u'value': u'20%', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': u'app_process', 
u'desc': u'\u65e5\u5fd7\u7a7a\u95f4\u4f7f\u7528\u7387'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': u'kevin_test', 
u'value': u'75%', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': u'app_process', u'desc': 
u'\u5e94\u7528\u7a0b\u5e8f\u7a7a\u95f4\u4f7f\u7528\u7387'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': u'kevin_test', 
u'value': u'2.30', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': u'app_process', u'desc': 
u'\u5e94\u7528\u6240\u5728\u78c1\u76d8\u8bfb\u901f\u7387\uff08Blk/s\uff09'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': 
u'kevin_test', u'value': u'16.41', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': u'app_process', 
u'desc': u'\u5e94\u7528\u6240\u5728\u78c1\u76d8\u5199\u901f\u7387\uff08Blk/s\uff09'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': 
u'kevin_test', u'value': u'1.31', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': u'app_process', 
u'desc': u'\u5e94\u7528\u6240\u5728\u78c1\u76d8TPS'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': u'kevin_test', u'value': 
u'\u5f53\u524d\u6ca1\u6709Tomcat\u8fdb\u7a0b', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', u'type': 
u'app_process', u'desc': u'TOMCAT\u5206\u914d\u6700\u5c0f\u5185\u5b58'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': u'kevin_test', 
u'value': u'\u5f53\u524d\u6ca1\u6709Tomcat\u8fdb\u7a0b', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', 
u'type': u'app_process', u'desc': u'TOMCAT\u5206\u914d\u6700\u5927\u5185\u5b58'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': 
u'kevin_test', u'value': u'\u5f53\u524d\u6ca1\u6709Tomcat\u8fdb\u7a0b', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99', 
u'type': u'app_process', u'desc': u'TOMCAT\u5206\u914d\u5e74\u8f7b\u4ee3\u7a7a\u95f4'}, {u'appname': u'bobo', u'@timestamp': u'1561030595', u'hostname': 
u'kevin_test', u'value': u'\u5f53\u524d\u6ca1\u6709Tomcat\u8fdb\u7a0b', u'checker': u'/root/100_app_index/app_process_index/20_app_process_index.sh@192.168.10.99',

======================================================
上面涉及到python的當前路徑調用問題（即上面討論的第1、2、三的python腳本）

這裏對python獲取當前目錄路徑和文件的方法作個記錄：
1、獲取當前路徑

1）使用sys.argv[0]
import sys
print sys.argv[0]
#輸出
#本地路徑
     
2）os模塊
import  os
print os.getcwd()                     #獲取當前工做目錄路徑
print os.path.abspath('.')            #獲取當前工做目錄路徑
print os.path.abspath('test.txt')     #獲取當前目錄文件下的工做目錄路徑
print os.path.abspath('..')           #獲取當前工做的父目錄 ！注意是父目錄路徑
print os.path.abspath(os.curdir)      #獲取當前工做目錄路徑

3）改變當前目錄
使用: os.chdir(path)。

4）組合路徑返回
os.path.join('file1','file2','file3')
合併獲得路徑 file1/file2/file3

2、得到當前目錄下全部文件

1）os.walk() 用於在目錄樹種遊走輸出目錄中的文件名字，向上或下；

語法
os.walk(top[, topdown=True[, onerror=None[, followlinks=False]]])

參數：
top -- 根目錄下的每個文件夾(包含它本身), 產生3-元組 (dirpath, dirnames, filenames)【文件夾路徑, 
       文件夾名字, 文件名】。

topdown --可選，爲True或者沒有指定, 一個目錄的的3-元組將比它的任何子文件夾的3-元組先產生 (目錄自上而下)。
          若是topdown爲 False, 一個目錄的3-元組將比它的任何子文件夾的3-元組後產生 (目錄自下而上)。

onerror -- 可選，是一個函數; 它調用時有一個參數, 一個OSError實例。報告這錯誤後，繼續walk,或者拋出exception終止walk。

followlinks -- 設置爲 true，則經過軟連接訪問目錄。

2）
import os
root = os.getcwd()

def file_name(file_dir):
    for root, dirs, files in os.walk(file_dir):
        print "-----------"
        print root    #os.walk()所在目錄
        print dirs    #os.walk()所在目錄的全部目錄名
        print files   #os.walk()所在目錄的全部非目錄文件名
        print " "

file_name(root)