openresty使用火焰圖排查性能問題

本文主要是講解如何在ubuntu安裝最新Systemtap.以及繪製火焰圖

安裝調試鏡像

# 導入 GPG key
# 16.04 and higher
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622

#older distributions
#sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01 

# 設置源
codename=$(lsb_release -c | awk  '{print $2}')
sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename}      main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

# 更新
sudo apt-get update

# 安裝調試鏡像
sudo apt-get install -y linux-image-$(uname -r)-dbgsym複製代碼

安裝最新版 systemtap

$ sudo apt-get install -y build-essential zlib1g-dev elfutils libdw-dev gettext

# https://sourceware.org/elfutils/ftp/?C=M;O=D
$ wget https://sourceware.org/elfutils/ftp/0.170/elfutils-0.170.tar.bz2
$ tar xf elfutils-0.170.tar.bz2

# https://sourceware.org/systemtap/ftp/releases/?C=M;O=D
$ wget https://sourceware.org/systemtap/ftp/releases/systemtap-3.1.tar.gz
$ tar zxf systemtap-3.1.tar.gz

$ cd systemtap-3.1

$ ./configure --prefix=/opt/stap --disable-docs \
    --disable-publican --disable-refdocs CFLAGS="-g -O2" \
    --with-elfutils=../elfutils-0.170

$ make -j$(getconf _NPROCESSORS_ONLN) && sudo make install

# export STAP_HOME=/opt/stap/
# export PATH=$STAP_HOME:$PATH

# stap -V

Systemtap translator/driver (version 3.1/0.170, non-git sources)
Copyright (C) 2005-2017 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.18 ... 4.10-rc8
enabled features: PYTHON2 PYTHON3 LIBXML2 NLS READLINE複製代碼

測試是否生效

# stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
Pass 1: parsed user script and 465 library scripts using 77388virt/46648res/5256shr/41840data kb, in 80usr/30sys/333real ms.
Pass 2: analyzed script: 1 probe, 1 function, 7 embeds, 0 globals using 260440virt/231204res/6736shr/224892data kb, in 1680usr/350sys/7050real ms.
Pass 3: translated to C into "/tmp/stap8Lyxq5/stap_e1c4934460a3e749f6deefe95dd50015_2729_src.c" using 260440virt/231404res/6936shr/224892data kb, in 10usr/0sys/5real ms.
Pass 4: compiled C into "stap_e1c4934460a3e749f6deefe95dd50015_2729.ko" in 5260usr/420sys/7185real ms.
Pass 5: starting run.
read performed
Pass 5: run completed in 0usr/20sys/486real ms.複製代碼

繪製火焰圖

下載各工具包

# git clone https://github.com/openresty/stapxx.git --depth=1 /opt/stapxx
# export STAP_PLUS_HOME=/opt/stapxx
# export PATH=$STAP_PLUS_HOME:$STAP_PLUS_HOME/samples:$PATH

# stap++ -e 'probe begin { println("hello") exit() }'

hello


# git clone https://github.com/openresty/openresty-systemtap-toolkit.git --depth=1 /opt/openresty-systemtap-toolkit

# git clone https://github.com/brendangregg/FlameGraph.git --depth=1 /opt/FlameGraph複製代碼

繪製火焰圖

# /opt/stapxx/samples/lj-lua-stacks.sxx --arg time=120 --skip-badvars -x `ps --no-headers -fC nginx|awk '/worker/ {print$2}'| shuf | head -n 1` > /tmp/tmp.bt （-x 是要抓的進程的 pid， 探測結果輸出到 tmp.bt）
# /opt/openresty-systemtap-toolkit/fix-lua-bt tmp.bt > /tmp/flame.bt (處理 lj-lua-stacks.sxx 的輸出，使其可讀性更佳)
# /opt/FlameGraph/stackcollapse-stap.pl /tmp/flame.bt > /tmp/flame.cbt
# /opt/FlameGraph/flamegraph.pl /tmp/flame.cbt > /tmp/flame.svg複製代碼

爲了突出效果，建議在運行stap++的時候，使用壓測工具，以便採集足夠的樣本

# ab -n 10000 -c 100 -k http://localhost/複製代碼

用瀏覽器打開 /tmp/flame.svg 儘可能用 chrome firefox別用國產亂七八糟瀏覽器.

openresty/stapxx

## 使用 stap++ --args xx.sxx查看具體參數

# stap++ --args /opt/stapxx/samples/lj-lua-stacks.sxx
    --arg depth=VALUE (default: 30)
    --arg detailed=VALUE (default: 0)
    --arg limit=VALUE (default: 1000)
    --arg min=VALUE (default: 2)
    --arg nointerp=VALUE (default: )
    --arg nojit=VALUE (default: )
    --arg probe=VALUE (default: timer.profile)
    --arg time=VALUE (default: )複製代碼

具體腳本用法，參見 openresty/stapxx#samples

openresty/openresty-systemtap-toolkit

這一系列腳本頗有用，好比能夠用來看共享內存大小，使用狀況，內存泄露狀況，哪裏泄露的，不過部分腳本須要在編譯的時候，開啓調試或者增長依賴。具體參見readme.

若是要使用ngx-leaked-pools須要用到dtrace

$ apt install systemtap-sdt-dev -y
$ ./configure --prefix=/etc/openresty \
  --with-dtrace-probes複製代碼

若是要用到ngx-pcrejit須要在編譯openresty時增長--with-pcre-opt=-g

從新編譯並make && make install 後會將原有的二進制文件重命名爲${openresty_home}/nginx/sbin/nginx.old，並建立一個${openresty_home}/nginx/sbin/nginx(新版)

$ kill -USR2 `cat /var/run/nginx.pid`複製代碼

經過ps -fC nginx或者ps -fC openresty查看新版本是否成功啓動

若是成功啓動，此時新舊版本同時接受請求

經過

$ kill -QUIT `cat /var/run/nginx.pid.oldbin`複製代碼

平滑殺掉舊版

更多資料請自行谷歌、百度。或者參閱 下面的參考鏈接

參考鏈接

• 白話火焰圖-火丁筆記
• Build Systemtap-openresty官方文檔
• 火焰圖-openresty最佳實踐
• Systemtap - ubuntu wiki
• openresty/stapxx
• openresty/openresty-systemtap-toolkit
• brendangregg/FlameGraph
• 虢兆坤- Nginx 的啓動、中止、平滑重啓、信號控制和平滑升級

博客 anjia.ml/2017/09/12/…
掘金 juejin.im/post/59ce27…
簡書 www.jianshu.com/p/008fde883…