SystemTap是一個Linux很是有用的調試(跟蹤/探測)工具,經常使用於Linux 內核或者應用程序的信息採集,好比:獲取一個函數裏面運行時的變 量、調用堆棧,甚至能夠直接修改變量的值,對診斷性能或功能問題非 常有幫助。SystemTap提供很是簡單的命令行接口和很簡潔的腳本語 言,以及很是豐富的tapset和例子。
定位(內核)函數位置 查看函數被調用時的調用堆棧、局部變量、參數 查看函數指針變量實際指的是哪一個函數 查看代碼的執行軌跡(哪些行被執行了) 查看內核或者進程的執行流程 調試內存泄露或者內存重複釋放 統計函數調用次數 ......
在網上找了個原理圖:javascript
SystemTap的處理流程有5個步驟:解析script文件(parse)、細化(elaborate)、script文件翻譯成C語言代碼(translate)、編譯C語言代碼(生成內核模塊)(build)、加載內核模塊(run)php
SystemTap依賴的package:
elfutils、gcc、kernel-devel、kernel-debuginfo
若是調用用戶態進程,還須要該程序有調試符號,不然沒法調試。
推薦使用最新穩定版的SystemTap,目前最新穩定版爲:systemtap-2.9.tar.gzhtml
5.1 stap命令java
stap [OPTIONS] FILENAME [ARGUMENTS] stap [OPTIONS] - [ARGUMENTS] stap [OPTIONS] –e SCRIPT [ARGUMENTS] 比較經常使用和有用的參數: -e SCRIPT Run given script. -l PROBE List matching probes. -L PROBE List matching probes and local variables. -g guru mode -D NM=VAL emit macro definition into generated C code -o FILE send script output to file, instead of stdout. -x PID sets target() to PID
Hello World:linux
root@j9 ~/stp# cat hello-world.stp probe begin { print("===Hello World===\n") } probe end { print("===GunLe===\n") } root@j9 ~/stp# stap hello-world.stp ===Hello World=== ^C===GunLe=== root@j9 ~/stp# stap -e 'probe begin { printf("Hello World!\n") exit() }' Hello World! root@j9 ~/stp#
5.2 staprun命令nginx
staprun [OPTIONS] MODULE [MODULE-OPTIONS]
stap命令與staprun命令的區別在於:
stap命令的操做對象是stp文件或script命令等,而staprun命令的操做對象是編譯生成的內核模塊。docker
6.1 probeapi
「probe」 <=> 「探測」, 是SystemTap進行具體地收集數據的關鍵字。數組
「probe point」 是probe動做的時機,也稱探測點。也就是probe程序監視的某事件點,一旦偵測的事件觸發了,則probe將今後處插入內核或者用戶進程中。
「probe handle」 是當probe插入內核或者用戶進程後所作的具體動做。bash
probe用法:
probe probe-point { statement }
在Hello World例子中begin和end就是probe-point, statement就是該探測點的處理邏輯,在Hello World例子中statement只有一行print,statement能夠是複雜的代碼塊。
探測點語法:
kernel.function(PATTERN) kernel.function(PATTERN).call kernel.function(PATTERN).return kernel.function(PATTERN).return.maxactive(VALUE) kernel.function(PATTERN).inline kernel.function(PATTERN).label(LPATTERN) module(MPATTERN).function(PATTERN) module(MPATTERN).function(PATTERN).call module(MPATTERN).function(PATTERN).return.maxactive(VALUE) module(MPATTERN).function(PATTERN).inline kernel.statement(PATTERN) kernel.statement(ADDRESS).absolute module(MPATTERN).statement(PATTERN) process(PROCESSPATH).function(PATTERN) process(PROCESSPATH).function(PATTERN).call process(PROCESSPATH).function(PATTERN).return process(PROCESSPATH).function(PATTERN).inline process(PROCESSPATH).statement(PATTERN)
PATTERN語法爲:
func[@file] func@file:linenumber
例如:
kernel.function("*init*") module("ext3").function("*") kernel.statement("*@kernel/time.c:296") process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request")
在return探測點能夠用$return獲取該函數的返回值。
inline函數沒法安裝.return探測點,也沒法用$return獲取其返回值。
6.2 基本語法
SystemTap腳本語法比較簡單,與C語言相似,只是每一行結尾";"是可選的。主要語句以下:
if/else、while、for/foreach、break/continue、return、next、delete、try/catch
其中:
next:主要在probe探測點邏輯處理中使用,調用此語句時,馬上從調用函數中退出。不一樣於exit()的是,next只是退出當前的調用函數,而此SystemTap並無終了,但exit()則會終止SystemTap。
6.2.1 變量
不須要明確聲明變量類型,腳本語言會根據函數參數等自動判斷變量是什麼類型的。
局部變量:在聲明的probe和block(」{ }「範圍內的部分)內有效。
全局變量:用」global「聲明的變量,在此SystemTap的整個動做過程當中都有效。全局變量的聲明位置沒有具體要求。須要注意的是,全局變量默認有鎖保護,使用過多會有性能損失,若是用全局變量保存指針,可能出現指針所指的內容被進程修改,在探測點中拿不到真正的數據。
獲取進程中的變量(全局變量、局部變量、參數)直接在變量名前面加$便可(後面會有例子)
6.2.2 註釋
# ...... : Shell語言風格 //...... : C++語言風格 /*......*/ : C語言風格
6.2.3 操做符
比較運算符、算數運算符基本上與C語言同樣,須要特別指出的是:
(1)、.操做符:鏈接兩個字符串,相似於php;
(2)、=~和!~:正則匹配和正則不匹配;
6.2.4 函數
函數定義例子:
function indent:string (delta:long){ return _generic_indent(-1, "", delta) } function _generic_indent (idx, desc, delta) { ts = __indent_timestamp () if (! _indent_counters[idx]) _indent_timestamps[idx] = ts depth = _generic_indent_depth(idx, delta) return sprintf("%6d (%d:%d) %s:%-*s", (ts - _indent_timestamps[idx]), depth, delta, desc, depth, "") } function strlen:long(s:string) %{ STAP_RETURN(strlen(STAP_ARG_s)); %}
官方有不少頗有用的函數,詳情請參考:https://sourceware.org/system...
以及在本機安裝了SystemTap以後在目錄/usr/local/share/systemtap/tapset/下也能夠看具體函數的實現以及一些奇特的用法。
7.1 定位函數位置
在一個大型項目中找出函數在哪裏定義有時頗有用,特別是一些比較難找出在哪裏定義的函數,好比內核或者glibc中的某個函數想要看其實現時,首先得找出其在哪一個文件的哪一行定義,用SystemTap一行命令就能夠搞定。
好比要看printf在glibc中哪裏定義的:
root@j9 ~# stap -l 'process("/lib/x86_64-linux-gnu/libc.so.6").function("printf")' process("/lib/x86_64-linux-gnu/libc-2.15.so").function("__printf@/build/buildd/eglibc-2.15/stdio-common/printf.c:29")
能夠看出printf是在printf.c第29行定義的。
再好比要看內核中recv系統的調用是在哪裏定義的:
root@j9 ~# stap -l 'kernel.function("sys_recv")' kernel.function("sys_recv@/build/buildd/linux-lts-trusty-3.13.0/net/socket.c:1868")
能夠看出recv是在socket.c第1868行定義的。
甚至能夠*號來模糊查找:
root@j9 ~# stap -l 'kernel.function("*recv")' kernel.function("__audit_mq_sendrecv@/build/buildd/linux-lts-trusty-3.13.0/kernel/auditsc.c:2062") kernel.function("audit_mq_sendrecv@/build/buildd/linux-lts-trusty-3.13.0/include/linux/audit.h:263") kernel.function("compat_sys_recv@/build/buildd/linux-lts-trusty-3.13.0/net/compat.c:762") kernel.function("i2c_master_recv@/build/buildd/linux-lts-trusty-3.13.0/drivers/i2c/i2c-core.c:1827") kernel.function("ip_cmsg_recv@/build/buildd/linux-lts-trusty-3.13.0/net/ipv4/ip_sockglue.c:147") kernel.function("kgdb_tty_recv@/build/buildd/linux-lts-trusty-3.13.0/drivers/tty/serial/kgdb_nmi.c:109") kernel.function("ppp_do_recv@/build/buildd/linux-lts-trusty-3.13.0/drivers/net/ppp/ppp_generic.c:1617") kernel.function("scm_recv@/build/buildd/linux-lts-trusty-3.13.0/include/net/scm.h:109") kernel.function("sys_recv@/build/buildd/linux-lts-trusty-3.13.0/net/socket.c:1868") kernel.function("tcp_event_data_recv@/build/buildd/linux-lts-trusty-3.13.0/net/ipv4/tcp_input.c:615") kernel.function("tcp_splice_data_recv@/build/buildd/linux-lts-trusty-3.13.0/net/ipv4/tcp.c:637") kernel.function("tpm_tis_recv@/build/buildd/linux-lts-trusty-3.13.0/drivers/char/tpm/tpm_tis.c:231") kernel.function("try_fill_recv@/build/buildd/linux-lts-trusty-3.13.0/drivers/net/virtio_net.c:615")
同理,也能夠用來定位用戶進程的函數位置:
好比tengine的文件ngx_shmem.c裏面爲了兼容各個操做系統而實現了三個版本的ngx_shm_alloc,用#if (NGX_HAVE_MAP_ANON)、#elif (NGX_HAVE_MAP_DEVZERO)、#elif (NGX_HAVE_SYSVSHM)、#endif來作條件編譯,那怎麼知道編譯出來的是哪一個版本呢,用SystemTap的話就很簡單了,不然要去grep一下這幾宏有沒有定義才知道了。
[root@cache4 tengine]# stap -l 'process("/home/admin/tengine/bin/nginx").function("ngx_shm_alloc")' process("/home/admin/tengine/bin/nginx").function("ngx_shm_alloc@src/os/unix/ngx_shmem.c:15")
7.2 查看可用探測點以及該探測點上可用的變量
在一些探測點上能獲取的變量比較有限,這是由於這些變量可能已經被編譯器優化掉了,優化掉的變量就獲取不到了。通常先用-L參數來看看有哪些變量能夠直接使用:
[root@cache4 tengine]# stap -L 'process("/home/admin/tengine/bin/nginx").function("ngx_shm_alloc")' process("/home/admin/tengine/bin/nginx").function("ngx_shm_alloc@src/os/unix/ngx_shmem.c:15") $shm:ngx_shm_t*
可見在該探測點上能夠直接使用$shm這個變量,其類型是ngx_shm_t*。
statement探測點也相似:
[root@cache4 tengine]# stap -L 'process("/home/admin/tengine/bin/nginx").statement("ngx_pcalloc@src/core/ngx_palloc.c:*")' process("/home/admin/tengine/bin/nginx").statement("ngx_pcalloc@src/core/ngx_palloc.c:395") $pool:ngx_pool_t* $size:size_t process("/home/admin/tengine/bin/nginx").statement("ngx_pcalloc@src/core/ngx_palloc.c:398") $pool:ngx_pool_t* $size:size_t process("/home/admin/tengine/bin/nginx").statement("ngx_pcalloc@src/core/ngx_palloc.c:399") $size:size_t process("/home/admin/tengine/bin/nginx").statement("ngx_pcalloc@src/core/ngx_palloc.c:404") $size:size_t $p:void*
7.3 輸出調用堆棧
用戶態探測點堆棧:print_ubacktrace()、sprint_ubacktrace()
內核態探測點堆棧:print_backtrace()、sprint_backtrace()
不帶s和帶s的區別是前者直接輸出,後者是返回堆棧字符串。
這幾個函數很是有用,在排查問題時能夠根據一些特定條件來過濾函數被執行時是怎麼調用進來的,好比排查tengine返回5xx時的調用堆棧是怎樣的:
#cat debug_tengine_5xx.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_http_finalize_request").call { if ($rc >= 500) { printf("rc: %d\n", $rc) print_ubacktrace() } } #stap debug_tengine_5xx.stp rc: 502 0x49af2e : ngx_http_finalize_request+0xe/0x480 [/home/admin/tengine/bin/nginx] 0x543305 : ngx_http_video_flv_send_rest+0xf5/0x380 [/home/admin/tengine/bin/nginx] 0x543187 : ngx_http_video_finalize_request+0x57/0xe0 [/home/admin/tengine/bin/nginx] 0x49828f : ngx_http_terminate_request+0x4f/0xc0 [/home/admin/tengine/bin/nginx] 0x49b760 : ngx_http_test_reading+0x50/0x130 [/home/admin/tengine/bin/nginx] 0x49779f : ngx_http_request_handler+0x1f/0x40 [/home/admin/tengine/bin/nginx] 0x47ea8f : ngx_epoll_process_events+0x2df/0x330 [/home/admin/tengine/bin/nginx] 0x4753f9 : ngx_process_events_and_timers+0x69/0x1c0 [/home/admin/tengine/bin/nginx] 0x47d4d8 : ngx_worker_process_cycle+0x138/0x260 [/home/admin/tengine/bin/nginx] 0x47a38a : ngx_spawn_process+0x1ca/0x5e0 [/home/admin/tengine/bin/nginx] 0x47c73c : ngx_start_worker_processes+0x7c/0x100 [/home/admin/tengine/bin/nginx] 0x47db5f : ngx_master_process_cycle+0x3af/0x9b0 [/home/admin/tengine/bin/nginx] 0x45a740 : main+0xa90/0xb50 [/home/admin/tengine/bin/nginx] 0x3623e1ecdd [/lib64/libc-2.12.so+0x1ecdd/0x38d000]
好比看看內核是怎麼收包的:
root@jusse ~# cat netif_receive_skb.stp probe kernel.function("netif_receive_skb") { printf("--------------------------------------------------------\n"); print_backtrace(); printf("--------------------------------------------------------\n"); } root@jusse ~# stap netif_receive_skb.stp -------------------------------------------------------- 0xffffffff8164dc00 : netif_receive_skb+0x0/0x90 [kernel] 0xffffffff8164e280 : napi_gro_receive+0xb0/0x130 [kernel] 0xffffffff81554537 : handle_incoming_queue+0xe7/0x100 [kernel] 0xffffffff815555d9 : xennet_poll+0x279/0x430 [kernel] 0xffffffff8164ee09 : net_rx_action+0x139/0x250 [kernel] 0xffffffff810702cd : __do_softirq+0xdd/0x300 [kernel] 0xffffffff8107088e : irq_exit+0x11e/0x140 [kernel] 0xffffffff8144e785 : xen_evtchn_do_upcall+0x35/0x50 [kernel] 0xffffffff8176c9ed : xen_hvm_callback_vector+0x6d/0x80 [kernel] --------------------------------------------------------
7.4 獲取函數參數
一些被編譯器優化掉的函數參數用-L去看的時候沒有找到,這樣的話在探測點裏面也不能直接用$方式獲取該參數變量,這時可使用SystemTap提供的_arg函數接口,是根據類型指定的,好比pointer_arg是獲取指針類型參數,int_arg是獲取整型參數,相似的還有long_arg、longlong_arg、uint_arg、ulong_arg、ulonglong_arg、s32_arg、s64_arg、u32_arg、u64_arg:
root@j9 ~# stap -L 'kernel.function("sys_open")' kernel.function("SyS_open@/build/buildd/linux-lts-trusty-3.13.0/fs/open.c:1011") $ret:long int root@j9 ~# cat sys_open.stp probe kernel.function("sys_open").call { printf("filename: %p(%s), flags: %d, mode: %x\n", pointer_arg(1), kernel_string(pointer_arg(1)), int_arg(2), int_arg(3)); } root@j9 ~# stap sys_open.stp filename: 0xc2081d2120(/proc/stat), flags: 524288, mode: 0 filename: 0x7facec00e838(/root/opt/libexec/systemtap/stapio), flags: 0, mode: 1b6 filename: 0x2219488(/var/log/auth.log), flags: 0, mode: 1b6 filename: 0x7facec00e838(/root/opt/libexec/systemtap/stapio), flags: 0, mode: 1b6 filename: 0x7fad10172c29(/etc/passwd), flags: 524288, mode: 1b6 ^C
再好比兩個函數的函數參數類型兼容也可使用這種方法獲取:
這兩個函數的參數徹底兼容,只是第二個參數命名不同而已,能夠像下面這麼用:
#cat debug_tengine_5xx.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_http_finalize_request").call, process("/home/admin/tengine/bin/nginx").function("ngx_http_special_response_handler").call { rc = int_arg(2) if (rc >= 500) { printf("rc: %d\n", rc) print_ubacktrace() } }
7.5 獲取全局變量
有時候用$能夠直接獲取到全局變量,但有時候又獲取不到,那能夠試試@var:
好比獲取nginx的全局變量ngx_cycyle:
root@j9 ~# cat get_ngx_cycle.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_process_events_and_timers").call { printf("ngx_cycle->connections: %d\n", $ngx_cycle->connections) exit() } root@j9 ~# stap get_ngx_cycle.stp semantic error: while processing probe process("/home/admin/tengine/bin/nginx").function("ngx_process_events_and_timers@src/event/ngx_event.c:225").call from: process("/home/admin/tengine/bin/nginx").function("ngx_process_events_and_timers").call semantic error: unable to find local 'ngx_cycle', [man error::dwarf] dieoffset 0x73ca8 in /home/admin/tengine/bin/nginx, near pc 0x434152 in ngx_process_events_and_timers src/event/ngx_event.c (alternatives: $cycle, $delta, $timer, $flags)): identifier '$ngx_cycle' at get_ngx_cycle.stp:3:44 source: printf("ngx_cycle->connections: %d\n", $ngx_cycle->connections) ^ Pass 2: analysis failed. [man error::pass2] root@j9 ~# cat get_ngx_cycle.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_process_events_and_timers").call { ngx_cycle = @var("ngx_cycle@src/core/ngx_cycle.c") printf("ngx_cycle->connections: %d\n", ngx_cycle->connections) exit() } root@j9 ~# stap get_ngx_cycle.stp ngx_cycle->connections: 19507312
7.6 獲取數據結構成員用法
typedef struct { size_t len; u_char *data; } ngx_str_t; struct ngx_http_request_s { ...... ngx_uint_t method; ngx_uint_t http_version; ngx_str_t request_line; ngx_str_t raw_uri; ngx_str_t uri; ...... };
上面這個是nginx裏面的http請求結構裏面的幾個成員,在C語言裏,若是r是struct ngx_http_request_t *,那麼要獲取uri的data是這樣的:r->uri.data,但在SystemTap裏面,不論是指針仍是數據結構,都是用->訪問其成員:
#cat get_http_uri.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request").call { printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri.len, $r->uri.data) } #stap get_http_uri.stp WARNING: never-assigned local variable 'len' (similar: data): identifier 'len' at get_http_uri.stp:2:57 source: printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri.len, $r->uri.data) ^ WARNING: never-assigned local variable 'data' (similar: len): identifier 'data' at :2:70 source: printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri.len, $r->uri.data) ^ semantic error: invalid operator: operator '.' at :2:56 source: printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri.len, $r->uri.data) ^ semantic error: type mismatch: expected long but found string: operator '.' at :2:56 source: printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri.len, $r->uri.data) ^ Pass 2: analysis failed. [man error::pass2] #cat get_http_uri.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request").call { printf("r->uri.len: %d, r->uri.data: %p\n", $r->uri->len, $r->uri->data) } #stap get_http_uri.stp r->uri.len: 1, r->uri.data: 0x1276f94 r->uri.len: 1, r->uri.data: 0x11d5fc4 r->uri.len: 1, r->uri.data: 0x124fd24 ^C
7.7 輸出整個數據結構
SystemTap有兩個語法能夠輸出整個數據結構:在變量的後面加一個或者兩個
$便可,例子以下:
#cat get_r_pool.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request").call { printf("$r->pool$: %s\n$r->pool$$: %s\n", $r->pool$, $r->pool$$) } #stap get_r_pool.stp $r->pool$: {.d={...}, .max=4016, .current=0x161acd0, .chain=0x0, .large=0x0, .cleanup=0x0, .log=0x161c690} $r->pool$$: {.d={.last="a", .end="", .next=0x1617650, .failed=0}, .max=4016, .current=0x161acd0, .chain=0x0, .large=0x0, .cleanup=0x0, .log=0x161c690}
其中r->pool的結構以下:
typedef struct { u_char *last; u_char *end; ngx_pool_t *next; ngx_uint_t failed; } ngx_pool_data_t; struct ngx_pool_s { ngx_pool_data_t d; size_t max; ngx_pool_t *current; ngx_chain_t *chain; ngx_pool_large_t *large; ngx_pool_cleanup_t *cleanup; ngx_log_t *log; #if (NGX_DEBUG_POOL) size_t size; ngx_pool_stat_t *stat; #endif };
ngx_pool_s包含告終構ngx_pool_data_t。變量後面加和$的區別是後者展開了裏面的結構而前者不展開,此用法只輸出基本數據類型的值。
7.8 輸出字符串指針
用戶態使用:user_string、user_string_n
內核態使用:kernel_string、kernel_string_n、user_string_quoted
#cat get_http_uri.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request").call { printf("r->uri: %s\nr->uri(n): %s\n", user_string($r->uri->data), user_string_n($r->uri->data, $r->uri->len)) } #stap get_http_uri.stp r->uri: /?id=1 HTTP/1.1 User-Agent r->uri(n): /
user_string_quoted是獲取用戶態傳給內核的字符串,代碼中通常有__user宏標記:
#cat sys_open.stp probe kernel.function("sys_open") { printf("filename: %s\n", user_string_quoted(pointer_arg(1))); } #stap sys_open.stp filename: "/var/log/auth.log" filename: "/proc/stat" filename: "/proc/uptime"
7.9 指針類型轉換
SystemTap提供@cast來實現指針類型轉換,好比能夠將void *轉成本身須要的類型:
#cat get_c_fd.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request_line").call { printf("c->fd: %d\n", @cast($rev->data, "ngx_connection_t")->fd) } #stap get_c_fd.stp c->fd: 3 c->fd: 28 c->fd: 30 c->fd: 32 c->fd: 34 ^C
7.10 定義某個類型的變量
一樣是用@cast,定義一個變量用來保存其轉換後的地址便可,用法以下:
#cat get_c.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request_line").call { c = &@cast($rev->data, "ngx_connection_t") printf("c->fd: %d, c->requests: %d\n", c->fd, c->requests) } #stap get_c.stp c->fd: 3, c->requests: 1 c->fd: 28, c->requests: 1 c->fd: 30, c->requests: 1 ^C
7.11 多級指針用法
root@j9 ~# cat cc_multi_pointer.c #include <stdio.h> struct test { int count; }; int main(int argc, char *argv[]) { struct test t = {.count = 5566}; struct test *pt = &t; struct test **ppt = &pt; printf("t.count: %d, pt->count: %d, ppt->count: %d\n", t.count, pt->count, (*ppt)->count); return 0; } root@j9 ~# gcc -Wall -g -o cc_multi_pointer ./cc_multi_pointer.c root@j9 ~# cat cc_multi_pointer.stp probe process("./cc_multi_pointer").statement("main@./cc_multi_pointer.c:13") { printf("$t->count: %d, $pt->count: %d, $ppt->count: %d", $t->count, $pt->count, $ppt[0]->count); } root@j9 ~# ./cc_multi_pointer t.count: 5566, pt->count: 5566, ppt->count: 5566 root@j9 ~# stap ./cc_multi_pointer.stp -c './cc_multi_pointer' t.count: 5566, pt->count: 5566, ppt->count: 5566 $t->count: 5566, $pt->count: 5566, $ppt->count: 5566
簡言之:經過[0]去解引用便可。
7.12 遍歷C語言數組
下面是在nginx處理請求關閉時遍歷請求頭的例子:
#cat debug_http_header.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_http_finalize_request").call { i = 0 headers_in_part = &$r->headers_in->headers->part headers = &@cast(headers_in_part->elts, "ngx_table_elt_t")[0] while (headers) { if (i >= headers_in_part->nelts) { if (!headers_in_part->next) { break } headers_in_part = headers_in_part->next; headers = &@cast(headers_in_part->elts, "ngx_table_elt_t")[0] i = 0 } h = &@cast(headers, "ngx_table_elt_t")[i] printf("%s: %s\n", user_string_n(h->key->data, h->key->len), user_string_n(h->value->data, h->value->len)) i += 1 } } #stap debug_http_header.stp User-Agent: curl/7.29.0 Host: 127.0.0.1:20090 Accept: */*
7.13 查看函數指針所指的函數名
獲取一個地址所對應的符號:
用戶態:usymname
內核態:symname
#cat get_c_handler.stp probe process("/home/admin/tengine/bin/nginx").function("ngx_http_process_request_line").call { c = &@cast($rev->data, "ngx_connection_t") printf("c->read->handlers: %s, c->write->handler: %s\n", usymname(c->read->handler), usymname(c->write->handler)) } #stap get_c_handler.stp c->read->handlers: ngx_http_process_request_line, c->write->handler: ngx_http_empty_handler ^C
7.14 修改進程中的變量
root@j9 ~# cat stap_set_var.c -n 1 #include <stdio.h> 2 3 typedef struct policy { 4 int id; 5 } policy_t; 6 7 int main(int argc, char *argv[]) 8 { 9 policy_t policy; 10 policy_t *p = &policy; 11 policy_t **pp; 12 13 p->id = 111; 14 15 printf("before stap set, p->id: %d\n", p->id); 16 17 pp = &p; 18 19 printf("after stap set, p->id: %d, (*pp)->id: %d\n", p->id, (*pp)->id); 20 21 return 0; 22 } root@j9 ~# gcc -Wall -g -o ./stap_set_var ./stap_set_var.c root@j9 ~# cat stap_set_var.stp probe process("./stap_set_var").statement("main@./stap_set_var.c:17") { $p->id = 222; printf("$p$: %s\n", $p$) } root@j9 ~# stap -g stap_set_var.stp -c ./stap_set_var before stap set, p->id: 111 after stap set, p->id: 222, (*pp)->id: 222 $p$: {.id=222} root@j9 ~#
能夠看出在第17行用SystemTap修改後的值在第19行就生效了。
須要注意的是stap要加-g參數在guru模式下才能修改變量的值。
7.15 跟蹤進程執行流程
thread_indent(n): 補充空格
ppfunc(): 當前探測點所在的函數
在call探測點調用thread_indent(4)補充4個空格,在return探測點調用thread_indent(-4)回退4個空格,效果以下:
#cat trace_nginx.stp probe process("/home/admin/tengine/bin/nginx").function("*@src/http/ngx_http_*").call { printf("%s -> %s\n", thread_indent(4), ppfunc()); } probe process("/home/admin/tengine/bin/nginx").function("*@src/http/ngx_http_*").return { printf("%s <- %s\n", thread_indent(-4), ppfunc()); } #stap trace_nginx.stp 0 nginx(11368): -> ngx_http_init_connection 21 nginx(11368): <- ngx_http_init_connection 0 nginx(11368): -> ngx_http_wait_request_handler 30 nginx(11368): -> ngx_http_create_request 41 nginx(11368): <- ngx_http_create_request 55 nginx(11368): -> ngx_http_process_request_line 72 nginx(11368): -> ngx_http_read_request_header 78 nginx(11368): <- ngx_http_read_request_header 91 nginx(11368): -> ngx_http_parse_request_line 99 nginx(11368): <- ngx_http_parse_request_line 109 nginx(11368): -> ngx_http_process_request_uri 115 nginx(11368): <- ngx_http_process_request_uri 127 nginx(11368): -> ngx_http_process_request_headers 138 nginx(11368): -> ngx_http_read_request_header 143 nginx(11368): <- ngx_http_read_request_header 155 nginx(11368): -> ngx_http_parse_header_line 163 nginx(11368): <- ngx_http_parse_header_line 178 nginx(11368): -> ngx_http_process_user_agent 185 nginx(11368): <- ngx_http_process_user_agent 192 nginx(11368): -> ngx_http_parse_header_line 198 nginx(11368): <- ngx_http_parse_header_line 208 nginx(11368): -> ngx_http_process_host 222 nginx(11368): -> ngx_http_validate_host 229 nginx(11368): <- ngx_http_validate_host 239 nginx(11368): -> ngx_http_set_virtual_server 252 nginx(11368): -> ngx_http_find_virtual_server 259 nginx(11368): <- ngx_http_find_virtual_server 263 nginx(11368): <- ngx_http_set_virtual_server 266 nginx(11368): <- ngx_http_process_host 274 nginx(11368): -> ngx_http_parse_header_line 279 nginx(11368): <- ngx_http_parse_header_line 287 nginx(11368): -> ngx_http_parse_header_line 292 nginx(11368): <- ngx_http_parse_header_line ..... 2072 nginx(11368): <- ngx_http_finalize_request 2076 nginx(11368): <- ngx_http_core_content_phase 2079 nginx(11368): <- ngx_http_core_run_phases 2083 nginx(11368): <- ngx_http_handler 2093 nginx(11368): -> ngx_http_run_posted_requests 2100 nginx(11368): <- ngx_http_run_posted_requests 2103 nginx(11368): <- ngx_http_process_request 2107 nginx(11368): <- ngx_http_process_request_headers 2111 nginx(11368): <- ngx_http_process_request_line 2114 nginx(11368): <- ngx_http_wait_request_handler 0 nginx(11368): -> ngx_http_keepalive_handler 26 nginx(11368): -> ngx_http_close_connection 79 nginx(11368): <- ngx_http_close_connection 83 nginx(11368): <- ngx_http_keepalive_handler
7.16 查看代碼執行路徑
pp(): 輸出當前被激活的探測點
#cat ngx_http_process_request.stp probe process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:*") { printf("%s\n", pp()) } #stap ngx_http_process_request.stp process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2762") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2768") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2771") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2773") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2774") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2783") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2835") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2840") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2841") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2842") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2843") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2846") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2847") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2848") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2850") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2852") process("/home/admin/tengine/bin/nginx").statement("ngx_http_process_request@src/http/ngx_http_request.c:2853") ^C
能夠看出該函數哪些行被執行了。
7.17 巧用正則匹配過濾
在排查問題時,能夠利用一些正則匹配來獲取本身想要的信息,好比下面是隻收集*.j9.com的堆棧:
# cat debug_tengine_5xx.stp probe process("/home/admin/tengine/bin/t-coresystem-tengine-cdn").function("ngx_http_finalize_request").call { rc = $rc if (rc < 0) { host = "(null)" if ($r->headers_in->server->len != 0) { host = user_string_n($r->headers_in->server->data, $r->headers_in->server->len) } else { cscf = &@cast($r->srv_conf, "ngx_http_core_srv_conf_t")[@var("ngx_http_core_module@src/http/ngx_http_core_module.c")->ctx_index] if (cscf->server_name->len != 0) { host = user_string_n(cscf->server_name->data, cscf->server_name->len) } } if (host =~ ".*\.j9\.com") { printf("rc: %d, host: %s\n", rc, host) print_ubacktrace() } } } #stap debug_tengine_5xx.stp WARNING: Missing unwind data for module, rerun with 'stap -d /lib64/libc-2.12.so' rc: -4, host: www.j9.com 0x49af2e : ngx_http_finalize_request+0xe/0x480 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x492eab : ngx_http_core_content_phase+0x2b/0x130 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x48e74d : ngx_http_core_run_phases+0x3d/0x50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x514c3c : ngx_http_lua_socket_tcp_read+0x44c/0x590 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x513150 : ngx_http_lua_socket_tcp_handler+0x30/0x50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x475b96 : ngx_event_process_posted+0x36/0x40 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x47d4d8 : ngx_worker_process_cycle+0x138/0x260 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x47a38a : ngx_spawn_process+0x1ca/0x5e0 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x47c73c : ngx_start_worker_processes+0x7c/0x100 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x47db5f : ngx_master_process_cycle+0x3af/0x9b0 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x45a740 : main+0xa90/0xb50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x3623e1ecdd [/lib64/libc-2.12.so+0x1ecdd/0x38d000] rc: -4, host: cdn.j9.com 0x49af2e : ngx_http_finalize_request+0xe/0x480 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x492eab : ngx_http_core_content_phase+0x2b/0x130 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x48e74d : ngx_http_core_run_phases+0x3d/0x50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x514c3c : ngx_http_lua_socket_tcp_read+0x44c/0x590 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x513150 : ngx_http_lua_socket_tcp_handler+0x30/0x50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x475b96 : ngx_event_process_posted+0x36/0x40 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x47d4d8 : ngx_worker_process_cycle+0x138/0x260 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x47a38a : ngx_spawn_process+0x1ca/0x5e0 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x47c73c : ngx_start_worker_processes+0x7c/0x100 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x47db5f : ngx_master_process_cycle+0x3af/0x9b0 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x45a740 : main+0xa90/0xb50 [/home/admin/tengine/bin/t-coresystem-tengine-cdn] 0x3623e1ecdd [/lib64/libc-2.12.so+0x1ecdd/0x38d000]
7.18 關聯數組用法
SystemTap的關聯數組必須是全局變量,須要用global進行聲明,其索引能夠支持多達9項索引域,各域間以逗號隔開。支持 =, ++ 與 +=操做,其默認的初始值爲0。
例如:
root@j9 ~# cat stap_array.stp global reads probe vfs.read { reads[execname(), pid()] ++ } probe timer.s(3) { foreach ([execname, pid] in reads) { printf("%s(%d) : %d \n", execname, pid, reads[execname, pid]) } print("============================\n") delete reads } root@j9 ~# stap stap_array.stp stapio(18716) : 16 rsyslogd(770) : 1 docker(743) : 3 IFSWatch(5594) : 30 QThread(5594) : 6 AliYunDunUpdate(1057) : 4 sshd(15118) : 1 sshd(15191) : 1 ============================ stapio(18716) : 16 sshd(15191) : 3 docker(743) : 3 IFSWatch(5594) : 30 sshd(15118) : 2 QThread(5594) : 12 AliYunDunUpdate(1057) : 8 ============================ ^C root@j9 ~/systemtap#
也能夠用+、-進行排序:
root@j9 ~# cat stap_array.stp global reads probe vfs.read { reads[execname(), pid()] ++ } probe timer.s(3) { foreach ([execname, pid+] in reads) { printf("%s(%d) : %d \n", execname, pid, reads[execname, pid]) } print("============================\n") delete reads } root@j9 ~# stap stap_array.stp docker(743) : 3 rsyslogd(770) : 1 AliYunDunUpdate(1057) : 12 IFSWatch(5594) : 30 QThread(5594) : 12 sshd(15118) : 2 sshd(15191) : 2 stapio(19021) : 16 ============================ docker(743) : 3 AliYunDunUpdate(1057) : 12 IFSWatch(5594) : 30 QThread(5594) : 6 sshd(15118) : 1 sshd(15191) : 19 stapio(19021) : 16 ============================ ^C root@j9 ~#
7.19 調試內存泄漏以及內存重複釋放
probe begin { printf("=============begin============\n") } //記錄內存分配和釋放的計數關聯數組 global g_mem_ref_tbl //記錄內存分配和釋放的調用堆棧關聯數組 global g_mem_bt_tbl probe process("/lib/x86_64-linux-gnu/libc.so.6").function("__libc_malloc").return, process("/lib/x86_64-linux-gnu/libc.so.6").function("__libc_calloc").return { if (target() == pid()) { if (g_mem_ref_tbl[$return] == 0) { g_mem_ref_tbl[$return]++ g_mem_bt_tbl[$return] = sprint_ubacktrace() } } } probe process("/lib/x86_64-linux-gnu/libc.so.6").function("__libc_free").call { if (target() == pid()) { g_mem_ref_tbl[$mem]-- if (g_mem_ref_tbl[$mem] == 0) { if ($mem != 0) { //記錄上次釋放的調用堆棧 g_mem_bt_tbl[$mem] = sprint_ubacktrace() } } else if (g_mem_ref_tbl[$mem] < 0 && $mem != 0) { //若是調用free已經失衡,那就出現了重複釋放內存的問題,這裏輸出當前調用堆棧,以及這個地址上次釋放的調用堆棧 printf("MMMMMMMMMMMMMMMMMMMMMMMMMMMM\n") printf("g_mem_ref_tbl[%p]: %d\n", $mem, g_mem_ref_tbl[$mem]) print_ubacktrace() printf("last free backtrace:\n%s\n", g_mem_bt_tbl[$mem]) printf("WWWWWWWWWWWWWWWWWWWWWWWWWWWW\n") } } } probe end { //最後輸出產生泄漏的內存是在哪裏分配的 printf("=============end============\n") foreach(mem in g_mem_ref_tbl) { if (g_mem_ref_tbl[mem] > 0) { printf("%s\n", g_mem_bt_tbl[mem]) } } }
詳細請看:http://blog.csdn.net/wangzuxi...
7.20 嵌入C代碼
在進程fork出子進程時打印出進程id和進程名:
root@jusse ~/systemtap# cat copy_process.stp function getprocname:string(task:long) %{ struct task_struct *task = (struct task_struct *)STAP_ARG_task; snprintf(STAP_RETVALUE, MAXSTRINGLEN, "pid: %d, comm: %s", task->pid, task->comm); %} function getprocid:long(task:long) %{ struct task_struct *task = (struct task_struct *)STAP_ARG_task; STAP_RETURN(task->pid); %} probe kernel.function("copy_process").return { printf("copy_process return: %p, pid: %d, getprocname: %s, getprocid: %d\n", $return, $return->pid, getprocname($return), getprocid($return)); } root@jusse ~/systemtap# stap -g copy_process.stp copy_process return: 0xffff880039f61800, pid: 12212, getprocname: pid: 12212, comm: bash, getprocid: 12212 copy_process return: 0xffff880039f61800, pid: 12212, getprocname: pid: 12212, comm: bash, getprocid: 12212 copy_process return: 0xffff880039f63000, pid: 12213, getprocname: pid: 12213, comm: cc_epoll, getprocid: 12213 copy_process return: 0xffff880039f63000, pid: 12213, getprocname: pid: 12213, comm: cc_epoll, getprocid: 12213 copy_process return: 0xffff8800081a9800, pid: 12214, getprocname: pid: 12214, comm: cc_epoll, getprocid: 12214 copy_process return: 0xffff8800081a9800, pid: 12214, getprocname: pid: 12214, comm: cc_epoll, getprocid: 12214 copy_process return: 0xffff8800004d8000, pid: 12215, getprocname: pid: 12215, comm: cc_epoll, getprocid: 12215 copy_process return: 0xffff8800004d8000, pid: 12215, getprocname: pid: 12215, comm: cc_epoll, getprocid: 12215 copy_process return: 0xffff880000564800, pid: 12216, getprocname: pid: 12216, comm: cc_epoll, getprocid: 12216 copy_process return: 0xffff880000564800, pid: 12216, getprocname: pid: 12216, comm: cc_epoll, getprocid: 12216 copy_process return: 0xffff880000566000, pid: 12217, getprocname: pid: 12217, comm: cc_epoll, getprocid: 12217 copy_process return: 0xffff880000566000, pid: 12217, getprocname: pid: 12217, comm: cc_epoll, getprocid: 12217
有三個須要注意的地方:
1)、SystemTap腳本里面嵌入C語言代碼要在每一個大括號前加%前綴,是%{…… %} 而不是%{ …… }%;
2)、獲取腳本函數參數要用STAP_ARG_前綴;
3)、通常long等返回值用STAP_RETURN,而string類型返回值要用snprintf、strncat等方式把字符串複製到STAP_RETVALUE裏面。
7.21 調試內核模塊
這小節就不細講了,這篇博客 (http://blog.chinaunix.net/uid-14528823-id-4726046.html) 寫得很詳細,這裏只copy兩個關鍵點過來記錄一下:
要調試本身的內核模塊,須要注意的有兩個關鍵點:
1)、使用SystemTap調試內核模塊,探測點的編寫格式示例爲:
module("ext3").function("ext3_*")
2)、須要將本身的模塊cp到/lib/modules/uname -r/extra目錄中,不然找不到符號,若是/lib/modules/uname -r/目錄下沒有extra這個目錄,本身mkdir一下就能夠。
7.22 一些錯誤提示及解決辦法
錯誤提示1:
ERROR: MAXACTION exceeded near keyword at debug_connection.stp:86:9 ERROR: MAXACTION exceeded near operator '->' at debug_connection.stp:84:30
解決辦法:
加上stap參數:-DMAXACTION=102400,若是還報這種類型的錯誤,只需把102400調成更大的值便可。
錯誤提示2:
WARNING: Number of errors: 0, skipped probes: 82
解決辦法:
加上-DMAXSKIPPED=102400和-DSTP_NO_OVERLOAD參數
還有一些能夠去掉限制的宏:
MAXSTRINGLEN:這個宏會影響sprintf的buffer大小,默認爲512字節。
MAXTRYLOCK:對全局變量進行try lock操做的次數,超過則次數還拿不到鎖則放棄和跳過該探測點,默認值爲1000.全局變量多的時候能夠把這個宏開大一點。
(完)