Go -- 經過GOTRACEBACK生成程序崩潰後core文件的方法(gcore gdb)

寫一個錯誤的c程序

 

package dlsym import "testing" func Test_intercept(t *testing.T) { Intercept("gethostbyname\x00") }

package dlsym // #cgo CFLAGS: -I. // #include <stddef.h> // #include "dlsym_wrapper.h" import "C" import "unsafe" func Intercept(symbol string) { ptr := unsafe.Pointer(&([]byte(symbol)[0])) C.intercept((*C.char)(ptr), C.size_t(len(symbol))) }

#include <dlfcn.h> #include <stddef.h> #include <stdio.h> void intercept(char *symbol, size_t symbol_len) { symbol = NULL; // will cause SIGSEGV printf("%s\n", symbol); fflush(stdout); }

編譯測試爲可執行文件

go test -c github.com/taowen/go-lib c/dlsym # will produce executable dlsym.test

這個是用於分析coredump的時候得到符號表使用的。

執行測試，得到coredump

GOTRACEBACK=crash ./dlsym.test # produced /tmp/core_dlsym.test.29937

若是找不到coredump的位置，執行以前先設置好coredump的寫出條件

echo '/tmp/core_%e.%p' | sudo tee /proc/sys/kernel/core_pattern ulimit -c unlimited # coredump can be any large

用gdb分析coredump

gdb dlsym.test /tmp/core_dlsym.test /tmp/core_dlsym.test.29937

• 用 bt full 查看全部的frame

• 用 frame <number> 查看指定的frame

• 用 print <symbol> 查看指定的變量的值

 

 

 

經過cgo調用C語言庫時會出現程序崩潰的狀況，因而就但願可以生成core文件來查看程序崩潰時的堆棧信息。那麼Golang程序如何在崩潰後生成core文件呢？答案就是GOTRACEBACK這個環境變量。

 

       關於GOTRACEBACK環境變量的詳細說明，能夠參考官方 文檔在runtime一節的鏈接，這裏僅列出文檔中較爲核心的說明以下（Golang版本爲1.6）。根據文檔的說明咱們能夠知道GOTRACEBACK的可選值爲：none、single、all、system和crash，其中關於crash的說明就指出了在Unix系統上，程序崩潰會經過SIGABRT信號觸發一次core dump。

 

GOTRACEBACK=none omits the goroutine stack traces entirely. 

GOTRACEBACK=single (the default) behaves as described above. 

GOTRACEBACK=all adds stack traces for all user-created goroutines. 

GOTRACEBACK=system is like 「all」 but adds stack frames for run-time functions and shows goroutines created internally by the run-time. 

GOTRACEBACK=crash is like 「system」 but crashes in an operating system-specific manner instead of exiting. For example, on Unix systems, the crash raises SIGABRT to trigger a core dump.

 

       另外，雖然官方文檔給出了GOTRACEBACK環境變量在不一樣選值下的行爲說明，但看起來實在是有些不知所云，這裏咱們不妨經過一個簡單的程序來驗證一下，畢竟程序跑出來的效果才比較真實。如下是驗證程序的代碼，其功能很是簡單，啓動兩個goroutine，一個正常跑10秒鐘，一個睡5秒鐘以後自行panic。

 

package main

import (

"fmt"

"time"

)

func saferoutine(c chan bool) {

for i := 0; i < 10; i++ {

fmt.Println("Count:", i)

time.Sleep(1 * time.Second)

}

c <- true

}

func panicgoroutine(c chan bool) {

time.Sleep(5 * time.Second)

panic("Panic, omg ...")

c <- true

}

func main() {

c := make(chan bool, 2)

go saferoutine(c)

go panicgoroutine(c)

for i := 0; i < 2; i++ {

<-c

}

}

 

       編譯以上驗證程序，而後在GOTRACEBACK取值爲none的狀況下執行，其結果以下。由此可知在取值爲none時，程序僅僅打印出panic消息，以後程序就退出了。

 

# go build testgotraceback.go 

# env GOTRACEBACK=none ./testgotraceback 

Count: 0

Count: 1

Count: 2

Count: 3

Count: 4

panic: Panic, omg ...

 

       如下爲GOTRACEBACK取值爲single狀況下的執行結果，此時還會打印出發生panic的goroutine（也就是main.panicgoroutine）的調用棧信息，而single也是GOTRACEBACK的默認值。

 

# env GOTRACEBACK=single ./testgotraceback

Count: 0

Count: 1

Count: 2

Count: 3

Count: 4

panic: Panic, omg ...

 

goroutine 6 [running]:

panic(0x4b8e00, 0xc82000a310)

/home/ubuntu/repository/oo/go/src/runtime/panic.go:464 +0x3e6

main.panicgoroutine(0xc82004e070)

/home/ubuntu/go/src/testcode/testgotraceback.go:18 +0x78

created by main.main

/home/ubuntu/go/src/testcode/testgotraceback.go:25 +0x79

 

       如下爲GOTRACEBACK取值爲all狀況下的執行結果，此時不只發生panic的goroutine（也就是main.panicgoroutine）的調用棧信息會被打印出來，主進程（main.main）和正常goroutine（也就是main.saferoutine）的調用棧信息也都被打印出來了。

 

# env GOTRACEBACK=all ./testgotraceback

Count: 0

Count: 1

Count: 2

Count: 3

Count: 4

panic: Panic, omg ...

 

goroutine 6 [running]:

panic(0x4b8e00, 0xc82000a300)

/home/ubuntu/repository/oo/go/src/runtime/panic.go:464 +0x3e6

main.panicgoroutine(0xc82004e070)

/home/ubuntu/go/src/testcode/testgotraceback.go:18 +0x78

created by main.main

/home/ubuntu/go/src/testcode/testgotraceback.go:25 +0x79

 

goroutine 1 [chan receive]:

main.main()

/home/ubuntu/go/src/testcode/testgotraceback.go:27 +0xa9

 

goroutine 5 [sleep]:

time.Sleep(0x3b9aca00)

/home/ubuntu/repository/oo/go/src/runtime/time.go:59 +0xf9

main.saferoutine(0xc82004e070)

/home/ubuntu/go/src/testcode/testgotraceback.go:11 +0x177

created by main.main

/home/ubuntu/go/src/testcode/testgotraceback.go:24 +0x57

 

       如下爲GOTRACEBACK取值爲system狀況下的執行結果，此時除了咱們明顯知道的三個goroutine的調用棧信息會被打印出來之外，不少隱形存在的系統級goroutine的信息也被打印出來了。由於本博主還不太瞭解Golang運行期機制，這裏就不對這些系統級的goroutine一一作介紹了，但最明顯的是有用來作垃圾回收的相關協程。

 

# env GOTRACEBACK=system ./testgotraceback

Count: 0

Count: 1

Count: 2

Count: 3

Count: 4

panic: Panic, omg ...

 

goroutine 6 [running]:

panic(0x4b8e00, 0xc820064090)

/home/ubuntu/repository/oo/go/src/runtime/panic.go:464 +0x3e6 fp=0xc820028778 sp=0xc8200286f8

main.panicgoroutine(0xc820056070)

/home/ubuntu/go/src/testcode/testgotraceback.go:18 +0x78 fp=0xc8200287b8 sp=0xc820028778

runtime.goexit()

/home/ubuntu/repository/oo/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8200287c0 sp=0xc8200287b8

created by main.main

/home/ubuntu/go/src/testcode/testgotraceback.go:25 +0x79

 

goroutine 1 [chan receive]:

runtime.gopark(0x52b150, 0xc8200560c8, 0x5005c0, 0xc, 0x17, 0x3)

/home/ubuntu/repository/oo/go/src/runtime/proc.go:262 +0x163 fp=0xc82003de30 sp=0xc82003de08

runtime.goparkunlock(0xc8200560c8, 0x5005c0, 0xc, 0x17, 0x3)

/home/ubuntu/repository/oo/go/src/runtime/proc.go:268 +0x54 fp=0xc82003de68 sp=0xc82003de30

runtime.chanrecv(0x4b4780, 0xc820056070, 0x0, 0xc82003df01, 0x400000)

/home/ubuntu/repository/oo/go/src/runtime/chan.go:470 +0x49f fp=0xc82003def0 sp=0xc82003de68

runtime.chanrecv1(0x4b4780, 0xc820056070, 0x0)

/home/ubuntu/repository/oo/go/src/runtime/chan.go:355 +0x2b fp=0xc82003df20 sp=0xc82003def0

main.main()

/home/ubuntu/go/src/testcode/testgotraceback.go:27 +0xa9 fp=0xc82003df50 sp=0xc82003df20

runtime.main()

/home/ubuntu/repository/oo/go/src/runtime/proc.go:188 +0x2b0 fp=0xc82003dfa0 sp=0xc82003df50

runtime.goexit()

/home/ubuntu/repository/oo/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82003dfa8 sp=0xc82003dfa0

 

goroutine 2 [force gc (idle)]:

runtime.gopark(0x52b150, 0x585640, 0x500ce0, 0xf, 0x14, 0x1)

/home/ubuntu/repository/oo/go/src/runtime/proc.go:262 +0x163 fp=0xc820026758 sp=0xc820026730

runtime.goparkunlock(0x585640, 0x500ce0, 0xf, 0xc820000314, 0x1)

/home/ubuntu/repository/oo/go/src/runtime/proc.go:268 +0x54 fp=0xc820026790 sp=0xc820026758

runtime.forcegchelper()

/home/ubuntu/repository/oo/go/src/runtime/proc.go:229 +0xb8 fp=0xc8200267c0 sp=0xc820026790

runtime.goexit()

/home/ubuntu/repository/oo/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8200267c8 sp=0xc8200267c0

created by runtime.init.4

/home/ubuntu/repository/oo/go/src/runtime/proc.go:218 +0x2b

 

goroutine 3 [GC sweep wait]:

runtime.gopark(0x52b150, 0x585760, 0x4ff370, 0xd, 0x41bf14, 0x1)

/home/ubuntu/repository/oo/go/src/runtime/proc.go:262 +0x163 fp=0xc820026f48 sp=0xc820026f20

runtime.goparkunlock(0x585760, 0x4ff370, 0xd, 0x14, 0x1)

/home/ubuntu/repository/oo/go/src/runtime/proc.go:268 +0x54 fp=0xc820026f80 sp=0xc820026f48

runtime.bgsweep(0xc820056000)

/home/ubuntu/repository/oo/go/src/runtime/mgcsweep.go:63 +0xb1 fp=0xc820026fb8 sp=0xc820026f80

runtime.goexit()

/home/ubuntu/repository/oo/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820026fc0 sp=0xc820026fb8

created by runtime.gcenable

/home/ubuntu/repository/oo/go/src/runtime/mgc.go:191 +0x53

 

goroutine 4 [finalizer wait]:

runtime.gopark(0x52b150, 0x59fc00, 0x500a40, 0xe, 0x14, 0x1)

/home/ubuntu/repository/oo/go/src/runtime/proc.go:262 +0x163 fp=0xc820027718 sp=0xc8200276f0

runtime.goparkunlock(0x59fc00, 0x500a40, 0xe, 0x14, 0x1)

/home/ubuntu/repository/oo/go/src/runtime/proc.go:268 +0x54 fp=0xc820027750 sp=0xc820027718

runtime.runfinq()

/home/ubuntu/repository/oo/go/src/runtime/mfinal.go:158 +0xaa fp=0xc8200277c0 sp=0xc820027750

runtime.goexit()

/home/ubuntu/repository/oo/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8200277c8 sp=0xc8200277c0

created by runtime.createfing

/home/ubuntu/repository/oo/go/src/runtime/mfinal.go:139 +0x60

 

goroutine 5 [sleep]:

runtime.gopark(0x52b150, 0x585780, 0x4fded0, 0x5, 0x643513, 0x2)

/home/ubuntu/repository/oo/go/src/runtime/proc.go:262 +0x163 fp=0xc82003ee80 sp=0xc82003ee58

runtime.goparkunlock(0x585780, 0x4fded0, 0x5, 0x13, 0x2)

/home/ubuntu/repository/oo/go/src/runtime/proc.go:268 +0x54 fp=0xc82003eeb8 sp=0xc82003ee80

time.Sleep(0x3b9aca00)

/home/ubuntu/repository/oo/go/src/runtime/time.go:59 +0xf9 fp=0xc82003ef00 sp=0xc82003eeb8

main.saferoutine(0xc820056070)

/home/ubuntu/go/src/testcode/testgotraceback.go:11 +0x177 fp=0xc82003efa8 sp=0xc82003ef00

runtime.goexit()

/home/ubuntu/repository/oo/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82003efb0 sp=0xc82003efa8

created by main.main

/home/ubuntu/go/src/testcode/testgotraceback.go:24 +0x57

 

goroutine 7 [syscall]:

runtime.notetsleepg(0x585798, 0x682c29, 0x0)

/home/ubuntu/repository/oo/go/src/runtime/lock_futex.go:205 +0x4e fp=0xc820028f38 sp=0xc820028f10

runtime.timerproc()

/home/ubuntu/repository/oo/go/src/runtime/time.go:209 +0xde fp=0xc820028fc0 sp=0xc820028f38

runtime.goexit()

/home/ubuntu/repository/oo/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820028fc8 sp=0xc820028fc0

created by runtime.addtimerLocked

/home/ubuntu/repository/oo/go/src/runtime/time.go:116 +0x11f

 

       如下爲GOTRACEBACK取值爲crash狀況下的執行結果，固然在執行以前咱們須要修改core文件大小爲unlimited，此時打印出來的內容與system的狀況下基本同樣，除了最後那行表明core dump的提示。另外，在當前目錄下，咱們想要的core文件出現了！因此在Linux系統下，crash與system的區別就是，前者能夠生成core文件。

 

# ulimit -c unlimited

# env GOTRACEBACK=crash ./testgotraceback

...

goroutine 21 [runnable]:

runtime.notetsleepg(0x585798, 0x11b22e, 0x0)

/home/ubuntu/repository/oo/go/src/runtime/lock_futex.go:205 +0x4e fp=0xc820024738 sp=0xc820024710

runtime.timerproc()

/home/ubuntu/repository/oo/go/src/runtime/time.go:209 +0xde fp=0xc8200247c0 sp=0xc820024738

runtime.goexit()

/home/ubuntu/repository/oo/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8200247c8 sp=0xc8200247c0

created by runtime.addtimerLocked

/home/ubuntu/repository/oo/go/src/runtime/time.go:116 +0x11f

Aborted (core dumped)

# ls -l

total 5400

-rw------- 1 ubuntu ubuntu 1994752 Apr 20 18:11 core

-rwxrwxr-x 1 ubuntu ubuntu 2295416 Apr 20 17:38 testgotraceback

-rw-rw-r-- 1 ubuntu ubuntu     395 Apr 20 17:38 testgotraceback.go

 

       經過gdb命令，咱們能夠看到程序發生崩潰時的具體位置，這樣即便崩潰點發生在C語言庫中，也能夠快速定位問題，簡直就是so nice！

 

# gdb ./testgotraceback core

(gdb) bt

#0  runtime.raise () at /home/ubuntu/repository/oo/go/src/runtime/sys_linux_amd64.s:110

#1  0x0000000000438588 in runtime.dieFromSignal (sig=6) at /home/ubuntu/repository/oo/go/src/runtime/signal1_unix.go:192

#2  0x00000000004386af in runtime.crash () at /home/ubuntu/repository/oo/go/src/runtime/signal1_unix.go:247

#3  0x000000000042886e in runtime.dopanic_m (gp=0xc820001380, pc=4357542, sp=859530495736)

    at /home/ubuntu/repository/oo/go/src/runtime/panic.go:642

#4  0x000000000044ad72 in runtime.dopanic.func1 () at /home/ubuntu/repository/oo/go/src/runtime/panic.go:517

#5  0x0000000000453569 in runtime.systemstack () at /home/ubuntu/repository/oo/go/src/runtime/asm_amd64.s:291

#6  0x000000000042c6b0 in runtime.startTheWorldWithSema () at /home/ubuntu/repository/oo/go/src/runtime/proc.go:983

#7  0x000000c82001f500 in ?? ()

#8  0x0000000000000000 in ?? ()

 

參考連接：http://dave.cheney.net/2015/11/29/a-whirlwind-tour-of-gos-runtime-environment-variables