Windbg程序調試系列4-Live Debugging

時間  2019-11-05
標籤 windbg 程序 調試 系列 live debugging 简体版
原文   原文鏈接

上篇博文中給你們分享了使用Windbg分析線程阻塞問題：html

Windbg程序調試系列3-線程阻塞問題前端

本篇中咱們繼續，跟你們分享附加進程實時調試-Live Debugging。session

先說一下使用Windbg附加進程實時調試的應用場景和注意事項：app

應用場景：oop

  • 集成測試環境，影響異常後，分析異常和線程上下文的執行堆棧、參數狀況；
  • 生產環境：短期內調試程序異常，查看異常上下文和參數，可是調試時間不能過久。

注意事項：附加進程調試會阻塞請求，調試後新的請求被阻塞住，前端調用受影響，所以要謹慎、權衡利弊，開發測試環境能夠，生產環境要謹慎。測試

附加進程實時調試的套路：ui

  1. F6 Attache進程
  2. 加載SOS, .loadby sos clr
  3. 啓用異常捕獲 sxe clr
  4. 運行，捕獲&查看異常 g  !pe
  5. 查看異常所在線程堆棧!clrstack, 查看異常所在線程堆棧的參數和變量 !clrstack -a / !dso
  6. 禁用異常捕獲 sxd clr
  7. 退出調試 qd

動手實操：this

1. F6 Attache進程spa

打開Windbg，按下F6鍵，選擇要調試的進程，支持輸入進程ID：插件

2. 加載SOS, .loadby sos clr

.loadby sos clr

3. 啓用異常捕獲 sxe clr

sxe clr

4. 運行，捕獲&查看異常 g  !pe

g

上圖的輸出中，咱們發現瞭如下異常信息：SocketException：遠程追究強迫關閉了一個現有的鏈接

0:032> !pe
Exception object: 000000d79aedb7a8
Exception type:   System.Net.Sockets.SocketException
Message:          遠程主機強迫關閉了一個現有的鏈接。
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80004005

異常Exception對象的內存地址：

000000d79aedb7a8

5. 查看異常所在線程堆棧!clrstack, 查看異常所在線程堆棧的參數和變量 !clrstack -a / !dso

0:032> !pe
Exception object: 000000d79aedb7a8
Exception type:   System.Net.Sockets.SocketException
Message:          遠程主機強迫關閉了一個現有的鏈接。
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80004005
0:032> !clrstack
OS Thread Id: 0x15968 (32)
        Child SP               IP Call Site
000000d7b927ea88 00007ffe61c395fc [HelperMethodFrame: 000000d7b927ea88] 
000000d7b927eb70 00007ffe52e68f0f *** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.30319_64\System\47e0be927382f169f5de470fab0ceb7d\System.ni.dll
System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) [f:\dd\NDP\fx\src\net\System\Net\Sockets\NetworkStream.cs @ 513]
000000d7b927ebe0 00007ffe53e59c04 *** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\5d0c037297cc1a64b52ce43b45c2ac2e\mscorlib.ni.dll
System.IO.BufferedStream.ReadByte() [f:\dd\ndp\clr\src\BCL\system\io\bufferedstream.cs @ 814]
000000d7b927ec10 00007ffe53e2183c System.IO.BinaryReader.ReadByte() [f:\dd\ndp\clr\src\BCL\system\io\binaryreader.cs @ 143]
000000d7b927ec40 00007ffdf6188eb5 RabbitMQ.Client.Impl.Frame.ReadFrom(RabbitMQ.Util.NetworkBinaryReader)
000000d7b927ecb0 00007ffdf6188e13 RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()
000000d7b927ed10 00007ffdf6188cfc RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()
000000d7b927ed50 00007ffdf6188a9f RabbitMQ.Client.Framing.Impl.Connection.MainLoop()
000000d7b927edb0 00007ffe53e34740 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 954]
000000d7b927ee80 00007ffe53e345d4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 902]
000000d7b927eeb0 00007ffe53e345a2 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 891]
000000d7b927ef00 00007ffe53ebcf62 System.Threading.ThreadHelper.ThreadStart() [f:\dd\ndp\clr\src\BCL\system\threading\thread.cs @ 111]
000000d7b927f158 00007ffe55325a03 [GCFrame: 000000d7b927f158] 
000000d7b927f4a8 00007ffe55325a03 [DebuggerU2MCatchHandlerFrame: 000000d7b927f4a8]

上面的輸出中，咱們能看到異常所在的線程是32號線程，線程堆棧以下，RabbitMQ的通信異常（心跳線程鏈接RabbitMQ Server異常）

繼續看線程堆棧的變量：

!clrstack -a
0:032> !clrstack -a
OS Thread Id: 0x15968 (32)
        Child SP               IP Call Site
000000d7b927ea88 00007ffe61c395fc [HelperMethodFrame: 000000d7b927ea88] 
000000d7b927eb70 00007ffe52e68f0f System.Net.Sockets.NetworkStream.Read(Byte[], Int32, Int32) [f:\dd\NDP\fx\src\net\System\Net\Sockets\NetworkStream.cs @ 513]
    PARAMETERS:
        this = <no data>
        buffer = <no data>
        offset = <no data>
        size = <no data>
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

000000d7b927ebe0 00007ffe53e59c04 System.IO.BufferedStream.ReadByte() [f:\dd\ndp\clr\src\BCL\system\io\bufferedstream.cs @ 814]
    PARAMETERS:
        this (<CLR reg>) = 0x000000d79a3c2f18
    LOCALS:
        <no data>
        <no data>

000000d7b927ec10 00007ffe53e2183c System.IO.BinaryReader.ReadByte() [f:\dd\ndp\clr\src\BCL\system\io\binaryreader.cs @ 143]
    PARAMETERS:
        this = <no data>
    LOCALS:
        <no data>

000000d7b927ec40 00007ffdf6188eb5 RabbitMQ.Client.Impl.Frame.ReadFrom(RabbitMQ.Util.NetworkBinaryReader)
    PARAMETERS:
        reader (<CLR reg>) = 0x000000d79a3c2f70
    LOCALS:
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>
        <no data>

000000d7b927ecb0 00007ffdf6188e13 RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()
    PARAMETERS:
        this = <no data>
    LOCALS:
        <no data>
        0x000000d7b927ece0 = 0x0000000000000000
        0x000000d7b927ecd8 = 0x000000d79a3c2f70

000000d7b927ed10 00007ffdf6188cfc RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()
    PARAMETERS:
        this (<CLR reg>) = 0x000000d79a3c31b8
    LOCALS:
        <no data>
        <no data>

000000d7b927ed50 00007ffdf6188a9f RabbitMQ.Client.Framing.Impl.Connection.MainLoop()
    PARAMETERS:
        this (0x000000d7b927edb0) = 0x000000d79a3c31b8
    LOCALS:
        0x000000d7b927ed8c = 0x0000000000000000
        <no data>
        <no data>
        <no data>
        <no data>

000000d7b927edb0 00007ffe53e34740 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 954]
    PARAMETERS:
        executionContext = <no data>
        callback = <no data>
        state = <no data>
        preserveSyncCtx = <no data>
    LOCALS:
        <no data>
        <no data>
        <no data>

000000d7b927ee80 00007ffe53e345d4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 902]
    PARAMETERS:
        executionContext = <no data>
        callback = <no data>
        state = <no data>
        preserveSyncCtx = <no data>

000000d7b927eeb0 00007ffe53e345a2 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 891]
    PARAMETERS:
        executionContext = <no data>
        callback = <no data>
        state = <no data>

000000d7b927ef00 00007ffe53ebcf62 System.Threading.ThreadHelper.ThreadStart() [f:\dd\ndp\clr\src\BCL\system\threading\thread.cs @ 111]
    PARAMETERS:
        this = <no data>

000000d7b927f158 00007ffe55325a03 [GCFrame: 000000d7b927f158] 
000000d7b927f4a8 00007ffe55325a03 [DebuggerU2MCatchHandlerFrame: 000000d7b927f4a8]

能看到部分參數的內存地址，可是不少是no data，這樣的話，咱們嘗試用另外一個命令，查詢線程上下文中的全部變量信息：　

!dso

0:032> !dso
OS Thread Id: 0x15968 (32)
RSP/REG          Object           Name
000000D7B927E8D0 000000d79aedb7a8 System.Net.Sockets.SocketException
000000D7B927E948 000000d79aedb7a8 System.Net.Sockets.SocketException
000000D7B927E9A8 000000d79aedb7a8 System.Net.Sockets.SocketException
000000D7B927E9B0 000000d79aedb7a8 System.Net.Sockets.SocketException
000000D7B927E9E0 000000d79aedb7a8 System.Net.Sockets.SocketException
000000D7B927E9F0 000000d79aedb7a8 System.Net.Sockets.SocketException
000000D7B927EA78 000000d79aedb850 System.Text.StringBuilder
000000D7B927EAC0 000000d79aedb850 System.Text.StringBuilder
000000D7B927EAD0 000000d79aedb7a8 System.Net.Sockets.SocketException
000000D7B927EAD8 000000d79aedb7a8 System.Net.Sockets.SocketException
000000D7B927EB20 000000d79aedb7a8 System.Net.Sockets.SocketException
000000D7B927EB30 000000d79a3c2ed8 System.Net.Sockets.NetworkStream
000000D7B927EB60 000000d79a809eb0 System.Byte[]
000000D7B927EB70 000000d799fd1420 System.String    
000000D7B927EB78 000000d79aeda2e8 RabbitMQ.Client.Framing.BasicProperties
000000D7B927EB80 000000d79aeda568 System.Byte[]
000000D7B927EBB0 000000d79a017a90 System.Threading.ContextCallback
000000D7B927EBB8 000000d79a3c2f18 System.IO.BufferedStream
000000D7B927EBC0 000000d79a3c2f70 RabbitMQ.Util.NetworkBinaryReader
000000D7B927EBC8 000000d79a3c3ff0 System.Threading.ThreadHelper
000000D7B927EBF0 000000d79aeda0a8 System.String    HSF-ServiceState
000000D7B927EBF8 000000d799fd1420 System.String    
000000D7B927EC00 000000d79a3c2f70 RabbitMQ.Util.NetworkBinaryReader
000000D7B927EC10 000000d79aeda2e8 RabbitMQ.Client.Framing.BasicProperties
000000D7B927EC20 000000d79aeda2e8 RabbitMQ.Client.Framing.BasicProperties
000000D7B927EC28 000000d79a017a90 System.Threading.ContextCallback
000000D7B927EC30 000000d79a3c6a00 RabbitMQ.Client.Framing.Impl.Model
000000D7B927EC50 000000d79a3c3ff0 System.Threading.ThreadHelper
000000D7B927EC78 000000d79a017a90 System.Threading.ContextCallback
000000D7B927EC80 000000d79a3c2bc0 RabbitMQ.Client.Impl.SocketFrameHandler
000000D7B927EC88 000000d79a3c2f70 RabbitMQ.Util.NetworkBinaryReader
000000D7B927EC90 000000d79a3c3ff0 System.Threading.ThreadHelper
000000D7B927ECB0 000000d79a017a90 System.Threading.ContextCallback
000000D7B927ECC0 000000d79a3c6688 RabbitMQ.Client.Impl.Session
000000D7B927ECC8 000000d79aedb430 RabbitMQ.Client.Impl.Frame
000000D7B927ECD8 000000d79a3c2f70 RabbitMQ.Util.NetworkBinaryReader
000000D7B927ECF0 000000d79a3c31b8 RabbitMQ.Client.Framing.Impl.Connection
000000D7B927ED80 000000d79a017a90 System.Threading.ContextCallback
000000D7B927EDB0 000000d79a3c31b8 RabbitMQ.Client.Framing.Impl.Connection
000000D7B927EDF0 000000d79a3c3f90 System.Threading.Thread
000000D7B927EE48 000000d79a3c3ff0 System.Threading.ThreadHelper
000000D7B927EE50 000000d79a3c4150 System.Threading.ExecutionContext
000000D7B927EE58 000000d79a017a90 System.Threading.ContextCallback
000000D7B927EEE8 000000d79a3c3ff0 System.Threading.ThreadHelper
000000D7B927EEF0 000000d79a3c4150 System.Threading.ExecutionContext
000000D7B927F100 000000d79a3c4018 System.Threading.ThreadStart

 線程上下文中全部的變量的內存地址，咱們都能看到，接下來看每一個對象的信息，你們應該都會用了  !do objectAddress， 若是加載的Mex插件，能夠用牛逼的  !do2

 

0:032> !do2 000000d79a3c6688 
0x000000d79a3c6688 RabbitMQ.Client.Impl.Session
  0000  _shutdownLock                    : 000000d79a3c66d0 (System.Object)
  0008  _sessionShutdown                 : 000000d79a3c6cf8 (System.EventHandler<RabbitMQ.Client.ShutdownEventArgs>)
  0010  <CloseReason>k__BackingField     : NULL
  0018  <CommandReceived>k__BackingField : 000000d79a3c6c50 (System.Action<RabbitMQ.Client.Impl.ISession,RabbitMQ.Client.Impl.Command>)
  0020  <Connection>k__BackingField      : 000000d79a3c31b8 (RabbitMQ.Client.Framing.Impl.Connection)
  0028  <ChannelNumber>k__BackingField   : 1 (System.Int32)
  0030  m_assembler                      : 000000d79a3c6790 (RabbitMQ.Client.Impl.CommandAssembler)
0:032> !do 000000d79a3c6688 
Name:        RabbitMQ.Client.Impl.Session
MethodTable: 00007ffdf6257888
EEClass:     00007ffdf6264ce8
Size:        72(0x48) bytes
File:        C:\TeldApp\HSF\HSF.Host-BillCM-9086\RabbitMQ.Client.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ffe53fd6fd8  4000283        8        System.Object  0 instance 000000d79a3c66d0 _shutdownLock
00007ffdf6250760  4000284       10 ...RabbitMQ.Client]]  0 instance 000000d79a3c6cf8 _sessionShutdown
00007ffe53fd9428  4000285       30         System.Int32  1 instance                1 <ChannelNumber>k__BackingField
00007ffdf62506c0  4000286       18 ...ShutdownEventArgs  0 instance 0000000000000000 <CloseReason>k__BackingField
00007ffdf6272e78  4000287       20 ...RabbitMQ.Client]]  0 instance 000000d79a3c6c50 <CommandReceived>k__BackingField
00007ffdf62549a0  4000288       28 ...g.Impl.Connection  0 instance 000000d79a3c31b8 <Connection>k__BackingField
00007ffdf62713b0  4000289       38 ....CommandAssembler  0 instance 000000d79a3c6790 m_assembler

咱們繼續看Connection對象：000000d79a3c31b8 

0:032> !do 000000d79a3c31b8 
Name:        RabbitMQ.Client.Framing.Impl.Connection
MethodTable: 00007ffdf62549a0
EEClass:     00007ffdf6260b78
Size:        232(0xe8) bytes
File:        C:\TeldApp\HSF\HSF.Host-BillCM-9086\RabbitMQ.Client.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ffe53fd6fd8  4000247        8        System.Object  0 instance 000000d79a3c32a0 m_eventLock
00007ffdf6257c48  4000248       10 ...Client.Impl.Frame  0 instance 000000d79a3c32b8 m_heartbeatFrame
00007ffe53fe7378  4000249       18 ....ManualResetEvent  0 instance 000000d79a3c32f8 m_appContinuation
0000000000000000  400024a       20                       0 instance 0000000000000000 m_callbackException
00007ffe53954820  400024b       28 ...bject, mscorlib]]  0 instance 000000d79a3c51d8 m_clientProperties
00007ffdf62506c0  400024c       30 ...ShutdownEventArgs  0 instance 0000000000000000 m_closeReason
00007ffe53ff9a38  400024d       c2       System.Boolean  1 instance                0 m_closed
0000000000000000  400024e       38                       0 instance 0000000000000000 m_connectionBlocked
00007ffdf6250760  400024f       40 ...RabbitMQ.Client]]  0 instance 000000d79a3c6750 m_connectionShutdown
00007ffe53953ff0  4000250       48 ...tArgs, mscorlib]]  0 instance 0000000000000000 m_connectionUnblocked
00007ffdf622f7d0  4000251       50 ...ConnectionFactory  0 instance 000000d79a3c2698 m_factory
00007ffdf6255cb8  4000252       58 ...mpl.IFrameHandler  0 instance 000000d79a3c2bc0 m_frameHandler
00007ffe53ff40a0  4000253       c8          System.Guid  1 instance 000000d79a3c3280 m_id
00007ffdf625a5d0  4000254       60 ...nt.Impl.ModelBase  0 instance 000000d79a3c3860 m_model0
00007ffe53ff9a38  4000255       c3       System.Boolean  1 instance                1 m_running
00007ffdf6257a48  4000256       68 ....Impl.MainSession  0 instance 000000d79a3c35f0 m_session0
00007ffdf6259050  4000257       70 ...pl.SessionManager  0 instance 000000d79a3c6018 m_sessionManager
00007ffdf6258068  4000258       78 ...RabbitMQ.Client]]  0 instance 000000d79a3c3370 m_shutdownReport
00007ffe53fd5570  4000259       c0        System.UInt16  1 instance               60 m_heartbeat
00007ffe53ff3f28  400025a       d8      System.TimeSpan  1 instance 000000d79a3c3290 m_heartbeatTimeSpan
00007ffe53fd9428  400025b       b8         System.Int32  1 instance                0 m_missedHeartbeats
00007ffe53ff4248  400025c       80 ...m.Threading.Timer  0 instance 000000d79a3c61f0 _heartbeatWriteTimer
00007ffe53ff4248  400025d       88 ...m.Threading.Timer  0 instance 000000d79a3c6428 _heartbeatReadTimer
00007ffe53ff4de8  400025e       90 ...ng.AutoResetEvent  0 instance 000000d79a3c33a8 m_heartbeatRead
00007ffe53ff4de8  400025f       98 ...ng.AutoResetEvent  0 instance 000000d79a3c33f8 m_heartbeatWrite
00007ffe53ff9a38  4000260       c4       System.Boolean  1 instance                0 m_inConnectionNegotiation
00007ffdf6258e90  4000261       a0 ...nsumerWorkService  0 instance 000000d79a3c3448 <ConsumerWorkService>k__BackingField
00007ffe53ff21f0  4000262       bc        System.UInt32  1 instance           131072 <FrameMax>k__BackingField
00007ffdf625ace0  4000263       a8 ...AmqpTcpEndpoint[]  0 instance 0000000000000000 <KnownHosts>k__BackingField
00007ffe53954820  4000264       b0 ...bject, mscorlib]]  0 instance 000000d79a3c5468 <ServerProperties>k__BackingField
0:032> !do2 000000d79a3c31b8 
0x000000d79a3c31b8 RabbitMQ.Client.Framing.Impl.Connection
  0000  m_eventLock                          : 000000d79a3c32a0 (System.Object)
  0008  m_heartbeatFrame                     : 000000d79a3c32b8 (RabbitMQ.Client.Impl.Frame)
  0010  m_appContinuation                    : 000000d79a3c32f8 (System.Threading.ManualResetEvent)
  0018  m_callbackException                  : NULL
  0020  m_clientProperties                   : 000000d79a3c51d8 (System.Collections.Generic.Dictionary<System.String,System.Object>)
  0028  m_closeReason                        : NULL
  0030  m_connectionBlocked                  : NULL
  0038  m_connectionShutdown                 : 000000d79a3c6750 (System.EventHandler<RabbitMQ.Client.ShutdownEventArgs>)
  0040  m_connectionUnblocked                : NULL
  0048  m_factory                            : 000000d79a3c2698 (RabbitMQ.Client.ConnectionFactory)
  0050  m_frameHandler                       : 000000d79a3c2bc0 (RabbitMQ.Client.Impl.SocketFrameHandler)
  0058  m_model0                             : 000000d79a3c3860 (RabbitMQ.Client.Framing.Impl.Model)
  0060  m_session0                           : 000000d79a3c35f0 (RabbitMQ.Client.Impl.MainSession)
  0068  m_sessionManager                     : 000000d79a3c6018 (RabbitMQ.Client.Impl.SessionManager)
  0070  m_shutdownReport                     : 000000d79a3c3370 (RabbitMQ.Util.SynchronizedList<RabbitMQ.Client.ShutdownReportEntry>)
  0078  _heartbeatWriteTimer                 : 000000d79a3c61f0 (System.Threading.Timer)
  0080  _heartbeatReadTimer                  : 000000d79a3c6428 (System.Threading.Timer)
  0088  m_heartbeatRead                      : 000000d79a3c33a8 (System.Threading.AutoResetEvent)
  0090  m_heartbeatWrite                     : 000000d79a3c33f8 (System.Threading.AutoResetEvent)
  0098  <ConsumerWorkService>k__BackingField : 000000d79a3c3448 (RabbitMQ.Client.ConsumerWorkService)
  00a0  <KnownHosts>k__BackingField          : NULL
  00a8  <ServerProperties>k__BackingField    : 000000d79a3c5468 (System.Collections.Generic.Dictionary<System.String,System.Object>)
  00b0  m_missedHeartbeats                   : 0 (System.Int32)
  00b4  <FrameMax>k__BackingField            : 131072 (System.UInt32)
  00b8  m_heartbeat                          : 60 (System.UInt16)
  00ba  m_closed                             : False (System.Boolean)
  00bb  m_running                            : True (System.Boolean)
  00bc  m_inConnectionNegotiation            : False (System.Boolean)
  00c0  m_id                                 : 000000d79a3c3280 1c8d8e96-0c6a-4c62-84b7-9feb241b67ad (System.Guid)
  00d0  m_heartbeatTimeSpan                  : 000000d79a3c3290 00:00:15 (System.TimeSpan)

相似的，其餘的內存變量也能夠看，幫助咱們分析問題。

找到問題的大體緣由後，咱們須要禁用異常捕獲，退出調試，若是直接關閉Windbg，此時附加調試的進程就被強制退出了，必定要注意，請使用下面的指令：

6. 禁用異常捕獲 sxd clr
7. 退出調試 qd

sxd  clr
qd

以上就是跟你們分享的附加進程實時調試，總結一下套路：

  1. F6 Attache進程
  2. 加載SOS, .loadby sos clr
  3. 啓用異常捕獲 sxe clr
  4. 運行，捕獲&查看異常 g  !pe
  5. 查看異常所在線程堆棧!clrstack, 查看異常所在線程堆棧的參數和變量 !clrstack -a / !dso
  6. 禁用異常捕獲 sxd clr
  7. 退出調試 qd

分享給你們。

 

周國慶

2018/11/4

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程