Linux監控工具介紹系列——OSWatcher Black Box

 

OSWatcher Balck Box簡介

 

OSWatcher Black Box (oswbb)是Oracle開發、提供的一個小巧，可是實用、強大的系統工具，它能夠用來抓取操做系統的性能指標，用於輔助監控系統的資源使用。其安裝部署、卸載都很是簡單；資源消耗也比較小，原理也十分簡單，它經過調用OS的的一些命令（例如vmstat、iostat等）來採集、存儲CPU/Memory/Swap/Disk IO/Nentwork相關數據。安裝和運行oswbba能夠幫助在性能診斷時提供豐富多樣的各種性能數據、圖文報表支持。

 

OSWatcher 在4.0的版本時被命名爲OSWatcher Black Box,簡稱爲oswbb，同時增長了數據分析功能，即OSWatcher Black Box Analyzer (OSWbba)這個繪圖和分析工具，其捆綁在 OS Watcher Black Box當中。替代了以前的OSWg。 也就是說在OSWatcher 4.0 以前是：OSWatcher 和 OSWg的關係，OSWatcher 4.0 後變成了： OSWbb 與 OSWbba 的關係。

 

OSWatcher Black Box（oswbb）支持多個操做系統，也分Linux與Window版本，固然這兩個版本有所差異，本篇只講述Linux版本。另外，OSWatcher Black Box（oswbb）由兩個部分組成：

 

1. oswbb: 一個Unix的 shell script腳本集合，其用來收集和歸檔數據，從而幫助定位問題。

 

2. oswbba: 一個Java工具來自動分析數據，提供建議，而且生成一個包含圖形的 html 文檔。

 

OSWatcher Black Box（oswbb）的官方下載地址以及相關資料以下（Oracle Metalink上的資料須要帳號）

 

How To Start OSWatcher Black Box (OSWBB) Every System Boot Using RPM oswbb-service(文檔 ID 580513.1)

OSWatcher Analyzer User Guide (文檔 ID 461053.1)

 

 

    官方介紹文檔對oswbb，oswbba的介紹以下：

 

OSWatcher (oswbb) is a collection of UNIX shell scripts intended to collect and archive operating system and network metrics to aid support in diagnosing performance issues. OSWatcher operates as a set of background processes on the server and gathers OS data on a regular basis, invoking such Unix utilities as vmstat, netstat and iostat. OSWatcher can be downloaded from this note. OSWatcher is also included in the RAC-DDT script file, but is not installed by RAC-DDT. For more information on RAC-DDT see RAC-DDT User Guide. OSWatcher is installed on each node where data is to be collected. Installation instructions for OSWatcher are provided in this user guide.

 

The OSWatcher Analyzer (oswbba) is a graphing and analysis utility which comes bundled with OSWatcher v4.0 and higher. oswbba allows the user to graphically display data collected, generate reports containing these graphs and provides a built in analyzer to analyze the data and provide details on any performance problems it detects. The ability to graph and analyze this information relieves the user of manually inspecting all the files.

 

NOTE: oswbba replaces the utility OSWg. This was done to eliminate the confusion caused by having multiple tools in support named OSWatcher. oswbba is only supported for data collected by oswbb and no other tool.

 

 

安裝OSWatcher

 

Linux平臺的安裝簡單到了不能再簡單，以下所示，解壓安裝包生成oswbb文件夾。固然你最好將其部署或移動到合適的目錄。

 

[oracle@DB-Server monitoring]$ tar -xvf oswbb734.tar

[oracle@DB-Server monitoring]$ ls -lrt

total 6196

drwxr-xr-x 6 oracle oinstall    4096 Jul 25 22:22 oswbb

-rw-r--r-- 1 oracle oinstall 6318080 Nov  8 02:33 oswbb734.tar

[oracle@DB-Server monitoring]$ cd oswbb

[oracle@DB-Server oswbb]$ ls –lrt

 

 

 

卸載OSWatcher

 

 

卸載OSWatche也是很是簡單，只要稍懂Linux的應該都會。

 

1：卸載前先中止OSWatcher

 

./stopOSWbb.sh

 

2：刪除OSWatcher的目錄oswbb

 

   rm –fr oswbb

 

 

 

啓動OSWatcher

 

啓動OSWatcher也很是簡單，只須要執行startOSWbb.sh腳本便可，以下所示

 

 

  ./startOSWbb.sh 10  2

 

後面參數表示10秒採集一次數據，只保留最後採集2個小時的數據在歸檔文件中。首次啓動的日誌最好作一下檢查，注意是否存在Warning信息，以下測試所示，檢查時發現ifconfig命令找不到（oracle帳戶下沒法運行ifconfig命令）

[oracle@DB-Server oswbb]$ ./startOSWbb.sh 10 2

[oracle@DB-Server oswbb]$ Setting the archive log directory to/home/oracle/monitoring/oswbb/archive

Testing for discovery of OS Utilities...

VMSTAT found on your system.

IOSTAT found on your system.

MPSTAT found on your system.

Warning... IFCONFIG not found on your system. No IFCONFIG data will be collected.

NETSTAT found on your system.

TOP found on your system.

Testing for discovery of OS CPU COUNT

oswbb is looking for the CPU COUNT on your system

CPU COUNT will be used by oswbba to automatically look for cpu problems

CPU COUNT found on your system.

CPU COUNT = 4

Discovery completed.

Starting OSWatcher v7.3.3  on Sun Dec 4 08:01:57 EST 2016

With SnapshotInterval = 10

With ArchiveInterval = 2

OSWatcher - Written by Carl Davis, Center of Expertise,

Oracle Corporation

For questions on install/usage please go to MOS (Note:301137.1)

If you need further assistance or have comments or enhancement

requests you can email me Carl.Davis@Oracle.com

Data is stored in directory: /home/oracle/monitoring/oswbb/archive

Starting Data Collection...

oswbb heartbeat:Sun Dec 4 08:02:02 EST 2016

oswbb heartbeat:Sun Dec 4 08:02:12 EST 2016

 

 ./startOSWbb.sh

 

若是沒有指定參數，默認參數爲30,48，意味着30秒採集一次數據，只保留最後48小時的數據到歸檔文件當中。其實startOSWbb.sh能夠定義四個參數

 

參數1： 指定多少秒採集一次數據。

參數2： 指定採集的數據文件在歸檔路徑保留多少個小時

參數3： 可選參數，打包壓縮工具， 在完成收集後OSW 將使用其來打包壓縮歸檔文件

參數4： 可選參數，指定採集歸檔數據的輸出目錄，默認爲系統變量OSWBB_ARCHIVE_DEST的值。

 

 

 

使用上面方式啓動OSWatcher，會被輸出信息一直刷屏。基本上不多使用這種方式，通常使用nohup啓動，這樣可讓OSW可以在後臺持續運行並在當前會話終止後不會被掛斷。固然若是系統重啓，那麼OSWatcher就會中止。

 

 nohup ./startOSWbb.sh 30 48 &

 

 

第一次啓動OSWatcher會在oswbb目錄下建立 gif、archive、tmp、locks目錄。其歸檔文件夾和osw<工具名> 子文件夾會被建

 

 

 

OSWatcher採集的數據都存放在archive（目錄路徑）下，下面會生成oswiostat、oswmeminfo、oswmpstat等10個目錄

 

[oracle@DB-Server archive]$ tree

.

|-- oswifconfig

|-- oswiostat

|   |-- DB-Server.localdomain_iostat_16.12.09.1300.dat

|   |-- DB-Server.localdomain_iostat_16.12.09.1400.dat

|   `-- DB-Server.localdomain_iostat_16.12.09.1500.dat

|-- oswmeminfo

|   |-- DB-Server.localdomain_meminfo_16.12.09.1300.dat

|   |-- DB-Server.localdomain_meminfo_16.12.09.1400.dat

|   `-- DB-Server.localdomain_meminfo_16.12.09.1500.dat

|-- oswmpstat

|   |-- DB-Server.localdomain_mpstat_16.12.09.1300.dat

|   |-- DB-Server.localdomain_mpstat_16.12.09.1400.dat

|   `-- DB-Server.localdomain_mpstat_16.12.09.1500.dat

|-- oswnetstat

|   |-- DB-Server.localdomain_netstat_16.12.09.1300.dat

|   |-- DB-Server.localdomain_netstat_16.12.09.1400.dat

|   `-- DB-Server.localdomain_netstat_16.12.09.1500.dat

|-- oswprvtnet

|-- oswps

|   |-- DB-Server.localdomain_ps_16.12.09.1300.dat

|   |-- DB-Server.localdomain_ps_16.12.09.1400.dat

|   `-- DB-Server.localdomain_ps_16.12.09.1500.dat

|-- oswslabinfo

|   |-- DB-Server.localdomain_slabinfo_16.12.09.1300.dat

|   |-- DB-Server.localdomain_slabinfo_16.12.09.1400.dat

|   `-- DB-Server.localdomain_slabinfo_16.12.09.1500.dat

|-- oswtop

|   |-- DB-Server.localdomain_top_16.12.09.1300.dat

|   |-- DB-Server.localdomain_top_16.12.09.1400.dat

|   `-- DB-Server.localdomain_top_16.12.09.1500.dat

`-- oswvmstat

    |-- DB-Server.localdomain_vmstat_16.12.09.1300.dat

    |-- DB-Server.localdomain_vmstat_16.12.09.1400.dat

    `-- DB-Server.localdomain_vmstat_16.12.09.1500.dat

 

10 directories, 24 files

 

採集的數據文件命名格式爲OSWatcher 輸出文件格式爲：<節點名>_<操做系統工具名>_YY.MM.DD.HH24.dat  。以下所示，

 

 

 

配置OSWatcher自啓動

 

OSWatcher在系統重啓事後，是沒法自動重啓的，若是須要設置OSWatcher開機自啓動，須要安裝、配置osw-service這個RPM包。這個安裝包能夠從How To Start OSWatcher Black Box (OSWBB) Every System Boot Using RPM oswbb-service (文檔 ID 580513.1)出下載。

 

[root@DB-Server tmp]# rpm -ivh oswbb-service-7.2.0-1.noarch.rpm

Preparing...                ########################################### [100%]

   1:oswbb-service          ########################################### [100%]

[root@DB-Server tmp]#

 

安裝osw-service這個RPM包後，須要配置/etc/oswbb.conf文件後，才能配置OSWatcher自啓動。以下所示：

 

 

[root@DB-Server ~]#

[root@DB-Server ~]# /sbin/chkconfig oswbb on

[root@DB-Server ~]# /sbin/service oswbb start

Starting OSWatcher: [  OK  ]

[root@DB-Server ~]#

 

 

關閉OSWatcher

 

  關閉OSWatcher也是很是簡單，網上有種說法：stopOSWbb.sh 是Oracle惟一支持的用於中止OSW的方法，其實否則，應該說有兩種方法

 

  1：執行腳本stopOSWbb.sh關閉OSWatcher

 

./stopOSWbb.sh

 

  2：kill進程方法 

[root@DB-Server ~]# ps -ef | grep -i OSW

oracle   24863     1  0 16:02 pts/1    00:00:00 /bin/sh ./OSWatcher.sh 30 48

oracle   24904 24863  0 16:03 pts/1    00:00:00 /bin/sh ./OSWatcherFM.sh 48 /home/oracle/monitoring/oswbb/archive

root     25330 18699  0 16:05 pts/2    00:00:00 grep -i osw

[root@DB-Server ~]# kill 24863

[root@DB-Server ~]# kill 24904

[root@DB-Server ~]# ps -ef | grep -i OSW

root     25342 18699  0 16:05 pts/2    00:00:00 grep -i osw

[root@DB-Server ~]# 

 

 

若是你查看stopOSWbb.sh腳本，你會發現它中止OSWatcher也是經過kill OSWatcher program相關進程的方法，以下所示：

 

[oracle@DB-Server oswbb]$ more stopOSWbb.sh 

#!/bin/sh

######################################################################

# stopOSW.sh

# This is the script which terminates all processes associated with

# the OSWatcher program.

######################################################################

# Kill the OSWatcher processes

######################################################################

PLATFORM=`/bin/uname`

case $PLATFORM in

  AIX)

    kill -15 `ps -ef | grep OSWatch | grep -v grep | awk '{print $2}'` 

    ;;

  *)

    kill -15 `ps -e | grep OSWatch | awk '{print $1}'`

    ;;

esac

 

 

OSWatcher bba使用總結

 

OSWatcher now provides an analysis tool oswbba which analyzes the log files produced by OSWatcher. This tool allows OSWatcher to be selfanalyzing.

This tool also provides a graphing capability to graph the data and to produce a html profile. See the "Graphing and Analyzing the Output" section below.

 

oswbba is written in java and requires as a minimum java version 1.4.2 or higher. oswbba can run on any Unix X Windows or PC Windows platform. An X Windows environment is required because oswbba uses Oracle Chartbuilder which requires it.

 

OSWatcher bba 是一個Java語言寫的應用程序，須要安裝Java 1.4.2 或更高的版本。oswbba可以在任何有X Windows的Unix平臺或Windows平臺上運行， X Windows環境是必須的，由於oswbba須要用到Oracle Chartbuilder組件，而這個組件須要它。

 

[root@DB-Server oswbb]# java -version

java version "1.8.0_111"

Java(TM) SE Runtime Environment (build 1.8.0_111-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

[root@DB-Server oswbb]# java -jar oswbba.jar -i /home/oracle/monitoring/oswbb/archive/

或

[root@DB-Server oswbb]# java -jar Xmx512M oswbba.jar -i /home/oracle/monitoring/oswbb/archive/

 

若是你只想生成某個時間段的報表，你可使用參數-B  -E 以下案例所示

 

[root@DB-Server oswbb]#java -jar  -Xmx256m oswbba.jar -i /home/oracle/scripts/oswbb/archive  -B Dec  7 15:30:00 2016  -E Dec 7 17:00:00 2016

 

在處理過程OSWatcher有下面一些參數可供選擇，以下所示。根據具體須要選擇。 生成對應的html報表後，將其下載到本機，你就能夠作一些分析了。

 

 

 

以下截圖所示，這些圖表僅僅是其中部分，OSWatcher給人印象最深的就是豐富的圖表，而不是一堆枯燥的數字。

 

 

 

 

 

 

 

 

固然oswbba也能生成一個很是詳細的分析報告，是文本格式，以下部分截圖所示。

 

 

 

其實使用OSWatcher的使用並無什麼難度，反而是你要理解報表或分析報告中的指標意義，可以結合AWR、ASH以及其餘數據庫診斷信息來分析、診斷問題。這個纔是有難度的。關於具體指標的詳細說明，OSWatcher Analyzer User Guide (文檔 ID 461053.1) 中已有詳細解說，在此就不班門弄斧了。

 

 

 

 

OSWatcher常見應用問題

 

1： 如何判斷OSWatcher是否正在運行

 

    判斷OSWatcher是否正在運行，很是簡單，使用ps命令查看是否存在OSWatcher的相關進程便可。以下所示

 

[oracle@DB-Server ~]$ ps -ef | grep -i OSWatcher

oracle   23532     1  0 08:01 pts/2    00:00:14 /bin/sh ./OSWatcher.sh 10 2

oracle   23587 23532  0 08:02 pts/2    00:00:00 /bin/sh ./OSWatcherFM.sh 2 /home/oracle/monitoring/oswbb/archive

oracle   25808 24564  0 09:22 pts/3    00:00:00 grep --color=auto -i OSWatcher

[oracle@DB-Server ~]$

 

2: 沒有設置JAVA環境變量，會遇到下面錯誤：

 

[oracle@mylnx02 oswbb]$ java -jar oswbba.jar -i /home/oracle/oswbb/archive -B Dec 21 09:00:00 2016  -E Dec 21 10:00:00 2016

Validating times in the archive...

Starting OSW Analyzer V7.3.3

OSWatcher Analyzer Written by Oracle Center of Expertise

Copyright (c)  2014 by Oracle Corporation

Parsing Data. Please Wait...

ERROR. You do not have a legitimate version of java in your PATH.

Linux users please download and install java from java.sun.com or 

see the oswbba README for instructions on how to use the version of java

that comes shipped with the Oracle database.

以下所示，設置JAVA環境變量後，而後執行oswbba的相關命令便可。

 

[oracle@mylnx02 oswbb]$  export PATH=$ORACLE_HOME/jre/1.4.2/bin:$PATH

[oracle@mylnx02 oswbb]$  java -jar oswbba.jar -i /home/oracle/oswbb/archive -B Dec 21 09:00:00 2016  -E Dec 21 10:00:00 2016

Validating times in the archive...

Starting OSW Analyzer V7.3.3

OSWatcher Analyzer Written by Oracle Center of Expertise

Copyright (c)  2014 by Oracle Corporation

Parsing Data. Please Wait...

Scanning file headers for version and platform info...

Parsing file getlnx14.gfg1.esquel.com_iostat_16.12.21.0800.dat ...

Parsing file getlnx14.gfg1.esquel.com_iostat_16.12.21.0900.dat ...

 

 

3：運行"java -jar oswbba.jar ..."命令時報"Exception in thread "main" java.lang.OutOfMemoryError"

 

這個是由於JAVA的堆棧設置過小的緣故，加載到內存中的數據量過於龐大，須要設置參數-Xmx 能夠用-Xmx256m或-Xmx512m 以下所示

 

java -jar  -Xmx256m oswbba.jar -i /home/oracle/scripts/oswbb/archive

 

oswbba parses all the archive files in memory prior to generating graphs or performing an analysis. If you have a large amount of files to parse you may need to allocate more memory in the java heap. If you experience any error messages regarding out of memory such as java.lang.OutOfMemoryError, you may have to increase the size of the java heap. To increase the size of the java heap use the -Xmx flag.

 

 

4：oswbba是否能夠在命令行界面生成html報告文件？ 答案是能夠，固然你會遇到不少雜七雜八問題，下面列舉了一些我遇到的問題

 

1： 」No X11 DISPLAY variable was set, but this program performed an operation which requires it「

 

 

 

若是是root帳號，只要執行export DISPLAY=:0.0命令後，便可在SecureCRT或 命令窗口生成報表文件。以下所示：

 

[root@DB-Server oswbb]# export DISPLAY=:0.0

[root@DB-Server oswbb]# java -jar oswbba.jar -i /home/oracle/monitoring/oswbb/archive/

Starting OSW Analyzer V7.3.3

OSWatcher Analyzer Written by Oracle Center of Expertise

Copyright (c)  2014 by Oracle Corporation

Parsing Data. Please Wait...

Scanning file headers for version and platform info...

Parsing file DB-Server.localdomain_iostat_16.12.04.0800.dat ...

Parsing file DB-Server.localdomain_iostat_16.12.04.0900.dat ...

Parsing file DB-Server.localdomain_vmstat_16.12.04.0800.dat ...

Parsing file DB-Server.localdomain_vmstat_16.12.04.0900.dat ...

Parsing file DB-Server.localdomain_netstat_16.12.04.0800.dat ...

Parsing file DB-Server.localdomain_netstat_16.12.04.0900.dat ...

Parsing file DB-Server.localdomain_top_16.12.04.0800.dat ...

Parsing file DB-Server.localdomain_top_16.12.04.0900.dat ...

Parsing file DB-Server.localdomain_ps_16.12.04.0800.dat ...

Parsing file DB-Server.localdomain_ps_16.12.04.0900.dat ...

Parsing Completed.

Enter 1 to Display CPU Process Queue Graphs

Enter 2 to Display CPU Utilization Graphs

Enter 3 to Display CPU Other Graphs

Enter 4 to Display Memory Graphs

Enter 5 to Display Disk IO Graphs

Enter 6 to Generate All CPU Gif Files

Enter 7 to Generate All Memory Gif Files

Enter 8 to Generate All Disk Gif Files

Enter L to Specify Alternate Location of Gif Directory

Enter T to Alter Graph Time Scale Only (Does not change analysis dataset)

Enter D to Return to Default Graph Time Scale

Enter R to Remove Currently Displayed Graphs

Enter A to Analyze Data

Enter S to Analyze Subset of Data(Changes analysis dataset including graph time scale)

Enter P to Generate A Profile

Enter X to Export Parsed Data to File

Enter Q to Quit Program

Please Select an Option:1

Enter 1 to Display CPU Process Queue Graphs

Enter 2 to Display CPU Utilization Graphs

Enter 3 to Display CPU Other Graphs

Enter 4 to Display Memory Graphs

Enter 5 to Display Disk IO Graphs

Enter 6 to Generate All CPU Gif Files

Enter 7 to Generate All Memory Gif Files

Enter 8 to Generate All Disk Gif Files

Enter L to Specify Alternate Location of Gif Directory

Enter T to Alter Graph Time Scale Only (Does not change analysis dataset)

Enter D to Return to Default Graph Time Scale

Enter R to Remove Currently Displayed Graphs

Enter A to Analyze Data

Enter S to Analyze Subset of Data(Changes analysis dataset including graph time scale)

Enter P to Generate A Profile

Enter X to Export Parsed Data to File

Enter Q to Quit Program

Please Select an Option:P

Enter a unique profile directory name or enter <CR> to accept default name:kkk.html

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Run_Queue.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Block_Queue.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Cpu_Idle.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Cpu_System.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Cpu_User.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Cpu_Wa.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Cpu_Interrupts.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Context_Switches.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Memory_Swap.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Memory_Free.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_Memory_Page_In_Rate.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_ST.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_RPS.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_WPS.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_PB.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_PBTP_1.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_PBTP_2.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_PBTP_3.gif

Generating file profile/DB-Server.localdomain_kkk.html/OSW_profile_files/OSWg_OS_IO_TPS.gif

 

上面使用的是root用戶，若是是oracle用戶，仍是有可能還會遇到下面問題

 

 

 

此時須要切換到root帳號，執行下面命令，而後切換到oracle用戶下執行命令

 

[oracle@DB-Server oswbb]$ su - root

Password: 

[root@DB-Server ~]# export DISPLAY=:0.0

[root@DB-Server ~]# xhost local:oracle

non-network local connections being added to access control list

 

 

 

固然，在SecureCRT中，你不能選擇1-5這幾個選項，不然依然會遇到 Can't connect to X11 window server using ':0.0'這個錯誤。這個只能在圖形化界面才能執行。例如經過VNC來生成。

 

 

 

java -jar  -Xmx256m oswbba.jar -i /home/oracle/oswbb/archive -B Mar 5 16:00:00 2016  -E Mar 5 16:30:00 2016

 

java -jar  -Djava.awt.headless=true -Xmx256m oswbba.jar -i /home/oracle/oswbb/archive -B Mar 5 16:00:00 2016  -E Mar 5 16:30:00 2016

 

 

 

 

參考資料：

 

OSWatcher（包括：[視頻]） (文檔 ID 1526578.1)

OSWatcher (Includes: [Video]) (文檔 ID 301137.1) How To Start OSWatcher Black Box (OSWBB) Every System Boot Using RPM oswbb-service(文檔 ID 580513.1) OSWatcher Analyzer User Guide (文檔 ID 461053.1)