Linux性能分析

生產環境服務器變慢,診斷思路和性能評估

整機:top

代碼java

public class JavaDemo2 {
    public static void main(String[] args) {
        while (true){
            System.out.println(new java.util.Random().nextInt(77778888));
        }
    }
}

top命令查看ios

[root@192 ~]# top
top - 11:28:50 up 7 min,  2 users,  load average: 0.73, 0.39, 0.18
Tasks: 230 total,   1 running, 229 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  2.3%sy,  0.0%ni, 96.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1906280k total,   649948k used,  1256332k free,    21164k buffers
Swap:  2097148k total,        0k used,  2097148k free,   203820k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                              
  3672 root      20   0 3158m  94m  11m S 15.6  5.1   0:13.67 java                                                                                 
  3397 root      20   0  100m 5388 3396 S  9.3  0.3   0:09.00 sshd                                                                                 
    39 root      20   0     0    0    0 S  1.7  0.0   0:00.25 events/4                                                                             
    37 root      20   0     0    0    0 S  1.0  0.0   0:00.39 events/2                                                                             
    35 root      20   0     0    0    0 S  0.3  0.0   0:01.27 events/0                                                                             
    36 root      20   0     0    0    0 S  0.3  0.0   0:00.34 events/1                                                                             
    41 root      20   0     0    0    0 S  0.3  0.0   0:00.24 events/6   
[root@192 ~]# uptime
11:29:09 up 7 min,  2 users,  load average: 0.52, 0.36, 0.18

CPU:vmstat

[root@192 ~]# vmstat -n 2  3
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1313028  21228 203868    0    0    54     3  112 1620  0  1 98  0  0       
 0  0      0 1312732  21228 203896    0    0     0     0 2579 25940  1  3 96  0  0      
 0  0      0 1312856  21228 203896    0    0     0     0 2677 24290  1  3 97  0  0

1564976316953

id:處於空閒的CPU百分比c++

wa:系統等待IO的CPU時間百分比服務器

st:來自於一個虛擬機偷取的CPU時間百分比網絡

查看額外的CPU信息dom

全部CPU核信息ssh

[root@192 ~]# mpstat -P  ALL 2
Linux 2.6.32-642.el6.x86_64 (192.168.1.101)     08/05/2019      _x86_64_        (8 CPU)


11:42:40 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
11:42:42 AM  all    1.07    0.00    2.89    0.00    0.00    0.00    0.00    0.00   96.04
11:42:42 AM    0    0.00    0.00    1.52    0.00    0.00    0.00    0.00    0.00   98.48
11:42:42 AM    1    6.09    0.00    9.14    0.00    0.00    0.00    0.00    0.00   84.77
11:42:42 AM    2    0.50    0.00    4.02    0.00    0.00    0.00    0.00    0.00   95.48
11:42:42 AM    3    1.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00   97.00

每一個進程使用CPU的用量分解信息性能

[root@192 ~]# pidstat -u 1 -p 3790
Linux 2.6.32-642.el6.x86_64 (192.168.1.101)     08/05/2019      _x86_64_        (8 CPU)

11:45:00 AM       PID    %usr %system  %guest    %CPU   CPU  Command
11:45:01 AM      3790    6.00   10.00    0.00   16.00     7  java
11:45:02 AM      3790    4.00   11.00    0.00   15.00     7  java
11:45:03 AM      3790    9.00   15.00    0.00   24.00     7  java
11:45:04 AM      3790    7.00   14.00    0.00   21.00     7  java

內存:free

應用程序可用內存數flex

推薦使用free -m線程

[root@192 ~]# free -g
             total       used       free     shared    buffers     cached
Mem:             1          0          1          0          0          0
-/+ buffers/cache:          0          1
Swap:            1          0          1
[root@192 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          1861        553       1308          1         20        199
-/+ buffers/cache:        333       1528
Swap:         2047          0       2047

1564976920565

查看額外

[root@192 ~]# pidstat -p 3823 -r 2
Linux 2.6.32-642.el6.x86_64 (192.168.1.101)     08/05/2019      _x86_64_        (8 CPU)

11:51:16 AM       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
11:51:18 AM      3823      1.00      0.00 3234728  47068   2.47  java
11:51:20 AM      3823      2.50      0.00 3234728  47068   2.47  java
11:51:22 AM      3823      0.50      0.00 3234728  47068   2.47  java
11:51:24 AM      3823      2.50      0.00 3234728  47068   2.47  java
11:51:26 AM      3823      0.50      0.00 3234728  47068   2.47  java

1564977136597

硬盤:df

查看磁盤剩餘空間

[root@192 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        15G  5.6G  8.4G  41% /
tmpfs           931M   72K  931M   1% /dev/shm
/dev/sda1       190M   39M  142M  22% /boot

磁盤IO:iostat

[root@192 ~]# iostat -xdk 2 3
Linux 2.6.32-642.el6.x86_64 (192.168.1.101)     08/05/2019      _x86_64_        (8 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
scd0              0.00     0.00    0.38    0.00     1.50     0.00     8.00     0.00    0.67    0.67    0.00   0.67   0.03
sda              34.33     4.94   47.36    3.18  1631.23    32.44    65.84     0.07    1.38    1.22    3.72   0.88   4.43

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.50    0.00     4.00     0.00    16.00     0.00    1.00    1.00    0.00   1.00   0.05

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

1564985628042

查看額外

[root@192 ~]# pidstat -d 2 -p 3485
Linux 2.6.32-642.el6.x86_64 (192.168.1.101)     08/05/2019      _x86_64_        (8 CPU)

02:16:10 PM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
02:16:12 PM      3485      0.00      0.00      0.00  java
02:16:14 PM      3485      0.00      0.00      0.00  java
02:16:16 PM      3485      0.00      0.00      0.00  java
02:16:18 PM      3485      0.00      0.00      0.00  java
02:16:20 PM      3485      0.00      0.00      0.00  java
02:16:22 PM      3485      0.00      0.00      0.00  java
02:16:24 PM      3485      0.00      0.00      0.00  java

1564985817582

網絡IO:ifstat

默認本地沒有,須要下載ifstat

安裝ifstat

[root@192 ifstat-1.1]# yum install flex byacc  libpcap ncurses ncurses-devel libpcap-devel gcc-c++
wget http://gael.roualland.free.fr/ifstat/ifstat-1.1.tar.gz
tar xzvf ifstat-1.1.tar.gz
cd ifstat-1.1
./configure
make
make install
[root@192 ifstat-1.1]# ifstat 1
       eth0       
 KB/s in  KB/s out
    0.06      0.13
    0.06      0.12
    0.06      0.12
    0.06      0.12
    0.06      0.12
    0.06      0.12
    0.12      0.17
    0.06      0.12

生產環境出現CPU佔用太高,分析思路和定位

public class JavaDemo2 {
    public static void main(String[] args) {
        while (true){
            System.out.println(new java.util.Random().nextInt(77778888));
        }
    }
}

先用top命令找出CPU佔比最高的

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                              
  4294 root      20   0 3158m  37m  11m S 16.6  2.0   0:03.28 java                                                                                 
  3428 root      20   0  100m 5444 3396 S 14.3  0.3   0:22.85 sshd                                                                                 
    35 root      20   0     0    0    0 S  1.0  0.0   0:03.18 events/0                                                                             
    41 root      20   0     0    0    0 S  0.3  0.0   0:00.52 events/6                                                                             
  2012 root      20   0  199m 5192 4288 S  0.3  0.3   0:00.32 ManagementAgent                                                                      
     1 root      20   0 19344 1544 1228 S  0.0  0.1   0:01.83 init                                                                                 
     2 root      20   0     0    0    0 S  0.0  0.0   0:00.02 kthreadd

ps -ef或者jps進一步定位

[root@192 ifstat-1.1]# jps
4320 Jps
4294 JavaDemo2

定位到具體線程或者代碼

[root@192 ifstat-1.1]# ps -mp 4294 -o THREAD,tid,time
USER     %CPU PRI SCNT WCHAN  USER SYSTEM    TID     TIME
root     18.6   -    - -         -      -      - 00:00:15
root      0.0  19    - futex_    -      -   4294 00:00:00
root     15.6  19    - n_tty_    -      -   4295 00:00:13
root      0.2  19    - futex_    -      -   4296 00:00:00
root      0.2  19    - futex_    -      -   4297 00:00:00
root      0.3  19    - futex_    -      -   4298 00:00:00
root      0.2  19    - futex_    -      -   4299 00:00:00
root      0.3  19    - futex_    -      -   4300 00:00:00
root      0.3  19    - futex_    -      -   4301 00:00:00
root      0.3  19    - futex_    -      -   4302 00:00:00
root      0.2  19    - futex_    -      -   4303 00:00:00
root      0.0  19    - futex_    -      -   4304 00:00:00
root      0.0  19    - futex_    -      -   4305 00:00:00
root      0.0  19    - futex_    -      -   4306 00:00:00
root      0.0  19    - futex_    -      -   4307 00:00:00
root      0.0  19    - futex_    -      -   4308 00:00:00
root      0.1  19    - futex_    -      -   4309 00:00:00
root      0.0  19    - futex_    -      -   4310 00:00:00
root      0.0  19    - futex_    -      -   4311 00:00:00
root      0.0  19    - futex_    -      -   4312 00:00:00
root      0.0  19    - futex_    -      -   4313 00:00:00

-m:顯示全部的線程

-p: pid進程使用CPU的時間

-o:該參數後是用戶自定義格式

線程ID轉換爲16進制格式(英文小寫格式)

[root@192 ifstat-1.1]# printf "%x\n" 4295
10c7

jstack 進程ID|grep tid(16進制線程ID小寫英文) -A60

[root@192 ifstat-1.1]# jstack 4294 | grep 10c7 -A60
"main" #1 prio=5 os_prio=0 tid=0x00007f7f68009000 nid=0x10c7 runnable [0x00007f7f6f117000]
   java.lang.Thread.State: RUNNABLE
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:326)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        - locked <0x00000000e2e0e6e0> (a java.io.BufferedOutputStream)
        at java.io.PrintStream.write(PrintStream.java:482)
        - locked <0x00000000e2e0bbd8> (a java.io.PrintStream)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        - locked <0x00000000e2e0e800> (a java.io.OutputStreamWriter)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.newLine(PrintStream.java:546)
        - eliminated <0x00000000e2e0bbd8> (a java.io.PrintStream)
        at java.io.PrintStream.println(PrintStream.java:737)
        - locked <0x00000000e2e0bbd8> (a java.io.PrintStream)
        at JavaDemo2.main(JavaDemo2.java:4)

"VM Thread" os_prio=0 tid=0x00007f7f68080000 nid=0x10d0 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f7f6801e800 nid=0x10c8 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f7f68020000 nid=0x10c9 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f7f68022000 nid=0x10ca runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f7f68024000 nid=0x10cb runnable 

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f7f68025800 nid=0x10cc runnable 

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f7f68027800 nid=0x10cd runnable 

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f7f68029800 nid=0x10ce runnable 

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f7f6802b000 nid=0x10cf runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007f7f680e8000 nid=0x10d9 waiting on condition 

JNI global references: 5

參考

找到最耗CPU的java線程 - - ITeye博客

本文由博客一文多發平臺 OpenWrite 發佈!

相關文章
相關標籤/搜索