技术流ken

运维拯救世界

Linux系统性能瓶颈分析工具vmstat–技术流ken

vmstat是一个很好用的检测系统性能工具,没有过多的参数,直接一个vmstat命令即可,不过我们一般加上-w表示宽格式输出。然后再附加上侦测时间即可

例如:

[root@ken1 docker]# vmstat -w 3 100

表示每3秒检测一次并输出系统信息,一共输出100次。

这样的格式的命令很好用,接下来我们运行一下这个命令并对输出的数据进行分析

procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
 r  b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st
 1  0            0      1693508          948      1846672    0    0    28    89   84  124   0   1  98   0   0

 

参数讲解


r:The number of runnable processes (running or waiting for run time).
  表示当前运行队列中运行或等待CPU时间片的进程的数目,代表线程处于可运行状态,但CPU还未能执行。大家不要误认为等待CPU时间片意味着这个进程没有运行,实际上某一时刻1个CPU只能有一个进程占用,其他的进程只能排着队等着,此时这些排队等待CPU资源的进程依然是运行状态。这个值如果长期大于系统CPU的逻辑个数,说明CPU不足,需要增加CPU的个数。
b: The number of processes in uninterruptible sleep.
  表示处在非中断睡眠状态的进程数。通俗的说就是表示在等待资源的进程数,比如正在等待I/O或者内存交换等。举个例子,当磁盘读写非常频繁时,写入数据就会非常慢,此时CPU运算很快就结束了,但进程需要把计算的结果写入磁盘,这样进程的任务才算完成,那此时这个进程只能慢慢地等待磁盘了,这样这个进程就是这个b状态。该数值如果长时间大于1,则需要关注一下了。
us: Time spent running non-kernel code.
  表示用户进程消耗的CPU利用率的百分比。us的值比较高时,说明用户进程消耗的CPU时间多,但是如果长期大于50%,就需要考虑优化程序和算法
sy: Time spent running kernel code.
  表示内核进程消耗的CPU时间的百分比,sy的值越高时,说明内核消耗的CPU资源很多。
注意:us+sy:参考值为80%,如果us+sy这个值大于80%说明可能存在CPU资源不足。
id: Time spent idle.  Prior to Linux 2.5.41, this includes IO-wait time.
  CPU空闲时间的百分比。如果这个值很小,表示CPU没有空闲时间,一直处于忙碌状态。
wa: Time spent waiting for IO.  Prior to Linux 2.5.41, included in idle  
  所有可运行状态线程被阻塞在等待IO请求的百分比
cs: The number of context switches per second
  当前kernel system中,发生上下文切换的数目。系统中越多的上下文切换,在处理器的调度管理下,内核将得到更多的工作。
in: The number of interrupts per second, including the clock。
   当前中断被处理的数目
swpd :the amount of virtual memory used. KB
  当前虚拟内存使用的总额(KB)。当空闲内存极少的时候,更多的数据将被转换到交换设备中。
free:the amount of idle memory. KB
  当前内存中可用的空间字节数
buff:the amount of idle memory.
  表示(即将写入磁盘的)缓冲大小,KB
cache:the amount of memory used as cache
  表示(从磁盘中读取的)缓冲大小,KB
si: Amount of memory swapped in from disk (/s)
  从交换区写入内存的字节数总额,KB
so: Amount of memory swapped to disk (/s)
  从内存写入交换区的字节数总额。KB
注意:如果内存不够用了,这两列的数值比较高。说明内存中的数据频繁交换到交换分区swap中,这往往对系统性能影响较大。
bi: Blocks received from a block device (blocks/s)
  读磁盘,KB
bo:Blocks sent to a block device (blocks/s)
  写磁盘,KB
注意:如果磁盘IO压力很大,这两列的数值会比较高。
可以使用iostat -x命令来查看详细信息。

 

案例一、

最近使用sysbench在做mysql数据库的压力测试,已经设置了较大的innodb_buffer_pool的值。在测试过程中我们对系统的性能进行了分析:

[root@ken1 docker]#vmstat 3 100
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b    swpd   free     buff  cache    si   so    bi    bo   in   cs   us sy  id wa st
67  0      0  24610424   1476 2424032    0    0    95   101    1    2    3  0  97  0  0
64  0      0  24610356   1476 2424072    0    0     0     0 51859 28739 69  31  0  0  0
62  0      0  24609760   1476 2424832    0    0   235     0 51815 28698 68  32  0  0  0
59  0      0  24609592   1476 2424832    0    0     0     0 51894 28751 69  31  0  0  0
63  0      0  24608740   1476 2424864    0    0     5     0 52030 28753 68  32  0  0  0

这个例子很明显是个系统性能方面的问题。我们来分析一下:

1、从上面的r列可以看出,很多线程在等待CPU的执行。b列看出I/O或内存方面没有对CPU造成大的影响,说明瓶颈不在IO设备上,

2、并且我们通过wa列也可以得出结论IO请求比不高。我们再来关注us列和sy列,us列的值比较高,说明当前用户的mysql进程占用CPU资源比较高,sy说明内核进程占用比也比较高。造成内核进程占用比高的原因是内核在进行上下文的切换,不难看出确实此时的cs列的请求数挺高的。

3、us+sy的值大概有100%,我们的mysql在单台机器上测试的,与网络的关系不大。这个值高于80%,需要多考虑一下CPU方面了。

4、cs的上下文切换高于in的中断数目,说明内核中相当数量的时间都开销在中断请求数目上。

因此得出结论:我们这台机器的性能瓶颈大概是在CPU上,因为CPU的逻辑个数确实不多,才4个。因此造成CPU的忙碌运行。接下来我们以提高CPU的执行效率为主来考虑。

 

案例二、

procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
 r  b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st
 1            0     19135476       132432      6400624    0    0     0 36949 37619 20303  79  19   1   0   0
 1            0     19130916       132432      6404780    0    0     0 36441 37495 20000  79  20   1   0   0
 1            0     19127628       132436      6409060    0    0     0 35024 37774 20280  80  19   1   0   0
 2            0     19123456       132436      6412836    0    0     0 36896 37451 20089  78  21   1   0   0
 2            0     19119352       132436      6417288    0    0     0 35761 37466 20085  79  20   1   0   0

 

这个例子我们也来分析一下:

1、首先通过r列可以得出结论说明线程等待CPU执行的个数比较多,那么此时CPU肯定是忙碌的状态。

2、通过id列可以看出来CPU空闲时间确实挺少的,说明CPU确实忙碌。

3、通过内存的那几列可以看出内存不是瓶颈的因素

4、通过bo列可以看出此时在进行写磁盘操作,那么我们可以得出结论有可能是磁盘的瓶颈导致CPU的瓶颈,那么我们就要开始验证一下是否是这样子的原因。

5、通过b列可以看出这个值偶尔大于1,再看看wa列,wa列则表示IO等待所占用的CPU的百分比,不过wa列的值挺小的,说明瓶颈不在磁盘上。

6、因此我们可以得出结论是CPU自己达到了瓶颈,我们可以更换高配置的CPU来解决问题。

 

案例三:

procs memoryswapiosystemcpur
r  b     swpd  free  buff  cache   si     so  bi  bo  in    cs   us    sy  id  wa
0     1250  3248  45820 1488472 30    132 992  0   2437 7657  23   50   0  23
0   1376  3256  45820  1488888 57    245 416  0   2391 7173  10   90   0  0
0   1582  1688  45828  1490228 63    131 1348 76  2432 7315  10   90   0 10
2   3981  1848  45468  1489824 185   56  2300 68  2478 9149  15   12   0 73
2   10385 2400  44484  1489732 0     87  1112 20  2515 11620  0   12   0 88
2   12671 2280  43644  1488816 76    51  1812 204 2546 11407 20   45   0 35

 

1、从r列与id列可以看出CPU处于忙碌的状态。

2、从swap列可以看出值在增大,说明有大量数据从内存转换到交换空间。再从free列可以看出空闲内存减少,是因为有大量请求(bi)转换到内存中。

3、内存free的减少导致系统写入swpd设备的块数目(so)和swap空间在不断增加。

4、从wa列可以看出大量的IO请求导致CPU的效率低下。

5、因此我们得出结论是I/O请求导致的CPU的效率低下,(当然也有可能CPU的执行效率也不高,但是你的先排除除了CPU之外的设备的瓶颈,最后在查看CPU的瓶颈)

 

案例四

# vmstat 1 10
procs memory swap io system cpu
r b swpd   free  buff  cache    si so bi  bo   in   cs   us sy  id  wa
0 249844 19144 18532 1221212  0  0  7    3   22   17   25  8  17  18
1 249844 17828 18528 1222696  0  0 40448 8   1384 1138  13  7  65  14
1 249844 18004 18528 1222756  0  0 13568 4   623  534   3  4   56  37
0 249844 17840 18528 1223200  0  0 35200 0   1285 1017  17 7  56   20
0 249844 22488 18528 1218608  0  0 38656 0   1294 1034  17 7  58   18
1 249844 21228 18544 1219908  0  0 13696 484   609 559   5  3  54   38
1 249844 17752 18544 1223376  0  0 36224 4   1469 1035  10 6  67   17
1 249844 17856 18544 1208520  0  0 28724 0   950  941   33 12 49   7
0 249844 17748 18544 1222468  0  0 40968 8   1266 1164  17 9  59   16
0 249844 17912 18544 1222572  0  0 41344 12   1237 1080  13 8  65   13

 

1、从内存方面我们看出来,swpd列的值始终没有变化,并且si和so的值也没有变化。

2、空闲内存free的值虽然不高,但是由于swap没有发生变化说明与内存的关系不大。

3、CPU方面也没有太大的问题,尽管有一些运行队列r,但处理器还有始终50%多的idle(CPU id)

4、CPU还有平均20%的I/O等待情况。

5、结论:这是一个I/O瓶颈的问题。

 

案例五、

[root@:vg_adn_tidbCkhsTest:54.158.254.36:172.31.30.62 /dev/data]#vmstat -w 3 100
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
 r  b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st
 1            0       221736       132832     14142460    0    0    91   120    1    1   3   0  97   0   0
 0            0       224572       132836     14128024    0    0     0 78357 43930 57823  59  25  10   6   0
 3            0       226932       132836     14112496    0    0     0 78351 45233 58913  64  27   6   3   0
 1            0       231532       132832     14096368    0    0     0 78271 43600 55667  63  26   7   3   0
 1            0       230788       132832     14084196    0    0     8 76889 45326 61036  61  27   9   4   0
 2            0       229664       132832     14073200    0    0     0 78984 44279 54078  65  26   6   3   0
 1            0       212588       132836     14078364    0    0     0 75559 44687 54993  65  26   6   3   0
 2            0       212308       132828     14066684    0    0     0 82573 44591 57873  63  26   8   3   0
 0            0       208936       132832     14056368    0    0     0 91479 43920 54727  64  27   6   3   0
 1            0       211868       132832     14041520    0    0     0 92304 42619 55316  65  26   6   2   0
 1            0       232676       132824     14017956    0    0     0 114328 27546 34453   2   4  72  21   0
 3            0       225940       132832     14024544    0    0     0 91408 21580 24824   3   4  67  26   0
 2            0       232532       132828     14013068    0    0     0 92092 34802 47947  18  10  46  25   0
 1            0       226364       132828     14011320    0    0     0 73961 39283 50904  41  18  27  14   0
 0            0       216000       132824     14012016    0    0     0 75463 45527 60788  61  25  10   4   0
 2            0       230808       132824     13984236    0    0     0 71920 44469 56487  65  27   5   3   0

 

 

这是一个在做mysql数据库性能压测的例子,我们使用sysbench工具对其进行insert操作,在128线程下进行的压测,数据库表是100万行数据,一共10个表。

1、既然是做insert操作,想必磁盘IO肯定会比较高,因此我们先看bo列,bo列的值基本在70Mb/s的速度之上。

2、我们再看r列,128线程的操作,此时的r列值不算太高,说明CPU是在高负载下运行。并且id列的值有一部分时刻也没有接近于0。

3、当bo列的值某一时刻开始增高的时候,说明IO吞吐量此时加大了。这个时候r列的值几乎为0,说明之前CPU的负载过高是磁盘IO过低导致的。而现在磁盘IO升高了那么此时的CPU已经能完成磁盘写操作(我们来看id列,确实CPU此时空闲了),既然磁盘这一时刻的性能变高了,吞吐量变大了,那么不难得出b值肯定会高,事实上也是b列的值确实增高了,wa的值也增高了。

4、因此我们可以看出这个压测主要影响在磁盘的性能上,如果有必要我们可以更换更好的高吞吐量的硬盘。

 

本片部分内容节选自系链接,大家也可以看看:https://blog.csdn.net/achang21/article/details/44041535

 

8 thoughts on “Linux系统性能瓶颈分析工具vmstat–技术流ken

  1. I simply wanted to thank you so much all over again. I do not know what I would’ve taken care of in the absence of those strategies discussed by you about that situation. It actually was an absolute daunting concern in my circumstances, nevertheless noticing a well-written approach you managed that took me to weep for joy. Extremely grateful for this help as well as expect you find out what a powerful job you are undertaking training many others through a site. I am certain you’ve never got to know any of us.

  2. I am glad for commenting to let you be aware of what a wonderful discovery my child enjoyed studying your site. She learned several issues, including what it’s like to possess a very effective coaching style to have the rest really easily comprehend certain tortuous topics. You really surpassed my expectations. Thank you for giving these precious, dependable, edifying and also fun tips on that topic to Julie.

  3. I precisely desired to thank you very much once more. I am not sure the things I would have gone through without the type of secrets shown by you directly on such area of interest. It truly was an absolute frightful concern for me personally, but viewing this specialized technique you handled the issue made me to leap for fulfillment. I’m happy for the support and even expect you comprehend what a powerful job you’re undertaking teaching men and women using a site. More than likely you’ve never come across all of us.

  4. Thank you so much for providing individuals with an exceptionally pleasant possiblity to read in detail from here. It can be so kind plus jam-packed with amusement for me personally and my office acquaintances to visit your web site the equivalent of three times in a week to find out the newest things you have. Not to mention, we are certainly happy concerning the astonishing creative concepts you serve. Some 3 points on this page are basically the finest I’ve ever had.

  5. I as well as my buddies were found to be checking out the excellent ideas located on the website while all of the sudden developed a terrible suspicion I never expressed respect to the web site owner for those tips. Those men appeared to be as a result joyful to read through them and have clearly been using them. Many thanks for really being quite kind and also for deciding upon varieties of ideal resources most people are really needing to understand about. My personal honest apologies for not expressing gratitude to earlier.

  6. I am just writing to let you know of the really good experience our princess enjoyed reading through your blog. She realized many pieces, which included what it is like to possess an ideal coaching style to get others completely have an understanding of some multifaceted subject matter. You undoubtedly exceeded her expectations. Many thanks for displaying those practical, dependable, explanatory not to mention cool tips on your topic to Jane.

  7. My spouse and i were very fortunate that Louis managed to do his investigation through the entire precious recommendations he obtained out of the blog. It’s not at all simplistic just to possibly be freely giving tips which often men and women have been trying to sell. And now we take into account we’ve got the writer to be grateful to for this. All the explanations you made, the straightforward site navigation, the relationships your site make it easier to engender – it’s got everything astonishing, and it’s aiding our son and the family believe that this subject is brilliant, which is extremely mandatory. Many thanks for all!

  8. I wish to express some appreciation to you just for rescuing me from such a instance. Because of exploring throughout the the web and seeing principles that were not beneficial, I believed my entire life was gone. Living devoid of the solutions to the difficulties you’ve solved as a result of your main website is a crucial case, as well as ones that would have adversely affected my entire career if I hadn’t discovered your web blog. Your personal expertise and kindness in maneuvering all the details was precious. I don’t know what I would have done if I hadn’t come across such a subject like this. I can now look forward to my future. Thank you very much for your reliable and results-oriented help. I won’t be reluctant to endorse your blog to any person who needs to have guidance about this topic.

发表评论

电子邮件地址不会被公开。