3atv精品不卡视频,97人人超碰国产精品最新,中文字幕av一区二区三区人妻少妇,久久久精品波多野结衣,日韩一区二区三区精品

歡迎訪問 生活随笔!

生活随笔

當前位置: 首頁 > 编程资源 > 编程问答 >内容正文

编程问答

【OS】NMON的简介和使用

發布時間:2024/3/26 编程问答 29 豆豆
生活随笔 收集整理的這篇文章主要介紹了 【OS】NMON的简介和使用 小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

【OS】NMON的簡介和使用





目前NMON已開源,以sourceforge為根據地,網址是http://nmon.sourceforge.net。







1.?目的

本文介紹操作系統監控工具Nmon的概念、使用方式及使用參數。指導運維人員通過nmon工具監視AIX/Linux操作系統資源使用情況,收集監控結果及產生的數據文件,制作相關系統性能分析報告。

2.?Nmon簡介

Nmon (Nigel’s Monitor)是由IBM 提供、免費監控 AIX 系統與 Linux 系統資源的工具。該工具可將服務器系統資源耗用情況收集起來并輸出一個特定的文件,并可利用 excel 分析工具(nmon analyser)進行數據的統計分析。

2.1??軟件特性

nmon 工具可以在一個屏幕上顯示所有重要的性能優化信息,并動態地對其進行更新。這個高效的工具可以工作于任何啞屏幕、telnet 會話、甚至撥號線路。另外,它不會消耗大量的 CPU 周期,通常低于百分之二(在更新的計算機上,其 CPU 使用率將低于百分之一)。nmon使用啞屏幕,在屏幕上對數據進行顯示,并且每隔兩秒鐘對其進行更新。用戶可以很容易地將這個時間間隔更改為更長或更短的時間段。如果拉伸窗口,并在 X Windows、VNC、PuTTY 或類似的窗口中顯示這些數據,nmon 工具可以同時輸出大量的信息。

nmon 工具還可以將相同的數據捕獲到一個文本文件,便于以后對報告進行分析和繪制圖形。輸出文件采用電子表格的格式 (.csv)。

目前nmon已開源,以sourceforge為根據地,網址是http://nmon.sourceforge.net

2.2??軟件組成

Nmon使用需要nmon工具和nmonanalyser分析程序兩者配合使用。nmon工具生成性能數據文件,然后monanalyser以nmon生成的數據文件作為輸入,輸出為Excel?電子表格,并自動地生成相應的圖形,使得我們能夠直觀地觀察OS性能(CPU、IO和內存等)的變化過程。

2.3??運行環境

??? nmon 工具運行于:

·?AIX? 4.1.5、4.2.0、4.3.2 和 4.3.3(nmon Version 9a:該版本的功能已經確定,并且不會對其進行進一步的開發。)

·?AIX 5.1、5.2 和 5.3(nmon Version 10:該版本現在支持 AIX 5.3 和基于 POWER5? 處理器的計算機,并且提供了 SMT 和共享 CPU 微分區的支持。)

·?pSeries? p5 和 OpenPower? 上的 Linux? SUSE SLES 9、Red Hat EL 3 和 4、Debian

·?Linux SUSE、Red Hat 和許多最新的 x86(32 位模式的 Intel 和 AMD)上的發布版

·?zSeries? 或 mainframe 上的 Linux SUSE 和 Red Hat

nmon 工具大約每六個月更新一次,或者在可用的新的操作系統發布版中對其進行更新。

2.4??軟件功能

nmon 工具可以為 AIX 和 Linux 性能專家提供監視和分析性能數據的功能,其中包括:

·?CPU 使用率

·?內存使用情況

·?內核統計信息和運行隊列信息

·?磁盤 I/O 速度、傳輸和讀/寫比率

·?文件系統中的可用空間

·?磁盤適配器

·?網絡 I/O 速度、傳輸和讀/寫比率

·?頁面空間和頁面速度

·?CPU 和 AIX 規范

·?消耗資源最多的進程

·?IBM HTTP Web 緩存

·?用戶自定義的磁盤組

·?計算機詳細信息和資源

·?異步 I/O,僅適用于 AIX

·?工作負載管理器 (WLM),僅適用于 AIX

·?IBM TotalStorage? Enterprise Storage Server? (ESS) 磁盤,僅適用于 AIX

·?網絡文件系統 (NFS)

·?動態 LPAR (DLPAR) 更改,僅適用于面向 AIX 或 Linux 的 pSeries p5 和 OpenPower

2.5??軟件獲取

nmon工具和nmonanalyser工具都可以在IBM的Wike頁面上下載到。

1)?Nmon下載:
位置:可從IBM Wiki上下載
http://www-941.haw.ibm.com/collaboration/wiki/display/WikiPtype/nmon
下載頁面如下:
下載 (31.17 KB)2009-2-22 23:49

例如測試的系統是AIX5.3,那么就可以下載nmon4aix12e.zip ,下載后可以看到壓縮包里是一些文件,如下:

下載 (8.54 KB)

2009-2-22 23:49

其實nmon就是shell腳本,nmon文件運行時調用其他的文件,生成性能數據,這個工具運行時也是通過執行nmon腳本接受參數。

2)?Nmonanalyser下載:
位置:可從可從IBM Wiki上下載
http://www-941.haw.ibm.com/collaboration/wiki/display/Wikiptype/nmonanalyser
下載頁面如下:

下載 (26.08 KB)

2009-2-22 23:49

例如下載V3.3版本的Nmonanalyser

3.?Nmon使用

3.1??下載軟件

1)?下載之前須確定操作系統內核版本,本文以公司服務器192.168.40.212為例。登錄服務器使用命令獲取操作系統版本信息:

?

查看可知為Enterprise Linux Server release 5.5。

2)?登錄IBM官方網站(http://nmon.sourceforge.net/pmwiki.php?n=Site.Download)下載相應版本的nmon工具:nmon_linux_14g.tar.gz?

3.2??安裝軟件

1)?用root用戶登錄系統,建立目錄:#mkdir? /nmon

2)?通過FTP將下載的nmon工具上傳至服務器 192.168.40.212目錄/nmon下。?

3)?修改tar包權限:#chmod +x??nmon_linux_14g.tar.gz?

4)?解壓文件:#tar xvfZ? nmon_linux_14g.tar.gz?

5)?執行授權命令:#chmod? +x? nmon_x86_rhel54

3.3??運行界面

1)?運行./nmon_x86_rhel54,便可進入nmon的監控界面,如下圖:

?

2)?鍵入“c”查看系統CPU使用情況

?

3)?鍵入“m”查看系統內存使用情況

?

4)? 鍵入“d”查看系統磁盤I/O情況

?

5)?鍵入“h”查看幫助信息

?

3.4??使用方法

3.4.1?實時監控

在完成配置后,只需要“#./nmon”命令即可運行程序,通過一些快捷鍵調取關心的系統資源指標進行顯示,如按鍵“C”可以查看CPU相 關信息;按鍵“D”可以查看磁盤信息;按鍵“T”可以查看系統的進程信息;“M”對應內存、“N”對應網絡等等,完整的快捷鍵對應內容可以通過幫助(按鍵 “H”)查看,可以顯示Linux系統CPU、內存、進程信息,包括了CPU的用戶、系統、等待和空閑狀態值,可用內存、緩存大小以及進程的CPU消耗等詳細指標。?

該種方式顯示信息實時性強,能夠及時掌握系統承受壓力下的運行情況,每顆CPU利用率是多少、內存使用多少、網絡流量多少、磁盤讀寫……這些數據均是實時刷新,一目了然。

3.4.2?后臺監控

為了配合性能測試,我們往往需要將一個時間段內系統資源消耗情況記錄下來,這時可以使用命令在遠程窗口執行命令:

./nmon/ nmon_x86_rhel5? -f -N -m /nmon/log? -s 30 -c 120
其中各參數表示:
? -f 按標準格式輸出文件:_YYYYMMDD_HHMM.nmon
? -N include NFS sections
? -m 切換到路徑去保存日志文件
? -s 每隔n秒抽樣一次,這里為30
? -c 取出多少個抽樣數量,這里為120,即監控=120*(30/60/60)=1小時
??? 根據小時計算這個數字的公式為:c=h*3600/s,比如要監控10小時,每隔30秒采樣一次,則c=10*3600/30=1200

該命令啟動后,會在nmon所在目錄下生成監控文件,并持續寫入資源數據,直至360個監控點收集完成——即監控1小時,這些操作均自動完成,無需手工干 預,測試人員可以繼續完成其他操作。如果想停止該監控,需要通過“#ps –ef|grep nmon”查詢進程號,然后殺掉該進程以停止監控。

3.4.3?定時任務

??? 除配合性能測試的短期監控,我們也可以實現對系統的定期監控,作為運營維護階段的參考。定期監控實現如下:

1)?執行命令:#crontab? –e

2)?在最后一行添加如下命令:
0 8 * * 1,2,3,4,5? /nmon/nmon_x86_rhel5? -f -N -m /nmon/log? -s 30 -c 1200
表示:
周一到周五,從早上08點開始,監控10個小時(到18:00整為止),輸出到/nmon/log

4.?Nmon監控結果介紹

4.1??生成結果文件

???? 通過后臺監控和定期監控,我們可以得到擴展名為nmon的監控文件,這些文件記錄著系統資源的數據,需要配合分析工具(nmon analyser)進行解讀。

1)?使用FTP工具從服務器上取下生成結果文件/nmon/log/sjfx212_120318_1723.nmon到本機。

2)?打開nmon_analyser.zip 包下的nmon analyser v33g.xls 文件,點擊Analyse nomn data按鈕,選擇之前get下來的sjfx212_120318_1723.nmon文件。

Excel可能禁止運行宏了,點“安全警告旁邊的選項”,允許運行宏:

??? ???

3)?生成分析結果文件sjfx212_120318_1723.nmon.xlsx,并生成統計圖,直觀顯示系統資源情況。

4.2??主要性能參數介紹

u?系統匯總(對應excel標簽的‘SYS_SUMM’):其中藍線為cpu占有率變化情況;粉線為磁盤IO的變化情況。

u?磁盤讀寫情況匯總(對應excel標簽的‘DISK_SUMM’):?其中藍色為磁盤讀的速率KB/sec;紫色為磁盤寫的速率KB/sec。

u?內存情況匯總(對應excel標簽的‘MEM’): 曲線表示內存剩余量(MB)

4.3??頁面介紹

簡單介紹生成結果的各個頁面:

1)?系統匯總頁面:對應標簽頁(SYS_SUMM)

頁面顯示項主要有主機名,執行日期,系統cpu使用情況(藍線),系統I/O情況(粉紅線),其中坐標左縱軸為系統cpu(user%+sys%)使用率,橫軸為運行時長(下圖為一個小時),右縱軸為系統磁盤傳輸(Disk xfers),坐標下側為統計信息:系統I/O情況(一個周期內的平均值、最大值、出現最大值的時間),系統CPU使用情況。

選項

說明

備注

User%

用戶進程時間在CPU 開銷時間百分比

如果一個CPU 被充分使用,利用率分類之間均衡的比例應該是:

65% - 70% User Time

30% - 35% System Time

0% - 5%?? Idle Time

Wait%

所有進程線程被阻塞等待完成一次IO 請求所占CPU 開銷idle的時間百分比

Sys%

線程和中斷在CPU 開銷時間百分比

Idle%

CPU 空閑時間的百分比

CPU%

CPU 利用率的百分比

?

注釋:

如果系統CPU有IO wait存在,說明可能IO或內存方面存在瓶頸,其中主要導致IO Wait的主要原因如下:

ü?內存不夠而引起頻繁的的數據交換,導致數據存取存在交換空間的 I/O瓶頸

ü?硬盤數據分布不合理

2)?系統信息:標簽頁(AAA)

頁面信息主要包括:執行命令,主機CPU數(4),操作系統內核版本信息,主機名等信息。

?

3)?系統詳細信息:標簽頁(BBBP)

?頁面信息主要包括:操作系統版本,主機磁盤信息,主機CPU型號、主頻信息,內存信息,網卡信息等。

?

4)?CPU使用情況:標簽頁(CPU_ALL,CPU_SUMM,CPU001,CPU002,CPU003,CPU004)

主機CPU使用情況匯總以及單顆CPU的運行情況。

?

?

5)?磁盤讀寫情況匯總:標簽頁(DISK_SUMM,DISKBSIZE,DISKBUSY,DISKREAD,DISKWRITE,DISKXFER)

?磁盤的讀、寫及I/O統計信息,系統各磁盤分區的讀寫情況。其中DISKBUSY頁主要反映系統本地磁盤使用,DISK_SUMM包括本地和存儲(ESS, EMC, FASt及HDS)上所有磁盤使用情況。

?

?

?

6)?內存使用情況:標簽頁(MEM)

系統內存空閑、使用,swap、cached等統計信息。

?

7)?系統網絡情況:標簽頁(NET,NETPACKET)

反映系統的網絡運行情況,系統各個網絡適配器讀寫的數據包數

?

8)?系統進程:標簽頁(PROC)

反映系統運行線程及等待切換的線程平均數。其中RunQueue - 每個處理器應該運行隊列不超過1-3個線程。

?

?

9)?活動虛擬內存總量:標簽頁(VM)

Linux操作系統特有指標,主要包括系統/proc/vmstat文件中信息,兩張圖片主要顯示系統分頁文件(pagefile)和swap分區運行情況。如果系統總是存在大量換出頁(pgpgout/s)KB數,說明系統需要更多內存。

?

?

10)?頁式調度:標簽頁(PAGE)

記錄系統(AIX)頁式調度的情況。這一欄主要記錄系統換頁(paging)情況和頁面掃描:自由比率(page scan:free ratio)。其中系統換頁頻率應該不大于5次/秒,而當頁面掃描:自由比率持續大于4時,需要重點關注下系統內存和分頁空間的使用情況。

?

說明:

大量的內存交換操作會極大地影響系統的性能,尤其是在當數據庫文件創建在文件系統上時(JFS and JFS2)。在這種情況下經常訪問的數據,即在SGA中存在,也同樣在文件的緩存中存在。這種相同的數據在內存中緩存兩次的情況,會降低內存的使用效率,從而使內存頻繁進行交換操作,造成系統的I/O瓶頸,降低整個系統的性能。

11)?采集時間:標簽頁(ZZZZ)

記錄nmon工具采集系統信息的時間點。

?

5.?Nmon監控案例介紹

本節介紹通過nmon工具發現系統性能下降問題的常見現象和處理流程。

5.1??常見現象和產生原因

性能下降(Performance Degradation),主要是指系統的性能隨時間而逐漸下降(這里假定在系統性能下降的過程中系統的負載狀況沒有明顯變化)。系統運行過程中占用的CPU或內存隨時間增加也屬于廣義的性能下降問題。

在生產環境中,通常由終端客戶最先感覺到并報告性能下降問題。所以狹義的性能下降問題主要是指系統運行指標隨時間變化,比如吞吐率隨時間下降或頁面響應時間隨時間上升,或者兩者兼而有之。

下面列舉一些引起性能下降問題的原因:

ü?應用程序資源使用問題。主要是內存使用問題,即由于應用服務器的內存碎片問題或內存泄漏問題,導致垃圾回收的開銷隨時間增大。也有可能是因為磁盤臨時文件積累造成磁盤訪問開銷增大。

ü?應用程序設計問題。由于應用程序的設計存在可擴展性或可靠性問題,導致運行開銷隨時間或業務對象的積累而增大。

ü?數據庫訪問問題。該問題又可以分為許多類型,如調優參數問題、表結構或索引設計問題、垃圾數據問題等。其共同特點是導致應用程序利用特定操作訪問數據庫的開銷隨時間而增大。

ü?服務器軟件資源使用問題。雖然可能性很小,但是應用服務器、數據庫服務器等服務器程序也是軟件程序,也有可能存在性能下降問題。這些服務器程序在自身測試過程中可能遺漏了某些性能問題,而在用戶特定的執行狀況下觸發了這些問題,結果導致這些服務器程序使用的操作系統資源泄漏而出現性能下降問題。

ü?測試用例設計問題。性能測試中有可能發現一些“假”的性能下降問題。比如測試用例設計時假設在測試執行過程中系統負載保持恒定,但實際的測試用例實現導致系統負載或特定頁面的處理內容隨時間增多,也可能導致測試工具的測試報告中出現性能下降問題。

5.2??實例介紹

5.2.1?示例一

?

5-1? nmon操作系統監視匯總信息圖

由此信息圖可發現在12小時測試進行過程中,系統的磁盤傳輸(Disk xfers)逐漸增大,與此同時系統CPU占用率逐漸下降。進一步檢查單個CPU的使用情況,發現1號CPU的Wait狀態占用率明顯增大,如圖5-2所示。這說明CPU占用率逐漸下降是由于等待磁盤I/O引起的。

?

5-2? nmon單個CPU使用情況圖

接下來分析磁盤傳輸匯總信息,如圖5-3所示,可以看出磁盤寫數據量沒有明顯增加,但是磁盤讀數據量明顯隨時間而增加。

?

5-3? nmon磁盤傳輸匯總情況圖

憑磁盤傳輸匯總信息,在排除由應用服務器讀取導致的性能下降問題后,基本可以肯定不斷增加的磁盤讀取操作是由數據庫引起的。

隨后,分析DB2的快照監視器的監視結果,可以發現DB2的緩沖池(Buffer pool)的數據和索引物理讀(physical read)的比例非常高。如下例所示:

Buffer pool data logical reads???????????? = 5502388

Buffer pool data physical reads??????????? = 430671

Buffer pool temporary data logical reads?? = 0

Buffer pool temporary data physical reads? = 0

……

可以看到緩沖池的物理讀比例(即緩沖池不命中率)高達7%,這遠遠大于1%的警戒線。而且物理讀比例有隨時間增加的趨勢(通過不同時間的快照信息對比發現)。

至此可以懷疑性能下降問題是由于DB2的緩沖池配置參數設置不當引起的。考察數據庫配置參數信息發現,該數據庫的BUFFPAGE參數值為10000。與該測試用例使用的數據規模相比,這個參數值明顯偏小。于是將BUFFPAGE參數值增大10倍,變為100000,重新運行性能測試,發現性能下降問題基本消失。

5.2.2?示例二

本實例為通過nmon監控河北稅源管理平臺數據倉庫收集的數據。對照nmonanalyser工具生成結果進行分析。

1)?收集服務器基本信息

?

?

?

?????? 通過以上數據可以知道河北數據倉庫服務器基本配置信息如下:

ü?主機名:HE_SSGLY_DB_01

ü?操作系統版本:AIX 5.3.0.44?build?5300-04

ü?操作系統內核:HW-type=CHRP=Common H/W Reference Platform Bus=PCI LPAR=Dynamic Multi-Processor 64 bit

ü?主機型號:IBM p5 595 (9119-595)

ü?網絡配置:IP Address: 75.16.16.191?Sub Netmask: 255.255.248.0?Gateway: 75.16.16.100

ü?主機存儲:EMC存儲

ü?主機物理內存:49152 MB

ü?網卡信息:2塊網卡 速率為1024M/S

2)?文件系統使用情況

?

?

?????? 通過以上數據可以知道河北數據倉庫服務器文件系統相關信息如下:

ü?文件系統分區大小及使用情況

ü?文件系統掛載點信息

ü?文件系統類型為JFS2(Journaled File System 2)文件系統

3)?系統資源使用情況

?

????? 觀察上圖,可以發現服務器cpu使用率較高的時段為8:00-11:30、14:00-17:20,而系統I/O開銷較高的時段均出現在21:00-次日5:50。以上信息基本符合服務器晚上進行數據ETL處理、白天工作時段多進行平臺一般事務處理的情況。進一步觀察服務器cpu使用情況:

?

?

?

???? 發現CPU利用率較高均為用戶進程,單獨查看CPU-11注意到22:00-0:00這個時段內CPU的Wait狀態占用率明顯增大, user狀態占用率卻很低,此時系統I/O明顯升高,說明這個時段內有大量磁盤I/O發生,CPU占用率逐漸下降是由于等待磁盤I/O引起的。實際情況中,這個時段服務器正在進行ETL數據處理,的確有大量數據傳輸和磁盤讀寫發生。

?

5.2.3?示例三

?? ? 介紹個利用nmon進行系統實時監控的例子。

???? 把nmon腳本上傳到服務中,直接運行即可,執行命令如下:#./nmon或者#/tmp/nmon/nmon

???? 以下是aix5.3下使用nmon的一些截圖:

?

?? 分別輸入c、t、n、m,可以了解系統cpu,內存,消耗資源最高的線程的使用情況。

?




Links:

NMON home page

http://www-941.haw.ibm.com/collaboration/wiki/display/Wikiptype/nmon

NMON_Analyser home page

http://www-941.haw.ibm.com/collaboration/wiki/display/Wikiptype/nmonanalyser

User Forum

http://www.ibm.com/developerworks/forums/dw_forum.jsp?forum=749&cat=56





一.nmon工具介紹

nmon 是一個分析aix和linux性能的免費工具(主要是ibm為自己的aix操作系統開發的,但是也可以用在linux操作系統),而nmon_analyser是nmon的一個工具可以把nmon生成的報告轉化成excel報表的形式供查看

nmon 工具可以在一個屏幕上顯示所有重要的性能優化信息,并動態地對其進行更新。它并不會消耗大量的 CPU ,通常低于百分之二

?

下載地址:

http://nmon.sourceforge.net/pmwiki.php?n=Site.Download

http://sourceforge.net/projects/nmon/files/?source=navbar

?

注意選擇和自己機器對應的版本:

linux查看系統版本號:

1.方法1

對于linux系統而已,有成百上千個發行版。對于發行版的版本號查看方法

如以centos為例。輸入lsb_release -a即可

該命令適用于所有的linux,包括Redhat、SuSE、Debian等發行版

?

?

2.

?

3.

?

查看系統位數

同樣對于Linux系統而言也有32和64位之分,用上述查看內核方法,輸入

uname -a ?或 more /proc/version

在內核版本后面會有一個X86_64就是表示64為系統啦

# uname -a
x86_64則說明你是64位內核, 跑的是64位的系統.
i386, i686說明你是32位的內核, 跑的是32位的系統

?

查看內核的幾種方法:

1. uname -a#顯示詳細的內核信息,

Linux localhost.localdomain 2.6.18-92.1.6.el5xen #1 SMP Wed Jun 25 12:56:52 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

2. cat /etc/issue#顯示版本類型

Scientific Linux SL release 5.2 (Boron)
Kernel \r on an \m

3. cat /proc/version

Linux version 2.6.18-92.1.6.el5xen (brewbuilder@norob.fnal.gov) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Jun 25 12:56:52 EDT 2008

?

?

查看操作系統位數:

1.ls /?#如果有lib64或這個目錄,那操作系統就是64位的
2.getconfig LONG_BIT?若輸出32即為32位系統,64為64位系統

32位的系統中int類型和long類型一般都是4字節,

64位的系統中int類型還是4字節的,但是long已變成了8字節。

inux系統中可 用"getconf WORD_BIT"和"getconf LONG_BIT"獲得word和long的位數。
64位系統中應該分別得到32和64。

3.?uname -a中若為X86示意為64位系統,i386等位32位系統

?

?

?

二.下載安裝

?

nmon默認下載之后只是一個可執行測的文件,但是下載時根據不同的系統來進行區分的

大家可以根據自己的系統選擇相應的版本。

執行命令 more /etc/issue 查看系統版本。

?

首先將下載的nmon_x86_64_centos6文件上傳到linux服務器上

?

[root@localhost source]#cp nmon_x86_64_centos6 /usr/bin

[root@localhost source]#cd /usr/bin

[root@localhost source]# chmod 777 nmon_x86_64_centos6

[root@localhost source]# ./nmon_x86_64_centos6(切換到nmon_x86_rhel45文件所在的目錄)

[root@localhost source]# env

[root@localhost source]# mv?nmon_x86_64_centos6 nmon

[root@localhost source]# nmon(此時nmon就是全局命令)

也有可能

?

監控設置

15分鐘=900s

#alias nmon15='nmon -f -s3 -c300 -m /root/qumf/'

?

?

nmon工具命令的使用:

#nmon -s 300 -c 288 -f -m /tmp

-s 300:表示每300秒采集一次數據,

-c 288 :表示采集288次,300*288=86400秒,剛好是1天的數據,這樣運行一次這個程序就會生成一個一天的數據文件,

-m /tmp: 表示生成的數據文件的路徑

-f :表示生成的數據文件名中有時間

也可以讓其自動監控,每天一個文件夾,每小時一個文件,每分鐘或5分采樣一次

?

?

?

參數解釋:
-s10 每 10 秒采集一次數據。
-c60 采集 60 次,即為采集十分鐘的數據。
-f 生成的數據文件名中包含文件創建的時間。
-m 生成的數據文件的存放目錄。

?

nmon -f -s 10 -c 60

-f 表示生成的數據文件名中有時間;

-s 10 表示每 10 秒采集一次數據;

-c 60 表示采集 60 次,10*60=600 秒,

剛好是 10分鐘的數據,這樣運行一次這個程序就會生成一個采集 10分鐘數據的文件。該行命令將在當前目錄中創建輸出文件,其名稱為:_date_time.nmon", hostname" 是這臺主機的主機名。

?

更多用法
crontab -e
0 0 * * * /usr/local/nmo -s300 -c288 -f -m /home/nmon/ > /dev/null 2>&1
表示:
300*288=86400 秒,正好是一天的數據。
0 8 * * 1,2,3,4,5 /usr/local/nmon -f -N -m /home/nmon/log -s 30 -c 1200
表示:
周一到周五,從早上08點開始,監控10個小時(到18:00整為止),輸出到/home/nmon/log。

?

?

范例1:

自動按天采集數據:
在 crontab 中增加一條記錄:
0 0 * * * nmon -s300 -c288 -f -m /home/ > /dev/null 2>&1:
300*288=86400 秒,正好是一天的數據。

范例2:
A、執行命令:#crontab -e
B、在最后一行添加如下命令:
?0 8 * * 1,2,3,4,5 /nmon/scriptn/mon_x86_rhel52 -f -N -m /nmon/log -s 30 -c 1200
表示:
?周一到周五,從早上08點開始,監控10個小時(到18:00整為止),輸出到/nmon/log

?

三、nmon數據采集

1、數據采集

為了實時監控系統在一段時間內的使用情況并將結果記錄下來,我們可以通過運行以下命令實現:

?

1 #./ nmon -f -t -s30 -c 180

參數說明:

l-f:按標準格式輸出文件:_YYYYMMDD_HHMM.nmon;

l-t:輸出中包括占用率較高的進程;

l-s30:每30秒進行一次數據采集

l-c180:一共采集180次

輸入命令回車后,將自動在當前目錄生成一個hostname_timeSeries.nmon的文件,nmon輸出文件的命名方式是服務器名_日期時間.nmon

特別說明:

該命令啟動后,會在nmon所在目錄下生成監控文件,并持續寫入資源數據,直至360個監控點收集完成——即監控1小時,這些操作均自動完成,無需手工干預,測試人員可以繼續完成其他操作。如果想停止該監控,需要通過“#ps–ef|grepnmon”查詢進程號,然后殺掉該進程以停止監控。

?

[root@localhost source]# hostname

linux_test

[root@localhost source]# ./nmon -f -s 10 -c 60

[root@localhost source]# ps -ef | grep nmon

root 17815 1 0 08:22 pts/1 00:00:00 ./nmon -f -s 10 -c 60

root 17888 6977 0 08:22 pts/1 00:00:00 grep nmon

[root@localhost source]# ls linux_test_120724_0822.nmon

linux_test_120724_0822.nmon

當我們執行相關命令后,就在當前目錄生成了一個以本主機名linux_test開頭,以執行日期為規則,nmon結尾的文件,并且我們通過ps命令會看到相關的nmon進程,這里我們當10分鐘過去以后我們會發現相關nmon進程會消失的。

linux_test_120724_0822.nmon 就是我們所生成的數據文件,所有的信息都在linux_test_120724_0822.nmon記錄,而且我們通過more命令后發現都是我們看不懂的一些文本,這就需要我們把其轉化成我們能看懂的excel格式的文件。首先我們把linux_test_120724_0822.nmon文件導出到我們的windows本地。然后我們從http://www.ibm.com/developerworks/wikis/display/Wikiptype/nmonanalyser

下載nmonanalyser軟件到windows本地。打開解壓縮文件后,我們會發現2個文件,一個是nmonanalyse的說明word格式的說明穩定,另一個是excel格式的nmonanalyse文件。

我們這里所需要的就是excel格式問文件,我們雙擊打開。

?

四.生成圖形化結果

下載 nmon analyser (生成性能報告的免費工具):

下載地址:

https://www.ibm.com/developerworks/community/wikis/form/anonymous/api/wiki/61ad9cf2-c6a3-4d2c-b779-61ff0266d32a/page/b7fc61a1-eef9-4756-8028-6e687997f176/attachment/721e9797-b5fc-41d7-9b2f-5bd2aa2c8f7d/media/nmon_analyser_34a.zip

?

解壓之后可以得到一個nmon analyser v34a.xls,

雙擊打開nmon analyser v34a.xls;

設置excel宏的安全級別調低
選中上圖中的選項,

打開nmon_analyser中的excel,然后載入nmon數據,保存輸出的excel即可。

?

?

動態顯示:







 1、概述

  監控,在檢查系統問題或優化系統性能工作上是一個不可缺少的部分。通過操作系統監控工具監視操作系統資源的使用情況,間接地反映了各服務器程序的運行情況。根據運行結果分析可以幫助我們快速定位系統問題范圍或者性能瓶頸點。

  nmon是一種在AIX與各種Linux操作系統上廣泛使用的監控與分析工具,相對于其它一些系統資源監控工具來說,nmon所記錄的信息是比較全面的,它能在系統運行過程中實時地捕捉系統資源的使用情況,并且能輸出結果到文件中,然后通過nmon_analyzer工具產生數據文件與圖形化結果。

  nmon所記錄的數據包含以下一些方面(也是我們在尋找問題過程中所關注的資源點):

  ● cpu占用率

  ● 內存使用情況

  ● 磁盤I/O速度、傳輸和讀寫比率

  ● 文件系統的使用率

  ● 網絡I/O速度、傳輸和讀寫比率、錯誤統計率與傳輸包的大小

  ● 消耗資源最多的進程

  ● 計算機詳細信息和資源

  ● 頁面空間和頁面I/O速度

  ● 用戶自定義的磁盤組

  ● 網絡文件系統

  另外在AIX操作系統上,nmon還能監控到其他的一些信息,如異步I/O等。

  2、下載安裝nmon

  如何獲取nmon呢?我們可以在IBM的官方網站上免費下載獲取,下載網址為:http://www.ibm.com/developerworks/wikis/display/WikiPtype/nmon。

  nmon的安裝步驟如下:

  1)用root用戶登錄到系統中;

  2)建目錄:#mkdir /test;

  3)把nmon用ftp上傳到/test,或者通過其他介質拷貝到/test目錄中;

  4)執行授權命令:#chmod +x nmon。

  3、nmon數據采集

  3.1 數據采集

  為了實時監控系統在一段時間內的使用情況并將結果記錄下來,我們可以通過運行以下命令實現:

  #./ nmon -f -t -s 30 -c 180

  n -f:按標準格式輸出文件:_YYYYMMDD_HHMM.nmon;

  n -t:輸出中包括占用率較高的進程;

  n -s 30:每30秒進行一次數據采集

  n -c 180:一共采集180次

  輸入命令回車后,將自動在當前目錄生成一個hostname_timeSeries.nmon的文件,如果hosname為test1,生產的文件為:test1_090308_1313.nmon。


  通過sort命令可以將nmon結果文件轉換為csv文件:

  # sort -A test1_090308_1313.nmon > test1_090308_1313.csv

  執行完sort命令后即可在當前目錄生產test1_090308_1313.csv文件。

  3.2 生成圖形化結果

  為了分析nmon監控獲得的結果,IBM還提供了相應的圖形化分析工具nmon_analyser,通過nmon analyser.xls工具可以把監控的結果文件轉換成excel文件,方便分析系統的各項資源占用情況。

  nmon analyser.xls工具的使用方法如下:

  (1)打開nmon analyser.xls工具;

  (2)調整excel宏安全性:工具-宏-安全性

  (修改安全級別與可靠發行商)

  (選擇)安全級別:低

  (勾上)信任所有安裝的加載項和模板

  (勾上)信任對于“Visual Baisc項目”的訪問

  (3)修改完后,確定-關閉nmon analyser.xls,重新打開;

  (4)點擊Analyse nmon data按鈕,加載之前下載的test1_090308_1313.csv文件。

  以下是分析結果的截圖:

  以上就是nmon的簡單描述與使用介紹,大家可以根據自己所采集到的結果分析系統的情況。






nmon for Linux?- nmon is short for?Nigel's performance?Monitor for Linux on?POWER, x86, x86_64, Mainframe & now ARM (Raspberry Pi)


STOP PRESS: nmon for Linux Hits 500,000 downloads July 2017

This systems administrator, tuner, benchmark tool gives you a huge amount of important performance information in one go. It can output the data in two ways

  • On screen?(console, telnet, VNC, putty or X Windows) using curses for low CPU impact which is updated once every two seconds. You hit single characters on you keyboard to enable/disable the various sorts of data.
    • You can display the CPU, memory, network, disks (mini graphs or numbers), file systems, NFS, top processes, resources (Linux version & processors) and on Power micro-partition information.
    • For lots of examples, see the "Screen shots" from the left menu.
    • As you can see on the left lmon12e now in colour
  • Save the data to a comma separated file?for analysis and longer term data capture.
    • Use nmonchart (from this website) to generate a Googlechart webpage.
    • Use this together with nmon Analyser Microsoft Excel spreadsheet, which loads the nmon output file and automatically creates dozens of graphs ready for you to study or write performance reports.
    • Filter this data, add it to a rrd database (using an excellent freely available utility called rrdtool). This graphs the data to .gif or .png files plus generates the webpage .html file and you can then put the graphs directly on a website automatically on AIX with no need of a Windows based machine.
    • Directly put the data into a rrd database or other database for your own analysis
  • Latest version nmon for Linux is?16f
    Download the?precompiled binaries?or?nmon source code

  • More details

    • nmon is a single binary for
      • each operating system (Red Hat, SUSE, Ubuntu, Fedora, OpenSUSE etc.) and
      • each platform (Power, Mainframe, arm, x86 or x86_64).
    • Installing is very easy - just start the right executable.
      • Or rename the version you need to?/usr/bin/nmon?and then type: nmon
    • Why use five or six tools when one free tool can give you everything you need!!
    • For the pre-compiled versions - click on?Download
    • For the source code & compiling - click on?Compiling nmon

    On-screen

    When using nmon via a terminal session you can see the performance data directly on the screen and updated every second. You should if possible, stretch the terminal window to be longer to see more stats at one time. Here is a sample example from a Raspberry Pi 2 running Ubuntu 15.10 and nmon v16b. I typed "cCUd" to display this data.

    For more screen shots take the left-hand side menu option?Screen shots?or click?Screen shots.


    Data Analysis

    Once you save the nmon data you have a number of options to analyse and graph the statistics:

    • nmonchart?tool/script - see left hand menu
      • Nigel's nmonchart tools is quick and simple to convert a nmon output file to a webpage file .html that you can open with a browser directly or add to a website to share.
      • It takes a second or too and generates very nice looking graphs.
      • It is implemented in Korn shell script so you can add features (please share your updates).
      • The Clever part is using the Google.com Charting Javascript Library and your browser to do the actual graphing.
      • This this works on your PC, tablet or even larger mobile phone regardless of operating system.
      • Click here to find out more?nmonchart
    • nmon Analyser Excel Spread-sheet Download
      • This is the original tool and been developed over many years by Stephen Atkins
      • You can request support via the?Performance Tools Forum
      • However, Linux users might not like the idea of using the Microsoft Excel Spreadsheet and automating the creation of graphs can be tricky.
      • Sample Graphs out of the many (see screen shots for more and larger examples:
      • CPU Compared to Disk I/O
      • Disk Read and Write with I/O per second
      • Hot Disk analysis with Average, Weighted Average and Peak values
      • Network Read (top half) and Write (bottom half) Transfer Rates
    • nmon Consolidator Excel Spread-sheet Download
      • This is a newer tool and can combine nmon output files. It is by Stephen Atkins
      • Again its Microsoft Excel Spreadsheet
    • nmon2rrd
      • Microsoft free tool
      • This tool uses the excellent rrdtool to generate all the graphs and a website .html file.
      • Download it from the?nmon for AIX Wiki
      • This allows the automated analysis on many machines and viewing via a Browser.

    Now - Open Source

    nmon for Linux is a single source code file of 5000 lines and single makefile. This will enable you to compile nmon for your precise Linux version (if you can't find what you want in the binaries) and open a few other possibilities:

    • Fixing my code - be gentle, please.
    • Removing magic numbers i.e. constants that can catch us out as machines get larger
    • Developing for some strange environments like machines with no disks, blades that boot from NFS, internal Linux based engines within disks subsystems, embedded machines.
    • Who knows we may get nmon for Linux within the Linux Distro's - any one know how to go about that?

    Thanks for your support, suggestions, testing and I hope this starts a whole new wave of development and interest.


    History

    • nmon for Linux?was an internal project at IBM for many years and was released to open source under GPL on 27th July 2009.
    • Sourceforge.net?is being used to host the project, see?http://sourceforge.net/projects/nmon
    • nmon for AIX?does has a similar online look, file format but was always complete different source code.
      • It is now integrated into AIX topas command from
        • AIX 5.3 TL09
        • AIX 6.1 TL02.
      • nmon for AIX is not open source.
      • For more information?nmon for AIX Wiki



    Documentation

    nmon of Linux Documentation?- Updated 21st Nov 2016

    Ha ha ha ha - you are joking right :-)


    This page contains the following sections:

  • Hardware and Linux Supported
  • Getting Started via YouTube Videos - Including nmon for Linux and nmon for AIX
  • Getting Started - If you prefer to read the absolute minimum
  • nmon Command help - nmon for Linux -? and -h command output for the full syntax
  • nmon Support
  • Other sources


  • Hardware and Linux Supported

    • Platforms = hardware
      • POWER
      • x86_64 = AMD64 - 63 bit
      • Mainframe
      • x86 = 32 bit dropping off rapidly
      • ARM Raspberry Pi 2+3
      • Others . . . include embedded processors running Linux
    • Linux Distro’s
      • Ubuntu
      • Debian
      • SUSE SLES
      • OpenSUSE
      • Red Hat RHEL
      • Fedora
      • Centos
      • Many others . . .


    YouTube?Videos about nmon for Linux

    Many People prefer to watch a YouTube Video to learn - here are the six videos on nmon for both Linux and AIX operating systems. These are all the details you need to know to use nmon well. Note: nmonchart creates a .html file of all the graphs - it is one of many nmon graphing tools.

    nmon for Linux

  • nmon for Linux Starter Pack?20 minutes
  • nmon for Linux Data Capture?15 minutes
  • nmonchart to graph your nmon data files?22 minutes
  • Many nmon users use both Linux and AIX so here are the AIX equivalent videos and many nmon for Linux user use the nmon Analyser (Microsoft Excel spreadsheet)

    nmon for AIX

  • nmon Starter Pack Monitoring Online?14 minutes
  • nmon Starter Pack for AIX Data Capture?15 minutes
  • nmon Starter Pack for AIX Analyser?10 minutes


  • nmon for Linux Getting Started - If you prefer to read the absolute minimum

    Below assumes that you are logged on your system, that you have renamed your nmon binary file to just "nmon", that the nmon file has execute permission (chmod ugo+x nmon) and it is in your PATH.

    Using nmon for Linux Online

    • Just start nmon for Linux with: nmon
    • To stop it, just type: q
    • To get on screen hints type: h
      • and h again to remove the hints
    • Most of the rest are toggled commands i.e. type c to see the CPU stats and type c again to remove CPU stats.
    • The various stats come out in a set order (you can't control this) starting with CPU then memory and finally top processes at the bottom as there can be many processes this tends to fill up the rest of the window
    • Note if you make the window larger you can see more lines of output - this works in X Windows, VMC and Putty.
    • For memory stats type m
    • For disk graphs type d and you will see a 50 column graph of the read and write busy percentages
    • For disk numbers type D and if you type D again you see different information eventually typing D will close this section
    • For top processes there are different modes for the order of displaying the processes and different information, See the top line of the Top Processes section for further details.

    Using nmon for Linux in data capture mode

    • Start by capturing a small sample file. Type: nmon -f -s2 -c 30
      • The -f means you want the data saved to a file and not displayed on the screen.
      • The -s 2 means you want to capture data every 2 seconds
      • The -c 30 means you want thirty data points or snap shots
      • This means that after a few seconds collecting the configuration nmon for Linux will run for 2 x 30 = 60 seconds and stop. At the end some further configuration data is collected.
    • As nmon for Linux start up briefly check you system and options and then disconnect from you terminal sessions.
    • It then runs like a daemon process in the background. The point is that if you log out or get disconnected then nmon will complete the data file capture - this is a good thing.
    • If you want to be sure nmon is still running you can't use a simple "ps" because it is not associated with your log on session. Use "ps -ef | grep nmon" instead.
    • It is a common mistake to try to start using nmon for Linux output file before nmon has finished - this results in a either incomplete data which messes up a later tool or if you asked to a longer time between snap shots then there may be no data actually in the file, which confuses all tools trying to analyse the file.
    • Once nmon for Linux has finished and to build confidence try:
    • Use: grep ZZZZ yourfile.nmon
      • This should output the one line for each snapshot and date and time it happened.
    • Edit the nmon file with vi. You will notice it is a simple text file The start of each line defines the content of the line and then the values are separated with commas. This means the file can be imported in to a spread sheet. If you want to manually import the file, make sure you sort the file first (with the um er "sort" command). This sort means all the lines of a particular type are together. A sort is not required by most of the nmon for Linux analysing tools as they perform the function themselves.



    Hint for nmon version 16d


    # ./nmon -h

    Hint for nmon version 16d

    Full Help Info : nmon -hOn-screen Stats: nmonData Collection: nmon -f [-s ] [-c ] [-t|-T]Capacity Plan : nmon -x

    Interactive-Mode:

    Read the Welcome screen & at any time type: "h" for more helpType "q" to exit nmon

    For Data-Collect-Mode

    -f Must be the first option on the line (switches off interactive mode)Saves data to a CSV Spreadsheet format .nmon file in then local directoryNote: -f sets a defaults -s300 -c288 which you can then modifyFurther Data Collection Options:-s time between data snapshots-c of snapshots before exiting-t Includes Top Processes stats (-T also collects command arguments)-x Capacity Planning=15 min snapshots for 1 day. (nmon -ft -s 900 -c 96)

    End of Hints


    Full Help Information for nmon 16d


    For Interactive and Data Collection Mode:

    User Defined Disk Groups (DG) - This works in both modesIt is a work around Linux issues, where disks & partitions are mixed up in /proc files& drive driver developers use bizarre device names, making it trick to separate them.-g Use this file to define the groups- On each line: group-name (space separated list)- Example line: database sdb sdc sdd sde- Up to 64 disk groups, 512 disks per line- Disks names can appear more than one group-g auto - Will generate a file called "auto" with just disks from "lsblk|grep disk" outputFor Interactive use define the groups then type: g or GFor Data Capture defining the groups switches on data collection

    Data-Collect-Mode = spreadsheet format (i.e. comma separated values)

    Note: Use only one of f, F, R, x, X or z to switch on Data Collection modeNote: Make it the first argument then use other options to modify the defaultsNote: Don't collect data that you don't want - it just makes the files too largeNote: Too many snapshots = too much data and crashes Analyser and other toolsNote: 500 to 800 snapshots make a good graph on a normal size screenRecommended normal minimal options: snapshots every 2 minutes all day:Simple capture: nmon -f -s 120 -c 720With Top Procs: nmon -fT -s 120 -c 720Set the directory: nmon -fT -s 120 -c 720 -m /home/nag/nmonCapture a busy hour: nmon -fT -s 5 -c 720 -m /home/nag/nmon

    For Data-Collect-Mode Options

    -f spreadsheet output format [note: default -s300 -c288]output file is _YYYYMMDD_HHMM.nmon-F same as -f but user supplied filenameNot recommended as the default file name is perfectThe other options in alphabetical order:-a Include Accelerator GPU stats-b Online only: for black and white mode (switch off colour)-c The number of snapshots before nmon stops-d To set the maximum number of disks [default 256]Ignores disks if the systems has 100's of disk or the config is odd!-D Use with -g to add the Disk Wait/Service Time & in-flight stats-f and -F See above-g User Defined Disk Groups (see above) - Data Capture: Generates BBBG & DG lines-g auto See above but makes the file "auto" for you of just the disks like sda etc.-h This help output-I Set the ignore process & disks busy threshold (default 0.1%)Don't save or show proc/disk using less than this percent-l Disks per line in data capture to avoid spreadsheet width issues. Default 150. EMC=64.-m nmon changes to this directory before saving to fileUseful when starting nmon via cron-M Adds MHz stats for each CPU thread. Some POWER8 model CPU cores can be different frequencies-N Include NFS Network File System for V2, V3 and V4-p nmon outputs the PID when it starts. Useful in scripts to capture the PID for a later safe stop.-r Use in a benchmark to record the run details for later analysis [default hostname]-R Old rrdtool format used by some - may be removed in the future. If you use this email Nigel-s Time between snap shots - with "-c count" decides duration of the data capture-t Include Top Processes in the output-T As -t plus it saves command line arguments in UARG section-U Include the Linux 10 CPU utilisation stats (CPUUTIL lines in the file)-V Print nmon version & exit immediatelyTo manually load nmon files into a spreadsheet:sort -A *nmon >stats.csvTransfer the stats.csv file to your PCStart spreadsheet & then Open with type=comma-separated-value ASCII fileThis puts every datum in a different cellNow select the data of one type (same 1st column) and graph itThe nmon Analyser & other tools do not need the file sorted.

    Capacity Planning mode - use cron to run each day

    -x Sensible spreadsheet output for one dayEvery 15 mins for 1 day ( i.e. -ft -s 900 -c 96)-X Sensible spreadsheet output for busy hourEvery 30 secs for 1 hour ( i.e. -ft -s 30 -c 120)-z Like -x but the output saved in /var/perf/tmp assuming root user

    Interactive Mode Keys in Alphabetical Order

    Start nmon then type the letters below to switch on & off particular statsThe stats are always in the same order on-screenTo see more stats: make the font smaller or use two windowsKey --- Toggles on off to control what is displayed ---b = Black and white mode (or use -b command line option)c = CPU Utilisation stats with bar graphs (CPU core threads)C = CPU Utilisation as above but concise wide view (up to 192 CPUs)d = Disk I/O Busy% & Graphs of Read and Write KB/sD = Disk I/O Numbers including Transfers, Average Block Size & Peaks (type: 0 to reset)g = User Defined Disk Groups (assumes -g when starting nmon)G = Change Disk stats (d) to just disks (assumes -g auto when starting nmon)h = This help informationj = File Systems including Journal File Systemsk = Kernel stats Run Queue, context-switch, fork, Load Average & Uptimel = Long term Total CPU (over 75 snapshots) via bar graphsL = Large and =Huge memory page statsm = Memory & Swap statsM = MHz for machines with variable frequency 1st=Threads 2nd=Cores 3=Graphsn = Network stats & errors (if no errors it disappears)N = NFS - Network File System1st NFS V2 & V3, 2nd=NFS4-Client & 3rd=NFS4-Servero = Disk I/O Map (one character per disk pixels showing how busy it is)Particularly good if you have 100's of disksp = PowerVM LPAR Stats from /proc/ppc64/lparcfgq = Quitr = Resources: Machine type, name, cache details & OS version & Distro + LPARt = Top Processes: select the data & order 1=Basic, 3=Perf 4=Size 5=I/O=root onlyu = Top Process with command line detailsU = CPU utilisation stats - all 10 Linux stats:user, user_nice, system, idle, iowait, irq, softirq, steal, guest, guest_nicev = Experimental Verbose mode - tries to make recommendationsV = Virtual Memory statsKey --- Other Interactive Controls ---+ = Double the screen refresh time- = Halves the screen refresh time0 = Reset peak counts to zero (peak highlight with ">")1 = Top Processes mode 1 Nice, Priority, Status3 = Top Processes mode 3 CPU, Memory, Faults4 = Top Processes mode 4 as 3 but order by memory5 = Top Processes mode 5 as 3 but order by I/O (if root user)6 = Highlights 60% row on Long Term CPU view7 = Highlights 70% row on Long Term CPU view8 = Highlights 80% row on Long Term CPU view9 = Highlights 90% row on Long Term CPU view. = Minimum mode i.e. only busy disks and processes shownspace = Refresh screen now

    Interactive Start-up Control

    If you find you always type the same toggles every time you startthen place them in the NMON shell variable. For example:export NMON=cmdrtn

    Other items for Interactive and Data Collection mode:

    a) To limit the processes nmon lists (online and to a file)either set NMONCMD0 to NMONCMD63 to the program namesor use -C cmd:cmd:cmd etc. example: -C ksh:vi:syncd

    Other items for Data Collection mode:

    b) To you want to stop nmon use: kill -USR2 c) Use -p and nmon outputs the background process pidd) If you want to pipe nmon output to other commands use a FIFO:mkfifo /tmp/mypipenmon -F /tmp/mypipe &tail -f /tmp/mypipee) If nmon fails please report it with:1) nmon version like: 16d2) the output of: cd /proc; cat cpuinfo meminfo partitions stat vmstat3) some clue of what you were doing4) I may ask you to run the debug version or collect data filesf) If box & line characters are letters then check: terminal emulator & $TERMg) External Data Collectors - nmon will execute a command or script at each snapshot timeThey must output to a different file which is merge afterwards with the nmon outputSet the following shell variables:NMON_START = script to generate CVS Header test line explaining the columnsGenerate: TabName,DataDescription,Column_name_and_units,Column_name_and_units ...NMON_SNAP = script for each snapshots data, the parameter is the T0000 snapshot numberGenerate: TabName,T00NN,Data,Data,Data ...NMON_END = script to clean up or finalise the dataNMON_ONE_IN = call NMON_START less often (if it is heavy in CPU terms)Once capture done: cat nmon-file data-file >merged-file ; ready for Analyser or other toolsThe nmon Analyser will automatically do its best to graph the data on a new Tab sheetDeveloper: Nigel Griffiths See http://nmon.sourceforge.net Feedback welcome - On the current release onlyNo warranty given or implied. Copyright GPLv3

    nmon for Linux Support

    If you:

    • need help running nmon or understanding the data saved
    • have a suggestion for improvements
    • have bug fixes or want to report errors
    • want to extend nmon for Linux on to other platforms
    • want to include nmon on your distro (not a problem, I would just like to know)

    getting in touch at the?nmon for Linux - Help Forum

    • Alternatively use the?Performance Tools Forum




    NMON_Analyser User Guide for V4.6


    Preface

    NMON_Analyser is designed to complement NMON (Nigel’s Monitor) in analysing and reporting performance problems; it produces graphs for virtually all sections of output created using the “spreadsheet output” mode of NMON as well as doing some additional analyses for ESS, EMC and FAStT subsystems. ?It will also work with files produced by topasout and with other tools that produce data in “NMON” format. It is written in VBA for Excel and will work with Excel 2007 or later. ???It may work also on Excel 2003 with the required Microsoft updates to support .xlsx files https://support.microsoft.com/kb/924074?wa=wsignin1.0

    ?

    NMON was originally written by Nigel Griffiths (nag@uk.ibm.com) and is now (since AIX 5.3 TL09 and AIX 6.1 TL02) part of topas. ?NMON_Analyser was originally written by Stephen Atkins with contributions from many people – including Ralf Schmidt-Dannert and Markus Fehling, both of IBM. ?Currently nmon Analyzer is maintained by Ron McCargar (mccargar@us.ibm.com.)

    ?

    Support for NMON_Analyser is provided on a best efforts basis. ?Please direct questions to the User Forum (see below) rather than contact the author direct. ???

    ?

    Links:

    NMON_Analyser home page https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power%20Systems/page/nmon_analyser

    User Forum https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000749&ps=25


    Collecting data using NMON

    Please make sure you have the latest versions of both NMON and NMON_Analyser before starting a new engagement. ?If you want automatic notification of a new release of the Analyser send a note to steve_atkins@uk.ibm.com?and I’ll add you to my distribution list. ?Nigel maintains a similar list for NMON.

    ?

    You will need to have root privileges in order to get a complete set of data on the BBBP sheet. ?In order to collect data for the DISKBUSY sheets you need to make sure that iostat data collection is enabled:

    chdev -l sys0 -a iostat=true

    ?

    For spreadsheet output mode (comma separated values) use the following flags when invoking nmon:

    -f ??????????spreadsheet output format [note: default -s300 -c288]

    ???????? Output file is _YYYYMMDD_HHMM.nmon

    -F?? same as -f but user supplied filename

    -c???? number of snapshots

    -d? ??? requests disk service and wait times (DISKSERV and DISKWAIT)

    -i ??Ignore processes using less than this amount of CPU when generating TOP section – useful for reducing data volumes

    -g file containing disk group definitions

    -l?? number of hdisks per sheet - defaults to 150, maximum 250. ?See notes

    -m?

    NMON changes to this directory before saving the file


    -r??? goes into spreadsheet file [default hostname]

    -s??? interval between snap shots

    -x? capacity planning (15 mins for 1 day = -fdt -s900 -c96)

    -t ????????include top processes in the output

    -T ????????as?–t plus saves command line arguments in UARG section

    -A ????????include data for async I/O (PROCAIO) sections

    -Dprevents DISK sections being produced (useful when Disk Groups are being used because there are too many hdisks to process)

    -E ????????stops ESS sections being produced (necessary when Disk Groups are being used because there are too many vpaths to process)

    -J prevents JFS sections being produced (prevents Excel errors when you have more than 255 filesystems)

    -Lincludes LARGEPAGE section

    -Ninclude NFS sections

    -Sinclude WLM sections with subclasses

    -Winclude WLM sections without subclasses

    -Yinclude SUMMARY section (very efficient alternative to –t?if PID level data is not required)

    ?

    ????????example: nmon_aix51 -F asterix.nmon -r Test1 -s6 -c12


    Notes:

    1.?The –f?(or –F) flag must appear first.

    2.?The value of the -l?flag controls the number of hdisks per sheet on the DISK sheets and per line on the BBBD sheet. ??There are two factors to consider when choosing this value. ?Excel has a limit of 256 columns per sheet; however, both NMON and NMON_Analyser use some columns, so the upper limit is really 250. ??The second factor is that Excel VBA has an upper limit of 2048 bytes for input line length. ??This particularly affects users of EMC systems that use long hdisk names (e.g. hdiskpower123). ??The default of 150 is safe for such systems. ??Other users may set the value to 250 in order to reduce the number of output sheets.

    3.?Consider the value of the -s?flag very carefully. ??The shorter the interval between snapshots, the more variable the values for each resource will be. ?If you use an interval of 1 second, don’t be surprised to see many of your disks hitting 100% busy for short periods. ?For normal monitoring, 10-minute intervals (-c 600) provide a good balance.

    4.?The graphs produced by NMON_Analyser look best when the number of snapshots (specified by the -c?flag) is 300 or less. ???

    5.?The TOP section (produced by specifying the -t?flag) can generate large amounts of output and the size of the output can grow exponentially if a large value is specified for the -c?flag. ??If you want the TOP section then specify no more than 250 snapshots - ideally less.

    Collecting data using topas (xmwlm)

    You need AIX V5.3 TL5 Service pack 4 with APAR IY87993 or later. ???Note that only the output using the –a?flag can be analysed. ?In particular, cross-partition statistics cannot be analysed; if you wish to get a report for the entire machine, collect data from each LPAR separately and then use NMON_Consolidator to merge the data. ?Commands like the following are required to collect the data.

    ?

    topasout -a /etc/perf/daily/xmwlm.yymmdd

    :

    cp /etc/perf/daily/xmwlm.yymmdd_01 hostname.ddmmyy.topasout.csv

    Using NMON_Analyser

    ·?FTP the input file to your PC – ideally using the ASCI or TEXT options to make sure that lines are terminated with the CRLF characters required by Windows applications.

    ·?Open the NMON_Analyser spreadsheet and specify the options you want on the “Analyser” and “Settings” sheets (see below). Save the spreadsheet if you want to make these options your personal defaults.

    ??Click on the "Analyse nmon data" button and find/select the .nmon file(s) to be processed. ?You may select several files in the same directory. ?If you wish to process several files in different directories you may wish to consider using the “FILELIST” option described below.

    ??You may see the message SORT command failed for “filename”?if the file has >65K lines and the filename (or directory name) contains blanks or special characters. ?Either rename the file/directory or just pre-sort the file before using the Analyser.

    Analyser options

    ?

    GRAPHSThe first option is?either ALL?or LIST. ??If the value is LIST?then only those sheets which appear in the LIST on the Settings sheet will have graphs drawn for them. ???This option is particularly useful if the graphs are to be printed/published or to reduce the amount of memory/fonts/disk space required when analysing files from large systems.

    The second is?either CHARTS, PICTURES, PRINT?or?WEB. ??The meaning of these are as follows:

    ·?CHARTS produce Excel charts in-place on the selected sheets

    ·?PICTURES graphs will be produced on a separate “Charts” sheets as pictures. ?Selecting this option can reduce the size of the output file by up to 90%.

    ·?PRINTimplies PICTURES. ?Pictures will be printed to the designated printer (see “Printing Options” below)

    ·?WEB implies PICTURES. ?Automated web publishing (see “Web Publishing” below)

    INTERVALSspecifies the first and last time interval to be processed. ??Intervals outside this range will be discarded after parsing. ??Note that these are numbers between 1-999999 and are not time values. ??Setting a value of 2 for the first interval is useful in discarding the very large numbers that often appear at the start of an NMON collection run with AIX. ??If you have used a splitter program on the input file, or if you are analysing data from a LINUX system, then you should leave this as 1.

    TIMESspecifies the first and last time/date to be processed. ??Samples outside this range will be discarded after parsing. ??They can be specified in any form recognised by Excel as time/date values; e.g. ?

    14:00:1016:15:30

    ?4-Aug-12 6-Aug-12

    18:00 28/6/1204:00

    ????Notes:?

    ·?Specifying a date without a time is the same as specifying a time of 00:00.

    ·?If the second time is less than the first then Analyser will assume the second time is in the next calendar day

    ·?If there are no qualifying intervals in the file a message will be issued:

    “Invalid values for FIRST/LAST - values reset to 1/999999”

    and the entire file will be processed.

    MERGEspecifying YES?here results in NMON_Analyser merging all of the input files to form a single file. ??The input files must be unsorted. By default the Analyser will delete the TOP and UARG sections during the merge process; specify TOP?to prevent this from happening but be aware that processing time will be increased and that if the TOP section exceeds the maximum number of lines per sheet (depending on the version of Excel) then data will be lost. Specify KEEP?to stop the merged file from being deleted at the end of the run. ?Specify ONLY?if you simply want to create a merged file for future analysis.

    PIVOTspecifying YES?here results in NMON_Analyser creating ?a Pivot Chart from the specified sheet after all other processing (including printing/publishing the other charts) has completed. ??See additional parameters on the “Settings” sheet.

    ESSspecifying NO?here results in NMON_Analyser bypassing the additional analysis performed for ESS subsystems. ??This will result in faster analysis and can allow larger files to be analysed successfully when “out of memory” errors occur.

    FILELISTthe name of a control file containing a list of nmon output files to be processed by the Analyser. ?Leave this field blank for normal operation. ?The name must be fully qualified (e.g. c:\nmon\testcases\filenames.txt). ??The names specified in the text file must contain full path information. ?Wildcard characters may be included in the filenames so long as they conform to Windows standards. For example:

    c:\nmon\testcases\*.nmon

    NB: if you save the spreadsheet with a value in this field, the Analyser will automatically begin execution the next time you open it. ?This is defined as “batch mode” (see Appendix). ?You can stop the execution by pressing Ctrl+Break or by deleting/renaming the Control File.

    Batch Processing Options

    The following fields can be found on the “Settings” sheet.

    ?

    REPROCChange this to NO?if you want to bypass processing of input files which may have been processed in a previous run. ??This is useful if you make use of wildcards in the batch control file. ?Note that the REPROC option only takes effect when you have specified more than one input file. ?????

    OUTDIRthe name of an existing directory in which output files will be saved by default. This is primarily intended for batch operation (see FILELIST?above) but also works to set the default directory for interactive sessions. ?If the directory does not exist (or OUTDIR is blank) then output files are saved back to the same directory as the corresponding input files.

    Example: C:\NMON\Analyser\Output\

    Formatting Options

    The following fields can be found on the “Settings” sheet.

    ?

    BBBFontEnter the name of a fixed pitch font to be used for formatting the BBBC and BBBP sheets. ?Courier?provides acceptable results.

    GWIDTH Change the values in this row to make the generated graphs bigger or smaller. ?The default value of 0 means that the Analyser will dynamically size the graphs according to your screen size, font settings or page size. ??Be careful not to set a value larger than your page width when printing.

    GHEIGHT? Change the values in this row to make the generated graphs bigger or smaller. ?If you specify a value here you must also specify a value for GWIDTH.

    LIST? A comma-separated list a sheets for which the Analyser is to draw graphs. ?Only used if the GRAPHS option is set to LIST. ??The list can contain any valid wildcard characters recognised by Excel, e.g. “EMC*” ???

    Note: graphs are always drawn for SYS_SUMM, CPU_SUMM and DISK_SUMM

    NOLISTThe default is?KEEP. ?If you change this to DELETE?then all sheets which do not appear in LIST will be deleted after analysis. ?This can dramatically reduce the size of files that are to be kept for long periods.?

    CPUmaxSpecifies the maximum number of CPUnnn, PCPUnnn and SCPUnnn sheets that will be generated. ???The default value of 0 will choose all sheets for an LPAR using dedicated processors and number equal to the SMT mode for an LPAR using shared processors. ???

    REORDERspecifying YES?here results in NMON_Analyser reordering the sheets to improve navigation to more relevant information.

    TOPDISKSthe maximum number of hdisks/vpaths to include on disk graphs. ?A value of 0 produces graphs containing all the hdisks on a sheet (up to 250). ??Graphs containing more than 50 hdisks will be automatically scaled to fit and may therefore exceed the size of the screen.

    xToDFormat to be used for timestamps on Time of Day graphs. ?Anything acceptable to Excel as a Number Format Code ?may be entered. ??Default is hh:mm. ?Note that ?the date is also available within the timestamp and you may therefore use something like dd-mmm-yy hh:mm if, for example, you have merged multiple NMON files together. ?If you use something other than the default string you may need to increase the value of GHEIGHT – experiment with different values if you don’t see what you expect.

    SORTDEFAULT ?This setting indicates if the 1st graph on 'default' sheets (ones not handled else where by the Analyzer) are sorted. ?Note:?If Yes, this will also reorder the columns.

    Pivot Chart

    These parameters are used to construct a pivot chart. ????The required parameters are: Sheetname, PageField, RowField, ColumnField, DataField and xlFunction (can be COUNT, SUM, MIN, AVG, MAX). ?This is primarily useful for the TOP and SUMMARY sheets but might prove useful for other, possibly user-supplied, data sheets. ??????

    Printing Options

    The following fields can be found on the “Settings” sheet. Note - these only take effect if you select PRINT?for the OUTPUT option on the Analyser sheet.

    LSCAPEChange to YES?if you want the Analyser to set the page orientation to Landscape. ??By default the Analyser will fit one chart per page when printing landscape. ?

    COPIESSet to the number of copies to be printed.

    PRINTERThe name of the printer. ?Specifying a value of PREVIEW will cause the Analyser to invoke the Excel print preview function - useful for testing. ???You may also specify DEFAULT to print to the system default printer or the name of network printer.

    The Analyser adds page headers and footers.

    Web Publishing Options

    The following fields can be found on the “Settings” sheet.

    PNGChange to NO?if your browser can’t handle the PNG graphics format. ?Graphics will be generated as GIF files.

    SUBDIRIf this is?YES then?all supporting files, such as background textures and graphics, are organized in a separate folder. If this is NO?then supporting files are saved in the same folder as the Web page.

    WEBDIRthe name of an existing directory in which HTML files will be saved by default. ?If the directory does not exist (or OUTDIR is blank) then output files are saved back to the same directory as the corresponding input file.?

    Example: C:\NMON\Analyser\HTML\??

    Interpreting the output sections

    Notes on the “Weighted Average” as used in the Analyser

    Several graphs produced by the Analyser show average, weighted average and maximum values. ??Although everyone understands averages and maximums, the concept of a weighted average is a little more difficult to grasp. ?

    ?

    One of the problems we are faced with in analysing sample data is that resources on the target system may be idle for long periods during the collection. ?For example, the NMON data collection may be started some time before the system reaches peak utilisation and may not be stopped until the workload being monitored has long since finished. ??Although this does not affect the maximums it can severely affect the accuracy of the averages.

    ?

    The idea of a weighted average is to apply a weighting factor to each snapshot to indicate how relevant that snapshot is to the average. ???In NMON_Analyser, we use the value of the measurement itself as the weighting factor. ??In effect, this produces a figure that shows how busy a resource is when it is active. ?For example: ?a database log disk is only active during the middle part of a benchmark. ??We record the following figures for %tm_act (DISKBUSY):

    ?

    Snapshot

    %tm_act

    1

    0

    2

    0

    3

    0

    4

    0

    5

    0

    6

    6.1

    7

    6.3

    8

    6.5

    9

    9.1

    10

    5.9

    11

    0

    12

    0

    13

    0

    14

    0

    15

    0

    ?

    The average for this set of data is 2.3 and the weighted average is 7.0. ??The weighted average gives a better picture of how busy the disk is while logging is taking place. ??NMON_Analyser uses the weighted average as a sort key when sorting the contents of disk sheets.

    ?

    Note that, occasionally, NMON generates very large numbers for the first interval and this heavily skews the weighted average values. ??If you see this problem occurring then change the ?value for the FIRST parameter to 2 in order to exclude the first interval completely. ??This is only a problem for AIX.

    SYS_SUMM

    This section is entirely generated by the Analyser and contains a useful summary of data taken from other sheets. ???Note the the avg/max values for User%, Sys%, Wait% and Idle% are independent and will not add up to 100%. ??The CPU% column shows User% + Sys % for each line. ??

    ?

    For non-partitioned or dedicated CPU partitions the graph shows the total CPU Utilisation (%usr + %sys) together with the Disk I/O rate (taken from the DISKXFER sheet) by time of day. ??For micro-partitions the graph shows the number of physical CPUs being used instead of CPU%.

    ?

    The value “Max:Avg” is simply the maximum value divided by the average. ?If monitored over a long period of time the value for CPU% can be useful in spotting a system reaching saturation level (the ratio will steadily decrease). ?If you have historical data then the value can also be useful in determining how much latent demand is present in a system running at saturation level.

    AAA

    The AAA section is generated by NMON at the start of the data collection and contains information about the system and NMON itself – contents vary by release, following is for 12e

    ?

    AIXthis is the release / maintenance level of AIX being used on the target system as reported by the lslpp?command.

    buildthe particular build of nmon used to collect this data

    commandthe command line used to invoke NMON and the date when it finished.

    cpus? the number of CPUs in the system and the number active at the start of data collection.

    datedate at the start of the collection.

    disks_per_linethe setting of the?-l?flag on the NMON command line or the default value.

    hardwarethe processor technology used in the target system.

    hostthe hostname of the target system.

    Intervalthe time (in seconds) between snapshots.

    kernelinformation from the kernel - useful in identifying the type of kernel (32-bit or 64-bit) and whether this is an LPAR.

    LPARNumberName the LPAR number followed by the name

    MachineTypemachine type and model of the system

    prognamethe name of the NMON executable.

    runnametaken from the NMON command line if specified using the -r?parameter, else defaults to hostname.

    SerialNumber the machine serial number

    snapshots?the number of snapshots - this is used by the Analyser. ??The Analyser will modify this value to match the number of snapshots actually found in the input file.

    stealthis value will be a 1 if running on Linux and Steal% CPU is provided in the CPUnnn and CPU_ALL data.

    subversion?detailed information about the nmon version used to collect this data – including the date and time it was created.

    timetime as shown by the system clock at the start of the collection. Also see date value.

    timestampsizethe number of characters used for timestamps in each record. ?The default is 5.

    TLthe Technology Level of the AIX release

    userthe name of the user executing the NMON command.

    versionthe version of NMON used to collect this data.

    VIOSthe release/maintenance level if ?this is a VIOS LPAR.

    analyser the version of NMON_Analyser used to generate the output file together with the elapsed time (in seconds) for processing this file.

    environmentthe version of Excel you are using.

    parms the values of ?most of the user options specified on the “Analyser” sheet.

    settings the values of ?most of the user options specified on the “Settings” sheet.

    elapsed the execution time of the Analyser. ?I use this for tuning.

    NodeNamethe name of the LPAR’s node name

    ?

    NMON_Analyser deletes the NOTES?lines generated by NMON.

    ?BBBB

    The BBBB sheet lists all of the disks listed in the ODM together with the capacity (in Gbytes) and the adapter type (SCSI/SSA/Fibre) as reported by lsdev. ??Note that some fibre-attached devices do not report their capacity to AIX.

    NMON_Analyser deletes the column containing the sort key generated by NMON on all BBB sheets.

    BBBC

    The BBBC sheet shows the output from the lspv?command for all local disks at the start of the data collection. The Analyser highlights the hdisk name using a bold font and sets the sheet to use the fixed-pitch font specified on the NLS sheet (default Courier) in order to improve readability.

    BBBD

    The BBBD sheet shows a list of all I/O adapters listed in the ODM together with the hdisks addressed through that adapter. ?

    BBBE

    The BBBE sheet contains data extracted from the lsdev?command and shows the mapping between vpaths and hdisks. ?NMON_Analyser uses this information to construct the ESSBUSY, ESSRSIZE and ESSWSIZE sheets.

    BBBG

    The BBBG sheet contains details of the NMON disk group mappings.

    BBBL

    The BBBL sheet is only produced if the operating system is running in a partitions and contains details of the configuration of the LPAR at the start of the collection run.

    BBBN

    The BBBN describes each network adapter in the system and shows the name, speed and MTU size.

    BBBP

    The BBBP sheet contains the un-interpreted output from the?emstat?and lsattr?commands. ?Note that to get output from these commands requires NMON to be running with root privileges.

    The Analyser sets the sheet to use the fixed-pitch font specified on the NLS sheet (default Courier) in order to improve readability. ?

    BBBR

    This sheet records dynamic LPAR reconfiguration events during the collection run.

    BBBV

    This sheet lists all of the volume groups present at the start of the collection run.

    CPUnnn

    These sheets show %usr, %sys, %wait?and %idle?by time of day for each logical processor. ?Note that for micropartitions the Idle%?and Wait% figures will include times when the physical processor was ceded to the shared pool.

    NMON_Analyser generates a graph and a column headed “CPU%” containing the sum of %usr?and %sys?for use on the CPU_SUMM sheet. ??The Analyser also adds blank intervals for CPUs that are varied online during the collection interval so that the graphs cover the entire collection period. ??

    ?

    If REORDER?is set to YES?on the Control sheet the Analyser will move all CPUnn sheets to the end of the file.

    CPU_ALL

    This sheet shows the average utilisation for all physical threads by time of day. ??Note that for micro-partitions CPU%?is a measure of utilisation vs the entitlement but is adjusted by libperfstat so that it never exceeds 100%; this makes the numbers virtually useless for analysis of uncapped partitions and you may choose to look at the charts on the LPAR sheet instead.??Note also that micro-partitions generally record very little Idle%?or Wait%?because they will normally cede their timeslice to other LPARs rather than waste CPU time waiting for work. ???If the CPUmax?value is lower than the number of physical threads in use then the graph title will show the number of threads excluded.

    ?

    The second graph shows the number of active CPUs by time of day and is useful in determining whether CPUs have been varied on/off during the collection period. ?If the system has SMT enabled then the Analyser shows logical CPUs rather than physical CPUs and the legend reports “SMT=ON”.

    CPU_SUMM

    The Analyser generates this sheet from data on the CPUnnn sheets. ??It gives a breakdown of CPU Utilisation by thread (logical processor) and by core over the collection period. ??The chart can be very useful in identifying situations in which the system is thread-starved (i.e. too few threads to fully utilise the logical processors) or where the workload is dominated by a small number of ?single-threaded processes.

    ?

    Note that if CPUs have been dynamically reconfigured during the collection period, these figures reflect only those intervals when the CPU was varied on.

    General notes for DISK, ESS, EMC, FASt and DG sheets

    The DISK sheets record device statistics for each hdisk in the system. ??If there are more than ?hdisks in the system (see "Collecting data using NMON" above) NMON generates multiple output sections. ??DISKBUSY will therefore contain device busy statistics for the first group of hdisks, DISKBUSY1 for the next, DISKBUSY2 for the next and so on. ?

    ?

    The Analyser will normally sort the contents of the sheet in ascending sequence using the weighted average values as a sort key. ?However, if a storage subsystem is detected then sorting of the DISK sheets is disabled and the ESS/EMC/FASt/DG sheet contents are sorted instead. ??

    ?

    If the number of hdisks (or vpaths) on the sheet exceeds the value specified for TOPDISKS?(see “Analyser options” above), the graph will only show information for the specified number of disks and a warning will appear in the graph title.

    ?

    If REORDER?is set to YES?on the Control sheet the Analyser will move all but the DISKBUSY, DISK_SUMM and DISKSERV sheets to the end of the file when a storage subsystem or Disk Group is detected. ??The rationale behind this being that the ESS/EMC/FASt/DG sheets contain the most useful data and that the DISK sheets merely replicate it. ??However, the DISKBUSY sheet is useful for checking the activity on system disks (normally local) and the DISK_SUMM sheet gives total data rates for the system (local + subsystem disks).

    ?Note that NMON and NMON_Analyser can only handle a maximum of 250 vpaths in a system. ?If you have more than this then you need to use the Disk Groups feature of NMON to select the vpaths that are of primary interest.

    EMC/PowerPath subsystems

    NMON_Analyser detects the presence of an EMC/PowerPath subsystem by scanning the input file for the string “hdiskpower” before starting the analysis. ??EMC PowerPath creates devices called “hdiskpowern” which each map to multiple hdisks. ??They are therefore comparable to the vpaths generated by ESS/SDD. ??However, unlike vpaths, hdiskpower devices appear to AIX (and, therefore NMON) as real disks and NMON therefore records their activity on the DISK sheets. ??

    ?

    NMON_Analyser removes all hdiskpower entries from DISK sheets and moves them to new sheets beginning with the letters EMC. ??For example, hdiskpower entries found on DISKBUSY1 are simply moved to a new sheet called EMCBUSY1. ?If a DISK sheet contains only hdiskpower devices, that sheet is simply renamed. ?

    ?

    All device statistics reported by the Analyser (e.g. System I/O rates) are correct and as NMON excludes hdiskpower activity from the IOADAPT statistics these figures are also correct. ?

    FAStT subsystems

    NMON_Analyser detects the presence of a FAStT subsystem by scanning the input file for the string “dac” before starting the analysis. ?They are handled in the same way as EMC/PowerPath subsystems except that the dac devices are moved to sheets with names beginning "FASt".

    Note that the Analyser is not able to correctly handle systems having both EMC and FASt subsystems.

    DGBUSY

    This sheet records the average value of device busy for each hdisk in the NMON Disk Group.

    DGREAD

    This sheet records the average data rate (Kbytes/sec) for read operations to each NMON Disk Group.

    DGSIZE

    This sheet records the average data transfer size (block size), in Kbytes, for read/write operations to each NMON Disk Group. ?

    DGWRITE

    This sheet records the average data rate (Kbytes/sec) for write operations to each NMON Disk Group.

    DGXFER

    This sheet records the total I/O operations per second to each NMON Disk Group.

    DISKBSIZE

    These sheets record the average data transfer size (block size), in Kbytes, for read/write operations on each hdisk in the system. ?If this number is not very close to the stripe size for the device there may be a problem that could be solved by increasing the value of numclust.

    DISKBUSY

    These sheets record device busy for each hdisk in the system. ?This is the same as the %tm_act?value recorded by iostat. Note that if this sheet contains all zero values then it means you forgot to enable iostat collection before starting nmon:

    chdev -l sys0 -a iostat=true

    DISKREAD

    These sheets record the data rate (Kbytes/sec) for read operations on each hdisk in the system.

    DISKSERV

    These sheets record the service times (in milliseconds) for read/write transfers to each hdisk in the system.

    DISKWAIT

    These sheets record the queue times (in milliseconds) for read/write transfers to each hdisk in the system.

    DISKWRITE

    These sheets record the data rate (Kbytes/sec) for write operations on each hdisk in the system.

    DISKXFER

    These sheets record the I/O operations per second for each hdisk in the system. ??This is the same as the tps?value recorded by iostat.

    DISK_SUMM

    The Analyser creates this sheet. ??It shows the total data rates (reads and writes) in Kbytes/sec plus total I/O rates for all hdisks in the system. ???The figures on this sheet are accurate for all systems including ESS, EMC, FASt and HDS configurations. ??

    ?

    These data are displayed on the chart; the IO/sec data are also graphically displayed on the AAA sheet.

    DONATE

    This sheet records physical processor usage and donation to the shared pool. ?Only present for dedicated LPARs running on ?POWER6 systems.

    EMCBSIZE/FAStBSIZE

    This sheet records the average data transfer size (blocksize), in Kbytes, for read/write operations to each esoteric device in a system using EMC/PowerPath or FAStT.

    EMCBUSY/FAStBUSY

    These sheets record device busy for each esoteric device in a system using EMC/PowerPath or FAStT. ?

    EMCREAD/FAStREAD

    These sheets record the data rate (Kbytes/sec) for read operations to each esoteric device in a system using EMC/PowerPath or FAStT.

    EMCWRITE/FAStWRITE

    This sheet records the data rate (Kbytes/sec) for write operations to each esoteric device in a system using EMC/PowerPath or FAStT.

    EMCXFER/FAStXFER

    These sheets record the I/O operations per second to each esoteric device in a system using EMC/PowerPath or FAStT.

    EMCSERV/FAStSERV

    The Analyser creates this sheet. ??It shows estimated services times (not response times) for each esoteric device over the collection interval. ??The service time is derived from the device busy and the transfer rate taken from the corresponding BUSY and XFER sheets. ??Intervals where the transfer rate is below SVCXLIM?are ignored in order to improve the accuracy of the estimate.

    ERROR

    This sheet shows all errors reported by nmon during the collection run. ??FIRST/LAST intervals are ignored for this sheet. ??Timestamps are not converted to time values.

    ESSBSIZE

    This sheet is only present if ESS?is set to YES?and records the average data transfer size (blocksize), in Kbytes, for read/write operations on each vpath in the system. The data on this sheet are calculated by NMON_Analyser as the average of the DISKBSIZE values for component hdisks as recorded on the BBBE sheet.

    ESSBUSY

    This sheet is only present if ESS?is set to YES?and records device busy for each vpath in a system using ESS. ?The data on this sheet are calculated by NMON_Analyser as the average of the DISKBUSY values for component hdisks as recorded on the BBBE sheet. ?

    ESSREAD

    This sheet records the data rate (Kbytes/sec) for read operations on each vpath in the system. ?This information is provided by NMON.

    ESSWRITE

    This sheet records the data rate (Kbytes/sec) for write operations on each vpath in the system. ?This information is provided by NMON.

    ESSXFER

    These sheets record the I/O operations per second for each vpath in the system. ??This information is provided by NMON.

    FCREAD

    This sheet records the data rate (MBytes/sec) for read operations on each Fibre Channel adapter in the system. ?

    FCWRITE

    This sheet records the data rate (MBytes/sec) for write operations on each Fibre Channel adapter in the system.

    FCXFERIN

    This sheet records the read operations per second for each Fibre Channel adapter in the system.

    FCXFEROUT

    This sheet records the write operations per second for each Fibre Channel adapter in the system. ??

    FILE

    This sheet contains a subset of the fields reported by NMON on the Kernel Internal Statistics panel. ??These are the same values as reported by the sar?command.

    ?

    All fields are rates/sec.

    igettranslations of i-node numbers to pointers to the i-node structure of a file or device.?This is reported as iget/s?by the sar -a?command. Calls to iget occur when a call to to namei has failed to find a pointer in the i-node cache. This figure should therefore be as close to 0 as possible. ?

    nameicalls to the directory search routine that finds the address of a v-node given a path name. This is reported as lookuppn/s?by the sar -a?command.

    dirblknumber of 512-byte blocks read by the directory search routine to locate a directory entry for a specific file. This is reported as dirblk/s?by the sar -a?command.

    readchcharacters transferred by read system call. This is reported as rchar/s?by the sar -c?command

    writechcharacters transferred by write system call. This is reported as wchar/s?by the sar -c?command.

    ttyrawchtty input queue characters. This is reported as rawch/s?by the sar -y?command.

    ttycanchtty canonical input queue characters. This field is always 0 (zero) for AIX Version 4 and later versions.

    ttyoutchtty output queue characters. This is reported as outch/s?by the sar -y?command

    ?

    NMON_Analyser produces two graphs - one showing rates/sec for readch?and writech?by time of day and one showing rates/sec for iget, namei?and dirblk. ?

    FRCA

    This sheet is only generated if FRCA is loaded on the target system. NMON_Analyser produces a graph showing the cache hit ratio (as a percentage). ?If FRCA is not loaded, NMON generates no data and the Analyser consigns the redundant header record to the StrayLines sheet.

    IOADAPT

    For each I/O adapter listed on the BBBC sheet, contains the data rates for both read and write operations (Kbytes/sec) and total number of I/O operations performed. On AIX 5.1 and later, this information is reported by the iostat -A?command. NMON_Analyser reorders the columns on the sheet for easier graphing.

    ?

    The Analyser generates three graphs. ??Note the area charts can be easily converted to line charts if required. ??Simply right click on the white space within the chart area, then select Chart Type>Line>OK.

    IP

    This sheet only appears for topasout. ?

    JFSFILE

    For each file system, this sheet shows what percentage of the space allocation is being used during each interval. ?These figures are the same as the %Used?value reported by the df?command. The column headings show the mount point; sheet BBBC can be used to cross-reference to the file system/LV. ?

    JFSINODE

    For each file system, this sheet shows what percentage of the Inode allocation is being used during each interval. ?These figures are the same as the %Iused?value reported by the df?command. The column headings show the mount point; sheet BBBC can be used to cross-reference to the file system/LV.

    LAN

    This sheet only appears for topasout. ?

    LARGEPAGE

    The graph shows Usedpages and Freepages over time.

    ?

    Columns on the sheet are as follows:

    ?

    Freepagesthe number of large pages on the free list.

    Usedpagesthe number of large pages currently in use.

    Pagesthe number of large pages in the pool.

    HighWaterthe maximum number of pages used since the last reboot.

    SizeMBthe size of a large page in Mbytes.

    ?

    LPAR

    The first ?graph shows the number of physical processors used by this partition vs the entitlement. ??For an uncapped partition the number of physical processors may exceed the entitlement but can never exceed the number of virtual processors allocated to the partition. ??For AIX the graph also shows the number of unfolded virtual processors (AIX will “fold” - stop dispatching work to - excess processors in order to minimise scheduling costs).

    ?

    Note that the ratio of physical processor to entitlement (shown as %entc in the output of the lparstat?command) will generally be higher than CPU%?on the CPU_ALL sheet. ??The reason for this is that a partition that is within its entitlement may wait for a short period of time before ceding a processor that enters an I/O wait or becomes idle. ??This can eliminate unnecessary context switches.

    ?

    The second graph shows CPU utilisation as a percentage of virtual processors – for AIX this is broken down in to usr%, sys% and wait%. ??This level of detail is not available for Linux or releases of NMON prior to version 12.

    ?

    The third graph is only present for AIX systems and shows cpu utilisation of the shared pool by this and other partitions. ??The area marked “UsedPoolCPU%” represents the percentage of the shared pool that has been used by this partition, while the area marked “other% “ represents the percentage used by all other partitions. ??Note that if the partition is not authorised to see utilisation of the shared pool then the pool will appear to be 100% utilised.

    ?

    Columns on the sheet are as follows:

    PhysicalCPUphysical cores consumed by AIX

    VirtualCPUsnumber of Virtual CPUs allocated to the LPAR

    logicalCPUsnumber of threads (i.e.?Virtual CPUs multiplied by SMT mode)

    poolCPUsnumber of cores in the pool that this LPAR occupies

    entitled the number of cores guaranteed to be available to this LPAR

    weightthe priority of this LPAR when competing for unused CPU cycles

    PoolIdle? the number of ?unused cores in the Pool

    usedAllCPU%? percentage of active cores in the machine that this LPAR is using

    usedPoolCPU? percentage of cores in the pool that this LPAR is using

    SharedCPU? 1 if the LPAR is not a dedicated CPU

    Capped? 1 if the LPAR is capped

    EC_User%? percentage of Entitlement used in User mode

    EC_Sys%? percentage of Entitlement used in System mode

    EC_Wait%? percentage of Entitlement waiting for I/O

    EC_Idle%? percentage of Entitlement used in Idle mode

    VP_User%? percentage of Virtual CPU used in User mode

    VP_Sys%? percentage of Virtual CPU used in System mode

    VP_Wait%? percentage of Virtual CPU waiting for I/O

    VP_Idle%? percentage of Virtual CPU spent in Idle mode

    Folded? no. of Virtual CPUs unused for efficiency reasons

    CPU_Pool_id? the Id of the CPU pool, useful if there are multiple pools

    MEM

    The main graph on this sheet shows the amount of Real Free memory in Mbytes by time of day. ?This would be the same as dividing the fre?values reported by vmstat?over the same interval by 256. The small graph shows the amount of real memory. This is useful in determining if dynamic reconfiguration has been used during the collection period. ?

    ?

    For AIX, other columns on the sheet are as follows:

    ?

    Real Freethe percentage of real pages on the free list.

    Virtual Freethe percentage of unallocated virtual slots on the paging spaces.

    Real Free (MB)the amount of memory on the free list in Mbytes.

    Virtual Free (MB)the amount of unallocated space on the paging spaces.

    Real Total (MB)the total amount of memory available to AIX.

    Virtual Total (MB)the total amount of space allocated for paging spaces. ?

    ?

    Note: you can calculate the amount of memory used during an interval simply by subtracting the Real Free (MB) value?from the?Real Total (MB) value. ???This will, however, include file pages. ??The graph on the MEMUSE sheet gives a more accurate assessment of memory used by programs (computational pages).

    MEMUSE

    Except for %comp, the values on this sheet are the same as would be reported by the vmtune?command. ?

    ?

    %numpermthe percentage of real memory allocated to file pages.

    %minpermvalue specified on the vmtune?command or system default of 20%. This will normally be constant for the run unless the vmtune?or rmss?commands are used during collection.

    %maxpermvalue specified on the vmtune?command or system default of 80%. This will normally be constant for the run unless the vmtune?or rmss?commands are used during collection.

    minfreethe minimum number of pages AIX is to keep on the free list. ?Specified on the vmtune?command or system default of maxfree - 8.

    maxfreethe maximum number of pages AIX is steal in order to replenish the free list. ?Specified on the vmtune?command or system default.

    %compThe percentage of real memory allocated to computational pages. NMON_Analyser calculates this value. ?Computational pages are those backed by page space and include working storage and program text segments. ??They exclude data, executable and shared library files.

    ?

    The Analyser generates two graphs. ?The first shows the split between computational and file pages by time of day. ?The second plots the values of %numperm, %minperm, %maxperm and?%comp.

    ??

    If %numperm?falls below %minperm?then computational pages will be stolen. ??If %numperm?rises above %maxperm?then computational pages cannot be stolen. ?Low values for both %minperm?and %maxperm?indicate that the system has been tuned for a database server. ??You may also want to check the setting of STRICT_MAXPERM?on the BBBP sheet (if present).

    MEMNEW

    The graph shows the allocation of memory split into the three major categories: pages used by user processes, file system cache, and pages used by the system (kernel).

    ?

    Process%the percentage of real memory allocated to user processes

    FSCache%the percentage of real memory allocated to file system cache

    System%the percentage of real memory used by system segments

    Free%the percentage of unallocated real memory

    User%the percentage of real memory used by non-system segments

    MEMPAGES4K/64K/16MB/16GB

    These sheets are only present for AIX. ?They show various statistics for different page sizes in use within the system. ???The Analyser will delete the sheets for pages sizes not currently in use.

    ?

    For the MEMPAGES64KB sheet a graph is drawn showing the use of ?both 4KB and 64KB pages within the system – starting with AIX V6.1 (and supporting hardware) these page sizes are selected dynamically based on memory access patterns.

    MEMREAL

    This sheet only appears for topasout. The Analyser adds a column showing Real Free memory in Mbytes. ??

    MEMVIRT

    This sheet only appears for topasout.

    NET?

    This sheet shows the data rates, in Kbytes/sec, for each network adapter in the system (including SP switch if present). ?This is the same as produced by the netpmon –O dd?command. ?NMON_Analyser adds one column for each adapter showing the total data rate (read + write) and two columns showing Total Read and Total Write. ?Note that the Total Write is calculated as a negative number for graphing. ?

    ?

    The Analyser generates three graphs. ??The first graph shows total network traffic broken down as Total-Read and Total-Write. ?The writes are shown below the X-Axis.

    ?

    Note the area chart can be easily converted to a line chart if required. ??Simply right click on the white space within the chart area, then select Chart Type>Line>OK.

    NETPACKET

    This sheet shows the number of read/write network packets for each adapter. This is the same as produced by the netpmon –O dd?command. ??

    NETSIZE

    This sheet shows the average packet size in bytes for each network adapter in the system.

    NFS sheets

    There are separate sheets for NFS2, NFS3 and NF4 client/server. ???The Analyser will delete empty sheets. ??

    PAGE

    This sheet has the paging statistics as recorded by NMON.

    ?

    faultsthe number of page faults per second. This is not a count of page faults that generate I/O, because some page faults can be resolved without I/O.

    pginthe total rate/sec of page-in operations to both paging space and file systems during the interval.

    pgoutthe total rate/sec of page-out operations to both paging space and file systems during the interval.?

    pgsinthe rate/sec of page-in operations from paging space during the interval. ?This is the same as the pi?value reported by vmstat. If pgsin?is consistently higher than pgsout?this may indicate thrashing. ??

    pgsoutthe rate/sec of page-out operations to paging space during the interval. This is the same as the po?value reported by vmstat.

    reclaimsfrom NMON 10 onwards this field is the same as the?fr?value reported by?vmstat and represents the number of pages/sec freed by the page-replacement routine.

    scansthe number of pages/sec examined by the page replacement routine. ?This is the same as the sr?value reported by vmstat. Page replacement is initiated when the number of free pages falls below minfree?and stops when the number of free pages exceeds maxfree.

    cyclesthe number of times/sec the page replacement routine had to scan the entire Page Frame Table in order to replenish the free list. ?This is the same as the cy?value reported by vmstat?but note that vmstat?reports this number as an integer whereas nmon?reports it as a real number.

    fsincalculated by the Analyser as pgin-pgsin?for graphing

    fsoutcalculated by the Analyser as pgout-pgsout?for graphing

    sr/frcalculated by the Analyser as scans/reclaims?for graphing

    ?

    NMON_Analyser produces two graphs. ??The first shows paging operations to/from paging space. ?The ideal here would be no more than 5 operations/sec per page space (see the BBBC sheet for details). ??The second graph shows the scan:free rate. ??Memory may be over-committed when this figure is >4 although you also need to examine the MEM and PAGE sheets as well.

    POOLS

    This sheet contains information about the shared pool in which the LPAR is running. ??Most of the data will only be present if ?“Allow performance information collection.” is set in the LPAR properties.

    ??

    shcpus_in_systhe number of cores allocated to the global shared pool

    max_pool_capacitythe maximum number of VPs defined for this pool

    entitled_pool_capacitythe entitlement for this pool (includes reserve entitlement)

    pool_max_timesame as max_pool_capacity?but may vary in value if the pool definition is changed during the collection run.

    pool_busy_timethe average number of cores in use by this shared pool during the interval

    shcpu_tot_timethe average number of cores available to the global shared pool (including shared dedicated resources?) during the interval

    shcpu_busy_timethe average number of core in use within the global shared pool

    pool_idthis identifier of this pool

    entitledthe entitlement of this LPAR

    PROC

    This sheet contains a subset of the fields reported by NMON on the Kernel Internal Statistics panel. ?The RunQueue?and Swap-in?fields are average values for the interval. All other fields are rates/sec:

    ??

    RunQueuethe average number of kernel threads in the run queue. This is reported as runq-sz?by the sar -q?command and is reported as RunQueue?on the nmon Kernel Internal Statistics panel. ??A value that exceeds 3x the number of CPUs may indicate CPU constraint.

    Swap-inthe average number of kernel threads waiting to be paged in. This is reported as swpq-sz?by the sar -q?command.

    pswitchthe number of context switches. This is reported as pswch/s?by the sar -w?command.

    syscallthe total number of system calls. This is reported as scall/s?by the sar -c?command.

    readthe number of read system calls. This is reported as sread/s?reported by the sar -c?command.

    writethe number of write system calls. This is reported as swrit/s?by the sar -c?command.

    forkthe number of fork system calls. This is reported as fork/s?by the sar -c?command.??

    execthe number of exec system calls. This is reported as exec/s?by the sar -c?command.

    rcvintthe number of tty receive interrupts. This is reported as revin/s?by the sar -y?command.

    xmtintthe number of tty transmit interrupts. This is reported as xmtin/s?by the sar -y?command.

    semthe number of IPC semaphore primitives (creating, using and destroying). This is reported as sema/s?by the sar -m?command.

    msgthe number of IPC message primitives (sending and receiving). This is reported as msg/s?by the sar -m?command.

    ?

    NMON_Analyser produces three graphs - one showing the average length of the RunQueue and the number of?swap-ins by time of day, another showing rates/sec for pswitch?and syscalls?by time of day and a third showing rates/sec for forks?and execs. ?

    ?

    The graph for forks/execs can be useful when monitoring web server systems. ?

    PROCAIO

    This sheet contains information about the number of asynchronous I/O processes available and active (i.e. using more than 0.1% of the CPU) . ????It also shows the amount of CPU being used by the AIO processes during the collection interval.

    ?

    Two graphs are produced. ??The second uses two y-axes. ??The number of running aio processes is shown against the first axis and the amount of cpu used is shown against the second.

    RAWCPUTOTAL, RAWLPAR

    These sheets contain a dump of various counters such as context switches and phantom interrupts. ?

    TCPUDP

    This sheet only appears for topasout.

    TOP

    This sheet is only generated if you specify the -t??flag on the NMON command line. ?The output is similar to that produced using the ps v?command. ??Note that, because of the limitation of having only 65,000 lines on a single sheet, some data may be omitted for very large files and this may mean that entire PIDs or even commands may be missing from the analysis.

    ?

    Note that data are only present for processes that consumed a significant amount of CPU during an interval. ?The TOP sheet does not represent a complete view of the system.

    ?

    NMON_Analyser does the following:

    ??reorders the columns for easier processing. ??

    ??Sorts the data on the sheet into COMMAND?name order - using TIME?as a minor sort key. ?

    ??Creates a table at the end of the sheet summarising the data by command name and used for graphing.

    ?

    You can see the detail section by scrolling to the top of the sheet. ??The summary table is largely obscured by the graphs and so you will need to move (or delete) them for easier viewing.

    ?

    PIDin the detail section this is the process ID of a specific invocation of a command. ?In the summary table this is the command name.

    %CPUin the detail section this is the utilisation of a single processor (rather than of the system) by that PID during the interval. In the summary table this is the average amount of CPU used by all invocations of the command during the collection period.

    %Usrin the detail section this is the average amount of User-mode CPU used by that PID during the interval.

    %Sysin the detail section this is the average amount of Kernel-mode CPU used by that PID during the interval.

    Threadsthe number of (software) threads being used by this command.

    Sizethe average amount of paging space (in Kbytes) allocated for the data section (private segment + shared library data pages) for one invocation of this command. ?This is the same as the SIZE?figure on the ps v?command. ?Note that if Size?is greater than ResData?it means some working segment pages are currently paged out.

    ResTextthe average amount of real memory (in Kbytes) used for the code segments of one invocation of this command. ??Note that multiple concurrent invocations will normally share these pages.

    ResDatathe average amount of real memory (in Kbytes) used for the data segments of one invocation of this command. ?A method of calculating real memory usage for a command is ResText?+ (ResData?* N).

    CharIOthis is the count of bytes/sec being passed via the read and write system calls. The bulk of this is reading and writing to disks but also includes data to/from terminals, sockets and pipes. Use this to work out which processes are doing the I/O.

    %RAMthis is an indication of what percentage of real memory this command is using. This is (ResText + ResData) / Real Mem; it is the same as the %MEM?value on the

    ps v?command. ???Due to rounding/truncation, and the large amounts of memory in modern systems, this is usually 0. ?

    Pagingsum of all page faults for this process. ?Use this to identify which process is causing paging but note that the figure includes asynchronous I/O and can be misleading.

    Commandname of the command

    WLMClassname of the Workload Partition or Workload Manager superclass to which this command has been allocated (64-bit kernel only). ??

    IntervalCPUgenerated by the Analyser. ??In the detail section this shows the total amount of CPU used by all invocations of a command in the time interval. ?It is calculated as the sum of CPU used by all PIDS running the same command divided by the number of active processors (physical cores) available during the interval. In the summary section this is broken down as Average, Weighted Average and Maximum and is used to generate the graph.

    WSetgenerated by the Analyser. ??In the detail section this shows the total amount of memory used by all invocations of a command recorded in the time interval. ?It is calculated as ResText?+ (ResData?* N) (where “N” is the number of copies of this command running concurrently during the interval). ?In the summary section this is broken down as Minimum, Average and Maximum and is used to generate the graph.

    Usergenerated by the Analyser if a UARG sheet is present. ?This contains the name of the user running the process. ?

    Arggenerated by the Analyser if a UARG sheet is present. ?This contains the complete argument string entered for the command.

    ?

    The Analyser generates four graphs using data in the generated table:

    ??A graph showing Average, Weighted Average and Maximum CPU Utilisation by command

    ??A graph showing Minimum, Average and Maximum Memory Utilisation by command

    ??A graph showing Average, Weighted Average and Maximum CHARIO by command

    ??A graph showing the CPU utilisation for each PID for each interval as a scatter chart. ?Note that this chart is only produced if there are fewer than 32,000 lines on the TOP sheet. See below for notes on interpreting this chart.

    Interpreting the %Processor by PID chart

    The purpose of the chart is to provide a link to the UARG sheet so that you can discover precisely which invocation of a command was responsible for using the CPU. It shows the processor utilisation (utilisation of a single CPU) by each PID captured on the TOP sheet. ?Note that a process can use more than 100% of a single CPU if it is multi-threaded. ??

    ?

    Active PIDs will create a cluster of points on the chart. ???The highest point will show the maximum amount of CPU used during any one snapshot. ??To find out which PID a point refers to, move the mouse to position the cursor above it and Excel will display a coordinate pair. ??The first coordinate is the PID – use this to refer to the UARG sheet to find precisely which command was being executed.

    UARG

    This sheet has the first 1,000 commands executed during the collection period. ??The commands are listed in time order. ??Note that commands appearing in the first interval may have been executing prior to the start of the collection.

    ?

    PIDthe process ID of a specific invocation of a command

    PPIDthe parent process ID

    COMMthe command being executed

    THCOUNTthe number of threads started by this process

    USERthe name of the user running this process

    GROUPthe name of the group to which the user belongs

    FullCommand the full command string entered by the user

    VM

    This sheet is only present for Linux systems and contains a dump of the /proc/vmstat file values.

    The two graphs shows file-backed paging (pgpgin/pgpgout) and swap space activity.

    WPAR sheets

    These sheets are only present for AIX V6 and record data for each Workload Partition in the system. ???They are not currently generated by the topas version of nmon.

    WLM sheets

    Contain details of CPU, Memory and I/O bandwidth used by each Superclass/Subclass defined to WLM during the collection run. ??The Analyser will extract subclasss data and create a new set of sheets for each class with more than one subclass. ?These sheets will be named “WLMCPU.class” etc. ?

    If this is a Micro-partition then the Analyser will also create a set of WLMPCPU sheets which will show the physical processor utilisation rather than %CPU utilisation.

    ZZZZ

    The Analyser uses the information on this sheet to automatically convert all NMON time stamps to actual time of day for easier analysis. For NMON10 or later a column is added which contains the date and time as a single value and this is used for the conversions. ?The number of rows on this sheet is used by the Analyser to reset the “snapshots” value on the AAA sheet in case the nmon run was terminated with SIGUSR2.

    ?Error Handling

    Error handling in NMON_Analyser is rudimentary. ???The Analyser can handle many input file errors, but occasionally the analysis will halt leaving you staring at a dialog box. ?Should this happen, please accept my apologies. ?However, before sending me a copy of the input file, please read the following:

    Common problems

    The most commonly reported problems arise from invalid input files. ??We also get problems reported where, for whatever reason, lines have been truncated, split or even duplicated. ??NMON_Analyser attempts to trap these errors and will report them on the “StrayLines” sheet. ?Check this sheet if the run stops unexpectedly.

    ?

    ??'No valid input! NMON run may have failed.'

    ?

    The most common cause of this message is that the NMON run failed and there really is no valid input. ??NMON initialises the output file by writing all of the section headings. ??If it subsequently fails, you will get an output file that consists purely of headings ?- with no data. ??Check the file by loading it into a word processor or, indeed, a spreadsheet (as a .csv file) before you send it to me.

    ?

    ??'Unexpected end of file.'

    ?

    This is only reported when processing files containing more ?lines than can be stored on one sheet and when SORTINP is set to NO. It is usually caused by the fact that lines ?are being terminated with a CR character instead of CRLF. ?Change your FTP option to ASCII ?or TEXT when sending the file to your PC. ?This problem only shows up with large files because the Analyser uses a different technique to read them than that used to read smaller files.

    ?

    Strangely, one of the most common problems I get is caused by the fact that the Analyser can’t handle files with a single data interval ?very well. ??If you want to test the package just let NMON run for a few minutes to get a reasonable data sample!

    Known bugs/problems (V3.4, topas_nmon)

    ??When analysing systems with a very large number of disks, Excel can stop with error “No more fonts can be added” or “Insufficient Resources”. ??Set GRAPHS?to LIST?and select only those sheets you are interested in using the LIST?option on the Settings sheet.

    ??The data can be sorted incorrectly with some versions of nmon (notably 14g) which generate variable length timestamps. ???Process each file separately and specify MERGE=YES to get the data sorted correctly.

    ??The PIVOT option does not seem to work with Excel 2007 or later

    ??The Analyser will crash if you edit the .csv file using Excel prior to the run and the file contains a TOP section. ??If you need to edit the input file, use a word processor.

    ??There are some issues with processing files from systems having both ESS and EMC subsystems attached. ?

    ??When analysing ESS subsystems with more than 253 vpaths some vpath data will be missing from the output. ?Use NMON Disk Groups to combine several vpaths into a single unit for reporting and use the –E?flag to prevent the ESS sections from being produced. ???Alternatively, set ESS?to NO?on the Control sheet and ignore warning messages about data truncation – only the first 253 vpaths will appear in the output.

    How to report a problem?

    Post the relevant information on the nmon forum (see the link in the introduction). ?It can help to include a copy of the original, unmodified?.nmon input file, plus the incomplete output spreadsheet, as a compressed (zipped) file but please consider your system security before doing so.

    ?

    If you have the ability to capture a screenshot then a copy of any dialog boxes also proves useful on occasion.

    ?

    Note:?Development is currently done on Microsoft Excel 2003 (11.8320.8221) SP3. ?It may not be possible to fix problems arising from the use of different releases.

    Excel/VBA Resources/Links

    This is a good source for Excel tips and it has some VBA examples as well:

    http://exceltips.vitalnews.com/

    ?

    This is the home of an excellent reference book:

    http://www.exceltip.com/

    ?

    And of course

    http://www.microsoft.com


    Appendix: Notes on Batch Operation

    If you regularly process large numbers of files, the operation of NMON_Analyser can be completely automated. ???Simply create a text file containing a list of nmon file names (using wild card characters as appropriate) and enter the name of this file into the FILELIST field of the Analyser control sheet. ???Specify the name of an existing directory in the OUTDIR field if you want all of the output files to end up in one place. ??Save the NMON_Analyser spreadsheet under a new name (this is recommended so that you can still use NMON_Analyser for interactive sessions). ?

    Now create Windows .bat files to invoke Excel (see the samples below). ??

    ?

    After processing the last input file, the Analyser will automatically close down Excel. ??Note, however, that this only happens if you load a copy of the Analyser that has a saved FILELIST name and?if there are no other open spreadsheets. ??This allows you to use the FILELIST option safely during an interactive session.

    Sample .bat files

    These sample batch control files are designed to use the pscp file from the PuTTY suite written and maintained by Simon Tatham at http://www.chiark.greenend.org.uk/~sgtatham/putty/

    My thanks to Jamie Dennis for providing them.

    getcsv.bat

    cd \NMON\RawData

    del *.csv

    d:\progra~1\putty\pscp -p -r -l userid host://Performance/NMON/Rawdata/*.csv .

    analyser.bat

    D:

    cd \NMON\FinishedData

    del d:\NMON\FinishedData\*.xls

    "D:\NMON\nmon analyser batch.xls"

    putxls.bat

    D:

    cd \NMON\FinishedData

    d:\progra~1\putty\pscp -p -r -l userid *.xls host:/Performance/NMON/

    Control.txt

    d:\NMON\RawData\*.csv

    NMON Analyser Batch.xls

    OUTDIR

    d:\NMON\FinishedData\

    FILELIST

    d:\NMON\FinishedData\Control.txt







    About Me

    ...............................................................................................................................

    ● 本文整理自網絡

    ● 本文在itpub(http://blog.itpub.net/26736162)、博客園(http://www.cnblogs.com/lhrbest)和個人微信公眾號(xiaomaimiaolhr)上有同步更新

    ● 本文itpub地址:http://blog.itpub.net/26736162/abstract/1/

    ● 本文博客園地址:http://www.cnblogs.com/lhrbest

    ● 本文pdf版及小麥苗云盤地址:http://blog.itpub.net/26736162/viewspace-1624453/

    ● 數據庫筆試面試題庫及解答:http://blog.itpub.net/26736162/viewspace-2134706/

    ● QQ群:230161599???? 微信群:私聊

    ● 聯系我請加QQ好友(646634621),注明添加緣由

    ● 于 2017-07-01 09:00 ~ 2017-07-31 22:00 在魔都完成

    ● 文章內容來源于小麥苗的學習筆記,部分整理自網絡,若有侵權或不當之處還請諒解

    ● 版權所有,歡迎分享本文,轉載請保留出處

    ...............................................................................................................................

    拿起手機使用微信客戶端掃描下邊的左邊圖片來關注小麥苗的微信公眾號:xiaomaimiaolhr,掃描右邊的二維碼加入小麥苗的QQ群,學習最實用的數據庫技術。





    總結

    以上是生活随笔為你收集整理的【OS】NMON的简介和使用的全部內容,希望文章能夠幫你解決所遇到的問題。

    如果覺得生活随笔網站內容還不錯,歡迎將生活随笔推薦給好友。

    99久久精品无码一区二区毛片 | 99久久久国产精品无码免费 | 亚洲乱亚洲乱妇50p | 国内精品久久久久久中文字幕 | 高清无码午夜福利视频 | 无码人妻少妇伦在线电影 | 狠狠色丁香久久婷婷综合五月 | 99久久婷婷国产综合精品青草免费 | 理论片87福利理论电影 | 亚洲成a人一区二区三区 | 久久精品国产一区二区三区肥胖 | 亚洲精品中文字幕乱码 | 一本一道久久综合久久 | 99久久人妻精品免费一区 | 久久亚洲精品中文字幕无男同 | 欧美丰满熟妇xxxx | 99久久久无码国产aaa精品 | 国产精品嫩草久久久久 | 波多野结衣av一区二区全免费观看 | 国产成人精品一区二区在线小狼 | 精品夜夜澡人妻无码av蜜桃 | 狠狠色欧美亚洲狠狠色www | 波多野结衣av一区二区全免费观看 | 中文字幕乱码中文乱码51精品 | 又大又硬又黄的免费视频 | 一二三四社区在线中文视频 | 国产色视频一区二区三区 | 精品午夜福利在线观看 | 一个人看的www免费视频在线观看 | 无码av中文字幕免费放 | 99精品视频在线观看免费 | 99国产精品白浆在线观看免费 | 蜜臀av在线播放 久久综合激激的五月天 | 亚洲自偷自偷在线制服 | 老子影院午夜伦不卡 | 国产精品久久久av久久久 | 鲁大师影院在线观看 | 免费乱码人妻系列无码专区 | 国产卡一卡二卡三 | 青青青手机频在线观看 | 精品国产青草久久久久福利 | 精品无码国产一区二区三区av | 老熟女重囗味hdxx69 | 天堂а√在线中文在线 | 亚洲成av人片在线观看无码不卡 | 精品人妻中文字幕有码在线 | 亚拍精品一区二区三区探花 | 国产办公室秘书无码精品99 | 欧美亚洲日韩国产人成在线播放 | 久久精品一区二区三区四区 | 国产精品久久久久影院嫩草 | 性开放的女人aaa片 | av人摸人人人澡人人超碰下载 | 国产高清av在线播放 | 国产又爽又猛又粗的视频a片 | 亚洲国产欧美在线成人 | 波多野结衣高清一区二区三区 | 亚洲一区二区三区偷拍女厕 | 四虎影视成人永久免费观看视频 | 无码国内精品人妻少妇 | 日韩av激情在线观看 | 女高中生第一次破苞av | 国产精品无码mv在线观看 | 永久免费观看美女裸体的网站 | 妺妺窝人体色www婷婷 | 亚洲日韩一区二区 | 欧美精品国产综合久久 | 国产无套内射久久久国产 | 日日噜噜噜噜夜夜爽亚洲精品 | 丁香啪啪综合成人亚洲 | 久久久久久av无码免费看大片 | 欧美激情一区二区三区成人 | 亚洲综合无码久久精品综合 | 国产又爽又黄又刺激的视频 | 学生妹亚洲一区二区 | 欧美自拍另类欧美综合图片区 | 免费人成网站视频在线观看 | 无码国产色欲xxxxx视频 | 亚洲精品午夜无码电影网 | 国产麻豆精品一区二区三区v视界 | 成人精品天堂一区二区三区 | 综合人妻久久一区二区精品 | 成熟人妻av无码专区 | 国产超级va在线观看视频 | 国产电影无码午夜在线播放 | 国产suv精品一区二区五 | 99久久99久久免费精品蜜桃 | 久久久久成人精品免费播放动漫 | 国精产品一品二品国精品69xx | 51国偷自产一区二区三区 | 欧美精品国产综合久久 | 三级4级全黄60分钟 | 国产又爽又猛又粗的视频a片 | 伊在人天堂亚洲香蕉精品区 | 午夜精品一区二区三区的区别 | 成人免费视频视频在线观看 免费 | 欧美日韩亚洲国产精品 | 丁香花在线影院观看在线播放 | 国精品人妻无码一区二区三区蜜柚 | 久久国产劲爆∧v内射 | 国产精品久久福利网站 | 欧美 亚洲 国产 另类 | 婷婷色婷婷开心五月四房播播 | 国产美女精品一区二区三区 | 麻豆果冻传媒2021精品传媒一区下载 | 亚洲小说春色综合另类 | 久精品国产欧美亚洲色aⅴ大片 | 麻豆人妻少妇精品无码专区 | 无码国产乱人伦偷精品视频 | 麻豆精品国产精华精华液好用吗 | 国产精品va在线播放 | 青青草原综合久久大伊人精品 | 一区二区三区乱码在线 | 欧洲 | 对白脏话肉麻粗话av | 亚洲日韩中文字幕在线播放 | 国产成人无码区免费内射一片色欲 | 国产精品视频免费播放 | 中文字幕亚洲情99在线 | 亚洲综合在线一区二区三区 | 漂亮人妻洗澡被公强 日日躁 | 欧洲熟妇色 欧美 | 无码人妻精品一区二区三区不卡 | 午夜无码人妻av大片色欲 | 国产激情精品一区二区三区 | 狂野欧美性猛交免费视频 | 在线观看国产一区二区三区 | 亚洲一区二区三区四区 | 亚洲熟妇色xxxxx亚洲 | 九九在线中文字幕无码 | 女人和拘做爰正片视频 | 国产成人人人97超碰超爽8 | 男女性色大片免费网站 | 色婷婷久久一区二区三区麻豆 | 国产精品爱久久久久久久 | 午夜成人1000部免费视频 | 亚洲成色www久久网站 | 欧美激情内射喷水高潮 | 欧美人妻一区二区三区 | 久久精品国产99精品亚洲 | 国产亚洲欧美日韩亚洲中文色 | 日本一卡2卡3卡4卡无卡免费网站 国产一区二区三区影院 | 妺妺窝人体色www婷婷 | 精品成在人线av无码免费看 | 亚洲男人av天堂午夜在 | 亚无码乱人伦一区二区 | 啦啦啦www在线观看免费视频 | 特级做a爰片毛片免费69 | 中文字幕乱码人妻二区三区 | 亚洲综合无码久久精品综合 | 国产成人精品一区二区在线小狼 | 久热国产vs视频在线观看 | 国内丰满熟女出轨videos | 熟妇女人妻丰满少妇中文字幕 | 精品久久久中文字幕人妻 | 久久久中文久久久无码 | 日本免费一区二区三区最新 | 日韩av无码一区二区三区不卡 | 国产午夜视频在线观看 | 最近免费中文字幕中文高清百度 | 一个人看的www免费视频在线观看 | 无码精品人妻一区二区三区av | 欧美日韩一区二区综合 | 欧美日韩人成综合在线播放 | 亚洲自偷自拍另类第1页 | 99re在线播放 | 97se亚洲精品一区 | 色婷婷久久一区二区三区麻豆 | 永久免费观看美女裸体的网站 | 人人妻人人澡人人爽欧美一区 | 国内丰满熟女出轨videos | 色 综合 欧美 亚洲 国产 | 国产 精品 自在自线 | 无码毛片视频一区二区本码 | 成人影院yy111111在线观看 | 欧美日韩色另类综合 | 久久久久se色偷偷亚洲精品av | 色诱久久久久综合网ywww | 人妻少妇精品无码专区二区 | 成人三级无码视频在线观看 | 久久人人97超碰a片精品 | 任你躁在线精品免费 | 无码av免费一区二区三区试看 | 在教室伦流澡到高潮hnp视频 | 亚洲成在人网站无码天堂 | 噜噜噜亚洲色成人网站 | 国产内射老熟女aaaa | 亚洲精品一区国产 | www国产亚洲精品久久久日本 | 久久久婷婷五月亚洲97号色 | 曰韩少妇内射免费播放 | 狠狠色欧美亚洲狠狠色www | 在线精品国产一区二区三区 | 综合人妻久久一区二区精品 | 99久久精品日本一区二区免费 | 国产成人无码一二三区视频 | 亚洲精品一区二区三区大桥未久 | 国产电影无码午夜在线播放 | 亚洲国产一区二区三区在线观看 | 亚洲无人区一区二区三区 | 国产疯狂伦交大片 | 久久久久99精品成人片 | 日韩精品乱码av一区二区 | 亚洲aⅴ无码成人网站国产app | 欧美变态另类xxxx | 亚洲无人区一区二区三区 | 无遮挡国产高潮视频免费观看 | 精品熟女少妇av免费观看 | 18精品久久久无码午夜福利 | 大色综合色综合网站 | 国产免费久久精品国产传媒 | 97精品国产97久久久久久免费 | 久青草影院在线观看国产 | 久久精品人妻少妇一区二区三区 | 免费男性肉肉影院 | 麻豆蜜桃av蜜臀av色欲av | 国产卡一卡二卡三 | 人妻天天爽夜夜爽一区二区 | 日日摸夜夜摸狠狠摸婷婷 | 日韩欧美成人免费观看 | 国产精品对白交换视频 | 亚洲一区二区三区偷拍女厕 | 蜜桃臀无码内射一区二区三区 | 又大又硬又黄的免费视频 | 国产成人无码专区 | 极品嫩模高潮叫床 | 国产深夜福利视频在线 | 麻豆md0077饥渴少妇 | 婷婷综合久久中文字幕蜜桃三电影 | 奇米影视888欧美在线观看 | 牲欲强的熟妇农村老妇女视频 | 97久久精品无码一区二区 | 国产精品永久免费视频 | 中文亚洲成a人片在线观看 | 精品国产精品久久一区免费式 | 精品欧美一区二区三区久久久 | 一本久久a久久精品亚洲 | 国产在线aaa片一区二区99 | 亚洲综合色区中文字幕 | 日韩av无码中文无码电影 | 欧美丰满熟妇xxxx | 久久亚洲中文字幕无码 | 国产激情艳情在线看视频 | 蜜桃av抽搐高潮一区二区 | 精品国产乱码久久久久乱码 | 无码人妻少妇伦在线电影 | 中文字幕日韩精品一区二区三区 | 熟女少妇在线视频播放 | 国产人成高清在线视频99最全资源 | 日韩精品无码一本二本三本色 | 一本久久伊人热热精品中文字幕 | 精品偷拍一区二区三区在线看 | 色综合久久中文娱乐网 | 人妻插b视频一区二区三区 | 亚洲综合色区中文字幕 | 精品成人av一区二区三区 | 图片小说视频一区二区 | 丰满诱人的人妻3 | 在线播放免费人成毛片乱码 | 无码精品国产va在线观看dvd | 成人试看120秒体验区 | 天堂亚洲免费视频 | 亚洲精品鲁一鲁一区二区三区 | √天堂资源地址中文在线 | v一区无码内射国产 | 狠狠亚洲超碰狼人久久 | 奇米综合四色77777久久 东京无码熟妇人妻av在线网址 | 久久人人97超碰a片精品 | www国产亚洲精品久久网站 | 极品尤物被啪到呻吟喷水 | 日本熟妇乱子伦xxxx | 男女性色大片免费网站 | 日产国产精品亚洲系列 | 成人精品视频一区二区三区尤物 | 给我免费的视频在线观看 | 亚洲人成影院在线无码按摩店 | 国产片av国语在线观看 | 精品久久8x国产免费观看 | 国产美女精品一区二区三区 | 亚洲欧洲日本综合aⅴ在线 | 亚洲欧美日韩国产精品一区二区 | 亚洲男人av天堂午夜在 | 国产精品欧美成人 | 午夜性刺激在线视频免费 | 鲁鲁鲁爽爽爽在线视频观看 | 精品国偷自产在线视频 | 精品国产一区二区三区av 性色 | 丰满少妇熟乱xxxxx视频 | 熟妇人妻激情偷爽文 | 国产性生交xxxxx无码 | 久久久成人毛片无码 | 久久国产精品二国产精品 | 中文字幕无码免费久久9一区9 | 一本久道久久综合婷婷五月 | 久久久久99精品成人片 | 久久久成人毛片无码 | 无码午夜成人1000部免费视频 | 国色天香社区在线视频 | 午夜福利不卡在线视频 | 国产成人精品无码播放 | 国产精品.xx视频.xxtv | 3d动漫精品啪啪一区二区中 | аⅴ资源天堂资源库在线 | 亚洲色欲久久久综合网东京热 | 中国大陆精品视频xxxx | 无套内射视频囯产 | 国产午夜视频在线观看 | 内射欧美老妇wbb | 99久久精品国产一区二区蜜芽 | 漂亮人妻洗澡被公强 日日躁 | 亚洲精品久久久久avwww潮水 | 伊人久久大香线焦av综合影院 | 亚洲 a v无 码免 费 成 人 a v | 国产一区二区三区日韩精品 | 2020久久香蕉国产线看观看 | 免费国产黄网站在线观看 | 无码成人精品区在线观看 | 日本爽爽爽爽爽爽在线观看免 | 亚洲欧美综合区丁香五月小说 | 亚洲国产综合无码一区 | 日韩成人一区二区三区在线观看 | 67194成是人免费无码 | 女人色极品影院 | 99视频精品全部免费免费观看 | 久久99精品久久久久久 | 日日天日日夜日日摸 | 国产精品久久久久久亚洲影视内衣 | 久久人人爽人人爽人人片ⅴ | 日韩无套无码精品 | 人妻人人添人妻人人爱 | 夜先锋av资源网站 | 在线观看国产一区二区三区 | 成人精品视频一区二区 | 亚洲国产欧美日韩精品一区二区三区 | 窝窝午夜理论片影院 | 天天摸天天透天天添 | 国产精品永久免费视频 | 色综合久久久久综合一本到桃花网 | 永久黄网站色视频免费直播 | 麻豆md0077饥渴少妇 | 国产人妻大战黑人第1集 | 久久久久人妻一区精品色欧美 | 少妇性荡欲午夜性开放视频剧场 | 国产舌乚八伦偷品w中 | 男女超爽视频免费播放 | 一本久道久久综合狠狠爱 | 国产人妻久久精品二区三区老狼 | 奇米综合四色77777久久 东京无码熟妇人妻av在线网址 | 精品无码av一区二区三区 | 久久久国产一区二区三区 | 人人妻人人澡人人爽欧美精品 | 亚洲成熟女人毛毛耸耸多 | 日本免费一区二区三区最新 | 国产欧美亚洲精品a | 夜精品a片一区二区三区无码白浆 | 久久久久成人精品免费播放动漫 | 久久人人97超碰a片精品 | 欧美怡红院免费全部视频 | 成熟妇人a片免费看网站 | 99久久久国产精品无码免费 | 亚洲国产精品久久久天堂 | 波多野结衣aⅴ在线 | 久久久精品456亚洲影院 | 精品无码成人片一区二区98 | 亚洲精品综合一区二区三区在线 | 亚洲精品一区二区三区婷婷月 | 亚洲乱码中文字幕在线 | 欧美阿v高清资源不卡在线播放 | av无码不卡在线观看免费 | 亚洲の无码国产の无码影院 | 中文字幕无码乱人伦 | 亚洲 激情 小说 另类 欧美 | 国产精品18久久久久久麻辣 | 日本护士毛茸茸高潮 | 国产人妻精品一区二区三区 | 中文字幕av伊人av无码av | 国产无遮挡又黄又爽免费视频 | 男女超爽视频免费播放 | 中文无码成人免费视频在线观看 | 亚洲一区av无码专区在线观看 | 精品一区二区三区无码免费视频 | 四虎影视成人永久免费观看视频 | 高潮毛片无遮挡高清免费 | 99久久精品午夜一区二区 | 精品无码av一区二区三区 | 精品无码国产自产拍在线观看蜜 | 亚洲综合无码一区二区三区 | 丰满肥臀大屁股熟妇激情视频 | 国产成人亚洲综合无码 | 兔费看少妇性l交大片免费 | 中文字幕无码av激情不卡 | 久久99精品国产麻豆 | 日韩成人一区二区三区在线观看 | 无码人妻少妇伦在线电影 | 久久精品99久久香蕉国产色戒 | 狠狠色噜噜狠狠狠7777奇米 | 国产真实乱对白精彩久久 | 永久免费精品精品永久-夜色 | 亚洲自偷自拍另类第1页 | 精品人人妻人人澡人人爽人人 | 性生交大片免费看l | 成人精品天堂一区二区三区 | 国产无遮挡又黄又爽又色 | 午夜福利试看120秒体验区 | 亚洲日本一区二区三区在线 | 国产精品99久久精品爆乳 | 国产在线无码精品电影网 | 亚洲色成人中文字幕网站 | 婷婷丁香六月激情综合啪 | 国产精品久久久一区二区三区 | 亚洲自偷精品视频自拍 | 成人一区二区免费视频 | 国产精品视频免费播放 | 97夜夜澡人人双人人人喊 | 久久视频在线观看精品 | 青青久在线视频免费观看 | 国产免费久久久久久无码 | 少妇太爽了在线观看 | 国产午夜无码视频在线观看 | 亚洲精品国偷拍自产在线麻豆 | 久久精品人人做人人综合 | 人人澡人人透人人爽 | 中文字幕日产无线码一区 | 国产精品久久久一区二区三区 | 成人性做爰aaa片免费看不忠 | 成人精品天堂一区二区三区 | 久久亚洲中文字幕精品一区 | 99国产精品白浆在线观看免费 | 国产av一区二区精品久久凹凸 | 高潮毛片无遮挡高清免费视频 | 色妞www精品免费视频 | 国产精品福利视频导航 | 国产午夜精品一区二区三区嫩草 | 亚洲呦女专区 | 97久久国产亚洲精品超碰热 | 夜精品a片一区二区三区无码白浆 | 国产综合在线观看 | 狂野欧美性猛交免费视频 | 国产成人综合在线女婷五月99播放 | 国产乱人伦偷精品视频 | 成人无码精品1区2区3区免费看 | 无码国模国产在线观看 | 久久久久成人片免费观看蜜芽 | 国产精品高潮呻吟av久久4虎 | 无码一区二区三区在线观看 | 樱花草在线播放免费中文 | 55夜色66夜色国产精品视频 | 少妇激情av一区二区 | 成人欧美一区二区三区黑人 | 精品一区二区三区无码免费视频 | 亚洲狠狠色丁香婷婷综合 | 老司机亚洲精品影院 | 亚洲国产成人av在线观看 | 日本精品久久久久中文字幕 | 在线天堂新版最新版在线8 | 婷婷色婷婷开心五月四房播播 | 国产无遮挡又黄又爽免费视频 | 十八禁真人啪啪免费网站 | 日韩精品久久久肉伦网站 | 久久aⅴ免费观看 | 亚洲精品一区三区三区在线观看 | 久久视频在线观看精品 | 亚洲色欲色欲欲www在线 | 沈阳熟女露脸对白视频 | 精品无人区无码乱码毛片国产 | 狠狠色丁香久久婷婷综合五月 | 又大又黄又粗又爽的免费视频 | 粗大的内捧猛烈进出视频 | 亚洲国产精品毛片av不卡在线 | 久久伊人色av天堂九九小黄鸭 | 少妇性l交大片 | 亚洲大尺度无码无码专区 | 精品一区二区不卡无码av | 国产乱人伦偷精品视频 | 亚洲精品国偷拍自产在线观看蜜桃 | 久精品国产欧美亚洲色aⅴ大片 | 久久久久亚洲精品中文字幕 | 欧美黑人乱大交 | 国产亚洲美女精品久久久2020 | 精品无码一区二区三区的天堂 | 55夜色66夜色国产精品视频 | 亚洲人成网站在线播放942 | 精品人人妻人人澡人人爽人人 | 久久 国产 尿 小便 嘘嘘 | 欧洲精品码一区二区三区免费看 | 亚洲欧美国产精品专区久久 | 国产精品无套呻吟在线 | 久久午夜夜伦鲁鲁片无码免费 | 最近免费中文字幕中文高清百度 | 国产精品久久久午夜夜伦鲁鲁 | 亚洲中文字幕无码中字 | 色综合天天综合狠狠爱 | 久久久婷婷五月亚洲97号色 | 大胆欧美熟妇xx | 国产精品无码一区二区桃花视频 | 国产又爽又黄又刺激的视频 | 国产偷自视频区视频 | 日本乱人伦片中文三区 | av无码久久久久不卡免费网站 | 欧美丰满熟妇xxxx | 无码精品人妻一区二区三区av | 亚洲国产成人av在线观看 | 国产精品亚洲综合色区韩国 | 久精品国产欧美亚洲色aⅴ大片 | 国产真实乱对白精彩久久 | 成人精品一区二区三区中文字幕 | 成人欧美一区二区三区黑人 | 久久99国产综合精品 | 妺妺窝人体色www在线小说 | 欧美一区二区三区视频在线观看 | 精品久久久久久亚洲精品 | 婷婷综合久久中文字幕蜜桃三电影 | 国产人妖乱国产精品人妖 | 麻豆人妻少妇精品无码专区 | 自拍偷自拍亚洲精品被多人伦好爽 | 狠狠色噜噜狠狠狠狠7777米奇 | 老熟女乱子伦 | 美女扒开屁股让男人桶 | 女人和拘做爰正片视频 | 欧美老熟妇乱xxxxx | yw尤物av无码国产在线观看 | 亚洲色偷偷男人的天堂 | 99久久婷婷国产综合精品青草免费 | 国产又粗又硬又大爽黄老大爷视 | 国产精品无码一区二区三区不卡 | 欧美日韩人成综合在线播放 | 国产精品无套呻吟在线 | 波多野结衣aⅴ在线 | 99久久精品日本一区二区免费 | 鲁鲁鲁爽爽爽在线视频观看 | 国产猛烈高潮尖叫视频免费 | 在线观看欧美一区二区三区 | 一个人看的www免费视频在线观看 | 亚洲色欲色欲天天天www | 亚洲精品国产精品乱码视色 | 性欧美牲交xxxxx视频 | 婷婷丁香六月激情综合啪 | 中文精品久久久久人妻不卡 | 超碰97人人做人人爱少妇 | 精品一区二区三区无码免费视频 | 动漫av一区二区在线观看 | 亚洲中文字幕无码中文字在线 | 久久婷婷五月综合色国产香蕉 | 未满小14洗澡无码视频网站 | 亚洲人成网站色7799 | 国产精品毛多多水多 | 少妇性荡欲午夜性开放视频剧场 | 国产精品久久久久久久9999 | 日韩亚洲欧美中文高清在线 | 成在人线av无码免观看麻豆 | 精品国产一区二区三区av 性色 | 性欧美熟妇videofreesex | 亚洲一区二区三区在线观看网站 | 亚洲va中文字幕无码久久不卡 | 丝袜 中出 制服 人妻 美腿 | 精品欧美一区二区三区久久久 | 伊人色综合久久天天小片 | aⅴ亚洲 日韩 色 图网站 播放 | 亚拍精品一区二区三区探花 | 午夜理论片yy44880影院 | 婷婷丁香五月天综合东京热 | 国产亚洲精品久久久久久大师 | 国产精品高潮呻吟av久久 | 国内精品人妻无码久久久影院 | 97久久超碰中文字幕 | 久久午夜夜伦鲁鲁片无码免费 | 奇米综合四色77777久久 东京无码熟妇人妻av在线网址 | 老司机亚洲精品影院无码 | 老司机亚洲精品影院 | 鲁大师影院在线观看 | 亚洲区小说区激情区图片区 | 午夜男女很黄的视频 | 国产精品久久久久无码av色戒 | 狂野欧美激情性xxxx | 国产成人久久精品流白浆 | 搡女人真爽免费视频大全 | 无码人妻精品一区二区三区下载 | 理论片87福利理论电影 | 人人妻人人澡人人爽欧美一区 | 亚洲а∨天堂久久精品2021 | 福利一区二区三区视频在线观看 | 一个人看的www免费视频在线观看 | 亚洲无人区午夜福利码高清完整版 | 无码av免费一区二区三区试看 | 无码国内精品人妻少妇 | 亚洲人交乣女bbw | 久久视频在线观看精品 | 国产精品国产自线拍免费软件 | 亚洲 a v无 码免 费 成 人 a v | 精品亚洲韩国一区二区三区 | 国产偷国产偷精品高清尤物 | 欧美肥老太牲交大战 | 一个人看的www免费视频在线观看 | 欧美老人巨大xxxx做受 | 国产suv精品一区二区五 | 日韩精品久久久肉伦网站 | 人妻aⅴ无码一区二区三区 | 精品无码一区二区三区的天堂 | 国精产品一品二品国精品69xx | 亚洲精品中文字幕久久久久 | 日韩欧美群交p片內射中文 | 国产精品亚洲lv粉色 | 任你躁国产自任一区二区三区 | 国产av一区二区三区最新精品 | 久久久国产精品无码免费专区 | 狠狠cao日日穞夜夜穞av | 欧美刺激性大交 | 国产午夜无码精品免费看 | 秋霞成人午夜鲁丝一区二区三区 | 亚洲综合无码一区二区三区 | 国产人妻久久精品二区三区老狼 | 国产成人人人97超碰超爽8 | 亚洲精品综合五月久久小说 | 亚洲中文字幕无码一久久区 | 国产麻豆精品一区二区三区v视界 | 国产偷国产偷精品高清尤物 | 97久久国产亚洲精品超碰热 | 少妇无套内谢久久久久 | 精品国精品国产自在久国产87 | 女人被爽到呻吟gif动态图视看 | 成人欧美一区二区三区黑人 | 美女扒开屁股让男人桶 | 免费观看又污又黄的网站 | 久久综合久久自在自线精品自 | 亚洲伊人久久精品影院 | 欧美 日韩 人妻 高清 中文 | 无码人妻丰满熟妇区毛片18 | 欧美 日韩 亚洲 在线 | 国产成人无码a区在线观看视频app | 久久www免费人成人片 | 成年女人永久免费看片 | 中文字幕+乱码+中文字幕一区 | 中文字幕无码乱人伦 | 亚洲日本va午夜在线电影 | 欧美人与禽猛交狂配 | 久久99国产综合精品 | 久久天天躁夜夜躁狠狠 | 99久久人妻精品免费二区 | 精品国产一区二区三区四区在线看 | 少妇性l交大片 | 亚洲成av人片天堂网无码】 | 在线成人www免费观看视频 | 国产成人无码av一区二区 | 女人被男人躁得好爽免费视频 | 精品欧洲av无码一区二区三区 | 亚洲理论电影在线观看 | 老司机亚洲精品影院 | 国产 浪潮av性色四虎 | 男女下面进入的视频免费午夜 | 久久亚洲中文字幕无码 | 丰满人妻翻云覆雨呻吟视频 | 丰满少妇高潮惨叫视频 | 日产精品高潮呻吟av久久 | 欧美日韩一区二区三区自拍 | 亚洲精品国偷拍自产在线观看蜜桃 | 大胆欧美熟妇xx | 性色av无码免费一区二区三区 | 久久综合九色综合欧美狠狠 | 波多野结衣av在线观看 | 国产成人av免费观看 | 精品国产国产综合精品 | 久久综合香蕉国产蜜臀av | 国产电影无码午夜在线播放 | 亚洲色在线无码国产精品不卡 | 亚洲va中文字幕无码久久不卡 | 色五月五月丁香亚洲综合网 | 99在线 | 亚洲 | 国产乱人无码伦av在线a | 国产黄在线观看免费观看不卡 | 丰满少妇女裸体bbw | 成人综合网亚洲伊人 | 综合人妻久久一区二区精品 | 四虎国产精品一区二区 | 伊人久久大香线焦av综合影院 | 欧美人与禽猛交狂配 | 欧美放荡的少妇 | 女高中生第一次破苞av | 国产精品手机免费 | 少妇被粗大的猛进出69影院 | 国产午夜福利100集发布 | 亚洲最大成人网站 | 国产精品内射视频免费 | 亚洲国产精品成人久久蜜臀 | 日本精品少妇一区二区三区 | 俺去俺来也在线www色官网 | 日本丰满护士爆乳xxxx | 伊人久久大香线焦av综合影院 | 在线观看欧美一区二区三区 | 国产又爽又猛又粗的视频a片 | 99国产欧美久久久精品 | 天天摸天天透天天添 | 午夜精品一区二区三区在线观看 | 久久午夜夜伦鲁鲁片无码免费 | 亚洲一区二区三区播放 | 丰满肥臀大屁股熟妇激情视频 | 乱人伦人妻中文字幕无码 | 无码国产激情在线观看 | 亚洲精品中文字幕乱码 | 亚洲а∨天堂久久精品2021 | 无遮挡啪啪摇乳动态图 | √天堂中文官网8在线 | 久久精品99久久香蕉国产色戒 | 性色欲情网站iwww九文堂 | 国产人妻精品一区二区三区不卡 | 久久午夜无码鲁丝片午夜精品 | 奇米影视888欧美在线观看 | 麻豆国产丝袜白领秘书在线观看 | 亚洲 高清 成人 动漫 | 亚洲码国产精品高潮在线 | 亚洲熟妇色xxxxx欧美老妇 | 久久人人爽人人爽人人片av高清 | 日本丰满熟妇videos | 国产av久久久久精东av | 亚洲热妇无码av在线播放 | 亚洲熟妇自偷自拍另类 | 欧美日韩在线亚洲综合国产人 | a国产一区二区免费入口 | 国产在线一区二区三区四区五区 | 国产绳艺sm调教室论坛 | 少妇一晚三次一区二区三区 | 欧美激情一区二区三区成人 | 亚洲中文字幕在线观看 | av在线亚洲欧洲日产一区二区 | 伊人久久婷婷五月综合97色 | 欧美第一黄网免费网站 | 中文字幕人妻无码一夲道 | 国产莉萝无码av在线播放 | av无码久久久久不卡免费网站 | 欧美日韩人成综合在线播放 | 国产乱人伦偷精品视频 | 少妇太爽了在线观看 | 一二三四在线观看免费视频 | 国产精品香蕉在线观看 | 久久久久久亚洲精品a片成人 | 国产又爽又黄又刺激的视频 | yw尤物av无码国产在线观看 | 狠狠噜狠狠狠狠丁香五月 | 国产无遮挡又黄又爽又色 | 色窝窝无码一区二区三区色欲 | 国产麻豆精品精东影业av网站 | 国产精品va在线观看无码 | 国产精品无码一区二区三区不卡 | yw尤物av无码国产在线观看 | а√天堂www在线天堂小说 | 老太婆性杂交欧美肥老太 | 久久综合给合久久狠狠狠97色 | 久久精品国产一区二区三区肥胖 | 99久久精品国产一区二区蜜芽 | 国产极品视觉盛宴 | 国内综合精品午夜久久资源 | 在线欧美精品一区二区三区 | 3d动漫精品啪啪一区二区中 | 骚片av蜜桃精品一区 | 欧美丰满熟妇xxxx性ppx人交 | 国产成人精品久久亚洲高清不卡 | 免费播放一区二区三区 | 国产疯狂伦交大片 | 精品欧美一区二区三区久久久 | 亚洲欧美精品aaaaaa片 | 国产精品对白交换视频 | 一本大道伊人av久久综合 | 人妻有码中文字幕在线 | 天干天干啦夜天干天2017 | 小鲜肉自慰网站xnxx | 成人精品视频一区二区三区尤物 | 丰腴饱满的极品熟妇 | 欧美性生交xxxxx久久久 | 狠狠色丁香久久婷婷综合五月 | 国产综合久久久久鬼色 | 四虎国产精品一区二区 | 亚洲第一网站男人都懂 | 呦交小u女精品视频 | 色婷婷香蕉在线一区二区 | 欧美 日韩 人妻 高清 中文 | 国产人妻久久精品二区三区老狼 | 欧美三级a做爰在线观看 | 久久无码专区国产精品s | 久久99精品久久久久久动态图 | 国产亚洲精品久久久闺蜜 | 男人和女人高潮免费网站 | 国产亚洲人成a在线v网站 | 亚洲人交乣女bbw | 精品一区二区三区无码免费视频 | 亚洲天堂2017无码中文 | 一本久道高清无码视频 | 亚洲精品无码国产 | 国产高潮视频在线观看 | 国产精品va在线观看无码 | 97久久精品无码一区二区 | 久久99精品久久久久久动态图 | 欧美三级不卡在线观看 | 18无码粉嫩小泬无套在线观看 | 99riav国产精品视频 | 午夜精品久久久内射近拍高清 | 国产精品va在线观看无码 | 亚洲爆乳精品无码一区二区三区 | 无码国产乱人伦偷精品视频 | 娇妻被黑人粗大高潮白浆 | 亚洲人成网站免费播放 | 色欲人妻aaaaaaa无码 | 在线 国产 欧美 亚洲 天堂 | 无码成人精品区在线观看 | 狠狠色欧美亚洲狠狠色www | 国产精品久久久一区二区三区 | 丰满少妇人妻久久久久久 | 久久人人爽人人爽人人片av高清 | 精品偷自拍另类在线观看 | 中文字幕日韩精品一区二区三区 | 国产成人精品必看 | 国产凸凹视频一区二区 | 51国偷自产一区二区三区 | 日本精品人妻无码77777 天堂一区人妻无码 | 装睡被陌生人摸出水好爽 | 色诱久久久久综合网ywww | 荫蒂添的好舒服视频囗交 | 美女扒开屁股让男人桶 | 野狼第一精品社区 | 亚洲精品一区二区三区婷婷月 | 一个人免费观看的www视频 | 国产av人人夜夜澡人人爽麻豆 | 国产精品人妻一区二区三区四 | 久久久久久国产精品无码下载 | 97资源共享在线视频 | 狠狠噜狠狠狠狠丁香五月 | 少妇人妻av毛片在线看 | 欧美自拍另类欧美综合图片区 | 色噜噜亚洲男人的天堂 | 国产亚洲精品久久久久久 | 好爽又高潮了毛片免费下载 | 强伦人妻一区二区三区视频18 | 对白脏话肉麻粗话av | 丰满少妇弄高潮了www | 欧美老人巨大xxxx做受 | 人人妻人人澡人人爽欧美一区 | 国产精品.xx视频.xxtv | 精品熟女少妇av免费观看 | 久激情内射婷内射蜜桃人妖 | 国内老熟妇对白xxxxhd | 又黄又爽又色的视频 | 丰满少妇熟乱xxxxx视频 | 老熟女重囗味hdxx69 | 亚洲爆乳大丰满无码专区 | 欧美精品免费观看二区 | 日本熟妇大屁股人妻 | 熟妇人妻中文av无码 | 日韩无套无码精品 | 一本一道久久综合久久 | 精品无码成人片一区二区98 | 国产精品无码一区二区桃花视频 | 国内综合精品午夜久久资源 | 理论片87福利理论电影 | 宝宝好涨水快流出来免费视频 | 亚洲人成影院在线无码按摩店 | 中文字幕人成乱码熟女app | 亚洲精品国产第一综合99久久 | a国产一区二区免费入口 | 无码人妻av免费一区二区三区 | 少妇高潮喷潮久久久影院 | 人妻少妇精品无码专区二区 | 东京热无码av男人的天堂 | 亚洲色欲色欲欲www在线 | 国产精品人人妻人人爽 | 日韩av无码一区二区三区 | 爆乳一区二区三区无码 | 人妻无码αv中文字幕久久琪琪布 | 国产精品久久久 | 午夜福利不卡在线视频 | 欧美野外疯狂做受xxxx高潮 | 天海翼激烈高潮到腰振不止 | 亚洲 另类 在线 欧美 制服 | 国产精品欧美成人 | 国产成人无码区免费内射一片色欲 | 久热国产vs视频在线观看 | 中文字幕无码热在线视频 | 亚洲va欧美va天堂v国产综合 | 国产高清不卡无码视频 | 精品无码国产一区二区三区av | 国产一区二区三区日韩精品 | 成人一区二区免费视频 | 午夜福利试看120秒体验区 | 无码毛片视频一区二区本码 | 欧美精品无码一区二区三区 | 日本大香伊一区二区三区 | 国产av一区二区精品久久凹凸 | 精品久久综合1区2区3区激情 | 亚洲精品成a人在线观看 | 国产成人精品久久亚洲高清不卡 | 亚洲精品国产第一综合99久久 | 国产人妻精品午夜福利免费 | 精品欧美一区二区三区久久久 | 少妇高潮喷潮久久久影院 | 色婷婷av一区二区三区之红樱桃 | 在线а√天堂中文官网 | 久久久久亚洲精品男人的天堂 | 中文字幕av无码一区二区三区电影 | 久久综合色之久久综合 | 正在播放东北夫妻内射 | 亚洲娇小与黑人巨大交 | 国产精品爱久久久久久久 | 免费网站看v片在线18禁无码 | av无码电影一区二区三区 | 嫩b人妻精品一区二区三区 | 2019午夜福利不卡片在线 | 又粗又大又硬又长又爽 | 国产又粗又硬又大爽黄老大爷视 | 亚洲gv猛男gv无码男同 | 熟女俱乐部五十路六十路av | 最近的中文字幕在线看视频 | 精品无人区无码乱码毛片国产 | 99久久久国产精品无码免费 | 国产精品无码成人午夜电影 | 国产农村妇女aaaaa视频 撕开奶罩揉吮奶头视频 | 真人与拘做受免费视频 | 精品国产国产综合精品 | 波多野结衣高清一区二区三区 | 人人妻人人澡人人爽欧美一区 | 国产亚洲人成在线播放 | 高中生自慰www网站 | 亚洲の无码国产の无码影院 | 久久午夜无码鲁丝片 | 成人性做爰aaa片免费看不忠 | 欧美兽交xxxx×视频 | 人人妻人人澡人人爽欧美精品 | 亚洲一区二区三区国产精华液 | 日韩欧美群交p片內射中文 | 天天av天天av天天透 | 丰满少妇女裸体bbw | 久久亚洲精品成人无码 | 成人欧美一区二区三区黑人免费 | 亚洲精品久久久久久久久久久 | 免费无码午夜福利片69 | 天天躁日日躁狠狠躁免费麻豆 | 亚洲春色在线视频 | 女人被男人躁得好爽免费视频 | 中国大陆精品视频xxxx | 最近中文2019字幕第二页 | 熟妇激情内射com | www成人国产高清内射 | 国产真人无遮挡作爱免费视频 | 色欲久久久天天天综合网精品 | 99精品国产综合久久久久五月天 | 少妇被黑人到高潮喷出白浆 | 奇米影视888欧美在线观看 | 国产午夜无码精品免费看 | 久久99热只有频精品8 | 日本一卡2卡3卡四卡精品网站 | 成人三级无码视频在线观看 | 亚洲爆乳精品无码一区二区三区 | 中文字幕 人妻熟女 | 国产婷婷色一区二区三区在线 | 国产又爽又猛又粗的视频a片 | 麻豆人妻少妇精品无码专区 | 国产成人综合色在线观看网站 | 亚洲精品国偷拍自产在线观看蜜桃 | 少妇人妻大乳在线视频 | 国产精品无码成人午夜电影 | 亚洲熟妇色xxxxx欧美老妇y | 76少妇精品导航 | 精品欧美一区二区三区久久久 | 男人扒开女人内裤强吻桶进去 | 日日鲁鲁鲁夜夜爽爽狠狠 | 亚洲精品无码人妻无码 | 欧美日本精品一区二区三区 | 国产乱子伦视频在线播放 | 亚洲人成影院在线无码按摩店 | 日日碰狠狠丁香久燥 | 天下第一社区视频www日本 | 欧美zoozzooz性欧美 | 日本精品人妻无码免费大全 | 丰满少妇熟乱xxxxx视频 | 精品无人国产偷自产在线 | 国产小呦泬泬99精品 | 国产亚洲美女精品久久久2020 | 精品一区二区三区波多野结衣 | 美女扒开屁股让男人桶 | 成人片黄网站色大片免费观看 | 九九久久精品国产免费看小说 | 亚洲综合精品香蕉久久网 | 亚洲精品一区三区三区在线观看 | 一本色道久久综合狠狠躁 | 性欧美大战久久久久久久 | 久久久成人毛片无码 | 亚洲色欲久久久综合网东京热 | 少妇人妻偷人精品无码视频 | 黑人大群体交免费视频 | 国产农村乱对白刺激视频 | 妺妺窝人体色www在线小说 | 欧美国产日韩久久mv | 亚洲综合精品香蕉久久网 | 97久久精品无码一区二区 | 中文字幕av无码一区二区三区电影 | 少妇无套内谢久久久久 | 久久精品丝袜高跟鞋 | 亚洲一区二区三区偷拍女厕 | 成人性做爰aaa片免费看不忠 | 欧美zoozzooz性欧美 | 四虎影视成人永久免费观看视频 | 亚洲熟妇色xxxxx欧美老妇y | 国模大胆一区二区三区 | 伊人久久婷婷五月综合97色 | 精品国精品国产自在久国产87 | 真人与拘做受免费视频一 | aa片在线观看视频在线播放 | 亚洲国产av美女网站 | 精品熟女少妇av免费观看 | 无码免费一区二区三区 | 久久国产精品偷任你爽任你 | 色综合久久88色综合天天 | 久久综合给久久狠狠97色 | 大乳丰满人妻中文字幕日本 | 成 人影片 免费观看 | 久青草影院在线观看国产 | 亚洲日韩av片在线观看 | 国产亲子乱弄免费视频 | 无码av免费一区二区三区试看 | 亚洲七七久久桃花影院 | 美女张开腿让人桶 | 无码人妻黑人中文字幕 | 在线视频网站www色 | 久久99热只有频精品8 | 99精品久久毛片a片 | 久久久久久久女国产乱让韩 | 又大又紧又粉嫩18p少妇 | 精品一二三区久久aaa片 | 久久亚洲精品中文字幕无男同 | 丝袜 中出 制服 人妻 美腿 | 国产一区二区三区影院 | 成年美女黄网站色大免费全看 | 亚洲国产精品毛片av不卡在线 | 大色综合色综合网站 | 欧美变态另类xxxx | 欧洲美熟女乱又伦 | 色婷婷av一区二区三区之红樱桃 | 久久成人a毛片免费观看网站 | 特黄特色大片免费播放器图片 | 精品欧洲av无码一区二区三区 | 国产精品人妻一区二区三区四 | 亚洲欧洲无卡二区视頻 | 大胆欧美熟妇xx | 欧美日本日韩 | 2020久久超碰国产精品最新 | 久久精品人人做人人综合 | 网友自拍区视频精品 | 中文精品无码中文字幕无码专区 | 国产特级毛片aaaaaa高潮流水 | 久久久久久久女国产乱让韩 | 精品国偷自产在线视频 | 少妇一晚三次一区二区三区 | 亚洲中文字幕va福利 | 国产特级毛片aaaaaaa高清 | 少妇高潮一区二区三区99 | 丁香花在线影院观看在线播放 | 中文字幕无码人妻少妇免费 | 秋霞成人午夜鲁丝一区二区三区 | 久久综合久久自在自线精品自 | 一本色道久久综合亚洲精品不卡 | 国产成人精品无码播放 | 色狠狠av一区二区三区 | 丰满护士巨好爽好大乳 | 亚洲国产一区二区三区在线观看 | 精品熟女少妇av免费观看 | 久久人人爽人人人人片 | 亚洲一区二区三区四区 | 男女作爱免费网站 | 国内精品九九久久久精品 | 动漫av一区二区在线观看 | v一区无码内射国产 | 亚洲无人区一区二区三区 | av无码电影一区二区三区 | 伊人久久大香线蕉av一区二区 | 中文字幕人妻无码一区二区三区 | 精品水蜜桃久久久久久久 | 内射白嫩少妇超碰 | 人妻少妇精品无码专区动漫 | 午夜无码人妻av大片色欲 | 久久国产精品萌白酱免费 | 人人妻人人澡人人爽人人精品浪潮 | 亚洲综合无码一区二区三区 | 国产日产欧产精品精品app | 色一情一乱一伦 | 国产明星裸体无码xxxx视频 | 蜜桃av蜜臀av色欲av麻 999久久久国产精品消防器材 | 国产亚洲视频中文字幕97精品 | 精品亚洲韩国一区二区三区 | 久久久精品456亚洲影院 | 久久99精品国产麻豆 | 国精产品一品二品国精品69xx | 国产性生大片免费观看性 | 色婷婷久久一区二区三区麻豆 | 狠狠色色综合网站 | 少妇太爽了在线观看 | 亚洲欧美国产精品专区久久 | 成人无码精品一区二区三区 | 亚洲精品综合一区二区三区在线 | 小泽玛莉亚一区二区视频在线 | 内射爽无广熟女亚洲 | 乱人伦中文视频在线观看 | 人人爽人人澡人人人妻 | 亚洲国产欧美日韩精品一区二区三区 | 黑森林福利视频导航 | 香港三级日本三级妇三级 | 亚洲va欧美va天堂v国产综合 | 国产亚洲欧美日韩亚洲中文色 | 午夜成人1000部免费视频 | 在线亚洲高清揄拍自拍一品区 | 青青草原综合久久大伊人精品 | 樱花草在线播放免费中文 | 欧美精品无码一区二区三区 | 狠狠cao日日穞夜夜穞av | 欧美第一黄网免费网站 | 日本一本二本三区免费 | 少妇一晚三次一区二区三区 | 亚洲第一网站男人都懂 | 亚洲狠狠色丁香婷婷综合 | 欧美人与动性行为视频 | 亚洲 激情 小说 另类 欧美 | 亚洲自偷自偷在线制服 | 激情综合激情五月俺也去 | 亚洲娇小与黑人巨大交 | ass日本丰满熟妇pics | 久久国产精品偷任你爽任你 | 又紧又大又爽精品一区二区 | 高潮毛片无遮挡高清免费视频 | 爽爽影院免费观看 | 成人女人看片免费视频放人 | 日产精品高潮呻吟av久久 | 久久久久亚洲精品中文字幕 | aⅴ亚洲 日韩 色 图网站 播放 | 伊人久久大香线蕉av一区二区 | 中文字幕乱码人妻无码久久 | 国产精品人人爽人人做我的可爱 | 无码成人精品区在线观看 | 久久综合香蕉国产蜜臀av | 久久久久免费精品国产 | 99久久久国产精品无码免费 | 欧美肥老太牲交大战 | 18黄暴禁片在线观看 | 国产精品久久久久影院嫩草 | 欧美精品在线观看 | 成人无码影片精品久久久 | 一区二区传媒有限公司 | 日日摸夜夜摸狠狠摸婷婷 | 国语精品一区二区三区 | 粉嫩少妇内射浓精videos | 成在人线av无码免观看麻豆 | 亚洲成av人综合在线观看 | 亚洲人成影院在线无码按摩店 | 日本大乳高潮视频在线观看 | 国产九九九九九九九a片 | 无套内谢的新婚少妇国语播放 | 精品久久久久久亚洲精品 | 亚洲熟妇色xxxxx欧美老妇y | 国产xxx69麻豆国语对白 | 又粗又大又硬毛片免费看 | 国产精品久久福利网站 | 永久免费精品精品永久-夜色 | 97资源共享在线视频 | 日韩欧美成人免费观看 | 荫蒂添的好舒服视频囗交 | 久9re热视频这里只有精品 | 国产精品无码一区二区桃花视频 | 妺妺窝人体色www在线小说 | 一个人看的视频www在线 | 国产精品无套呻吟在线 | 国产成人av免费观看 | 最近中文2019字幕第二页 | 东京一本一道一二三区 | 18无码粉嫩小泬无套在线观看 | 又大又硬又黄的免费视频 | 免费看少妇作爱视频 | av人摸人人人澡人人超碰下载 | 影音先锋中文字幕无码 | 露脸叫床粗话东北少妇 | 宝宝好涨水快流出来免费视频 | 亚洲gv猛男gv无码男同 | 欧美老熟妇乱xxxxx | 亚洲精品一区三区三区在线观看 | 精品人人妻人人澡人人爽人人 | 男女猛烈xx00免费视频试看 | 清纯唯美经典一区二区 | 性生交大片免费看l | 捆绑白丝粉色jk震动捧喷白浆 | 熟女少妇人妻中文字幕 | 色一情一乱一伦一区二区三欧美 | 大肉大捧一进一出好爽视频 | 未满成年国产在线观看 | 人妻互换免费中文字幕 | 超碰97人人射妻 | 婷婷色婷婷开心五月四房播播 | 麻豆国产人妻欲求不满 | 国产 精品 自在自线 | 小鲜肉自慰网站xnxx | 亚洲中文字幕久久无码 | 成人片黄网站色大片免费观看 | 国产精品美女久久久 | 久久国产36精品色熟妇 | 亚洲乱亚洲乱妇50p | 色综合久久中文娱乐网 | 久久久久成人精品免费播放动漫 | 国产艳妇av在线观看果冻传媒 | 中文字幕av伊人av无码av | 日韩 欧美 动漫 国产 制服 | 牲交欧美兽交欧美 | 在线播放亚洲第一字幕 | 久久久久久av无码免费看大片 | 最新国产乱人伦偷精品免费网站 | 国产无遮挡又黄又爽又色 | 久久综合香蕉国产蜜臀av | 国产精品美女久久久 | 国产精品无码一区二区三区不卡 | 久久综合给合久久狠狠狠97色 | 青青草原综合久久大伊人精品 | 俺去俺来也在线www色官网 | 台湾无码一区二区 | 国产成人一区二区三区别 | 亚洲精品久久久久久一区二区 | 美女毛片一区二区三区四区 | 人人妻人人澡人人爽欧美一区九九 | 国产疯狂伦交大片 | 青青草原综合久久大伊人精品 | 精品国产aⅴ无码一区二区 | 国精品人妻无码一区二区三区蜜柚 | 99久久久无码国产精品免费 | 欧美日韩视频无码一区二区三 | 精品国精品国产自在久国产87 | 十八禁真人啪啪免费网站 | 久久人人爽人人人人片 | 国产猛烈高潮尖叫视频免费 | 国产精品福利视频导航 | 亚洲欧美国产精品专区久久 | 精品久久久中文字幕人妻 | 国产做国产爱免费视频 | 亚洲综合精品香蕉久久网 | 亚洲第一无码av无码专区 | 99视频精品全部免费免费观看 | 欧美成人午夜精品久久久 | 久久97精品久久久久久久不卡 | 亚洲欧美日韩综合久久久 | 国产人妻大战黑人第1集 | a在线观看免费网站大全 | 国产精品亚洲综合色区韩国 | 黑森林福利视频导航 | 国产亚av手机在线观看 | 欧美变态另类xxxx | 好男人www社区 | 欧美日本免费一区二区三区 | 无码纯肉视频在线观看 | 狠狠噜狠狠狠狠丁香五月 | 午夜肉伦伦影院 | 国产一区二区不卡老阿姨 | 午夜成人1000部免费视频 | 欧美国产亚洲日韩在线二区 | 精品欧美一区二区三区久久久 | 真人与拘做受免费视频一 | 亚洲精品综合一区二区三区在线 | 亚洲国产精品一区二区美利坚 | 国产精品国产自线拍免费软件 | aⅴ亚洲 日韩 色 图网站 播放 | 日韩人妻少妇一区二区三区 | 亚洲色欲色欲天天天www | 午夜精品一区二区三区的区别 | 久久 国产 尿 小便 嘘嘘 | 77777熟女视频在线观看 а天堂中文在线官网 | 久久天天躁狠狠躁夜夜免费观看 | 精品无码一区二区三区的天堂 | 巨爆乳无码视频在线观看 | 国产精品无码永久免费888 | 亚洲国产成人av在线观看 | 欧洲极品少妇 | 国产精品丝袜黑色高跟鞋 | 老熟妇仑乱视频一区二区 | 亚洲精品无码国产 | www成人国产高清内射 | 亚洲一区二区三区无码久久 | 无码人妻精品一区二区三区不卡 | 成人亚洲精品久久久久软件 | 婷婷五月综合激情中文字幕 | 欧美日韩精品 | 国产精品久久久久影院嫩草 | 伊在人天堂亚洲香蕉精品区 | 日韩精品无码一区二区中文字幕 | 大屁股大乳丰满人妻 | 欧美日韩一区二区三区自拍 | 亚洲精品久久久久中文第一幕 | 人妻夜夜爽天天爽三区 | 99er热精品视频 | 午夜福利不卡在线视频 | 亚洲精品一区三区三区在线观看 | 四虎影视成人永久免费观看视频 | 日韩少妇白浆无码系列 | 人妻无码久久精品人妻 | 色婷婷欧美在线播放内射 | 捆绑白丝粉色jk震动捧喷白浆 | 奇米综合四色77777久久 东京无码熟妇人妻av在线网址 | 国产精品亚洲а∨无码播放麻豆 | 免费观看的无遮挡av | 性欧美大战久久久久久久 | 性啪啪chinese东北女人 | 少妇人妻大乳在线视频 | 久久精品国产99久久6动漫 | 大地资源网第二页免费观看 | 妺妺窝人体色www在线小说 | 国产激情一区二区三区 | 欧美日韩精品 | 精品偷拍一区二区三区在线看 | 国产激情艳情在线看视频 | 自拍偷自拍亚洲精品被多人伦好爽 | 伊人久久大香线蕉亚洲 | av人摸人人人澡人人超碰下载 | 人妻天天爽夜夜爽一区二区 | 免费男性肉肉影院 | 亚洲国产日韩a在线播放 | 欧美日韩一区二区综合 | 日韩精品成人一区二区三区 | 大地资源中文第3页 | 亚洲精品综合五月久久小说 | 少妇激情av一区二区 | 国产又粗又硬又大爽黄老大爷视 | av人摸人人人澡人人超碰下载 | 欧美日韩亚洲国产精品 | 亚洲国产成人av在线观看 | 精品一区二区不卡无码av | 漂亮人妻洗澡被公强 日日躁 | 亚洲自偷自拍另类第1页 | 四虎影视成人永久免费观看视频 | 欧美精品免费观看二区 | 国产精品久免费的黄网站 | 国产真实夫妇视频 | 一本精品99久久精品77 | 国产亚洲视频中文字幕97精品 | 国产人妖乱国产精品人妖 | 国产成人无码一二三区视频 | 国产乱子伦视频在线播放 | 丝袜足控一区二区三区 | 黑森林福利视频导航 | 成人影院yy111111在线观看 | 中文字幕无码免费久久99 | 成年美女黄网站色大免费视频 | 永久免费观看美女裸体的网站 | 免费人成网站视频在线观看 | 亚洲国产一区二区三区在线观看 | 少妇高潮喷潮久久久影院 | 国产成人精品优优av | 亚洲呦女专区 | 中文字幕无码热在线视频 | 国产无av码在线观看 | 成人免费无码大片a毛片 | 亚洲爆乳精品无码一区二区三区 | www国产亚洲精品久久久日本 | 亚洲熟妇色xxxxx欧美老妇 | 午夜精品一区二区三区的区别 | 乱码av麻豆丝袜熟女系列 | 日韩人妻无码一区二区三区久久99 | 天天爽夜夜爽夜夜爽 | 国产人妻大战黑人第1集 | 国产成人精品必看 | 成年美女黄网站色大免费视频 | 最新版天堂资源中文官网 | 亚洲人成网站色7799 | 欧美日韩人成综合在线播放 | 久久精品99久久香蕉国产色戒 | 一本色道久久综合狠狠躁 | 免费人成网站视频在线观看 | 日韩av激情在线观看 | 在线观看欧美一区二区三区 | 久激情内射婷内射蜜桃人妖 | 国产精品久久久 | 国产亚洲欧美日韩亚洲中文色 | 精品久久8x国产免费观看 | 国产av一区二区三区最新精品 | 在线播放无码字幕亚洲 | 清纯唯美经典一区二区 | 中文字幕无码av激情不卡 | 日韩少妇白浆无码系列 | 国产亚洲精品久久久久久久久动漫 | 色婷婷欧美在线播放内射 | 国产另类ts人妖一区二区 | √天堂资源地址中文在线 | 久久久国产一区二区三区 | 日本肉体xxxx裸交 | 99视频精品全部免费免费观看 | 色综合久久久久综合一本到桃花网 | 女人被爽到呻吟gif动态图视看 | 日日橹狠狠爱欧美视频 | 四虎国产精品免费久久 | 亚洲色偷偷偷综合网 | 亚洲欧洲日本综合aⅴ在线 | 中文字幕+乱码+中文字幕一区 | 成人无码视频在线观看网站 | 一个人看的www免费视频在线观看 | 久久天天躁夜夜躁狠狠 | 欧美丰满老熟妇xxxxx性 | 乱码av麻豆丝袜熟女系列 | 国产精品沙发午睡系列 | 桃花色综合影院 | 婷婷色婷婷开心五月四房播播 | 亚洲七七久久桃花影院 | 亚洲精品一区二区三区在线观看 | 国产成人精品视频ⅴa片软件竹菊 | 中文字幕无码免费久久99 | 色情久久久av熟女人妻网站 | 99国产精品白浆在线观看免费 | 对白脏话肉麻粗话av | 性生交大片免费看女人按摩摩 | 国产精品久久久久9999小说 | 无码人妻久久一区二区三区不卡 | 学生妹亚洲一区二区 | 国产无遮挡又黄又爽又色 | 欧美黑人乱大交 | 曰本女人与公拘交酡免费视频 | 欧美野外疯狂做受xxxx高潮 | 色综合视频一区二区三区 | 亚洲狠狠色丁香婷婷综合 | 精品成在人线av无码免费看 | 中文字幕久久久久人妻 | 日日天日日夜日日摸 | 麻花豆传媒剧国产免费mv在线 | 女人高潮内射99精品 | 国内精品人妻无码久久久影院蜜桃 | 乱码av麻豆丝袜熟女系列 | 天堂а√在线地址中文在线 | 性生交大片免费看l | 无码中文字幕色专区 | 九九热爱视频精品 | 18无码粉嫩小泬无套在线观看 | 麻豆av传媒蜜桃天美传媒 | 国产成人精品视频ⅴa片软件竹菊 | 丁香花在线影院观看在线播放 | 日韩精品久久久肉伦网站 | 成 人 免费观看网站 | 人人爽人人澡人人高潮 | 人妻少妇精品视频专区 | 久久久久久久久888 | 乌克兰少妇xxxx做受 | 精品国偷自产在线 | 一本久道久久综合婷婷五月 | 日韩亚洲欧美精品综合 | 精品一区二区三区无码免费视频 | 成人免费视频一区二区 | 老子影院午夜精品无码 | 国产国语老龄妇女a片 | 久热国产vs视频在线观看 | 亚洲中文字幕在线无码一区二区 | 国产精品18久久久久久麻辣 | 99久久久无码国产精品免费 | 在线a亚洲视频播放在线观看 | 亚洲日韩av一区二区三区四区 | 亚洲国产精品一区二区美利坚 | 久久亚洲日韩精品一区二区三区 | 欧美精品国产综合久久 | 国产精品亚洲一区二区三区喷水 | 亚洲最大成人网站 | 欧美日韩一区二区三区自拍 | 真人与拘做受免费视频 | 国产精品久久久久久无码 | 国产真实夫妇视频 | 日本丰满护士爆乳xxxx | 丰满护士巨好爽好大乳 | 亚洲成a人片在线观看无码 | 成熟女人特级毛片www免费 | 色婷婷综合中文久久一本 | 人人妻人人澡人人爽欧美精品 | 成人三级无码视频在线观看 | 中文亚洲成a人片在线观看 | 亚洲成av人综合在线观看 | 2020久久超碰国产精品最新 | 国产电影无码午夜在线播放 | 国产精品对白交换视频 | 麻豆国产丝袜白领秘书在线观看 | 日韩人妻系列无码专区 | 久久久婷婷五月亚洲97号色 | 久久99热只有频精品8 | 岛国片人妻三上悠亚 | 亚洲男女内射在线播放 | 捆绑白丝粉色jk震动捧喷白浆 | 少妇无码av无码专区在线观看 | 国产亲子乱弄免费视频 | 久久久久久av无码免费看大片 | 无码国产激情在线观看 | 无码人妻精品一区二区三区不卡 | 久久久久久亚洲精品a片成人 | 色老头在线一区二区三区 | 亚洲精品成a人在线观看 | 亚洲精品国产精品乱码视色 | 亚洲一区二区三区四区 | 欧洲熟妇精品视频 | 国产精品免费大片 | 精品久久综合1区2区3区激情 | 日欧一片内射va在线影院 | 国产在线一区二区三区四区五区 | 免费男性肉肉影院 | 亚洲s色大片在线观看 | 帮老师解开蕾丝奶罩吸乳网站 | 人人澡人摸人人添 | 日日天干夜夜狠狠爱 | 国产精品第一国产精品 | 中国女人内谢69xxxx | 亚洲日韩精品欧美一区二区 | 亚洲精品一区三区三区在线观看 | 欧美放荡的少妇 | 欧美性猛交内射兽交老熟妇 | 国产精品香蕉在线观看 | 波多野结衣一区二区三区av免费 | 久久精品一区二区三区四区 | 久久国产劲爆∧v内射 | 思思久久99热只有频精品66 | 小鲜肉自慰网站xnxx | 久激情内射婷内射蜜桃人妖 | 99久久人妻精品免费二区 | 夜夜夜高潮夜夜爽夜夜爰爰 | 中文精品无码中文字幕无码专区 | 欧美35页视频在线观看 | 成人av无码一区二区三区 | 免费观看的无遮挡av | 国产三级精品三级男人的天堂 | 67194成是人免费无码 | 少妇被粗大的猛进出69影院 | 中文字幕亚洲情99在线 | 亚洲精品一区二区三区婷婷月 | 亚洲s码欧洲m码国产av | 欧美日韩人成综合在线播放 | 丰满少妇人妻久久久久久 | 丝袜人妻一区二区三区 | a片在线免费观看 | 大地资源中文第3页 | 日韩精品成人一区二区三区 | 爱做久久久久久 | 久久国产精品偷任你爽任你 | 国产成人综合色在线观看网站 | 午夜熟女插插xx免费视频 | 国产精品人妻一区二区三区四 | 日韩人妻无码一区二区三区久久99 | 午夜男女很黄的视频 | 亚洲色成人中文字幕网站 | 又大又硬又爽免费视频 | 中文字幕无码人妻少妇免费 | 亚欧洲精品在线视频免费观看 | 中文字幕日韩精品一区二区三区 | 亚洲精品一区二区三区四区五区 | 极品尤物被啪到呻吟喷水 | 丰满岳乱妇在线观看中字无码 | 久久久亚洲欧洲日产国码αv | www一区二区www免费 | 麻豆精品国产精华精华液好用吗 | 一区二区三区乱码在线 | 欧洲 | 欧美真人作爱免费视频 | 国产美女极度色诱视频www | 小鲜肉自慰网站xnxx | 久久国产精品精品国产色婷婷 | 永久免费精品精品永久-夜色 | 扒开双腿疯狂进出爽爽爽视频 | 天天摸天天透天天添 | 草草网站影院白丝内射 | 曰本女人与公拘交酡免费视频 | 亚洲国产欧美在线成人 | 免费中文字幕日韩欧美 | 久久综合久久自在自线精品自 | 在线精品亚洲一区二区 | 内射巨臀欧美在线视频 | 国产精品美女久久久网av | 男女超爽视频免费播放 | 性欧美疯狂xxxxbbbb | 精品偷拍一区二区三区在线看 | 国产日产欧产精品精品app | 粉嫩少妇内射浓精videos | 国产精品久久久久影院嫩草 | 欧美熟妇另类久久久久久不卡 | 日韩精品无码一区二区中文字幕 | 1000部啪啪未满十八勿入下载 | 乌克兰少妇xxxx做受 | av小次郎收藏 | 中文无码精品a∨在线观看不卡 | 亚洲一区二区三区播放 | 国产综合久久久久鬼色 | 久久久无码中文字幕久... | 国产9 9在线 | 中文 | 三上悠亚人妻中文字幕在线 | 99久久精品午夜一区二区 | 久久99久久99精品中文字幕 | 国产精品久久久久影院嫩草 | 国产真实伦对白全集 | 欧美 丝袜 自拍 制服 另类 | 欧美人与善在线com | 国产精品.xx视频.xxtv | 成人精品视频一区二区三区尤物 | 亚洲熟悉妇女xxx妇女av | 日本一区二区更新不卡 | 55夜色66夜色国产精品视频 | 一本久道高清无码视频 | 大地资源网第二页免费观看 | 欧美 日韩 人妻 高清 中文 | 国产女主播喷水视频在线观看 | v一区无码内射国产 | 亚洲成av人片天堂网无码】 | аⅴ资源天堂资源库在线 | 精品无码av一区二区三区 | 精品久久久久香蕉网 | 亚洲中文字幕av在天堂 | 奇米影视888欧美在线观看 | 国产乱人偷精品人妻a片 | 国产在线无码精品电影网 | 天天拍夜夜添久久精品大 | 国产精品亚洲专区无码不卡 | 兔费看少妇性l交大片免费 | 巨爆乳无码视频在线观看 | 999久久久国产精品消防器材 | 国产午夜福利亚洲第一 | 99精品无人区乱码1区2区3区 | 人人澡人人妻人人爽人人蜜桃 | 少妇高潮一区二区三区99 | 欧美xxxxx精品 | 樱花草在线播放免费中文 | 亚洲va中文字幕无码久久不卡 | 97精品人妻一区二区三区香蕉 | 欧美喷潮久久久xxxxx | 蜜臀av无码人妻精品 | 中文字幕人妻丝袜二区 | 精品国产一区av天美传媒 | 一本色道久久综合亚洲精品不卡 | 国产亚洲欧美日韩亚洲中文色 | 免费观看的无遮挡av | 国产精品无码mv在线观看 | 中文字幕无码热在线视频 | 乱码午夜-极国产极内射 | 色五月五月丁香亚洲综合网 | 天堂一区人妻无码 | 国产卡一卡二卡三 | 亚洲欧美中文字幕5发布 | 欧美真人作爱免费视频 | 国产精品亚洲综合色区韩国 | 国产日产欧产精品精品app | 久久精品无码一区二区三区 | 国产情侣作爱视频免费观看 | 亚洲乱码国产乱码精品精 | 国产精品美女久久久久av爽李琼 | 精品偷自拍另类在线观看 | 高清不卡一区二区三区 | 免费无码一区二区三区蜜桃大 | 男人扒开女人内裤强吻桶进去 | 日本一区二区三区免费播放 | 国产三级久久久精品麻豆三级 | 国产成人综合在线女婷五月99播放 | 国产精品多人p群无码 | 国产后入清纯学生妹 | 亚洲精品一区国产 | 鲁鲁鲁爽爽爽在线视频观看 | 免费视频欧美无人区码 | 国内老熟妇对白xxxxhd | 无码av最新清无码专区吞精 | 亚洲色在线无码国产精品不卡 | 人妻插b视频一区二区三区 | 亚洲一区二区三区无码久久 | 少妇一晚三次一区二区三区 | 亚洲 欧美 激情 小说 另类 | 国产在线精品一区二区高清不卡 | 日韩成人一区二区三区在线观看 | 国产精品成人av在线观看 | 婷婷丁香五月天综合东京热 | 国产网红无码精品视频 | 精品乱子伦一区二区三区 | 沈阳熟女露脸对白视频 | 99精品国产综合久久久久五月天 | 小sao货水好多真紧h无码视频 | 久久综合激激的五月天 | 在线 国产 欧美 亚洲 天堂 | 大地资源网第二页免费观看 | 中文亚洲成a人片在线观看 | 在线亚洲高清揄拍自拍一品区 | 亚洲一区二区三区在线观看网站 | 国产激情无码一区二区 | 亚洲精品国产第一综合99久久 | 天天拍夜夜添久久精品大 | 99久久久国产精品无码免费 | 天天躁日日躁狠狠躁免费麻豆 | 亚洲区欧美区综合区自拍区 | 蜜臀aⅴ国产精品久久久国产老师 | 亚洲精品一区二区三区在线观看 |