Monday 18 April 2011

Debugging a device in ESX using esxtop.



There is plenty of information in vmware and yellow-bricks blogs on advanced esxtop components. I believe you would get a very good insight on esxtop usage from these sites.
However I would like to discuss a small section of esxtop, which would be very helpful if you would like to debug your storage devices or watch the IOPS that are exchanged over multiple paths between ESX and storage devices(Good to check the performance of VMW_PSP_RR).
To get the stats in esxtop for storage device
1. Execute "esxtop" on ESX console.
2. Type "u" to view storage device stats.
3. If your devices are identified by  naa IDs, then  enter SHIFT+ L.
4. Type "36", this will give you complete id of the storage device.
5. Now this is the best part. If you want to watch IOs flowing through multiple paths especially if you have configured VMW_PSP_RR for the device then type "SHIFT+P". An option would appear on the top of "esxtop"  to enter the device ID. Enter the device ID of the device for which you want to view the stats.
And you would see all the paths and IOs flowing through them for the specific device.


Often you might have observed failures while unclaiming a device.This is because device might be actively servicing IOs, or some world has the device open. To know if a world has the device open, you can try option "e"(just e) after step 4.
This would again ask for device info. Enter the device name and hit enter.
This time you would see a list of worlds working on the device.


After you get the world id, you can grep the process information as follows.
~ # ps -u | grep 91559
91559 5124 vix-async-pipe hostd
In my case world '91559' that is using the device is hostd.

1 comment:

  1. I guess this was an ESXi. In ESX hostd runs in the service console and you would see World ID 4096 instead. Service console activity is opaque to esxtop, and hostd runs inside the service console. On the other hand, in ESXi hostd runs in the vmkernel itself, vmkernel processes (AKA worlds) are not opaque to esxtop.

    ReplyDelete