COPYRIGHT
VMware ESX Server is Copyright 2006 VMware, Inc. All rights reserved.
SYNOPSIS
esxtop [-] [h] [v] [b] [s] [R vm-support_dir_path] [d delay] [n iter]
DESCRIPTION
esxtop provides a fine-grain look at resource utilization in real time. For more information, please refer to the official documentation, available at <http://www.vmware.com/info?id=193>.

Esxtop runs on the ESX Server Service Console. It can only be run by user root. It can be run in three different modes; interactive (default), batch and replay. Worlds in this document refer to ESX Server VMKernel schedulable entities, similar to processes in other operating systems.
Interactive Mode
This is the default mode for esxtop. This mode displays statistics under four broad categories. Each category is displayed on a seperate screen. There are screens displaying CPU, memory, storage and network resource usage. A help menu is available for each of the four screens.
In this mode there are several command line options available.
Command-line Options
h Prints help for esxtop command-line options. v Prints esxtop version number. s Tells esxtop to run in secure mode. This disables the interactive command to change delay between screen updates. d Specifies the delay between screen updates. Default is 5 seconds. Minimum is 2 seconds. You can change this with the interactive command 's'. If a delay of less than 2 seconds is specified then the delay is set to 2 seconds. n Number of iterations. Update the display 'n' number of times and then exit.

Common Statistics Description
Several statistics are displayed on the different screens while esxtop is running in interactive mode. Statistics listed in this section are common across all four screens. "uptime" This first line found at the top of each of the four esxtop screens displays the current time, time since last reboot and number of currently running Worlds.

On this first line CPU load averages are also displayed for the CPU, storage and network screens. The load averages over the past 1, 5 and 15 minutes are displayed. Load averages take into account both running and ready-to-run Worlds. A load average of 1.00 implies that all the physical CPUs are fully accounted for. A load average of 2.00 implies that the ESX Server may be in need of twice as many physical CPUs as currently available.

For the memory screen memory overcommit averages are also displayed on the first line. The memory overcommit averages over the past 1, 5 and 15 minutes are displayed. A memory overcommit average of 0.37 implies that the memory is overcommit by 37%.

Common Interactive Commands
Several single-key commands are recognized while esxtop is running in interactive mode. Commands listed in this section are common across all four screens. The command to specify the delay between updates is disabled if the s option has been given on the command line. All sorting interactive commands sort in descending order. "h or ?" Displays a help menu for the current screen giving a brief summary of commands, and the status of secure mode. space Immediately updates the current screen. ^L Erases and redraws the current screen. "f or F" Displays a screen to add or remove statistics columns (fields) to or from the current screen. See below for more information. "o or O" Displays a screen to change order of statistics columns on the current screen. See below for more information. # Change the number of statistics rows to show. You will be prompted to enter the number. Any value greater than 0 overrides automatic determination of the number of rows to show, which is based on window size measurement. This number updated on any one of the four esxtop screens will be used for all esxtop screens. s Change the delay between updates. You will be prompted to enter the delay time, in seconds, between updates. Fractional values are recognized down to microseconds. The default value is 5 seconds. The minimum value is 2 seconds. This command is not available in secure mode. W Write current setup to ~/.esxtop3rc. This is the recommended way to write an esxtop configuration file. See below for more information. q Quit. c switch to CPU resource utilization screen. m switch to memory resource utilization screen. d switch to storage (disk) resource utilization screen. n switch to network resource utilization screen.

The Statistics Columns (fields) and Order Screens
After pressing f, F, o or O you will be shown a screen specifying the field order on the top line and short descriptions of the field contents. The field order string uses the following syntax: if the letter in the field string corresponding to a field is upper case, the field will be displayed. This is also indicated by an asterisk in front of the field description. The order of the fields corresponds to the order of the letters in the string.

From the field select screen you can toggle the display of a field by pressing the corresponding letter.
From the order screen you may move a field to the left by pressing the corresponding upper case letter and to the right by pressing the lower case one.

Configuration File
esxtop reads its default configuration from ~/.esxtop3rc. This configuration file contains five lines. The first four lines contain lower and upper case letters to specify which fields in what order are to be displayed on the CPU, memory, storage and network screens. The letters correspond to the letters in the Fields or Order screens for the respective esxtop screens. Editing this file by hand is not recommended. Select fields and order in a running esxtop process and save this file using the W interactive command. The fifth line contains information on the other options. Most important, if you have saved a configuration in secure mode, you will not get an insecure esxtop without removing the lower 's' from this fifth line of your ~/.esxtop3rc. A digit specifies the delay time between updates. As in interactive mode, a lower 'c', 'm', 'd' and 'n' determines the screen esxtop starts on.

CPU Screen
This screen displays server-wide and per Resource Pool/World CPU utilization statistics. By default, World CPU utilization statistics are aggregated into Resource Pools of Worlds. For Worlds belonging to a Virtual Machine, these Resource Pools correspond to running Virtual Machines. All other Worlds are logically aggregated into Resource Pools to save vertical window space.

Statistics Descriptions PCPU(%) This line displays the percentage CPU utilization per physical CPU and the total average physical CPU utilization. LCPU(%) This line displays the percentage CPU utilization per logical CPU. The CPU used percentages for the logical CPUs belonging to a package add up to 100%. This line is displayed only if hyper-threading is present and enabled. CCPU(%) This line displays the percentages of total CPU time as reported by the ESX Server Service Console. us is for percentage user time, sy is for percentage system time, id is for percentage idle time and wa is for percentage wait time. cs/sec is for the context switches per second recorded by the ESX Server Service Console. ID The Resource Pool ID of the running World's Resource Pool or World ID of running World. GID The Resource Pool ID of running World's Resource Pool. NAME The name of running World's Resource Pool or name of running World. NMEM The number of members in running World's Resource Pool. If a Resource Pool is expanded using the interactive command e (see interactive commands below) then NMEM for all the resulting Worlds belonging to the Resource Pool will be 1 (some Resource Pool like the console Resource Pool have only 1 member). %STATE TIMES This is a set of CPU statistics made up of the following percentages. Note that for a World, the percentages are a percentage of one physical CPU. %USED The percentage physical CPU used by the Resource Pool/World. %SYS The percentage of time spent in the ESX Server VMKernel on behalf of the Resource Pool/World to process interrupts and to perform other system activities. This time is part of used time used to calculate %USED above. %OVRLP The Percentage of system time that was spent on behalf of some other Resource Pool/World while Resource Pool/World was scheduled. This time is not included in the %SYS for Resource Pool/World. For example, if Resource Pool A was currently scheduled and a network packet for Resource Pool B was being processed by the ESX Server VMKernel, then the time spent doing so appears as %OVRLP for Resource Pool A and %SYS for Resource Pool B. %RUN Percentage of total time scheduled. This time does not account for hyper-threading and system time. Hence, on a hyper-threading enabled server, the %RUN can be twice as large as %USED. %TWAIT The percentage of time the Resource Pool/World spent in wait state. Note that this percentage includes the percentage of time the Resource Pool/World was idle. %IDLE The percentage of time the Resource Pool/World was idle. Subtracting this percentage from %TWAIT above gives you the percentage of time the Resource Pool/World was waiting on some event. %RDY The percentage of time the Resource Pool/World was ready to run. %MLMTD Percentage of time the ESX Server VMKernel deliberately didn't run the Resource Pool/World because that would violate the Resource Pool/World's limit setting. Even though the Resource Pool/World is ready to run when it is prevented from running in this way, the %MLMTD time is not included in %RDY time. EVENT COUNTS/s This is a set of CPU statistics made up of the following, per second, event rates. These statistics are for internal to VMware use only. CPU ALLOC This is a set of CPU statistics made up of the following CPU allocation configuration parameters. AMIN Resource Pool/World attribute reservation. AMAX Resource Pool/World attribute limit. ASHRS Resource pool attribute shares. SUMMARY STATS This is a set of CPU statistics made up of the following CPU configuration parameters and statistics. Note that these statistics are only applicable to Worlds and not Resource Pools. AFFINITY BIT MASK Bit mask showing the current scheduling affinity for the World. HTSHARING current hyper-threading configuration. CPU The physical or logical processor the World was found to be running on. HTQ World is currently quaratined or not. 'N' implies no and 'Y' implies yes. TIMER/s The timer rate the World is currently requesting for.

Interactive Commands e Expand/Rollup Resource Pool/Worlds CPU statistics. Allows viewing CPU resource utilization statistics broken down by individual Worlds belonging to a Resource Pool. U Sort Resource Pools/Worlds by Resource Pool %USED column. R Sort Resource Pools/Worlds by Resource Pool %RDY column. N Sort Resource Pools/Worlds by GID column. This is the default sort order.
Memory Screen
This screen displays server-wide and per Resource Pool memory utilization statistics. As on the CPU screen, Resource Pools correspond to running Virtual Machines or Worlds that consume memory. Note the distinction between machine memory and physical memory in this section.

Interactive Commands M Sort Resource Pools by Resource Pool MEMSZ column. B Sort Resource Pools by Resource Pool MCTLSZ column. N Sort Resource Pools by GID column. This is the default sort order.

Storage Screen
This screen displays server-wide storage utilization statistics. On this screen statistics are aggregated per storage adapter by default. Statistics can also be viewed per storage channel, target, LUN or World using a LUN. See below for more information.

Statistics Descriptions ADAPTR The name of the storage adapter. CID The storage adapter channel id. This id is only visible if the corresponding adapter is expanded. See interactive command 'e' below. TID The storage adapter channel target id. This id is only visible if the corresponding adapter and channel are expanded. See interactive commands 'e' and 'a' below. LID The storage adapter channel target LUN id. This id is only visible if the corresponding adapter, channel and target are expanded. See interactive commands 'e', 'a' and 't' below. WID The storage adapter channel target LUN World id. This id is only visible if the corresponding adapter, channel, target and LUN are expanded. See interactive commands 'e', 'a', 't' and 'l' below. NCHNS The number of channels. NTGTS The number of targets. NLUNS The number of LUNs. NVMS The number of Worlds. SHARES The number of shares. BLKSZ The block size in bytes. This statistic is only applicable to LUNs. AQLEN The storage adapter queue depth. This is the maximum number of ESX Server VMKernel active commands that the adapter driver is configured to support. LQLEN The LUN queue depth. This is the maximum number of ESX Server VMKernel active commands that the LUN is allowed to have. WQLEN The World queue depth. This is the maximum number of ESX Server VMKernel active commands that the World is allowed to have. Note that this is a per LUN maximum for the World. ACTV The number of commands in the ESX Server VMKernel that are currently active. QUED The number of commands in the ESX Server VMKernel that are currently queued. %USD The percentage of queue depth (adapter, LUN or World, whichever is applicable) used by ESX Server VMKernel active commands. LOAD The ratio of ESX Server VMKernel active commands plus ESX Server VMKernel queued commands to queue depth (adapter, LUN or World, whichever is applicable). CMDS/s The number of commands issued per second. READS/s The number of read commands issued per second. WRITES/s The number of write commands issued per second. MBREAD/s The megabytes read per second. MBWRTN/s The megabytes written per second. DAVG/cmd The average device latency (millisecs) per command. KAVG/cmd The average ESX Server VMKernel latency (millisecs) per command. GAVG/cmd The average Guest OS latency (millisecs) per command. ABRTS/s The number of commands aborted per second. RESETS/s The number of commands reset per second.

Interactive Commands e Expand/Rollup storage adapter statistics. Allows viewing storage resource utilization statistics broken down by individual channels belonging to an expanded storage adapter. You will be prompted to enter the adapter name. a Expand/Rollup storage channel statistics. Allows viewing storage resource utilization statistics broken down by individual targets belonging to an expanded storage channel. You will be prompted to enter the adapter name first and then the channel id. Note that the channel adapter needs to be expanded before the channel itself can be expanded. t Expand/Rollup storage target statistics. Allows viewing storage resource utilization statistics broken down by individual LUNs belonging to an expanded storage target. You will be prompted to enter the adapter name first, then the channel id and then the target id. Note that the target channel and adapter need to be expanded before the target itself can be expanded. l Expand/Rollup storage LUN statistics. Allows viewing storage resource utilization statistics broken down by individual Worlds utilizing an expanded storage LUN. You will be prompted to enter the adapter name first, then the channel id, then the target id and finally the LUN id. Note that the LUN target, channel and adapter need to be expanded before the LUN itself can be expanded. r Sort by READS/s column. w Sort by WRITES/s column. R Sort by MBREAD/s column. T Sort by MBWRTN/s column. N Sort first by ADAPTR column, then by CID column within each ADAPTR, then by TID column within each CID, then by LID column within each TID and finally by WID column within each LID. This is the default sort order.
Network Screen
This screen displays server-wide network utilization statistics. On this screen statistics are arranged per port per configured virtual network device. For physical NIC statistics, see the row corresponding to the port that the physical NIC is connected to. For virtual NIC configured in a particular Virtual Machine statistics, see the row corresponding to the port that the virtual NIC is connected to.

Statistics Descriptions PORT ID The virtual network device port id. UPLINK Y implies the corresponding port is an uplink. N implies it is not. UP Y implies the corresponding link is up. N implies it is not. SPEED The link speed in MegaBits per second. FDUPLX Y implies the corresponding link is operating at full duplex. N implies it is not. USED BY The virtual network device port user. DTYP The virtual network device type. H implies HUB and S implies switch. DNAME The virtual network device name. PKTTX/s The number of packets transmitted per second. PKTRX/s The number of packets received per second. MbTX/s The MegaBits transmitted per second. MbRX/s The MegaBits received per second. %DRPTX The percentage of transmit packets dropped. %DRPRX The percentage of receive packets dropped.

Interactive Commands T Sort by MbTX/s column. R Sort by MbRX/s column. t Sort by PKTTX/s column. r Sort by PKTRX/s column. N Sort by PORT ID column. This is the default sort order.
Batch Mode
Allows collecting and saving resource utilization statistics in a file. Running in this mode is a two step process.
Step one requires running esxtop in interactive mode, switching to each of the four available screens, selecting the columns on each screen you are interested in and saving this configuration in the ~/.esxtop3rc file using the W interactive command. See Interactive Mode above.
Step two requires running esxtop in batch mode and redirecting the output to a file. This may be done as follows: esxtop -b > foo.csv. Note that the file name needs to have a .csv extension. Esxtop does not enforce this, but post-processing tools mentioned next require this.
Statistics collected in batch mode may be post-processed using such MS Windows applications like Excel and Perfmon.
In this mode, esxtop will not accept interactive commands. Esxtop in batch mode runs until it produces the number of iterations requested (see command-line option n below for more details) or until killed using CTRL c.
In this mode there are several command line options available.
Command-line Options
b Run esxtop in Batch mode. d Specifies the delay between statistics snapshots. Default is 5 seconds. Minimum is 2 seconds. If a delay of less than 2 seconds is specified then the delay is set to 2 seconds. n Number of iterations. Collect and save statistics this number of times and then exit.
Replay Mode
Replays resource utilization statistics collected using vm-support (see vm-support man page for more information). Running in this mode is a two step process.
Step one requires running vm-support in snapshot mode on the ESX Server Service Console. This may be done as follows: vm-support -S -d duration -i interval. The resulting zipped tar file needs to be un-zipped and un-tar'ed before use by esxtop replay mode.
Step two requires running esxtop in replay mode. This may be done as follows: esxtop -R vm-support_dir_path. Note that it is not required to run esxtop replay mode on the ESX Server Service Console.
Replay mode can be run to produce Batch Mode style output (see command-line option b below for more information).
In replay mode, esxtop accepts the same set of interactive commands as in Interactive Mode. Esxtop in replay mode runs until there are no more vm-support collected snapshots to be read or esxtop produces the number of iterations requested (see command-line option n below for more details).
In this mode there are several command line options available.
Command-line Options
R Specifies the path to the vm-support collected snapshot's directory. b Run esxtop in Batch mode. d Specifies the delay between screen updates. Default is 5 seconds. Minimum is 2 seconds. If a delay of less than 2 seconds is specified then the delay is set to 2 seconds. n Number of iterations. Update the display this number of times and then exit.
FILES
~/.esxtop3rc The personal configuration file.
DISCLAIMER
Based on top source code from the OpenBSD distribution.

Copyright (c) 1997, Jason Downs. All rights reserved.
Copyright (c) 1984, 1989, William LeFebvre, Rice University
Copyright (c) 1989, 1990, 1992, William LeFebvre,
Northwestern University

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.