In this second part of our monitoring Hyper-V article we’ll discuss the details about Processor, Memory, and Storage monitoring. In part 1 we discussed the Overall Health metric set. Let’s start with the Processor Metrics.
There are three processor-related metric sets:
- Hyper-V Hypervisor Logical Processor
- Hyper-V Hypervisor Root Virtual Processor
- Hyper-V Hypervisor Virtual Processor
The most important counter set you want to monitor is the Hyper-V Hypervisor Logical Processor. This counter set allows you to determine how much of the physical processor is being used. The Virtual Processor counter set only shows part of the Hyper-V Hypervisor Logical Processor, so you want to make sure to use the Logical Processor counter set. The most useful counters in this counter set are the following;
- %Guest Run
- %Hypervisor Run Time
- %Idle Run Time
- %Total Run Time
Note that there is one logical processor in Hyper-V that that has a larger workload than the other logical processors and that is LP0. This Logical Processor is where all interrupts in the system are directed to. If this LP hits 100% utilization there is likely an I/O bottleneck.
The Hyper-V Hypervisor Root Virtual Processor and Hyper-V Hypervisor Virtual Processor are just slices of the Logical Processor counter and can help you understand how much total CPU the root and guests are using on the system.
If you want to monitor the root CPU you should use the Processor counter set. According to Microsoft this counter suffers from some skew, but is useful because it gives an idea of how busy the root is. This is important because the root is involved in all I/O, meaning if the root CPU’s are saturated your whole system is likely saturated. In general you want to see the root CPU lower than 10% utilization and over 50% might indicate an issue.
The following are the counter sets you want to monitor;
- Hyper-V Hypervisor Partition
- Hyper-V Hypervisor Root Partition
- Hyper-V VM Vid Partition
The Hyper-v [Root] Partition counters determine how much memory the Hypervisor is managing and using on behalf of a VM which includes the guest address space but not all the memory in the worker process and VID partition. The Hyper-V Hypervisor [ROOT] Partition counter set indicates in the 1G GPA Pages and 2M GPA Pages metric whether or not a VM is using large memory pages which improves overall VM performance.
The Hyper-V VM Vid Partition counters account for the guest address space and any additional memory the VID needs to manage the VM.
The Memory counters are not specific to any VM and can be used to measure memory used to service I/O on behalf of a virtual machine.
The Partition Counter set also indicates in the Deposited Pages metric how much memory the hypervisor is using for managing the VM. The last counter that is interesting in the partition counter set is the Virtual Processors counter which tells you how many Virtual Processors a VM is configured to use.
The Hyper-V VM Vid Partition counters have two counters that you want to monitor. The Physical Pages Allocated is the total number of guest pages and VID pages needed to manage the VM. The Remote Physical Pages let you know on NUMA based systems if a VM is spanning multiple nodes. Note that Microsoft does not recommend such a configuration.
The Memory counter set allows you to monitor how much memory is consumed in the root. The root is responsible for managing all memory in Hyper-V. When a VM starts you will see the Available Bytes go down by at least the amount of memory given to the guest plus around another 16 – 64MB for guests meta data structures.
The following counters are useful to monitor:
- Available Bytes – Returns how much memory is remaining for guests.
- Pages / Sec – Measures memory pressure. This metric can tell you if processes are competing with each other for physical RAM.
You can use the network counters to monitor overall networking performance on the system. The most important thing to monitor is the total throughput counters to make sure the NICs are not getting saturated. There are four relevant counter sets:
- Network Interface
- Hyper-V Virtual Switch
- Hyper-V Legacy Network Adapter
- Hyper-V Virtual Network Adapter
The Network Interface measures the overall performance of physical device whereas the other counter sets represent the activity of the virtual switches and network adapters in the virtual machines. For the Network Interface the following are the most important counters to monitor:
- Bytes Total / Sec
- Offloaded Connections
- Packets / Sec
- Packets Outbound Errors
- Packets Receive Errors
The Hyper-V Virtual Switch counters are good to monitor because depending on your configured network, traffic might only exist on the virtual switch. This is the case with network traffic between guest to guest. The most important counters are:
The Hyper-V Virtual Network Adapter and the Hyper-V Legacy Network Adapter counter sets allow you to measure performance of a virtual machine. These counter sets are named with the friendly name of the VM plus the name of the network adapter followed by two GUIDs. The GUIDs are the internal id of the VM and adapter.
If you assign a Legacy Network Adapter to a VM then the counter set to use is the Hyper-V Legacy Network Adapter. In general you should not use this network adapter type because it is not enlightened (see our prior ‘Best Practices’ article). It creates a lot of CPU load in the root, and is slower than the Virtual Network Adapter. However, you need the Legacy Network Adapter to get a VM working before installing Integration Services. Once your VM is working with Integration Services you should use the Network Adapter and the “Hyper-V Virtual Network Adapter” counter set. Note that both Windows Server 2008 and Windows Server 2008 R2 have integration Services pre-installed.
The Hyper-V Legacy Network Adapter counters to monitor are:
- Bytes Dropped
- Bytes Sent / Sec
- Bytes Received / Sec
In the Hyper-V Virtual Network Adapter you should monitor;
- Bytes / Sec
- Packets / Sec
The storage counters are used to monitor overall disk performance on the system and each VM. There are:
- Physical Disk
- Hyper-V Virtual Storage Device
- Hyper-V Virtual IDE Controller
The Physical Disk counter set will give overall storage performance on the system. The other two metrics are for VM’s only. Of the Physical Disk counter set there are three metrics that are useful to monitor. The Current Disk Queue Length measures how busy the drives are. This counter should be around two per drive, depending on your drive configuration. A queue length of 32 might indicate this disk is the bottleneck in your system.
Another useful counter is the Disk Bytes / Sec. You can typically expect to see about 10MB/sec per drive; a safe number for most drives. For random workloads, look at the counter Disk Transfers/Sec. Expect to see about 100 I/O’s per second (IOPs) for each drive. There are drives that can do much better, like 180 IOPs and some that perform worse.
Because of the way storage works in Hyper-V, there are two Hyper-V storage counter sets. Hyper-V provides two virtual storage buses for virtual machines. One IDE and one SCSI. The Virtual IDE counters show up in the Hyper-V Virtual IDE Controller counter set, unless the Integration Services are loaded. In that case you’ll see the activity for both virtual IDE and SCSI in the Hyper-V Virtual Storage Device counter set. If you don’t have integration services installed, only the Hyper-V Virtual IDE Controller will show the VM disk activity.
It wouldn’t hurt to monitor all the counters of both Hyper-V Virtual IDE Controller and Hyper-V Virtual Storage Device. Below are the basic metrics for these two counter sets:
For Hyper-V Virtual IDE Controller there are:
- Read Bytes / Sec
- Write Bytes / Sec
- Read Sectors / Sec
- Write Sectors / Sec
For Hyper-V Virtual Storage Device there are:
- Error Count
- Flush Count
- Read Bytes / Sec
- Write Bytes / Sec
- Read Count
- Write Count
That’s it for our discussion about the Hyper-V metrics that can be used to help you understand what the system is doing. In the table below, we’ll list all the counters:
- \Hyper-V Virtual Machine Health Summary \*
- \Hyper-V Hypervisor\*
- \Hyper-V Hypervisor Logical Processor(*)\%Guest Run
- \Hyper-V Hypervisor Logical Processor(*)\%Hypervisor Run Time
- \Hyper-V Hypervisor Logical Processor(*)\%Idle Run Time
- \Hyper-V Hypervisor Logical Processor(*)\%Total Run Time
- \Hyper-V Hypervisor Root Virtual Processor (*)\%Guest Run
- \Hyper-V Hypervisor Root Virtual Processor (*)\%Hypervisor Run Time
- \Hyper-V Hypervisor Root Virtual Processor (*)\%Idle Run Time
- \Hyper-V Hypervisor Root Virtual Processor (*)\%Total Run Time
- \Hyper-V Hypervisor Virtual Processor (_Total)\*
- \Memory\Pages / Sec
- \Memory\Available Bytes
- \Hyper-V Hypervisor Partition(*)\2M GPA Pages
- \Hyper-V Hypervisor Partition(*)\Deposited Pages
- \Hyper-V Hypervisor Partition(*)\Virtual Processors
- \Hyper-V Hypervisor Root Partition(*)\*
- \Hyper-V VM Vid Partition(*)\Physical Pages Allocated
- \Hyper-V VM Vid Partition(*)\Remote Physical Pages
- \Network Interface(*)\*
- \Hyper-V Virtual Switch(*)\*
- \Hyper-V Legacy Network Adapter(*)\*
- \Hyper-V Virtual Network Adapter(*)\*
- \Physical Disk(*)\Current Disk Queue Length
- \Physical Disk(*)\Disk Bytes / sec
- \Physical Disk(*)\Disk Transfers/sec
- \Hyper-V Virtual Storage Device(*)\*
- \Hyper-V Virtual IDE Controller(*)\*
(_Total) means only the total should be collected.
(*) means collect all counters.
\* means collect all the counters in the counter set.
\<name> means only collect that counter.
We have provided a custom monitor in VBScript that uses remote WMI and the available performance counters. The most important counters that we discussed in this article are included in this monitor. You can download this script from GitHub.