Monday, May 9, 2016

DRS Advanced Settings

Recently, I performed a study on DRS, which focused on workload placement of virtual machines to load balance a cluster.

It demonstrated that by using DRS advanced options, we are able to balance memory usage across a cluster. In my next blog post, I examine memory consumption (usage) vs active memory (demand); and discuss the importance of understanding the difference and the impact on workload placement.

Key Terms

Active (Demand) - Amount of memory that is actively used, as estimated by the VMkernel based on recently touched memory pages.
  • Virtual machine:   Amount of guest “physical” memory actively used.
  • Host:  Sum of all active metrics for all powered-on virtual machines plus vSphere services (such as COS, vpxa) on the host.
Granted - The amount of machine memory or “physical” memory that is mapped for a virtual machine or a host.
  • Virtual machine:   Guest “physical” memory that is mapped to machine memory. Includes shared memory amount. Does not include overhead.
  • Host:   Sum of all granted metrics for all powered-on virtual machines, plus machine memory for vSphere services on the host.
Consumed - Amount of memory consumed by a virtual machine, host, or cluster.
  • Virtual machine:   Amount of guest physical memory consumed by the virtual machine for guest memory. Consumed memory does not include overhead memory. It includes shared memory and memory that might be reserved, but not actually used. Use this metric for charge-back purposes.
    vm consumed memory = memory granted - memory saved due to memory sharing
  • Host:   Amount of machine memory used on the host. Consumed memory includes Includes memory used by the Service Console, the VMkernel, vSphere services, plus the total consumed metrics for all running virtual machines.
    host consumed memory = total host memory - free host memory 
Usage - Memory usage as percentage of total configured or available memory.
  • Virtual machine:   Percentage of configured virtual machine “physical” memory.
    active ÷ virtual machine configured size
  • Host:   Percentage of available machine memory.
    consumed ÷ machine-memory-size
Idle – Consumed memory that is not actively used.

DRS Migration Thresholds – The threshold set to automatically migrate virtual machines from one host to another to optimize resource usage.
  • Level 1 – Apply only priority 1 migrations, vCenter will only apply recommendations that must be taken to satisfy cluster constraints like affinity rules and host maintenance.
  • Level 2 – Apply priority 1 and 2 migrations, vCenter will apply recommendations that promise a significant improvement to the cluster’s load balance.
  • Level 3 (default) – Apply priority 1, 2, and 3 migrations, vCenter will apply recommendations that promise at least good improvement to the cluster’s load balance.
  • Level 4 - Apply priority 1, 2, 3, and 4 migrations, vCenter will apply recommendations that promise even a moderate improvement to the cluster’s load balance.
  • Level 5 - vCenter will apply recommendations that promise even a slight improvement to the cluster’s load balance.

DRS Calculation

Before moving virtual machines, DRS calculates the benefits of migrating the guest based on:

Resource Usage
  • CPU usage
  • CPU read time
  • Active memory usage
  • Entitlements based on limits, shares, and reservations
Cost
  • CPU impact of vMotion at source
  • Memory consumption of "Shadow VM" at destination host
  • Memory reclamation impact at the destination host
Benefit
  • Positive benefits to the virtual machines at the source host
  • Positive performance gain for the migrated virtual machines at the destination host
  • Overall load at source and destination
A great resource on DRS is Duncan Eppings and Frank Dennemans book VMware vSphere 5.1 Cluster Deepdive.


VMware Test Lab Environment

My VMware test lab environment consists of 10 virtual machines on 5 nested ESXi hosts:


Before I setup DRS, I moved several virtual machines around to make an unbalanced cluster from a memory usage standpoint. 



In the below diagram, we can see that vmt-esx02 is the most heavily loaded host and vmt-esx03 is the least utilized.


DRS Migration Level 3 (Default) Test

By default, DRS uses active memory + 25% idle memory when calculating memory load balance. This is important to note, because in some of our tests we are going to change an advanced DRS setting to move to a memory usage model.

In my first test, I set DRS to fully automated with migration level 3 (moderate) using the default DRS setting; vCenter will apply recommendations that promise at least good improvement to the cluster’s load balance. 

There were no recommendations and no virtual machine moves.


DRS Migration Level 5 Test

In my next test, I set DRS to fully automated with migration level 5 (aggressive) using the default DRS setting. DRS will apply recommendations that promise even slight improvement to the cluster’s load balancing.


There were two recommendations, DRS recommended moving VMT-W2K1210 and VMT-W2K1207 to other hosts.



2-hours after the migrations, the memory consumption levels were more evenly distributed on the VMware test lab hosts.


3-hours after the initial moves, it moved VMT-W2K1206 to another host.


The memory isn’t quite as balanced, but still the load balance is much better than the original virtual machine placement and the unbalanced memory utilization.


12-hours after the migrations, there were no additional DRS moves, the workloads stayed fairly balanced, vmt-esx05 and vmt-esx03 have higher memory utilization.


DRS Migration Level 3 (Consumption) Test

Like mentioned earlier, DRS uses active memory + 25% idle memory when looking at balancing workload in a cluster. There is an advanced DRS option that can be set to use 100% of the idle memory, this changes the DRS load balancing to use consumed memory for load balancing recommendations. I used this advanced setting for two test scenarios.


When I set DRS to fully automated with migration level 3 (moderate) using memory consumption, it recommended that we make two virtual machine moves. The recommendation was the same as the DRS Migration Level 5 (Default) Test.


2-hours after the migrations, the memory consumption levels were more evenly distributed on the VMware test lab hosts.


12-hours after the migrations, the memory usage appeared to be slightly more balanced.


DRS Migration Level 5 (Consumption) Test

For my last test, I set DRS to fully automated with migration level 5 (aggressive) using memory consumption, it recommended that we make three virtual machine moves as illustrated below.


2-hours after the migrations, the memory consumption levels were very evenly distributed on the VMware test lab hosts.


12-hours after the migrations, there were no additional moves, and the memory consumption stayed very balanced.


DRS was able to spread the workload usage and make a balanced cluster with 2 to 3 vMotions, this is a small lab environment, but it was very efficient in minimizing the amount of virtual machine moves when maintaining a cluster with balanced memory usage.

With the advanced DRS option, you can configure DRS to be more aggressive with load balancing by setting 50% idle memory, 75% idle memory, or going with the consumption model like we did in this post with 100% idle memory. For CPU, you can set the advanced DRS setting of AggressiveCPUActive = 1 to react to bursty CPU workloads.
News: Top vBlog 2016 Trending: DRS Advanced Settings