It demonstrated that by using DRS advanced options, we are able to balance memory usage across a cluster. In my next blog post, I examine memory consumption (usage) vs active memory (demand); and discuss the importance of understanding the difference and the impact on workload placement.
Key Terms
Active (Demand) - Amount of memory that is actively used, as estimated by the VMkernel based on recently touched memory pages.
- Virtual machine: Amount of guest “physical” memory actively used.
- Host: Sum of all active metrics for all powered-on virtual machines plus
services (such as COS,vSphere ) on the host.vpxa
Granted - The amount of machine memory or “physical” memory that is mapped for a virtual machine or a host.
- Virtual machine: Guest “physical” memory that is mapped to machine memory. Includes shared memory amount. Does not include overhead.
- Host: Sum of all granted metrics for all powered-on virtual machines, plus machine memory for
services on the host.vSphere
Consumed - Amount of memory consumed by a virtual machine, host, or cluster.
- Virtual machine: Amount of guest physical memory consumed by the virtual machine for guest memory. Consumed memory does not include overhead memory. It includes shared memory and memory that might be reserved, but not actually used. Use this metric for charge-back purposes.
consumed memory = memory granted - memory saved due to memory sharingvm - Host: Amount of machine memory used on the host. Consumed memory includes Includes memory used by the Service Console, the VMkernel,
services, plus the total consumed metrics for all running virtual machines.vSphere
consumed memory = total host memory - free host memoryhost
Usage - Memory usage as percentage of total configured or available memory.
- Virtual machine: Percentage of configured virtual machine “physical” memory.
÷ virtual machine configured sizeactive - Host: Percentage of available machine memory.
÷ machine-memory-sizeconsumed
Idle – Consumed memory that is not actively used.
DRS Migration Thresholds – The threshold set to automatically migrate virtual machines from one host to another to optimize resource usage.
- Level 1 – Apply only priority 1 migrations,
will only apply recommendations that must be taken to satisfy cluster constraints like affinity rules and host maintenance.vCenter - Level 2 – Apply priority 1 and 2 migrations,
will apply recommendations that promise a significant improvementvCenter theto load balance.cluster’s - Level 3 (default) – Apply priority 1, 2, and 3 migrations,
will apply recommendations that promise at least good improvement to thevCenter load balance.cluster’s - Level 4 - Apply priority 1, 2, 3, and 4 migrations,
will apply recommendations that promise even a moderate improvementvCenter theto load balance.cluster’s - Level 5 -
will apply recommendations that promise even a slight improvement to thevCenter load balance.cluster’s
Before moving virtual machines, DRS calculates the benefits of migrating the guest based on:
Resource Usage
- CPU usage
- CPU read
time - Active memory usage
- Entitlements based on limits, shares, and reservations
Cost
- CPU impact of
at sourcevMotion - Memory consumption of "Shadow VM" at
destination host - Memory reclamation impact
the destination hostat
Benefit
- Positive benefits to the virtual machines
the source hostat - Positive performance gain for the migrated virtual machines
the destination hostat - Overall load at source and destination
A great resource on DRS is Duncan Eppings and Frank Dennemans book VMware vSphere 5.1 Cluster Deepdive .
VMware Test Lab Environment
My VMware test lab environment consists of 10 virtual machines on 5 nested ESXi hosts:
Before I setup DRS, I moved several virtual machines around to make an unbalanced cluster from a memory usage standpoint.
In the below diagram, we
can see that vmt -esx02 is the most heavily loaded host
and vmt -esx03 is the least utilized.
DRS Migration Level 3 (Default) Test
By default, DRS uses active memory + 25% idle memory when calculating memory load balance. This is important to note, because in some of our tests we are going to change an advanced DRS setting to move to a memory usage model.
In my first test, I set DRS to fully automated with migration level 3 (moderate) using the default DRS setting; vCenter will apply recommendations that promise at least good improvement to the cluster’s load balance.
There were no recommendations and no virtual machine moves.
DRS Migration Level 5 Test
In my next test, I set DRS to fully automated with migration level 5 (aggressive) using the default DRS setting. DRS will apply recommendations that promise even slight improvement to the cluster’s load balancing.
There were two recommendations, DRS recommended moving VMT-W2K1210 and VMT-W2K1207 to other hosts.
2-hours after the migrations, the memory consumption levels were more evenly distributed on the VMware test lab hosts.
3-hours after the initial
moves, it moved VMT-W2K1206 to another host.
The memory isn’t quite as balanced, but still the load balance is much better than the original virtual machine placement and the unbalanced memory utilization.
12-hours after the migrations, there were no additional DRS moves, the workloads stayed fairly balanced, vmt -esx05 and vmt -esx03 have higher memory utilization.
DRS Migration Level 3 (Consumption) Test
Like mentioned earlier, DRS uses
active memory + 25% idle memory when looking at balancing workload in a
cluster. There is an advanced DRS option that can be set to use 100% of the
idle memory, this changes the DRS load balancing to use consumed memory for
load balancing recommendations. I used this advanced setting for two test
scenarios.
When I set DRS to fully automated with migration level 3 (moderate) using memory consumption, it recommended that we make two virtual machine moves. The recommendation was the same as the DRS Migration Level 5 (Default) Test.
2-hours after the migrations, the memory consumption levels were more evenly distributed on the VMware test lab hosts.
12-hours after the migrations, the memory usage appeared to be slightly more balanced.
DRS Migration Level 5 (Consumption) Test
For my last test, I set DRS to fully automated with migration level 5 (aggressive) using memory consumption, it recommended that we make three virtual machine moves as illustrated below.
2-hours after the migrations, the memory consumption levels were very evenly distributed on the VMware test lab hosts.
12-hours after the migrations, there were no additional moves, and the memory consumption stayed very balanced.
DRS was able to spread the workload usage and make a balanced cluster with 2 to 3 vMotions , this is a small lab environment, but it was very efficient in minimizing the amount of virtual machine moves when maintaining a cluster with balanced memory usage.
With the advanced DRS option, you can configure DRS to be more aggressive with load balancing by setting 50% idle memory, 75% idle memory, or going with the consumption model like we did in this post with 100% idle memory. For CPU, you can set the advanced DRS setting of AggressiveCPUActive = 1 to react to bursty CPU workloads.