RAM Disk Full

One problem I have run into frequently is esx.problem.visorfs.ramdisk.full. This is when the ESX RAM disks that make up the ESXi host file system reaches its resource pool limit and the host becomes unresponsive. The virtual machines are online and available to the business users, but you are no longer capable of managing the host. In a corporate environment, that means you have the choice of restarting the server outside of normal change windows after hours, which will cause an outage to the virtual machines; or leaving the host online until the next approved change window and taking the chance that if something happens to the virtual machine during production hours you won’t be able to fix the problem.

If you catch this problem early enough, you may be able to get to the host before it has becomes fully inaccessible. The server may struggle with vMotion activity when putting it into maintenance mode, but you can move the virtual machines and restart the host without affecting the business community until you can get the underlying issue resolved.

Two of the issues I have come across recently are:

1.  VMware ESXi 5.x host becomes unresponsive when attempting a vMotion or a configuration change - This issue occurs when SNMPD is enabled and the /var/spool/snmp folder is filled with Simple Network Management Protocol (SNMP) trap files. This issue is resolved in ESXi 5.1 Patch 04. Detail article explaining the entire process can be found here - http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2040707
2.  ESXi ramdisk full due to /var/log/HpHelper.log file size - File (located at /var/log/) grows excessively. Error Unable to connect to the MKS: To resolve this issue, HP Support has provided an updated hp-ams VIB to stop the excessive logging to the hpHelper.log file. Detailed article explaining the entire process can be found here - http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2055924

William Lam created a great article that details VMkernal Observations that can be used for creating alarms - Other handy vSphere VOBs for creating vCenter Alarms. We are going to create an alarm for RAM disk full. vCenter alarms can be specified for a particular object and it will apply to all the child-objects beneath in the vCenter server tree. In our instance, we are going to create an alarm to monitor a specific event that applies to all the ESXi hosts.

To start, we are going to go into the vCenter Client and go to the Alarms tab and click on the Definitions view. In the Definitions view, you are going to right click and create a New Alarm...

On the General tab, you are going to give your alert a name and description. You will set the Alarm Type to Host for Monitor, and you click on Monitor for specific events occurring on this object, for example, VM powered on. Finally, check Enable this Alarm.

Next you are going to want Add a trigger. The event is going to be esx.problem.visorfs.ramdisk.full with the Status of Alert.

On the Actions tab, this is where we are going to setup the action taken when the alert hits a threshold. In the image below, I set an e-mail notification to send me an alert when the alarm goes from yellow to red. I have it configured to alert me Once. After this is completed, if you are using a distribution list, then the Ram Disk Full alerts will go to everyone in the distribution list and you will have advanced notification of the impending problem.

One last item, to have vCenter Server send an e-mail for a triggered alarm, you are going to want to verify that the Mail Sender Settings have been configured in vCenter with the SMTP server. To configure the setting, from the vCenter Client you will want to go to Administration from the vCenter toolbar and select vCenter Server Settings. You will then select the Mail option on the left hand list. Supply the SMTP Server and the Sender Account information, so that when you receive the e-mail you know where it came from.