Just Because You Can, Doesn’t Mean You Should

First, I haven’t been ignoring my duties to this blog by not posting articles, the time at my job has kept me away from posting regularly scheduled articles but it hasn’t stopped me from producing some great topics for future articles, this being one of these.

ESXi5.5 Purple Screen of Death = Oh My!

In our quest to migrating our environment to ESXi 5.5 and View 5.3 we had to do some maintenance on some of our file servers, the quick and dirty was to build massive TB VMDK’s for robocopy jobs as we migrated to newer File Servers. Part of this process included kicking off Veeam backups of these temporary File Servers. During the course of the reverse incremental job (multiple rounds of robocopy!) we encountered some PSOD’s (purple screen of death) on the temporary cluster where the file servers were located.

Seeing as this was my first experience with a PSOD, yes I know I’m so lucky! I proceeded to establish a SR, support request, with VMware and Veeam. Then I began retracing my steps, trying to understand if it was a misconfiguration, something I didn’t enter correctly, etc. The cause of the host failures surprised me, considering it is such a big selling feature of ESXi5.5. What we discovered is that when you have a VMDK that is larger than 1.9TB, snapshots are in the SESparse format. There can be a memory heap issue related to ESXi5.5 that can cause host failure. So when we kicked off our Veeam Backups, Veeam uses the VMware Snapshot model for backups, the snapshot files were in the SESparse format and after 45 minutes the hosts failed.

The resolution from VMware was to reduce all VMDK’s to below the 1.9TB threshold and to wait patiently for the release of a upcoming patch to ESXi5.5 coming in July. Which brings me to my final thoughts, I have flashbacks of Jerry Springer: Just because you can, doesn’t mean you should.

Just Because You Can, Doesn’t Mean You Should

One of the big selling points for me about ESXi5.5 was the support for larger than 2TB VMDK’s think of the possibilities that this could bring an organization: Large File Servers, Exchange Datastores, SQL Databases, etc. But why? Why would you want to subject yourself, your company to the risk of placing all of your important files on one big drive, why not spread that risk out across multiple stores, servers, etc. It flies in the face of KISS, keep it simple stupid, that my friend Brad Christian constantly reminds me of!

So going forward, tread cautiously with each new feature a software comes out with, it may be great on paper, but does it really fit for your organization, your initiatives, your systems?


vSphere Virtual Machine Management Book Review

vSphere Virtual Machine Management

Ask anyone in this industry and they will tell you those pivotal moments in their careers when they were inspired, mentored, coached by a select few that sling-shot them to great times. I have been fortunate to have several of those people that I got to work with or mentored by that have pushed me in the right direction in my career. I met Rebecca Fitzhugh while attending my vSphere 5.1 Install, Configure and Manage required classroom training in 2012. She was a delight to have as a teacher not only because of her vast knowledge on the topic of vSphere and vCenter, but also because she taught more than the manual and showcased real world scenarios, that we all know so well aren’t normally covered in these types of classes although VMware does a better job than other companies! At the conclusion of the class we exchanged information knowing that there would be opportunities to network in the future. Fast forward two years, Rebecca and I continue to network and she pushes me to pursue higher-level learning from VMware. Because of her suggestions, I have attained my VCA-Cloud, VCA-WM, VCA-DCV, VCP-DCV and VCP-DT. I am also currently studying for my VCAP-DTA and DCA. She truly has been one of those people that have inspired me to push further in my career and reinforce good values when working with my clients.

Rebecca approached me at VMworld 2013 in San Francisco about a book she was writing about vSphere Virtual Machines, I was intrigued about this great opportunity for her and pressed her to keep me in the loop should an opportunity come up where I could partially return the favor for the knowledge she gave me. I am proud to be writing this review for her, because I know the knowledge that is in this book has already been put to use by me and will be by countless others. The book itself comprises of 326 pages and 10 chapters and is a nice balance of introduction, technical and deep dive with regards to vSphere VM’s.

The book covers a lot of ground for all aspects of a VMware Admin from new to experienced. The author gives a thorough explanation of what a virtual machine is and the components that make one up, how to create virtual machines, provisioning, creating templates and how to configure resource pools. This is a great book for any admin that interacts with VMware vSphere and vCenter. I particularly appreciated the fact that most of the concepts in this book are current as of vSphere and vCenter 5.5. I highly recommend this book for any die-hard VMware person or someone brand new to the software suite. Grab a copy today and see for yourself!

As a special treat to those are interested, for a limited time, if you tweet the following you can receive a free copy of her book!

Learn how to optimize your virtual machine using this new guide http://bit.ly/1jTgF53  by @RebeccaFitzhugh @VMwarevSphere #OVF #VMware


Build Custom ESXi 5.5 ISO using PowerCLI


Here at work we are about to embark on a complete redesign of our virtual infrastructure from the ground up. As I’ve mentioned before with our IOPs issues with View, we went from PoC to Production without scaling all resources evenly. So starting from scratch means redoing everything, all the way down to standardizing ESXi ISO images. I have been attending vBrownBag’s (Professional VMware#AutomateAllThings  video session about how automation can help with everyday tasks to overhauling your environment.

Why not automate the creation of ESXi images for my various clusters? Great idea since each cluster has different NIC’s, Drivers, Graphics Cards, etc. so let’s throw all of my VIB’s into my image as opposed to doing it manually like I did during my vSGA evaluation.

To start you will need a few things to get started:

  • Vanilla ZIP instance of ESXi 5.5
  • All VIB’s that you want to load into your image
  • PowerCLI 5.5
  • MS PowerShell v2 (Running Windows 7 or 8 it will be installed)
  • Free Space on your Computer (kinda obvious but you never know!)

Now that we have everything let’s assume you have already installed PowerShell and PowerCLI on your workstation and are ready to start, if you haven’t here are some helpful links to aid you in installing those packages. Install PowerShell. Install PowerCLI. You probably received a prompt during the installation of PowerCLI talking about execution policy issues may arise, fear not we will take care of that right now. Navigate to your Start Menu or Desktop and locate the PowerShell icon, right click and select “Run as Administrator”. When PowerShell opens, run the following command:

Set-ExecutionPolicy RemoteSigned

Your screen should look something similar to this:

Screen Shot 2014-03-26 at 3.42.03 PM

You can close PowerShell since the remainder of the work will be done in PowerCLI, run PowerCLI (always run CLI as administrator) and you should see this:

Screen Shot 2014-03-26 at 3.43.29 PM

First let’s create a small folder structure to organize our custom ISO’s, me personally I am going to be building 3 ISO’s for three separate cluster configurations. I also wrote my scripts in Notebook++ so I can look at the individual lines for each function.

Screen Shot 2014-03-26 at 3.46.30 PM

Here is the framework for the script that I will run:

Get-PSSnapin -Name VMware.ImageBuilder

Add-EsxSoftwareDepot C:\ESXi_Depot\VMware-ESXi-5.5.0-1331820-depot.zip

Add-EsxSoftwareDepot C:\ESXi_Depot\Server_Cluster\igb-

Add-EsxSoftwareDepot C:\ESXi_Depot\Server_Cluster\ixgbe-3.18.7-offline_bundle-1327252.zip

Add-EsxSoftwareDepot C:\ESXi_Depot\Server_Cluster\NVIDIA-VMware-x86_64-319.65-1OEM.550.0.0.1331820-bundle.zip

Get-EsxImageProfile | Format-Table -AutoSize

New-EsxImageProfile -CloneProfile ESXi-5.5.0-1331820-standard -Name ESXi-5.5.0.ServerCluster.Customised -AcceptanceLevel PartnerSupported

Export-EsxImageProfile -ImageProfile ESXi-5.5.0.ServerCluster.Customised -ExportToIso -FilePath C:\ESXi_Depot\BOXX_Cluster\ESXi-5.5.0.ServerCluster.Customised.iso

Let’s go through each step of this process and explain what each function does.

Get-PSSnapin -Name VMware.ImageBuilder

Get-PSSnapin is the call function to bring VMware Image Builder into PowerCLI if you don’t have it already loaded in your instance of PowerCLI you will need to run this command.

Add-PSSnapin VMware.ImageBuilder

Add-EsxSoftwareDepot is the call to add my ESXi image (ZIP format) and then all of my VIB’s to my image, each VIB requires a separate line.  After you run this command you will get a spit of like this:

Screen Shot 2014-03-27 at 10.46.46 AM

Once you have added all of your VIB’s to the depot then we need to look at the actual ESXi image we introduced to the image builder and determine which profile we want to use, that is the intent of the Get-EsxImageProfile function. You will get a table of the profiles to choose from:

Screen Shot 2014-03-27 at 10.51.07 AM

Next we use the New-EsxImageProfile to clone the profile and name it what we want!

Screen Shot 2014-03-27 at 1.16.28 PM

Finally we run the Export-EsxImageProfile function to compile our depot of VIB’s to produce our custom ESXi ISO as shown here:

Screen Shot 2014-03-27 at 1.18.27 PM

Now I have my custom image that I can load to my hosts when I do my complete virtual infrastructure redesign.

Big thanks to vStrong.info for his article on VMware Image Builder and Alan Renouf on getting PowerCLI setup.

Veeam ONE Monitor Free Edition Review


As 2014 is about to start, December is always a time to have internal IT meetings on how to improve processes, workflows and responsiveness. This year our IT department experienced positives and negatives in those categories from power outages to unexpected server down times. The big takeaway from 2013 was being a more proactive team when it comes to our virtual system and resolving problems before they become downtime scenarios.

So as I set out to do research (read: Google) on proactive monitoring solutions for our virtual infrastructure I came across several good candidates like SolarWinds, Xangati, vCOPS and Veeam. I think money grows on trees, my bosses think otherwise so I decided to deploy VeeamONE Free Edition to see if it cut the mustard for what we wanted and if the free version is an actual solution or just a digital carrot dangling in front of the budget waiting to be purchased.

There are a few differences/limitations to the free version, I have highlighted them below, now that we have that covered let’s find out if VeeamONE is really free and usable.

Veeam ONE Monitor Free vs Paid


Installation was quite easy to stand up. Built out a VM with the necessary “hardware” requirements on a Windows 2008 R2 x64 box. Everything was pretty much next, next, next, except for a restriction we have on SQL creations, that was an easy fix. We simply ran the CREATE script that came packaged with the installer, ran it in SSMS and the database was created. Verified the default ports and linked VeeamONE to our primary vCenter server and assigned some users to the Users and Admin groups. Reboot once and everything came back up just fine.


Configuration was a breeze, there are two types of roles, Admins and Users, biggest difference I can see is Users are limited to what changes can be altered to events where as Admins have higher control. Email notifications are limited to the canned responses in the Free edition but is sufficient for what we wanted. Setup your SMTP settings and go!  With notifications you can include known KB articles with a specific issue, this is a helpful step for your lower tier Help Desk guys if they don’t live on VMware’s KB site like I do! If you have a broader SNMP capture system, VeeamONE links up nicely. On to views and the dashboard.

Views and Dashboard

This is where I personally think VeeamONE shines, you have 3 views to choose from: Infrastructure, Business and Data Protection.  In the Free Edition, Data Protection is unavailable as it relates to Veeam Backup & Replication and Enterprise Manager for a higher level view of your environment as it relates to your data integrity, this is a view we would like, but completely understand why Veeam left it out of the free edition. The Infrastructure View is where I live, it gives me a complete breakdown of my vCenter environment separated by Datacenter, Clusters, Hosts, VM’s, Resources and Datastores. As referenced in the free vs. paid chart, some notifications are limited but it is still a ton of information to get you closer to resolution.

My favorite is the Dashboard view, we are a VDI shop at so I built out a Kiosk Mode VM that auto loads the VeeamONE Client in full screen mode (pictured above) that gives me a dedicated station in my office to turn around and focus on a specific problem or event. Lately we have been testing some VDI users on a certain Synology DS3612XS with SSD’s (article coming soon!), it’s been nice to see statistics on performance and be alerted if the datastore spikes with latency.


VeeamONE Free Edition is a great compliment to your vCenter environment and has helped up isolate issues that we weren’t even looking for. Veeam has done a great job giving a lot of functionality in a free edition, there are some limitations that will make us seriously consider the paid version (Management likes reports!). But with some knowledge in PowerShell and PowerCLI vCheck can help with this! We have only had it up and running for 2 weeks and I obsessively knock out all the events that come across my inbox from the notification system. It has made us think twice about issues before diving in. I would highly recommend standing up the free version in your environment, what do you have to lose but a little more pro-activeness and maybe a different view on your vCenter environment!

VeeamONE Monitor Free Edition Link

Manually Remove vCenter Server from SSO


I was playing around in our test lab with Linked Mode vCenter Servers last week and ran across an error in the vSphere Web Client after I removed the second vCenter Server. The specific error I got was: Could not connect to one or more vCenter Server Systems: vcenter address:443/sdk

My first guess was that the uninstallation was successful but that SSO had held onto some remnants of the second vCenter so it needed to be manually unregistered with the Lookup Service. Here is what I did to get everything fixed.

Credit Due: Mark Almeida-Cardy at vi-admin.net has a great article about how to resolve this with vCenter 5.1 so I will use his post with updates for vCenter 5.5

VMware has a KB Article 2033238 that lays out the steps for vCenter 5.1 as well.

For Windows: <SSO install directory>\ssolscli\ssolscli listServices <Lookup Service URL>

For vCenter Server Appliance: /usr/lib/vmware-sso/bin/vi_regtool listServices <Lookup Service URL>

  1. In the list of services, locate the service entry that contains the address of the system where the solution was installed.
  2. Record the ownerId of the service entry.
  3. In the vSphere Web Client, navigate to Administration > SSO Users and Groups > Application Users and locate the application user with the same name as the ownerId you recorded.
  4. Right-click the user and select Delete Application User.
  5. At the command line, remove the service entry from the Lookup Service.
    1. Create a text file that contains the service ID.
      The service ID must be the only text in the file.
  6. Unregister the entry for the solution by running the unregisterService command.Note: It be necessary to Set your JAVA_HOME environmental variable (default jre location below).
    set JAVA_HOME=c:\program files\vmware\infrastructure\jre

For Windows: <SSO install directory>\ssolscli\ssolscli unregisterService -d <Lookup Service URL> -u “Lookup Service administrator user” -p “administrator password” -si <serviceId file>

For vCenter Server Appliance: </usr/lib/vmware-sso/bin/vi_regtool unregisterService -d <Lookup Service URL> -u “Lookup Service administrator user” -p “administrator password” -si <serviceId file>

Script I Used: ssolscli.cmd listServices https://VCENTER FQDN:7444/lookupservice/sdk > C:\sso_services.txt

Output txt file looked like this: 

Intializing registration provider…
Getting SSL certificates for https://VCENTER FQDN:7444/lookupservice/sdk
Anonymous execution
Found 15 services.

Service 1
serviceName=VMware vCenter Support Assistant, WebClientPluginPackage
endpoints={[url=https://IP ADDRESS:8443/plugin/package/ph-admin-ui.zip,protocol=http]}


Next let’s identify the services that we need to unregistered and copy/past the serviceId’s into another txt file (remember the name and location of this file)

Now we can run our unregistered script, mine looked like this: ssolscli unregisterService -d https://VCENTER FQDN:7444/lookupservice/sdk -u “LOOKUP SERVICE USERNAME” -p “PASSWORD” -si <FILE LOCATION>

Here is the result that I got:

C:\Program Files\VMware\Infrastructure\VMware\CIS\vmware-sso>ssolscli unregister
Service -d https://VCENTER FQDN:7444/lookupservice/sdk -u “LOOKUP SERVICE USERNAME” -p “PASSWORD” -si C:\sso_services.txt
Intializing registration provider…
Getting SSL certificates for https://VCENTER FQDN:7444/lookupservice/sdk
Service with id “Default-First-Site:cdda2053-438a-439d-95aa-b47081f94e42” is successfully unregistered
Service with id “Default-First-Site:31d628f8-60e7-4955-a9aa-fd3e3a24bb31” is successfully unregistered
Service with id “Default-First-Site:6e21c57b-da61-460b-b6b2-ef82a3647dad” is successfully unregistered
Return code is: Success 0

Second instance of vCenter has been remove and no error on start up of vSphere Web Client….Happy Admin!


Veeam B&R v7 Support for vSphere 5.5 is here!


Veeam has released Update 2 for Backup & Replication v7 today with full support for vSphere 5.5 and Hyper-V 2012 R2 (the first backup company to support both!), here is a breakdown of all the new and exciting features:


  • vSphere 5.5 support, including support for 62TB virtual disks and virtual hardware v10 virtual machines.
  • vCloud Director 5.5 support.
  • Support for Windows Server 2012 R2 and Windows 8.1 as guest virtual machines (VMs).
  • Added ability to limit maximum amount of active VM snapshots per datastore to prevent it from being overfilled with snapshot deltas. The default value of 4 active snapshots can be controlled with MaxSnapshotsPerDatastore (REG_DWORD) registry key.


  • Windows Server 2012 R2 Hyper-V and free Hyper-V Server 2012 R2 support, including support for Generation 2 virtual machines.
  • Support for Windows Server 2012 R2 and Windows 8.1 as guest virtual machines (VMs)
  • Support for System Center 2012 R2 Virtual Machine Manager (VMM)
  • Support for the installation of Veeam Backup & Replication and its components on Windows Server 2012 R2 and Windows 8.1.

Built-in WAN acceleration

  • Increased data processing performance up to 50% with hard drive based cache, and up to 3 times with SSD based cache. Multi-core CPU on source WAN accelerator is recommended to take full advantage of the enhanced data processing engine.


  • Added ability for source and target proxy servers to reconnect and resume replication when network connection between source and target site drops for a short period of time.


  • Added support for a number of enterprise-class tape libraries with partitioning functionality that allows presenting multiple tape library partitions to the same host.
  • Import/export slot interaction has been redesigned to add support for a number of IBM and Oracle tape libraries.

Application-aware processing

  • Added ability for application-aware processing logic to detect passive Microsoft Exchange DAG database present on the VM, and process it accordingly.
  • Added support for Exchange CCR clusters.

User interface

  • User interface should now remember size and positions off the main window, as well as all panels and columns.

I am proud to say that we upgraded our Backup Environment this morning and everything is running great. Big thanks to Veeam and their Engineering Team for releasing this so fast!

Here is a link to the KB article for this Update.

vSphere 5.5 Upgrade – Walkthrough and Notes

Since the announcement of vSphere 5.5 at VMworld, I have been waiting for the day to upgrade our environment. That day has finally come. We decided to go with the Simple Install since there isn’t anything custom when we install vCenter.

We have Horizon View in our environment, so the first step was to shutdown all workstations, linked-clones, etc and disable provisioning to not create additional clones. We shutdown all servers except our Firewall, Exchange, Primary File Server and vCenter servers (Apps and DB’s). Now that we have our environment in a controlled state, we are ready to start the upgrade process.  So lets get to it!

Download the vCenter 5.5 Install Package for Windows from VMware, mount the ISO or extract the contents to a temp location and select autorun.exe

Slide0Select the Simple Install Option, review the upgrade process and click Install to start the upgrade of Single Sign-On.


It has already detected the previous version, Click Next


Accept the End User License Agreement, who really reads that stuff! Click Next


Make sure that the FQDN, IP is accurate. vCenter Windows Server is joined to the Domain and Value is correct, Click Next


It has detected the previous instance of SSO so it will migrate the existing information, Click Next


Choose the local default domain account credentials for SSO, Click Next


Confirm the Site Name, Click Next


Choose the install location, Click Next


Here is an overview of the choices we made, time to Install


It will take a little bit of time to install, when it completes it will proceed to start installing vSphere Web Client components.


vSphere Web Client begins to install and register with SSO. Next up is the Inventory Service.


Pretty simply choice, keep your existing inventory or stay up all night and rebuild it. We chose to get some sleep tonight! Click Next


It’s ready to install the Inventory Services, Click Install


Install the components and watch the progress bar. It will immediately go into vCenter Upgrade next.


Click Next


Enter License, Click Next Slide3 \

Enter your SQL Database credentials for vCenter Database, Click Next


It prompts you that vSphere Update Manager isn’t compatible with 5.5 (we will upgrade that after we are done), Click OK


Upgrade the existing vCenter database, check the box, Click Next


Choose Automatic, Click Next


Unless you want a Domain Account to auto login, choose a separate login account for the vCenter Service Account.


Validate your ports, you have the option to increase the ephemeral ports. We didn’t increase them because we will not hit 2000+ VM’s before vSphere 6 comes out, Click Next


Everybody has a different take on this option, we chose Large so that we don’t have to hit an artificial limit on JVM Memory issues, to each his own, Click Next


Time to Install, Click Install


The database upgrade took a while…




vCenter has finished the upgrade process, are we done? Not just yet!

Time to install the Windows vSphere Client, just select the Client install option and click Install.


Next through the options since there isn’t any customization to apply.


vSphere Client is updated, to sign in!

So at this point we went to fire up vCenter Windows Client to start upgrading Update Manager but ran into an error where our AD accounts wouldn’t authenticate to vCenter. After a little research (Google!) we found this article from VMware. We have a AD Admin group nested into the Local Admin group, upgraded vCenter doesn’t like this.

It was an easy fix, log in using local admin to vSphere Web Client, go to vCenter permissions and add Domain Admin group (users to manage vCenter), then you are back in business. Now it’s time to upgrade vSphere Update Manager so we can start upgrading ESXi Hosts.


Choose vSphere Update Manager and Click Install


In the Installation has detected our existing version of Update Manager, Click OK


Click Next


Allow Update Manager to download baselines, patches and others after install, Click Next


Verify your vCenter Information, remember that vCenter account we entered earlier? Time to enter it again, it will validate and Click Next


SQL Connection Info, Click Next


Choose Yes I want to upgrade my VUM Database!, Click Next


Verify Port Settings, Click Next


Click Install


Watch the progress!


Installation is complete time to fire up the vSphere Windows Client to install the plugin.


Login using your credentials or use Windows Credentials. Notice the warning in the new version, you MUST use Windows VIC to manage Update Manager, Site Recovery Manager or an isolated Host.

Now we need to load vSphere Update Manager into our vSphere Client so navigate to Plug-ins > Manage Plug-ins


Choose Download and Install under the Status Column, It will fire up the Client Install


Click Next


Accept the End User License Agreement, Click Next


Click Install


Watch the progress!


Click Finish


Now we see that vSphere Update Manager has been installed and is “Enabled” in vSphere Client, time to start upgrading hosts!

First we need to create a Baseline to see which hosts are out of compliance. So let’s start there.

From the Home view select Update Manager from Solutions and Applications


Create a new Baseline


Choose a Name and select Host Upgrade for your baseline type, Click Next


Choose you ESXi Image, if you do not see the ESXi image in the list, go to Admin View > ESXi Images tab to import the ESXi 5.5 image, Click Next



Verify your settings and Click Finish, now we can run our baseline again the hosts we want to upgrade. For now we are only upgrading our Production Server Hosts because we are waiting on Teradici to update the Firmware of the APEX 2800 PCoIP Offload Cards for vSphere 5.5


Final Thoughts

So we simulated almost everything before we did the upgrade in Production, aside from the Nested AD Group, we didn’t hit any snags. I hope this walk through has been beneficial. If you have any questions or comments please post them.

vSphere 5.5 Upgrade – Ready, Aim, Fire!

I was hoping to have my DFW VMUG User Conference Download article finished today but I am waiting on my slide decks to put the final pieces together. So look for that article in the next few days, spoiler alert: the User Conference was awesome!

I want to upgrade vSphere 5.5 all night long!

Tonight we will be upgrading our vCenter environment to 5.5. As our team was going through our checklist I ran across some good tidbits of information, a lot of which came from my previous post. The two biggest items so far have been the which order to upgrade the components and to make sure there would be any hiccups during the upgrade. The component upgrade process we are going to follow is based on KB Article 2057795. Here is the upgrade order I plan to take:

5.5 Upgrade

So far we are aware of one issue going into the upgrade, there is a Single Sign-On issue related to AD authentication when running SSO on Windows 2012, you can find the reference here. I’m glad we caught this because we were planning to migrate our vCenter server onto 2012. There is a fix, but will wait to do the OS upgrade for another night.

I will be following up with my notes on how the upgrade process went.

Wish me luck!

RTFM or KB’s – vSphere 5.5

So vSphere 5.5 is out and you are all pumped to get it loaded into your production environment. I know I am! There are a few important articles that you should check out before you start down the 5.5 road. Rick Blythe has compiled an awesome list for us to reference! Enjoy!

PS: Thanks Wayne Conrad for the link on Google+

Links inserted from: http://blogs.vmware.com/kb/2013/09/vsphere-5-5-is-here-kbs-you-need-to-know-about.html

The first grouping contains absolute ‘must know’ information, the second grouping gets a bit more into details.

Also not to be missed: