The case of the flapping VMware Secure Token Service

sick-vmwareSo after upgrading to vCenter 5.5.0b we encountered a problem where the VMware Secure Token Service would not stay started. It would start and then immediately fail. Some initial poking around lead to looking at the STS logs in C:\ProgramData\VMware\CIS\runtime\VMwareSTS\logs. After checking the catalina log for the current date (catalina.2014-03-21.log) I noticed a bunch of SEVERE errors like the following:

SEVERE [WrapperSimpleAppMain] org.apache.coyote.AbstractProtocol.init Failed to initialize end point associated with ProtocolHandler [“http-bio-7080”]

This error was in the vpxd log:

Unable to create SSO facade: No connection could be made because the target machine actively refused it.

And finally a few java errors: Address already in use: JVM_Bind <null>:7080

Staring at those errors lead me to remember where I’ve seen “7080” before. Long ago vCenter Converter Standalone was installed on the system and during its configuration port 7080 was selected. As it turns out this port is needed in order for the Secure Token Service to run but its nowhere to be found in the Required ports for vCenter 5.5 KB article. You can check what ports are being used by vCenter Converter by looking at the XML located at use C:\ProgramData\VMware\VMware vCenter Converter Standalone\converter-server.xml and drill down to the proxySvc\ports\http node.

Stopping the vCenter Converter services and/or changing the port resolves this issue. This probably wont be true for most of you so look for any services using port 7080 (netstat -abn might help).

vCenter Operations Manager IP Pool Error

Following the vCenter Operations Manager 5.8.1 installation and deployment guide leads you to a notice that in order for the deployment of the vApp to work properly you must create an IP Pool and associate it to the portgroup where the vApp is to be connected. IP Pools are created at the Datacenter level. After creating the pool and deploying the app I was all set to power up the vApp. At power on an error was returned:

Cannot initialize property ‘vami.netma-sk0.VM_1’. Network has no associated network protocol profile.

Googling this error will lead you to a few places where its mentioned the issue is that you did not create an IP Pool. The problem was I did in fact create this pool. What the issue turned out to be was we have multiple dVswitches for different clusters that have the same portgroup names. Even though I triple checked the correct portgroup where the vApp was located did indeed have an IP Pool associated this did not rectify the error. The fix was to go back to the IP Pool configuration section, right click on the pool, and edit the properties. Once inside go to the associations tab and select all portgroups that have similar names.

Another quick fact about this IP Pool is you do not need to select Enable IP Pool inside of the pool settings. This checkbox option is only necessary if you intend to specify a range of IP’s.

Warning to ESXi 5.5U1 upgraders

Warning_esxi55Just a few reminders out there for those looking to upgrade to ESXi 5.5U1 from anything that is not 5.5. Keep in mind with this version that VMware removed drivers for devices that are not on the HCL. This includes a few NICs like Realtek and Marvell and possibly a few SATA controllers. In order to prevent you from this disaster the best way to accomplish the upgrade is using the profile update esxcli command. Details to follow soon!