he other day one of my co-workers had some issues logging on to vCAC. It would accept his logon credentials but would just refresh the page and ask to logon again. Seeing similar issues in the past on vCenter web logon it seemed that it was an SSO issue. I decided to have a look at the identity appliance. Everything looked fine on the identity appliance. I then headed over to the vCAC appliance. I could see that a couple of the web services visible on the vCAC appliance web logon screen were in a failed state.
Going over the log files it was very clear what the problem was…
javax.xml.ws.soap.SOAPFaultException: Invalid credentials
I then used JXplorer to test my SSO credentials and they are all good.
Had a look at “email@example.com” user in the LDAP browser to check if the account was locked or expired and they all looked in order, no lockout or expired account.
I know what you thinking…. “Just re-register the vCAC appliance to the SSO server” believe me I tried. I just kept on getting the same error in the log files. Strange thing is that during the process of re-registering the vCAC appliance back to the SSO server it accepts the credentials, with an a error message. “Unable to establish a connection to “SSO-Appliance.domain.local:7444” (that’s not my actual servers DNS details).
A week ago my colleague replaced all the certificates with signed certificates using our internal CA. What a mission that was. There doesn’t seem to be any good documentation or good guides on how to do this. The couple we were able to find were helpful but needed you to at least have a good understanding about certificates and how they worked. I think I should do a step-by-step guide for this, anyway getting back to the problem. After he replaced them everything worked as expected, for about a week. I created blueprints, deployed some new virtual machines without any issues.
So I don’t think it could be the certificates. I have gone over all them a couple of times and they look fine.
Decided to do a full restore of both appliances. If this doesn’t work I guess I will get VMware support to have a look at it.
Will update this post when I eventually fix this.
UPDATE – Problem solved.
I decided to open a case with VMware tech support. The engineer tried to assist but turns out our version of vCAC 6.0.1 has reach EOL support. I actually didn’t even realize that this version of vCAC only had support for 13 months. Needles to say VMware tried to assist but eventually told me to update before they would continue with the support request.
I decided to go over everything again before I updated. I noticed that when listing the internal solution users within the keystore on the vCAC appliance, there was something wrong with internal solution users certificates. Some seemed to be expired. “BTW when asked to enter a password just hit enter.”
vcachost01:/etc/vcac # keytool -list -keystore vcac.keystore
cafe, Mar 28, 2016, PrivateKeyEntry,
Certificate fingerprint (SHA1): CA:16:72:1F:BD:E8:2B:82:FE:56:85:8D:92:F5:DE:A7:BE:6D:B5:5D
csp-admin, Mar 28, 2016, PrivateKeyEntry,
Certificate fingerprint (SHA1): C7:C1:17:E9:15:22:06:C0:3B:12:DB:15:43:0E:B0:7D:EA:01:A5:44
websso, Apr 09, 2016, trustedCertEntry,
Certificate fingerprint (SHA1): B4:78:36:CE:E5:C9:74:8F:B8:3B:24:9B:BB:C0:FB:32:8C:62:F2:F9
apache, Mar 29, 2016, PrivateKeyEntry,
Certificate fingerprint (SHA1): 7B:C0:DC:80:78:09:EB:49:90:72:60:AD:61:5D:32:C3:B0:B7:60:3E
This led me to believe that this could be the issue. So this is what i did to resolve it. Not sure if this is supported by VMware but hey, product was EOL and I really did not feel like rebuilding the environment on a Friday.
Perform these steps on the vCAC appliance.
You should see the internal solution users that have expired certificates. In my case these were the culprits.
I needed these solution users to recreated with new certificates. Fortunately when you register vCAC with the SSO identity appliance, these solution users are created if they are missing.
Before editing any files make a copy of the file.
vcachost01:cp /etc/vcac/solution-users.properties /etc/vcac/solution-users.properties.bkp
Now edit the file :
Remove the solution users with expired certificates and save your file.
Now re-register the vCAC appliance back to the identity appliance using the command:
vcac-config -e register-with-sso –tenant vsphere.local –user firstname.lastname@example.org –password ‘yourpass’
Once this command has executed without any errors your vCAC appliance should be up and running again.
I have decided to plan an update to vRA 7.x in the next couple of weeks and will be doing a detailed upgrade guide.
Please feel free to leave a comment or any suggestions.