VM operations hang due to expired VSM service account password
search cancel

VM operations hang due to expired VSM service account password

book

Article ID: 316456

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
On vCenter Server 8.0U2, when attempting to perform VM operations, the operation hangs and completes after an unknown amount of time or fails to complete.
  • Examples of affected VM operations:
    • Powering on
    • Rebooting
    • Cloning
    • Deployment of VMs via OVA/OVF
  • In some cases an error message is shown saying  
    Failed to clone state for the entity '<VM_Template_Name>' on extension vService Manager

    for example:
 
image.png
  • In /var/log/vmware/vsm/vsm.log, you may see: 
    2023-11-02T23:11:57.450-04:00 INFO [Thread-4] ServiceUtil.java 137 - Acquiring SAML token for user [email protected]
2023-11-02T23:11:57.787-04:00 ERROR [Thread-4] SoapBindingImpl.java 185 - SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-vsm-758251d3-01bc-41a9-9269-c1b8287facba, Domain: vsphere.local} Please see the server log to find more detail regarding exact cause of the failure.
..
com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983) [libwstclient.jar:?]
      at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) [libwstclient.jar:?]
      at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:155) [libwstclient.jar:?]
      at com.vmware.vsm.utils.ServiceUtil.getSamlTokenForSvcAccount(ServiceUtil.java:138) [vsm.jar:?]
      at com.vmware.vsm.vc.VCenterListener.loadSamlTokenAndPrivateKey(VCenterListener.java:473) [vsm.jar:?]
      at com.vmware.vsm.vc.VCenterListener.initializeConnection(VCenterListener.java:268) [vsm.jar:?]
      at com.vmware.vsm.vc.VCenterListener.run(VCenterListener.java:301) [vsm.jar:?]
2023-11-02T23:11:57.843-04:00 INFO [Thread-4] ServiceUtil.java 142 - Password expired. Resetting service account password
 
  • OVF deployment may hang at power on 
In /var/log/vmware/vpxd/vpxd.log you will see events similar to below 

2024-02-21T18:34:51.721-06:00 info vpxd[27763] [Originator@6876 sub=VmProv opID=TxId: 0625c4dc-8450-4f08-aee2-12d5add051b0-47-01] P
owering on VM '[vsanDatastore] 6b96d665-ca86-9992-f3f8-043201173960/Interconnect-C3-IX-R1.vmx' on host <host fqdn/ip>
2024-02-21T18:34:51.725-06:00 warning vpxd[27763] [Originator@6876 sub=vmomi.soapStub[258] opID=TxId: 0625c4dc-8450-4f08-aee2-12d5a
dd051b0-47-01] SOAP request returned HTTP failure; <<io_obj p:0x00007fdab0421b10, h:182, <TCP '127.0.0.1 : 37532'>, <TCP '127.0.0.1
 : 15007'>>, /vsm/ovfConsumer/>, method: notifyPowerOn; code: 500(Internal Server Error)
2024-02-21T18:34:51.725-06:00 info vpxd[27763] [Originator@6876 sub=OvfConsumers opID=TxId: 0625c4dc-8450-4f08-aee2-12d5add051b0-47
-01] Failed to invoke OVF stub adapter, will re-try after login; N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.S
ecurityError
--> )
--> [context]zKq7AVECAQAAAIUcWQEsdnB4ZAAAxbVTbGlidm1hY29yZS5zbwAAUglDAIwxRACaSEsBoy8XbGlidm1vbWkuc28AAX6eJQFZICABOk4gAZ3HH4JJ8TUBdn
B4ZACCscNFAoKExUUCAX02GoNhyScBbGlidmltLXR5cGVzLnNvAIJGprsBgrXTuwGCNIHUAYLM9SACgn+wGwKC670kAoJI3SQCgkr9JAKCfQhiAoLPGGICgvkvYQKCUothA
oLJH9QBgk0M1QGChw3VAYJTDtUBgrIO1QGD5v4xAYK5d2MCAdXDG4IgokMCgn0IYgKCzxhiAoL5L2ECgpQCYgIA5ss3APkkOACTwFEEro4AbGlicHRocmVhZC5zby4wAAUv
3g9saWJjLnNvLjYA[/context]
  • At the same time in /var/log/vmware/vsm/vsm.log
2024-02-21T18:34:51.724-06:00 ERROR [pool-5-thread-1] VsmActivationValidator.java 267 - Failed to validate user: only vpxd-svc-acct requests allowed and not


Environment

VMware vCenter Server 8.0.2

Cause

The root cause of the problem is missing jar files from the classpath of the VSM service.

Resolution

This has been fixed in vCenter 8.0 U2b. 

VMware vCenter Server 8.0 Update 2b Release Notes

Workaround:
As a workaround, restart the VSM service to recreate the service account. 
  1. SSH to vCenter via root
  2. Restart the VSM service: 
    # service-control --restart vsm
OR 
  • Login to the VAMI of vCenter and restart the VMware Service Manager 

Note: Every time the VSM service is restarted, a new service account and password is created. 


Powered by Wolken Software