Select Page

Using Thycotic Secret Server to Rotate VMware ESXi Root Passwords

Using Thycotic Secret Server to Rotate VMware ESXi Root Passwords

One of my security related projects this coming year will be to reduce (remove?) the usage of identical passwords across similar devices. For example, how many of you utilize the same root password for ESXi servers or the same password for the local administrator account on servers and desktops? We do, and so had every place I’ve worked for previously. In a time of almost daily breach announcements, it’s no longer sufficient to maintain this concept moving forward. We fixed it using Thycotic’s Secret Server.

For us, Secret Server was originally looked at as a centralized password management tool for our team who, at the time, were all using different products installed on their local workstations. Although we got by, it wasn’t an ideal solution and proved to become an issue at times, sparking the search for a solution. We quickly found that Secret Server also has tons of other features, one of which is managing and remotely rotating credentials for both service accounts and a devices local accounts. This had been a goal of mine since I started and after a week long proof of concept, we were impressed and purchased the product.

The following demonstrates how to configure Secret Server to rotate a ESXi server’s root password.

password rotation

1. Using the Discovery feature of Secret Server, import the ESXi servers you want to manage. For me, I’m going to choose both of the ESXi servers in our lab. You can find this under Admin –> Discovery.

  1. Create a new Discovery Source, choosing the type as “VMware ESX/ESXi Discovery Source” and follow through the wizard. Make sure to add a secret so Secret Server may authenticate properly and read the list of current users.
  2. Run Discovery by hitting the Run Now button.
  3. Enter the Discovery Network View page by clicking the Discovery Network View button.
  4. On the left, expand the Discovery Source you created and click on a server. In the right pane, you should see a list of user accounts available on the server. If the status displays “Not Scanned”, click the Rescan icon next to the server name.
  5. Select the root account and click Import – *Note: Do not change the VPXUSER account as it is the account vCenter uses to communicate with the host. This password is already random and is rotated by default.
  6. Follow the Import wizard and choose the desired actions. This will create a new secret in the selected folder in which you can manage and modify later as shown below.

new secret for lab002

 

2. Navigate to the newly created secret and choose Edit, opening the screen that you see above. Here you can see the details of the current account including the password and it’s expiration. In this case the VMware ESX/ESXi Secret Template has an expiration of 30 days which this secret has inherited.

  1. Click on the Remote Password Changing tab.
  2. To change the password immediately, click the Change Password Remotely button. Type in a new password or Generate a new one. Click Change.
  3. As shown by the vSphere Client’s Recent Tasks, Secret server has modified the root password as requested. The new password can be found by viewing it within the secret.
  4. You may also view Secret Server’s Remote Password Changing logs under Admin –> Remote Password Changing Log.

changed secret

 

3. Once you’ve added all your ESXi hosts using the same Secret Template, they will automatically be changed when the password expires as configured. Scheduling the frequency of a Secret’s expiration is done within the Secret Template while the time/date of the actual password change can be configured using the AutoChange Schedule. If you leave the AutoChange Schedule set to none, the password change will be queued for a change immediately after the expiration, assuming AutoChange is enabled. For more information on definitions on Expiration, AutoChange, and the AutoChange Schedule, including examples, check out Thycotic’s KB article here.

  • Secret Template – If you wish to modify the frequency of the expiration, you’ll want to change the value for Expiration Days in the template; navigate to  Admin –> Secret Templates. Choose your template and modify it accordingly. This will apply to all secrets utilizing this template.
  • AutoChange Schedule – If you wish to modify the actual time/day that the password will change, say Saturday at 2am, utilize the AutoChange Schedule feature.
  • AutoChange – As noted in the graphic below, be sure to enable AutoChange for your secrets. This can be done per secret, as noted below, or in bulk by selecting all desired secrets in the browser widget and selecting “Enable AutoChange” under the bulk operation drop down menu below.

enable autochange

 

Last but not least, if you ever get in a bind with your ESXi’s root account and you’re unable to retrieve it’s password, you can use this post to safely reset the host’s root account. Also, be sure to check out one of our general vSphere security posts here.

 

follow

About The Author

Bryan Krausen

Bryan Krausen is currently working as a Technical Architect with experience in a vast number of platforms.

Bryan has been active within the VMware vExpert community for several years and is the leader of the Louisville VMware User Group (VMUG) and Louisville AWS User Group.

3 Comments

  1. Great blog article.. I’m curious how the discovery works. Are they expecting SSH to be enabled on the ESXi server? Also, for your readers, seriously consider using Active Directory accounts on your ESXi servers and logging in as your AD account and not root. Why? Because then an audit trail will be left showing who did what. It’s kinda hard to pin down the “who” when using a generic account like root. To do that, you’d need Secret Server to send a log to a SIEM or Log Insight saying “Mike Foley just claimed the root account for ESXi Host “foobar””. Only then would you be able to associate what Mike did with the root account. That’s too much work when you’re digging through logs for forensic evidence. If someone from Secret Server wants to reach out to me to discuss how it interfaces to ESXi I’d appreciate that. mfoley at vmware dot com.

    Reply
    • btkrausen

      The discovery scans work by either manual discovery by entering name/IP or scanning a subnet using SSH Discovery. I used manual discovery since we only have about 30-35 hosts.

      Regarding the audit trail, Secret Server has extensive auditing built right into the product. It allows you to quickly pinpoint who added, modified, displayed, or copied a password. See the screenshot below that shows the options for reporting.

      Reply
  2. The idea of not having the same password stored across multiple systems is a solid design goal, one that’s been part of existing mgmt protocols such as SNMPv3 since 2002 as any attack that gets access to one system should not be used to gain access to related systems.

    Reply

Leave a reply

Your email address will not be published.