VMware Tools Upgrade failed on Linux VMs

The other day I was working on a project updating vApp components and after successfully upgrading vCenter, ESXi hosts, and NSX Managers it was time to upgrade the VMware Tools for all the VMs. I powered on the VMs and noticed that 8 of the 10 linux machines failed with the same error as shown in Figure-1. I will be the first one to admit that my linux-ninja skills are subpar at best. When I saw this error message I was excited because this gave me a chance to learn some new linux commands and gain some much needed experience points to level up my linux-ninja skills.
1
Figure-1 Error upgrading Vmware Tools. Vix error code = 21009

I thought hmm that’s odd let’s take a look at the vmware-tools-upgrader.log file and see what’s going on. Typically when searching through log files I like to leverage egrep. Below is the command with some common verbiage typically found inside log files.

egrep -wi ‘warning|fail|failure|error|cannot’ /var/log/vmware-tools-upgrader.log

Egrep quickly returned results shown in Figure-2 and to confirm the current disk size I ran df-h on the server and noticed only 41M were available on the disk…YIKES, as shown in Figure-3. Then ran pvs to take a look at the physical volume and notice there is 0 space left to expand…OUCH shown in Figure-4. To get around this we need to create another physical volume. A useful command that can provide the current size of the disk is fdisk -l, notice in Figure-5 the disk size is 2GB. Before we proceed let’s take a look at the logical volumes and see how many we have on the disk using lvdisplay. As you can see in Figure-6, we have 2 logical volumes one is our OS/file (lv_root) and the other is our swap (lv_swap). In order to increase the disk size we need to first shutdown the VM. Right click on the VM -> Edit Settings -> Virtual Hardware -> Hard disk 1 and increase the size from 2GB to 3GB as shown in Figure-7. Once the change has been applied power on the VM and verify the disk size using fdisk -l as shown in Figure-8.
2
Figure-2
Cannot mkdir: No Space left on device
3
Figure-3
Checking available space on disk
4
Figure-4 Confirm physical volume free space
5
Figure-5 Current disk size
6
Figure-6
Show all logical volumes
7
Figure-7
Edit Settings for VM
8
Figure-8
Confirmation of increase in disk size

To begin the process of extending the disk space of this VM we need to issue the cfdisk /dev/sda command as shown in Figure-9 which launches cfdisk utility. Cfdisk is a partition editor and allows us to create both primary and logical partitions. In our case, we can see the newly added 1GB (1073.75MB) of disk space. Let’s create a new primary partition by highlighting the Free Space at the bottom -> New -> Primary -> accept default values -> press Enter. Once done notice in Figure-10 “sda3” appears. Next, lets write changes to disk by selecting Write -> read the warning -> type ‘yes‘. The VM will need to be rebooted in order for the changes to take affect.
9
Figure-9
Launch cfdisk utility
10
Figure-10
Create primary partition with cfdisk utility

On to creating a physical volume using pvcreate /dev/sda3 shown in Figure-11. If we run the pvs we should see the newly created physical volume shown in Figure-12. Let’s see the current volume group size by using vgdisplay as shown in Figure-13. Now to extend VolGroup using the newly created physical volume by issuing the command vgextend VolGroup /dev/sda3 see Figure-14. To confirm the size we again run the vgdisplay command shown in Figure-15.
11
Figure-11
Create new physical volume
12
Figure-12
Confirming physical volume creation
13
Figure-13
vgdisplay
14
Figure-14
Extend volume group VolGroup
15
Figure-15
After extending VolGroup

Now that VolGroup has been extended lets proceed with increasing lv_root. Remember when we ran lvdisplay see Figure-6, we saw two logical volumes lv_root followed by lv_swap attempting to extend lv_root in this current state will fail as shown in Figure-16 because lv_swap is for a lack of a better term “in our way”. So we need to disable and remove lv_swap by using the following commands shown in Figure-17:

  1. swapoff /dev/VolGroup/lv_swap
  2. lvm lvremove /dev/VolGroup/lv_swap

With lv_swap gone lets extend lv_root using lvextend -L+1G /dev/VolGroup/lv_root as shown in Figure-18
16
Figure-16
Insufficient free space
17
Figure-17
Disable and remove lv_swap
18
Figure-18
Extending logical volume lv_root

Great! Let’s confirm lv_root has been increased by using lvdisplay shown in Figure-19. Ok, now that we have extended lv_root all should be good, right? Not exactly because the file system isn’t aware of the increase we did in the previous step shown in Figure-20 using the df -h . To increase the file system let’s run resize2fs /dev/VolGroup/lv_root as shown in Figure-21 and again confirm by using df -h shown in Figure-22.
19
Figure-19
lv_root size increased
20
Figure-20
Current file system size
21
Figure-21
Resizing the file system
22
Figure-22
File system size increased

What has been described to this point might seem like a lot but once you have done this a few time it goes really quick  😉 . We are entering the final stretch just need to recreate the lv_swap we deleted earlier and activate it. As you can see in Figure-23 we issue the lvm lvcreate VolGroup -n lv_swap -L 200M command. Once done like everything else we have done up to this point we confirm lv_swap was created by using our favorite lvdisplay command shown in Figure-24. Awesome, lv_swap is created but we must not forget to activate it otherwise our swap cache won’t work. Let’s issue the following commands to ensure the lv_swap is functioning shown in Figure-25:

  1. mkswap /dev/VolGroup/lv_swap
  2. swapon /dev/VolGroup/lv_swap
  3. swapon -s

Hurray we successfully extended our disk inside our CentOS VM let’s do a quick cleanup of the temp directory using rm -rf /tmp/* shown in Figure-26 before attempting  the VMware Tools upgrade again. Success! We can now successfully upgrade VMware Tools on our CentOS VM as shown in Figure-27.
23
Figure-23
Recreate lv_swap logical volume
24
Figure-24
Confirmation lv_swap was created
25
Figure-25
Activate lv_swap cache
26
Figure-26
Cleaning up the temp directory
27
Figure-27
VMware Tools successfully upgraded

Command cheat sheet used to complete task:

 

 

1 comments On VMware Tools Upgrade failed on Linux VMs

  • I cam across the same error code when I tried to upgrade the VCSA 6.0 VM since it was flagged as having outdated tools. My problem wasn’t an issue with disk space but rather:

    VMware Tools cannot be installed, since they have already been installed using
    a package-based mechanism (rpm or deb) on this system. If you wish to continue,
    you must first remove the currently installed VMware Tools using the
    appropriate packaged-based mechanism, and then restart this installer

    This message was in the upgrader log file.

Leave a reply:

Your email address will not be published.

Site Footer

Sliding Sidebar

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.