Nutanix : CVMs are running out of space (sic!)


Background

Trying to upgrade to the latest AOS, NCC, Hypervisor .... but you are running out of space on your cluster and pre-checks are failing ? There is a fix !

Let's do some cleanup

[Edit Oct 25th, 2021] : Nutanix recently issued a script that is dealing with this exact issue. This is not by default installed in the CVM and it is fully explained in KB1540 available here. It only works on AHV. Use the below with caution !

Logs are taking a lot of space, same for temp files and old binaries. Those simple commands can be ran on each of the CVMs to save space :

nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       9.8G  5.2G  4.1G  56% /
devtmpfs         14G     0   14G   0% /dev
tmpfs            14G  4.0K   14G   1% /dev/shm
tmpfs            14G  696K   14G   1% /run
tmpfs            14G     0   14G   0% /sys/fs/cgroup
/dev/sda3        40G   23G   17G  59% /home
tmpfs           2.8G     0  2.8G   0% /run/user/1000
/dev/sdd1       917G  238G  670G  27% /home/nutanix/data/stargate-storage/disks/9XG843J5
/dev/sdc1       917G  248G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG86131
/dev/sdb1       917G  249G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG8619W
/dev/sde1       917G  248G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG8618V
/dev/sda4       308G  148G  157G  49% /home/nutanix/data/stargate-storage/disks/BTTV451306A6400HGN


In the above example, /home has 17 Gb of available space. Let initiate the cleanup.


nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ allssh rm -rf /home/nutanix/data/cores/*
================== 192.168.8.183 =================
================== 192.168.8.184 =================
================== 192.168.8.185 =================
nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ allssh rm -rf /home/nutanix/data/binary_logs/*
================== 192.168.8.183 =================
================== 192.168.8.184 =================
================== 192.168.8.185 =================
nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ allssh rm -rf /home/nutanix/data/ncc/installer/*
================== 192.168.8.183 =================
================== 192.168.8.184 =================
================== 192.168.8.185 =================
nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ allssh rm -rf /home/nutanix/data/log_collector/*
================== 192.168.8.183 =================
================== 192.168.8.184 =================

================== 192.168.8.185 =================


Note : this tip and trick has been provided directly by the Nutanix Support !

After the cleanup, here is the new available space :

nutanix@NTNX-15SM15xxxxxx-A-CVM:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       9.8G  5.2G  4.1G  56% /
devtmpfs         14G     0   14G   0% /dev
tmpfs            14G  4.0K   14G   1% /dev/shm
tmpfs            14G  696K   14G   1% /run
tmpfs            14G     0   14G   0% /sys/fs/cgroup
/dev/sda3        40G   22G   18G  55% /home
tmpfs           2.8G     0  2.8G   0% /run/user/1000
/dev/sdd1       917G  238G  670G  27% /home/nutanix/data/stargate-storage/disks/9XG843J5
/dev/sdc1       917G  248G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG86131
/dev/sdb1       917G  249G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG8619W
/dev/sde1       917G  248G  660G  28% /home/nutanix/data/stargate-storage/disks/9XG8618V
/dev/sda4       308G  148G  157G  49% /home/nutanix/data/stargate-storage/disks/BTTV451306A6400HGN

We have a gain of 1 Gb. Hoora ! ;)

Time to deploy the latest firmware !

If ever this is not sufficient to install the new firmware, you'd better have to contact the Nutanix support directly, you may have missed some gigantic files in the way that could be deleted or moved elsewhere. Always contact the Nutanix support at first, especially on the commercial product. This trick also apply to the Community Edition.

Comments

  1. Thank you! Your posts concerning nutanix CE are quite useful.

    ReplyDelete
    Replies
    1. My pleasure, I just realized that I have a lot of comments on almost every articles. I should read the "moderate section" a little bit more often ;)

      Delete

Post a Comment

Thank you for your message, it has been sent to the moderator for review...

What's hot ?

ShredOS : HDD degaussing with style

ThingSpeak : Create some useful formulas