Rubrik : Bug on AHV Export




Background

When we had an issue on an AHV VM, the first reflex was to restore it from last night backup. It was very quick and we were all happy ... unless we realized that no networking was connected. We opened a case with Nutanix to troubleshoot the issue and ....

We have found a bug !

After a couple of investigation and forensic, we have discovered the below phenomenon.

Here is a sample of the faulty VM with the original VLAN order :

Here is the exported VM VLAN order :


We can see the difference, the VLAN ID order is inverted. It was not very obvious at first, but when looking at the ARP table on the hypervisor managing that VM, we have seen that MAC address were inverted.
Now, we need to understand what's wrong and how to solve it. Indeed, what's wrong ? The export itself ? or the backups ? If this is the backups it means all our backups are corrupt since a long time and this is serious.

Other option, it can be an issue when the newly exported VM is created. In that case, a hotfix can definitely help.

At this stage we can be sure we have found a bug and not a small one. It could affect other customers with different VLANs created the same way as we did.

We have involved Rubrik support and it has been escalated to Engineering department. After some times reviewing the code they confirmed there is indeed a "sort" instruction on the export module that should be removed. They managed to correct the code, do the Q/A and we have been delivered with a hotfix. The internal issue number is CDM-271956.

The hotfix has been installed today (2nd Nov 2021) and the exact CDM version number is 6.0.1-12877_HP798644

The hotfix can only be applied by Rubrik support and it will be incorporated into any CDM release starting from 6.0.2.

I can confirm the hotfix is indeed solving our issue on restoring AHV VMs with multiple vNIC. The AHV version does not matter, this is not an issue on the Nutanix side.

I hope this is also helping other customers.




Comments

What's hot ?

RClone : Mount Google Drive on any File System