Extremely bad performance after upgrade from Netbackup 8.0 to 8.1.1

Hi

We upgraded our netbackup from 8.0 to 8.1.1 august 29th. I know that there is a new MSDP rolling conversion.
It's been 3 weeks since upgrade now and backups run like crap. We have about 200 servers, more then half of them does not been backuped for a week, every day loads of jobs fails due to passed backup windows.

I have read that "Backup can be slow during MSDP rolling conversion", but as I see it, backup does not run at all.
They start and gets Active and then nothing happen for days.

I have called support and the solution is "wait for the conversion to finish", OK. No backups for perhaps 1-2 month. Great
solution. I do not think it will gets done in 2-3 days like the Veritas support told me. MSDP size is 64 TB, and for 14 days conversion progress is now at 8%

All Netbackup 8.1.1 running on Windows 2012 servers.

No, I am not happy with this upgrade.

//Peter

Tags (2)
6 Replies

Re: Extremely bad performance after upgrade from Netbackup 8.0 to 8.1.1

Hi Peter,

We had a terrible time upgrading our appliances from 8.0 to 8.1.1 due to the conversion. My experiences were as follows:

  • As long as the conversion process was in its initial stages <20-40%, backups would perform very poorly. This is partially due to the fact that fingerprints for incoming data needs to be converted on the fly. The jobs would sometimes complete, but more often than not they would not finish.
  • We had a lot of VMware Accelerator jobs, which were failing in a large scale. This was caused by the issue above, which overloaded spoold and forced it to restart. 

Our actions to mitigate these problems were:

  • Disable any secondary SLP processing until conversion finished. 
  • Disable Accelerator for all VMware jobs, as the Accelerator proccesing is happening on the backup host (media server) as opposed to the client. 
  • Set the conversion mode to Fast, and let it run around the clock. Please note that if the spoold proccess crashes and restarts, then the conversion will default to normal again. 

We have an MSDP pool of 186GB, which finished after less than two weeks. 

I hope this might help you, its not an easy upgrade, but after doing the above actions we managed to get through it relatively unscathed.

Caspar

Re: Extremely bad performance after upgrade from Netbackup 8.0 to 8.1.1

Hi

Thanks for replying.

Yes, we have alot of accelator jobs too. I will try do disable it and se how it goes.
I have already change Secondary SLP processing and conversion mode to Fast.

I have installed a EEB patch that I have received from Veritas, hope that fix something.

https://www.veritas.com/support/en_US/article.100043461

Peter

 

Re: Extremely bad performance after upgrade from Netbackup 8.0 to 8.1.1

You're welcome Smiley Happy 

Please note that it will only be VMware (perhaps Hyper-V also, not sure) type Accelerator jobs where you will need to disable it. Disabling Accelerator on a normal MS-Windows or Standard policy will not have any effect as far as I see it as their processing happens on the client machines.

I wrote a basic script to mail me the dataconversion progress every hour, including the current conversion mode and the status for spoold. If I could see that the conversion mode was Normal and spoold had restarted since the last time I knew that action was required. I can see you are primarily running Windows, but you can probably convert it if you wish to use it. 

Caspar

Re: Extremely bad performance after upgrade from Netbackup 8.0 to 8.1.1

Hi Gents,

I am expierencing the exact same issue you have described below after upgrading from 7.7.1 to 8.1.1 ( NBU 8.1.1 running on Windows Server 2012r2) 

I have one master and 2 media servers in two different sites each media server hosts a msdp 64tb in size. 

I do have a support ticket open with Veritas but I firmly believe they are stringing me along hoping the conversion just finishes. Truly disgusted with this whole upgrade, if I could I would dump Veritas and go with another solution. I opened tickets with the Netbackup team, InfoScale team and my storage vendor ( since Veritas tried to suggest the issue could be with my storage array that is hosting all my msdp volumes).

Have tried some of the same tweaks and suggested changes you both have mentioned. Turning on fast mode and some other tweaks. Installing EEB patch (did not help).

I am approaching week 4 now and the conversion is still ongoing and I have backup and replication jobs failing daily. 

Unfortuanately, no other choice but to let this conversion finish. When that happens is anyones guess.

Hope you had better luck

Highlighted

Re: Extremely bad performance after upgrade from Netbackup 8.0 to 8.1.1

Hi how we know the status of msdp conversion? running background?

Re: Extremely bad performance after upgrade from Netbackup 8.0 to 8.1.1

Hi @jjimenez2 

Have you read through this section of the Upgrade Guide? 

About MSDP rolling data conversion
......
You can manage and monitor the rolling data conversion using the crcontrol
command. More information about its use is available.
See the NetBackup Deduplication Guide and the NetBackup Commands Reference Guide.