PBX goes down and EMM hangs in stopping state

[Android]

OK here's the scenario.  Two virtual machines running Windows 2003 server.  Baking up the VM's using option 3 (Full VM and File) causes the bpbkar32 process handles to jump over 2 million.  Immediately afterward, PBX goes down.  Then, EMM goes into a stopping state and hangs there.  The services cannot be restarted manually.  The only way to bring get NBU running again is to reboot the server. 

NBU master/media server is Windows 2003 server (32-bit).  This server is also the proxy server. 

[Android]

I should add that switching the VCB backup type from 3 to 1 allws the backups to complete without any issues or impact on the services.

[Anonymous]

Although it is supported, some best practices I have read recommend to separate the two apart  - 1 master server and 1 media/vcb proxy server.

You could make the master server a virtual machine itself.

First off recommend the following versions of software to be installed:

NetBackup 6.5.4
VMware VCB 1.5 (ESX 3.5U2 or latest 3.5U4) / VCB 1.5.1 (vSphere or ESX4)
vCenter Server 2.5U4

However, it would seem that the File part of the backup option is where it causes the server to go into process problems.
I would increase logging to level 5 on the master server host properties.
Create the %PROGRAMFILES%/Veritas/NetBackup/logs/bpbkar/ directory

rerun the backup and check back in this log directory.

Also troubleshoot by limit the jobs in policy to 1.

Type 3 FullVM with File for incrementals, it might be doing the indexing of the files inside the vmdk backup, where it creates all the bpbkar32 processes. It has done the full snapshot copy to the staging area by this point.

I would recommend you dont have any Anti-Virus software running and/or realtime scanning on this server or at the least disabled,

If you can use the SAN method in the snapshot client options.


Here is a handy flowchart that helps outline what is going on.

[Android]

I realize best practice is to have the media/proxy on a seperate physical server.  However this is not my environment and I can only suggest configuration changes.  Thus far I haven't had a great response to that suggestion. 

The strange thing is there are more than a hundred VM clients all using VCB that use this same proxy server without any issues.  Alot of them are Windows 2003 VM's that are using option 3 as well.  They backup every night without any issues. 

There are two clients that experience this behavior amongst many that do not.

[Anonymous]

Are VMware Tools installed in these two VM's and status OK in VI Client Summary page?

Also some relative technotes to the bpbkar process for a fullvm.

DOCUMENTATION: NTFS file system and system memory considerations for VMware proxy server used by Net...

Each instance of bpbkar that does a Mapped FullVM backup is going to allocate a bitmap in memory for the purpose of mapping disk extents:

-  The larger the NTFS file system used as the snapshot mount point, the more the memory requirement for the bitmap.

-  The smaller the allocation size for NTFS file system (specified during file system creation) the more the memory requirement for the bitmap.

As per the current design, 1 Gigabytes of memory can map an NTFS file system of size 4.2TB. However, this space is just for the bitmap alone from one bpbkar process.



EXIT STATUS 36: VMware VCB backups: ERR - can't allocate bitmap

GENERAL ERROR: VMware VCB Type 3 incremental backup schedules or Type 0 backups fail with Status 156...

GENERAL ERROR: VMWare Consolidated Backup type 2 or 3 backups fail to index files level information,...

BUG REPORT: A VMware Consolidated Backup (VCB) can hang or fail reporting status code 11 (system cal...