Vmware backup failling with status code 6

[Abhisheknetback]

Dear All,

One of the my vm backup is failling with status code 6 but in the same policy rest of backup is completing successfully . jobdetails are below and some log are also attached kindly assist on this .

Enviornment Details :-

Master and media Server windows 2008 R2,

netbackup 7.5.0.5 on master media

vmware backup is configured through vcenter

Thank You

6/4/2014 11:43:08 PM - Info nbjm(pid=8476) starting backup job (jobid=429485) for client WFBZAEHO1, policy VMWARE_VM_VCAEHO_GROUPC_CBT_PRD, schedule DAILY-INCR  
6/4/2014 11:43:08 PM - estimated 15346348 Kbytes needed
6/4/2014 11:43:08 PM - Info nbjm(pid=8476) started backup (backupid=WFBZAEHO1_1401910988) job for client WFBZAEHO1, policy VMWARE_VM_VCAEHO_GROUPC_CBT_PRD, schedule DAILY-INCR on storage unit medhoae3_ho_stu1
6/4/2014 11:43:10 PM - started process bpbrm (4864)
6/4/2014 11:43:11 PM - Info bpbrm(pid=4864) WFBZAEHO1 is the host to backup data from     
6/4/2014 11:43:11 PM - Info bpbrm(pid=4864) reading file list from client        
6/4/2014 11:43:12 PM - Info bpbrm(pid=4864) starting bpbkar32 on client         
6/4/2014 11:43:12 PM - connecting
6/4/2014 11:43:12 PM - connected; connect time: 00:00:00
6/4/2014 11:43:13 PM - Info bpbkar32(pid=11200) Backup started           
6/4/2014 11:43:13 PM - Info bpbkar32(pid=11200) CONTINUE BACKUP received.          
6/4/2014 11:43:13 PM - Info bptm(pid=8820) start            
6/4/2014 11:43:13 PM - Info bptm(pid=8820) using 524288 data buffer size        
6/4/2014 11:43:13 PM - Info bptm(pid=8820) setting receive network buffer to 2098176 bytes      
6/4/2014 11:43:13 PM - Info bptm(pid=8820) using 256 data buffers         
6/4/2014 11:43:18 PM - Info bptm(pid=8820) start backup           
6/4/2014 11:43:19 PM - begin writing
6/5/2014 12:18:42 AM - Info bptm(pid=8820) waited for full buffer 1 times, delayed 136103 times    
6/5/2014 12:18:42 AM - Info bptm(pid=8820) EXITING with status 90 <----------        
6/5/2014 12:18:42 AM - Info bpbkar32(pid=11200) bpbkar waited 0 times for empty buffer, delayed 0 times.   
6/5/2014 12:18:45 AM - Error bpbrm(pid=4864) could not send server status message       
6/5/2014 12:18:46 AM - Critical bpbrm(pid=4864) unexpected termination of client WFBZAEHO1        
6/5/2014 12:18:46 AM - Error bpbrm(pid=4864) cannot send mail to braju@unb.ae,Satish.K.Ajmera@unb.ae,DataCenter@unb.ae,criticala@unb.ae,omukthar@unb.ae        
6/5/2014 12:18:46 AM - end writing; write time: 00:35:27
the backup failed to back up the requested files(6)

[Nathan_Kippen]

Have you tried suggestions in following tech article:

http://www.symantec.com/docs/HOWTO44491

[Mark_Solutions]

From the bpbkar log the operation timed out:

12:20:43.779 PM: [3872.5456] <2> tar_base::V_vTarMsgW: INF - Snapshot BLIB type: ON

12:55:46.752 PM: [3872.5456] <4> tar_backup::backup_done_state: INF - number of file directives not found: 0

12:55:46.752 PM: [3872.5456] <4> tar_backup::backup_done_state: INF - number of file directives found: 1

12:55:46.752 PM: [3872.1492] <4> tar_base::keepaliveThread: INF - keepalive thread terminating (reason: WAIT_OBJECT_0)

12:55:46.752 PM: [3872.5456] <4> tar_base::stopKeepaliveThread: INF - keepalive thread has exited. (reason: WAIT_OBJECT_0)

That 35 minute gap caused the failure.

It may be a resource issue or maybe something hung - so you could increase the timeouts but possible better to try and see why nothing happened for 35 minutes - same for the one in your post, just different times!

[Abhisheknetback]

Dear Mark ,

When i unchecked the enable block -level incremental backup option on vm-ware tab in policy .

backup completed succesfuly , and with this option backup is keep on failling ,and  in the same policy rest of the client backup completing sucessfuly with same option.

for your information, schedule  is configured for difrential incremental .

kindly suggest

Thank You

[Nathan_Kippen]

This might be the issue:

http://www.symantec.com/docs/TECH188895

Solution

Solution 1

To increase the Virtual Center SessionTimeout perform the below operation: (Note the below increases the timeout to 60 minutes, higher values maybe needed)

1. Stop the VMware VirtualCenter Server service.
2. Open the vpxd.cfg file using a text editor.
   1. The vpxd.cfg file is located on the VMware vCenter Server by default at %ALLUSERPROFILE%\Application Data\VMware\VMware VirtualCenter\vpxd.cfg
      1. On Windows Server 2008, this would generally be C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg
      2. On Windows Server 2003, this would generally be C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg
3. Add the following <soap> elements within the <vmacore> </vmacore> tags:
   <soap>
   <!-- Minutes before idle sessions time out -->
   <sessionTimeout>60</sessionTimeout>
   </soap>
4. Start the VMware VirtualCenter Server service.

Solution 2

Check the virtual machine for orphaned snapshots.

[Mark_Solutions]

Odd that it is just thos one client - it is as if it is struggling to read the CBT information.

So has it ever worked?

Is that client configured with CBT on every disk?

Is it the same vmx version as the other servers that do work?

Is it a BIG client - I am thinking that if it is very large or has a lot of changes going on that this client needs more time to read the CBT information and so need its timeouts increasing.

[Abhisheknetback]

Dear Friends,

Sorry for dealy response ,

I removed the same client from policy and readd it so now backup is runing fine .

Thanks for your all support