VM Policy backup failing on one media server. Error 196

[stwali005]

All,

I have a Policy setup up called LN1-VMware setup up. Recently, half of the jobs are failing with a 196 error. Obviously looking into the error its an out of window error - however, some of them are working, some are not. And in the logs, the snapshots are created within a minute and then they dont do anything until the next morning when the window closes.

I have checked the window and it is run from 9pm to 6am the next day which should be enough time to complete.

The issue seems to be with one of the media servers. Someof the vm's are successfully backed up on NB02, but the ones with NB01 selected all fail with 196.

I have logged into that server and all seems fine - other policies are backing up on that media server. The only issue i can see is when i run the nbemmcmd command from NB01 i get an error "Failed to inisitalize EMM connection" Running the same command on the master server they can see NB01 and NB02 fine.

I just cannot find any difference in the setups to work out why some of my VM's are backing up to media server NB02 successfully - but dont even seem to start on NB01.

Is there a way to force this policy to use NB02 media server? Or any suggestions of where i can look?

Many thanks

[RiaanBadenhorst]

Hi,

The error on NB01 indicates there is something wrong with that media server and its not talking to the master. Have you restarted the services on NB01? Do that and rerun the nbemmcmd -listhosts -verbose command and past it here.

If you want to send the backups to NB02 only then you should look at the storage unit (group) that is configured in the policy. Currently its probably using a group so change it the policy so it uses the NB02 storage unit only.

If you like please post your policy output (bppllist "policy name" -U) and bpstulist -U

[Marianne]

Are the jobs Active at the point where snapshot is taken? Or queued?

I would expect to see indication in Job Details that the job was originally queued as well as the reason for the job being queued.

Please post all text in one of these failed jobs.

What nbemmcmd command are you running from NB01?

Pease run these 2 commands on the master and show us the output:

nbemmcmd -listhosts -verbose

nbemmcmd -getemmserver

[stwali005]

Successfyl VM Backup for Policy LN1_VMWare - pointing to nb02

11/07/2014 19:16:23 - Info nbjm(pid=8696) starting backup job (jobid=1507318) for client xxxxx, policy LN1_VMWare, schedule Weekly_Backup  
11/07/2014 19:16:24 - estimated 20424195 Kbytes needed
11/07/2014 19:16:24 - Info nbjm(pid=8696) started backup (backupid=xxxxx_1405102583) job for client xxxxx, policy LN1_VMWare, schedule Weekly_Backup on storage unit ln1emcdd01-su
11/07/2014 19:16:25 - started process bpbrm (11436)
11/07/2014 19:16:26 - Info bpbrm(pid=11436) xxxxxx is the host to backup data from     
11/07/2014 19:16:26 - Info bpbrm(pid=11436) reading file list from client        
11/07/2014 19:16:27 - Info bpbrm(pid=11436) starting bpbkar32 on client         
11/07/2014 19:16:27 - connecting
11/07/2014 19:16:27 - connected; connect time: 00:00:00
11/07/2014 19:16:29 - Info bpbkar32(pid=7548) Backup started           
11/07/2014 19:16:29 - Info bpbkar32(pid=7548) CONTINUE BACKUP received.          
11/07/2014 19:16:29 - Info bptm(pid=8528) start            
11/07/2014 19:16:29 - Info bptm(pid=8528) using 262144 data buffer size        
11/07/2014 19:16:29 - Info bptm(pid=8528) setting receive network buffer to 1049600 bytes      
11/07/2014 19:16:29 - Info bptm(pid=8528) using 64 data buffers         
11/07/2014 19:16:33 - Info bptm(pid=8528) start backup           
11/07/2014 19:16:34 - begin writing
11/07/2014 19:17:02 - Info bpbkar32(pid=7548) INF - Transport Type = san       
11/07/2014 19:24:20 - Info bptm(pid=8528) waited for full buffer 22756 times, delayed 28759 times    
11/07/2014 19:24:21 - Info bpbkar32(pid=7548) bpbkar waited 0 times for empty buffer, delayed 0 times.   
11/07/2014 19:24:32 - Info bptm(pid=8528) EXITING with status 0 <----------        
11/07/2014 19:24:33 - Info bpbrm(pid=11436) validating image for client ln1vaaa01        
11/07/2014 19:24:34 - end writing; write time: 00:08:00
the requested operation was successfully completed(0)

[stwali005]

Failed backup on Policy LN1_VMWare - times out - does not have a media server assigned. Error 196

11/07/2014 19:01:04 - Info nbjm(pid=8696) starting backup job (jobid=1507213) for client xxxxx, policy LN1_VMWare, schedule Weekly_Backup  
11/07/2014 19:01:04 - Info nbjm(pid=8696) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1507213, request id:{EA8D1A9A-7284-4343-A212-309771DCEF89})  
11/07/2014 19:01:04 - requesting resource xxxxddxx-su
11/07/2014 19:01:04 - requesting resource nb.domain.com.NBU_CLIENT.MAXJOBS.xxxx
11/07/2014 19:01:04 - requesting resource nb.domain.com.NBU_POLICY.MAXJOBS.LN1_VMWare
11/07/2014 19:01:05 - Info nbrb(pid=1468) Limit has been reached for the logical resource nb.domain.com.NBU_POLICY.MAXJOBS.LN1_VMWare    
client backup was not attempted because backup window closed(196)

Backup still running, so terminated by myself

11/07/2014 19:01:34 - Info nbjm(pid=8696) starting backup job (jobid=1507283) for client xxxxxx, policy LN1_VMWare, schedule Weekly_Backup  
11/07/2014 19:01:34 - estimated 2274432 Kbytes needed
11/07/2014 19:01:34 - Info nbjm(pid=8696) started backup (backupid=ln1vvsm02_1405101694) job for client xxxxxxx, policy LN1_VMWare, schedule Weekly_Backup on storage unit xxemcddxx-su
11/07/2014 19:01:36 - started process bpbrm (11880)
11/07/2014 19:01:37 - Info bpbrm(pid=11880) xxxxx is the host to backup data from     
11/07/2014 19:01:37 - Info bpbrm(pid=11880) reading file list from client        
11/07/2014 19:01:37 - Info bpbrm(pid=11880) starting bpbkar32 on client         
11/07/2014 19:01:37 - connecting
11/07/2014 19:01:37 - connected; connect time: 00:00:00
11/07/2014 19:02:22 - Info bpbkar32(pid=9756) Backup started           
11/07/2014 19:02:22 - Info bpbkar32(pid=9756) CONTINUE BACKUP received.          
11/07/2014 19:02:22 - Info bptm(pid=13208) start            
11/07/2014 19:02:22 - Info bptm(pid=13208) using 262144 data buffer size        
11/07/2014 19:02:22 - Info bptm(pid=13208) setting receive network buffer to 1049600 bytes      
11/07/2014 19:02:22 - Info bptm(pid=13208) using 64 data buffers         
11/07/2014 19:02:26 - Info bptm(pid=13208) start backup           
11/07/2014 19:02:27 - begin writing
14/07/2014 09:30:57 - Critical bptm(pid=13208) sts_close_handle failed: 2060046 plugin error        
14/07/2014 09:31:08 - end writing; write time: 2 14:28:41
termination requested by administrator(150)

[stwali005]

I ran those commands you requested but i am restricted to upload anything from work.

-getemmserver completed successfully and found all the hosts i would expect - including nb01 and nb02. We also have lncnb01 and lncnb02.

-listhosts - verbose returned returned a long successfull list - is there anything specific you are looking for from these commands that i can type up for you?

thanks

[RiaanBadenhorst]

ok, question, can you back something else using NB01? Even back some files located on NB01 to NB01.

Test that and report back

[stwali005]

There is a few successful backups on NB01 yes. However i must admit - not nearly as nuch as NB02 and NBC01 and NBC02. Not sure why this is the case though.

14/07/2014 20:00:00 - Info nbjm(pid=8696) starting backup job (jobid=1515648) for client ln1xxxxx, policy LN1_Physical, schedule Daily_Backup  
14/07/2014 20:00:00 - Info nbjm(pid=8696) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1515648, request id:{2BADFB0F-924A-4225-A3F3-3942077F997B})  
14/07/2014 20:00:00 - requesting resource ln1dd-su
14/07/2014 20:00:00 - requesting resource lncvnbdomain.com.NBU_CLIENT.MAXJOBS.ln1xxxxdomain.com
14/07/2014 20:00:00 - requesting resource lncvnbdomain.com.NBU_POLICY.MAXJOBS.LN1_Physical
14/07/2014 20:00:00 - granted resource lncvnbdomain.com.NBU_CLIENT.MAXJOBS.ln1xxxdomain.com
14/07/2014 20:00:00 - granted resource lncvnbdomain.com.NBU_POLICY.MAXJOBS.LN1_Physical
14/07/2014 20:00:00 - granted resource MediaID=@aaaab;DiskVolume=ln1xxxddxx-lsu;DiskPool=ln1xxxdd01-dp;Path=ln1xxxdd01-lsu;StorageServer=ln1xxxdd01.domain.com;MediaServer=ln1nb01.domain.com
14/07/2014 20:00:00 - granted resource ln1xxxdd01-su
14/07/2014 20:00:06 - started
14/07/2014 20:00:07 - estimated 7717383 Kbytes needed
14/07/2014 20:00:07 - Info nbjm(pid=8696) started backup (backupid=ln1xxx.domain.com_1405364406) job for client ln1xxx.domain.com, policy LN1_Physical, schedule Daily_Backup on storage unit ln1emcdd01-su
14/07/2014 20:00:08 - started process bpbrm (13340)
14/07/2014 20:00:14 - Info bpbrm(pid=13340) ln1xxx.domain.com is the host to backup data from     
14/07/2014 20:00:14 - Info bpbrm(pid=13340) reading file list from client        
14/07/2014 20:00:14 - connecting
14/07/2014 20:00:17 - Info bpbrm(pid=13340) starting bpbkar32 on client         
14/07/2014 20:00:17 - connected; connect time: 00:00:03
14/07/2014 20:00:19 - Info bpbkar32(pid=11756) Backup started           
14/07/2014 20:00:19 - Info bptm(pid=12664) start            
14/07/2014 20:00:20 - Info bptm(pid=12664) using 262144 data buffer size        
14/07/2014 20:00:20 - Info bptm(pid=12664) setting receive network buffer to 1049600 bytes      
14/07/2014 20:00:20 - Info bptm(pid=12664) using 64 data buffers         
14/07/2014 20:00:23 - Info bptm(pid=12664) start backup           
14/07/2014 20:00:24 - Info bptm(pid=12664) backup child process is pid 14696.14992       
14/07/2014 20:00:24 - Info bptm(pid=14696) start            
14/07/2014 20:00:24 - begin writing
14/07/2014 20:00:56 - Info bpbkar32(pid=11756) change journal NOT enabled for <C:\>       
14/07/2014 20:02:34 - Info bpbkar32(pid=11756) change journal NOT enabled for <D:\>       
14/07/2014 20:04:04 - Info bptm(pid=12664) waited for full buffer 2519 times, delayed 13918 times    
14/07/2014 20:04:04 - Info bptm(pid=12664) EXITING with status 0 <----------        
14/07/2014 20:04:05 - Info bpbrm(pid=13340) validating image for client ln1xxx.domain.com       
14/07/2014 20:04:07 - end writing; write time: 00:03:43
the requested operation was successfully completed(0)

[stwali005]

My Last post hasn't been approved?

It is a strange one becuase i can see SQL servers being backed up successfully using the NB01 media server. But i must admit the number of backups is way less then that of NB02.

I am logging a call with our 3rd party vendor - hopefully i can get a webex session and get to the bottom of this.

Thanks

[RiaanBadenhorst]

hi,

Sorry i didn't see your post earlier. What I wanted to determine is whether the media server was working. Can you try a new policy with just a single vm as a test. Make sure to set the backup host to 01, and also the storage unit.

Are you using a storage unit group in your policy? Why is it going to both 02 and 01?

[stwali005 · Accepted solution]

Morning All,

Yesterday i found out from our network team that we had an issue with one of our switches. the switch / ports are connected to our netbackup, and even though it should have failed over to the other channel - it didn't seem to work all the time. This is probably why i was seeing VMware fail on nb01 but some backups backup fine on NB01. (it was intermittent failures)

The network team had resolved the issue yesterday and the backups ran 100% last night. Some inter-team communication would have been nice :)

Many thanks all for your help with this - really do appreciate it.

[John_Meaders]

Hi stawali005,

Please mark your comment as the solution.  It will be helpful to others that run in to similar occurences to see that network issues caused those type of problems.  I'm glad it is all fixed now!

John

[stwali005]

Morning John,

I cant see an option to do that, the only action is reply or mark as offensive?

Thanks