Getting error message as "media manager killed by signal(82)"

[siddharth_khata]

Hello,

We are facing issue while taking backup of VM on VTL. We have configured mhVTL on media server. Getting error message as "media manager killed by signal(82)". We are successfully able to take backup on disk but not on VTL.

Setup details :

Netbackup version : 7.5

Master server : Windows 2008 R2

Please see below logs for details :

12/19/2013 5:46:58 PM - Info nbjm(pid=2704) starting backup job (jobid=1592) for client tsm-vm6, policy Windows_Testcollector_1month_Vmware, schedule full_1month 
12/19/2013 5:46:58 PM - estimated 0 Kbytes needed
12/19/2013 5:46:58 PM - Info nbjm(pid=2704) started backup (backupid=tsm-vm6_1387455418) job for client tsm-vm6, policy Windows_Testcollector_1month_Vmware, schedule full_1month on storage unit tsm-vm8-hcart2-robot-tld-2
12/19/2013 5:47:00 PM - started process bpbrm (27451)
12/19/2013 5:47:00 PM - Info bpbrm(pid=27451) tsm-vm6 is the host to backup data from    
12/19/2013 5:47:00 PM - Info bpbrm(pid=27451) reading file list from client       
12/19/2013 5:47:00 PM - Info bpbrm(pid=27451) starting bpbkar on client        
12/19/2013 5:47:03 PM - connecting
12/19/2013 5:47:03 PM - connected; connect time: 00:00:00
12/19/2013 5:47:11 PM - Info bpbkar(pid=3192) Backup started          
12/19/2013 5:47:11 PM - Info bpbrm(pid=27451) bptm pid: 27454         
12/19/2013 5:47:11 PM - Info bpbkar(pid=3192) CONTINUE BACKUP received.         
12/19/2013 5:47:11 PM - Info bptm(pid=27454) start           
12/19/2013 5:47:11 PM - Info bptm(pid=27454) using 65536 data buffer size       
12/19/2013 5:47:11 PM - Info bptm(pid=27454) using 30 data buffers        
12/19/2013 5:47:12 PM - Info bptm(pid=27454) start backup          
12/19/2013 5:47:12 PM - Info bptm(pid=27454) backup child process is pid 27463      
12/19/2013 5:47:12 PM - Info bptm(pid=27454) Waiting for mount of media id B00000 (copy 1) on server tsm-vm8.
12/19/2013 5:47:14 PM - Error bptm(pid=27454) media manager terminated during mount of media id B00000, possible media mount timeout
12/19/2013 5:47:15 PM - mounting B00000
12/19/2013 5:47:16 PM - Error bptm(pid=27454) media manager terminated by parent process      
12/19/2013 5:47:33 PM - Critical bpbrm(pid=27451) unexpected termination of client tsm-vm6       
12/19/2013 5:47:34 PM - Info bpbkar(pid=0) done. status: 82: media manager killed by signal    
12/19/2013 5:47:37 PM - end writing
media manager killed by signal(82)

Please let us know the solution to resolve this issue.

[Marianne]

We need to see the bptm log.

The timeout seems weird as it happens only 2 seconds after the mount request:

12/19/2013 5:47:12 PM - Info bptm(pid=27454) Waiting for mount of media id B00000 (copy 1) on server tsm-vm8.
12/19/2013 5:47:14 PM - Error bptm(pid=27454) media manager terminated during mount of media id B00000, possible media mount timeout
12/19/2013 5:47:15 PM - mounting B00000

Was Media Mount timeout configured in Host Properties? If so, what is the value?

Any errors in Windows Event Viewer System and/or Application logs?

Does robtest work if testing robot functionality from cmd?

Robtest commands that can be used to test the SCSI functionality of a robot 
http://www.symantec.com/docs/TECH83129

[siddharth_khata]

Media mount time was set to 5 min.

We are not getting any errors in Windows Event Viewer System and / or Application Logs.

Robtest giving status as below

C:\Program Files\Veritas\Volmgr\bin>robtest.exe
No robots are configured

[Marianne]

No robots are configured

????

Please show us output of these commands (in same volmgr\bin directory):

scan

tpconfig -l

5 minute mount timeout is never a good idea - change it to 15 minutes minimum.

As a matter of interest - have you disabled UAC on this server?
We have seen various wierd and wonderful issues caused by UAC....

[siddharth_khata]

we are getting this error message with latest run.

12/19/2013 7:20:55 PM - Info nbjm(pid=3612) starting backup job (jobid=1598) for client tsm-vm6, policy Windows_Testcollector_1month_Vmware, schedule full_1month 
12/19/2013 7:20:56 PM - estimated 0 Kbytes needed
12/19/2013 7:20:56 PM - Info nbjm(pid=3612) started backup (backupid=tsm-vm6_1387461055) job for client tsm-vm6, policy Windows_Testcollector_1month_Vmware, schedule full_1month on storage unit tsm-vm8-hcart-robot-tld-1
12/19/2013 7:20:57 PM - started process bpbrm (3687)
12/19/2013 7:21:00 PM - Info bpbrm(pid=3687) tsm-vm6 is the host to backup data from    
12/19/2013 7:21:00 PM - Info bpbrm(pid=3687) reading file list from client       
12/19/2013 7:21:00 PM - Info bpbrm(pid=3687) starting bpbkar on client        
12/19/2013 7:21:00 PM - connecting
12/19/2013 7:21:01 PM - connected; connect time: 00:00:01
12/19/2013 7:21:10 PM - Info bpbkar(pid=1808) Backup started          
12/19/2013 7:21:10 PM - Info bpbrm(pid=3687) bptm pid: 3688         
12/19/2013 7:21:10 PM - Info bpbkar(pid=1808) CONTINUE BACKUP received.         
12/19/2013 7:21:10 PM - Info bptm(pid=3688) start           
12/19/2013 7:21:11 PM - Info bptm(pid=3688) using 65536 data buffer size       
12/19/2013 7:21:11 PM - Info bptm(pid=3688) using 30 data buffers        
12/19/2013 7:21:11 PM - Info bptm(pid=3688) start backup          
12/19/2013 7:21:11 PM - Info bptm(pid=3688) backup child process is pid 3697      
12/19/2013 7:21:11 PM - Info bptm(pid=3688) Waiting for mount of media id A00000 (copy 1) on server tsm-vm8.
12/19/2013 7:21:12 PM - mounting A00000
12/19/2013 7:21:19 PM - Info bptm(pid=3688) media id A00000 mounted on drive index 1, drivepath /dev/nst6, drivename STK.T10000B.001, copy 1
12/19/2013 7:21:20 PM - mounted; mount time: 00:00:08
12/19/2013 7:21:20 PM - positioning A00000 to file 3
12/19/2013 7:21:20 PM - positioned A00000; position time: 00:00:00
12/19/2013 7:21:20 PM - begin writing
12/19/2013 7:23:23 PM - Error bptm(pid=3688) media manager terminated by parent process      
12/19/2013 7:23:25 PM - Critical bpbrm(pid=3687) unexpected termination of client tsm-vm6       
12/19/2013 7:23:25 PM - Info bpbkar(pid=0) done. status: 82: media manager killed by signal    
12/19/2013 7:23:26 PM - end writing; write time: 00:02:06
media manager killed by signal(82)

[Mark_Solutions]

Looks like something is crashing here to be honest - do you have anti virus installed? - if so have you set all the exclusions for netbackup directories and processes - and if McAfee have you disabled the inbuild firewall driver - it is known to kill of NetBackup services

[Marianne]

So, what happened in between Media Mount timeout, 'No robots are configured', and latest backup attempt that now says 'unexpected termination of client tsm-vm6' ?

You need to enable logs.

On media server: bptm and bpbrm

On client: bpcd and bpbkar

Please copy log files to reflect process name (e.g. bpbrm.txt) and upload as File Attachments.

[Mark_Solutions]

just spotted something here....

client name ...... tsm-vm6 ... i guess the "vm" means it is a virtual server on VMware?

storage unit name ..... tsm-vm8-hcart-robot-tld-1 .. so presumably media server name is tsm-vm8 ... is this a virtual server with tape drives mapped?

I think you will find in terms of VMware and NetBackup that is unsupported because it doesnt work properly - so you are probably just dropping the connection to the tape drives if they are mapped through VMware

Please confirm if this is the case

[Mark_Solutions]

I see it is mhVTL - so may be OK on VMware - has it ever worked at all?

If not i would reccomend rebuilding the VTL and try again - i have used this software VTL image and occasionally it needs a fresh build to get it to work

If you have no data on it yet try a fresh build

[siddharth_khata]

We are taking backup of tsm-vm6 virtual machine on virtual media server tsm-vm8. VTL is configured on tsm-vm8.

Do you mean that we can not configure VTL on virtual machine? We should configure it on physical machine only?

[siddharth_khata]

We have installed fresh build of VTL. And that was working for other type of backups. It is giving error for VMWare backups only.

[Marianne]

Is the VTL configured on the same datastores that you are trying to backup with VMware policy?

[siddharth_khata]

We have only one ESX server on which 10 VMs are created. We have configured 8th VM as media server & installed mhVTL on it. And we are taking backup of 6th VM on VTL.

Does it causes the problem?

Do I need to configure VTL on some other machine out of this ESX server?

[Marianne]

Are you doing VMware-type backup or MS-Windows backup? 

Is the VTL located on the same datastore as the VM that you are backing up?

[Deb_Wilmot]

Hmmm...

Note the following: "12/19/2013 7:20:56 PM - estimated 0 Kbytes needed"

Maybe this is a new vmware client..

You may want to check out the backup host bptm and bpfis logs.  It's very possible that there is nothing to backup (maybe a problem with the snapshot data???) and that is why you're seeing the error.

So on tsm-vm8, create the bpcd, bpbrm, bpfis and bptm directories in netbackup logs (not sure the platform - assume linux or windows). 

Set the verbose level to 5 in the GUI, media server host properties, logging tab (make sure bpbrm and bptm are  set to 5 or the same as global).

Rerun the backup.

Review the logs relating to the backup.

A status 82 means that something else terminated and bptm received a termination request due to that.  The bptm logs will probably not tell the story - most likely these other logs will. 

[siddharth_khata]

We are doing VMware-type backups.

MS-Windows type backups are working fine.

VTL is located on different machine but on same ESX server.

[Marianne]

VTL is located on different machine but on same ESX server.

My question was about the datastore on the ESX server and if the VTL and VMware hosts that need to be backed up are on the same datastore....

Ensure that the VTL is on a different datastore to the VMs that you need to backup.