NDMP Backup Failures - TpErrno = Robot operation failed

[LucasFroes]

Hello all,

I'm having problems with NDMP backup. "TpErrno = Robot operation failed"

Netbackup version 7.1.0.4 (Master Server), 7.1.0.4 (Media Server)

TAPE LIBRARY - STORAGETEK SL-150

TAPE DRIVE - HP ULTRIUM 5

Detailed Status

-----------------------------------------------------------------------------------------------------------------------

20/03/2014 18:50:05 - Info nbjm(pid=7488) starting backup job (jobid=310839) for client s-san1-b.sp.local, policy POL-SEDE-FILE-SERVER-SAN1, schedule DIFERENCIAL_DIARIO_04_QUINTA  
20/03/2014 18:50:05 - Info nbjm(pid=7488) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=310839, request id:{3DEDAE78-3BC7-4554-AB15-924C9BE3ECA3})  
20/03/2014 18:50:05 - requesting resource snbumediap2-hcart-robot-tld-0-S-SAN1-B.sp.local
20/03/2014 18:50:05 - requesting resource snbup1.sp.local.NBU_CLIENT.MAXJOBS.s-san1-b.sp.local
20/03/2014 18:50:05 - requesting resource snbup1.sp.local.NBU_POLICY.MAXJOBS.POL-SEDE-FILE-SERVER-SAN1
20/03/2014 18:50:05 - granted resource snbup1.sp.local.NBU_CLIENT.MAXJOBS.s-san1-b.sp.local
20/03/2014 18:50:05 - granted resource snbup1.sp.local.NBU_POLICY.MAXJOBS.POL-SEDE-FILE-SERVER-SAN1
20/03/2014 18:50:05 - granted resource 012455
20/03/2014 18:50:05 - granted resource HP.ULTRIUM5-SCSI.005
20/03/2014 18:50:05 - granted resource snbumediap2-hcart-robot-tld-0-S-SAN1-B.sp.local
20/03/2014 18:50:07 - estimated 0 Kbytes needed
20/03/2014 18:50:07 - Info nbjm(pid=7488) started backup job for client s-san1-b.sp.local, policy POL-SEDE-FILE-SERVER-SAN1, schedule DIFERENCIAL_DIARIO_04_QUINTA on storage unit snbumediap2-hcart-robot-tld-0-S-SAN1-B.sp.local
20/03/2014 18:50:08 - started process bpbrm (1604)
20/03/2014 18:50:09 - Info bpbrm(pid=1604) s-san1-b.sp.local is the host to backup data from     
20/03/2014 18:50:09 - Info bpbrm(pid=1604) reading file list from client        
20/03/2014 18:50:09 - Info bpbrm(pid=1604) starting ndmpagent on client         
20/03/2014 18:50:09 - Info ndmpagent(pid=1264) Backup started           
20/03/2014 18:50:09 - Info bptm(pid=1436) start            
20/03/2014 18:50:09 - Info bptm(pid=1436) using 30 data buffers         
20/03/2014 18:50:09 - Info bptm(pid=1436) using 65536 data buffer size        
20/03/2014 18:50:09 - connecting
20/03/2014 18:50:09 - connected; connect time: 00:00:00
20/03/2014 18:50:10 - Info bptm(pid=1436) start backup           
20/03/2014 18:50:10 - Info bptm(pid=1436) Waiting for mount of media id 012455 (copy 1) on server snbumediap2.sp.local. 
20/03/2014 18:50:10 - mounting 012455
20/03/2014 18:51:12 - Error bptm(pid=1436) error requesting media, TpErrno = Robot operation failed     
20/03/2014 18:51:15 - Warning bptm(pid=1436) media id 012455 load operation reported an error     
20/03/2014 18:51:16 - current media 012455 complete, requesting next resource Any
20/03/2014 18:53:12 - current media -- complete, awaiting next media Any Reason: Drives are in use, Media Server: snbumediap2.sp.local, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: SEDE-DIARIA-QUINTA, Storage Unit: snbumediap2-hcart-robot-tld-0-S-SAN1-B.sp.local, Drive Scan Host: N/A
    
21/03/2014 07:51:40 - Info bptm(pid=1436) Waiting for mount of media id 012455 (copy 1) on server snbumediap2.sp.local. 
21/03/2014 07:51:40 - granted resource 012455
21/03/2014 07:51:40 - granted resource HP.ULTRIUM5-SCSI.002
21/03/2014 07:51:40 - granted resource snbumediap2-hcart-robot-tld-0-S-SAN1-B.sp.local
21/03/2014 07:51:40 - mounting 012455
21/03/2014 07:52:32 - Error bptm(pid=1436) error requesting media, TpErrno = Robot operation failed     
21/03/2014 07:52:35 - Warning bptm(pid=1436) media id 012455 load operation reported an error     
21/03/2014 07:52:36 - current media 012455 complete, requesting next resource Any
21/03/2014 07:54:38 - current media -- complete, awaiting next media Any Reason: Drives are in use, Media Server: snbumediap2.sp.local, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: SEDE-DIARIA-QUINTA, Storage Unit: snbumediap2-hcart-robot-tld-0-S-SAN1-B.sp.local, Drive Scan Host: N/A
    
21/03/2014 13:59:59 - Error ndmpagent(pid=1264) terminated by parent process         
21/03/2014 14:00:00 - Info ndmpagent(pid=0) done            
21/03/2014 14:00:01 - Error ndmpagent(pid=1264) NDMP backup failed, path = UNKNOWN       
21/03/2014 14:00:01 - Error ndmpagent(pid=1264) connection 00000000015FB3E0 ndmp_message_process_one failed, status = 18 (NDMP_XDR_DECODE_ERR)     
21/03/2014 14:00:01 - end writing
21/03/2014 14:00:02 - Error ndmpagent(pid=1264) connection 0000000001603500 ndmp_message_process_one failed, status = 18 (NDMP_XDR_DECODE_ERR)     
termination requested by administrator(150)

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Already checked:

DENSITY - hcart and hcart2.

NDMP GLOBAL SETTINGS - USERNAME AND PASSWORD.

INTREGRITY OF DEVICES.

FIRMWARE - SL-150 2.01

FIRMWARE - HP ULTRIUM 5 - Y65S

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

C:\Program Files\Veritas\Volmgr\bin>tpautoconf.exe -verify s-san1-b.sp.local
Connecting to host "s-san1-b.sp.local" as user "root"...
Waiting for connect notification message...
Opening session--attempting with NDMP protocol version 4...
Opening session--successful with NDMP protocol version 4
  host supports MD5 authentication
Getting MD5 challenge from host...
Logging in using MD5 method...
Host info is:
  host name "s-san1-b"
  os type "NetApp"
  os version "NetApp Release 8.1.2 7-Mode"
  host id "0151741797"
Login was successful
Host supports LOCAL backup/restore
Host supports 3-way backup/restore

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

I'm just having trouble making the NDMP backup, the remaining jobs are normal.

[SymTerry]

Are you able to look at the physical robot? Do you have a web interface to the robot and can get errors? 

"media id 012455 load operation reported an error" means that NetBackup requested a load of 012455 and the hardware responded with a load operation error. something is going on with the physical load of that tape. The drive might already have a tape in it, the tape is damaged, the drive is damaged, its the wrong type of tape for the drive....

[inn_kam]

change the media

current media complete

current media 012455 complete, requesting next resource Any
21/03/2014 07:54:38 - current media -- complete, awaiting next media Any Reason: Drives are in use, Media Server: snbumediap2.sp.local, 
     Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A, 
     Volume Pool: SEDE-DIARIA-QUINTA, Storage Unit: snbumediap2-hcart-robot-tld-0-S-SAN1-B.sp.local, Drive Scan Host: N/A
    
21/03/2014 13:59:59 - Error ndmpagent(pid=1264) terminated by parent process   

[LucasFroes]

Hi Symterry, 

We have access to the console and robot detected a problem with the drive, but this drive has been replaced. 
In robot console have no evidence of problems. Have made tests on the device and no problem was detected. The other backups are performed successfully, and physical drives are not offline in netbackup.

[LucasFroes]

inn kam,

I've done tests with a new media and had the same problem.

[VerJD]

@Lucas Del Rey,

If you replace drives in the library, the web interface should report that everything looks good. However, the configuration in NetBackup needs to be updated because the serial numbers for the old drives are still retained in the EMM database.

This should help you resolve the inconsistencies in the device configuration...

How to update NetBackup for a replaced tape drive without deleting and re-adding the drive

- http://www.symantec.com/docs/TECH34296

[LucasFroes]

@SymJD,

After replacing the drive, I deleted and reconfigured the drive using the setup wizard. 

The logical drives are offline when I start the backup.

[SymTerry]

If you have run the wizard and reinstalled the drives, then try robtest to mount a tape. I want to make sure that tapes can be mounted and unmounted. Review TECH83129 for the commands.

[VerJD]

@Lucas Del Rey,

For media server 'snbumediap2.sp.local', please check for ghost devices (medium changers & tape drives) and share your findings, or the outcome.

How to delete "ghost" or "phantom" devices from the Windows Device Manager when using Symantec NetBackup (tm) - TECH93292

To identify and remove the ghost devices from the Windows Device Manager, do the following:

1. On the media server/robot control host, open the command prompt with 'Run as Administrator' and run these commands:

C:\> set devmgr_show_nonpresent_devices=1
C:\> start devmgmt.msc

2. Then, select 'View' from the drop down and select to 'Show Hidden Devices'.

At this point, any ghost tape devices will be seen with lighter, transparent icon and can be removed. This is done by right-clicking the ghost tape device and selecting "Uninstall". A reboot of the machine should be performed following this action.

[VerJD]

@Lucas Del Rey,

Did you find any ghost devices? Have you consulted with your hardware vendor? Please post an update.

[LucasFroes · Accepted solution]

Hey guys,

Sorry for the delay in responding, but fix the problem downgrading firmware StorageTek SL-150 (2.01) -> (1.50). For some reason the new firmware is not 100% compatible with the netbackup 7.1.0.4

Thanks for everyone's help.

[VerJD]

@Lucas Del Rey, We're glad to hear the issue was fix! We appreciate you sharing and marking the solution as well. Thanks for the update! Have a great week and weekend!