Forum Discussion

john10's avatar
john10
Level 6
11 years ago

NDMP Backup is Failed with 99 status code

Hello All,

We are running NBU 7.0 on win-2k8 Master and Media Server 7.0 on 2k8 and we configured NDMP Backup for Media server, previously backup is running fine for "vol" but suddenly after writing some amout of data the backup failed with 99, error code in Detailed status it showing as "I/O ERROR (for /vol/SnapMgr_Windows_IE10NT3KS001_backup_3)" is this related to NDMP issue or Netbackup issue ?

Probe and Verify is working fine from Master and Media server. and in ndmpagent logs it showing as ""DUMP: Using Full Volume Dump" and also "backup aborted". could any one help out on this. please find the attached detaield status and "ndmpagent" logs.

Failed Detaield job details :-

5/11/2014 9:56:42 PM - requesting resource ie11w0015-hcart2-robot-tld-1-ie11ast0001
5/11/2014 9:56:42 PM - requesting resource ie10nt3ks060.NBU_CLIENT.MAXJOBS.IE11AST0001
5/11/2014 9:56:42 PM - requesting resource ie10nt3ks060.NBU_POLICY.MAXJOBS.NDMP_IE11AST0001_Aggr3_One
5/11/2014 9:56:42 PM - granted resource ie10nt3ks060.NBU_CLIENT.MAXJOBS.IE11AST0001
5/11/2014 9:56:42 PM - granted resource ie10nt3ks060.NBU_POLICY.MAXJOBS.NDMP_IE11AST0001_Aggr3_One
5/11/2014 9:56:42 PM - granted resource 7095L5
5/11/2014 9:56:42 PM - granted resource IBM.ULTRIUM-HH5.000
5/11/2014 9:56:42 PM - granted resource ie11w0015-hcart2-robot-tld-1-ie11ast0001
5/11/2014 9:56:42 PM - estimated 12789425668 Kbytes needed
5/11/2014 9:56:44 PM - started process bpbrm (9732)
5/11/2014 9:56:45 PM - connecting
5/11/2014 9:56:45 PM - connected; connect time: 00:00:00
5/11/2014 9:56:49 PM - mounting 7095L5
5/11/2014 9:57:43 PM - mounted; mount time: 00:00:54
5/11/2014 9:57:43 PM - positioning 7095L5 to file 8
5/11/2014 9:58:44 PM - positioned 7095L5; position time: 00:01:01
5/11/2014 9:58:44 PM - begin writing
5/12/2014 2:48:23 AM - current media 7095L5 complete, requesting next resource Any
5/12/2014 2:48:24 AM - granted resource 7064L5
5/12/2014 2:48:24 AM - granted resource IBM.ULTRIUM-HH5.001
5/12/2014 2:48:24 AM - granted resource ie11w0015-hcart2-robot-tld-1-ie11ast0001
5/12/2014 2:48:26 AM - mounting 7064L5
5/12/2014 2:49:12 AM - mounted; mount time: 00:00:46
5/12/2014 2:49:12 AM - positioning 7064L5 to file 1
5/12/2014 2:49:16 AM - positioned 7064L5; position time: 00:00:04
5/12/2014 2:49:16 AM - begin writing
5/12/2014 8:11:31 AM - current media 7064L5 complete, requesting next resource Any
5/12/2014 8:11:31 AM - granted resource 7077L5
5/12/2014 8:11:31 AM - granted resource IBM.ULTRIUM-HH5.000
5/12/2014 8:11:31 AM - granted resource ie11w0015-hcart2-robot-tld-1-ie11ast0001
5/12/2014 8:11:33 AM - mounting 7077L5
5/12/2014 8:12:32 AM - mounted; mount time: 00:00:59
5/12/2014 8:12:32 AM - positioning 7077L5 to file 1
5/12/2014 8:12:36 AM - positioned 7077L5; position time: 00:00:04
5/12/2014 8:12:36 AM - begin writing
5/12/2014 11:18:15 AM - Error ndmpagent(pid=9600) NDMP_LOG_ERROR 135 DATA: Backup terminated: EVENT: I/O ERROR (for /vol/SnapMgr_Windows_IE10NT3KS001_backup_3)   
5/12/2014 11:18:16 AM - Error ndmpagent(pid=9600) NDMP backup failed, path = /vol/SnapMgr_Windows_IE10NT3KS001_backup_3       
5/12/2014 11:18:17 AM - Error bptm(pid=9556) none of the NDMP backups for client IE11AST0001 completed successfully   
5/12/2014 11:22:49 AM - Error bptm(pid=9556) write error on media id 7077L5, drive index 0, writing header block, 0
5/12/2014 11:22:49 AM - Error bptm(pid=9556) SUSPENDED media id 7077L5, could not terminate correctly     
5/12/2014 11:22:49 AM - end writing; write time: 03:10:13
NDMP backup failure(99)

 

Verify and probe out puts from Media serevr :-

D:\>cd "Program Files\Veritas\Volmgr\bin"

D:\Program Files\Veritas\Volmgr\bin>tpautoconf.exe -verify IE11AST00
Connecting to host "IE11AST0001" as user "root"...
Waiting for connect notification message...
Opening session--attempting with NDMP protocol version 4...
Opening session--successful with NDMP protocol version 4
  host supports MD5 authentication
Getting MD5 challenge from host...
Logging in using MD5 method...
Host info is:
  host name "ie11ast0001"
  os type "NetApp"
  os version "NetApp Release 8.1.2P3 7-Mode"
  host id "0151758733"
Login was successful
Host supports LOCAL backup/restore
Host supports 3-way backup/restore

D:\Program Files\Veritas\Volmgr\bin>tpautoconf.exe -probe IE11AST000
Host "IE11AST0001" SCSI device model "SPECTRA PYTHON":
  Device "mc0" attributes=0x0
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]L1
    SERIAL_NUMBER=911200451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126L1
    ALIAS 0=mc0
Host "IE11AST0001" tape device model "IBM LTO 5 ULTRIUM":
  Device "nrst1l" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st1
  Device "nrst1m" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st1
  Device "nrst1h" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st1
  Device "nrst1a" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st1
  Device "rst1l" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st1
  Device "rst1m" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st1
  Device "rst1h" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st1
  Device "rst1a" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st1
  Device "urst1l" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st1
  Device "urst1m" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st1
  Device "urst1h" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st1
  Device "urst1a" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st1
Host "IE11AST0001" tape device model "IBM LTO 5 ULTRIUM":
  Device "nrst2l" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st2
  Device "nrst2m" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st2
  Device "nrst2h" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st2
  Device "nrst2a" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st2
  Device "rst2l" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st2
  Device "rst2m" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st2
  Device "rst2h" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st2
  Device "rst2a" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st2
  Device "urst2l" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st2
  Device "urst2m" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st2
  Device "urst2h" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st2
  Device "urst2a" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st2

D:\Program Files\Veritas\Volmgr\bin>

 

7.1.x and Earlier
Backup and Recovery
NetBackup
Windows Server (2003-2008)
  • mtaormina's avatar
    mtaormina
    11 years ago

    This message is always a red flag for me:

    5/12/2014 11:22:49 AM - Error bptm(pid=9556) write error on media id 7077L5, drive index 0, writing header block, 0

    Usually this means that the write protect switch is set ON for that tape media.

  • pri3006's avatar
    pri3006
    Level 4
    11 years ago

    Do you see the failure showing up only for /vol/SnapMgr_Windows_IE10NT3KS001_backup_3?

    Check the filer side logs as well (ndmpd debug file, messages file, etc.). At first look, it seems like a problem on the filer.

    This technote may help you with regards to logging on the filer:

    http://www.symantec.com/business/support/index?page=content&id=TECH178502

     

     

     

  • john10's avatar
    john10
    Level 6
    11 years ago

    Hello pri3006,

    We only had 1 volume in that box,and i'm worried if the problem is from storage side will backup can write for 1 day ? and fail with 99?

  • pri3006's avatar
    pri3006
    Level 4
    11 years ago

    You could also try doing a local dump to null device for that volume (the syntax present in the technote I mentioned in my previous post) to isolate the issue.

  • mtaormina's avatar
    mtaormina
    Level 3
    11 years ago

    This message is always a red flag for me:

    5/12/2014 11:22:49 AM - Error bptm(pid=9556) write error on media id 7077L5, drive index 0, writing header block, 0

    Usually this means that the write protect switch is set ON for that tape media.

  • john10's avatar
    john10
    Level 6
    11 years ago

    Hello All,

    Thanks for your valuable suggestions and now the backup is working fine after changing the new set of tapes.