cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

NDMP Backup is Failed with 99 status code

john10
Level 6

Hello All,

We are running NBU 7.0 on win-2k8 Master and Media Server 7.0 on 2k8 and we configured NDMP Backup for Media server, previously backup is running fine for "vol" but suddenly after writing some amout of data the backup failed with 99, error code in Detailed status it showing as "I/O ERROR (for /vol/SnapMgr_Windows_IE10NT3KS001_backup_3)" is this related to NDMP issue or Netbackup issue ?

Probe and Verify is working fine from Master and Media server. and in ndmpagent logs it showing as ""DUMP: Using Full Volume Dump" and also "backup aborted". could any one help out on this. please find the attached detaield status and "ndmpagent" logs.

Failed Detaield job details :-

5/11/2014 9:56:42 PM - requesting resource ie11w0015-hcart2-robot-tld-1-ie11ast0001
5/11/2014 9:56:42 PM - requesting resource ie10nt3ks060.NBU_CLIENT.MAXJOBS.IE11AST0001
5/11/2014 9:56:42 PM - requesting resource ie10nt3ks060.NBU_POLICY.MAXJOBS.NDMP_IE11AST0001_Aggr3_One
5/11/2014 9:56:42 PM - granted resource ie10nt3ks060.NBU_CLIENT.MAXJOBS.IE11AST0001
5/11/2014 9:56:42 PM - granted resource ie10nt3ks060.NBU_POLICY.MAXJOBS.NDMP_IE11AST0001_Aggr3_One
5/11/2014 9:56:42 PM - granted resource 7095L5
5/11/2014 9:56:42 PM - granted resource IBM.ULTRIUM-HH5.000
5/11/2014 9:56:42 PM - granted resource ie11w0015-hcart2-robot-tld-1-ie11ast0001
5/11/2014 9:56:42 PM - estimated 12789425668 Kbytes needed
5/11/2014 9:56:44 PM - started process bpbrm (9732)
5/11/2014 9:56:45 PM - connecting
5/11/2014 9:56:45 PM - connected; connect time: 00:00:00
5/11/2014 9:56:49 PM - mounting 7095L5
5/11/2014 9:57:43 PM - mounted; mount time: 00:00:54
5/11/2014 9:57:43 PM - positioning 7095L5 to file 8
5/11/2014 9:58:44 PM - positioned 7095L5; position time: 00:01:01
5/11/2014 9:58:44 PM - begin writing
5/12/2014 2:48:23 AM - current media 7095L5 complete, requesting next resource Any
5/12/2014 2:48:24 AM - granted resource 7064L5
5/12/2014 2:48:24 AM - granted resource IBM.ULTRIUM-HH5.001
5/12/2014 2:48:24 AM - granted resource ie11w0015-hcart2-robot-tld-1-ie11ast0001
5/12/2014 2:48:26 AM - mounting 7064L5
5/12/2014 2:49:12 AM - mounted; mount time: 00:00:46
5/12/2014 2:49:12 AM - positioning 7064L5 to file 1
5/12/2014 2:49:16 AM - positioned 7064L5; position time: 00:00:04
5/12/2014 2:49:16 AM - begin writing
5/12/2014 8:11:31 AM - current media 7064L5 complete, requesting next resource Any
5/12/2014 8:11:31 AM - granted resource 7077L5
5/12/2014 8:11:31 AM - granted resource IBM.ULTRIUM-HH5.000
5/12/2014 8:11:31 AM - granted resource ie11w0015-hcart2-robot-tld-1-ie11ast0001
5/12/2014 8:11:33 AM - mounting 7077L5
5/12/2014 8:12:32 AM - mounted; mount time: 00:00:59
5/12/2014 8:12:32 AM - positioning 7077L5 to file 1
5/12/2014 8:12:36 AM - positioned 7077L5; position time: 00:00:04
5/12/2014 8:12:36 AM - begin writing
5/12/2014 11:18:15 AM - Error ndmpagent(pid=9600) NDMP_LOG_ERROR 135 DATA: Backup terminated: EVENT: I/O ERROR (for /vol/SnapMgr_Windows_IE10NT3KS001_backup_3)   
5/12/2014 11:18:16 AM - Error ndmpagent(pid=9600) NDMP backup failed, path = /vol/SnapMgr_Windows_IE10NT3KS001_backup_3       
5/12/2014 11:18:17 AM - Error bptm(pid=9556) none of the NDMP backups for client IE11AST0001 completed successfully   
5/12/2014 11:22:49 AM - Error bptm(pid=9556) write error on media id 7077L5, drive index 0, writing header block, 0
5/12/2014 11:22:49 AM - Error bptm(pid=9556) SUSPENDED media id 7077L5, could not terminate correctly     
5/12/2014 11:22:49 AM - end writing; write time: 03:10:13
NDMP backup failure(99)

 

Verify and probe out puts from Media serevr :-

D:\>cd "Program Files\Veritas\Volmgr\bin"

D:\Program Files\Veritas\Volmgr\bin>tpautoconf.exe -verify IE11AST00
Connecting to host "IE11AST0001" as user "root"...
Waiting for connect notification message...
Opening session--attempting with NDMP protocol version 4...
Opening session--successful with NDMP protocol version 4
  host supports MD5 authentication
Getting MD5 challenge from host...
Logging in using MD5 method...
Host info is:
  host name "ie11ast0001"
  os type "NetApp"
  os version "NetApp Release 8.1.2P3 7-Mode"
  host id "0151758733"
Login was successful
Host supports LOCAL backup/restore
Host supports 3-way backup/restore

D:\Program Files\Veritas\Volmgr\bin>tpautoconf.exe -probe IE11AST000
Host "IE11AST0001" SCSI device model "SPECTRA PYTHON":
  Device "mc0" attributes=0x0
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]L1
    SERIAL_NUMBER=911200451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126L1
    ALIAS 0=mc0
Host "IE11AST0001" tape device model "IBM LTO 5 ULTRIUM":
  Device "nrst1l" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st1
  Device "nrst1m" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st1
  Device "nrst1h" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st1
  Device "nrst1a" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st1
  Device "rst1l" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st1
  Device "rst1m" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st1
  Device "rst1h" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st1
  Device "rst1a" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st1
  Device "urst1l" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st1
  Device "urst1m" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st1
  Device "urst1h" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st1
  Device "urst1a" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:115:0090a5:00451f]
    SERIAL_NUMBER=101500451F
    ELECTRICAL_NAME=IE11SS0002:1-5.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st1
Host "IE11AST0001" tape device model "IBM LTO 5 ULTRIUM":
  Device "nrst2l" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st2
  Device "nrst2m" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st2
  Device "nrst2h" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st2
  Device "nrst2a" attributes=(0x4) RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st2
  Device "rst2l" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st2
  Device "rst2m" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st2
  Device "rst2h" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st2
  Device "rst2a" attributes=(0x5) REWIND RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st2
  Device "urst2l" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 4/800GB
    ALIAS 0=st2
  Device "urst2m" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-3(ro)/4 8/1600GB cmp
    ALIAS 0=st2
  Device "urst2h" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 1600GB
    ALIAS 0=st2
  Device "urst2a" attributes=(0x6) UNLOAD RAW
    WORLD_WIDE_NAME=WWN[2:113:0090a5:00451f]
    SERIAL_NUMBER=101300451F
    ELECTRICAL_NAME=IE11SS0002:1-4.126
    DENSITY=LTO-5 3200GB cmp
    ALIAS 0=st2

D:\Program Files\Veritas\Volmgr\bin>

 

1 ACCEPTED SOLUTION

Accepted Solutions

mtaormina
Level 3

This message is always a red flag for me:

5/12/2014 11:22:49 AM - Error bptm(pid=9556) write error on media id 7077L5, drive index 0, writing header block, 0

Usually this means that the write protect switch is set ON for that tape media.

View solution in original post

6 REPLIES 6

pri3006
Level 4
Certified

Do you see the failure showing up only for /vol/SnapMgr_Windows_IE10NT3KS001_backup_3?

Check the filer side logs as well (ndmpd debug file, messages file, etc.). At first look, it seems like a problem on the filer.

This technote may help you with regards to logging on the filer:

http://www.symantec.com/business/support/index?page=content&id=TECH178502

 

 

 

john10
Level 6

Hello pri3006,

We only had 1 volume in that box,and i'm worried if the problem is from storage side will backup can write for 1 day ? and fail with 99?

pri3006
Level 4
Certified

You could also try doing a local dump to null device for that volume (the syntax present in the technote I mentioned in my previous post) to isolate the issue.

mtaormina
Level 3

This message is always a red flag for me:

5/12/2014 11:22:49 AM - Error bptm(pid=9556) write error on media id 7077L5, drive index 0, writing header block, 0

Usually this means that the write protect switch is set ON for that tape media.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

nbmpagent logs cannot be read 'as is'.

You need to read them with vxlogview, e.g.

vxlogview -o 134 -b <start-date> -e <end-date> > ndmpagent.txt

Surround the date by single quotes in UNIX and double quotes in Windows.
For example:
-b ’1/1/2010 12:00:00 AM’

Various status 99 TNs available:

http://www.symantec.com/docs/TECH56492

http://www.symantec.com/docs/TECH37502

http://www.symantec.com/docs/TECH178502 

john10
Level 6

Hello All,

Thanks for your valuable suggestions and now the backup is working fine after changing the new set of tapes.