Status 84 media write error on NBU 7.5

[16ris10]

 2 node clustered master server running nbu 7.5 on rhel6. 4 other media servers.

currently having an issue with tape backups. its a new setup and not in production. i recently installed evrything. i tested my disk backups and they ran successful and the test was to backup one of the filesystem of the media server. i had no issues backing up one of the media server. i dont have any client currentlty in the environment.

the topology of the environment is to have the tape backups only from the clustered master server. and all disk backups from the media server.

01/25/2013 13:51:17 - Info nbjm (pid=28809) starting backup job (jobid=14) for client media01.domain.com, policy policy_1_asp, schedule tmp_bkup
01/25/2013 13:51:17 - Info nbjm (pid=28809) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=14, request id:{F744DA40-6730-11E2-BCAD-2D8C392DDC40})
01/25/2013 13:51:17 - requesting resource nbu-hcart2-robot-tld-0
01/25/2013 13:51:17 - requesting resource nbu.domain.com.NBU_CLIENT.MAXJOBS.media01.domain.com
01/25/2013 13:51:17 - requesting resource nbu.domain.com.NBU_POLICY.MAXJOBS.policy_1_asp
01/25/2013 13:51:18 - Info bpbrm (pid=29725) media01.domain.com is the host to backup data from
01/25/2013 13:51:18 - Info bpbrm (pid=29725) reading file list from client
01/25/2013 13:51:18 - Info bpbrm (pid=29725) starting bpbkar on client
01/25/2013 13:51:18 - Info bpbkar (pid=49560) Backup started
01/25/2013 13:51:18 - Info bpbrm (pid=29725) bptm pid: 29727
01/25/2013 13:51:18 - granted resource  nbu.domain.com.NBU_CLIENT.MAXJOBS.media01.domain.com
01/25/2013 13:51:18 - granted resource  nbu.domain.com.NBU_POLICY.MAXJOBS.policy_1_asp
01/25/2013 13:51:18 - granted resource  M00001
01/25/2013 13:51:18 - granted resource  Drive000
01/25/2013 13:51:18 - granted resource  nbu-hcart2-robot-tld-0
01/25/2013 13:51:18 - estimated 0 kbytes needed
01/25/2013 13:51:18 - Info nbjm (pid=28809) started backup (backupid=media01.domain.com_1359147078) job for client media01.domain.com, policy policy_1_asp, schedule tmp_bkup on storage unit nbu-hcart2-robot-tld-0
01/25/2013 13:51:18 - started process bpbrm (pid=29725)
01/25/2013 13:51:18 - connecting
01/25/2013 13:51:18 - connected; connect time: 0:00:00
01/25/2013 13:51:19 - Info bptm (pid=29727) start
01/25/2013 13:51:19 - Info bptm (pid=29727) using 65536 data buffer size
01/25/2013 13:51:19 - Info bptm (pid=29727) using 30 data buffers
01/25/2013 13:51:19 - Info bptm (pid=29727) start backup
01/25/2013 13:51:19 - Info bptm (pid=29727) backup child process is pid 29737
01/25/2013 13:51:19 - Info bptm (pid=29727) Waiting for mount of media id M00001 (copy 1) on server nbu.domain.com.
01/25/2013 13:51:19 - mounting M00001
01/25/2013 13:52:04 - Info bptm (pid=29727) media id M00001 mounted on drive index 0, drivepath /dev/nst3, drivename Drive000, copy 1
01/25/2013 13:52:04 - mounted M00001; mount time: 0:00:45
01/25/2013 13:52:04 - positioning M00001 to file 1
01/25/2013 13:52:53 - Error bptm (pid=29727) write error on media id M00001, drive index 0, writing header block, Input/output error
01/25/2013 13:52:53 - Info bptm (pid=29727) EXITING with status 84 <----------
01/25/2013 13:52:53 - Error bpbrm (pid=29725) from client media01.domain.com: ERR - Cannot write to STDOUT. Errno = 104: Connection reset by peer
01/25/2013 13:52:54 - Info bpbkar (pid=49560) done. status: 84: media write error
01/25/2013 13:52:54 - end writing
01/25/2013 13:53:38 - job 14 was restarted as job 15
media write error  (84)

logs to be posted in the following post. if any required. like bptm. bpbrm. bpbkar.

[16ris10]

[root@master01 bin]# ./tpconfig -d
Id  DriveName           Type   Residence
      Drive Path                                                       Status
****************************************************************************
0   Drive000             hcart2 TLD(0)  DRIVE=4
      /dev/nst3                                                        UP
1   Drive001             hcart2 TLD(0)  DRIVE=3
      /dev/nst2                                                        UP
2   Drive002             hcart2 TLD(0)  DRIVE=2
      /dev/nst1                                                        UP
3   Drive003             hcart2 TLD(0)  DRIVE=1
      /dev/nst0                                                        UP

Currently defined robotics are:
  TLD(0)     robotic path = /dev/sg1

EMM Server = nbu.domain.com


[root@master01 bin]# ./vmoprcmd -d

                                PENDING REQUESTS

                                     

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart2   TLD                -                     No       -         0
  1 hcart2   TLD                -                     No       -         0
  2 hcart2   TLD                -                     No       -         0
  3 hcart2   TLD                -                     No       -         0

                             ADDITIONAL DRIVE STATUS

Drv DriveName            Shared    Assigned        Comment
  0 Drive000              No       -
  1 Drive001              No       -
  2 Drive002              No       -
  3 Drive003              No       -



[root@master01 bin]# ./scan -tape
************************************************************
*********************** SDT_TAPE    ************************
************************************************************
------------------------------------------------------------
Device Name  : "/dev/nst3"
Passthru Name: "/dev/sg5"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 5-SCSI  I5CS"
Vendor ID  : "HP      "
Product ID : "Ultrium 5-SCSI  "
Product Rev: "I5CS"
Serial Number: "STU"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 10
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "/dev/nst2"
Passthru Name: "/dev/sg4"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 5-SCSI  I5CS"
Vendor ID  : "HP      "
Product ID : "Ultrium 5-SCSI  "
Product Rev: "I5CS"
Serial Number: "PQR"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 10
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "/dev/nst1"
Passthru Name: "/dev/sg3"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 5-SCSI  I5CS"
Vendor ID  : "HP      "
Product ID : "Ultrium 5-SCSI  "
Product Rev: "I5CS"
Serial Number: "ABC"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 10
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0
------------------------------------------------------------
Device Name  : "/dev/nst0"
Passthru Name: "/dev/sg2"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "HP      Ultrium 5-SCSI  I5CS"
Vendor ID  : "HP      "
Product ID : "Ultrium 5-SCSI  "
Product Rev: "I5CS"
Serial Number: "XYZ"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 10
Removable      : Yes
Device Supports: SCSI-6
Flags : 0x0
Reason: 0x0

[16ris10]

i just fired a new backup for the loggings bpbrm on the media server/client:

04:51:46.622 [45565] <4> bpbkar: INF - BACKUP START 45565
04:51:46.622 [45565] <4> bpbkar: INF - Estimate:-1 -1
04:51:46.624 [45565] <2> bpbkar add_to_filelist: starting sizeof(filelistrec) <128>
04:51:46.624 [45565] <4> bpbkar: start to backup filelist /tmp ,nb_fscp_enabled is 0
04:51:46.624 [45565] <4> bpbkar: INF - Processing /tmp
04:51:47.362 [45565] <2> fscp_is_tracked: disabled tla_init
04:53:20.139 [45565] <16> bpbkar: ERR - Cannot write to STDOUT. Errno = 104: Connection reset by peer
04:53:20.139 [45565] <16> bpbkar: ERR - bpbkar FATAL exit status = 24: socket write failed
04:53:20.139 [45565] <4> bpbkar: INF - EXIT STATUS 24: socket write failed
04:53:20.139 [45565] <4> bpbkar: INF - setenv FINISHED=0

bptm on the master active server's node master01 to be followed as in attachement at verbose 5.

[16ris10]

bptm log at verbose 5. on the active node of master (master01) (virtual name nbu) has some <16>s in there. :(

bpbrm also attached from the master/media.

[Yasuhisa_Ishika]

Are there any relevant messages in /var/log/messages, especially something relating to SCSI reservation?

I assume you are using some sort of VTL(odd drive serial number!). Is it functional at OS level?
Mount tapes from robtest, and try to read/write with OS tar command if possible.

[Mark_Solutions]

I am assuming that you have edited that output? especially as pointed out with regards to the serial numbers?

I first thoughts here are:

Wrong media type - LTO3 or lower in an LTO5 drive

WORM tape

Write Protected tape

Tape used in another backup system and no overwrite set (though this hould have been logged and it is not - so unlikely)

Tell us about your hardware and tapes so that we can assist further - it just cannot write a header to the tape so that is where to look

[16ris10]

no, i actually edited the names of the serial drives, are they important to be mentioned? ok, let me try the robtest. am not sure of the tar thingie to do. :S.

[16ris10]

yes your assumption is corrected. serial numbeers with other names have been edited. even the IPs. yes they're all LTO5 tape drives. how do i correct this now? WORM tape? let me see research on that and write protected tape aswell. these tapes are not used in another backup system. this is new environment. and we have put in 5-6 tapes for testing only. it highly possible that oeverwrite is not set since i never did this. its all new.

hardware. library is eml 245 with 4 lto5 drives. with encryption. tapes are as you can see in those output above. if not i'll get them for you in a moment.

[16ris10]

and no we're not using VTL. its physical eml245e tape library..

[Mark_Solutions]

Just re-read this thread and noticed what i believe is an issue ....

If what i have gathered is correct you have only disk on the Media Servers and only tape on the Master.

So ... first point is that if you configure tapes on a Master you have to configure it for each node - so configure it whilst running n node 1 the fail over and configure on node 2 so that both nodes are configured - both nodes also need to be able to properly resolve the short and FQDN name of the virtual server

Second point .. and the real problem you have ... is that you are running NetBackup 7.5 with a clustered Master which is fine ... but then you add tape to it which also makes it a media server .. and clustered media servers are no longer supported in NetBackup 7.5 .. so you cannot use disk or tape on the Master - it just needs to be a Master.

From the NetBackup 7.5 Release notes:

NetBackup 7.5 media server installations cannot be clustered. However, you
can upgrade existing NetBackup 6.x clustered media servers to version 7.5 and
they remain clustered.

Sorry but your system is not supported so you need to replan you topology - if i have read you first thread correctly anyway

Hope this helps?!

[16ris10]

very interesting and we never realised this released note fact. yes you have gathered it correct. master 2 node cluster on tapes only. and media server for disk only.

to your 1st point: yes both the nodes are configured for tapes. both can be resolved. this is not the issue in our case.

to your 2nd point. i agree on that, please tell me one thing, is this not suported here mean that its not supported by the symantec as yet, or it is impossible todo with this realese of 7.5? i mean not now, but may be in a month or two they might start supporting it? i have not configured disk on the master. and only through media server only and backups worked fine. nice. so that means. i can ask for an HBA(since we dont have them on any media server) on a media server and then we can have tapes backups too right? just cause of this blunder that we've made. would it be wrong to tell the upper management or whoever responsible to take out the existing HBA from the master and have it on one of the media servers? 

[16ris10]

why did sort risk assesment pick up this fact that it cannot be of media server role here in our environment. grr... what to do now?

[Mark_Solutions]

OK - so you do have an issue!

I don't think that it wouldn't work, although you are clearly having problems, as it does say that if you have a 6.5 clustered Media Server that they would still support it after a 7.5 upgrade - but i can see even that being withdrawn in the next release (7.6 or 8)

They are unlikely to ever support it now as they actually withdrew support for clustered Media Servers in 7.5

So as this is a new system i would get away from you current toplogy and, presuming the other media servers are not clustered, put the tape onto one of those.

Better still use fibre and a switch and share the drives between your media servers so that they can all write theor own data to tape.

So keep your clusetered Master as just a Master and the media servers as media server and you will be all good - shared storage option to share the tape drives would really be the way to go

Hope this helps

[16ris10]

hmmm... ok. i can forward a recommendation to the folks here,

but the thing is, can i make this work atleast? i haven't done any configuratino yet, thats the reason i believe its not working. its totally un-configured. all i did was run the wizard to nind tape drives and thats it. tape drives and robot was visible on the os. so had no issue there. now im getting these write errors. because of what? not because its not supported. but because there is something still not configured.

[16ris10]

how can i check if i still have ths hared storage option license installed. wat to look for?

[16ris10]

thanks for your advice. i have forwarded the recommendation. can you please also tell, how HBAs do i need for this? he says he's gonna order few right away. do i need 2 on each media server . or 2 on 1 media server would only? currently there are two coming our way ina few days. or its just one HBA needed foer the HBA media server?

[Mark_Solutions]

OK - you have 4 tape drive plus disk

If it is just the tape drive to connect and they look to be LTO5 then you dont want more than 2 per HBA port - so maybe a dual port HBA for each media server would do the trick for you.

I am assuming that the disk is connected else where - either way you dont want the disk and the tape drives on the same HBA port.

For the SSO option it depends how you are licensed. If you have a capacity license then they are covered in that, if you have traditional licensing then ask your symantec partner that sells you the licenses to get an IBR report from Symantec

You need to do this in writing (an email is fine)

An IBR report is an Install Base Report and will tell you exactly what licenses you own for the NetBackup so you can see if you have SSO, but being a new project of a reasonable size you may well have a capacity licnsese - there should be one or more pdf files that were sent through with the license keys in.

You can also open  the admin console and use Help - License keys to see if the Shared Storage Open shows up - but you still need to know how many you have as you need one per drive (so need 4)

You will also need everything to go through a fibre switch so that all media servers get access to all four tape drive

What you have should work just fine - just needs setting up correctly

[16ris10]

have a look at this output. i entered F for all active licence keys.

[root@media04 admincmd]# ./get_license_key

        License Key Utility
        -------------------

        A) Add a License Key
        D) Delete a License Key
        F) List Active License Keys
        L) List Registered License Keys
        H) Help
        q) Quit License Key Utility

Enter a letter: f

Enter the name of the host (default is media04):

Active NetBackup Features
=========================

License Key:    THIS-ISAL-ICEN-CEKE-YSOE-DITI-NGIT-LOLH-EHEH-E
Date Added:     Jun 23, 2012 at 23:12:54
Host:           media04
Product:        NetBackup Enterprise Server
Client count:   See license certificate.
Expires:        No Expiration Date
Feature:        Base NetBackup
Feature:        Additional clients
Feature:        NDMP
Feature:        Shared Storage Option
Feature:        MS Exchange extension
Feature:        MS SQL Server extension
Feature:        DB2 extension
Feature:        Lotus Notes extension
Feature:        Oracle extension
Feature:        Informix extension
Feature:        Sybase extension
Feature:        SAP extension
Feature:        Encryption (Legacy DES 40-bit)
Feature:        Encryption (Legacy DES 56-bit)
Feature:        DataStore
Feature:        Library Based Tape Drives
Feature:        Vault
Feature:        Inline Tape Copy
Feature:        MS SharePoint Agent
Feature:        Snapshot Client
Feature:        StorageTek ACS Robotic Libraries
Feature:        Fujitsu LMF Robotic Libraries
Feature:        IBM ATL Robotic Libraries
Feature:        ADIC DAS/SDLC Robotic Libraries
Feature:        Microsoft RSM Robotic Libraries
Feature:        Remote Media Server Support
Feature:        Robotic Library Sharing Support
Feature:        Remote Client Support
Feature:        Open File Backup
Feature:        Encryption
Feature:        Bare Metal Restore
Feature:        Virtual Tape Option
Feature:        OpenStorage Disk Option
Feature:        Flexible Disk Option
Feature:        PureDisk MS SQL Server Agent
Feature:        Enterprise Vault Agent
Feature:        PureDisk MS Exchange Agent
Feature:        SAN Client
Feature:        Replication Director

=========================

[16ris10]

L selected. for all registerd.

        License Key Utility
        -------------------

        A) Add a License Key
        D) Delete a License Key
        F) List Active License Keys
        L) List Registered License Keys
        H) Help
        q) Quit License Key Utility


Enter a letter: L

Enter the name of the host (default is media04):

Registered License Keys
=======================

License Key:    THIS-ISAL-ICEN-CEKE-YSOE-DITI-NGIT-LOLH-EHEH-E
Date Added:     Jun 23, 2012 at 23:12:54
Host:           media04
Product:        NetBackup Enterprise Server
Client count:   See license certificate.
Expires:        No Expiration Date
Feature:        Replication Director (Active)
Feature:        SAN Client (Active)
Feature:        PureDisk MS Exchange Agent (Active)
Feature:        Enterprise Vault Agent (Active)
Feature:        PureDisk MS SQL Server Agent (Active)
Feature:        Flexible Disk Option (Active)
Feature:        OpenStorage Disk Option (Active)
Feature:        Virtual Tape Option (Active)
Feature:        Bare Metal Restore (Active)
Feature:        Encryption (Active)
Feature:        Open File Backup (Active)
Feature:        Remote Client Support (Active)
Feature:        Robotic Library Sharing Support (Active)
Feature:        Remote Media Server Support (Active)
Feature:        Microsoft RSM Robotic Libraries (Active)
Feature:        ADIC DAS/SDLC Robotic Libraries (Active)
Feature:        IBM ATL Robotic Libraries (Active)
Feature:        Fujitsu LMF Robotic Libraries (Active)
Feature:        StorageTek ACS Robotic Libraries (Active)
Feature:        Snapshot Client (Active)
Feature:        MS SharePoint Agent (Active)
Feature:        Inline Tape Copy (Active)
Feature:        Vault (Active)
Feature:        Library Based Tape Drives (Active)
Feature:        DataStore (Active)
Feature:        Encryption (Legacy DES 56-bit) (Active)
Feature:        Encryption (Legacy DES 40-bit) (Active)
Feature:        SAP extension (Active)
Feature:        Sybase extension (Active)
Feature:        Informix extension (Active)
Feature:        Oracle extension (Active)
Feature:        Lotus Notes extension (Active)
Feature:        DB2 extension (Active)
Feature:        MS SQL Server extension (Active)
Feature:        MS Exchange extension (Active)
Feature:        Shared Storage Option (Active)
Feature:        NDMP (Active)
Feature:        Additional clients (Active)
Feature:        Base NetBackup (Active)

[16ris10]

am gonna do the mt and tar thingie now on the tapes. btw, tape drives have encyption feature on them. might be the encryption thingie coming in between this write thing.