Status 84 media write error on NBU 7.5

[16ris10]

 2 node clustered master server running nbu 7.5 on rhel6. 4 other media servers.

currently having an issue with tape backups. its a new setup and not in production. i recently installed evrything. i tested my disk backups and they ran successful and the test was to backup one of the filesystem of the media server. i had no issues backing up one of the media server. i dont have any client currentlty in the environment.

the topology of the environment is to have the tape backups only from the clustered master server. and all disk backups from the media server.

01/25/2013 13:51:17 - Info nbjm (pid=28809) starting backup job (jobid=14) for client media01.domain.com, policy policy_1_asp, schedule tmp_bkup
01/25/2013 13:51:17 - Info nbjm (pid=28809) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=14, request id:{F744DA40-6730-11E2-BCAD-2D8C392DDC40})
01/25/2013 13:51:17 - requesting resource nbu-hcart2-robot-tld-0
01/25/2013 13:51:17 - requesting resource nbu.domain.com.NBU_CLIENT.MAXJOBS.media01.domain.com
01/25/2013 13:51:17 - requesting resource nbu.domain.com.NBU_POLICY.MAXJOBS.policy_1_asp
01/25/2013 13:51:18 - Info bpbrm (pid=29725) media01.domain.com is the host to backup data from
01/25/2013 13:51:18 - Info bpbrm (pid=29725) reading file list from client
01/25/2013 13:51:18 - Info bpbrm (pid=29725) starting bpbkar on client
01/25/2013 13:51:18 - Info bpbkar (pid=49560) Backup started
01/25/2013 13:51:18 - Info bpbrm (pid=29725) bptm pid: 29727
01/25/2013 13:51:18 - granted resource  nbu.domain.com.NBU_CLIENT.MAXJOBS.media01.domain.com
01/25/2013 13:51:18 - granted resource  nbu.domain.com.NBU_POLICY.MAXJOBS.policy_1_asp
01/25/2013 13:51:18 - granted resource  M00001
01/25/2013 13:51:18 - granted resource  Drive000
01/25/2013 13:51:18 - granted resource  nbu-hcart2-robot-tld-0
01/25/2013 13:51:18 - estimated 0 kbytes needed
01/25/2013 13:51:18 - Info nbjm (pid=28809) started backup (backupid=media01.domain.com_1359147078) job for client media01.domain.com, policy policy_1_asp, schedule tmp_bkup on storage unit nbu-hcart2-robot-tld-0
01/25/2013 13:51:18 - started process bpbrm (pid=29725)
01/25/2013 13:51:18 - connecting
01/25/2013 13:51:18 - connected; connect time: 0:00:00
01/25/2013 13:51:19 - Info bptm (pid=29727) start
01/25/2013 13:51:19 - Info bptm (pid=29727) using 65536 data buffer size
01/25/2013 13:51:19 - Info bptm (pid=29727) using 30 data buffers
01/25/2013 13:51:19 - Info bptm (pid=29727) start backup
01/25/2013 13:51:19 - Info bptm (pid=29727) backup child process is pid 29737
01/25/2013 13:51:19 - Info bptm (pid=29727) Waiting for mount of media id M00001 (copy 1) on server nbu.domain.com.
01/25/2013 13:51:19 - mounting M00001
01/25/2013 13:52:04 - Info bptm (pid=29727) media id M00001 mounted on drive index 0, drivepath /dev/nst3, drivename Drive000, copy 1
01/25/2013 13:52:04 - mounted M00001; mount time: 0:00:45
01/25/2013 13:52:04 - positioning M00001 to file 1
01/25/2013 13:52:53 - Error bptm (pid=29727) write error on media id M00001, drive index 0, writing header block, Input/output error
01/25/2013 13:52:53 - Info bptm (pid=29727) EXITING with status 84 <----------
01/25/2013 13:52:53 - Error bpbrm (pid=29725) from client media01.domain.com: ERR - Cannot write to STDOUT. Errno = 104: Connection reset by peer
01/25/2013 13:52:54 - Info bpbkar (pid=49560) done. status: 84: media write error
01/25/2013 13:52:54 - end writing
01/25/2013 13:53:38 - job 14 was restarted as job 15
media write error  (84)

logs to be posted in the following post. if any required. like bptm. bpbrm. bpbkar.

[Mark_Solutions]

It does show SSO but can't tell from that how many

Open you admin console (a Windows one if you have one - Java may do the same but not sure) and go to Help - License keys

In here there is an option to show the capacity based license summary

If that shows up relevant stuff then you are capacity based and you have what ever you need - if not you still need to find your certificates or get an IBR report

[16ris10]

heres the screenshot.

and hey. i installed the mt-st. but when i use the command. it gives me a

[root@master01 bin]# mt -f /dev/nst0 rewind
/dev/nst0: Input/output error

[16ris10]

am i supposed to use passthru name or the device name?

Device Name  : "/dev/nst2"
Passthru Name: "/dev/sg4"

[Yasuhisa_Ishika]

/dev/nstX for mt - do not use paththru device for mt.

BTW, have you already checked /var/log/messages as I mentioned before?

[Marianne]

You seem to only have NetBackup Enterprise and Enterprise Disk license installed.

You also need Library Based Tape Drive license (QTY = 4) 
as well as Shared Storage Option license (QTY = 4)

These 2 licenses need to be added to both Master server nodes as well as all Media servers.
You then need to delete current Device config and start from scratch.
vmoprcmd -d on all servers need to show 
Shared
 Yes
for all tape drives.

Once you have added correct licenses and all devices show up correctly, and you still experience status 84's, let us then start troubleshooting.
Ensure all of the following logs are enabled:
bptm on all media servers
VERBOSE entry in vm.conf (/usr/openv/volmgr) on all media servers (including cluster nodes). Do this before re-running Device Config to ensure Media Manager processes are started in verbose mode (-v).

[Mark_Solutions]

As Marianne says you are not capacity based and so need all licenses for each component you install

I would suggest getting your IBR report or finding out if someone at your place has registered on the Licensing portal where you can access your keys

It may even be worth phoning customer support to get helk setting you up on the licensing portal and if they have your company details they may well be able to populate your portal for you so you know what you have.

Once you have all licenses get them added to the Master and Media Servers, redeploy your tape library and drives to the Media Servers and you will be all good

[Marianne]

If this installation is meant to become your new production installation, you may as well use drive and SSO licenses from your current production installation.

[16ris10]

yes, i did nothing relating to tape drives or robot..

[16ris10]

hmmm... i have the receipt, the certificate where the keys are listed. the thing is, what I saw in that certificate. 2 keys. when i was installing the master/media server. only worked. the other one never worked. not sure what that was for. i'll see if i can crop that for you here.

mariiane. mark;s recommendation is to run tape backups from the master server as its not supported yet. initially our plan was to have disk backups from the media and tape from the master only. but upon mark's recommendation we on friday installed hba's on the 2 media servers. but since we have an encryption device in the middle. we are not able to tar to the tape from the OS too. so there is a problem with talking to library anyway. 84 might of because of that.

our media server are not clustered. its just the master.

[16ris10]

i have set up liceienceing portal. and i have the certificate too. softcopy. it has two licence number. when i was installing master media. only 1 worked for both. not sure what the other is for. its not for opscenter or any other product. its mentioned in the row something related to netbackup. let me see if i show you certificate in anyway so resolve this two license number thingie in the certificate.

btw sir. we have installed 1 HBA in 2 of the media servers. we have thought about sharing 1 hba for 2 media servers. so all togther 4 would be suported in just 2. master still has 2 hba for the tapes. as of now. we are having problem with the encryption device. and i guess that must have been the reason for 84 status. i tried to tar, and it didn't work. so this is the reason of my judgement. we're in process of resolving network issue too. cause its not pingable too.

[16ris10]

really we can do that? how? our old nbu environment which we're going to migrate over months is running on 6.5.6.

[16ris10]

marriane, mark asked me to open the capacity tab. in the registered tab i see those whcih you are mentioning. :S

see, below.

[16ris10]

mark what marriane has mentioned. i have those licenses registered. pls see the screenshot below.

[Marianne]

**** EDIT ****

We can now see Shared Storage Option in your licensing screenshot.

Have you deleted all devices and re-run device config?

Important to verify that vmoprcmd output shows drives as Shared.
Remember to switch cluster to second node and re-run device config there as well.

If you are still seeing status 84 after correct config, ensure that bptm log folder exists on ALL media servers (including cluster nodes) as well as VERBOSE entry in all servers' vm.conf.
Verify that media manager processes run with '-v'.
bptm as well as system logs will be needed to troubleshoot status 84.

[16ris10]

marianne. sso was always there. its was just that it isn't in the capacity tab. anyway.. marianne & mark. you wanna know what was giving us 84 status? it was the encryption device preventing the master to talk to the tape lib. now that, that device is fixed i can take tape backups from the master itself. :). since mark suggested to have media take backups. i have HBAs on two of the media server now. now i need to configure all media servers to take tape backups too. so sharing need to be done here on media servers. as of now. i dont see shared menioned in the output. but let me delete and reconfiggure. i will create a new topic for this and close this one.. i really see nobody's post to the resolution so wouldn't be marking any post as solution. sorry guys..