status: 83: media open error

Your Dedup Environment is down.  Maybe it is out of free space, or Content Router queue processing has failed 5 times.

Does the following command run from the Appliance:

# /usr/openv/pdde/pdcr/bin/crcontrol --dsstat

# /usr/openv/pdde/pdcr/bin/crcontrol --getmode

Is the Disk Volume up:

# /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk -U

Is the Disk Pool Up:

# /usr/openv/netbackup/bin/admincmd/nbdevquery -listdp -stype PureDisk -U

Virutal ed,

I thought i had it docuemnted on the new maintance password for the appliance after the 2.1 update but apparently not (ghrr).  So that mean i am unable to try the commands you sent me.

I do know that the disk volume is up and it appears that my appliances are not full. They are 40tb appliances and my netbackup console and the admin page for the appliance only shows them only using about 20%.

 

The wierd thing is i can restart all the services on the appliances and my backup start working as normal for about two streams and then everything goes back to status 83.  I dont know buddy but thank you for helping.

I contacted support and 5 hours later they called me and asked that send them some logs.  But they have yet to email me what they want? 

I found some errors on the appliance saying the the pure_disk volume has been marked down.  So Ed your right but i dont know how to stop this from happening.

 

Im guessing this is all caused by me upgrading to 7.1.0.2.  Everything was working fine until i did that two weeks ago.  dangit!!

Stanley

It may just be a co-incidence as the appliances have bene know to suddently have issues and they do tend to need a little tuning.

Obviously to do that you will need to actually log into it via putty (you can try root or admin and P@ssw0rd) as it may have been re-set during the patch.

Once logged in go to the Support menu, then Maintenance, then type elevate - this will give you file level access to the appliance

Now work through the following - lots here so dont skip anything ...

First change the keep alive time if they are not set to 510, 3 and 3:

# cat /proc/sys/net/ipv4/tcp_keepalive_time
  7200
# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75
# cat /proc/sys/net/ipv4/tcp_keepalive_probes
9

change the setting:

# echo 510 > /proc/sys/net/ipv4/tcp_keepalive_time
# echo 3 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 3 > /proc/sys/net/ipv4/tcp_keepalive_probes

To keep persistent after a reboot see below – use vi editor:

The changes would be rendered persistent with an addition such as the following to /etc/sysctl.conf

## Keepalive at 8.5 minutes

# start probing for heartbeat after 8.5 idle minutes (default 7200 sec)
net.ipv4.tcp_keepalive_time=510

# close connection after 4 unanswered probes (default 9)
net.ipv4.tcp_keepalive_probes=3 

# wait 45 seconds for reponse to each probe (default 75
net.ipv4.tcp_keepalive_intvl=3

Don’t need a restart to take effect - and then run : chkconfig boot.sysctl on

Second create this file with the value of 800 inside:

/usr/openv/netbackup/db/config/DPS_PROXYDEFAULTRECVTMO

Third enable compression for de-dupe:

/usr/openv/lib/ost-plugins/pd.conf

Edit line so Compression =1

Fourth make sure that the automatic tuning has been done (it should run on first boot up but has been know not to):

cd to: /opt/NBUAppliance/scripts/
type the following once in that directory:
../bin/perl tune.pl

Fifth go to usr/openv/netbackup/db/config and make sure that there are no SIZE or NUMBER DATA BUFFER files present - if there are make a note of their values and then rename them (you can try putting these back later if you have any performance issues

Now type exit and exit to come back to the normal menu and issue a reboot command

See how it goes - i have found that all of these setting together make the applicances totally stable - just the BUFFERS i have not checked recently as 2.1a adds some in i believe

Hope this helps

Mark,

Thank you so much for the detailed instructions.  They are spot on!!!  I am rebooting my first appliance now and will run some backups to see what happens. 

Every time i upgrade my master it always takes about a week of doing these "performance tuning" steps before everything starts working again.

Once again thank you so much for taking the time to type these instructions out.  The password you posted is correct and i am going to document that for later use.

Mark,

I double checked that all the settings you provided stayed after the reboot and then attempted to run some backups.  They are still failing with status 213 and 2074s.  I am at a loss on what is causing this.  i have requested that my ticket be escalated so hopefully i will get with support here soon and figure out whats going on. 

I will update after i hear something back from support.

After you had done this work did you UP the Volume Pools as they may still have been marked down in NetBackup.

They are a bit odd sometimes as the GUI shows them as UP when they aren't so set them down and then set them back up in the GUI and try again.

Ed,

I tried the commands you suggested but they did not run.  Said they were invalid.

Also my appliance is at 2.1a patch level.  I may have forgotten to mention that.

Which commands were invalid?

I guessed it was 2.1a as the password worked.

Also down the pool and up it again to see if that works

i tried all the commands and i get "command not found" message. 

Yes P@ssw0rd is the new root password for 2.1a

 

I have restarted my appliances serveral times.  is this the same as "downing" the pools?