Forum Discussion

Yuvi's avatar
Yuvi
Level 5
12 years ago

Tape drive path and drive down issue.

Hi All,

Is there any facility to up the drive path and drive down via cmd prompt in Windows netbackup server of 7.1 version.

And i had 12 drives shared for 4 media servers , for ever full backup one path or a drive came down randomly and i need a idea to up the drive automatically when it comes down and use it without manual interaction ..

  • Marianne is right, you should determine WHY the tape drives keep going down.

    Here is a good check list to help

     

    Verify
    ---------

    1) Update ALL drivers ( tape drives, robot , tape library and HBA)
         -  Nothing should be older than 2010.
         -    ALWAYS obtain drivers from the manufacturer . Symantec has not updated their drivers since 2007
     
    2) Power cycle everything . (Make sure library is down for at least 3 min)


    3) Check and Remove any Phantom Devices -

    From command prompt run :
    set devmgr_show_nonpresent_devices=1
    start devmgmt.msc
    Remove any tape drives, library or robots that are “grayed out”

    REF : http://www.symantec.com/docs/TECH93292

    4) Install the latest device mappings - http://www.symantec.com/docs/TECH129536
         -  Symantec is constantly updating them so be sure to check back for newer versions.
         -  To confirm what version you are on :
             C:\Program Files\VERITAS\Volmgr\bin\tpext -get_dev_mappings_ver


    5)  Check to see if the drives need cleaning.
         -  Even if you think the drives are clean (by looking at the library’s GUI ) , they must been in a 'clean state' on the Netbackup side.
         -  From the Volmgr\bin\tpclean -L
            If the output contains the message, NEEDS CLEANING you must clean the drive.
            You can do clean the drive in the admin console or from a command line. Please review the following tech notes:
            http://www.symantec.com/docs/HOWTO43740 - tpclean
            http://www.symantec.com/docs/TECH17537 - How to use NetBackup's drive cleaning utility, tpclean, with a Library
            http://www.symantec.com/docs/TECH37409 - How to set up automatic tape cleaning in NetBackup Enterprise Server

    6) Does the OS see the devices ?
         - Check the OS Device Monitor and make sure they are listed and do not have any yellow exclamation marks.
         - From a command line :  install_path\Volmgr\bin\scan
                * This tells us what the OS sees

    7) Does Netbackup see the devices configured ?
         - From a command line :  install_path\Volmgr\bin\tpconfig -d
                * This tells us what Netbackup sees as being configured

    8) If using McAfee antivirus , be sure to disable or follow http://www.symantec.com/docs/TECH56658 to make exceptions for Netbackup    

    processes.  IF this is not done, the most common error you will see is a status 84.

    9) For servers that are still running Windows 2003 or ealier, be sure that the ‘Removable storage service’ is stopped and disabled.
     


    Configuration
    ----------------

    If you have had an issue with drives inconsistently going down, it is sometimes best to delete all devices and reconfigure

    NOTE : This works best if it is Windows and SSO is NOT being used.


    10) CONFRIM NO BACKUPS ARE RUNNING

    11) Release all allocations to devices
       
       From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbrbutil -resetAll
       
       (If backups are running, this will cause them to fail. This is why no backups should be running )

    12) Delete all devices - Ref http://www.symantec.com/docs/TECH64327
       
       From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbemmcmd -deletealldevices  
       
         - This will delete ALL devices.  

       From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbemmcmd -deletealldevices -machinename -machinetype media
       
         - This will delete all devices on the media server you specified above .


    13) Use the device configuration wizard to configure your devices.
         - Be sure there are no errors during the wizard or any "unused element" . If so, drag the tape drive to the robot in the wizard.

    14) Inventory the robot(s).



    Trouble Shooting
    -----------------

    15) Look for link state errors on HBA and tape devices

    16) Look in the System Event viewer for any hardware related errors including tape alerts.

    17) Confirm the following folders have been created in the NetBackup logs directory on the server that contains the devices.


    <Install_path>\NetBackup\logs\bptm
    <Install_path>\NetBackup\logs\bpbrm
    <Install_path>\volmgr\debug\daemon
    <Install_path>\volmgr\debug\ltid
    <Install_path>\volmgr\debug\reqlib
    <Install_path>\volmgr\debug\robots

    18) To increase logging :

    Increase verbosity to 5 in the server host properties -> logging.

    Add the line 'VERBOSE' to <Install_path>\volmgr\vm.conf

    create the following touch files:

    <Install_path>\volmgr\ROBOT_DEBUG
    <Install_path>\volmgr\AVRD_DEBUG
    <Install_path>\volmgr\DRIVE_DEBUG
    <Install_path>\volmgr\SSO_DEBUG

    19) Run through issue and review logs.  Be sure to change logging back to 0 and remove VERBSOE from vm.conf

    20) If opening a case with support , include the following :
         - Enviorment Information (use the server where the devices are located at)
           Server Name :
           OS :
           NBU Version
         - Details for the failed job
         - All logs in #17.
         - NBSU - <NetBackup install path>\NetBackup\bin\support\nbsu -c -t
         - Has anything changes since this issue started occurring ?  A recent upgrade ? Install ?  Hardware replacement?


    Additional Tech Notes
    ---------------------

    http://www.symantec.com/docs/HOWTO33906 - NetBackup media management best practices on Windows
    http://www.symantec.com/docs/TECH124594 - Description of Tape Alerts and code definitions
    http://www.symantec.com/docs/TECH169477 - Troubleshooting Robot or Drive Issues in NetBackup

5 Replies

  • Vmoprcmd - up to make the drives/path up/down. Actuall as you have 12 drives shared for 4 media server every dirve will have 4 paths going to media servers. If 1 path goes down then the backups would be still running from other paths, just check if it is showing missing path for that server. If that happens configure it again and should be fine. If it is again and again going down you might need to look for errors on the OS or connectivity.

  • I agree with sazz - 'vmoprcmd -up ...' will do it for you, but there is no automated way in NBU to do this.

    I have seen on the old veritas-bu mailing list that guys had scripts that they scheduled at OS level to check for DOWN drives at regular intervals and up them.

    I have never agreed with this method - there is a REASON why drives are being DOWN'ed. NBU is not randomly DOWN'ing drives just because it 'feels' like it.

    There are MANY reasons why drives are DOWN'ed - rather put all troubleshooting methods in place to find out WHY drives are being DOWN'ed and fix/eliminate issues causing this.

    Create bptm logs on all media servers.
    Add VERBOSE entry to vm.conf on all media servers and restart NBU.
    Device-level errors will by logged in Windows Event Viewer System and Application logs, and to syslog on Unix/Linux servers (e.g. /var/log/messages on Linux).
    I/O errors and TapeAlerts will be logged to bptm log.

  • Marianne is right, you should determine WHY the tape drives keep going down.

    Here is a good check list to help

     

    Verify
    ---------

    1) Update ALL drivers ( tape drives, robot , tape library and HBA)
         -  Nothing should be older than 2010.
         -    ALWAYS obtain drivers from the manufacturer . Symantec has not updated their drivers since 2007
     
    2) Power cycle everything . (Make sure library is down for at least 3 min)


    3) Check and Remove any Phantom Devices -

    From command prompt run :
    set devmgr_show_nonpresent_devices=1
    start devmgmt.msc
    Remove any tape drives, library or robots that are “grayed out”

    REF : http://www.symantec.com/docs/TECH93292

    4) Install the latest device mappings - http://www.symantec.com/docs/TECH129536
         -  Symantec is constantly updating them so be sure to check back for newer versions.
         -  To confirm what version you are on :
             C:\Program Files\VERITAS\Volmgr\bin\tpext -get_dev_mappings_ver


    5)  Check to see if the drives need cleaning.
         -  Even if you think the drives are clean (by looking at the library’s GUI ) , they must been in a 'clean state' on the Netbackup side.
         -  From the Volmgr\bin\tpclean -L
            If the output contains the message, NEEDS CLEANING you must clean the drive.
            You can do clean the drive in the admin console or from a command line. Please review the following tech notes:
            http://www.symantec.com/docs/HOWTO43740 - tpclean
            http://www.symantec.com/docs/TECH17537 - How to use NetBackup's drive cleaning utility, tpclean, with a Library
            http://www.symantec.com/docs/TECH37409 - How to set up automatic tape cleaning in NetBackup Enterprise Server

    6) Does the OS see the devices ?
         - Check the OS Device Monitor and make sure they are listed and do not have any yellow exclamation marks.
         - From a command line :  install_path\Volmgr\bin\scan
                * This tells us what the OS sees

    7) Does Netbackup see the devices configured ?
         - From a command line :  install_path\Volmgr\bin\tpconfig -d
                * This tells us what Netbackup sees as being configured

    8) If using McAfee antivirus , be sure to disable or follow http://www.symantec.com/docs/TECH56658 to make exceptions for Netbackup    

    processes.  IF this is not done, the most common error you will see is a status 84.

    9) For servers that are still running Windows 2003 or ealier, be sure that the ‘Removable storage service’ is stopped and disabled.
     


    Configuration
    ----------------

    If you have had an issue with drives inconsistently going down, it is sometimes best to delete all devices and reconfigure

    NOTE : This works best if it is Windows and SSO is NOT being used.


    10) CONFRIM NO BACKUPS ARE RUNNING

    11) Release all allocations to devices
       
       From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbrbutil -resetAll
       
       (If backups are running, this will cause them to fail. This is why no backups should be running )

    12) Delete all devices - Ref http://www.symantec.com/docs/TECH64327
       
       From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbemmcmd -deletealldevices  
       
         - This will delete ALL devices.  

       From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbemmcmd -deletealldevices -machinename -machinetype media
       
         - This will delete all devices on the media server you specified above .


    13) Use the device configuration wizard to configure your devices.
         - Be sure there are no errors during the wizard or any "unused element" . If so, drag the tape drive to the robot in the wizard.

    14) Inventory the robot(s).



    Trouble Shooting
    -----------------

    15) Look for link state errors on HBA and tape devices

    16) Look in the System Event viewer for any hardware related errors including tape alerts.

    17) Confirm the following folders have been created in the NetBackup logs directory on the server that contains the devices.


    <Install_path>\NetBackup\logs\bptm
    <Install_path>\NetBackup\logs\bpbrm
    <Install_path>\volmgr\debug\daemon
    <Install_path>\volmgr\debug\ltid
    <Install_path>\volmgr\debug\reqlib
    <Install_path>\volmgr\debug\robots

    18) To increase logging :

    Increase verbosity to 5 in the server host properties -> logging.

    Add the line 'VERBOSE' to <Install_path>\volmgr\vm.conf

    create the following touch files:

    <Install_path>\volmgr\ROBOT_DEBUG
    <Install_path>\volmgr\AVRD_DEBUG
    <Install_path>\volmgr\DRIVE_DEBUG
    <Install_path>\volmgr\SSO_DEBUG

    19) Run through issue and review logs.  Be sure to change logging back to 0 and remove VERBSOE from vm.conf

    20) If opening a case with support , include the following :
         - Enviorment Information (use the server where the devices are located at)
           Server Name :
           OS :
           NBU Version
         - Details for the failed job
         - All logs in #17.
         - NBSU - <NetBackup install path>\NetBackup\bin\support\nbsu -c -t
         - Has anything changes since this issue started occurring ?  A recent upgrade ? Install ?  Hardware replacement?


    Additional Tech Notes
    ---------------------

    http://www.symantec.com/docs/HOWTO33906 - NetBackup media management best practices on Windows
    http://www.symantec.com/docs/TECH124594 - Description of Tape Alerts and code definitions
    http://www.symantec.com/docs/TECH169477 - Troubleshooting Robot or Drive Issues in NetBackup

  • Yuvi,

    This was happening to me frequently last year. Follow the steps outlined above to resolve. You'll have to take the outage.  I danced around taking an outage for a while. I even created an alert in OpsCenter to let me know when the drives were down. Nothing worked until I deleted the ghost devices from Device Monitor, rebooted the servers, and ran the Configure Storage Devices wizard.  The failing jobs got to be too much for me.  It's one of the hazards of working in a Windows environment.

  • Hopefully Yuvi will EVENTUALLY get back to us...
    I have even PM'ed him with a long list of his unresolved posts - no response....