20 Steps to Analyze , Correct and Troubleshoot Device issues in Windows
The following 20 steps will help analyze, correct and troubleshoot many different device issues in Window Enviorments.
Just a few Examples (but not limited to ) :
- Status codes : 84 , 96 , 2009
- Drives keep going down or randomly go down.
- No drives available
V E R I F Y
1) Update ALL drivers ( tape drives, robot , tape library and HBA)
- Nothing should be older than 2010.
- ALWAYS obtain drivers from the manufacturer . Symantec has not updated their drivers since 2007
2) Power cycle everything . (Make sure library is down for at least 3 min)
3) Check and Remove any Phantom Devices -
From command prompt run :
set devmgr_show_nonpresent_devices=1
start devmgmt.msc
Remove any tape drives, library or robots that are “grayed out”
REF : http://www.symantec.com/docs/TECH93292
4) Install the latest device mappings - http://www.symantec.com/docs/TECH129536
- Symantec is constantly updating them so be sure subscribe to the tech note for to receive alerts when a new version is released.
- To confirm what version you are on :
C:\Program Files\VERITAS\Volmgr\bin\tpext -get_dev_mappings_ver
5) Check to see if the drives need cleaning.
- Even if you think the drives are clean (by looking at the library’s GUI ) , they must been in a 'clean state' on the Netbackup side.
- From the Volmgr\bin\tpclean -L
If the output contains the message, NEEDS CLEANING you must clean the drive.
You can clean the drive in the admin console or from a command line. Please review the following tech notes:
http://www.symantec.com/docs/HOWTO43740 - tpclean
http://www.symantec.com/docs/TECH17537 - How to use NetBackup's drive cleaning utility, tpclean, with a Library
http://www.symantec.com/docs/TECH37409 - How to set up automatic tape cleaning
6) Does the OS see the devices ?
- Check the OS Device Monitor and make sure they are listed and do not have any yellow exclamation marks.
- From a command line : install_path\Volmgr\bin\scan
* This tells us what the OS sees
* IF the OS does not see the devices, Netbackup will not see them. Please contact the hardware manufacturer
7) Does Netbackup see the devices configured ?
- From a command line : install_path\Volmgr\bin\tpconfig -d
* This tells us what Netbackup sees as being configured
8) If using McAfee antivirus , be sure to disable or follow http://www.symantec.com/docs/TECH56658 to make exceptions for Netbackup processes. IF this is not done, the most common error you will see is a status 84.
9) For servers that are still running Windows 2003 or earlier, be sure that the ‘Removable storage service’ is stopped and disabled.
C O N F I G U R A T I O N
If you have had an issue with drives inconsistently going down, it is sometimes best to delete all devices and reconfigure NOTE : This works best if it is Windows and SSO is NOT being used.
10) CONFIRM NO BACKUPS ARE RUNNING
11) Release all allocations to devices
From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbrbutil -resetAll
(If backups are running, this will cause them to fail. This is why no backups should be running )
12) Delete all devices - Ref http://www.symantec.com/docs/TECH64327
OPTION 1 – Delete ALL devices.
From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbemmcmd -deletealldevices
OPTION 2 – Delete all devices on a specified media server.
From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbemmcmd -deletealldevices -machinename <server name> -machinetype media
13) Use the device configuration wizard to configure your devices.
- Be sure there are no errors during the wizard or any "unused element" . If so, drag the tape drive to the robot in the wizard.
14) Inventory the robot(s).
T R O U B L E S H O O T I N G
15) Look for link state errors on HBA and tape devices
16) Look in the System Event viewer for any hardware related errors including tape alerts.
17) Confirm the following folders have been created in the NetBackup logs directory on the server that contains the devices.
<Install_path>\NetBackup\logs\bptm
<Install_path>\NetBackup\logs\bpbrm
<Install_path>\volmgr\debug\daemon
<Install_path>\volmgr\debug\ltid
<Install_path>\volmgr\debug\reqlib
<Install_path>\volmgr\debug\robots
18) To increase logging :
Increase verbosity to 5 in the server host properties -> logging.
Add the line 'VERBOSE' to <Install_path>\volmgr\vm.conf
create the following touch files:
<Install_path>\volmgr\ROBOT_DEBUG
<Install_path>\volmgr\AVRD_DEBUG
<Install_path>\volmgr\DRIVE_DEBUG
<Install_path>\volmgr\SSO_DEBUG
19) Run through issue and review logs. Be sure to change logging back to 0 and remove VERBSOE from vm.conf
20) Information to gather to send to support for troubleshooting :
- Environment Information (use the server where the devices are located at)
Server Name :
OS :
NBU Version
- Details for the failed job
- All logs in #17.
- NBSU - <NetBackup install path>\NetBackup\bin\support\nbsu -c -t
- Has anything changes since this issue started occurring ? A recent upgrade ? Install ? Hardware replacement?
Additional Tech Notes
---------------------
http://www.symantec.com/docs/HOWTO33906 - NetBackup media management best practices on Windows
http://www.symantec.com/docs/TECH124594 - Description of Tape Alerts and code definitions
http://www.symantec.com/docs/TECH169477 - Troubleshooting Robot or Drive Issues in NetBackup
http://www.symantec.com/docs/DOC5186 - NetBackup 7.5 Device Configuration Guide
http://www.symantec.com/docs/DOC3656 - NetBackup 7.1 Device Configuration Guide