on 02-10-2013 12:22 PM
The following 20 steps will help analyze, correct and troubleshoot many different device issues in Window Enviorments.
Just a few Examples (but not limited to ) :
- Status codes : 84 , 96 , 2009
- Drives keep going down or randomly go down.
- No drives available
V E R I F Y
1) Update ALL drivers ( tape drives, robot , tape library and HBA)
- Nothing should be older than 2010.
- ALWAYS obtain drivers from the manufacturer . Symantec has not updated their drivers since 2007
2) Power cycle everything . (Make sure library is down for at least 3 min)
3) Check and Remove any Phantom Devices -
From command prompt run :
set devmgr_show_nonpresent_devices=1
start devmgmt.msc
Remove any tape drives, library or robots that are “grayed out”
REF : http://www.symantec.com/docs/TECH93292
4) Install the latest device mappings - http://www.symantec.com/docs/TECH129536
- Symantec is constantly updating them so be sure subscribe to the tech note for to receive alerts when a new version is released.
- To confirm what version you are on :
C:\Program Files\VERITAS\Volmgr\bin\tpext -get_dev_mappings_ver
5) Check to see if the drives need cleaning.
- Even if you think the drives are clean (by looking at the library’s GUI ) , they must been in a 'clean state' on the Netbackup side.
- From the Volmgr\bin\tpclean -L
If the output contains the message, NEEDS CLEANING you must clean the drive.
You can clean the drive in the admin console or from a command line. Please review the following tech notes:
http://www.symantec.com/docs/HOWTO43740 - tpclean
http://www.symantec.com/docs/TECH17537 - How to use NetBackup's drive cleaning utility, tpclean, with a Library
http://www.symantec.com/docs/TECH37409 - How to set up automatic tape cleaning
6) Does the OS see the devices ?
- Check the OS Device Monitor and make sure they are listed and do not have any yellow exclamation marks.
- From a command line : install_path\Volmgr\bin\scan
* This tells us what the OS sees
* IF the OS does not see the devices, Netbackup will not see them. Please contact the hardware manufacturer
7) Does Netbackup see the devices configured ?
- From a command line : install_path\Volmgr\bin\tpconfig -d
* This tells us what Netbackup sees as being configured
8) If using McAfee antivirus , be sure to disable or follow http://www.symantec.com/docs/TECH56658 to make exceptions for Netbackup processes. IF this is not done, the most common error you will see is a status 84.
9) For servers that are still running Windows 2003 or earlier, be sure that the ‘Removable storage service’ is stopped and disabled.
C O N F I G U R A T I O N
If you have had an issue with drives inconsistently going down, it is sometimes best to delete all devices and reconfigure NOTE : This works best if it is Windows and SSO is NOT being used.
10) CONFIRM NO BACKUPS ARE RUNNING
11) Release all allocations to devices
From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbrbutil -resetAll
(If backups are running, this will cause them to fail. This is why no backups should be running )
12) Delete all devices - Ref http://www.symantec.com/docs/TECH64327
OPTION 1 – Delete ALL devices.
From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbemmcmd -deletealldevices
OPTION 2 – Delete all devices on a specified media server.
From the command line: Install_path\Veritas\NetBackup\bin\admincmd>nbemmcmd -deletealldevices -machinename <server name> -machinetype media
13) Use the device configuration wizard to configure your devices.
- Be sure there are no errors during the wizard or any "unused element" . If so, drag the tape drive to the robot in the wizard.
14) Inventory the robot(s).
T R O U B L E S H O O T I N G
15) Look for link state errors on HBA and tape devices
16) Look in the System Event viewer for any hardware related errors including tape alerts.
17) Confirm the following folders have been created in the NetBackup logs directory on the server that contains the devices.
<Install_path>\NetBackup\logs\bptm
<Install_path>\NetBackup\logs\bpbrm
<Install_path>\volmgr\debug\daemon
<Install_path>\volmgr\debug\ltid
<Install_path>\volmgr\debug\reqlib
<Install_path>\volmgr\debug\robots
18) To increase logging :
Increase verbosity to 5 in the server host properties -> logging.
Add the line 'VERBOSE' to <Install_path>\volmgr\vm.conf
create the following touch files:
<Install_path>\volmgr\ROBOT_DEBUG
<Install_path>\volmgr\AVRD_DEBUG
<Install_path>\volmgr\DRIVE_DEBUG
<Install_path>\volmgr\SSO_DEBUG
19) Run through issue and review logs. Be sure to change logging back to 0 and remove VERBSOE from vm.conf
20) Information to gather to send to support for troubleshooting :
- Environment Information (use the server where the devices are located at)
Server Name :
OS :
NBU Version
- Details for the failed job
- All logs in #17.
- NBSU - <NetBackup install path>\NetBackup\bin\support\nbsu -c -t
- Has anything changes since this issue started occurring ? A recent upgrade ? Install ? Hardware replacement?
Additional Tech Notes
---------------------
http://www.symantec.com/docs/HOWTO33906 - NetBackup media management best practices on Windows
http://www.symantec.com/docs/TECH124594 - Description of Tape Alerts and code definitions
http://www.symantec.com/docs/TECH169477 - Troubleshooting Robot or Drive Issues in NetBackup
http://www.symantec.com/docs/DOC5186 - NetBackup 7.5 Device Configuration Guide
http://www.symantec.com/docs/DOC3656 - NetBackup 7.1 Device Configuration Guide
very good! just a driver down question ,study. create the following touch files:(good)
Hi Jason, are you asking what you should look for in those logs ?
yes, would you give me some suggest?
tks,Dyneshia.
In most logs, you can look for :
<16>
<32>
fail
error
ERR
WRN
cannot
Here are some "compelte" common errors :
<Install_path>\volmgr\debug\daemon
<16> peer_hostname: gethostbyaddr() h_errno: (11004)
<Install_path>\volmgr\debug\ltid
<16> emmlib_SetRobotStatus: (0) UpdateLibrary failed, emmError = 2007087, nbError = 0
<16> emmlib_UpdateMachineState: (0) CORBA call threw exception.
<16> SendEmmHeartbeat: (-) Translating EMM_ERROR_CorbaException(3000004) to 334 in the device management context
<16> WriteEntry: (-) Translating EMM_ERROR_DriveAlreadyAllocated(2005004) to 258 in the device management context
<Install_path>\volmgr\debug\robots
<6> MultipleEjectLoop: eject aborted, exit_status = 44
<5> MultipleEjectLoop: eject_aborted
<3> move_medium: Invalid operation in move_medium(), op = 24
If the article helped you be sure to vote it up :)
very good post , would you please write a public technote as well just as this one .
Definitely :) Its been on my to do list.
so good ! I met a difficult about my Netbackup on media server recently.But I cann't solve it by my effort.I describe the problem following:
my operation system is windows 2003 on media server. my tapes is on EMC DD670.I give two drives to media server(media server name is qzmail).The hardware scan for the drives on operation system is right.But in the c:\Program Files\Verteris\Volmgr\bin\tpconfig -d ,the drives are have many paths. Then,I use the command like c:\Program Files\Vertris\Volmgr\bin\tpconfig -delete -drpath -path {1,0,0,14} -asciimame IBM.ULIRIUM-TD.043 to delete the wrong path. then tpconfig -d ,the drives and the paths are right ,but scan.exe show that the wrong path are not delete. The problem is how to produce? And how to solve it.I am so crazy with it and have no idea about it.
It seems like you are using SSO ? Please move topic to the forum. I tried to do it for you, however could not.
https://www-secure.symantec.com/connect/backup-and-recovery/forums/netbackup
Yes,I am using SSO.Thank you for your help.Now,I solved the problem by rebooting the operating system,then the resault of scan is right.Maybe something temp files of theoperating system need to eliminate.Though it is difficult for the production system,sometime rebooting is a very good method to solve problems.
Aesome Post dyneshia , 1 stop shop for tape drive issue fix , but we also need ACSLS issues to posted in.
cheer,
Rajesh Kumar
how to troubleshoot the drive down issue?