This entry is intended to cover a number of issues that I see time and time again with members posting questions on the forum, and from my experience as a NBU TSE.
The advice below is intended ...
(1) To make your life easier
(2) To assist in making a quicker resolutions
A common question - Where is the <command> I cannot find it ?
The vast majorty of NetBackup binaries are located in the following directories, therfore by simple looking, you should find the command. You could also search, using the search feature in windows, or the find command in Unix.
There are also so 'scripts' available in /usr/openv/netbackup/bin/goodies
There are also some 'scripts' available in <install path>\veritas\netbackup\bin\goodies
It is even easier, if you add these paths to the PATH variable of the operating system. This allows you to run commands without having to type in the comlete path each time, or, cd to the directory.
Add the directories to the PATH variable in the .profile file in the home directory of the user you log in as.
For example ;
Add the directories the the PATH user variable (Right click computer, Properties, Advanced System Settings, Environment Variables (Path is under System variable ), For example.
C:\Perl\site\bin;C:\Perl\bin;C:\Oracle\Ora92\bin;C:\Program Files\Oracle\jre\1.3.1\bin;C:\Program Files\Oracle\jre\1.1.8\bin;C:\Program Files\RSA SecurID Token Common;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files\Symantec\pcAnywhere\;C:\Program Files\QuickTime\QTSystem\;C:\Program Files\Enterprise Vault\EVClient\;C:\Program Files\Symantec\Workspace Virtualization\;C:\Program Files\Veritas\NetBackup\bin;C:\Program Files\Veritas\NetBackup\bin\admincmd
Starting a post with a question such as ...
My media is getting frozon during a job, what do I do ?
.. is virtually useless, EVERY post should be started with information simailar to these questions ...
Initial Information recommended for all issues:
1. The exact error message (if an error is given)
2. During which type of operation does the problem occur ? For example, Backup, Duplication, Vault, Restore etc.
3. At what point in the job does the issue occur, for example, right at the beginning.
4. Which system(s) are involved? For example, Master Server, Media Server - Please provide system names.
5. Please submit the support report from the master and any relevant Media Servers in the environment.
6. Is this a reoccurring issue ?
7. What was the system doing at this time ?
8. Please send in the contents of the details tab in the Activity Monitor for the job.
9. What logs or screenshots are available?
10. Has this worked previously? If so, when was the last time it DID work?
11. Have there been any environmental; system; or configuration changes?
12. When did the problem begin?
13. How often does the problem occur?
14. Is a similar configuration working properly?
Is NetBackup the cause
Many NetBackup issues, commonly, Tape/ Library issues and Networking issues are nothing to do with NBU, and the issue cannot be fixed through NetBackup commands.
NetBackup often reports and issue, but is nothing to do with the cause.
If you uninstall NetBackup, what are you left with ??? Hopefully ...
An operating system
Storage devices, tape or disk
So. straight away, it can been seen that that these three things are totally separate from NBU.
Clearly there is 'interaction' between them, for example, tape drives are 'configured' in NetBackup after all, but NetBAckup does not control these and if the advice given is to investigate from a hardware perspective, or OS issue, this is exactly what needs to be done.
In my experience, the majority of Network and Tape /Library issues have nothing to do with NetBackup, and the investigation should start with the vendor / network support.
NetBackup Engineers are not hardware engineers. If you have a hardware issue, speak to hardware type people ...
Document your system.
Any backup environment should be documented, with the documents kept up-to-date. An excellent start to this is to run :
nbsu -c -t every day and keep the outputs, (perhaps back them up to tape.)
You will note I have not told you exactly where this command is, please see point (A).
On occassion, we will ask for logs ...
See this technote, it contains everything you need to know.
When you send logs in, if the issue happens at say 15.00, we don't need the past weeks worth of logs, edit the log down so you send in for example, 14.30 - 16-00. (Note, sometimes the logs will be needed for a longer period, but usually / generally, the logs areound the time of the issue are fine).
Also, please rename logs to the process name (NOT for the case of Unified logs (see technote above).)
NetBackup defines different (legacy) logs simply by the folder name they are in, which isn't ideal. This means if we ask for say bptm and bpbrm log, both will have exactly the same filename (log.<date>). Please rename these to <process_name>.log.<date> )
The nbcplogs command is very good, it simplifies the collections of logs.
Have a go at learning to use this command.
You should be able to set up logs without assistance, all that should be required is that we ask, pleased send bptm, bpbrm logs from the media server, or, please send vx logs 111, 116, 117, 118. You should automatically know that these should be gathered, probably with the verbose levels increased.
Every log sent in for a problem, must cover the same time period. It is no good sending in bptm from Monday and bpbrm from Tuesday, yes, people actually do this ...
If we ask for (example) bpmedialist -m <media id>, please DO NOT send the binary. Please run the command and send the output.
I'm sure your binaries are very nice, but I have some of my own - thanks.
Our favorite question is "What changed". The usual answer is "Nothing has changed".
Generally speaking, NBU does NOT break on it's own, 99% of the time, something HAS changed.
I've not yet worked out if people just have no idea who is changeing what and when, or if they just deny making any changes - but if a change has been made, we will find it, it just may take considerable time, adding days or even weeks to the time it takes to resolve an issue.
It's better to be truthful and say "I am not aware of any changes" than categorically deny that any changes have been made.
If you make a change, write it down somewhere (preferably in a change file on the system) - see (I) above
Backups fail, get used to this idea, it is part of being a backup administrator.
Do NOT start making changes because of a single failure, this leads to broken systems.
If you make a change, and it does not fix the problem, revert it. DO NOT MAKE MULTIPLE CHANGES WITHOUT RECORDING THEM.
It can be very very difficult to find an issue, if multiple changes have been made that cause the same symptoms.
Consider this :
I go home and the light does not work. (Unknown to me, the fuse is broken)
I have three possibilities
The light switch is broken
The light bulb is broken
The fuse is broken
I decide to change the bulb, unfortunately the bulb I put in is also broken.
(I have now introduced another 'fault' that gives the same symptoms as the originally fault).
Next I change the fuse, but due to the broken buld, the light does not work. Unknown to me, I have now fixed the originally issue, but I cannot tell due to the faulty bulb fitted,
So, from this example you can see how it is possoible to to completely break a system - avoid doing this.
NetBackup Engineers are not network engineers, if you have a network issue, go and talk to network type people
Just because NetBackup reports and error, DOES NOT mean it caused the error. This is particularly true for network, tape and library issues.
Read this - http://www.symantec.com/docs/TECH169477
If your systems/ issue fits a description in the technote, no matter how many times you ask, the answer will be the same.
Hardware vendors often blame NBU, especially when it comes to tape/ library issues. This is why the technote in (O) was written, to help prove that NBU is not at fault.
Any backup system MUST be designed before being installed, else it WILL fail. Backup design does NOT start with pressing buttons, it starts with a peice of papaer and a pen.
If you want to know how to do something, try looking in the manuals.
MASTER Compatibility list: http://www.symantec.com/docs/TECH59978
Master Download List : http://www.symantec.com/docs/TECH65429
Late Breaking News Links : http://www.symantec.com/docs/TECH74904
NBU 6.5 Documentation: http://www.symantec.com/docs/TECH52878
NBU 7.0 Documentation: http://www.symantec.com/docs/TECH126327
NBU 7.1 Documentation: http://www.symantec.com/docs/TECH154178
NBU 7.5 Documentation: http://www.symantec.com/docs/DOC5138
HOWTO Videos: http://www.symantec.com/docs/HOWTO41825
These are the 'minimum' skills thay ANY NetBackup Administrator should have ...
1/ Create and correctly collect NetBackup logs for a given time period ( see http://www.symantec.com/docs/TECH75805 )
2/ Understand when NetBackup probably isn't the cause (see http://www.symantec.com/docs/TECH169477 )
3/ Understand Operating system commands to list attached hardware, change /create directory, edit a file, configure network
4/ Understand basic NetBackup concepts :
Setting up policies
Configure devices via Device Wizard
Use and undersnd trhe meaning of results in scan/ tpautoconf / scsi_command (hint, see point (O)
Understand on which machine the basic NBU processes run
Run backups from the GUI
Run restores from the GUI
Master = nbemm/ bprd/ nbjm/ nbemm
Media = bpbrm / bptm
Client = bpcd / bpbkar / tar
Understand the basic principles of a catalog backup and how to set it up
Understand that NetBackup DOES NOT CONTROL your network, we ONLY use what we are given
Understand that NetBackup does NOT write/ read to/ from tape - this is done by the operating system
Understand the idea of 'density' (eg. hcart, hcart2 etc ...)
Understand barcode rules, media id generation rules and how to set them up
Understand how to inventory a library
Understand how to use the robtest command and the meaning of the output
Undersnad the difference between multi-streaming and multi-plexing
Understand the idea of storage units
Understand the idea of media pools
If you are stuck on any of these, first thing, see point (R)
It is not the job of people on the forum (or Symantec support) to read the manuals for you
If something is unsupported (ie. not in the manuals) DO NOT DO IT.
When you are asked to do steps 1, 2, 3, 4, 5 Please ensure you do ALL the steps, and not just the ones you feel like doing.
We don't ask for the steps because we have nothing better to do, we ask for them because we need ALL of them.
Please do not make changes just because you can. Any change to a system should be fully justified, researched, and recorded accurately in the system documentation.
The system documentation should be so detailed, it contains every change made and when, and the current setting of EVERYTHING that has been changed from the default - are you doing this ?