NetBackup Basics and how to make YOUR life easier ...
This entry is intended to cover a number of issues that I see time and time again with members posting questions on the forum, and from my experience as a NBU TSE. The advice below is intended ... (1) To make your life easier (2) To assist in making a quicker resolutions (A) Command Issues *************** A common question - Where is the <command> I cannot find it ? The vast majorty of NetBackup binaries are located in the following directories, therfore by simple looking, you should find the command. You could also search, using the search feature in windows, or the find command in Unix. Unix/ Linux /usr/openv/netbackup/bin /usr/openv/netbackup/bin/support /usr/openv/netbackup/bin/admincmd /usr/openv/volmgr/bin There are also so 'scripts' available in /usr/openv/netbackup/bin/goodies Windows <install path>\veritas\netbackup\bin <install path>\veritas\netbackup\bin\support <install path>\veritas\netbackup\bin\admincmd <install path>\veritas\volmgr\bin There are also some 'scripts' available in <install path>\veritas\netbackup\bin\goodies It is even easier, if you add these paths to the PATH variable of the operating system. This allows you to run commands without having to type in the comlete path each time, or, cd to the directory. Unix: Add the directories to the PATH variable in the .profile file in the home directory of the user you log in as. For example ; PATH=$PATH:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin:/usr/openv /netbackup/bin/goodies:/usr/openv/netbackup/bin/support:/usr/openv/volmgr/bin:/u sr/openv/java:/netbackup:/opt/sfw/bin:/netbackup/scripts:/opt/VRTSaz/bin/vxazd:/ opt/VRTSat/bin/:/opt/VRTSaz/bin/:/opt/vormetric/mseo/server/bin:/opt/vormetric/m seo/agent/bin:/usr/openv/db/bin Windows: Add the directories the the PATH user variable (Right click computer, Properties, Advanced System Settings, Environment Variables (Path is under System variable ), For example. C:\Perl\site\bin;C:\Perl\bin;C:\Oracle\Ora92\bin;C:\Program Files\Oracle\jre\1.3.1\bin;C:\Program Files\Oracle\jre\1.1.8\bin;C:\Program Files\RSA SecurID Token Common;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program Files\Symantec\pcAnywhere\;C:\Program Files\QuickTime\QTSystem\;C:\Program Files\Enterprise Vault\EVClient\;C:\Program Files\Symantec\Workspace Virtualization\;C:\Program Files\Veritas\NetBackup\bin;C:\Program Files\Veritas\NetBackup\bin\admincmd (B) Insufficient information ************************ Starting a post with a question such as ... My media is getting frozon during a job, what do I do ? .. is virtually useless, EVERY post should be started with information simailar to these questions ... Initial Information recommended for all issues: 1. The exact error message (if an error is given) 2. During which type of operation does the problem occur ? For example, Backup, Duplication, Vault, Restore etc. 3. At what point in the job does the issue occur, for example, right at the beginning. 4. Which system(s) are involved? For example, Master Server, Media Server - Please provide system names. 5. Please submit the support report from the master and any relevant Media Servers in the environment. 6. Is this a reoccurring issue ? 7. What was the system doing at this time ? 8. Please send in the contents of the details tab in the Activity Monitor for the job. 9. What logs or screenshots are available? 10. Has this worked previously? If so, when was the last time it DID work? 11. Have there been any environmental; system; or configuration changes? 12. When did the problem begin? 13. How often does the problem occur? 14. Is a similar configuration working properly? (C) Is NetBackup the cause ********************** Many NetBackup issues, commonly, Tape/ Library issues and Networking issues are nothing to do with NBU, and the issue cannot be fixed through NetBackup commands. NetBackup often reports and issue, but is nothing to do with the cause. Consider this: If you uninstall NetBackup, what are you left with ??? Hopefully ... An operating system A network Storage devices, tape or disk So. straight away, it can been seen that that these three things are totally separate from NBU. Clearly there is 'interaction' between them, for example, tape drives are 'configured' in NetBackup after all, but NetBAckup does not control these and if the advice given is to investigate from a hardware perspective, or OS issue, this is exactly what needs to be done. In my experience, the majority of Network and Tape /Library issues have nothing to do with NetBackup, and the investigation should start with the vendor / network support. (D) NetBackup Engineers are not hardware engineers. If you have a hardware issue, speak to hardware type people ... (E) Document your system. Any backup environment should be documented, with the documents kept up-to-date. An excellent start to this is to run : nbsu -c -t every day and keep the outputs, (perhaps back them up to tape.) You will note I have not told you exactly where this command is, please see point (A). (F) Logs: On occassion, we will ask for logs ... See this technote, it contains everything you need to know. http://www.symantec.com/docs/TECH75805 When you send logs in, if the issue happens at say 15.00, we don't need the past weeks worth of logs, edit the log down so you send in for example, 14.30 - 16-00. (Note, sometimes the logs will be needed for a longer period, but usually / generally, the logs areound the time of the issue are fine). Also, please rename logs to the process name (NOT for the case of Unified logs (see technote above).) NetBackup defines different (legacy) logs simply by the folder name they are in, which isn't ideal. This means if we ask for say bptm and bpbrm log, both will have exactly the same filename (log.<date>). Please rename these to <process_name>.log.<date> ) The nbcplogs command is very good, it simplifies the collections of logs. Have a go at learning to use this command. http://www.symantec.com/docs/HOWTO43784 You should be able to set up logs without assistance, all that should be required is that we ask, pleased send bptm, bpbrm logs from the media server, or, please send vx logs 111, 116, 117, 118. You should automatically know that these should be gathered, probably with the verbose levels increased. (G) Every log sent in for a problem, must cover the same time period. It is no good sending in bptm from Monday and bpbrm from Tuesday, yes, people actually do this ... (H) If we ask for (example) bpmedialist -m <media id>, please DO NOT send the binary. Please run the command and send the output. I'm sure your binaries are very nice, but I have some of my own - thanks. (I) Our favorite question is "What changed". The usual answer is "Nothing has changed". Generally speaking, NBU does NOT break on it's own, 99% of the time, something HAS changed. I've not yet worked out if people just have no idea who is changeing what and when, or if they just deny making any changes - but if a change has been made, we will find it, it just may take considerable time, adding days or even weeks to the time it takes to resolve an issue. It's better to be truthful and say "I am not aware of any changes" than categorically deny that any changes have been made. (J) If you make a change, write it down somewhere (preferably in a change file on the system) - see (I) above (K) Backups fail, get used to this idea, it is part of being a backup administrator. Do NOT start making changes because of a single failure, this leads to broken systems. (L) If you make a change, and it does not fix the problem, revert it. DO NOT MAKE MULTIPLE CHANGES WITHOUT RECORDING THEM. It can be very very difficult to find an issue, if multiple changes have been made that cause the same symptoms. Consider this : I go home and the light does not work. (Unknown to me, the fuse is broken) I have three possibilities The light switch is broken The light bulb is broken The fuse is broken I decide to change the bulb, unfortunately the bulb I put in is also broken. (I have now introduced another 'fault' that gives the same symptoms as the originally fault). Next I change the fuse, but due to the broken buld, the light does not work. Unknown to me, I have now fixed the originally issue, but I cannot tell due to the faulty bulb fitted, So, from this example you can see how it is possoible to to completely break a system - avoid doing this. (M) NetBackup Engineers are not network engineers, if you have a network issue, go and talk to network type people (N) Just because NetBackup reports and error, DOES NOT mean it caused the error. This is particularly true for network, tape and library issues. (O) Read this - http://www.symantec.com/docs/TECH169477 If your systems/ issue fits a description in the technote, no matter how many times you ask, the answer will be the same. (P) Hardware vendors often blame NBU, especially when it comes to tape/ library issues. This is why the technote in (O) was written, to help prove that NBU is not at fault. (Q) Any backup system MUST be designed before being installed, else it WILL fail. Backup design does NOT start with pressing buttons, it starts with a peice of papaer and a pen. (R) If you want to know how to do something, try looking in the manuals. MASTER Compatibility list: http://www.symantec.com/docs/TECH59978 Master Download List : http://www.symantec.com/docs/TECH65429 Late Breaking News Links : http://www.symantec.com/docs/TECH74904 NBU 6.5 Documentation: http://www.symantec.com/docs/TECH52878 NBU 7.0 Documentation: http://www.symantec.com/docs/TECH126327 NBU 7.1 Documentation: http://www.symantec.com/docs/TECH154178 NBU 7.5 Documentation: http://www.symantec.com/docs/DOC5138 HOWTO Videos: http://www.symantec.com/docs/HOWTO41825 (S) These are the 'minimum' skills thay ANY NetBackup Administrator should have ... 1/ Create and correctly collect NetBackup logs for a given time period ( see http://www.symantec.com/docs/TECH75805 ) 2/ Understand when NetBackup probably isn't the cause (see http://www.symantec.com/docs/TECH169477 ) 3/ Understand Operating system commands to list attached hardware, change /create directory, edit a file, configure network 4/ Understand basic NetBackup concepts : Setting up policies Configure devices via Device Wizard Delete devices Use and undersnd trhe meaning of results in scan/ tpautoconf / scsi_command (hint, see point (O) Understand on which machine the basic NBU processes run Run backups from the GUI Run restores from the GUI Master = nbemm/ bprd/ nbjm/ nbemm Media = bpbrm / bptm Client = bpcd / bpbkar / tar Understand the basic principles of a catalog backup and how to set it up Understand that NetBackup DOES NOT CONTROL your network, we ONLY use what we are given Understand that NetBackup does NOT write/ read to/ from tape - this is done by the operating system Understand the idea of 'density' (eg. hcart, hcart2 etc ...) Understand barcode rules, media id generation rules and how to set them up Understand how to inventory a library Understand how to use the robtest command and the meaning of the output Undersnad the difference between multi-streaming and multi-plexing Understand the idea of storage units Understand the idea of media pools If you are stuck on any of these, first thing, see point (R) (T) It is not the job of people on the forum (or Symantec support) to read the manuals for you (U) If something is unsupported (ie. not in the manuals) DO NOT DO IT. (V) When you are asked to do steps 1, 2, 3, 4, 5 Please ensure you do ALL the steps, and not just the ones you feel like doing. We don't ask for the steps because we have nothing better to do, we ask for them because we need ALL of them. (W) Please do not make changes just because you can. Any change to a system should be fully justified, researched, and recorded accurately in the system documentation. The system documentation should be so detailed, it contains every change made and when, and the current setting of EVERYTHING that has been changed from the default - are you doing this ? Regards, Martin
Frequently Asked Questions on NetBackup Accelerator
NetBackup Accelerator is an exciting feature introduced in NetBackup 7.5 and NetBackup Appliances software version 2.5. This blog is not a substitute for NetBackup documentation. NetBackup Accelerator transforms the way organizations do backups, I am compiling a list of frequently asked questions from NetBackup Community forums and providing answers.
Day and Night: NetBackup 5220 vs. that other product
I have worked for NetBackup Technical Support as a Staff engineer for long time. Customers don’t call us when things are working great; they need our help when they are not getting what they want from the product. Naturally, a seasoned engineer in support is continuously dealing with weaknesses or defects in the product more than its strengths.
General database backup error troubleshooting
When troubleshooting database backups, it is important to remember that a database backup is in principle an user initiated backup, so the backup is actually started from the client by the script/database backup api. The database backup is more sensitive to name resolution than file backup, partly because of the above Places to look for information/things to do: The activity monitor, probably where you would see the error first. Talk with your DBA <- This is a very very important step!! The bphdb log folder which gives the script exit status, environment information and other things there can help figure out what the issue is. The dbclient log folder with 777 permission which gives detailed information about the backup done by the database api e.g. rman The database api's log e.g. rman messagelog The database's log e.g the alert log for Oracle The OS system logs, but often this a copy of the messages from the database's logs If none of this gives an indication of the problem run the script from a prompt if possible, as this might give an output not catched by any of the logs In case of layered scripts like Oracle, first run the cmd/shell script as administrator/root, then run the rman script as administrator/oracle user Sometimes it is necessary to run the scripts one line of time to catch the error In my experience this approach enables you to find most of the issues with the database backups