Legacy log files
RHEL 5.9, 64 bit Master Server. Windows 2008 Standard 64 bit Media Servers. NBU 7.5.0.6 all around.
1. Where can I get good explanations of the kind of information I can expect to find in a given legacy log?
2. Where can I find detailed information about where and what log files to retrieve when troubleshooting a problem.
i.e. For problem "A" look in logs x, y and z on your client, a and b on the Media server and c on the Master. (I understand that I will have to make determinations of the kind of entries I'm looking for given "problem "A"" but once I've determined what kinds of log entries I'm after I need to know what log files to look in.)
3. What log files should only exist / relavent on a client, on a MS or on a Master server, 2 of the three or all three.
I've read through NBU manuals and guides and searched the web but haven't been able to find this kind of specific information only what process / daemon a log file belongs to and if lucky some very brief explanation (four to ten words). I believe the info must be there but I'm spinning an lot of time trying to dig it out myself.
This has frustrated me for some time and what has prompted this post is that I am trying to definitively determine what the interfaces IPs and host names are used between client and MS plus between client and Master for specific backups. So help with this in the short term would be valuable.
Terry
These are pretty much impossible questions
1. Where can I get good explanations of the kind of information I can expect to find in a given legacy log?
It doesn't exist - NBU is too complex to document like ths. The best advice is to know which processes communicate with each other (see troubleshooting guide for process flows) and then when a failure happens you know (hopefuly) where abouts to start looking. Activity monitor often shows which process failed. Sometimes this can be misleading eg. backups fail with status 6, but the real status code will be somewhere else ... So an error can happen in one log, and then filter through others, but only one of the later failures is reported in act monitor. It can be a game of seek and ye will hopfully find ...
2. Where can I find detailed information about where and what log files to retrieve when troubleshooting a problem.
See answer above - it doesn't exsist in any offical documentation. Again use process flows to see what talks to what as a starting point. If I'm stuck, I usually enable everything on test machnine, run the job and then simpy see which logs have changed. It's not perfect as some logs (eg bpdbm) are constantly chaging but it's another way to get some idea, and this genearlly doesn't work on the vx logs as they constantly channge (well, the common ones do).
3. What log files should only exist / relavent on a client, on a MS or on a Master server, 2 of the three or all three.
It depends which processes you want to monitor.
On a master I would keep nbemm, nbjm, nbrb, nbpem, bpdbm and bprd as minimum. However bpdbm / jm /rb/ emm get very large at high verbose levels so it's usually not possible to leave them running.
Then again, you could add in mds (which logs into emm) .. but then again, this isn't always needed and makes the log bigger. vx logs 156 and 137 are often needed for comms type issues (also on media) but it would be impossible to turn these up for long on a busy system as they log into many of the other logs and make them massive (I'm talking GBs of logs)
Then if you use other features, eg. SLP then a load of other logs come into play,
For SLP I could list about 18 logs that you could enable to monitor SLPs across the master and media - but depending on where the problem is, you might only need maybe 3 or 4, so as you see, this is an impossible question to answer.
media - bpdm, bptm, bpbrm tpcommand ltid roobots
client - tar , bpbkar, bpcd
Again, certainly for the media server you could add in a load of vx logs depending on what features you use.
Unfortunately the answer really is just experience - I'd advise to tackle one issue at a time, and then make a note of which logs you needed . Hopefully the details above will help a little.
- Hi Terry, I think if you ask these questions, you would get many different answers. Sure ... you can probably do this without logs bptestbpcd -verbose -host
This would show the connection Eg run on the master with the media as the would show the ip going out of the aster to the ip going into the media. Can also use -host instead of -client. Can do this also from media to client Client to media or master will be fun though, as I don't think the client has the bptestbpcd binary - so log time ... Exact logs differ between versions, as connections at 7.5 (in fact I think from 6.5.6) go via PBX, previously for example we connected direct to bpcd on client for example. I'll give you the main logs of processes that talk to each other between the servers - a good keyword to search is 'connect' (upper case in unix) This should cover 6.x as well as 7.x Backup Master nbjm / bpdbm / pbx / vnetd Media bpbrm / bptm / pbx / vnetd Client bpcd / bpbkar / pbx / vnetd Restore - similar to above but tar on client instead of bpbkar - but connection from client comes into bprd on master if user backup is started from client It gets a bit confusing ... in older versions of NBU the media server process bpbrm would connect direct to bpcd on the client for example. Now, it connects to PBX which hands off to bpcd, so bpbrm still talks to bpcd, but 'is introduced to it' via PBX (one reason for this was to reduce the number of ports needing to be open in firewalls). So, think of pbx as a 'dating service' ... 'Mr. bpbrm, I'd like to introduce you to Miss. bpcd ' However, you would still see 'Connect from 10.20.30.40 in bpcd ... (I think, sorry brain dead tonight ...). To be honest, I don't usually end up looking in pbx of vnetd much, usually get the info in the other logs. PBX has a different product id and oid -p 50936 -o 103 Its Debug levels are different as well, by default it's set to 10 which I think is max. My 2c....
The most important part of troubleshooting and creating logs, is to understand the process flow. This can be found in Appendix A of the Troubleshooting Guide (along with definition of processes that you found).
If we know that bpbrm on the media server connects to bpcd on the client at the start of the backup, then we understand that these are the logs where we will find IP addresses used by media server and client for connection and backup. The bpcd log will show us how client resolves media server IP to hostname and compares with Server entries. So, these 2 logs are needed when we troubleshoot connection issues.
If we know that bpbkar on client sends data to bptm on the media server and metadata to bpbrm on the media server, we understand that these are the logs that we need to troubleshoot backups that fail after data transfer has started.If we know that all restore requests is firstly sent to bprd on the master, then we understand that troubleshooting of restores starts with this log.
Once restore parameters have been confirmed with bpdbm on the master, bprd hands the restore instruction to bpbrm on the media server, from where connection is made to bpcd on the client, which starts tar process and bpbrm in the meantime starts bptm to read data and sends data to tar on the client. So, now we know which logs we need to troubleshoot restores.Here is how I use logs: https://www-secure.symantec.com/connect/forums/how-use-logs#comment-6505511
Process flow in pdf format: NetBackup 7.x Process Flow
I apprecieate the time you put into your answer. I suspect that you are right but I wanted to put it out to the community anyway. Every once in awhile luck floats.
