BMR savecfg failing with Status 26 after upgrading Master server to NetBackup 7.5.0.4

[AKopel]

Hi we have upgraded most of our Master servers to Netbackup 7.5.0.4 and since then, the parent job fails with a status 1 (the internal error is Status 26).

We have a ticket open with Symantec but just checking to see if anyone else has seen anything similar.

Thanks!

AK

Here is a snippet of "Detailed Status"

10/15/2012 8:31:25 PM - begin Bare Metal Restore, Start Notify Script
10/15/2012 8:31:25 PM - Info RUNCMD(pid=6884) started            
10/15/2012 8:31:26 PM - Info RUNCMD(pid=6884) exiting with status: 0         
Status 0
10/15/2012 8:31:26 PM - end Bare Metal Restore, Start Notify Script; elapsed time: 00:00:01
10/15/2012 8:31:26 PM - begin Bare Metal Restore, BMR Save
10/15/2012 8:31:26 PM - started process bpbrm (8792)
10/15/2012 8:31:30 PM - collecting BMR information
10/15/2012 8:31:30 PM - connecting
10/15/2012 8:31:31 PM - connected; connect time: 00:00:01
10/15/2012 8:31:31 PM - transferring BMR information to the master server
10/15/2012 8:31:31 PM - connecting
10/15/2012 8:31:31 PM - connected; connect time: 00:00:00
Status 26
10/15/2012 8:31:36 PM - end Bare Metal Restore, BMR Save; elapsed time: 00:00:10
10/15/2012 8:31:36 PM - begin Bare Metal Restore, Policy Execution Manager Preprocessed
Status 0
10/15/2012 8:34:52 PM - end Bare Metal Restore, Policy Execution Manager Preprocessed; elapsed time: 00:03:16
10/15/2012 8:34:52 PM - begin Bare Metal Restore, End Notify Script
10/15/2012 8:34:53 PM - Info RUNCMD(pid=10040) started            
10/15/2012 8:34:53 PM - Info RUNCMD(pid=10040) exiting with status: 0         
Status 0
10/15/2012 8:34:53 PM - end Bare Metal Restore, End Notify Script; elapsed time: 00:00:01
Status 1
10/15/2012 8:34:53 PM - end Parent Job; elapsed time: 00:03:28
the requested operation was partially successful(1)

The job was successfully completed, but some files may have been
busy or unaccessible. See the problems report or the client's logs for more details.

[Marianne]

Not sure if this has changed in recent versions (the last time I have worked closely with customer using BMR was in 6.5) but my experience has been that BMR master and clients had to be on the exact same NBU version for BMR backups to work.

[AKopel]

That's not the case, but either way, the failure happens regardless of the Client Version (we have tried upgrading all the clients as well...)

[mandar_khanolka]

I believe you have BMR master already setup on your NB master using "bmrsetupmaster" command. It is also fine if BMR master is setup before the upgrade.

Can you please enable debug level 6 log by setting debuglevel=6 in nblog.conf file on your nb master server? And restart bmr master service. Clear existing log in logs/bmrd folder and take bmr enabled backup.

Provide here the bmrd/*.log file generated.

thanks.

mandar

[trv]

There is indeed something fishy with BMR in NBU 7.5.0.4. I have the same issue and I have found that bpbrm is in fact segfaulting during bmr data collection. Are you by any chance running NBU on rhel 5.8 too ?

Anyway, here is the backtrace:

Core was generated by `bpbrm -backup -collect_bmr_info -c SOMECLIENT -cl win_host_sr'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000003248a78900 in strlen () from /lib64/libc.so.6
(gdb) backtrace
#0  0x0000003248a78900 in strlen () from /lib64/libc.so.6
#1  0x0000003248a46e77 in vfprintf () from /lib64/libc.so.6
#2  0x0000003248a6875a in vsnprintf () from /lib64/libc.so.6
#3  0x00002adcea11d828 in V_vsnprintf () from /usr/openv/lib/libsts.so
#4  0x00002adcec4fde89 in vovgetmsg () from /usr/openv/lib/libnbbaseST.so
#5  0x00002adcec4fe184 in ovgetmsg () from /usr/openv/lib/libnbbaseST.so
#6  0x000000000048f11d in handleBmr ()
#7  0x000000000046dbd0 in main ()
 

[mandar_khanolka]

Ohh. Did you escalated this to Symantec Support? This certainly looks like some bug to me.

thanks.

mandar

[trv]

Nope - not yet, it's not critical for us as we can simply disable bmr collection or just ignore the error - the backup itself is working just fine. But I will do it later for sure.

[Marianne]

Easiest way to increase logging level without need for restarting anything:

vxlogcfg -a --prodid 51216 --orgid 119 -s DebugLevel=6 -s DiagnosticLevel=6

[mandar_khanolka]

That would be great if you can raise a service ticket for this observed problem.

thanks.

mandar

[AKopel]

Ahh!! Good catch! Our master is on Windows Server 2008 R2, but yes, looking at my app log, bpbrm.exe is core dumping here as well!

I'll escalate our case:

If you open a case as well, my case number is 600-868-553 to reference as likely the same issue.

AK

[Peter_Jakobs]

Having the same problem after upgrading to 7.5.0.4, but only with clients running Windows 2003 as a virtual machine.

Master server is Solaris 10, media server Windows 2008 R2.

Upgrading the client to version 7.5.0.4 did not change anything.

Peter

[AKopel]

Good catch... just looked at ours and it's also W2K3 clients doing it...

[marcelg]

Same problem here since installing 7.5.0.4 - SuSE Linux Master Server

bpbrm[24391] general protection ip:7f9c5af8c722 sp:7ffff244b618 error:0 in libc-2.11.1.so[7f9c5af0e000+155000]

Some W2K3 and W2K8 R2 clients fail, some succeed.

[Dip]

I am planning to upgrade to 7.5.0.4 in a week or two in three NBU Domains. I am currently running 7.1.0.1. We do have BMR enabled in a large W2K3 and W2k8 Environment. Please let me know if you have a fix or workaround for this issue.

[jim_dalton]

Oh dear. BMR problems again it seems.

I've said it before and I'll say it again. BMR is not suitable for DR. Dont rely on it.

Do Symantec ever do any genuine organised systematic testing or is that left to the customer?

I'm on Sol10 master/media, I very recently upgraded to 7504...the reason I did this was ...you guessed it...to overcome issues wih previous versions of BMR. 

And I also see that my BMR phases are ending status 1.

Seriously hacked off customer.

[marcelg]

In our case its failing on specific hardware - HP BL460c G7

[mandar_khanolka]

Hi Jim,

Please can you elaborate more on the issue you are facing.

Hi marcelg,

HP BL460c G7 + WinPE based recovery has some issues due to the HW and microsoft WinPE driver issues.

Can you explain on the problem you are facing? Is your NW is not coming up during recovery env?

Thanks.

Mandar

[AKopel]

Still no progress yet. May have to escalate case to get some movement on it..

[jim_dalton]

Mandar...I will respond in time, but I've got a number of issues with 7504.

[marcelg]

@mandar_khanolkar: Problem is pretty much the same as what everyone else is experiencing, i.e. backups incomplete, bpbrm -collect_bmr_info crashes.  Opened a case, but progress has been...slow