BMR savecfg failing with Status 26 after upgrading Master server to NetBackup 7.5.0.4

[AKopel]

Hi we have upgraded most of our Master servers to Netbackup 7.5.0.4 and since then, the parent job fails with a status 1 (the internal error is Status 26).

We have a ticket open with Symantec but just checking to see if anyone else has seen anything similar.

Thanks!

AK

Here is a snippet of "Detailed Status"

10/15/2012 8:31:25 PM - begin Bare Metal Restore, Start Notify Script
10/15/2012 8:31:25 PM - Info RUNCMD(pid=6884) started            
10/15/2012 8:31:26 PM - Info RUNCMD(pid=6884) exiting with status: 0         
Status 0
10/15/2012 8:31:26 PM - end Bare Metal Restore, Start Notify Script; elapsed time: 00:00:01
10/15/2012 8:31:26 PM - begin Bare Metal Restore, BMR Save
10/15/2012 8:31:26 PM - started process bpbrm (8792)
10/15/2012 8:31:30 PM - collecting BMR information
10/15/2012 8:31:30 PM - connecting
10/15/2012 8:31:31 PM - connected; connect time: 00:00:01
10/15/2012 8:31:31 PM - transferring BMR information to the master server
10/15/2012 8:31:31 PM - connecting
10/15/2012 8:31:31 PM - connected; connect time: 00:00:00
Status 26
10/15/2012 8:31:36 PM - end Bare Metal Restore, BMR Save; elapsed time: 00:00:10
10/15/2012 8:31:36 PM - begin Bare Metal Restore, Policy Execution Manager Preprocessed
Status 0
10/15/2012 8:34:52 PM - end Bare Metal Restore, Policy Execution Manager Preprocessed; elapsed time: 00:03:16
10/15/2012 8:34:52 PM - begin Bare Metal Restore, End Notify Script
10/15/2012 8:34:53 PM - Info RUNCMD(pid=10040) started            
10/15/2012 8:34:53 PM - Info RUNCMD(pid=10040) exiting with status: 0         
Status 0
10/15/2012 8:34:53 PM - end Bare Metal Restore, End Notify Script; elapsed time: 00:00:01
Status 1
10/15/2012 8:34:53 PM - end Parent Job; elapsed time: 00:03:28
the requested operation was partially successful(1)

The job was successfully completed, but some files may have been
busy or unaccessible. See the problems report or the client's logs for more details.

[Dip]

I just upgraded my Test NBU environment from 7.1.0.1 to 7.5.0.4 and performed full backups of some of the OSs and they were successful. However, this environment is idel all the time as it is used for test purposes only so not sure if having load in backup environment will result differently for BMR data collection.

Below is what I have tested o far and all is well.

Windows 2003 32bit     VM Guest      NBU7.1 Client

Windows 2003 32bit     VM Guest      NBU7.1.0.1 Client

Windows 2008 R2 64bit     VM Guest      NBU7.1.0.1 Client

RedHat Enterprise 5.7     DL585 G5       NBU7.1.0.1 Client

[jim_dalton]

Heres what I'm seeing: Im caught between thinking its worked and its failed:

10/29/2012 21:00:00 - Info nbjm (pid=25197) starting backup job (jobid=1176056) for client w2kmango, policy PROD_W2K_Servers, schedule Daily_Inc
10/29/2012 21:00:00 - Info nbjm (pid=25197) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1176056, request id:{9A5FB620-220B-11E2-8E3E-00144FC1D6A8})
10/29/2012 21:00:00 - requesting resource Master-wunebro-SL500-LTO4
10/29/2012 21:00:00 - requesting resource wunebro.NBU_CLIENT.MAXJOBS.w2kmango
10/29/2012 21:00:00 - requesting resource wunebro.NBU_POLICY.MAXJOBS.PROD_W2K_Servers
10/29/2012 21:00:03 - granted resource  wunebro.NBU_CLIENT.MAXJOBS.w2kmango
10/29/2012 21:00:03 - granted resource  wunebro.NBU_POLICY.MAXJOBS.PROD_W2K_Servers
10/29/2012 21:00:03 - granted resource  004631
10/29/2012 21:00:03 - granted resource  IBM.ULTRIUM-TD4.002
10/29/2012 21:00:03 - granted resource  Master-wunebro-SL500-LTO4
10/29/2012 21:00:10 - estimated 782486 kbytes needed
10/29/2012 21:00:10 - begin Parent Job
10/29/2012 21:00:10 - begin Bare Metal Restore: Start Notify Script
10/29/2012 21:00:10 - Info RUNCMD (pid=22664) started
10/29/2012 21:00:10 - Info RUNCMD (pid=22664) exiting with status: 0
Operation Status: 0
10/29/2012 21:00:10 - end Bare Metal Restore: Start Notify Script; elapsed time 0:00:00
10/29/2012 21:00:10 - begin Bare Metal Restore: Bare Metal Restore Save
10/29/2012 21:00:12 - started process bpbrm (pid=22672)
10/29/2012 21:00:16 - collecting BMR information
10/29/2012 21:00:16 - connecting
10/29/2012 21:00:16 - connected; connect time: 0:00:00
10/29/2012 21:00:16 - transfering BMR information to the master server
10/29/2012 21:00:16 - connecting
10/29/2012 21:00:16 - connected; connect time: 0:00:00
10/29/2012 21:00:58 - BMR information transfer successful
10/29/2012 21:00:58 - Info bmrsavecfg (pid=0) done. status: 0: the requested operation was successfully completed
10/29/2012 21:00:58 - end writing
Operation Status: 0
10/29/2012 21:00:58 - end Bare Metal Restore: Bare Metal Restore Save; elapsed time 0:00:48
10/29/2012 21:00:58 - begin Bare Metal Restore: Policy Execution Manager Preprocessed
Operation Status: 1
10/29/2012 21:25:57 - end Bare Metal Restore: Policy Execution Manager Preprocessed; elapsed time 0:24:59
10/29/2012 21:25:57 - begin Bare Metal Restore: End Notify Script
10/29/2012 21:25:57 - Info RUNCMD (pid=718) started
10/29/2012 21:25:57 - Info RUNCMD (pid=718) exiting with status: 0
Operation Status: 0
10/29/2012 21:25:57 - end Bare Metal Restore: End Notify Script; elapsed time 0:00:00
Operation Status: 1
10/29/2012 21:25:57 - end Parent Job; elapsed time 0:25:47
the requested operation was partially successful  (1)
 

The data backup itself ends with status 1; it has issues with some of the data on the server. I need to look deeper...

[Dip]

Below copy paste from you above log is the BMR Config log upload. It was started at 21:00:10 and was completed at 21:00:58. this includes collecting configuration from client and sending it to Master server successfully. Then another Job kicked of which actually performed backup of your client, that job was parcially completed. If you check job detail on that second job you will see what was skipped.

Your Backup was successful as far as BMR is concern, the actual backup had skipped some files.

10/29/2012 21:00:10 - begin Bare Metal Restore: Bare Metal Restore Save
10/29/2012 21:00:12 - started process bpbrm (pid=22672)
10/29/2012 21:00:16 - collecting BMR information
10/29/2012 21:00:16 - connecting
10/29/2012 21:00:16 - connected; connect time: 0:00:00
10/29/2012 21:00:16 - transfering BMR information to the master server
10/29/2012 21:00:16 - connecting
10/29/2012 21:00:16 - connected; connect time: 0:00:00
10/29/2012 21:00:58 - BMR information transfer successful
10/29/2012 21:00:58 - Info bmrsavecfg (pid=0) done. status: 0: the requested operation was successfully completed

10/29/2012 21:00:58 - end writing
Operation Status: 0
10/29/2012 21:00:58 - end Bare Metal Restore: Bare Metal Restore Save; elapsed time 0:00:48

[jim_dalton]

Hello Dip

But....

10/29/2012 21:00:58 - begin Bare Metal Restore: Policy Execution Manager Preprocessed
Operation Status: 1
 

This is _before_ the data backup , so it can only refer to BMR phase?

The data backup didnt finish til 21:25, the status 1 above cant possibly refer to anything but the BMR step, dont you agree?

The messages are somewhat confusing:

10/29/2012 21:25:57 - end Bare Metal Restore: End Notify Script; elapsed time 0:00:00
Operation Status: 1
 

Why is it bringing up the subject of BMR after the data phase? As I understood it, the BMR phase is complete before the data phase. Searching for info regards BMR start/end notify info...!

Jim

[Dip]

I missed that Status:1 after the BMR information transfered. You are correct, something did not work as far as BMR is concerned in this job.

[mandar_khanolka]

Looking at the job details provided by Jim, I do see that BMR configuration backup has been successfully executed. If BMR backup fails, "BMR information transfer successful" message should not come.

I hope you can see the client entry even under Bare metal restore -> clients menu in your admin GUI.

I suspect somthing is wrong with NB policy execution manager here showing this error.

Only debug logs can reveal the problem.

Thanks.

Mandar

[jim_dalton]

Mandar

Yes Ive check the status in the BMR clients and the timestamps suggest the BMR config information is indeed correctly transferred.

Ive raised a call with support , but support is painfully slow these days, getting information from support in a timely manner just doesnt happen, the forum is much more responsive.

Jim

[jim_dalton]

To go back to the original thread, I also have just four clients exhibiting AKopel's behaviour, ie error 26 on save so I'm interested in a fix for that. I'm so glad I upgraded to 7.5.0.4.

Jim

[mandar_khanolka]

Hi Jim

Is it possible for you to provide the support case number here?

Thanks.

Mandar

[jim_dalton]

I have received some feedback from support , we are making progress.

The error 26 issue seems to be related to the client and the server talking to eachother: the bundle is created ok and can be imported onto the master if done manually ie having copied it manually.

Error 1...nothing to tell as yet. 

Jim

[Peter_Jakobs]

Anybody has any news?

Another ten days have passed, support is disappeared. case number 420-213-118

I really doubt that my comapny is going to pay so much next year for 'support'

Peter

[AKopel]

We FINALLY got our case escalated and they are collecting the application core dumps for bpbrm... hopefully will have some progress soon.

AK

[marcelg]

Same here, i.e. finally got the case escalated.  What I find mystifying is why it took so long for support to acknowledge that it might be a bug.  In what world is an easily repeatable core dump not a bug?

[dezzer]

We are currently experiencing the same issues under case 02818877. All with HP BL460c G7 servers.

Unable to do the BMR dumps until we raise a change for the registry edits.

Not the only ones out there with this issue.

Must have been some code change in the NetBackup software, as these backups worked fine before we upgraded to 7.5.0.4.

[marcelg]

Hi dezzer,

What registry changes do you make?

[dezzer]

Been told to do this: Please enable the registry key "LocalDumps" by following the instruction shown in: http://msdn.microsoft.com/en-us/library/windows/desktop/bb787181%28v=vs.85%29.aspx So we can get BMR process crash dumps being saved.

[dezzer]

We need to set a registry keyto get the BMR dumps working: From Symantec support: Please enable the registry key "LocalDumps" by following the instruction shown in: http://msdn.microsoft.com/en-us/library/windows/desktop/bb787181%28v=vs.85%29.aspx We will then need to send them the logs for bpbrm.exe crashes.

[Stavros41]

Same here

Just deployed NBU 7.5.0.4 at a customer and exactly the same problem.

Tested servers so far are all HP 460c G7 Servers

[marcelg · Accepted solution]

Hi,

Symantec support has provided an EEB (etrack 2991238) that resolves this issue.

[AKopel]

We just got the same EEB and it also resolved it for us!

AK