07-22-2015 12:09 PM
Having a heck of a time trying to figure this out. 14 of my 21 VMware policies are failing with a status code 200 for some reason. Credentials are good and we can do a test query with no issues.
Windows 2012 VMware Backup Host
5.5 Update 2
7.6.0.2 and I verified the VDDK is 5.5.0
I have attached all of the logs that would actually run. I cranked VXMS up to 5, but no log was created as I am guessing it just doesnt get that far into the process.
Detailed Job Status:
07/22/2015 11:00:59 - Info nbjm (pid=1310) starting backup job (jobid=3391315) for client MEDIA_SERVER, policy Omaha_VMware_INFOBLOX, schedule Full_Daily
07/22/2015 11:00:59 - Info nbjm (pid=1310) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=3391315, request id:{D84F647C-308A-11E5-9F49-002128A6F97C})
07/22/2015 11:00:59 - requesting resource XP53TAPE009
07/22/2015 11:00:59 - requesting resource xp53tape001.NBU_POLICY.MAXJOBS.Omaha_VMware_INFOBLOX
07/22/2015 11:00:59 - granted resource xp53tape001.NBU_POLICY.MAXJOBS.Omaha_VMware_INFOBLOX
07/22/2015 11:00:59 - estimated 11832060 kbytes needed
07/22/2015 11:00:59 - begin Parent Job
07/22/2015 11:00:59 - begin Application Resolver: Start Notify Script
07/22/2015 11:01:00 - Info RUNCMD (pid=15674) started
07/22/2015 11:01:00 - Info RUNCMD (pid=15674) exiting with status: 0
Operation Status: 0
07/22/2015 11:01:00 - end Application Resolver: Start Notify Script; elapsed time 0:00:01
07/22/2015 11:01:00 - begin Application Resolver: Step By Condition
Operation Status: 0
07/22/2015 11:01:00 - end Application Resolver: Step By Condition; elapsed time 0:00:00
07/22/2015 11:01:00 - begin Application Resolver: Resolver Discovery
Operation Status: 0
07/22/2015 12:31:04 - end Application Resolver: Resolver Discovery; elapsed time 1:30:04
07/22/2015 12:31:04 - begin Application Resolver: Policy Execution Manager Preprocessed
07/22/2015 12:31:04 - Warning nbpem (pid=1367) VM retrieval failed, server = [53apps178vm].
Operation Status: 200
07/22/2015 12:31:04 - end Application Resolver: Policy Execution Manager Preprocessed; elapsed time 0:00:00
07/22/2015 12:31:04 - begin Application Resolver: Stop On Error
Operation Status: 0
07/22/2015 12:31:04 - end Application Resolver: Stop On Error; elapsed time 0:00:00
07/22/2015 12:31:04 - begin Application Resolver: End Notify Script
07/22/2015 12:31:04 - Info RUNCMD (pid=27555) started
07/22/2015 12:31:04 - Info RUNCMD (pid=27555) exiting with status: 0
Operation Status: 0
07/22/2015 12:31:04 - end Application Resolver: End Notify Script; elapsed time 0:00:00
Operation Status: 200
07/22/2015 12:31:04 - end Parent Job; elapsed time 1:30:05
scheduler found no backups due to run (200)
07-22-2015 09:03 PM
What type of schedule are you using? Please share the details of the start end times.
I see your job is traversing midnight so that could maybe be an issue.
07-23-2015 06:46 AM
The schedule was never an issue before. These policies have been in place for quite awhile and have always worked.
I probably should have added before that the VM team did the upgrade to update 2 for vSphere the day before we started seeing these issues.
Policy details for one of the policies that is failing...
Policy Name: Omaha_VMware5
Policy Type: VMware
Active: yes
Effective date: 11/08/2012 10:10:06
File Restore Raw: yes
Application Consistent: yes
Mult. Data Streams: no
Client Encrypt: no
Checkpoint: no
Policy Priority: 9
Max Jobs/Policy: 3
Disaster Recovery: 0
Collect BMR info: no
Residence: XP53TAPE009_SLP
Volume Pool: Daily
Server Group: *ANY*
Keyword: (none specified)
Data Classification: -
Residence is Storage Lifecycle Policy: yes
Application Discovery: yes
Discovery Lifetime: 28800 seconds
ASC Application and attributes: (none defined)
Granular Restore Info: no
Ignore Client Direct: no
Enable Metadata Indexing: no
Index server name: NULL
Use Accelerator: yes
HW/OS/Client: VMware VMware MEDIA_SERVER
Include: vmware:/?filter=Displayname Equal "04sna011" OR Displayname Equal "04sna012" OR Displayname Equal "04sna013" OR Displayname Equal "04sna014" OR Displayname Equal "33apps004vm" OR Displayname Equal "33apps009" OR Displayname Equal "33apps047" OR Displayname Equal "33apps081" OR Displayname Equal "33apps093" OR Displayname Equal "33apps094" OR Displayname Equal "33apps095" OR Displayname Equal "33apps096vm" OR Displayname Equal "33apps097vm" OR Displayname Equal "33apps221vm" OR Displayname Equal "33apps266vm" OR Displayname Equal "33apps334vm" OR Displayname Equal "33apps368" OR Displayname Equal "33apps413vm" OR Displayname Equal "33ctx220vm" OR Displayname Equal "33web063vm" OR Displayname Equal "33web110vm" OR Displayname Equal "33web124vm" OR Displayname Equal "53apps038vm" OR Displayname Equal "53APPS040" OR Displayname Equal "53apps419vm" OR Displayname Equal "53apps475vm" OR Displayname Equal "53apps546vm" OR Displayname Equal "53apps547vm" OR Displayname Equal "53apps548vm" OR Displayname Equal "53apps550vm" OR Displayname Equal "53apps580vm" OR Displayname Equal "53apps581vm" OR Displayname Equal "53apps595vm" OR Displayname Equal "53apps596vm" OR Displayname Equal "53apps597vm" OR Displayname Equal "53apps598vm" OR Displayname Equal "53apps599vm" OR Displayname Equal "53apps603vm" OR Displayname Equal "53apps604vm" OR Displayname Equal "53apps605vm" OR Displayname Equal "53apps607vm" OR Displayname Equal "53apps608vm" OR Displayname Equal "53apps610vm" OR Displayname Equal "53apps611vm" OR Displayname Equal "53apps624vm" OR Displayname Equal "53apps650vm" OR Displayname Equal "53apps681vm" OR Displayname Equal "53apps682vm" OR Displayname Equal "53apps740vm" OR Displayname Equal "53apps758vm" OR Displayname Equal "53apps776vm" OR Displayname Equal "53ctx106vm" OR Displayname Equal "53ctx251vm" OR Displayname Equal "53ctx252vm" OR Displayname Equal "53ctx253vm" OR Displayname Equal "53ctx255vm" OR Displayname Equal "53ctx256vm" OR Displayname Equal "53ctx265vm" OR Displayname Equal "53ctx266vm" OR Displayname Equal "53ctx282vm" OR Displayname Equal "53ctx283vm" OR Displayname Equal "53ctx284vm" OR Displayname Equal "53ctx285vm" OR Displayname Equal "53ctx286vm" OR Displayname Equal "53ctx287vm" OR Displayname Equal "53eislog03vm" OR Displayname Equal "53eislog04vm" OR Displayname Equal "53hts003vm" OR Displayname Equal "53risk003vm" OR Displayname Equal "53risk004vm" OR Displayname Equal "53web144vm" OR Displayname Equal "53web149vm" OR Displayname Equal "53web165vm" OR Displayname Equal "53web166vm" OR Displayname Equal "53web168vm" OR Displayname Equal "53web185vm" OR Displayname Equal "p53aceapp001t0w" OR Displayname Equal "p53crmapp001t0w" OR Displayname Equal "p53crmapp002t0w" OR Displayname Equal "p53crmapp003t0w" OR Displayname Equal "p53crmapp004t0w" OR Displayname Equal "p53crmetl001t0w" OR Displayname Equal "p53crmsrs001t0w" OR Displayname Equal "p53crmsrs002t0w" OR Displayname Equal "p53dcpapp001t0w" OR Displayname Equal "p53cprapp001t0w" OR Displayname Equal "p53cprweb001t0w" OR Displayname Equal "p53mtiapp003t0w" OR Displayname Equal "p53mtiapp004t0w" OR Displayname Equal "p53mtiora001t0x" OR Displayname Equal "xp33intapp01vm" OR Displayname Equal "xp53app0409vm" OR Displayname Equal "xp53app0485vm" OR Displayname Equal "xp53app0554vm" OR Displayname Equal "xp53app0555vm" OR Displayname Equal "xp53app0598vm" OR Displayname Equal "xp53app0774vm" OR Displayname Equal "xp53app0779vm" OR Displayname Equal "xp53app0780vm" OR Displayname Equal "xp53app0901" OR Displayname Equal "xp53app0902" OR Displayname Equal "xp53app0903" OR Displayname Equal "xp53app0904" OR Displayname Equal "xp53app0905" OR Displayname Equal "xp53app0906" OR Displayname Equal "xp53app0923" OR Displayname Equal "xp53app0924" OR Displayname Equal "xp53app0925" OR Displayname Equal "xp53app0926" OR Displayname Equal "xp53app0927" OR Displayname Equal "xp53app122vm" OR Displayname Equal "xp53app123vm" OR Displayname Equal "xp53apps0957vm" OR Displayname Equal "xp53db0599vm" OR Displayname Equal "xp53db0921vm" OR Displayname Equal "xp53inf0546vm" OR Displayname Equal "xp53info777vm" OR Displayname Equal "xp53ing0777vm" OR Displayname Equal "xp53ucscentral01" OR Displayname Equal "xp53web007vm" OR Displayname Equal "xp53web0558vm" OR Displayname Equal "xp53xmlfw02vm" OR Displayname Equal "xp53xmlfw12vm" OR Displayname Equal "xp53xmlfw13vm"
Schedule: Full_Weekly
Type: Full Backup
Calendar sched: Enabled
Included Dates-----------
Saturday, Week 2
Saturday, Week 3
Saturday, Week 4
Saturday, Week 5
Excluded Dates----------
No specific exclude dates entered
No exclude days of week entered
PFI Recovery: 0
Maximum MPX: 10
Retention Level: 3 (1 month)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required)
Volume Pool: Weekly
Server Group: (same as specified for policy)
Residence is Storage Lifecycle Policy: 0
Schedule indexing: 0
Daily Windows:
Saturday 12:00:00 --> Monday 16:00:00
Schedule: Incremental
Type: Cumulative Incremental Backup
Calendar sched: Enabled
Allowed to retry after run day
Included Dates-----------
Monday, Week 1
Tuesday, Week 1
Wednesday, Week 1
Thursday, Week 1
Friday, Week 1
Monday, Week 2
Tuesday, Week 2
Wednesday, Week 2
Thursday, Week 2
Friday, Week 2
Monday, Week 3
Tuesday, Week 3
Wednesday, Week 3
Thursday, Week 3
Friday, Week 3
Monday, Week 4
Tuesday, Week 4
Wednesday, Week 4
Thursday, Week 4
Friday, Week 4
Monday, Week 5
Tuesday, Week 5
Wednesday, Week 5
Thursday, Week 5
Friday, Week 5
Excluded Dates----------
EXCLUDE DATE 0 - 06/26/2013
No exclude days of week entered
PFI Recovery: 0
Maximum MPX: 10
Retention Level: 3 (1 month)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required)
Volume Pool: Daily
Server Group: (same as specified for policy)
Residence is Storage Lifecycle Policy: 0
Schedule indexing: 0
Daily Windows:
Monday 18:00:00 --> Tuesday 10:00:00
Tuesday 18:00:00 --> Wednesday 10:00:00
Wednesday 18:00:00 --> Thursday 10:00:00
Thursday 18:00:00 --> Friday 10:00:00
Friday 18:00:00 --> Saturday 10:00:00
Schedule: Full_Monthly
Type: Full Backup
Calendar sched: Enabled
Included Dates-----------
Saturday, Week 1
Excluded Dates----------
No specific exclude dates entered
No exclude days of week entered
PFI Recovery: 0
Maximum MPX: 10
Retention Level: 0 (1 week)
Number Copies: 1
Fail on Error: 0
Residence: XP53TAPE009_SLP_OFFSITE
Volume Pool: (same as policy volume pool)
Server Group: (same as specified for policy)
Residence is Storage Lifecycle Policy: 1
Schedule indexing: 0
Daily Windows:
Saturday 12:00:00 --> Monday 16:00:00
07-23-2015 07:49 AM
Pretty odd, but in some of the policies that said MEDIA_SERVER we have changed that to the actual VMware Backup Host for the "Netbackup host to perform automatic virtual machine selection" setting and they are now running. I have no idea why this is fixing anything.
07-23-2015 10:07 AM
Weird. Anyways, working now :)
07-23-2015 10:11 AM
Yeah thats what I said. Was hoping to find out what might be linked to that part of the policy to see why its fixing the problem.
08-06-2015 08:37 AM
I have this problem as an open case for weeks. I have made the change you made and will see if it fixes my problem also.
Did you ever sometimes get rc=4232 also?
08-06-2015 08:56 AM
I dont recall seeing that error no. I can say that since making the change to point directly at the vmware backup host we have not seen anymore issues.
08-06-2015 11:00 PM
In your event logs, do you see any bpbkar.exe crashing, or simply not exiting (hanging) in task manager?
Best to patch your VM backup host up to 7.6.0.4, as it has some fixes on Vmware backup including this one:
http://www.symantec.com/docs/TECH223790
08-07-2015 07:16 AM
I dont have an issue with hung bpbkar, but rather bpfis.
08-07-2015 07:33 AM
I am at 7.6.1.2.
If the problem persists (I have made the change already) I wil check bpbkar.
08-10-2015 08:53 AM
Unfortunately this not not help my problem.
I dont see any hung bpbkar processes.