cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Veritas.com
Support

VMware backup failures with status code 200

backup-botw
Level 6

‎07-22-2015 12:09 PM

Having a heck of a time trying to figure this out. 14 of my 21 VMware policies are failing with a status code 200 for some reason. Credentials are good and we can do a test query with no issues.

Windows 2012 VMware Backup Host

5.5 Update 2

7.6.0.2 and I verified the VDDK is 5.5.0

I have attached all of the logs that would actually run. I cranked VXMS up to 5, but no log was created as I am guessing it just doesnt get that far into the process.

Detailed Job Status:

07/22/2015 11:00:59 - Info nbjm (pid=1310) starting backup job (jobid=3391315) for client MEDIA_SERVER, policy Omaha_VMware_INFOBLOX, schedule Full_Daily
07/22/2015 11:00:59 - Info nbjm (pid=1310) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=3391315, request id:{D84F647C-308A-11E5-9F49-002128A6F97C})
07/22/2015 11:00:59 - requesting resource XP53TAPE009
07/22/2015 11:00:59 - requesting resource xp53tape001.NBU_POLICY.MAXJOBS.Omaha_VMware_INFOBLOX
07/22/2015 11:00:59 - granted resource  xp53tape001.NBU_POLICY.MAXJOBS.Omaha_VMware_INFOBLOX
07/22/2015 11:00:59 - estimated 11832060 kbytes needed
07/22/2015 11:00:59 - begin Parent Job
07/22/2015 11:00:59 - begin Application Resolver: Start Notify Script
07/22/2015 11:01:00 - Info RUNCMD (pid=15674) started
07/22/2015 11:01:00 - Info RUNCMD (pid=15674) exiting with status: 0
Operation Status: 0
07/22/2015 11:01:00 - end Application Resolver: Start Notify Script; elapsed time 0:00:01
07/22/2015 11:01:00 - begin Application Resolver: Step By Condition
Operation Status: 0
07/22/2015 11:01:00 - end Application Resolver: Step By Condition; elapsed time 0:00:00
07/22/2015 11:01:00 - begin Application Resolver: Resolver Discovery
Operation Status: 0
07/22/2015 12:31:04 - end Application Resolver: Resolver Discovery; elapsed time 1:30:04
07/22/2015 12:31:04 - begin Application Resolver: Policy Execution Manager Preprocessed
07/22/2015 12:31:04 - Warning nbpem (pid=1367) VM retrieval failed, server = [53apps178vm].
Operation Status: 200
07/22/2015 12:31:04 - end Application Resolver: Policy Execution Manager Preprocessed; elapsed time 0:00:00
07/22/2015 12:31:04 - begin Application Resolver: Stop On Error
Operation Status: 0
07/22/2015 12:31:04 - end Application Resolver: Stop On Error; elapsed time 0:00:00
07/22/2015 12:31:04 - begin Application Resolver: End Notify Script
07/22/2015 12:31:04 - Info RUNCMD (pid=27555) started
07/22/2015 12:31:04 - Info RUNCMD (pid=27555) exiting with status: 0
Operation Status: 0
07/22/2015 12:31:04 - end Application Resolver: End Notify Script; elapsed time 0:00:00
Operation Status: 200
07/22/2015 12:31:04 - end Parent Job; elapsed time 1:30:05
scheduler found no backups due to run  (200)

 

vmlogs.zip
5303 KB
0 Kudos
11 REPLIES 11

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

‎07-22-2015 09:03 PM

What type of schedule are you using? Please share the details of the start end times.

 

I see your job is traversing midnight so that could maybe be an issue.
 

0 Kudos

backup-botw
Level 6

‎07-23-2015 06:46 AM

The schedule was never an issue before. These policies have been in place for quite awhile and have always worked.

I probably should have added before that the VM team did the upgrade to update 2 for vSphere the day before we started seeing these issues.

Policy details for one of the policies that is failing...

Policy Name:       Omaha_VMware5

  Policy Type:         VMware
  Active:              yes
  Effective date:      11/08/2012 10:10:06
  File Restore Raw:    yes
  Application Consistent:    yes
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     9
  Max Jobs/Policy:     3
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           XP53TAPE009_SLP
  Volume Pool:         Daily
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    yes
  Application Discovery:      yes
  Discovery Lifetime:      28800 seconds
  ASC Application and attributes: (none defined)

  Granular Restore Info:  no
  Ignore Client Direct:  no
  Enable Metadata Indexing:  no
  Index server name:  NULL
  Use Accelerator:  yes
  HW/OS/Client:  VMware        VMware        MEDIA_SERVER

  Include:  vmware:/?filter=Displayname Equal "04sna011" OR Displayname Equal "04sna012" OR Displayname Equal "04sna013" OR Displayname Equal "04sna014" OR Displayname Equal "33apps004vm" OR Displayname Equal "33apps009" OR Displayname Equal "33apps047" OR Displayname Equal "33apps081" OR Displayname Equal "33apps093" OR Displayname Equal "33apps094" OR Displayname Equal "33apps095" OR Displayname Equal "33apps096vm" OR Displayname Equal "33apps097vm" OR Displayname Equal "33apps221vm" OR Displayname Equal "33apps266vm" OR Displayname Equal "33apps334vm" OR Displayname Equal "33apps368" OR Displayname Equal "33apps413vm" OR Displayname Equal "33ctx220vm" OR Displayname Equal "33web063vm" OR Displayname Equal "33web110vm" OR Displayname Equal "33web124vm" OR Displayname Equal "53apps038vm" OR Displayname Equal "53APPS040" OR Displayname Equal "53apps419vm" OR Displayname Equal "53apps475vm" OR Displayname Equal "53apps546vm" OR Displayname Equal "53apps547vm" OR Displayname Equal "53apps548vm" OR Displayname Equal "53apps550vm" OR Displayname Equal "53apps580vm" OR Displayname Equal "53apps581vm" OR Displayname Equal "53apps595vm" OR Displayname Equal "53apps596vm" OR Displayname Equal "53apps597vm" OR Displayname Equal "53apps598vm" OR Displayname Equal "53apps599vm" OR Displayname Equal "53apps603vm" OR Displayname Equal "53apps604vm" OR Displayname Equal "53apps605vm" OR Displayname Equal "53apps607vm" OR Displayname Equal "53apps608vm" OR Displayname Equal "53apps610vm" OR Displayname Equal "53apps611vm" OR Displayname Equal "53apps624vm" OR Displayname Equal "53apps650vm" OR Displayname Equal "53apps681vm" OR Displayname Equal "53apps682vm" OR Displayname Equal "53apps740vm" OR Displayname Equal "53apps758vm" OR Displayname Equal "53apps776vm" OR Displayname Equal "53ctx106vm" OR Displayname Equal "53ctx251vm" OR Displayname Equal "53ctx252vm" OR Displayname Equal "53ctx253vm" OR Displayname Equal "53ctx255vm" OR Displayname Equal "53ctx256vm" OR Displayname Equal "53ctx265vm" OR Displayname Equal "53ctx266vm" OR Displayname Equal "53ctx282vm" OR Displayname Equal "53ctx283vm" OR Displayname Equal "53ctx284vm" OR Displayname Equal "53ctx285vm" OR Displayname Equal "53ctx286vm" OR Displayname Equal "53ctx287vm" OR Displayname Equal "53eislog03vm" OR Displayname Equal "53eislog04vm" OR Displayname Equal "53hts003vm" OR Displayname Equal "53risk003vm" OR Displayname Equal "53risk004vm" OR Displayname Equal "53web144vm" OR Displayname Equal "53web149vm" OR Displayname Equal "53web165vm" OR Displayname Equal "53web166vm" OR Displayname Equal "53web168vm" OR Displayname Equal "53web185vm" OR Displayname Equal "p53aceapp001t0w" OR Displayname Equal "p53crmapp001t0w" OR Displayname Equal "p53crmapp002t0w" OR Displayname Equal "p53crmapp003t0w" OR Displayname Equal "p53crmapp004t0w" OR Displayname Equal "p53crmetl001t0w" OR Displayname Equal "p53crmsrs001t0w" OR Displayname Equal "p53crmsrs002t0w" OR Displayname Equal "p53dcpapp001t0w" OR Displayname Equal "p53cprapp001t0w" OR Displayname Equal "p53cprweb001t0w" OR Displayname Equal "p53mtiapp003t0w" OR Displayname Equal "p53mtiapp004t0w" OR Displayname Equal "p53mtiora001t0x" OR Displayname Equal "xp33intapp01vm" OR Displayname Equal "xp53app0409vm" OR Displayname Equal "xp53app0485vm" OR Displayname Equal "xp53app0554vm" OR Displayname Equal "xp53app0555vm" OR Displayname Equal "xp53app0598vm" OR Displayname Equal "xp53app0774vm" OR Displayname Equal "xp53app0779vm" OR Displayname Equal "xp53app0780vm" OR Displayname Equal "xp53app0901" OR Displayname Equal "xp53app0902" OR Displayname Equal "xp53app0903" OR Displayname Equal "xp53app0904" OR Displayname Equal "xp53app0905" OR Displayname Equal "xp53app0906" OR Displayname Equal "xp53app0923" OR Displayname Equal "xp53app0924" OR Displayname Equal "xp53app0925" OR Displayname Equal "xp53app0926" OR Displayname Equal "xp53app0927" OR Displayname Equal "xp53app122vm" OR Displayname Equal "xp53app123vm" OR Displayname Equal "xp53apps0957vm" OR Displayname Equal "xp53db0599vm" OR Displayname Equal "xp53db0921vm" OR Displayname Equal "xp53inf0546vm" OR Displayname Equal "xp53info777vm" OR Displayname Equal "xp53ing0777vm" OR Displayname Equal "xp53ucscentral01" OR Displayname Equal "xp53web007vm" OR Displayname Equal "xp53web0558vm" OR Displayname Equal "xp53xmlfw02vm" OR Displayname Equal "xp53xmlfw12vm" OR Displayname Equal "xp53xmlfw13vm"

  Schedule:              Full_Weekly
    Type:                Full Backup
    Calendar sched: Enabled
     Included Dates-----------
        Saturday, Week 2
        Saturday, Week 3
        Saturday, Week 4
        Saturday, Week 5
     Excluded Dates----------
        No specific exclude dates entered
        No exclude days of week entered
    PFI Recovery:        0
    Maximum MPX:         10
    Retention Level:     3 (1 month)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         Weekly
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:   0
    Daily Windows:
          Saturday   12:00:00  -->  Monday     16:00:00

  Schedule:              Incremental
    Type:                Cumulative Incremental Backup
    Calendar sched: Enabled
      Allowed to retry after run day
     Included Dates-----------
        Monday, Week 1
        Tuesday, Week 1
        Wednesday, Week 1
        Thursday, Week 1
        Friday, Week 1
        Monday, Week 2
        Tuesday, Week 2
        Wednesday, Week 2
        Thursday, Week 2
        Friday, Week 2
        Monday, Week 3
        Tuesday, Week 3
        Wednesday, Week 3
        Thursday, Week 3
        Friday, Week 3
        Monday, Week 4
        Tuesday, Week 4
        Wednesday, Week 4
        Thursday, Week 4
        Friday, Week 4
        Monday, Week 5
        Tuesday, Week 5
        Wednesday, Week 5
        Thursday, Week 5
        Friday, Week 5
     Excluded Dates----------
      EXCLUDE DATE 0 - 06/26/2013
        No exclude days of week entered
    PFI Recovery:        0
    Maximum MPX:         10
    Retention Level:     3 (1 month)
    Number Copies:       1
    Fail on Error:       0
    Residence:           (specific storage unit not required)
    Volume Pool:         Daily
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:   0
    Daily Windows:
          Monday     18:00:00  -->  Tuesday    10:00:00
          Tuesday    18:00:00  -->  Wednesday  10:00:00
          Wednesday  18:00:00  -->  Thursday   10:00:00
          Thursday   18:00:00  -->  Friday     10:00:00
          Friday     18:00:00  -->  Saturday   10:00:00

  Schedule:              Full_Monthly
    Type:                Full Backup
    Calendar sched: Enabled
     Included Dates-----------
        Saturday, Week 1
     Excluded Dates----------
        No specific exclude dates entered
        No exclude days of week entered
    PFI Recovery:        0
    Maximum MPX:         10
    Retention Level:     0 (1 week)
    Number Copies:       1
   Fail on Error:       0
    Residence:           XP53TAPE009_SLP_OFFSITE
    Volume Pool:         (same as policy volume pool)
    Server Group:        (same as specified for policy)
    Residence is Storage Lifecycle Policy:         1
    Schedule indexing:   0
    Daily Windows:
          Saturday   12:00:00  -->  Monday     16:00:00

 

0 Kudos

backup-botw
Level 6

‎07-23-2015 07:49 AM

Pretty odd, but in some of the policies that said MEDIA_SERVER we have changed that to the actual VMware Backup Host for the "Netbackup host to perform automatic virtual machine selection" setting and they are now running. I have no idea why this is fixing anything.

0 Kudos

RiaanBadenhorst
Moderator
Moderator
Partner    VIP    Accredited Certified

‎07-23-2015 10:07 AM

Weird. Anyways, working now :)

0 Kudos

backup-botw
Level 6

‎07-23-2015 10:11 AM

Yeah thats what I said. Was hoping to find out what might be linked to that part of the policy to see why its fixing the problem.

0 Kudos

sclind
Moderator
Moderator
   VIP   

‎08-06-2015 08:37 AM

I have this problem as an open case for weeks.  I have made the change you made and will see if it fixes my problem also.

Did you ever sometimes get rc=4232 also?

0 Kudos

backup-botw
Level 6

‎08-06-2015 08:56 AM

I dont recall seeing that error no. I can say that since making the change to point directly at the vmware backup host we have not seen anymore issues.

0 Kudos

watsons
Level 6

‎08-06-2015 11:00 PM

In your event logs, do you see any bpbkar.exe crashing, or simply not exiting (hanging) in task manager?

Best to patch your VM backup host up to 7.6.0.4, as it has some fixes on Vmware backup including this one:

http://www.symantec.com/docs/TECH223790

0 Kudos

backup-botw
Level 6

‎08-07-2015 07:16 AM

I dont have an issue with hung bpbkar, but rather bpfis.

0 Kudos

sclind
Moderator
Moderator
   VIP   

‎08-07-2015 07:33 AM

I am at 7.6.1.2.

If the problem persists (I have made the change already) I wil check bpbkar.

 

0 Kudos

sclind
Moderator
Moderator
   VIP   

‎08-10-2015 08:53 AM

Unfortunately this not not help my problem.

I dont see any hung bpbkar processes.

0 Kudos