Netbackup VMware failures with 6 and 156

Question

We recently just upgraded from 7.7.2 to 8.1.2 and we have been experiencing&nbsp;this issue where some VMs in the policy will run and others will fail with 6 and 156.&nbsp;VMs will flip flop with failure and succeed but all have this in its job details:&nbsp;Sep&nbsp;2,&nbsp;2019&nbsp;8:24:11&nbsp;PM&nbsp;-&nbsp;Info&nbsp;nbjm&nbsp;(pid=49500)&nbsp;starting&nbsp;backup&nbsp;job&nbsp;(jobid=324433)&nbsp;for&nbsp;client&nbsp;xxxxxxx,&nbsp;policy&nbsp;VSPHERE_VMIP_DMZ_xxxxxxx,&nbsp;schedule&nbsp;Diff-IncrSep&nbsp;2,&nbsp;2019&nbsp;8:24:11&nbsp;PM&nbsp;-&nbsp;Info&nbsp;nbjm&nbsp;(pid=49500)&nbsp;requesting&nbsp;STANDARD_RESOURCE&nbsp;resources&nbsp;from&nbsp;RB&nbsp;for&nbsp;backup&nbsp;job&nbsp;(jobid=324433,&nbsp;request&nbsp;id:{62E04202-A3F2-4555-BE68-5B2325894649})Sep&nbsp;2,&nbsp;2019&nbsp;8:24:11&nbsp;PM&nbsp;-&nbsp;requesting&nbsp;resource&nbsp;&nbsp;DataDomainSep&nbsp;2,&nbsp;2019&nbsp;8:24:11&nbsp;PM&nbsp;-&nbsp;requesting&nbsp;resource&nbsp;&nbsp;xxxxxxxx.NBU_CLIENT.MAXJOBS.xxxxxxSep&nbsp;2,&nbsp;2019&nbsp;8:24:11&nbsp;PM&nbsp;-&nbsp;granted&nbsp;resource&nbsp;&nbsp;xxxxxxx.NBU_CLIENT.MAXJOBS.xxxxxxxxxSep&nbsp;2,&nbsp;2019&nbsp;8:24:11&nbsp;PM&nbsp;-granted&nbsp;resource&nbsp;&nbsp;MediaID=@aaaab;DiskVolume=xxxxxxxx;DiskPool=xxxxxxx;Path=xxxxxxxx;StorageServer=xxxxxdedupe;MediaServer=xxxxxSep&nbsp;2,&nbsp;2019&nbsp;8:24:11&nbsp;PM&nbsp;-&nbsp;granted&nbsp;resource&nbsp;&nbsp;xxxxxxxSep&nbsp;2,&nbsp;2019&nbsp;8:24:11&nbsp;PM&nbsp;-&nbsp;estimated&nbsp;128863274&nbsp;kbytes&nbsp;neededSep&nbsp;2,&nbsp;2019&nbsp;8:24:11&nbsp;PM&nbsp;-&nbsp;Info&nbsp;nbjm&nbsp;(pid=49500)&nbsp;started&nbsp;backup&nbsp;(backupid=xxxx_1567470251)&nbsp;job&nbsp;for&nbsp;client&nbsp;xxxxx,&nbsp;policy&nbsp;VSPHERE_VMIP_DMZ_xxxxx,&nbsp;schedule&nbsp;Diff-Incr&nbsp;on&nbsp;storage&nbsp;unit&nbsp;xxxxxx&nbsp;using&nbsp;backup&nbsp;host&nbsp;xxxxxxxSep&nbsp;2,&nbsp;2019&nbsp;8:24:11&nbsp;PM&nbsp;-&nbsp;started&nbsp;process&nbsp;bpbrm&nbsp;(pid=289432)Sep&nbsp;2,&nbsp;2019&nbsp;8:24:12&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bpbrm&nbsp;(pid=289432)&nbsp;xxxxxx&nbsp;is&nbsp;the&nbsp;host&nbsp;to&nbsp;backup&nbsp;data&nbsp;fromSep&nbsp;2,&nbsp;2019&nbsp;8:24:12&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bpbrm&nbsp;(pid=289432)&nbsp;reading&nbsp;file&nbsp;list&nbsp;for&nbsp;clientSep&nbsp;2,&nbsp;2019&nbsp;8:24:13&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bpbrm&nbsp;(pid=289432)&nbsp;accelerator&nbsp;enabledSep&nbsp;2,&nbsp;2019&nbsp;8:24:16&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bpbrm&nbsp;(pid=289432)&nbsp;starting&nbsp;bpbkar32&nbsp;on&nbsp;clientSep&nbsp;2,&nbsp;2019&nbsp;8:24:16&nbsp;PM&nbsp;-&nbsp;connectingSep&nbsp;2,&nbsp;2019&nbsp;8:24:16&nbsp;PM&nbsp;-&nbsp;connected;&nbsp;connect&nbsp;time:&nbsp;0:00:00Sep&nbsp;2,&nbsp;2019&nbsp;8:24:31&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bpbkar32&nbsp;(pid=288800)&nbsp;Backup&nbsp;startedSep&nbsp;2,&nbsp;2019&nbsp;8:24:32&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bpbkar32&nbsp;(pid=288800)&nbsp;accelerator&nbsp;enabled&nbsp;backup,&nbsp;archive&nbsp;bit&nbsp;processing:&lt;disabled&gt;Sep&nbsp;2,&nbsp;2019&nbsp;8:24:33&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bptm&nbsp;(pid=293528)&nbsp;startSep&nbsp;2,&nbsp;2019&nbsp;8:24:33&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bptm&nbsp;(pid=293528)&nbsp;using&nbsp;1048576&nbsp;data&nbsp;buffer&nbsp;sizeSep&nbsp;2,&nbsp;2019&nbsp;8:24:33&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bptm&nbsp;(pid=293528)&nbsp;setting&nbsp;receive&nbsp;network&nbsp;buffer&nbsp;to&nbsp;4195328&nbsp;bytesSep&nbsp;2,&nbsp;2019&nbsp;8:24:33&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bptm&nbsp;(pid=293528)&nbsp;using&nbsp;256&nbsp;data&nbsp;buffersSep&nbsp;2,&nbsp;2019&nbsp;8:24:36&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bptm&nbsp;(pid=293528)&nbsp;start&nbsp;backupSep&nbsp;2,&nbsp;2019&nbsp;8:24:38&nbsp;PM&nbsp;-&nbsp;begin&nbsp;writingSep&nbsp;2,&nbsp;2019&nbsp;8:26:52&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bpbkar32&nbsp;(pid=288800)&nbsp;INF&nbsp;-&nbsp;Backing&nbsp;up&nbsp;vCenter&nbsp;server&nbsp;xxxxxxx,&nbsp;ESX&nbsp;host&nbsp;xxxxxxxx,&nbsp;BIOS&nbsp;UUID&nbsp;sxxx,&nbsp;Instance&nbsp;UUID&nbsp;xxxxx,&nbsp;Display&nbsp;Name&nbsp;xxxxx,&nbsp;Hostname&nbsp;xxxxxxxxxxSep&nbsp;2,&nbsp;2019&nbsp;8:28:20&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bpbkar32&nbsp;(pid=288800)&nbsp;accelerator&nbsp;sent&nbsp;0&nbsp;bytes&nbsp;out&nbsp;of&nbsp;0&nbsp;bytes&nbsp;to&nbsp;server,&nbsp;optimization&nbsp;0.0%Sep&nbsp;2,&nbsp;2019&nbsp;8:28:20&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bptm&nbsp;(pid=293528)&nbsp;waited&nbsp;for&nbsp;full&nbsp;buffer&nbsp;1&nbsp;times,&nbsp;delayed&nbsp;14136&nbsp;timesSep&nbsp;2,&nbsp;2019&nbsp;8:28:20&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bptm&nbsp;(pid=293528)&nbsp;EXITING&nbsp;with&nbsp;status&nbsp;90&nbsp;&lt;----------Sep&nbsp;2,&nbsp;2019&nbsp;8:28:21&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bpbkar32&nbsp;(pid=288800)&nbsp;bpbkar&nbsp;waited&nbsp;0&nbsp;times&nbsp;for&nbsp;empty&nbsp;buffer,&nbsp;delayed&nbsp;0&nbsp;times.Sep&nbsp;2,&nbsp;2019&nbsp;8:28:23&nbsp;PM&nbsp;-&nbsp;Error&nbsp;bpbrm&nbsp;(pid=289432)&nbsp;could&nbsp;not&nbsp;send&nbsp;server&nbsp;status&nbsp;message&nbsp;to&nbsp;clientSep&nbsp;2,&nbsp;2019&nbsp;8:28:26&nbsp;PM&nbsp;-&nbsp;Critical&nbsp;bpbrm&nbsp;(pid=289432)&nbsp;unexpected&nbsp;termination&nbsp;of&nbsp;client&nbsp;xxxxxxxxxSep&nbsp;2,&nbsp;2019&nbsp;8:28:26&nbsp;PM&nbsp;-&nbsp;Info&nbsp;bpbkar32&nbsp;(pid=0)&nbsp;done.&nbsp;status:&nbsp;6:&nbsp;the&nbsp;backup&nbsp;failed&nbsp;to&nbsp;back&nbsp;up&nbsp;the&nbsp;requested&nbsp;filesSep&nbsp;2,&nbsp;2019&nbsp;8:28:26&nbsp;PM&nbsp;-&nbsp;end&nbsp;writing;&nbsp;write&nbsp;time:&nbsp;0:03:48User&nbsp;backup&nbsp;failed&nbsp; (6)

lowell_palecek · Answer

Investigate why bpbkar32 sent 0 bytes to bptm. Check the bpbkar log on the backup host. Is it reasonable for there to be nothing to back up (e.g., nothing new on an incremental backup)?
Status 90 (no data received) from bptm is consistent with bpbkar32 logging that it sent no data.
Investigate why bpbrm logged "could not send to client" and "unexpected termination of client." I suppose without proof that the client bpbrm referred to was bpbkar32 on the backup host. Had bpbkar32 exited by then? Did bpbkar32.exe crash?

Forum Discussion

Netbackup VMware failures with 6 and 156

Related Content

vmware credential validation failure

Netbackup failure with code 24

Archive Task Failure

Netbackup-error: vmware credential validation failed

Netbackup VMware licensing requirements