Critical bpbrm (pid=2404) from client FcWin2K12: open of C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.FcWin2K12_1456353194.1.0 failed, errno 2

[nimUser]

Hi

    I am doing a VSS backup of Hyper V Cluster Running on Windows 2012 R2. The virtual machine being backedup is as well 2012 R2 with integration services installed.

We find the 2 status code 156 and 48 those as per the job the backup is successful.

Here is the complete details of  job

02/24/2016 14:33:14 - Info nbjm (pid=364) starting backup job (jobid=38) for client FcWin2K12, policy fcWinDaytona, schedule Full
02/24/2016 14:33:14 - Info nbjm (pid=364) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=38, request id:{EA933AD8-D602-4383-B567-4B82F7660CC9})
02/24/2016 14:33:14 - requesting resource NBU-stu
02/24/2016 14:33:14 - requesting resource nbu76-win2012.miracle.ns.NBU_CLIENT.MAXJOBS.FcWin2K12
02/24/2016 14:33:14 - requesting resource nbu76-win2012.miracle.ns.NBU_POLICY.MAXJOBS.fcWinDaytona
02/24/2016 14:33:14 - granted resource  nbu76-win2012.miracle.ns.NBU_CLIENT.MAXJOBS.FcWin2K12
02/24/2016 14:33:14 - granted resource  nbu76-win2012.miracle.ns.NBU_POLICY.MAXJOBS.fcWinDaytona
02/24/2016 14:33:14 - granted resource  MediaID=@aaaab;DiskVolume=F:\;DiskPool=NBU;Path=F:\;StorageServer=nbu76-win2012.miracle.ns;MediaServer=nbu76-win2012.miracle.ns
02/24/2016 14:33:14 - granted resource  NBU-stu
02/24/2016 14:33:20 - estimated 16836546 kbytes needed
02/24/2016 14:33:20 - begin Parent Job
02/24/2016 14:33:20 - begin Hyper-V: Start Notify Script
02/24/2016 14:33:24 - Info RUNCMD (pid=2928) started
02/24/2016 14:33:25 - Info RUNCMD (pid=2928) exiting with status: 0
Operation Status: 0
02/24/2016 14:33:25 - end Hyper-V: Start Notify Script; elapsed time 0:00:05
02/24/2016 14:33:25 - begin Hyper-V: Step By Condition
Operation Status: 0
02/24/2016 14:33:25 - end Hyper-V: Step By Condition; elapsed time 0:00:00
Operation Status: 0
02/24/2016 14:33:25 - end Parent Job; elapsed time 0:00:05
02/24/2016 14:33:25 - begin Hyper-V: Create Snapshot
02/24/2016 14:33:29 - started process bpbrm (pid=7136)
02/24/2016 14:33:30 - Info bpbrm (pid=7136) FcWin2K12 is the host to backup data from
02/24/2016 14:33:35 - Info bpbrm (pid=7136) reading file list for client
02/24/2016 14:33:35 - Info bpbrm (pid=7136) start bpfis on client
02/24/2016 14:33:35 - Info bpbrm (pid=7136) Starting create snapshot processing
02/24/2016 14:33:36 - Info bpfis (pid=1672) Backup started
02/24/2016 14:37:05 - Info bpfis (pid=1672) done. status: 0
02/24/2016 14:37:05 - end Hyper-V: Create Snapshot; elapsed time 0:03:40
02/24/2016 14:37:05 - Info bpfis (pid=1672) done. status: 0: the requested operation was successfully completed
02/24/2016 14:37:05 - end writing
Operation Status: 0
02/24/2016 14:37:05 - Info nbjm (pid=364) snapshotid=FcWin2K12_1456353194
02/24/2016 14:37:05 - begin Hyper-V: Policy Execution Manager Preprocessed
Operation Status: 0
02/24/2016 14:42:58 - end Hyper-V: Policy Execution Manager Preprocessed; elapsed time 0:05:53
02/24/2016 14:42:58 - begin Hyper-V: Validate Image
Operation Status: 0
02/24/2016 14:43:03 - end Hyper-V: Validate Image; elapsed time 0:00:05
02/24/2016 14:43:03 - begin Hyper-V: Delete Snapshot
02/24/2016 14:43:08 - started process bpbrm (pid=2404)
02/24/2016 14:43:09 - Info bpbrm (pid=2404) Starting delete snapshot processing
02/24/2016 14:43:38 - Info bpfis (pid=8540) Backup started
02/24/2016 14:43:53 - Critical bpbrm (pid=2404) from client FcWin2K12: open of C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.FcWin2K12_1456353194.1.0 failed, errno 2
02/24/2016 14:43:56 - Info bpfis (pid=8540) done. status: 156

02/24/2016 14:43:56 - end Hyper-V: Delete Snapshot; elapsed time 0:00:53
02/24/2016 14:43:56 - Info bpfis (pid=8540) done. status: 156: snapshot error encountered
02/24/2016 14:43:56 - end writing
Operation Status: 156

02/24/2016 14:43:56 - begin Hyper-V: End Notify Script
02/24/2016 14:44:00 - Info RUNCMD (pid=1552) started
02/24/2016 14:44:01 - Info RUNCMD (pid=1552) exiting with status: 0
Operation Status: 0
02/24/2016 14:44:01 - end Hyper-V: End Notify Script; elapsed time 0:00:05
Operation Status: 0
02/24/2016 14:44:15 - Error bpbrm (pid=6744) bpcd on FcWin2K12 exited with status 48: client hostname could not be found
02/24/2016 14:44:19 - Info  (pid=0) done. status: 48: client hostname could not be found
the requested operation was successfully completed  (0)
 

[nbutech]

Create bpbrm and bpfis legacy logging at verbose = 5 and retry the job

Does this happens with only 1 virtual machine which is win 2012...

Are there any other vm's with win 2012 and do they work...?

[nbutech]

Check is the  VSS Hyper V Writer is stable

vssadmin list writers...also check event viewer

[nimUser]

This happens with every Windows 2012.

[nimUser]

C:\Users\administrator>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {ee0951ce-20b8-4697-a348-35b4be47804b}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {7f2d96e0-ee8c-48e1-ac3a-f6563741b78e}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Hyper-V VSS Writer'
   Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Instance Id: {b447372a-8bcb-4222-8f12-987eedcdcac7}
   State: [5] Waiting for completion
   Last error: No error

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {6b7f91a7-7d7d-4cd6-8bb4-ba264b8149d3}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {edf5fa85-ab92-45fd-b365-8998276eb419}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {59e6d9d4-04a1-43ff-873a-8657e6bb3857}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {ba6227bd-441a-4977-8357-a86761cdb52f}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {0419008e-0bcc-4647-bd45-0420fe175c19}
   State: [1] Stable
   Last error: No error

Writer name: 'Cluster Shared Volume VSS Writer'
   Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
   Writer Instance Id: {bfca470d-5128-462b-a624-df4af77483ee}
   State: [1] Stable
   Last error: No error

Writer name: 'Cluster Database'
   Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
   Writer Instance Id: {ddc5ac00-5db6-464b-af7d-a9e05a66feee}
   State: [1] Stable
   Last error: No error

C:\Users\administrator>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Provider name: 'Microsoft CSV Shadow Copy Helper Provider'
   Provider type: Software
   Provider Id: {26d02d81-6aac-4275-8504-b9c6edc5261d}
   Version: 1.0.0.1

Provider name: 'Microsoft CSV Shadow Copy Provider'
   Provider type: Software
   Provider Id: {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff}
   Version: 1.0.0.1

Provider name: 'Nimble Storage VSS Provider'
   Provider type: Hardware
   Provider Id: {5253d47a-55bb-4b3a-928d-5f4baec6fbf4}
   Version: 2.3.2.287

Provider name: 'CommVault VSS Hardware Provider Service'
   Provider type: Hardware
   Provider Id: {68e1af3a-03cf-402e-a2f3-a595b4e7ac72}
   Version: 1.0.0.1

Provider name: 'Microsoft File Share Shadow Copy provider'
   Provider type: Fileshare
   Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
   Version: 1.0.0.1

Provider name: 'Galaxy VSS Provider Service'
   Provider type: Software
   Provider Id: {a4b3467a-2462-442b-8699-bbad76968c76}
   Version: 1.0.0.1

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

BTW the storage is from Nimble and Nimble VSS Hardware Provider is being used.

[nbutech]

https://www.veritas.com/support/en_US/article.000067923

To use a hardware array snapshot, make sure that the hardware array's VSS provider supports the snapshots that involve the Hyper-V writer. Check the release notes of the array vendor or VSS provider.

[nimUser]

Yes. I have checked and it works perfectly fine with both Commvault as well as Veeam.