ERR - Error opening the snapshot disks using given...

tuan_nguyen1 · ‎06-08-2017

Dear everybody,

Please help me to check this issue with VMWare backup:

===

6/8/2017 8:41:17 PM - Info bpbrm(pid=12872) HCM-MAIL-MB2 is the host to backup data from

6/8/2017 8:41:17 PM - Info bpbrm(pid=12872) reading file list for client

6/8/2017 8:41:17 PM - Info bpbrm(pid=12872) accelerator enabled

6/8/2017 8:41:17 PM - Info bpbrm(pid=12872) There is no complete backup image match with track journal, a regular full backup will be performed.

6/8/2017 8:41:17 PM - Info bpbrm(pid=12872) starting bpbkar on client

6/8/2017 8:41:18 PM - Info bpbkar(pid=12879) Backup started

6/8/2017 8:41:18 PM - Info bpbrm(pid=12872) bptm pid: 12880

6/8/2017 8:41:18 PM - Info bptm(pid=12880) start

6/8/2017 8:41:28 PM - Info bptm(pid=12880) using 1048576 data buffer size

6/8/2017 8:41:28 PM - Info bptm(pid=12880) using 512 data buffers

6/8/2017 8:41:30 PM - Info bptm(pid=12880) start backup

6/8/2017 8:42:05 PM - Info nbjm(pid=7032) starting backup job (jobid=11739) for client HCM-MAIL-MB2, policy mb2, schedule Full_monthly

6/8/2017 8:42:05 PM - estimated 0 Kbytes needed

6/8/2017 8:42:05 PM - Info nbjm(pid=7032) started backup (backupid=HCM-MAIL-MB2_1496929325) job for client HCM-MAIL-MB2, policy mb2, schedule Full_monthly on storage unit dedup_pool-stu using backup host mediaserver

6/8/2017 8:42:06 PM - started process bpbrm (12872)

6/8/2017 8:42:07 PM - connecting

6/8/2017 8:42:07 PM - connected; connect time: 0:00:00

6/8/2017 8:42:21 PM - begin writing

6/8/2017 8:45:37 PM - Error bpbrm(pid=12872) from client HCM-MAIL-MB2: ERR - Error opening the snapshot disks using given transport mode: san Status 23

6/8/2017 8:45:38 PM - Critical bpbrm(pid=12872) from client HCM-MAIL-MB2: FTL - cleanup() failed, status 6

6/8/2017 8:45:40 PM - Error bptm(pid=12880) media manager terminated by parent process

6/8/2017 8:47:20 PM - Info mediaserver(pid=12880) StorageServer=PureDisk:mediaserver; Report=PDDO Stats for (mediaserver): scanned: 3 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%, cache disabled

6/8/2017 8:47:20 PM - Info bpbkar(pid=0) done. status: 6: the backup failed to back up the requested files

6/8/2017 8:48:10 PM - end writing; write time: 0:05:49

the backup failed to back up the requested files (6)

VxMS log: https://1drv.ms/u/s!ApOpf2tIugUZgYs9ljgp2ZSZDZps-A

Thank you.

ndt

Thiago_Ribeiro · ‎06-08-2017

Hi,

what transport mode type that are you using? Other thing, give us more informations about your environment?

Example

NBU Master/Media version;

Vsphere version;

Regards,

Thiago Ribeiro

Lowell_Palecek · ‎06-08-2017

>Error bpbrm(pid=12872) from client HCM-MAIL-MB2...

>Info bpbkar(pid=0) done. status: 6: the backup failed to back up the requested files

When bpbrm reports an error "from client," find out more about the client process. Look for what comes before this error in the bpbkar log on HCM-MAIL-MB2.

Lowell_Palecek · ‎06-08-2017

See this thread for a discussion of the same error:

https://vox.veritas.com/t5/NetBackup/vmware-backup-failed-using-san-transport-mode/td-p/831940/jump-...

Marianne · ‎06-09-2017

Please give us all relevant info:

What is configured as VM backup host?
Master? Media server? a client?

Can you confirm that datastores are zoned/mapped to the backup host and can be seen at OS-level?

Was SAN transport mode explicitly selected in the policy?
Have you tried nbd?

Are other VMs backing up fine?
If so, which transport mode?

See this TN:

VMware Transport Modes: Best practices and troubleshooting
http://www.veritas.com/docs/000094725

PS:
Are you aware that NBU 7.6.x ran out of support 4 months ago?

Handy NetBackup Links

tuan_nguyen1 · ‎06-09-2017

Hi,

@Thiago_Ribeiro

More about my environment:

- transport mode: san
- We have Exchange DAG with 2 nodes running on vmware machine
- Master and media server have the same netbackup version
- I create 2 type of policy to backup DAG
Exchange backup with VMware
Exchange agent (like physical)

- As the previous post, we try to backup VM only (HCM-MAIL-MB2) for testing and getting more info about this issue.

- When I run backup non-VMware, there is only HCM-MAIL-MB1 was taken snapshot and backup completed. HCM-MAIL-MB2 go to fail with below message:

=== log ===
6/8/2017 7:10:29 PM - Info bpbrm(pid=4304) HCM-MAIL-MB2 is the host to backup data from
6/8/2017 7:10:29 PM - Info bpbrm(pid=4304) reading file list for client
6/8/2017 7:10:29 PM - Info bpbrm(pid=4304) start bpfis on client
6/8/2017 7:10:29 PM - Info bpbrm(pid=4304) Starting create snapshot processing
6/8/2017 7:10:30 PM - Info bpfis(pid=5316) Backup started
6/8/2017 7:11:04 PM - Info nbjm(pid=7032) starting backup job (jobid=11733) for client HCM-MAIL-MB2, policy hcm-mail-dag, schedule full
6/8/2017 7:11:04 PM - Info nbjm(pid=7032) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=11733, request id:{AF13163A-DBD7-4948-85D2-2D69DEBEED5E})
6/8/2017 7:11:04 PM - requesting resource dedup_pool-stu
6/8/2017 7:11:04 PM - requesting resource masterserver.NBU_CLIENT.MAXJOBS.HCM-MAIL-MB2
6/8/2017 7:11:04 PM - requesting resource masterserver.NBU_POLICY.MAXJOBS.hcm-mail-dag
6/8/2017 7:11:04 PM - granted resource masterserver.NBU_CLIENT.MAXJOBS.HCM-MAIL-MB2
6/8/2017 7:11:04 PM - granted resource masterserver.NBU_POLICY.MAXJOBS.hcm-mail-dag
6/8/2017 7:11:04 PM - granted resource MediaID=@aaaa8;DiskVolume=PureDiskVolume;DiskPool=dedup_pool;Path=PureDiskVolume;StorageServer=mediaserver;MediaServer=mediaserver
6/8/2017 7:11:04 PM - granted resource dedup_pool-stu
6/8/2017 7:11:04 PM - estimated 0 Kbytes needed
6/8/2017 7:11:04 PM - begin Parent Job
6/8/2017 7:11:04 PM - begin Exchange 14 Snapshot, Step By Condition
Status 0
6/8/2017 7:11:04 PM - end Exchange 14 Snapshot, Step By Condition; elapsed time: 0:00:00
6/8/2017 7:11:04 PM - begin Exchange 14 Snapshot, Read File List
Status 0
6/8/2017 7:11:04 PM - end Exchange 14 Snapshot, Read File List; elapsed time: 0:00:00
6/8/2017 7:11:04 PM - begin Exchange 14 Snapshot, Create Snapshot
6/8/2017 7:11:04 PM - started
6/8/2017 7:11:17 PM - started process bpbrm (4304)
6/8/2017 7:54:10 PM - Critical bpbrm(pid=4304) from client HCM-MAIL-MB2: FTL - vfm_freeze: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze
6/8/2017 7:54:10 PM - Critical bpbrm(pid=4304) from client HCM-MAIL-MB2: FTL -
6/8/2017 7:54:10 PM - Critical bpbrm(pid=4304) from client HCM-MAIL-MB2: FTL - snapshot preparation failed, status 156
6/8/2017 7:54:10 PM - Warning bpbrm(pid=4304) from client HCM-MAIL-MB2: WRN - Microsoft Information Store:\HCM DB04 is not frozen
6/8/2017 7:54:11 PM - Info bpfis(pid=5316) done. status: 156
6/8/2017 7:54:11 PM - end Exchange 14 Snapshot, Create Snapshot; elapsed time: 0:43:07
6/8/2017 7:54:11 PM - Info bpfis(pid=5316) done. status: 156: snapshot error encountered
6/8/2017 7:54:12 PM - Info bpbrm(pid=8338) Starting delete snapshot processing
6/8/2017 7:54:13 PM - Info bpfis(pid=21420) Backup started
6/8/2017 7:54:15 PM - Critical bpbrm(pid=8338) from client HCM-MAIL-MB2: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.hcm-mail-dag_1496923864.1.0
6/8/2017 7:54:16 PM - Info bpfis(pid=21420) done. status: 4207
6/8/2017 7:54:16 PM - end Parent Job; elapsed time: 0:43:12
6/8/2017 7:54:16 PM - Info bpfis(pid=21420) done. status: 4207: Could not fetch snapshot metadata or state files
6/8/2017 7:55:00 PM - end writing
Status 156
6/8/2017 7:55:00 PM - end Exchange 14 Snapshot, Create Snapshot; elapsed time: 0:43:56
6/8/2017 7:55:00 PM - begin Exchange 14 Snapshot, Stop On Error
Status 0
6/8/2017 7:55:00 PM - end Exchange 14 Snapshot, Stop On Error; elapsed time: 0:00:00
6/8/2017 7:55:00 PM - begin Exchange 14 Snapshot, Delete Snapshot
6/8/2017 7:55:01 PM - started process bpbrm (8338)
6/8/2017 7:55:05 PM - end writing
Status 4207
6/8/2017 7:55:05 PM - end Exchange 14 Snapshot, Delete Snapshot; elapsed time: 0:00:05
Status 156
6/8/2017 7:55:05 PM - end Parent Job; elapsed time: 0:44:01
snapshot error encountered (156)

=============

So, I try to backup HCM-MAIL-MB2 only (VMware backup) --> error code 6, 23 as my first post.

Then, I try to run backup DAG with VMware nbd mode, It seems that running but with very slow speed. The Application State Capture on HCM-MAIL-MB2:

=== log ===
6/9/2017 6:00:18 PM - Info nbjm(pid=7032) starting backup job (jobid=11767) for client HCM-MAIL-MB2.pvep.com.vn, policy hcm-mail-dag_VMware, schedule full
6/9/2017 6:00:18 PM - Info nbjm(pid=7032) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=11767, request id:{17BD3A27-B64D-4BEC-9FCA-B5B3D069AFA5})
6/9/2017 6:00:18 PM - requesting resource dedup_pool-stu
6/9/2017 6:00:18 PM - requesting resource masterserver.NBU_CLIENT.MAXJOBS.HCM-MAIL-MB2.pvep.com.vn
6/9/2017 6:00:18 PM - granted resource masterserver.NBU_CLIENT.MAXJOBS.HCM-MAIL-MB2.pvep.com.vn
6/9/2017 6:00:20 PM - estimated 0 Kbytes needed
6/9/2017 6:00:20 PM - begin Parent Job
6/9/2017 6:00:20 PM - begin Application State Capture, Step By Condition
Status 0
6/9/2017 6:00:20 PM - end Application State Capture, Step By Condition; elapsed time: 0:00:00
6/9/2017 6:00:20 PM - begin Application State Capture, Application State Capture
6/9/2017 6:43:17 PM - Critical ascc(pid=5380) Exchange: vfm_freeze: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze
6/9/2017 6:43:17 PM - Critical ascc(pid=5380) Exchange:
6/9/2017 6:43:17 PM - Info ascc(pid=5380) Exchange: snapshot preparation successful, with the following information: Found the Symantec VSS Provider installed on this VM.
6/9/2017 6:43:17 PM - Warning ascc(pid=5380) Exchange: snapshot preparation successful, with the following condition: It appears that the Symantec VSS Provider did not execute properly on the previous VM snapshot.
6/9/2017 6:43:17 PM - Critical ascc(pid=5380) Exchange: snapshot preparation failed, status 156
Status 13
6/9/2017 6:44:06 PM - end Application State Capture, Application State Capture; elapsed time: 0:43:46
Status 13
6/9/2017 6:44:06 PM - end Parent Job; elapsed time: 0:43:46
file read failed (13)

======

on MB1:

=== log ===
6/9/2017 6:00:18 PM - Info nbjm(pid=7032) starting backup job (jobid=11766) for client HCM-MAIL-MB1.pvep.com.vn, policy hcm-mail-dag_VMware, schedule full
6/9/2017 6:00:18 PM - Info nbjm(pid=7032) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=11766, request id:{426EA02C-D18F-43D4-8A93-66B4AB05A9DC})
6/9/2017 6:00:18 PM - requesting resource dedup_pool-stu
6/9/2017 6:00:18 PM - requesting resource masterserver.NBU_CLIENT.MAXJOBS.HCM-MAIL-MB1.pvep.com.vn
6/9/2017 6:00:18 PM - granted resource masterserver.NBU_CLIENT.MAXJOBS.HCM-MAIL-MB1.pvep.com.vn
6/9/2017 6:00:18 PM - estimated 0 Kbytes needed
6/9/2017 6:00:18 PM - begin Parent Job
6/9/2017 6:00:18 PM - begin Application State Capture, Step By Condition
Status 0
6/9/2017 6:00:18 PM - end Application State Capture, Step By Condition; elapsed time: 0:00:00
6/9/2017 6:00:18 PM - begin Application State Capture, Application State Capture
6/9/2017 6:00:31 PM - Info ascc(pid=5372) Exchange: snapshot preparation successful, with the following information: Found the Symantec VSS Provider installed on this VM.
6/9/2017 6:00:31 PM - Warning ascc(pid=5372) Exchange: snapshot preparation successful, with the following condition: It appears that the Symantec VSS Provider did not execute properly on the previous VM snapshot.
6/9/2017 6:00:31 PM - Warning ascc(pid=5372) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\HCM Archive' is not supported because volume <L:\> is on a GPT disk and is not supported.
6/9/2017 6:00:31 PM - Warning ascc(pid=5372) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\HCM Archive' is not supported because volume <M:\> is on a GPT disk and is not supported.
6/9/2017 6:00:32 PM - Warning ascc(pid=5372) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\HCM Archive' is not supported because volume <L:\> is on a GPT disk and is not supported.
6/9/2017 6:00:32 PM - Warning ascc(pid=5372) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\HCM DB01' is not supported because volume <D:\> is on a GPT disk and is not supported.
6/9/2017 6:00:32 PM - Warning ascc(pid=5372) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\HCM DB01' is not supported because volume <E:\> is on a GPT disk and is not supported.
6/9/2017 6:00:32 PM - Warning ascc(pid=5372) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\HCM DB01' is not supported because volume <D:\> is on a GPT disk and is not supported.
6/9/2017 6:00:32 PM - Warning ascc(pid=5372) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\HCM DB02' is not supported because volume <F:\> is on a GPT disk and is not supported.
6/9/2017 6:00:32 PM - Warning ascc(pid=5372) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\HCM DB02' is not supported because volume <G:\> is on a GPT disk and is not supported.
6/9/2017 6:00:32 PM - Warning ascc(pid=5372) Exchange: snapshot preparation successful, with the following condition: GRT Functionality for Database 'Microsoft Information Store:\HCM DB02' is not supported because volume <F:\> is on a GPT disk and is not supported.
6/9/2017 6:00:33 PM - Info ascc(pid=5372) Exchange: Host HCM-MAIL-MB1.pvep.com.vn successfully protected Exchange and cataloged under HCM-MAIL-MB1
Status 1
6/9/2017 6:01:22 PM - end Application State Capture, Application State Capture; elapsed time: 0:01:04
Status 1
6/9/2017 6:01:22 PM - end Parent Job; elapsed time: 0:01:04
the requested operation was partially successful (1)

The job was successfully completed, but some files may have been
busy or inaccessible. See the problems report or the client's logs for more details.
===

@Lowell_Palecek: Thank you, I will check

@Marianne:

What is configured as VM backup host?
--> Mediaserver

Can you confirm that datastores are zoned/mapped to the backup host and can be seen at OS-level?
--> Yes, they all were mapped to mediaserver

Was SAN transport mode explicitly selected in the policy?
Have you tried nbd?
--> I tried with nbd and find the error on Application State Capture job as above (the backup job still writting now with slow speed)

Are other VMs backing up fine?
If so, which transport mode?
--> Yes, I have some other VMs still backing up fine with san transport mode

Are you aware that NBU 7.6.x ran out of support 4 months ago?
--> No, I dont know this info before read your note.

Thank you all.

NDT

Thiago_Ribeiro · ‎06-09-2017

Hi ,
Thanks for answer. So take a look below.
- When I run backup non-VMware, there is only HCM-MAIL-MB1 was taken snapshot and backup completed. HCM-MAIL-MB2 go to fail with below message:
Log.
6/8/2017 7:54:10 PM - Critical bpbrm(pid=4304) from client HCM-MAIL-MB2: FTL - vfm_freeze: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze
For me according this log message above, it a problem with Windows VSS. Check the VSS status using vssadmin list writers command. Probably the Writer (Microsoft Exchange Writer) or ( Microsoft Exchange Replica Writer), its depending on your Exchange version e-or configuration. After check, take a look these articles below.
Example
Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {2999e0a2-76fa-4a2a-a0e5-16094458a1b6}
State: [1] Waiting for completion
Last error: No error
1 - https://www.veritas.com/support/en_US/article.TECH181190
2 - https://support.symantec.com/en_US/article.TECH129148.html
#########################################################################################
- So, I try to backup HCM-MAIL-MB2 only (VMware backup) --> error code 6, 23 as my first post.Then, I try to run backup DAG with VMware nbd mode, It seems that running but with very slow speed. The Application State Capture on HCM-MAIL-MB2:
About VMware backup using san transport mode,you said the Luns are presented for VMware backup Host (media server), can you check this TN https://www.veritas.com/support/en_US/article.TECH183072 to check if everything is ok?
You said that is using Netbackup 7.6.x version, as @Marianne, said this version is not anymore supported by Veritas...Another thing NBU 7.6.x.x version have a lot problems with VMware backup, look this TN https://www.veritas.com/support/en_US/article.TECH199999 and see if this applies to your environment.
#########################################################################################
For this problem I have some questions:
Are you using Windows VSS or Symantec VSS Provider for this backup?
Are you backing up DAG passive mailbox databases? If yes, take a look this TN https://www.veritas.com/support/en_US/article.000025499
Log
6/9/2017 6:43:17 PM - Critical ascc(pid=5380) Exchange: vfm_freeze: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze
Log
6/9/2017 6:43:17 PM - Critical ascc(pid=5380) Exchange:
6/9/2017 6:43:17 PM - Info ascc(pid=5380) Exchange: snapshot preparation successful, with the following information: Found the Symantec VSS Provider installed on this VM.
6/9/2017 6:43:17 PM - Warning ascc(pid=5380) Exchange: snapshot preparation successful, with the following condition: It appears that the Symantec VSS Provider did not execute properly on the previous VM snapshot.
 
Regards,
Thiago Ribeiro

tuan_nguyen1 · ‎06-09-2017

Hi @Thiago_Ribeiro

My VSS information on MB2:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {88013dac-bad5-4131-92e9-7db9bf3d0748}
   State: [1] Stable
   Last error: No error

Writer name: 'Cluster Shared Volume VSS Writer'
   Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
   Writer Instance Id: {7a424d39-c059-42cf-9e64-0d0e556a8453}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {c2b13d20-3b72-4fb1-aecd-547e290d79b5}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {5a9f2e03-cc99-4698-8608-4de96cd7a823}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {cb7efdf7-98c5-4bdd-8a92-d24c3ba2204b}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {e25bdc1f-313b-4cd4-b1e5-53cdba03fd94}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {1319324c-5fca-455b-bf1c-8e220d9fc797}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {70bcebc6-690e-464f-bc26-2b1bde5a06b6}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Config Writer'
   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Writer Instance Id: {fe811791-5993-4ab7-9404-e1521f2f943b}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {0df1d139-05d6-4d59-ab73-4ac1a08b8620}
   State: [1] Stable
   Last error: No error

Writer name: 'MSMQ Writer (MSMQ)'
   Writer Id: {7e47b561-971a-46e6-96b9-696eeaa53b2a}
   Writer Instance Id: {6f62cb14-48d0-4cb0-b0bb-6006016991e2}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {2bc428e7-6e10-4e8b-a915-e116b13bccc6}
   State: [1] Stable
   Last error: No error

Writer name: 'Cluster Database'
   Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
   Writer Instance Id: {b280f25c-b6ca-474e-97bd-b0b2b2e36060}
   State: [1] Stable
   Last error: No error

Thank you.

NDT

tuan_nguyen1 · ‎06-09-2017

Hi,

bpbrm log when backup MB2 go to error: https://1drv.ms/u/s!ApOpf2tIugUZgYs-GF8iNW9zjH5cCQ

Regards,

NDT

Thiago_Ribeiro · ‎06-12-2017

Hi,

<2> db_CLIENTsend: reset client protocol version from 0 to 8
06:39:36.444 [12947] <2> db_getCLIENT: db_CLIENTreceive: no entity was found 227
06:39:36.444 [12947] <2> verify_client: ../bpbrm.c.42259: db_getCLIENT failed for CLIENT: HCM-MAIL-MB2
06:39:36.444 [12947] <2> verify_client: ../bpbrm.c.42429: db_getCLIENT failed: 227 227 0x000000e3
#############################################################
<2> bpcr_get_version_rqst: bpcd version: 07600004

<2> local_getHostInfo: hostname comparison failed
[12947]
<2> local_getHostInfo: hostname comparison failed

#############################################################
<2> bpbrm write_filelist: wrote CONTINUE on COMM_SOCK
06:42:00.629 [13010] <16> get_bpfis_msg: from client HCM-MAIL-MB1: ERR - Send bpfis state to masterserver failed. status = 44
##############################################################

Im checking this log. that you post, for now take a look this errors above. Its seems a client name resolution problem, can you run this commands below to check if there isnt any problem?

From master server:

Windows
<installpath>\Program Files\Veritas\NetBackup\bin\bpclntcmd -hn <cilent hostname>
<installpath>\Program Files\Veritas\NetBackup\bin\bpclntcmd -ip <client IP>
<installpath>\Program Files\Veritas\NetBackup\bin\admincmd\bptestbpcd -client <client hostname> -verbose
<installpath>\Program Files\Veritas\NetBackup\bin\admincmd\bptestbpcd -client <client hostname> -debug

From your Windows client:

<installpath>\Program Files\Veritas\NetBackup\bin\bpclntcmd -pn (this should return your cilent host as recognized by master, error if it does not match)

Regards,

Thiago Ribeiro

tuan_nguyen1 · ‎06-13-2017

Hi @Thiago_Ribeiro

From master server:

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd.exe -hn hcm-mail-dag
host hcm-mail-dag: hcm-mail-dag at 10.32.3.117
aliases:     hcm-mail-dag     10.32.3.117

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd.exe -hn hcm-mail-mb2
host hcm-mail-mb2: hcm-mail-mb2 at 10.32.3.114
aliases:     hcm-mail-mb2     10.32.3.114

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -ip 10.32.3.117
host 10.32.3.117: hcm-mail-dag at 10.32.3.117
aliases:     hcm-mail-dag     10.32.3.117

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -ip 10.32.3.114
host 10.32.3.114: hcm-mail-mb2 at 10.32.3.114
aliases:     hcm-mail-mb2     10.32.3.114

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client hcm-mail-mb2
1 1 1
192.168.10.238:44286 -> 10.32.3.114:1556
192.168.10.238:44287 -> 10.32.3.114:1556
192.168.10.238:44288 -> 10.32.3.114:1556

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client hcm-mail-dag
1 1 1
192.168.10.238:44290 -> 10.32.3.117:1556
192.168.10.238:44291 -> 10.32.3.117:1556
192.168.10.238:44292 -> 10.32.3.117:1556

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client hcm-mail-dag
-debug
12:13:44.236 [9608.5524] <2> bptestbpcd: VERBOSE = 0
12:13:44.236 [9608.5524] <2> read_client: dname=., offline=0, online_at=0 offlin
e_at=0
12:13:44.236 [9608.5524] <2> read_client: dname=.., offline=0, online_at=0 offli
ne_at=0
12:13:44.236 [9608.5524] <2> read_client: dname=CO_0, offline=0, online_at=0 off
line_at=0
12:13:44.236 [9608.5524] <2> read_client: dname=exdb_info.HCM Archive, offline=0
, online_at=0 offline_at=0
12:13:44.236 [9608.5524] <2> read_client: dname=exdb_info.HCM DB01, offline=0, o
nline_at=0 offline_at=0
12:13:44.236 [9608.5524] <2> read_client: dname=exdb_info.HCM DB02, offline=0, o
nline_at=0 offline_at=0
12:13:44.236 [9608.5524] <2> read_client: dname=exdb_info.HCM DB03, offline=0, o
nline_at=0 offline_at=0
12:13:44.236 [9608.5524] <2> read_client: dname=exdb_info.HCM DB04, offline=0, o
nline_at=0 offline_at=0
12:13:44.251 [9608.5524] <2> read_client: dname=host_info, offline=0, online_at=
0 offline_at=0
12:13:44.251 [9608.5524] <2> read_client: dname=OA_1495609400, offline=0, online
_at=1495609400 offline_at=0
12:13:44.251 [9608.5524] <2> db_freeEXDB_INFO: ?
12:13:44.267 [9608.5524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:13:44.267 [9608.5524] <2> logconnections: BPCD CONNECT FROM 192.168.10.238.44
303 TO 10.32.3.117.1556 fd = 488
12:13:44.282 [9608.5524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:13:44.298 [9608.5524] <8> do_pbx_service: [vnet_connect.c:2156] via PBX VNETD
 CONNECT FROM 192.168.10.238.44304 TO 10.32.3.117.1556 fd = 512
12:13:44.298 [9608.5524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnet
d.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
12:13:44.314 [9608.5524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnet
d.c:457] ipc_string 32315
12:13:44.376 [9608.5524] <2> bpcr_get_version_rqst: bpcd version: 07600004
1 1 1
192.168.10.238:44303 -> 10.32.3.117:1556
192.168.10.238:44304 -> 10.32.3.117:1556
12:13:44.392 [9608.5524] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:13:44.407 [9608.5524] <8> do_pbx_service: [vnet_connect.c:2156] via PBX VNETD
 CONNECT FROM 192.168.10.238.44305 TO 10.32.3.117.1556 fd = 516
12:13:44.407 [9608.5524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnet
d.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
12:13:44.423 [9608.5524] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnet
d.c:457] ipc_string 32317
192.168.10.238:44305 -> 10.32.3.117:1556
<2>bptestbpcd: EXIT status = 0
12:13:44.454 [9608.5524] <2> bptestbpcd: EXIT status = 0

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client hcm-mail-mb2
-debug
12:14:28.878 [1272.9060] <2> bptestbpcd: VERBOSE = 0
12:14:28.910 [1272.9060] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:14:28.910 [1272.9060] <2> logconnections: BPCD CONNECT FROM 192.168.10.238.44
309 TO 10.32.3.114.1556 fd = 488
12:14:28.925 [1272.9060] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:14:28.941 [1272.9060] <8> do_pbx_service: [vnet_connect.c:2156] via PBX VNETD
 CONNECT FROM 192.168.10.238.44310 TO 10.32.3.114.1556 fd = 512
12:14:28.941 [1272.9060] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnet
d.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
12:14:28.956 [1272.9060] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnet
d.c:457] ipc_string 28099
12:14:29.019 [1272.9060] <2> bpcr_get_version_rqst: bpcd version: 07600004
1 1 1
192.168.10.238:44309 -> 10.32.3.114:1556
192.168.10.238:44310 -> 10.32.3.114:1556
12:14:29.035 [1272.9060] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:14:29.050 [1272.9060] <8> do_pbx_service: [vnet_connect.c:2156] via PBX VNETD
 CONNECT FROM 192.168.10.238.44311 TO 10.32.3.114.1556 fd = 516
12:14:29.050 [1272.9060] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnet
d.c:440] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
12:14:29.050 [1272.9060] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnet
d.c:457] ipc_string 28101
192.168.10.238:44311 -> 10.32.3.114:1556
<2>bptestbpcd: EXIT status = 0
12:14:29.097 [1272.9060] <2> bptestbpcd: EXIT status = 0


C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client hcm-mail-mb2
-verbose
1 1 1
192.168.10.238:44316 -> 10.32.3.114:1556
192.168.10.238:44317 -> 10.32.3.114:1556
PEER_NAME = masterserver
HOST_NAME = hcm-mail-mb2.pvep.com.vn
CLIENT_NAME = hcm-mail-mb2.pvep.com.vn
VERSION = 0x07600004
PLATFORM = win_x64
PATCH_VERSION = 7.6.0.4
SERVER_PATCH_VERSION = 7.6.0.4
MASTER_SERVER = masterserver
EMM_SERVER = masterserver
NB_MACHINE_TYPE = CLIENT
192.168.10.238:44318 -> 10.32.3.114:1556

C:\Program Files\Veritas\NetBackup\bin\admincmd>bptestbpcd -client hcm-mail-dag
-verbose
1 1 1
192.168.10.238:44320 -> 10.32.3.117:1556
192.168.10.238:44321 -> 10.32.3.117:1556
PEER_NAME = masterserver
HOST_NAME = hcm-mail-mb1.pvep.com.vn
CLIENT_NAME = hcm-mail-mb1.pvep.com.vn
VERSION = 0x07600004
PLATFORM = win_x64
PATCH_VERSION = 7.6.0.4
SERVER_PATCH_VERSION = 7.6.0.4
MASTER_SERVER = masterserver
EMM_SERVER = masterserver
NB_MACHINE_TYPE = CLIENT
192.168.10.238:44322 -> 10.32.3.117:1556

From your Windows client:

Thiago_Ribeiro · ‎06-13-2017

Hi @tuan_nguyen1,

Thanks for answers. Can you check these TNs below?

To this one, check this TN - https://www.veritas.com/support/en_US/article.000025159

g_vixInterfaceLogger:libvix.cpp:1844 <DEBUG> : [VFM_ESINFO] 2017-06-08T20:41:32.894+07:00 [41F17940 info 'Libs'] VThreadBase detected multiple threads.

#########################################################################################

To this one check this TN - https://www.veritas.com/support/en_US/article.000116073

vdOpen:VixInterface.cpp:495 <ERROR> : VixDiskLib_Open() error: 13
openLeafSnapshotDisks:VixGuest.cpp:478 <DEBUG> : vdOpen() error = 13. Calling closeLeafSnapshotDisks()

Are you able to connect on 443 port from VMware backup host to Vcenter? I hope that some of these TN can help you.

Regards,

Thiago

HunkerDownOneMo · ‎06-13-2017

Do your VMs have duplicate BIOS UUIDs with any other VMs in your enviornment? If you are using SAN transport and NBU 773 or 8.0, you are hosed and will need to use NBD or another transport method - Veritas has not fixed this issue since 773.

Thiago_Ribeiro · ‎06-13-2017

Hi @tuan_nguyen1,

what transport mode have you marked in your nbu policy? Can you send us a screenshot?

Regards,

Thiago

tuan_nguyen1 · ‎06-13-2017

Hi @Thiago_Ribeiro,

Please check https://1drv.ms/w/s!ApOpf2tIugUZgYtNTQUd7zkNHvSMDw

Thank you.

NDT

Thiago_Ribeiro · ‎06-20-2017

Hi @tuan_nguyen1,

I was checking the VxMs logs that you posted previous, and I would like to know if you already check the TN below?

TN - https://www.veritas.com/support/en_US/article.000116073

VxMs Logs

06/08/2017 20:45:32 : openLeafSnapshotDisks:VixGuest.cpp:486 <TRACE> : vdOpen() succeess
06/08/2017 20:45:32 : openLeafSnapshotDisks:VixGuest.cpp:487 <INFO> : Transport mode in effect = san
06/08/2017 20:45:32 : openLeafSnapshotDisks:VixGuest.cpp:514 <TRACE> : Calling vdGetInfo()
06/08/2017 20:45:32 : g_vixInterfaceLogger:libvix.cpp:1844 <DEBUG> : [VFM_ESINFO] VixDiskLib: VixDiskLib_GetInfo: Retrieve disk info.

06/08/2017 20:45:32 : g_vixInterfaceLogger:libvix.cpp:1844 <DEBUG> : [VFM_ESINFO] VixDiskLib: VixDiskLib_FreeInfo: Clean up VixDiskLib.

06/08/2017 20:45:32 : openLeafSnapshotDisks:VixGuest.cpp:467 <DEBUG> : Calling vdOpen(Disk: [HUS110N-0004-MB2-DB01] HCM-MAIL-MB2/HCM-MAIL-MB2_1-000001.vmdk, Flags: 0x4)
06/08/2017 20:45:32 : vdOpen:VixInterface.cpp:467 <INFO> : Calling VixDiskLib_Open()
06/08/2017 20:45:36 : vdOpen:VixInterface.cpp:474 <DEBUG> : Done with VixDiskLib_Open(): 0
06/08/2017 20:45:36 : vdOpen:VixInterface.cpp:495 <ERROR> : VixDiskLib_Open() error: 13
06/08/2017 20:45:36 : openLeafSnapshotDisks:VixGuest.cpp:478 <DEBUG> : vdOpen() error = 13. Calling closeLeafSnapshotDisks()
06/08/2017 20:45:36 : vdClose:VixInterface.cpp:103 <TRACE> : Calling VixDiskLib_Close(): 464689424

Regards,

Thiago

VOX

ERR - Error opening the snapshot disks using given transport mode: san Status 23