12-21-2020 10:07 PM - edited 12-21-2020 10:08 PM
VMWARE backup failed with the backup failed to back up the requested files (6)
Error:
12/22/2020 13:43:09 - Info nbjm (pid=7540) starting backup job (jobid=279169) for client CNBJLNX400, policy 600-Cnbjmgt02, schedule Full
12/22/2020 13:43:10 - estimated 5072527900 kbytes needed
12/22/2020 13:43:10 - Info nbjm (pid=7540) started backup (backupid=CNBJLNX400_1608615789) job for client CNBJLNX400, policy 600-Cnbjmgt02, schedule Full on storage unit cnbjbup11-hcart2-robot-tld-12 using backup host cnbjbup11.corpusers.net
12/22/2020 13:43:14 - started process bpbrm (pid=2824)
12/22/2020 13:43:20 - Info bpbrm (pid=2824) CNBJLNX400 is the host to backup data from
12/22/2020 13:43:20 - Info bpbrm (pid=2824) reading file list for client
12/22/2020 13:43:20 - connecting
12/22/2020 13:43:21 - Info bpbrm (pid=2824) starting bpbkar32 on client
12/22/2020 13:43:21 - connected; connect time: 0:00:00
12/22/2020 13:43:22 - Info bpbkar32 (pid=8648) Backup started
12/22/2020 13:43:22 - Info bptm (pid=3300) start
12/22/2020 13:43:22 - Info bpbkar32 (pid=8648) archive bit processing:<enabled>
12/22/2020 13:43:23 - Info bptm (pid=3300) using 262144 data buffer size
12/22/2020 13:43:23 - Info bptm (pid=3300) setting receive network buffer to 262144 bytes
12/22/2020 13:43:23 - Info bptm (pid=3300) using 256 data buffers
12/22/2020 13:43:25 - Info bpbkar32 (pid=8648) INF - Backing up vCenter server cnbjmgt02.corpusers.net, ESX host cnbjvmh04b.corpusers.net, BIOS UUID 423cde11-26ab-409f-1c6d-4148808464a7, Instance UUID 503c954f-dd22-725b-2311-649ce929f00c, Display Name CNBJLNX400, Hostname cnbjlnx400
12/22/2020 13:43:25 - Info bptm (pid=3300) start backup
12/22/2020 13:43:25 - Info bptm (pid=3300) Waiting for mount of media id HT1UV2 (copy 1) on server cnbjbup11.corpusers.net.
12/22/2020 13:43:25 - mounting HT1UV2
12/22/2020 13:43:36 - Info bpbkar32 (pid=8648) bpbkar waited 0 times for empty buffer, delayed 0 times.
12/22/2020 13:43:39 - Error bpbrm (pid=2824) could not send server status message
12/22/2020 13:43:42 - Critical bpbrm (pid=2824) unexpected termination of client CNBJLNX400
12/22/2020 13:43:43 - Info bpbkar32 (pid=0) done. status: 6: the backup failed to back up the requested files
12/22/2020 13:43:43 - end writing
the backup failed to back up the requested files (6)
Can anyone please help me on this
12-21-2020 10:27 PM
Before we look at the log, please give us more info.
NBU version?
VCenter version?
OS on Master/media/backup host?
(Unix is not supported as backup host)
OS on VM being backed up?
Is the problem with this VM only, or with others as well?
Has backup worked before?
12-21-2020 10:38 PM
Hi Marianne
NBU version? 7.7.2
VCenter version? - 6.7.0
OS on Master/media/backup host? - Windows 2008 R2
OS on VM being backed up? Linux cnbjlnx400 4.4.0-189-generic #219-Ubuntu SMP Tue Aug 11 12:26:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Is the problem with this VM only, or with others as well? - Yes only with one VM server
Has backup worked before? - Yes till yesterday it is working fine
12-21-2020 11:36 PM
Do you know why there is 23 minutes difference between Job Details and VxMS log?
Could you please upload bpbkar log where we can trace PID 8648 ?
In all honesty, I am battling to find anything useful in VxMS log.
There are these warnings and errors towards the end of the file, but I have no idea what it means. Google search of the warnings and errors find nothing.
13:20:44.0149 : buildRvpBitmap:VixGuest.cpp:1998 <WARN> : Virtual disk optimization disabled
13:20:44.0305 : allocMemory:RvpBitmapX64.cpp:531 <ERROR> : Exception: VERIFY_WIN32_HANDLE
13:20:44.0305 : log:Error.cpp:249 <WARN> : Error: 0x00001070 occured in file D:/NB/7.7.2/src/vxms/plugin/common/map/rvp/RvpBitmapX64.cpp, at line 531
13:20:44.0305 : log:Error.cpp:249 <WARN> : Error: 0x00001070 occured in file D:/NB/7.7.2/src/vxms/plugin/common/map/rvp/RvpBitmapX64.cpp, at line 531
13:20:44.0305 : vixMapObjCtl:VixCoordinator.cpp:977 <WARN> : Returning: 4208
13:20:44.0305 : vix_map_objctl:libvix.cpp:1216 <WARN> : Returning: 4208
Looking at Status Code Ref Guide for status 4208, I'm not sure how relevant this is, since the master server seems to be your media server and backup host (client):
Message: Could not send snapshot metadata or statefiles
Explanation: Snapshot metadata that is created by the client cannot be transferred on the master server.
Recommended Action: Verify the following:
The NetBackup client can communicate with the NetBackup master.
The state file location on the NetBackup client has write permission. The typical location of the NetBackup state file on the NetBackup client is as follows:
Windows: C:\Program Files\Veritas\NetBackup\db\snapshot\<client>
UNIX: /usr/openv/netbackup/db/snapshot/<client>
12-21-2020 11:44 PM
You may also want to go through the list of possible reasons for Status 6 in the Status Code Ref Guide:
https://www.veritas.com/content/support/en_US/doc/44037985-127664609-0/v15096798-127664609
See if anything is relevant here.
12-22-2020 01:56 AM
Hello,
I agree with @Marianne ,vxms logs doesn't match with the detaillled status (there is a delay of 23 minutes) did you share a snip of logs of a first retry and detailled status of the second?
I can see that you used san, did you try with nbd?
please share bpbkar logs.
what are types of Filesystem on the vm ?
12-22-2020 06:54 AM
Just a long shot here, but how full is your TEMP space?
12-22-2020 06:00 PM
I ahve cleared all teh files in Temp folder
12-22-2020 09:06 PM
I Have tried clearing the path and initiated the backup but again it got failed with same error
12-22-2020 09:10 PM
The location has full permission access
12-22-2020 09:16 PM
bpbkar form master and media server correct?
12-22-2020 09:19 PM
root file system
12-22-2020 10:16 PM
@IT-Infrastructu bpbkar from backup host backup host cnbjbup11 (which seems to be the media server as well).
So, the same server where you collected VxMS log.
When you upload the log, please post Job Details of a failed job from the same date.
About filesystem - the question was about the Linux filesystem TYPE. e.g. ext4, XFS, etc.
01-02-2021 08:42 PM
One thing to ask/check - the NBU version and VMware versions you mention are not compatible/supported (assuming you listed them correctly). VMware 6.7 was only supported when NBU 8.1 came out. Now you do mention that the vCenter version is 6.7 which may mean you have a supported vSphere version in use - but something else to check.
Did the VM team by chance recent;y upgrade VMware?
The issue yuou are seeing may be some boundary condition you have reached on that VM that now makes it incompatible with the VDDK version that NetBackup uses.