06-10-2020 11:06 AM
Client OS - Microsoft Windows Server 2008 R2 Enterprise
Master server OS : Red Hat Enterprise Linux Server"
NBU version : 8.1.2
Backup selection : ALL_LOCAL_DRIVES
Hello Team - Any idea why exclude drives still showing up in activity monitor and failing with status 25 & 71? This occurrence recently started 2 days ago after working fine for a while now. Could this be some bug related? Has anyone came across this before?
Solved! Go to Solution.
06-26-2020 01:49 PM
Hello All,
Very sorry for getting back late...Thank you so much for the contributions. Turns out even though these drives were disconnected from the servers, they are still mapped to the server which could be seen from "Disk management" under server manager. I went ahead to to remove both drives from exclusion list, restarted nbu services and that solved the issue .Thanks again all for your contributions.
06-10-2020 12:00 PM
Can you post the Detailed Status from the failing jobs, as well as the Exclusion List for the failing client? Is the failure occurring on an UNC path or CIFS share? If so, look at this: https://www.veritas.com/content/support/en_US/article.100046446
Lots of possible options, such as removable media being added/removed, new drive being mapped, etc.
Was this client recently patched? If everything was working before and suddenly began failing, that could be the culprit.
06-10-2020 12:43 PM
@EthanH - I think I pretty much have an idea what's going on. But meantime, We can eliminate error 71. But am still curious reason for exit status 25. detailed status below.
To answer your question :
Exclude drive = M:\ ( This drive is shared by 2 systems in cluster. This is a mapped drive from vmax)
Client regularly get patched which hasn't cause any issue before
===================================================================
Jun 10, 2020 1:40:27 AM - Info bpbrm (pid=219119) mickey is the host to backup data from
Jun 10, 2020 1:40:27 AM - Info bpbrm (pid=219119) reading file list for client
Jun 10, 2020 1:40:28 AM - Info bpbrm (pid=219119) accelerator enabled
Jun 10, 2020 1:40:32 AM - Info bpbrm (pid=219119) starting bpbkar on client
Jun 10, 2020 1:40:34 AM - Info bpbkar (pid=6576) Backup started
Jun 10, 2020 1:40:34 AM - Info bpbrm (pid=219119) bptm pid: 219247
Jun 10, 2020 1:40:34 AM - Info bpbkar (pid=6576) change time comparison:<enabled>
Jun 10, 2020 1:40:34 AM - Info bpbkar (pid=6576) accelerator enabled backup, archive bit processing:<disabled>
Jun 10, 2020 1:40:34 AM - Info bpbkar (pid=6576) not using change journal data for <M:\>: not supported for non-local volumes / file systems
Jun 10, 2020 1:40:34 AM - Info bptm (pid=219247) start
Jun 10, 2020 1:40:35 AM - Info bptm (pid=219247) using 262144 data buffer size
Jun 10, 2020 1:40:35 AM - Info bptm (pid=219247) using 30 data buffers
Jun 10, 2020 1:40:36 AM - Info bptm (pid=219247) start backup
Jun 10, 2020 1:40:43 AM - Info bpbrm (pid=219119) from client mickey: TRV - object not found for file system backup: M:
Jun 10, 2020 1:40:44 AM - Info bpbkar (pid=6576) accelerator sent 2048 bytes out of 2048 bytes to server, optimization 0.0%
Jun 10, 2020 1:40:44 AM - Error bptm (pid=219247) cannot add fragment to image database, error = cannot connect on socket
Jun 10, 2020 1:40:45 AM - Info bptm (pid=219247) EXITING with status 25 <----------
Jun 10, 2020 1:40:45 AM - Info mouse (pid=219247) StorageServer=PureDisk:mouse; Report=PDDO Stats for (mouse): scanned: 3 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%, cache disabled
Jun 10, 2020 1:40:45 AM - Error bpbrm (pid=219119) could not send server status message to client
Jun 10, 2020 1:40:46 AM - Info bpbkar (pid=6576) done. status: 25: cannot connect on socket
Jun 10, 2020 1:48:23 AM - Info nbjm (pid=25361) starting backup job (jobid=12992158) for client mickey, policy mouse_windows_2AM_Gold_v1, schedule mouse_Incremental_Daily
Jun 10, 2020 1:48:23 AM - Info nbjm (pid=25361) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=12992158, request id:{FEF7858C-AADD-11EA-A0AF-4ABA3473E158})
Jun 10, 2020 1:48:23 AM - requesting resource stu_disk_mouse
Jun 10, 2020 1:48:23 AM - requesting resource xyz.NBU_CLIENT.MAXJOBS.mickey
Jun 10, 2020 1:48:23 AM - granted resource xyz.NBU_CLIENT.MAXJOBS.mickey
Jun 10, 2020 1:48:23 AM - granted resource MediaID=@aaadw;DiskVolume=PureDiskVolume;DiskPool=dp_disk_mouse;Path=PureDiskVolume;StorageServer=mouse;MediaServer=mouse
Jun 10, 2020 1:48:23 AM - granted resource stu_disk_mouse
Jun 10, 2020 1:48:23 AM - estimated 0 kbytes needed
Jun 10, 2020 1:48:23 AM - Info nbjm (pid=25361) resumed backup (backupid=mickey_1591765395) job for client mickey, policy mouse_windows_2AM_Gold_v1, schedule mouse_Incremental_Daily on storage unit stu_disk_mouse
Jun 10, 2020 1:48:23 AM - started process bpbrm (pid=219119)
Jun 10, 2020 1:48:30 AM - connecting
Jun 10, 2020 1:48:31 AM - connected; connect time: 0:00:00
Jun 10, 2020 1:48:46 AM - end writing
cannot connect on socket (25)
06-10-2020 12:53 PM
What is the output of bptestbpcd -client mickey -verbose -debug?
06-10-2020 01:41 PM
@EthanH - Be aware there's not issue with client communication with master server. Other local Drives backup on the server are backing up as expected. My curiosity is basically center around why Excluded M:\ is showing up on and failing with 25. Am running couple of test in my dev environment. Will update my findings or solution
Here's truncated bptestbpcd outputs.
sudo /usr/openv/netbackup/bin/admincmd/bptestbpcd -client mickey -verbose -debug
PEER_NAME = xyz
HOST_NAME = mickey
CLIENT_NAME = mickey
VERSION = 0x07710000
PLATFORM = win_x64
PATCH_VERSION = 7.7.1.0
SERVER_PATCH_VERSION = 7.7.1.0
MASTER_SERVER = xyz
EMM_SERVER = xyz
NB_MACHINE_TYPE = CLIENT
SERVICE_TYPE = UNKNOWN
PROCESS_HINT =
<2>bptestbpcd: EXIT status = 0
15:55:35.399 [9973] <2> bptestbpcd: EXIT status = 0
06-10-2020 01:52 PM
Good luck! Status 25 is a tricky one to troubleshoot.
06-10-2020 03:07 PM
I think;
You should check,
*hostname resolution issues
* is client connected the master through ip which you want.
* clear host cache.
* check hosts file if client use it.
Good Luck.
06-10-2020 11:51 PM
I would start by looking at media server logs.
like there is some sort of comms issue between the media server and the master server after backup has been writing for a few seconds:
Jun 10, 2020 1:40:44 AM - Error bptm (pid=219247) cannot add fragment to image database, error = cannot connect on socket
...
Jun 10, 2020 1:40:45 AM - Error bpbrm (pid=219119) could not send server status message to client
Jun 10, 2020 1:40:46 AM - Info bpbkar (pid=6576) done. status: 25: cannot connect on socket
Can you get us level 3 logs of all of the following?
On media server: bptm and bpbrm
On client: bpbkar
06-11-2020 02:45 AM
Hey
Please put exclude list as M:\* I recall something that when I was having only a drive letter being excluded ie. M:\ I was seeing an error - although can't recall error code - but when I did add * so it would read M:\* the error was gone... I think this is not too much to add it there and test ;)
Good luck!
06-26-2020 01:49 PM
Hello All,
Very sorry for getting back late...Thank you so much for the contributions. Turns out even though these drives were disconnected from the servers, they are still mapped to the server which could be seen from "Disk management" under server manager. I went ahead to to remove both drives from exclusion list, restarted nbu services and that solved the issue .Thanks again all for your contributions.