cancel
Showing results for 
Search instead for 
Did you mean: 

NetBackup client-side deduplication multistreaming support

yobole
Level 6

I have a server 2012 client running Netbackup agent 7.5.0.6. the server has 4 volumes with size , 1TB , 1.5TB , 2 TB and TB . I have client side dedup enabled as well as change journal and accelerator.  However the large 6TB seems to fall over with a Failed write error 14 and an Incomplete status

All the other volume however complete ok

I know multistream support is possible with a maximum of 2 streams with have client side dedup enabled

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

yobole
Level 6

The backup seems to have kicked in . Found  the follwoing in the event log of the netbackup client .

“{Delayed Write Failed} Windows was unable to save all the data for the file \Device\HarddiskVolumeShadowCopy136\VxCJMon.dat; the data has been lost. This error may be caused if the device has been removed or the media is write-protected. “

Rebooting the server seems to have fixed it and and the backup of the laerge volume seems to  have kicked in .

The reference to Shadow Copy is a concern, is this required by NetBackup ?  the backups should be able to backup the live data a file at a time, there’s no need for this to be a point in time ??

 

View solution in original post

10 REPLIES 10

AAlmroth
Level 6
Partner Accredited

Status 14 is somewhat complicated to find the root cause for in many cases.

What happens if you create policy only backing up that 6TB partition? Disabling accelerator, client-side de-dup. Is the file system filled up? Enough room for the VSS snapshot? Are there many millions of files in file systems, where many are in the same directories?

By disabling all the "nice" features, and only trying this file system you may be able to trace the problem better in logs, such as bpbkar and Windows 2012 logs. 

Would it be possible to upgrade to 7.5.0.7, or even 7.6.0.1?

You could also be affected by the overhead that when using multi streams during baseline backup with client-side backup, that the memory and CPU processing of fingerprints takes to much of the resources. There is a technote on this.

/A

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Status 14 is either a 'real' File Read issue or else a network read error.

All text in Details tab will help to determine if this is file or network error.

Once we know what we are dealing with, we can continue with relevant troubleshooting.

yobole
Level 6

Still no joy  even with me disabling the "nice features" and runing the drive on its own 

 


1/16/2014 11:56:26 AM - started
1/16/2014 11:56:28 AM - started process bpbrm (11248)
1/16/2014 11:56:32 AM - Info bpbrm(pid=11248) xxx the host to backup data from    
1/16/2014 11:56:32 AM - Info bpbrm(pid=11248) reading file list from client       
1/16/2014 11:56:33 AM - connecting
1/16/2014 11:56:35 AM - Info bpbrm(pid=11248) starting bpbkar32 on client        
1/16/2014 11:56:35 AM - connected; connect time: 00:00:02
1/16/2014 11:56:37 AM - Info bpbkar32(pid=14648) Backup started          
1/16/2014 11:56:37 AM - Info bptm(pid=12860) start           
1/16/2014 11:56:39 AM - Info bptm(pid=12860) using 1048576 data buffer size       
1/16/2014 11:56:39 AM - Info bptm(pid=12860) setting receive network buffer to 4195328 bytes     
1/16/2014 11:56:39 AM - Info bptm(pid=12860) using 64 data buffers        
1/16/2014 11:56:41 AM - Info bptm(pid=12860) start backup          
1/16/2014 11:56:42 AM - Info bptm(pid=12860) backup child process is pid 10548.8096      
1/16/2014 11:56:42 AM - Info bptm(pid=10548) start           
1/16/2014 11:56:43 AM - begin writing
1/16/2014 11:58:41 AM - Info bpbkar32(pid=14648) change journal enabled for <D:\>   

Justs hangs for about 2 houes then

 Info bpbkar32(pid=4236) done. status: 14: file write failed 

All other drives are fine .. This 6TB however does have a lot files

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Mmmm not enough info there to tell us if it is network or file.

You are going to need logs to troubleshoot.

On client: bpbkar

On media server: bpbrm and bptm

Increase logging on client and media server to about level 3.

yobole
Level 6

15:19:58.973 [1340.14072] <8> file_to_cache_item: [vnet_addrinfo.c:6574] fopen() failed ERRNO=2 FILE=C:\Program Files\Veritas\NetBackup\var\host_cache\07a\286cae7a+vnetd,1,400,2,1,0+10.44.101.105.txt
15:19:58.973 [1340.14072] <2> retry_getaddrinfo: [vnet_addrinfo.c:908] using SVC=vnetd PORT=13724
15:19:58.973 [1340.14072] <2> retry_getaddrinfo: [vnet_addrinfo.c:912] adjusted service name 13724
15:19:58.973 [1340.14072] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1632] found via getaddrinfo NAME=10.44.101.105 SVC=vnetd
15:19:58.979 [1340.14072] <2> vnet_sortaddrs: [vnet_addrinfo.c:3954] sorted addrs: 1 0x1
15:19:58.979 [1340.14072] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4828] using interface  ANY
15:19:58.979 [1340.14072] <2> async_connect: [vnet_connect.c:1477] connect in progress 1 0x1
15:19:59.033 [1340.14072] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:19:59.575 [1340.14072] <8> do_pbx_service: [vnet_connect.c:2108] via PBX VNETD CONNECT FROM 10.44.101.183.50655 TO 10.44.101.105.1556 fd = 1128
15:19:59.575 [1340.14072] <2> async_connect: [vnet_connect.c:1644] connect async CONNECT FROM 10.44.101.183.50655 TO 10.44.101.105.1556 fd = 1128
15:19:59.575 [1340.14072] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO 10.44.101.105 10.44.101.105 vnetd VIA pbx
15:19:59.582 [1340.14072] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
15:19:59.800 [1340.14072] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:460] ipc_string 57139
15:19:59.995 [1340.14072] <2> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:471] hash_str1 a5ceb37d483579a13fe2d68283cc36c4
15:20:00.548 [1340.14072] <2> bpbrm Exit: OUT_SOCK from bpcr = 1340
15:20:00.548 [1340.14072] <2> bpbrm Exit: IN_SOCK from bpcr = 1128
15:20:00.748 [1340.14072] <2> bpcr_get_version_rqst: bpcd version: 07500004
15:20:00.949 [1340.14072] <2> bpcr_get_version_rqst: bpcd version: 07500004
15:20:01.350 [1340.14072] <2> bpcr_get_version_rqst: bpcd version: 07500004
15:20:01.551 [1340.14072] <2> set_job_details: Tfile (433832): LOG 1389885601 4 bpbkar32 8804 done. status: 14: file write failed

15:20:01.552 [1340.14072] <2> send_job_file: job ID 433832, ftype = 3 msg len = 67, msg = LOG 1389885601 4 bpbkar32 8804 done. status: 14: file write failed

15:20:01.552 [1340.14072] <2> bpbrm Exit: client backup EXIT STATUS 14: file write failed
15:20:01.594 [1340.14072] <2> vnet_close_socket_safely: [vnet.c:2029] safe close 0 0x0

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

We need ALL of these logs:

On client: bpbkar

On media server: bpbrm and bptm

Please copy logs to reflect process name (e.g. bpbkar.txt) and upload as File attachments.

yobole
Level 6

The server logs doent really say much

 

client logs (BPCD) (C:\Program Files\Veritas\NetBackup\logs\bpcd) seeing a lot of cannot open file and timeouts ,.. below  cannot see any bpbkar logs on the client

 

14:17:36.712 [5080.6020] <2> process_requests: offset to GMT 0

14:17:36.728 [5080.6020] <2> logconnections: BPCD ACCEPT FROM 10.44.101.183.58324 TO 10.44.101.105.1556 fd = 464

14:17:36.728 [5080.6020] <2> init_resilient_cache: [vnet_nbrntd.c:869] Initialize resilient cache. 0 0x0

14:17:36.728 [5080.6020] <2> process_requests: setup_sockopts complete

14:17:36.728 [5080.6020] <2> vnet_pcache_init_table: [vnet_private.c:235] starting cache size 200 0xc8

14:17:36.728 [5080.6020] <2> bpcd peer_hostname: Connection from host bak01lon1uk.uk.wspgroup.com (10.44.101.183) port 58324

14:17:36.728 [5080.6020] <2> vnet_pcache_init_table: [vnet_private.c:235] starting cache size 200 0xc8

14:17:36.728 [5080.6020] <4> bpcd valid_server: hostname comparison succeeded

14:17:36.743 [5080.6020] <2> process_requests: output socket port number = 1

14:17:37.055 [5080.6020] <2> vnet_in_resilient_network: [vnet_addrinfo.c:8771] ignoring local host 0 0x0

14:17:37.055 [5080.6020] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1585] in failed file cache ERR=10109 NAME=NULL SVC=testdaemon

14:17:37.055 [5080.6020] <2> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1275] vnet_cached_getaddrinfo_and_update() failed 6 0x6

14:17:37.055 [5080.6020] <8> vnet_cached_get_service_port: [vnet_addrinfo.c:2396] vnet_cached_getaddrinfo failed STAT=6 RV=10109 NAME=testdaemon

14:17:37.055 [5080.6020] <8> is_pbxable_service: [vnet_connect.c:2159] vnet_cached_get_service_port() failed 6 0x6

14:17:37.055 [5080.6020] <8> is_pbxable_service: [vnet_connect.c:2160] service 51937

14:17:37.055 [5080.6020] <8> file_to_cache_item: [vnet_addrinfo.c:6574] fopen() failed ERRNO=2 FILE=C:\Program Files\Veritas\NetBackup\var\host_cache\14f\6cf9fd4f+51937,1,400,2,1,0+127.0.0.1.txt

14:17:37.055 [5080.6020] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1632] found via getaddrinfo NAME=127.0.0.1 SVC=51937

14:17:37.071 [5080.6020] <2> vnet_sortaddrs: [vnet_addrinfo.c:3954] sorted addrs: 1 0x1

14:17:37.071 [5080.6020] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4795] Local [strong] check, using interface  ANY

14:17:37.071 [5080.6020] <2> async_connect: [vnet_connect.c:1477] connect in progress 1 0x1

14:17:37.071 [5080.6020] <2> async_connect: [vnet_connect.c:1644] connect async CONNECT FROM 127.0.0.1.51938 TO 127.0.0.1.51937 fd = 392

14:17:37.071 [5080.6020] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO 127.0.0.1 127.0.0.1 51937

14:17:37.071 [5080.6020] <2> vnet_receive_network_socket: [vnet_vnetd.c:1076] hash_str1 0c87eab7878fd4df89a31da2a91cfc09

14:17:37.071 [5080.6020] <2> vnet_receive_network_socket: [vnet_vnetd.c:1077] hash_str2 0cd2074b5e7694db08b3d7577085fa75

14:17:37.071 [5080.6020] <2> verify_hashes: [vnet_vnetd.c:1613] hash_str1 0c87eab7878fd4df89a31da2a91cfc09

14:17:37.071 [5080.6020] <2> verify_hashes: [vnet_vnetd.c:1615] hash_str2 0cd2074b5e7694db08b3d7577085fa75

14:17:37.071 [5080.6020] <8> verify_hashes: [vnet_vnetd.c:1641] hash_str 0c87eab7878fd4df89a31da2a91cfc09

14:17:37.087 [5080.6020] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:808] the socket is  412 0x19c

14:17:37.087 [5080.6020] <2> process_requests: Duplicated vnetd socket on stderr

14:17:37.087 [5080.6020] <2> process_requests: <---- NetBackup 7.5 0 ------------initiated

14:17:37.087 [5080.6020] <2> process_requests: VERBOSE = 5

14:17:37.087 [5080.6020] <2> process_requests: Not using VxSS authentication with bak01lon1uk.uk.wspgroup.com

14:17:37.133 [5080.6020] <2> process_requests:

14:17:37.274 [5080.6020] <2> process_requests: BPCD_GET_VERSION_RQST

14:17:37.477 [5080.6020] <2> process_requests: BPCD_DISCONNECT_RQST

14:17:37.477 [5080.6020] <2> bpcd exit_bpcd: exit status 0  ----------->exiting

14:17:37.477 [2564.2568] <2> vnet_pbxAcceptSocket: Accepted sock[488] from 10.44.101.183:58326

14:17:37.477 [2564.2568] <2> bpcd main: accept sock = 488

14:17:37.508 [6068.4732] <2> setup_debug_log: switched debug log file for bpcd

14:17:37.508 [6068.4732] <2> bpcd main: VERBOSE = 5

14:17:37.508 [6068.4732] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone

14:17:37.508 [6068.4732] <2> ReadKeyfile: keyfile C:\Program Files\Veritas\NetBackup\bin\keyfile.dat does not exist

 

14:17:37.508 [6068.4732] <2> process_requests: offset to GMT 0

14:17:37.524 [6068.4732] <2> logconnections: BPCD ACCEPT FROM 10.44.101.183.58326 TO 10.44.101.105.1556 fd = 488

14:17:37.524 [6068.4732] <2> init_resilient_cache: [vnet_nbrntd.c:869] Initialize resilient cache. 0 0x0

14:17:37.524 [6068.4732] <2> process_requests: setup_sockopts complete

14:17:37.524 [6068.4732] <2> vnet_pcache_init_table: [vnet_private.c:235] starting cache size 200 0xc8

14:17:37.524 [6068.4732] <2> vnet_cached_getnameinfo: [vnet_addrinfo.c:1904] found via getnameinfo OUR_HOST=IPSTR=10.44.101.183

14:17:37.524 [6068.4732] <2> bpcd peer_hostname: Connection from host (10.44.101.183) port 58326

14:17:37.524 [6068.4732] <2> bpcd valid_server: comparing

14:17:37.524 [6068.4732] <2> vnet_pcache_init_table: [vnet_private.c:235] starting cache size 200 0xc8

14:17:37.524 [6068.4732] <4> bpcd valid_server: hostname comparison succeeded

14:17:37.539 [6068.4732] <2> process_requests: output socket port number = 1

14:17:37.820 [6068.4732] <2> vnet_in_resilient_network: [vnet_addrinfo.c:8771] ignoring local host 0 0x0

14:17:37.820 [6068.4732] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1585] in failed file cache ERR=10109 NAME=NULL SVC=testdaemon

14:17:37.820 [6068.4732] <2> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1275] vnet_cached_getaddrinfo_and_update() failed 6 0x6

14:17:37.820 [6068.4732] <8> vnet_cached_get_service_port: [vnet_addrinfo.c:2396] vnet_cached_getaddrinfo failed STAT=6 RV=10109 NAME=testdaemon

14:17:37.820 [6068.4732] <8> is_pbxable_service: [vnet_connect.c:2159] vnet_cached_get_service_port() failed 6 0x6

14:17:37.820 [6068.4732] <8> is_pbxable_service: [vnet_connect.c:2160] service 51939

14:17:37.820 [6068.4732] <8> file_to_cache_item: [vnet_addrinfo.c:6574] fopen() failed ERRNO=2 FILE=C:\Program Files\Veritas\NetBackup\var\host_cache\14f\6cf9fd4f+51939,1,400,2,1,0+127.0.0.1.txt

14:17:37.820 [6068.4732] <2> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1632] found via getaddrinfo NAME=127.0.0.1 SVC=51939

14:17:37.836 [6068.4732] <2> vnet_sortaddrs: [vnet_addrinfo.c:3954] sorted addrs: 1 0x1

14:17:37.836 [6068.4732] <2> vnet_get_pref_netconnection: [vnet_addrinfo.c:4795] Local [strong] check, using interface  ANY

14:17:37.836 [6068.4732] <2> async_connect: [vnet_connect.c:1477] connect in progress 1 0x1

14:17:37.836 [6068.4732] <2> async_connect: [vnet_connect.c:1644] connect async CONNECT FROM 127.0.0.1.51940 TO 127.0.0.1.51939 fd = 396

14:17:37.836 [6068.4732] <2> connect_to_service: connect succeeded STATUS (0) SUCCESS FROM 0.0.0.0 TO 127.0.0.1 127.0.0.1 51939

14:17:37.836 [6068.4732] <2> vnet_receive_network_socket: [vnet_vnetd.c:1076] hash_str1 ab4ff315196ea318f4e2cfc1a51ceb2f

14:17:37.836 [6068.4732] <2> vnet_receive_network_socket: [vnet_vnetd.c:1077] hash_str2 b8cec65dc00b62d726167a9c3bcd7ec7

14:17:37.836 [6068.4732] <2> verify_hashes: [vnet_vnetd.c:1613] hash_str1 ab4ff315196ea318f4e2cfc1a51ceb2f

14:17:37.836 [6068.4732] <2> verify_hashes: [vnet_vnetd.c:1615] hash_str2 b8cec65dc00b62d726167a9c3bcd7ec7

14:17:37.836 [6068.4732] <8> verify_hashes: [vnet_vnetd.c:1641] hash_str ab4ff315196ea318f4e2cfc1a51ceb2f

14:17:37.851 [6068.4732] <2> vnet_check_resilient_socket: [vnet_nbrntd.c:808] the socket is  412 0x19c

14:17:37.851 [6068.4732] <2> process_requests: Duplicated vnetd socket on stderr

14:17:37.851 [6068.4732] <2> process_requests: <---- NetBackup 7.5 0 ------------initiated

14:17:37.851 [6068.4732] <2> process_requests: VERBOSE = 5

14:17:37.851 [6068.4732] <2> process_requests: Not using VxSS authentication with bak01lon1uk.uk.wspgroup.com

14:17:37.898 [6068.4732] <2> process_requests:

14:17:38.039 [6068.4732] <2> process_requests: BPCD_GET_VERSION_RQST

14:17:38.241 [6068.4732] <2> process_requests: BPCD_DISCONNECT_RQST

14:17:38.241 [6068.4732] <2> bpcd exit_bpcd: exit status 0  ----------->exiting

14:22:19.436 [4588.5704] <2> get_short_base: (1) cannot read (byte 1) from network: Connection timed out.

14:22:19.436 [4588.5704] <16> process_requests: token read: -3

14:22:19.436 [4588.5704] <4> process_requests:    h_errno = 10060 - Connection timed out.

14:22:19.436 [4588.5704] <16> bpcd main: process_requests returned 1

14:29:28.369 [2564.2568] <16> resync_host_cache: about to CreateProcess(..., "C:\Program Files\Veritas\NetBackup\bin\bpclntcmd.exe" -resync_host_cache 120, ...)

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Have you created bpbkar log folder?

Log folders do not exist by default. You need to create them.

Please have another look at request in my previous post: Check that all of these log folders exist.
If not, create them and retry backup.

On client: bpbkar

On media server: bpbrm and bptm

Please copy logs to reflect process name (e.g. bpbkar.txt) and upload as File attachments.

(Please do not copy entire log files in the body of forum posts.)

 

yobole
Level 6

The backup seems to have kicked in . Found  the follwoing in the event log of the netbackup client .

“{Delayed Write Failed} Windows was unable to save all the data for the file \Device\HarddiskVolumeShadowCopy136\VxCJMon.dat; the data has been lost. This error may be caused if the device has been removed or the media is write-protected. “

Rebooting the server seems to have fixed it and and the backup of the laerge volume seems to  have kicked in .

The reference to Shadow Copy is a concern, is this required by NetBackup ?  the backups should be able to backup the live data a file at a time, there’s no need for this to be a point in time ??

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

VSS is needed for Windows Open File Backup. 
WOFB is enabled by default for W2003 and later clients.

Details in NBU Admin Guide I.