network connection timed out (41)

[netbackup22]

4 sept. 2019 11:38:27 - Info nbjm (pid=5140) starting backup job (jobid=120485) for client srv-nav2009.snmvt.intra, policy P_DB_SQL, schedule Differential-Inc

4 sept. 2019 11:38:27 - Info nbjm (pid=5140) requesting MEDIA_SERVER_ONLY resources from RB for backup job (jobid=120485, request id:{9AA4F57B-2463-46A1-95DF-DF6DFB424F3C})

4 sept. 2019 11:38:27 - requesting resource sauv01py112-hcart-robot-tld-0

4 sept. 2019 11:38:27 - requesting resource sauv01py112.snmvt.intra.NBU_POLICY.MAXJOBS.P_DB_SQL

4 sept. 2019 11:38:27 - granted resource  sauv01py112.snmvt.intra.NBU_POLICY.MAXJOBS.P_DB_SQL

4 sept. 2019 11:38:27 - estimated 0 kbytes needed

4 sept. 2019 11:38:27 - Info nbjm (pid=5140) started backup (backupid=srv-nav2009.snmvt.intra_1567593507) job for client srv-nav2009.snmvt.intra, policy P_DB_SQL, schedule Differential-Inc on storage unit 

4 sept. 2019 11:38:27 - started process bpbrm (pid=9148)

4 sept. 2019 11:38:29 - Info bpbrm (pid=9148) srv-nav2009.snmvt.intra is the host to backup data from

4 sept. 2019 11:38:29 - Info bpbrm (pid=9148) reading file list for client

4 sept. 2019 11:38:30 - connecting

4 sept. 2019 11:38:33 - Info bpbrm (pid=9148) starting bphdb on client

4 sept. 2019 11:38:33 - Info bphdb (pid=5920) Backup started

4 sept. 2019 11:38:33 - connected; connect time: 0:00:00

4 sept. 2019 11:38:35 - Info dbclient (pid=5624) INF - Intelligent policy initiated by server <sauv01py112.snmvt.intra>

4 sept. 2019 11:38:36 - Info dbclient (pid=5624) INF - Intelligent policy <P_DB_SQL> is valid. The input file <C:\Program Files\Veritas\NetBackup\dbext\mssql\input.5920.0.0904119.113833> has been parsed successfully

4 sept. 2019 11:38:37 - Info dbclient (pid=5624) INF - <NAV2009> is a Failover Cluster Instance. Using virtual name <SRV-NAV2009.snmvt.intra>

4 sept. 2019 11:38:38 - Info dbclient (pid=5624) INF - Instance server name resolved to: <SRV-NAV2009>

4 sept. 2019 11:38:39 - Info dbclient (pid=5624) INF - Intelligent policy <P_DB_SQL>: skipping a <INCREMENTAL> backup on the master database since SQL Server will not allow it.

4 sept. 2019 11:38:40 - Info dbclient (pid=5624) INF - WARNING - Database <BASEPROD6> is in the <RECOVERY_PENDING> state and will generate backup failures.

4 sept. 2019 11:38:41 - Info dbclient (pid=5624) INF - Intelligent policy <P_DB_SQL> has resulted in <3> batch operation(s).

4 sept. 2019 11:38:41 - Info dbclient (pid=5624) INF - Intelligent policy <P_DB_SQL>: <3> operation(s) will protect incremental databases.

4 sept. 2019 11:38:42 - Info dbclient (pid=5624) INF - Intelligent policy <P_DB_SQL>: <1> databases(s) do not support the current operation and will be skipped.

4 sept. 2019 11:38:43 - Info dbclient (pid=5624) INF - Intelligent policy <P_DB_SQL>: <1> databases(s) are offline.  <0> of these will be skipped per the policy options.

4 sept. 2019 11:38:44 - Info dbclient (pid=5624) INF - Intelligent policy preprocessing has finished, created temporary batch file <__11_38_40_591_00.bch>, and submitted it to the backend agent for processing

4 sept. 2019 11:38:49 - Info dbclient (pid=5624) DBMS MSG - ODBC return code <-1>, SQL State <37000>, SQL Message <945><[Microsoft][SQL Native Client][SQL Server]La base de donn�es 'BASEPROD6' ne peut pas �tre ouverte, car des fichiers sont inaccessibles, ou la m�moire ou l'espace disque sont insuffisants. Pour plus d'informations, consultez le journal des erreurs de SQL Server.>.

4 sept. 2019 11:38:50 - Info dbclient (pid=5624) ERR - Error found executing <use "BASEPROD6">.

4 sept. 2019 11:38:50 - Info dbclient (pid=5624) ERR - Unable to determine FILESTREAM paths for database BASEPROD6.

4 sept. 2019 11:38:51 - Info dbclient (pid=5624) INF - OPERATION #1 of batch __11_38_40_591_00.bch FAILED with STATUS 1 (0 is normal). Elapsed time = 6(6) seconds.

4 sept. 2019 11:38:55 - Info dbclient (pid=5624) INF - BACKUP STARTED USING

4 sept. 2019 11:38:55 - Info dbclient (pid=5624) Microsoft SQL Server 2008 R2 (SP1) - 10.50.2789.0 (X64) 

4 sept. 2019 11:38:55 - Info dbclient (pid=5624) Sep 28 2011 17:10:21 

4 sept. 2019 11:38:55 - Info dbclient (pid=5624) Copyright (c) Microsoft Corporation

4 sept. 2019 11:38:55 - Info dbclient (pid=5624) Standard Edition (64-bit) on Windows NT 6.1 <X64> (Build 7601: Service Pack 1) 

4 sept. 2019 11:38:55 - Info dbclient (pid=5624) Batch = __11_38_40_591_00.bch, Op# = 2

4 sept. 2019 11:38:56 - Info dbclient (pid=5624) INF - Using backup image SQL04PY100.MSSQL7.SRV-NAV2009\NAV2009.inc.model.~.7.001of001.20190904113853..C

4 sept. 2019 11:38:56 - Info dbclient (pid=5624) INF - backup database "model" to VIRTUAL_DEVICE='VNBU0-5624-6676-1567593535' with  stats = 10, blocksize = 65536, maxtransfersize = 4194304, buffercount = 2, differential

4 sept. 2019 11:38:57 - Info dbclient (pid=5624) INF - Setting backup catalog name to: srv-nav2009.snmvt.intra

4 sept. 2019 11:38:57 - Info dbclient (pid=5624) INF - Number of stripes: 1, Number of buffers per stripe 2.

4 sept. 2019 11:38:58 - Info dbclient (pid=5624) INF - Created VDI object for SQL Server instance <NAV2009>. Connection timeout is <300> seconds.

4 sept. 2019 20:50:16 - Info bphdb (pid=5920) done. status: 41: network connection timed out

4 sept. 2019 20:50:16 - end writing

network connection timed out  (41)

 dear all 

what is the problem i have and how to resolve this later

[sdo]

My first guess is TCP keep alive.  Search this forum for topics on "keep alive" or "keepalive" or "keep-alive".  Remember, TCP keep alive is a "TCP stack" configuration setting... and "TCP stacks" exist on both operating systems and on network switches... so... just because your operating system may appear to have nice long TCP keep alive, maybe your network switch does not.

[Marianne]

We cannot tell you how how to resolve this, only how to troubleshoot.

Look at the 2 processes that are running on the SQL client: 
dbclient and bphdb.

You need to look at logs for these 2 processes to determine what is wrong.
Level 3 logs should be fine. 

Check above NBU logs, Windows Application logs as well as SQL errorlog and VDI logs to see what happened between these timestamps:

4 sept. 2019 11:38:58 - Info dbclient (pid=5624) INF - Created VDI object for SQL Server instance <NAV2009>. Connection timeout is <300> seconds.

4 sept. 2019 20:50:16 - Info bphdb (pid=5920) done. status: 41: network connection timed out

[Marianne]

@netbackup22 

More logs that will be helpful to troubleshoot the 'break in communication':

bpbrm and bptm on media server sauv01py112.  
Logging level 3 as well.
bpbrm will log metadata updates received from SQL client.
bptm will log each time full data buffer is received from SQL client.