partial SQL logs backup failing with error 23 & 27

[LIMRA]

Hi Experts

We have  few SQL  databases backup failing with error 23 & 27 but those all databases are running from same server.

Netbackup version  7.5.0.3

Master Server : Linux RHEL 2.6

Media Server : Windows 2008 R2

02/21/2014 12:31:02 - Info dbclient (pid=1164) INF - backup log "model" to VIRTUAL_DEVICE='VNBU0-1164-6196-1392946281' with  stats = 10, blocksize = 65536, maxtransfersize = 65536, buffercount = 1
02/21/2014 12:31:02 - Info dbclient (pid=1164) INF - Number of stripes: 1, Number of buffers per stripe 1.
02/21/2014 12:31:02 - Info dbclient (pid=1164) INF - Created VDI object for SQL Server instance <SQL1>. Connection timeout is <300> seconds.
02/21/2014 12:31:23 - Info dbclient (pid=1164) ERR - Error in VxBSACreateObject: 3.
02/21/2014 12:31:23 - Info dbclient (pid=1164)     CONTINUATION: - System detected error, operation aborted.
02/21/2014 12:31:23 - Info dbclient (pid=1164) ERR - Error in CPipeServer::CreateAgentMetadata: 6.
02/21/2014 12:31:23 - Info dbclient (pid=1164)     CONTINUATION: - The system cannot find the file specified.
02/21/2014 12:31:23 - Info dbclient (pid=1164) ERR - Error in VxBSACreateObject: 6.
02/21/2014 12:31:23 - Info dbclient (pid=1164)     CONTINUATION: - The handle used to associate this call with a previous VxBSAInit() call is invalid.

Regards!

[RamNagalla]

hello,

do you see anything in the event viewer?

are you trying to encrypt the backups if yes, see the bleow T/N once

http://www.symantec.com/business/support/index?page=content&id=TECH46812

[Marianne]

See if SQL logs contain any useful info:

ERRORLOG and vdi.log in <install_path>\mssql\log directory

I have also seen a similar post where the OP found 'something' online that pointed to a fix in NBU 7.5.0.5... 
Very confusing post that started with same error as yours and then turned into issue with media!
 https://www-secure.symantec.com/connect/forums/sql-backup-failing-0 

This post points to hanging dbbackex.exe that was resolved in NBU 7.5.0.4: 
https://www-secure.symantec.com/connect/forums/sql-jobs-randomly-failing-and-locking-databases-7501 

So, best to patch client to 7.5.0.5 or later.
Higher patch level on Client is supported as per Version Compatibility TN:  http://www.symantec.com/docs/TECH29677 
 

[LIMRA]

@ Nagalla :  eventviewer of client or media server?

How can I confirm it is trying to encrypt or not?

@ Marianne:    Is that from client or media server   - Note both are Windows

???

See if SQL logs contain any useful info:

ERRORLOG and vdi.log in <install_path>\mssql\log directory

It could not be media issue, backups writing to PDDO

And I am afraid If I would have problem like you mentioned  "hanging dbbackex.exe that was resolved in NBU 7.5.0.4:"  , its includes lot of effort to get done.

regards

[Marianne]

SQL logs are on the SQL server (NBU client) where the backups are failing.

I was merely referring to the media issue because I feel the wrong post was finally marked as Solution in that discussion. I wanted you to see that the OP seemed to have found 'something' pointing to a fix in NBU 7.5.0.5. Unfortunately no details. (Have you had a look at that post yet? )

Please check if there are still dbbackex.exe processes in Task Manager on the client after backup has completed/failed. This will confirm hanging/orphaned dbbackex.exe processes. 

If NBU 7.5.0.4 or later is going to fix your issue, it will be worth the effort, don't you think?

[LIMRA]

Hi Marrianne

We have plan in 2 or 3 weeks to upgrade master/media server to 7.5.0.7.

There are lot of  VM windows clients which having this issue, I have opened a case with symantec support as well which is still going. On one of the media server we found bptm.exe faulty in Windows Event Log and we have done test by removing that media server for few days, and then we did not see any error 27 since for any SQL child jobs, but few of the SQL parent job just stay active , neither failed nor kicked any  sql db/log child job. Still trying to workout that.

04/08/2014 19:30:01 - Info dbclient (pid=3580) INF - BACKUP STARTED USING
04/08/2014 19:30:01 - Info dbclient (pid=3580) Microsoft SQL Server 2008 R2 (SP2) - 10.50.4286.0 (X64)  
04/08/2014 19:30:01 - Info dbclient (pid=3580) May 29 2013 15:04:02  
04/08/2014 19:30:01 - Info dbclient (pid=3580) Copyright (c) Microsoft Corporation
04/08/2014 19:30:01 - Info dbclient (pid=3580) Enterprise Edition (64-bit) on Windows NT 6.1 <X64> (Build 7601: Service Pack 1) (Hypervisor)  
04/08/2014 19:30:01 - Info dbclient (pid=3580) Batch = F:\SQL1\Scripts\Netbackup\Db_transaction_bkup.bch, Op# = 1
04/08/2014 19:30:01 - Info dbclient (pid=3580) INF - Using backup image AUP13SQ003NABWI.MSSQL7.AUP13SQ003NABWI\SQL1.trx.EVVSDomJrnlVS1_16.~.7.001of001.20140408193001..C
04/08/2014 19:30:02 - Info dbclient (pid=3580) INF - backup log "EVVSDomJrnlVS1_16" to VIRTUAL_DEVICE='VNBU0-3580-1544-1396949402' with  stats = 10, blocksize = 65536, maxtransfersize = 65536, buffercount = 1
04/08/2014 19:30:02 - Info dbclient (pid=3580) INF - Number of stripes: 1, Number of buffers per stripe 1.
04/08/2014 19:30:02 - Info dbclient (pid=3580) INF - Created VDI object for SQL Server instance <SQL1>. Connection timeout is <300> seconds.
04/08/2014 19:37:03 - Info dbclient (pid=3580) ERR - Error in VxBSACreateObject: 3.
04/08/2014 19:37:03 - Info dbclient (pid=3580)     CONTINUATION: - System detected error, operation aborted.
04/08/2014 19:37:03 - Info dbclient (pid=3580) ERR - Error in CPipeServer::CreateAgentMetadata: 6.
04/08/2014 19:37:03 - Info dbclient (pid=3580)     CONTINUATION: - The system cannot find the file specified.
04/08/2014 19:37:04 - Info dbclient (pid=3580) ERR - Error in VxBSACreateObject: 6.
04/08/2014 19:37:04 - Info dbclient (pid=3580)     CONTINUATION: - The handle used to associate this call with a previous VxBSAInit() call is invalid.

[Marianne · Accepted solution]

Have you considered patching just one of these affected clients in the meantime?

You will see in Version Compatibility TN: http://www.symantec.com/docs/TECH29677  that higher point patch  on client is perfectly supported.

Have you looked at SQL Errorlog and VDI logs on one of these clients?

[LIMRA]

Hi  Marianne

we have upgraded the Netbackup Envo to 7.5.0.6  and looks the bug was dbback.exe was active , had to manually kill to start fresh backup. I guess. but sure  7.5.0.6 helps