07-23-2012 11:16 AM
Hello,
Our database (9i, 10g and 11g) backups on solaris 9 & 10 have intermittently started failing with a mix of the following errors :
rman-03009, ora-19513 and ora-27206 OR ORA-19502, ORA-27030 OR ORA-19511:
Error received from media manager layer, error text:
VxBSASendData: Failed with error:
Server Status: Communication with the server has not been initiated or the server status has not been retrieved from the serve
In the netbackup logs, some times, we find this error - "writeToServer: ERR - send() to server on socket failed: Broken pipe (32)".
We are on version 7.5.0.3 (on master and media servers). We have enabled logging for a few of the client machines and are trying
to identify the root cause. Does anyone have any suggestions/ideas/recommendations ?
Thank you !
07-23-2012 11:22 AM
In the dbclient logs, we do find following kind of messages.
05:01:31.846 [9856] <4> VxBSAQueryObject: INF - Object </PROD_h8ngpiq6_1_1> was not found in the NetBackup catalog.
05:01:31.846 [9856] <2> xbsa_QueryObject: INF - leaving (17)
05:01:31.846 [9856] <2> int_FindBackupImage: INF - PROD_h8ngpiq6_1_1 not found
We have synced the rman catalog, and staggered the backups, but, it hasn't helped.
07-23-2012 11:28 AM
The client software is - NetBackup 6.5 0
07-23-2012 11:31 AM
Please rename dbclient log to dbclient.txt and post entire log as attachment.
What are the Client Connect and Client Read timeout values on the media server?
The default of 300 (5 minutes) is normally insufficient for database backups. Increase both to 900 as a start.
Please also ensure that the following logs are enabled:
On master: bprd (NBU needs to be restarted for logging to be enabled)
On media server: bpbrm and bptm (no restart needed)
07-23-2012 11:55 AM
Hello Marianne,
I do appreciate your quick response. I'll check w/our netbackup team regarding about other settings and enabling logging for other processes and update at the earliest feasible.
Thank you,
Rajeev
07-23-2012 12:25 PM
fyi -
For the master server, client connect timeout=300 seconds, client read timeout =7200 seconds.
For the media servers, client connect timeout=300 seconds, client read timeout=7200 seconds.
Also, for both master and media servers, "Use OS dependent timeouts" option is checked.
Thank you !
07-23-2012 12:50 PM
Hi Marianne,
I have been told that logging for "bprd (on master), bpbrm and bptm (on media server)" is already enabled.
Plus, we are on win 2008 R2 (SP1).
Thank you !
07-23-2012 02:47 PM
Seems backup was doing well until we see 'broken pipe' in dbclient log:
15:15:02.810 [15173] <2> int_WriteData: INF - writing buffer # 20000 of size 262144 15:15:02.811 [15173] <4> dbc_put: INF - sending keepalive on NAME Socket 15:15:02.888 [15173] <4> dbc_put: INF - sending keepalive on NAME Socket 15:15:02.974 [15173] <16> writeToServer: ERR - send() to server on socket failed: Broken pipe (32) 15:15:02.974 [15173] <16> dbc_put: ERR - writeToServer() failed when sending keepalive