database backups failing

[RajPr]

Hello,

Our database (9i, 10g and 11g) backups on solaris 9 & 10 have intermittently started failing with a mix of the following errors :

rman-03009, ora-19513 and ora-27206 OR ORA-19502, ORA-27030 OR ORA-19511:

Error received from media manager layer, error text:
   VxBSASendData: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve

In the netbackup logs, some times, we find this error - "writeToServer: ERR - send() to server on socket failed: Broken pipe (32)".

We are on version 7.5.0.3 (on master and media servers). We have enabled logging for a few of the client machines and are trying

to identify the root cause. Does anyone have any suggestions/ideas/recommendations ?

Thank you !

[RajPr]

In the dbclient logs, we do find following kind of messages.

05:01:31.846 [9856] <4> VxBSAQueryObject: INF - Object </PROD_h8ngpiq6_1_1> was not found in the NetBackup catalog.

05:01:31.846 [9856] <2> xbsa_QueryObject: INF - leaving (17)

05:01:31.846 [9856] <2> int_FindBackupImage: INF - PROD_h8ngpiq6_1_1 not found

We have synced the rman catalog, and staggered the backups, but, it hasn't helped.

[RajPr]

The client software is - NetBackup 6.5 0

[Marianne]

Please rename dbclient log to dbclient.txt and post entire log as attachment.

What are the Client Connect and Client Read timeout values on the media server?

The default of 300 (5 minutes) is normally insufficient for database backups. Increase both to 900 as a start.

Please also ensure that the following logs are enabled:

On master: bprd (NBU needs to be restarted for logging to be enabled)

On media server: bpbrm and bptm (no restart needed)

[RajPr]

Hello Marianne,

I do appreciate your quick response. I'll check w/our netbackup team regarding about other settings and enabling logging for other processes and update at the earliest feasible.

Thank you,

Rajeev

[RajPr]

fyi -

For the master server, client connect timeout=300 seconds, client read timeout =7200 seconds.

For the media servers, client connect timeout=300 seconds, client read timeout=7200 seconds.

Also, for both master and media servers, "Use OS dependent timeouts" option is checked.

Thank you !

[RajPr]

Hi Marianne,

I have been told that logging for "bprd (on master), bpbrm and bptm (on media server)" is already enabled.

Plus, we are on win 2008 R2 (SP1).

Thank you !

[Marianne]

Seems backup was doing well until we see 'broken pipe' in dbclient log:

15:15:02.810 [15173] <2> int_WriteData: INF - writing buffer # 20000 of size 262144
15:15:02.811 [15173] <4> dbc_put: INF - sending keepalive on NAME Socket
15:15:02.888 [15173] <4> dbc_put: INF - sending keepalive on NAME Socket
15:15:02.974 [15173] <16> writeToServer: ERR - send() to server on socket failed: Broken pipe (32)
15:15:02.974 [15173] <16> dbc_put: ERR - writeToServer() failed when sending keepalive

I cannot say what happened here - just that NBU is reporting network failure....

You may want to compare the same period in media server bptm and bpbrm logs to see until what time data and metadata was received.

PS: Any plans to upgrade clients in the near future?