Ask the Experts: Oracle backup Failing Time & Time Again!

[Maddy777]

Just recently got engaged with a customer who has a specific Oracle RMAN backup failing time & time again.  It was observed that it ran fine for about 2 hours before it bombed. 

Here is the error message. 

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on c2 channel at 05/04/2015 20:26:37
ORA-19502: write error on file "db_GMEPRD11_878847280_642482_1", blockno 1491713 (blocksize=1024)
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: Error received from media manager layer, error text:
VxBSASendData: Failed with error:
Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve

We are suspecting it to be a NIC issue (Probably NIC Hanging) and possibly trying to swap NIC’s around, if we have a spare one.
Workaround formulated is:
1.        Swap the NIC with another to see if the problem moves.
2.       Replace the card and see if we still have the problem.

Please advise...

[RahulG · Accepted solution]

refer https://www-secure.symantec.com/connect/forums/server-status-communication-server-has-not-been-initiated-or-server-status-has-not-been-ret-0

[Maddy777]

Thanks Rahul. Going through the link...

[Nicolai · Accepted solution]

Firewalls in installation ?

If yes the ensure TCP_KEEPALIVE is configured on master, media and client

DOCUMENTATION: COMM_FAILURE as a consequence of reusing a transport that has been inactive across a firewall

http://www.symantec.com/docs/TECH125896

[Marianne]

To know what is happening from NBU point of view, we need all of the following:

All text in Job Details of failing job.

All of the following logs (create folders if they do not exist under netbackup/logs):

On master: bprd  (restart NBU after creating folder)

On media server: bptm and bpbrm

On Oracle client: dbclient.

Copy logs to .txt files (e.g. bprd.txt) and upload as File attachments.

[Maddy777]

Thanks Nicolai & Marianne!

[Marianne]

Have you created log folders yet? 

Impossible to troubleshoot without logs.