Oracle backups failure and archive log backups failed with error 6 after writing some data stream.

[Shas]

Good Morning All,

Stuck in an issue with Oracle DB failure. DB Size 1 TB.

Versions:

Master Server (Windows 2003 R2 :(  Netbackup 6.5.6 behaving as a Media server also .

Client  Server (solaris8) : Netbackup client 6.0 mp4 agent installed . Previously backups were working properly.Bptestbpcd output: See attachment,Please.

I am chasing the server team for get bphdb and dbclient logs.

Last night’s datafiles backup failed with the following messages :

input datafile fno=00070 name=/export/home16/app/oradata/DBG1/DBG1_data110.dbf
channel ch00: specifying datafile(s) in backupset
channel ch00: starting incremental level 0 datafile backupset
channel ch00: backup set complete, elapsed time: 02:26:57
released channel: ch00
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================

   VxBSASendData: Failed with error:

ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19502: write error on file "DBG1_bk_u6upnhqml_s35038_p1_t863562453", blockno 143383553 (blocksize=512)
   Failed to process backup file <DBG1_bk_u6upnhqml_s35038_p1_t863562453>
ORA-19511: Error received from media manager layer, error text:
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
RMAN-03009: failure of backup command on ch00 channel at 11/14/2014 00:17:30
ORA-19511: Error received from media manager layer, error text:

[Shas]

Attaching logs 

[Michael_G_Ander]

Please provide the logs in txt format and the job details

What version of Oracle is it ? ask as your Netbackup version is very old and don't support the newer Oracle versions

[Marianne]

We need to see what happened on NBU side. 

As a start, provide all text in Details tab of failed job.

I agree - you really need to upgrade to supported versions of OS, NBU (and probably Oracle as well).

*** EDIT ***

When you post the text in Details tab, just copy the logs for that specific failure to .txt file. Not entire folder. 
We need to see the one set of logs that corresponds with the error.

For example - we see this error in 1st dbclient log:

10:53:48.331 [27772] <16> serverResponse: ERR - server exited with status 199: operation not allowed during this time period

No idea if this is relevant to this issue or not...

Another error in dbclient log from a different day:

18:00:10.764 [12522] <16> serverResponse: ERR - server exited with status 196: client backup was not attempted because backup window closed

Backup window for Default-Application schedule should be open 24x7.

[jim_dalton]

Did it used to work? Probably did as you are asking for incremental, so what else has changed? Where are you writing to? Is the target healthy? The error is from Oracle not NB so thats really where you need to head.

Error suggests disk error / fault or full.

Jim

[Shas]

Yes it was working ...

I will update the with the new logs as suggested by Marianne and Michael :)

Today I did not get time to look after this failure as the other master server had an issue with dbm deadlock :(

[Shas]

Attaced the logs of recent achive Job failure. 

Note :I renamed the master , media and client name .

[Marianne · Accepted solution]

Strange that DBext_log shows success? But there should be more than one dbext log.

We need to see dbclient log as well.

Archive log backup succeeded, but failed on Control File and SPFILE autobackup.
This error looks like something is wrong on Oracle side?

INF - RMAN-03009: failure of Control File and SPFILE Autobackup command on ch00 channel at 11/19/2014 08:31:08
INF - ORA-00230: operation disallowed: snapshot control file enqueue unavailable

See:

https://docs.oracle.com/cd/B19306_01/backup.102/b14191/rcmtroub.htm#i1007443 

http://cgrabowy.wordpress.com/2014/04/18/58/ 

http://www.symantec.com/docs/TECH18161

[Shas]

http://www.symantec.com/business/support/index?page=content&id=TECH18161   This work for me .

DBA team has identified a database session from November 10^th that was causing the error.

Thanks Marianne , Michael and Jim for your valuable feedback. :)

[Marianne]

Great stuff! 

Always awesome to see issues resolved!