RMAN backups failing

[syrus]

Hi,

We are facing a weird issue with RMAN backups. DBA team is complaining that RMAN backups are failing with the error "ORA-19511- Error received from media manager layer". We, the backup team doesnot see any errors in activity monitor.Also we are not able to find any entries in activity monitor for the backupjob (whether it is success or failure or partially sucessfull etc).

Backup window is open for 24*7 and the DBA team triggers backup on friday evening. Usually it takes 4-5 hours to complete. This was working fine till last week and from this week onwards backups are not running.

Any thoughts ?

[RamNagalla]

does backup of file system getting successfull?

so how the backup are getting triggred ?

are they netbackup Scheduled... or from RMAN /3 rd party scheduler?

if you are not seeing the jobs in the netbackup Activity moniter, that indicates thet the master server (bprd) is not recieving the backup request...

test the communication between the client and Master server

bpclntcmd -pn  -from client

and also enable the dbclient log and test the backups again and attach the log to this post.

[Marianne]

We, the backup team doesnot see any errors in activity monitor.Also we are not able to find any entries in activity monitor for the backupjob (whether it is success or failure or partially sucessfull etc).

As per Nagalla's post - we need to see bprd log on the master server to see if backup request was received from the client.
If bprd log folder does not exist on the master, you need to create it and restart NBU to enable the log.

Along with master's bprd log, we need to see client's dbclient log to see client's connection attempt to master server.
Ensure dbclient log folder exists on the client.
If the client is Unix/Linux, the dbclient log folder needs 777 permission.

Before next backup attempt, ensure that log folders exist and double-check SERVER entries on the Client. If SERVER entries have been changed and a media server is accidently listed as 1st SERVER entry, the Oracle client will attempt to connect to bprd on the server that is listed 1st. If this is not the Master server, the connection attempt will fail.

Also check client's script for NB_ORA_SERVER entry - ensure that entry matches Master server name.

Please copy logs to .txt files (e.g. bprd.txt) and upload as File attachments after next failure.

[Nicolai]

Attach output from RMAN as well as a text file.

[syrus · Accepted solution]

Thank you all.

Issue has been resolved. IP address of the DB server was changed, because of which name resolutions were not happening properly.

Manually updated the host file with the new IP and backups are working fine now.

Once again thanks a lot for the valuable suggestions.