Netbackup- Status Code 48: the DB backups suddenly failed

[Iwan_Tamimi]

Hi All,

I just enountered strange problem, some of the DB backup suddely got problem, the db backup consist of MS_SQL and oracle backup using perl script. The backup policies are not new, they are running fine for quite long.  The error status code is 48. The funny thing the file system on the same server still running fine. I know this is something to do with the name resolution(?) but I have checked with the nslookup for both client and master/media server, looks ok.

What could be possible problem? Any idea how to resolve it?

(Master server is running on HPUX 11.11 with Netbackup enterprise server 6.5.6)

Thank you very much.

Iwan

[Marianne · Accepted solution]

MS_SQL and oracle backup using perl script.

A bit more info, please...

Are DB clients running backups to disk followed by bpbackup to backup the disk backups to NBU?

Or Agent backups started by external scheduler on the clients?

Please provide all text in Details tab of failed job.

Check at OS level on master and media server that forward and reverse lookup is correct between server(s) and client(s).

Maybe DNS and/or hosts files have changed?

You will need logging to troubleshoot the problem.
If user-initiated backups, bprd log on master server is first to check.
Next log is bpbrm on the media server(s).

PS: Are you aware that NBU 6.5 support ended in Oct last year?

[RamNagalla]

hi 

DO you have hosts file or DNS in place for the name resolutions?

if the if the resolution query too more time it may be possible for getting EC 48?

if you are using DNS its more likely possible.

please add the Client entires in etc/hosts file and check how it works

[Iwan_Tamimi]

Thanks Marianne for the quick response

>Are DB clients running backups to disk followed by bpbackup to backup the disk backups to NBU?Or >Agent backups started by external scheduler on the clients?

To make it easier maybe I concentrate on the MS-SQL agent backup first ignore the oracle. Yes our backup started by external scheduler but on the master server

>Please provide all text in Details tab of failed job.

Jan 8, 2013 4:28:56 PM - Error bprd (pid=20182) Unable to write progress log </C/ProgramÀ Files/Veritas/NetBackup/Logs/user_ops/dbext/logs/5180.0.1357633728> on client bccprj01-bck. Policy=BCCPRJ01_DB_BCCPRJSP_MSDB_HOT Sched=NONE
Jan 8, 2013 4:28:57 PM - Error bprd (pid=20182) CLIENT bccprj01-bck  POLICY BCCPRJ01_DB_BCCPRJSP_MSDB_HOT  SCHED NONE  EXIT STATUS 48 (client hostname could not be found)
 

>Check at OS level on master and media server that forward and reverse lookup is correct between server(s) and client(s).

Checked, looks fine

> Maybe DNS and/or hosts files have changed?

Yes there is a possibility, because the policies that failed are not new , just out of sudden it failed, but I don't know what to checked.

>You will need logging to troubleshoot the problem.
>If user-initiated backups, bprd log on master server is first to check.
>Next log is bpbrm on the media server(s).

I will check on it

> PS: Are you aware that NBU 6.5 support ended in Oct last year?

Hehe yes, that's why I am asking here, we are in the middle of upgrading.

Thanks Marrianne

[RamNagalla]

Jan 8, 2013 4:28:57 PM - Error bprd (pid=20182) CLIENT bccprj01-bck  POLICY BCCPRJ01_DB_BCCPRJSP_MSDB_HOT  SCHED NONE  EXIT STATUS 48 (client hostname could not be found)

bprd is not finding the host name... 

Please provide the output of below commands in master server

bpcltncmd -hn bccprj01-bck

bpclntcmd -Ip <IP of bccprj01-bck>

grep bccprj01 /etc/hosts

nslookup bccprj01-bck

[Iwan_Tamimi]

Hi Nagalla,

ebs1-bck@/> bpclntcmd -hn bccprj01-bck

host bccprj01-bck: bccprj01-bck at  xx.xx.81.199 (0xa9551c7)

aliases:

ebs1-bck@/> bpclntcmd -ip xx.xx.81.199

checkhaddr: host   : bccprj01-bck: bccprj01-bck at xx.xx.81.199 (0xa9551c7)

checkhaddr: aliases:

ebs1-bck@/> grep bccprj01 /etc/hosts

xx.xx.81.199   bccprj01-bck

ebs1-bck@/> nslookup bccprj01-bck

Using /etc/hosts on:  ebs1-bck

looking up FILES

Name:    bccprj01-bck

Address:  xx.xx.81.199

(notes: I mask our network, but it gave a consistence result)

Thank Nagalla.

Iwan 

[RamNagalla]

please provide the log of bprd for the process 20182

if the verbose is not 5, please raise up the verbose to 5 and let the job fail then collect the logs of bprd.

and also, how many hosts entires you have in /etc/hosts file?

if it has more entiries, could you move this client entires to beginning of the /etc/hosts fine and check how it working

[Iwan_Tamimi]

Hi All, Yesterday morning after midnight the problem resolved by itself , I didn't really fix it, I still don't know what wase the caused but here wassome facts: o The day the thing happened there aws a problem in one of our exchange server, I suspect it could be our secondary DNS server for Windows(The Windows DNS, the domain like company.local, our other DNS is in l Linux for serving company.com.sg), this Windows DNS manage by other department. o The problem only affecting the one that use SQL Agent and SAP agent. o During the problem I did nslookup or ping with the name from Master, Media and client to each other and were OK. o For the client the name resolving done on ..\etc\hosts (but I didn't check wheter it is also automatically registed on the Windows DNS) Anyway thank you very much for Marianne and Nagala for the supports. Regards, Iwan I will still monitor the situation if I found something I will post it to this forum

[Iwan_Tamimi]

I found something else, it could be the caused, at the same day I also edit the /etc/hosts I accidently put a note without a remark (#) then at night I did restore back the /etc/hosts.

Regards,

Iwan