Catalog backup is failing

[Zahid_Haseeb]

Environment

Single Machine Master/Media Netbackup Server(Enterprise) = 7.1

OS = Windows2008

Problem

Backup is failing

Please Note: I faced the same problem and resolved. Below is the discussion for reference but that was also not helped me.

https://www-secure.symantec.com/connect/forums/info-bpbkar32pid3124-done-status-44-network-write-failed-network-write-failed44

Activity Monitor

2/13/2013 3:24:56 PM - Info nbjm(pid=2240) starting backup job (jobid=165773) for client veritas-nbu, policy Catalog_Backup, schedule Full   
2/13/2013 3:24:56 PM - Info nbjm(pid=2240) requesting CATALOG_BACKUP_RESOURCE resources from RB for backup job (jobid=165773, request id:{ADB17D6B-49A9-4997-A6A0-090C3159CA90})   
2/13/2013 3:24:56 PM - requesting resource veritas-nbu.NBU_CATALOG.MAXJOBS
2/13/2013 3:24:56 PM - Info nbrb(pid=2980) Limit has been reached for the logical resource veritas-nbu.NBU_CATALOG.MAXJOBS     
2/13/2013 3:33:51 PM - granted resource veritas-nbu.NBU_CATALOG.MAXJOBS
2/13/2013 3:33:51 PM - estimated 86644432 Kbytes needed
2/13/2013 3:33:51 PM - begin Parent Job
2/13/2013 3:33:51 PM - begin Catalog Backup, Start Notify Script
2/13/2013 3:33:51 PM - Info RUNCMD(pid=4240) started             
2/13/2013 3:33:51 PM - Info RUNCMD(pid=4240) exiting with status: 0          
Status 0
2/13/2013 3:33:51 PM - end Catalog Backup, Start Notify Script; elapsed time: 00:00:00
2/13/2013 3:33:51 PM - begin Catalog Backup, DBM Query
Status 44
2/13/2013 5:24:57 PM - end Catalog Backup, DBM Query; elapsed time: 01:51:06
2/13/2013 5:24:57 PM - begin Catalog Backup, Stop On Error
Status 0
2/13/2013 5:24:57 PM - end Catalog Backup, Stop On Error; elapsed time: 00:00:00
2/13/2013 5:24:57 PM - begin Catalog Backup, End Notify Script
2/13/2013 5:24:57 PM - Info RUNCMD(pid=5048) started             
2/13/2013 5:24:57 PM - Info RUNCMD(pid=5048) exiting with status: 0          
Status 0
2/13/2013 5:24:57 PM - end Catalog Backup, End Notify Script; elapsed time: 00:00:00
Status 44
2/13/2013 5:24:57 PM - end Parent Job; elapsed time: 01:51:06
network write failed(44)

[Mark_Solutions]

Hi - sorry previous issue did not get resolved

As another thought on this one can you tell us where the DR file is saved to?

If it uses a remote path then check the path exists and that the credentials are good.

Try setting it to a local path if it is not already (<install path>\veritas\CAT_DR perhaps, but do create the directory first)

The network write error may just be the failure in creating the DR file

Hope this helps

[Zahid_Haseeb]

The DIrectory is created and the drive also have alot of space

[Mark_Solutions]

Could you show me the output of this please:

bppllist catalogpolicyname -U

[Zahid_Haseeb]

C:\Program Files\Veritas\NetBackup\bin\admincmd>bppllist.exe catalog_Manual -U
------------------------------------------------------------

Policy Name:       catalog_XXXXXX

  Policy Type:         NBU-Catalog
  Active:              yes
  Effective date:      09/08/2009 16:05:37
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          no
  Policy Priority:     0
  Max Jobs/Policy:     1
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           veritas-XXXX-hcart-robot-tld-6
  Volume Pool:         CatalogBackup
  Server Group:        *ANY*
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no
  Application Discovery:      no
  Discovery Lifetime:      0 seconds

  Granular Restore Info:  no
  Ignore Client Direct:  no
  HW/OS/Client:  PC            Windows2008   veritas-XXXXXX

  Include:  CATALOG_DRIVEN_BACKUP

  Schedule:          Full
    Type:            Full Backup
    Frequency:       every 1 day
    Maximum MPX:     1
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 24 (infinity)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:

Catalog Disaster Recovery Configuration:
  Email Address:   xxxxxx@xxxxxx.com
  Disk Path:       L:\Pri_Catalog-DR
  User Name:       (none specified)
  Pass Word:       (none specified)
  Critical policy: (none specified)


C:\Program Files\Veritas\NetBackup\bin\admincmd>
 

[Mark_Solutions]

OK - Thanks

Do you have the bpdbm, bptm and admin logs from the Master?

[Marianne]

I agree with Mark - you need logs to troubleshoot.

Zahid - I have just gone through your other post - how was the problem resolved?
You never came back to tell us?
Your last comment says:

Mark and Marianne let me do this and share the result

Post got locked after 6 months of no activity.

[Zahid_Haseeb]

Yap you are right. but unfortunately I dont remember yet what was the last words with Symantec that either he was also able to catch the problem ot not.

Let me share the logs for this ASAP.

[Zahid_Haseeb]

Time of Catalog Backup Failure

2/18/2013    6:14:39 PM    Started
2/18/2013    5:45:09 PM    Failed

two files attached and the admin file did not have generated any log at that time. I attached both files and password are PM to Marianne and Mark.

[Mark_Solutions]

Cant see anything in those - do you have the admin log?

[Zahid_Haseeb]

at that specific time there is no entry under admin log

[Mark_Solutions]

Maybe the bpbrm log then?

Can you also confirm that the DR files do get created at L:\Pri_Catalog-DR

Could you run an all log entries report from the admin console from 5:45 to 6:15 as well please

[Zahid_Haseeb]

- bpbrm log attached.

- Yes creating.

- attached.

[Mark_Solutions · Accepted solution]

OK - looking a everything i have seen, and re-reading your original thread i still feel you have the same issues.

A combination of possibly poor network and performance of the Master Server and maybe disk space again - though i think your error previosuly was peaged memory causing the issue,

So a few things you need to check and do - please go through them ALL:

1. Did you ever implement all of the TCPIP tuning parameters i suggested in the original post?

If not please do them and reboot your server so that they take effect.

2. I would also tune the memory on your master server as follows:

HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\

DWORD - PoolUsageMaximum  - Decimal value of 40

DWORD - PagedPoolSize Hex value of FFFFFFFF (this is 8 x F)

These again need a reboot to take effect so do them at the same time as the TCPIP ones.

3. Check disk space on the installation drive of the Master Server - keep more than 10% free.

4. check Anti Virus and Access Protection software is excluding all NetBackup directory and processes.

Do these, reboot, try the catalog backup again and come back to us.

[Marianne]

Only getting to this post now... Sorry - busy day!

I agree with Mark - please go through recommendations in previous post.
The one log we forgot to ask is master's bpbkar log...

If you have gone through the recommendations and the problem persists, please post a fresh set of logs, including bpbkar log.

*** EDIT ***

I was particularly interested in bpbrm log file, but it contains no record of 'catalog_...' policy.
We need a full set of logs collected after a failed catalog backup attempt.

[Zahid_Haseeb]

Thanks both for such a valuable interest in the problem. Let me do it and share the result.

@Mark I will do each thing one by one so that(if problem resolved) I may come to know that in which area there was problem.

[Zahid_Haseeb]

Problem Resolved:

I planned to do each step suggested above and verify Catalog Backup....So that I can come to know where was the problem. As per the first point (A combination of possibly poor network and performance of the Master Server and maybe disk space again)

I did free around 50GB of space (earlier its around 5 to 10GB free) and triggered the backup which got successful. Marking the post as solution + Thumb :)

[Mark_Solutions]

Great stuff! Glad it is resolved!

Was the previous issue resolved in a similar way do you remember as that thread is still unsolved

[Zahid_Haseeb]

emm Mark ...I increased the Browse time out to 6000 from 300 and that time the problem is resolved. At that time I was also astonished that through that way that should not be resolved but it happened :)