Hung backup - Possible reasons

[Bharath_Achar]

Hi All,

 I could see backup is active for morethan 9 hours but it is hung; nothing showing under kilobytes in activity monitor.

09/18/2012 07:25:37 - Info nbjm (pid=26555) starting backup job (jobid=1327163) for client mozart-bac.XXXXX.com, policy Lotus_notes_Mon, schedule Full_Backup
09/18/2012 07:25:38 - estimated 31429293 kbytes needed
09/18/2012 07:25:38 - Info nbjm (pid=26555) started backup job for client mozart-bak.advancemags.com, policy Lotus_notes_Mon, schedule Full_Backup on storage unit DISKPOOL5020_DE6
09/18/2012 07:25:38 - Info bpbrm (pid=28678) mozart-bac.XXXXX.comis the host to backup data from
09/18/2012 07:25:38 - Info bpbrm (pid=28678) reading file list from client
09/18/2012 07:25:38 - started process bpbrm (pid=28678)
09/18/2012 07:25:38 - connecting
09/18/2012 07:25:39 - Info bpbrm (pid=28678) starting bpbkar on client
09/18/2012 07:25:39 - connected; connect time: 0:00:00
09/18/2012 07:25:42 - Info bpbkar (pid=5528) Backup started
09/18/2012 07:25:42 - Info bpbrm (pid=28678) bptm pid: 28938
09/18/2012 07:25:42 - Info bptm (pid=28938) start
09/18/2012 07:25:43 - Info bptm (pid=28938) using 262144 data buffer size
09/18/2012 07:25:43 - Info bptm (pid=28938) using 30 data buffers
09/18/2012 07:25:44 - Info bptm (pid=28938) start backup
09/18/2012 07:25:52 - Info bptm (pid=28938) backup child process is pid 29447
09/18/2012 07:25:52 - begin writing

Please find the attachment for more information .

[revarooo]

Does a standard backup to this client also hang or is it just Lotus backups? Try it and let us know.

[Bharath_Achar]

Standard backup completed successfully.

The issue is with lotus notes backup.

Thanks and regards,

Bharath

[Marianne]

Enable bpbkar log on client.

Check bpbkar log to see what is happening when agent is supposed to generate data stream.

[Bharath_Achar]

Sure,

I would get back to you on this.

Regards,

Bharath

[Bharath_Achar]

Hi Marianne,

Please find the bpbkar logs with verbosity level 5 in the attachment and let us know your findings ASAP.

Regards,

Bharath

[Marianne]

Level 5 logs cannot be worked through ASAP... I don't have any tools  - just sift through manually....

Please tell us more about this client:
Has backups worked before?
If so, when did it stop working?
Anything changed on this client lately? (e.g. any OS or application updates installed without reboot?)

I see multiple backup jobs starting, some before previous one has completed/failed.

1:51:41.723 AM: [4116.4956] ... -class Lotus_notes_Mon -sched Full_Backup

1:58:42.931 AM: [5328.4328] ... -class FS_Windows_Mon_13 -sched Differential_Incremental 

2:17:17.789 AM: [1744.5904] ... -class Lotus_notes_Mon -sched Full_Backup 

4:29:24.610 AM: [4960.4828].... -class Lotus_notes_Mon -sched Cummulative_Incremental

11:59:33.154 AM: [4004.5708] ... -class Lotus_notes_Mon -sched Full_Backup 

Some of these jobs failed, some just kept sending 'keepalive' without any data transfered.

Any idea why these many jobs were kicked off?

Suggestion:

Check Task Manager on Notes client - look for bpbkar processes and kill all of them. 

Stop NBU services on this client and check in cmd window with 'netstat -a' that all NBU connections have terminated (nothing in CLOSE_WAIT state).

Start NBU services again and try another backup.

Maybe a good idea to reboot client instead of NBU restart?

[Bharath_Achar]

Hi,

Has backups worked before?

Yes, we have few cumulative incremental backups of lotus notes databse which are backed up succesfully.

If so, when did it stop working?

From 17/09/2012 full backups are in hung state.

We killed the hung process and jobs and triggered a manual full back but again it became hung.

Some of these jobs failed, some just kept sending 'keepalive' without any data transfered.

Any idea why these many jobs were kicked off?

We have configuresboth filesytem and lotus notes backup of client.

filesytem backup completed succesfully.

few lotus notes backup which are failed with status150 : are killed by us.

Anything changed on this client lately? (e.g. any OS or application updates installed without reboot?)

No.

check Task Manager on Notes client - look for bpbkar processes and kill all of them. 

Stop NBU services on this client and check in cmd window with 'netstat -a' that all NBU connections have terminated (nothing in CLOSE_WAIT state).

Start NBU services again and try another backup.

Tried it . It didnt work.

Maybe a good idea to reboot client instead of NBU restart?

Only this option is left.

As this is production server. we need to plan for rebooting this server.