firewall, card speed port

Foxtrot_Lima · ‎10-07-2009

Backing up Linux and hpux clients on a separated network with a media server. Master and media server is located on different networks with a firewall between.

Master server: NetBackup 6.5.1 on SLES 10 x86_64 SP2
Media server: NetBackup 6.5.1 SLES 9 x86_64 SP3

Backups randomly fails, some days all backups are completing without any errors, other days some backups fails on first try and on the next retry finishes OK. Some completely fails with Status 40 on all retries and when restarting the backups next morning they finishes.

From Detailed Status Tab:
2009-okt-07 00:00:02 - requesting resource netbackup-me5-STKSL8500-LTO3
2009-okt-07 00:00:02 - requesting resource netbackup-ma1.NBU_CLIENT.MAXJOBS.ux166002
2009-okt-07 00:00:02 - requesting resource netbackup-ma1.NBU_POLICY.MAXJOBS.ux_fil_backup_um
2009-okt-07 00:00:03 - awaiting resource netbackup-me5-STKSL8500-LTO3. Maximum job count has been reached for the storage unit.
2009-okt-07 00:00:05 - awaiting resource netbackup-me5-STKSL8500-LTO3. No drives are available.
2009-okt-07 00:00:36 - awaiting resource netbackup-me5-STKSL8500-LTO3. Maximum job count has been reached for the storage unit.
2009-okt-07 00:00:37 - awaiting resource netbackup-me5-STKSL8500-LTO3. No drives are available.
2009-okt-07 00:00:48 - awaiting resource netbackup-me5-STKSL8500-LTO3. Maximum job count has been reached for the storage unit.
2009-okt-07 00:00:50 - awaiting resource netbackup-me5-STKSL8500-LTO3. No drives are available.
2009-okt-07 00:00:54 - awaiting resource netbackup-me5-STKSL8500-LTO3. Maximum job count has been reached for the storage unit.
2009-okt-07 00:00:59 - awaiting resource netbackup-me5-STKSL8500-LTO3. No drives are available.
2009-okt-07 00:07:28 - Waiting for scan drive stop Drive-0.1.1.14-SL8500, Media server: netbackup-me5
2009-okt-07 00:07:28 - granted resource netbackup-ma1.NBU_CLIENT.MAXJOBS.ux166002
2009-okt-07 00:07:28 - granted resource netbackup-ma1.NBU_POLICY.MAXJOBS.ux_fil_backup_um
2009-okt-07 00:07:28 - granted resource A00769
2009-okt-07 00:07:28 - granted resource Drive-0.1.1.14-SL8500
2009-okt-07 00:07:28 - granted resource netbackup-me5-STKSL8500-LTO3
2009-okt-07 00:07:28 - estimated 1870315 kbytes needed
2009-okt-07 00:07:32 - connecting
2009-okt-07 00:07:32 - connected; connect time: 0:00:00
2009-okt-07 00:07:32 - begin writing
2009-okt-07 00:15:08 - Error bpbrm (pid=18031) could not write FILE ADDED message to stderr
2009-okt-07 00:15:30 - Error bpbrm (pid=18031) could not write FILE ADDED message to stderr
2009-okt-07 00:15:54 - Error bpbrm (pid=18031) could not write FILE ADDED message to stderr
2009-okt-07 00:16:19 - Error bpbrm (pid=18031) could not write FILE ADDED message to stderr
2009-okt-07 00:17:48 - Error bpbrm (pid=18031) could not write FILE ADDED message to stderr
network connection broken (40)

Here's a list of the clients and their backups and retries:

Status         Date/Time                                                                 Client

40           Wed Oct 07 02:00:23 CEST 2009                         ux166002
196             Tue Oct 06 06:00:03 CEST 2009                        ux166002
0                  Mon Oct 05 01:32:48 CEST 2009                         ux166002

0                           Wed Oct 07 00:07:17 CEST 2009                         ux225612
196                      Tue Oct 06 06:00:03 CEST 2009                        ux225612
0                           Mon Oct 05 01:24:01 CEST 2009                         ux225612

0                           Wed Oct 07 02:26:32 CEST 2009                         ux164006_backup
40 Wed Oct 07 00:20:06 CEST 2009                         ux164006_backup
196                      Tue Oct 06 06:00:03 CEST 2009                           ux164006_backup
0                           Mon Oct 05 01:18:31 CEST 2009                          ux164006_backup

40                      Wed Oct 07 00:20:06 CEST 2009                          ux225611
196                    Tue Oct 06 06:00:03 CEST 2009                            ux225611
0                         Mon Oct 05 01:00:42 CEST 2009                           ux225611

0                         Wed Oct 07 02:33:08 CEST 2009                          ux164002
174                    Wed Oct 07 00:30:12 CEST 2009                          ux164002
196                    Tue Oct 06 06:00:03 CEST 2009                          ux164002
0           Mon Oct 05 01:49:11 CEST 2009                          ux164002

40                      Wed Oct 07 02:00:23 CEST 2009                          ux164003
196                    Tue Oct 06 06:00:03 CEST 2009                          ux164003
0               Mon Oct 05 01:41:22 CEST 2009                          ux164003

0           Wed Oct 07 04:59:43 CEST 2009                          ux158702
13                       Wed Oct 07 00:40:15 CEST 2009                          ux158702
40                       Wed Oct 07 00:20:06 CEST 2009                          ux158702
196                    Tue Oct 06 06:00:03 CEST 2009                          ux158702
0           Mon Oct 05 01:23:30 CEST 2009                          ux158702

0                        Wed Oct 07 04:58:48 CEST 2009                          ux166001
13                      Wed Oct 07 00:40:15 CEST 2009                          ux166001
40                      Wed Oct 07 00:20:06 CEST 2009                          ux166001
196                  Tue Oct 06 06:00:03 CEST 2009                          ux166001
0           Mon Oct 05 01:23:06 CEST 2009                          ux166001

Nicolai · ‎10-07-2009

Try to redirect the backup traffic around the firewall to see if it changes anything (as a temporary fix you could connect a NIC from the media server to the other side of the firewall).

Backup traffic and firewall don't match very well. Way to often the firewall can't handle the traffic and you get all sort of problems.

zippy · ‎10-07-2009

firewall, card speed port speed on switch???

network connection broken (40)

network problem

VOX

Backups on a separated network randomly fails with Status 40