cancel
Showing results for 
Search instead for 
Did you mean: 

Backup Job it takes more than 70 hours

robertoaxity
Moderator
Moderator
   VIP   

Hello,

I have a filesytem backup policy (1.5tb), but this takes more than 70 hours, since it writes on average at 950-3000kb / sec, which affects the sap service, since it leaves the space taken from the filesystem and it fills up.

Master Server:

8.1.1

RHEL 6.7

Media Server (appliance 5240, the one who starts the backup)

8.1.1

RHEL 7.3

Client 

9.0

Suse Enterprise 15 SP 2

both the media server and the client have their 10gb network card with mtu 9000, and the trunk is a port channel (10gb)

Therefore, I no longer know where else to analyze or look for a solution to be able to increase the speed of backup for this client, since generally I have to cancel the job due to the time it takes and that means not having customer support

Can you help me? please.

Regards

6 REPLIES 6

StoneRam-Simon
Level 6
Partner    VIP    Accredited Certified

Is the 9.0 a typo??  Client should not be a higher version than the master or media server....

Is this a filesystem backup?  or is it application data (SAP) ?

What settings  / options have you set in the policy ?  

multiple streams?

accelerator?

What is in your filelist / includes ?

If you monitor the system during the backup, does it have high CPU usage / load?   Is the processor stuck in IO_WAIT a lot?

Are you low on memory?  is the system "swapping"?

If you look at the network stats during the backup are you getting a lot of dropped packets?

What is the network connectivity to the Client?  Are there firewalls / routers between the client and the media server?

How much "free space" is in the filesystem?

Are you using any Anti Virus software, if so is it configured to ignore NBU processes? 

Without any backups running on the client what does the network / cpu / memory /  disk performance look like (is the system already "maxed-out" on any of them?

 

Nicolai
Moderator
Moderator
Partner    VIP   

First thing to do is running a iperf between client and media server to network bandwidth. Software can't make hardware run faster.

You will likely need to install the iperf packages, but should be in the distros so easy to install

also see examples here :

https://www.cyberithub.com/iperf-commands-how-to-use-iperf-in-linux/

Run the bpbkar by hand for the file system area : 

/usr/openv/netbackup/bin/bpbkar  -nocont -nofileinfo -nokeepalives /var  > /dev/null 2> /tmp/file.out

Used /var here as example

Make sure you have created the bpbkar debug directory in  VERITAS\NetBackup\logs before starting. The command will return to command prompt immediately, but the process will be visible in task manager, and the debug log will grow in size as well.

if bpbkar run by hand takes the same amount of time as a real backup, you know the problem is on the client and know where to chase the next bottleneck.

robertoaxity
Moderator
Moderator
   VIP   

Is the 9.0 a typo?? Client should not be a higher version than the master or media server....
The client SO is not It does not allow less than 8.2 if the 9 was installed

Is this a filesystem backup? or is it application data (SAP) ?
Filesystem backup

What settings / options have you set in the policy ?

multiple streams?

uncheck

accelerator?
uncheck

What is in your filelist / includes ?
/backup/data

If you monitor the system during the backup, does it have high CPU usage / load? Is the processor stuck in IO_WAIT a lot?
not high cpu load

Are you low on memory? is the system "swapping"?
No, 2TB in total memory

If you look at the network stats during the backup are you getting a lot of dropped packets?
I uploaded a photo with the graph of the last 2 days, the peak was 149mb being a 10gb interface

What is the network connectivity to the Client? Are there firewalls / routers between the client and the media server?
Firewall, but with excludes

How much "free space" is in the filesystem?
now, 1.2TB

Are you using any Anti Virus software, if so is it configured to ignore NBU processes?
No

Without any backups running on the client what does the network / cpu / memory / disk performance look like (is the system already "maxed-out" on any of them?
i dont know

robertoaxity
Moderator
Moderator
   VIP   

Hello,

Run the bpbkar by hand for the file system area :

/usr/openv/netbackup/bin/bpbkar -nocont -nofileinfo -nokeepalives /var > /dev/null 2> /tmp/file.out

This is on master server, media server (appliance) or client?

 

Used /var here as example

Make sure you have created the bpbkar debug directory in VERITAS\NetBackup\logs before starting. The command will return to command prompt immediately, but the process will be visible in task manager, and the debug log will grow in size as well.

if bpbkar run by hand takes the same amount of time as a real backup, you know the problem is on the client and know where to chase the next bottleneck.

 

Nicolai
Moderator
Moderator
Partner    VIP   

Hi @robertoaxity 

On the client , you specify the problematic file system and observe how fast bpbkar can read the data off the disk without transferring the data to the media server (read data is send to /dev/null).

If bpbkar can read the data much faster the backup time, problem is not with the file system itself

If bpbkar read time is the same as backup time, the file system on the client is the limiting factor.

Since you have a firewall in-between master and media server, you really want to perform a iperf test to see how good or bad the firewall perform.

 

StoneRam-Simon
Level 6
Partner    VIP    Accredited Certified

Just to follow on from this, I was querying the client being 9.0 when you were stating the Master and media were both  8.1.1

I would like to understand this:-

 which affects the sap service, since it leaves the space taken from the filesystem and it fills up.

Where are you seeing space issues?  (client or media server?) 

If this is the client, is the space limitation in  a different filesytem to where the data you are backing up resides?  From what you have provided I'm envisaging a 2.7 TB filesystem of which 1.5TB has been used, is this correct? if not please provide the output of the command "df -k /backup/data" or possibly just a "df -k" so we can also determine if you have space issues in other paths that NBU may be using....