06-07-2021 06:58 AM
Hello,
I have a filesytem backup policy (1.5tb), but this takes more than 70 hours, since it writes on average at 950-3000kb / sec, which affects the sap service, since it leaves the space taken from the filesystem and it fills up.
Master Server:
8.1.1
RHEL 6.7
Media Server (appliance 5240, the one who starts the backup)
8.1.1
RHEL 7.3
Client
9.0
Suse Enterprise 15 SP 2
both the media server and the client have their 10gb network card with mtu 9000, and the trunk is a port channel (10gb)
Therefore, I no longer know where else to analyze or look for a solution to be able to increase the speed of backup for this client, since generally I have to cancel the job due to the time it takes and that means not having customer support
Can you help me? please.
Regards
06-07-2021 07:20 AM
Is the 9.0 a typo?? Client should not be a higher version than the master or media server....
Is this a filesystem backup? or is it application data (SAP) ?
What settings / options have you set in the policy ?
multiple streams?
accelerator?
What is in your filelist / includes ?
If you monitor the system during the backup, does it have high CPU usage / load? Is the processor stuck in IO_WAIT a lot?
Are you low on memory? is the system "swapping"?
If you look at the network stats during the backup are you getting a lot of dropped packets?
What is the network connectivity to the Client? Are there firewalls / routers between the client and the media server?
How much "free space" is in the filesystem?
Are you using any Anti Virus software, if so is it configured to ignore NBU processes?
Without any backups running on the client what does the network / cpu / memory / disk performance look like (is the system already "maxed-out" on any of them?
06-07-2021 07:38 AM
First thing to do is running a iperf between client and media server to network bandwidth. Software can't make hardware run faster.
You will likely need to install the iperf packages, but should be in the distros so easy to install
also see examples here :
https://www.cyberithub.com/iperf-commands-how-to-use-iperf-in-linux/
Run the bpbkar by hand for the file system area :
/usr/openv/netbackup/bin/bpbkar -nocont -nofileinfo -nokeepalives /var > /dev/null 2> /tmp/file.out
Used /var here as example
Make sure you have created the bpbkar debug directory in VERITAS\NetBackup\logs before starting. The command will return to command prompt immediately, but the process will be visible in task manager, and the debug log will grow in size as well.
if bpbkar run by hand takes the same amount of time as a real backup, you know the problem is on the client and know where to chase the next bottleneck.
06-07-2021 08:32 AM
Is the 9.0 a typo?? Client should not be a higher version than the master or media server....
The client SO is not It does not allow less than 8.2 if the 9 was installed
Is this a filesystem backup? or is it application data (SAP) ?
Filesystem backup
What settings / options have you set in the policy ?
multiple streams?
uncheck
accelerator?
uncheck
What is in your filelist / includes ?
/backup/data
If you monitor the system during the backup, does it have high CPU usage / load? Is the processor stuck in IO_WAIT a lot?
not high cpu load
Are you low on memory? is the system "swapping"?
No, 2TB in total memory
If you look at the network stats during the backup are you getting a lot of dropped packets?
I uploaded a photo with the graph of the last 2 days, the peak was 149mb being a 10gb interface
What is the network connectivity to the Client? Are there firewalls / routers between the client and the media server?
Firewall, but with excludes
How much "free space" is in the filesystem?
now, 1.2TB
Are you using any Anti Virus software, if so is it configured to ignore NBU processes?
No
Without any backups running on the client what does the network / cpu / memory / disk performance look like (is the system already "maxed-out" on any of them?
i dont know
06-07-2021 08:35 AM
Hello,
Run the bpbkar by hand for the file system area :
/usr/openv/netbackup/bin/bpbkar -nocont -nofileinfo -nokeepalives /var > /dev/null 2> /tmp/file.out
This is on master server, media server (appliance) or client?
Used /var here as example
Make sure you have created the bpbkar debug directory in VERITAS\NetBackup\logs before starting. The command will return to command prompt immediately, but the process will be visible in task manager, and the debug log will grow in size as well.
if bpbkar run by hand takes the same amount of time as a real backup, you know the problem is on the client and know where to chase the next bottleneck.
06-07-2021 11:56 PM - edited 06-07-2021 11:58 PM
On the client , you specify the problematic file system and observe how fast bpbkar can read the data off the disk without transferring the data to the media server (read data is send to /dev/null).
If bpbkar can read the data much faster the backup time, problem is not with the file system itself
If bpbkar read time is the same as backup time, the file system on the client is the limiting factor.
Since you have a firewall in-between master and media server, you really want to perform a iperf test to see how good or bad the firewall perform.
06-08-2021 05:38 AM
Just to follow on from this, I was querying the client being 9.0 when you were stating the Master and media were both 8.1.1
I would like to understand this:-
which affects the sap service, since it leaves the space taken from the filesystem and it fills up.
Where are you seeing space issues? (client or media server?)
If this is the client, is the space limitation in a different filesytem to where the data you are backing up resides? From what you have provided I'm envisaging a 2.7 TB filesystem of which 1.5TB has been used, is this correct? if not please provide the output of the command "df -k /backup/data" or possibly just a "df -k" so we can also determine if you have space issues in other paths that NBU may be using....