Duplicate job copy data from original location instead of source job location

We use BackupExec 22 Rev 1193.1009. BackupExec server (hardware server ProLiant BL460c Gen9) is connected to SAN and LTO8 library. Speed of Ethernet connection is 10Gb/s

At first backups are saved to SAN deduplication storage. Then - duplicate job runs. We intended that duplicate job would run from SAN backup, not from original location. The speed between SAN and tape is much faster than speed between original data location and tape.

We intended to make a system where 8 backups will run simultaneously to SAN. They are slow (because they copy data from original location) but we do not care - because we can run many backups simultaneously. THEN - when first stage backups are finished - they duplicate to tape. They run only in 3 streams (because library only has 3 drives), but we do not care - because the speed from SAN to LTO8 tape is fast.

But it seems, that duplicate backups run from original location, instead of SAN, where first stage of backups are saved. This is VERY VERY bad to us, because we need to backup about a hundred servers and some of them are large. If duplicate backup - copy data from original location, instead from SAN - we loose speed in about 6-8 times I think.

I opened topic about a year ago about this issue and I was informed that duplicate backups run from primary backup, not from original data location.

Does duplicate JOB copy data from source VM or fro... - VOX (veritas.com)

I tried to enable\disable "Direct copy to tape" checkbox, BacupExec still copy data in duplicate backup from original location.

Also I tried to save first stage of backup to regular disk storage (without deduplication). This is disk storage on BackupExec server itself - so speed must be very high. But BacupExec still copy data in duplicate backup from original location, not from disk storage.

I also checked the bandwidth of optical Ethernet which is connected to BackupExec server and it is utilized no more than on 30%

I tried to look through all options of backup, but didn't found anything related to problem - from where duplicate backups run: from original location or from first stage backup.

Is it possible to configure duplicate backups to run from first stage backups location? How to do it? This will increase overall speed of my backup system in 6-8 times.

backupexec

speed

Windows Server 2019

17 Replies

shaptala
Level 4
2 years ago
I have tested on these drivers:
IBMTape.w16_7008-x64_WHQL_Cert
IBMTape.x64_w16_6266
IBMTape.w22_7008-x64_WHQL
The version of BackupExec is 22, with hotfix
Yeah, really very strange problem and is very critical to us.
shaptala
Level 4
2 years ago
I found out that if I delete IBM TS4300 Tape Library IBM driver and leave old (2006 year) Microsoft driver IBM ULT3580-HH8 SCSI Sequential Device - there are no issues with speed: both on first write and all subsequent appends.
I will try 21 version of Veritas. May be there will be no problem there.
Or there is something other I can try?
There are some properties on the driver of IBM drives in Device Manager. But they are grayed out. I can not change them.
- Gurvinder
  Moderator
  2 years ago
  Seeing this post, I had tested this in our in-house setup and did not run into an append slow speed problem and hence it seems to me as something outside of BE but even then please can you share some screenshot of driver version with which you see this problem.
RogRubin
Moderator
3 years ago
Have you opened a job and then compared the different sections?
Is there one particular section which is taking longer?
- shaptala
  Level 4
  3 years ago
  Job consist of 1 file, which is located on local hard drive of the server. I would rather think in direction of fiber HBA may be?
shaptala
Level 4
3 years ago
We have time till Monday. After it - I will destroy this server and we will never know what is the cause of this problem.
PJ_Backup
Moderator
3 years ago
Is the initial write to the new scratch tape a full backup, and subsequent writes incremental or differential backups? If so that makes more sense as these jobs will always have a slower rate than the full backups (due to the overhead of the system determining which files are required to be backed up)

If 6GB is the size of the initial backup, what size are the subsequent appended backups? The smaller the backup size, the less reliable the speed statistics will be - eg 6GB is quite small so if subsequent amounts appended to the tape are even smaller the figure will be more greatly affected by the mount and dismount times as well as searching the tape to locate the correct location to start the next append.

The first full backup also has the advantage that this last process (winding the tape forward to get to the correct place to start the backup) is quicker as the backup just has to start at the beginning.
- shaptala
  Level 4
  3 years ago
  BTW - red squares are appends, green squares - are first writes to scratch tape. Opened a case in your support - now at the stage of uploading logs.
- shaptala
  Level 4
  3 years ago
  May be it's somehow related to new 22 version? Since we are using it.
- shaptala
  Level 4
  3 years ago
  No incremental or differential backups are even configured. Only Full Backup.
  The size of BOTH initial and subsequent backups are THE SAME. I attach the screenshot - please look at the size of backup - it is always 5.03 GB. Initially I made bigger backups (about 160 GB), but later start making smaller backups, because I make many speed test. Waiting each time is pain.
  Mount and dismount time, searching the tape to locate the correct location (I believe) - do not contribute to the value "Job rate" column.
  I do not care about mounting tape speed and searching speed. I only look to job rate and on appends it's slower.
  Screenshot attached.
shaptala
Level 4
3 years ago
This is file to tape backup. File size is about 6GB. This is iso file. The file is located LOCALLY on backup server. Ethernet is not involved, only Fiber. Because LTO8 is connected via Fiber.
RogRubin Of what do you think? This behavior looks very strangle.
RogRubin
Moderator
3 years ago
Regarding this point:
"While trying to resolve this further I discovered very strange thing: if I write file to new, scratch LTO8 media - the speed is very good. But subsequent appends are slow."
Is that still a duplicate job or do you see the same behavior if you test this with just a normal backup job directly to tape?
shaptala
Level 4
3 years ago
While trying to resolve this further I discovered very strange thing: if I write file to new, scratch LTO8 media - the speed is very good. But subsequent appends are slow. WTF???????
shaptala
Level 4
3 years ago
It seams that the problem is not related to location from which backup is run. I observe the same low speed even if backing up from local disk of BackupExec server. Even more: backup to HP LTO5 drive runs 3-4 times faster, then backup to IBM LTO8
My Fiber controller is QLogic with driver from QLogic 17.08.2015
Library is: IBM TS 4300. Firmware: 1.5.0.1-B00
Drives: IBM LTO 8 HH FC. Firmware: P381
Drives (as they look in Windows): IBM ULTRIUM 8 HH 3580 TAPE DRIVE
Driver of drives: From IBM, version: 7.0.0.8, date: 11.04.2022 (non-exclusive)
We recently changed backup server from old HP G5 server to ProLiant BL460c Gen9. We didn't move configuration. We installed everything from scratch. But on old server - there where no issues with speed. The speed on old server was about 10 times faster - if I backup file from local drive.
shaptala
Level 4
3 years ago
No ideas? Backup Exec is terrible importance to us.