08-24-2022 01:54 AM
We use BackupExec 22 Rev 1193.1009. BackupExec server (hardware server ProLiant BL460c Gen9) is connected to SAN and LTO8 library. Speed of Ethernet connection is 10Gb/s
At first backups are saved to SAN deduplication storage. Then - duplicate job runs. We intended that duplicate job would run from SAN backup, not from original location. The speed between SAN and tape is much faster than speed between original data location and tape.
We intended to make a system where 8 backups will run simultaneously to SAN. They are slow (because they copy data from original location) but we do not care - because we can run many backups simultaneously. THEN - when first stage backups are finished - they duplicate to tape. They run only in 3 streams (because library only has 3 drives), but we do not care - because the speed from SAN to LTO8 tape is fast.
But it seems, that duplicate backups run from original location, instead of SAN, where first stage of backups are saved. This is VERY VERY bad to us, because we need to backup about a hundred servers and some of them are large. If duplicate backup - copy data from original location, instead from SAN - we loose speed in about 6-8 times I think.
I opened topic about a year ago about this issue and I was informed that duplicate backups run from primary backup, not from original data location.
Does duplicate JOB copy data from source VM or fro... - VOX (veritas.com)
I tried to enable\disable "Direct copy to tape" checkbox, BacupExec still copy data in duplicate backup from original location.
Also I tried to save first stage of backup to regular disk storage (without deduplication). This is disk storage on BackupExec server itself - so speed must be very high. But BacupExec still copy data in duplicate backup from original location, not from disk storage.
I also checked the bandwidth of optical Ethernet which is connected to BackupExec server and it is utilized no more than on 30%
I tried to look through all options of backup, but didn't found anything related to problem - from where duplicate backups run: from original location or from first stage backup.
Is it possible to configure duplicate backups to run from first stage backups location? How to do it? This will increase overall speed of my backup system in 6-8 times.
08-24-2022 04:06 AM - edited 08-24-2022 04:07 AM
I saw the earlier post but please can you provide some screenshots which explain this line
"But it seems, that duplicate backups run from original location, instead of SAN, where first stage of backups are saved"
If the backup is kept on dedupe storage on SAN ( iscsi 10 Gbps ) then , they would be read out from BE server from that storage and then dupicated to Tape.
Also note - Backup will write less data and hence backup will be faster ( since dedupe is taking place ) but when you duplicate, BE has to read the entire set data ( (rehydrate) + fetch data from SAN) and write to Tape.
08-24-2022 06:47 AM
SAN is connected via Fibre Channel 8Gb/s. Is it a problem? Is this configuration able to proceed duplication to tape from SAN, not from original location?
Regarding deduplication - I tried to run 2nd stage duplicate job - both from deduplication SAN and normal SAN - the speed is the same. So deduplication might be not a problem.
08-25-2022 07:27 AM
No ideas? Backup Exec is terrible importance to us.
08-26-2022 01:47 PM - edited 08-26-2022 02:13 PM
It seams that the problem is not related to location from which backup is run. I observe the same low speed even if backing up from local disk of BackupExec server. Even more: backup to HP LTO5 drive runs 3-4 times faster, then backup to IBM LTO8
My Fiber controller is QLogic with driver from QLogic 17.08.2015
Library is: IBM TS 4300. Firmware: 1.5.0.1-B00
Drives: IBM LTO 8 HH FC. Firmware: P381
Drives (as they look in Windows): IBM ULTRIUM 8 HH 3580 TAPE DRIVE
Driver of drives: From IBM, version: 7.0.0.8, date: 11.04.2022 (non-exclusive)
We recently changed backup server from old HP G5 server to ProLiant BL460c Gen9. We didn't move configuration. We installed everything from scratch. But on old server - there where no issues with speed. The speed on old server was about 10 times faster - if I backup file from local drive.
08-31-2022 04:45 PM
While trying to resolve this further I discovered very strange thing: if I write file to new, scratch LTO8 media - the speed is very good. But subsequent appends are slow. WTF???????
09-01-2022 08:09 AM
Regarding this point:
"While trying to resolve this further I discovered very strange thing: if I write file to new, scratch LTO8 media - the speed is very good. But subsequent appends are slow."
Is that still a duplicate job or do you see the same behavior if you test this with just a normal backup job directly to tape?
09-01-2022 09:43 AM
This is file to tape backup. File size is about 6GB. This is iso file. The file is located LOCALLY on backup server. Ethernet is not involved, only Fiber. Because LTO8 is connected via Fiber.
@RogRubin Of what do you think? This behavior looks very strangle.
09-02-2022 04:01 AM
Is the initial write to the new scratch tape a full backup, and subsequent writes incremental or differential backups? If so that makes more sense as these jobs will always have a slower rate than the full backups (due to the overhead of the system determining which files are required to be backed up)
If 6GB is the size of the initial backup, what size are the subsequent appended backups? The smaller the backup size, the less reliable the speed statistics will be - eg 6GB is quite small so if subsequent amounts appended to the tape are even smaller the figure will be more greatly affected by the mount and dismount times as well as searching the tape to locate the correct location to start the next append.
The first full backup also has the advantage that this last process (winding the tape forward to get to the correct place to start the backup) is quicker as the backup just has to start at the beginning.
09-02-2022 04:28 AM
No incremental or differential backups are even configured. Only Full Backup.
The size of BOTH initial and subsequent backups are THE SAME. I attach the screenshot - please look at the size of backup - it is always 5.03 GB. Initially I made bigger backups (about 160 GB), but later start making smaller backups, because I make many speed test. Waiting each time is pain.
Mount and dismount time, searching the tape to locate the correct location (I believe) - do not contribute to the value "Job rate" column.
I do not care about mounting tape speed and searching speed. I only look to job rate and on appends it's slower.
Screenshot attached.
09-02-2022 05:28 AM
May be it's somehow related to new 22 version? Since we are using it.
09-02-2022 07:00 AM
BTW - red squares are appends, green squares - are first writes to scratch tape. Opened a case in your support - now at the stage of uploading logs.
09-03-2022 09:13 AM
We have time till Monday. After it - I will destroy this server and we will never know what is the cause of this problem.
09-05-2022 12:02 AM
Have you opened a job and then compared the different sections?
Is there one particular section which is taking longer?
09-05-2022 12:09 AM
Job consist of 1 file, which is located on local hard drive of the server. I would rather think in direction of fiber HBA may be?
09-11-2022 01:45 AM
I found out that if I delete IBM TS4300 Tape Library IBM driver and leave old (2006 year) Microsoft driver IBM ULT3580-HH8 SCSI Sequential Device - there are no issues with speed: both on first write and all subsequent appends.
I will try 21 version of Veritas. May be there will be no problem there.
Or there is something other I can try?
There are some properties on the driver of IBM drives in Device Manager. But they are grayed out. I can not change them.
09-11-2022 08:45 AM
09-11-2022 09:16 PM - edited 09-11-2022 09:17 PM
I have tested on these drivers:
IBMTape.w16_7008-x64_WHQL_Cert
IBMTape.x64_w16_6266
IBMTape.w22_7008-x64_WHQL
The version of BackupExec is 22, with hotfix
Yeah, really very strange problem and is very critical to us.