β05-13-2021 01:44 AM
Hi All,
Has anyone noticed any speed issues with BE 21.2 (21.0.1200.1899)
For example a backup of a VM that takes 40 seconds now takes over 4 mins, used to take 1 min with 21.1.
Look at the start and finish times compared to the VM backup times.
21.2
|
21.1
|
I have seem something strange in the log files between the 2 backups.
In 21.1 i'm seeing this :
[7060] 2021-05-01T17:02:04.290 [loops] - BackupBSDProcessor::ExtendedAddFile: m_bVirtSet 1 m_virtDataId 60 m_virtBedsId 60 m_virtBedsVer 0
[7060] 2021-05-01T17:02:04.295 [server] - MSG_END_VIRTUAL_BACKUP_SET: Restore Work Flow (RWF)
in 21.2 im seeing this :
BENGINE: [05/12/21 17:02:41] [5800] [loops] - BackupBSDProcessor::ExtendedAddFile: m_bVirtSet 1 m_virtDataId 14 m_virtBedsId 73 m_virtBedsVer 0
BENGINE: [05/12/21 17:02:50] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:02:55] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:00] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:05] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:10] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:15] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:20] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:25] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:30] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:35] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:40] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:45] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:50] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:55] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:00] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:05] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:10] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:15] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:20] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:25] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:30] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:35] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:40] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:42] [5800] [server] - MSG_END_VIRTUAL_BACKUP_SET: Restore Work Flow (RWF) is empty
I'm getting a lot of these ndmp readit errors, what ever they are. I don't have ndmp enabled in the options.
Can anyone shed any light on what is going off here as i'm guessing this is causing all the VM backups to slow down making the backups times about 40% longer.
cheers
Solved! Go to Solution.
β12-13-2021 12:40 AM
HI, Just an update on this issue. https://www.veritas.com/content/support/en_US/article.100051020
Last week Veritas release Backup Exec 21.4.
This fixes the issue i was having with slow back times after upgrading to 21.2.
My Backups are now completing roughly 30% faster with 21.4 bringing it back inline to what i was seeing with 21.1.
Cheers
Steve
β05-14-2021 03:21 AM
Hi,
I've seen this on previous versions where a version that was updated too ended up being slower. Make sure there are no new driver requirements for your backup target, and also check LiveUpdate to see if there are any updates available for the version you are now on.
Thanks!
β05-14-2021 04:33 AM
Hi Craig, thanks for the reply.
Nothing extra required that i can see in the guide or read me. I even found and updated to BE 21.2 FP2 (21.0.1200.1930) to see that that made a difference, but it hasn't.
It just looks like a ndmp issue with the new version and it is timing out, putting huge delays in the backups as it retries every 5 seconds.
It must be something to to with the 21.2 backup engine as some VM's i haven't updated the remote agent and they are showing the same slower backup running on 21.1 agent.
Cheers
Cheers
β05-24-2021 02:42 PM
I noticed this also since R2. Performing an agent based backup runs for us at ~25,000MB/min while Hyper-V based backups of the same VM barely achieve 3,000MB/min. Haven't been able to point at exactly what the difference might be yet here.
β06-01-2021 02:40 AM
Ah, so i'm not alone. Hopefully this issue will get resolved.
I have a support ticket open for this issue.
β06-06-2021 02:35 AM
Do you have random cloud tape drives or remote Linux storage present?
After removing the issue per this KB, affected backups in some cases are finally showing line speed: https://www.veritas.com/support/en_US/article.100032514
β06-10-2021 03:27 AM - edited β06-10-2021 03:29 AM
Hi,
No nothing like that. The speed issues are occurring when using backup to disk on windows servers.
The network speed is good (400MB/s) and the remote agents are working correctly.
Something fundamentally changed between 21.1 and 21.2 that has caused delays in the backup exec engine on the backup server.
As you can see from the 1st post the speed of the backup hasn't really changed 37 - 40 secs.
Just the overall processing time of the backup from 1 min to 4 mins.
So add this extra processing time to every VM and it soon adds up.
cheers
β06-24-2021 03:13 AM
We have not seen any concerns about slowness in VM backup after upgrade to Backup Exec 21.2. We suggest we check below things to possibly isolate the cause : 1. Does Backup Exec Remote Agent version of all VMs matches to the media server version i.e. 21.2 ? If not, I suggest we upgrade all remote agents and observe the speed. 2. In case you are observing the slowness for specific VM, I suggest you to disable all GRT options and give a try once. This would help to isolate any VM or workload specific issue. Thank you. Regards, Nagesh
β06-25-2021 04:35 AM - edited β06-25-2021 04:38 AM
Hi,
I've done some more testing today.
I exported a small VM (15GB) from our system to a new test server and installed 21.1
This server has backup exec and hvper-v installed so there is no network traffic as everything is local.
Ran the backup with debug logs enabled and catalog generation took 42 seconds on 21.1
[14052] 2021-06-25T10:02:12.989 [loops] - PDI backup: Starting catalog generation on server [TestHV]
[14052] 2021-06-25T10:02:54.208 [loops] - PDI backup: Catalog generation done
I upgraded straight to 21.2r2 and ran the backup with debug logs enabled and same catalog generation took 3 mins and 48 seconds.
[8408] 2021-06-25T11:02:11.799 [loops] - PDI backup: Starting catalog generation on server [TestHV]
[8408] 2021-06-25T11:05:59.772 [loops] - PDI backup: Catalog generation done
It's all seems to be pointing to the Instant GRT cataloging for VM's.
This particular VM was a domain controller, but i'm seeing the same slowness with SQL databases as well.
Disabling GRT on 21.2r2 does make it run faster, but still not as fast as 21.1 with instant GRT.
So i'm seeing exactly the same issue on different server with backup exec with backup exec 21.1 v 21.2.
β08-02-2021 03:16 AM
Just an update to the post. Issue is under investigation.
https://www.veritas.com/content/support/en_US/article.100051020
Cheers
β12-13-2021 12:40 AM
HI, Just an update on this issue. https://www.veritas.com/content/support/en_US/article.100051020
Last week Veritas release Backup Exec 21.4.
This fixes the issue i was having with slow back times after upgrading to 21.2.
My Backups are now completing roughly 30% faster with 21.4 bringing it back inline to what i was seeing with 21.1.
Cheers
Steve
β12-13-2021 12:57 AM - edited β12-13-2021 12:59 AM
.