cancel
Showing results for 
Search instead for 
Did you mean: 

Backup exec 21.2 Upgrade from 21.1 Speed Issue

steves_2
Level 4

Hi All,

Has anyone noticed any speed issues with BE 21.2 (21.0.1200.1899)

For example a backup of a VM that takes 40 seconds now takes over 4 mins, used to take 1 min with 21.1.

Look at the start and finish times compared to the VM backup times.

 

21.2

Backup started on 12/05/2021 at 17:01:45.
Backup Set Detail Information
Media Label: IMG000170
GRT backup set folder: E:\BEData\IMG000170
The standard processing method will be used for incremental and differential backups.
The data in Pagefile.sys will be skipped during backup because the data is transient and does not need to be restored.
Backup Exec has discovered and protected 'C:' on virtual machine 'XXXXXX-DC01.XXXXXX.local'.
Backup Exec has discovered and protected Microsoft Active Directory data on virtual machine 'XXXXXXl-DC01.XXXXXX.local'.
Backup completed on 12/05/2021 at 17:05:50.
Backup Set Summary
Backed up 3 files in 8 directories.
Processed 16,599,073,988 bytes in  40 seconds.
Throughput rate: 23745 MB/min

 

 

 

21.1

Backup started on 08/05/2021 at 17:01:07.
Backup Set Detail Information
Media Label: IMG000114
GRT backup set folder: E:\BEData\IMG000114
The standard processing method will be used for incremental and differential backups.
The data in Pagefile.sys will be skipped during backup because the data is transient and does not need to be restored.
Backup Exec has discovered and protected 'C:' on virtual machine 'XXXXXX-DC01.XXXXXX.local'.
Backup Exec has discovered and protected Microsoft Active Directory data on virtual machine 'XXXXXX-DC01.XXXXXX.local'.
Backup completed on 08/05/2021 at 17:02:08.
Backup Set Summary
Backed up 3 files in 8 directories.
Processed 16,210,747,454 bytes in  37 seconds.
Throughput rate: 25070 MB/min

 

 

I have seem something strange in the log files between the 2 backups.

In 21.1 i'm seeing this :

[7060] 2021-05-01T17:02:04.290 [loops] - BackupBSDProcessor::ExtendedAddFile: m_bVirtSet 1 m_virtDataId 60 m_virtBedsId 60 m_virtBedsVer 0
[7060] 2021-05-01T17:02:04.295 [server] - MSG_END_VIRTUAL_BACKUP_SET: Restore Work Flow (RWF)

in 21.2 im seeing this :

BENGINE: [05/12/21 17:02:41] [5800] [loops] - BackupBSDProcessor::ExtendedAddFile: m_bVirtSet 1 m_virtDataId 14 m_virtBedsId 73 m_virtBedsVer 0
BENGINE: [05/12/21 17:02:50] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:02:55] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:00] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:05] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:10] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:15] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:20] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:25] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:30] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:35] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:40] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:45] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:50] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:03:55] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:00] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:05] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:10] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:15] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:20] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:25] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:30] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:35] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:40] [5800] [ndmp\ndmpcomm] - ndmp_readit: Control connection socket (0x8e4) not readable!
BENGINE: [05/12/21 17:04:42] [5800] [server] - MSG_END_VIRTUAL_BACKUP_SET: Restore Work Flow (RWF) is empty

 

I'm getting a lot of these ndmp readit errors, what ever they are. I don't have ndmp enabled in the options.

Can anyone shed any light on what is going off here as i'm guessing this is causing all the VM backups to slow down making the backups times about 40% longer.

cheers

 

 

9 REPLIES 9

CraigV
Level 6
Partner    VIP    Accredited

Hi,

I've seen this on previous versions where a version that was updated too ended up being slower. Make sure there are no new driver requirements for your backup target, and also check LiveUpdate to see if there are any updates available for the version you are now on.

Thanks!

Hi Craig, thanks for the reply.

Nothing extra required that i can see in the guide or read me. I even found and updated to BE 21.2 FP2 (21.0.1200.1930) to see that that made a difference, but it hasn't.

It just looks like a ndmp issue with the new version and it is timing out, putting huge delays in the backups as it retries every 5 seconds.

It must be something to to with the 21.2 backup engine as some VM's i haven't updated the remote agent and they are showing the same slower backup running on  21.1 agent.

Cheers

 

Cheers

I noticed this also since R2.  Performing an agent based backup runs for us at ~25,000MB/min while Hyper-V based backups of the same VM barely achieve 3,000MB/min.  Haven't been able to point at exactly what the difference might be yet here.

Ah, so i'm not alone. Hopefully this issue will get resolved.

I have a support ticket open for this issue.

Do you have random cloud tape drives or remote Linux storage present?

After removing the issue per this KB, affected backups in some cases are finally showing line speed: https://www.veritas.com/support/en_US/article.100032514

 

 

Hi,

No nothing like that. The speed issues are occurring when using backup to disk on windows servers.

The network speed is good (400MB/s) and the remote agents are working correctly.

Something fundamentally changed between 21.1 and 21.2 that has caused delays in the backup exec engine on the backup server.

As you can see from the 1st post the speed of the backup hasn't really changed 37 - 40 secs.

Just the overall processing time of the backup from 1 min to 4 mins.

So add this extra processing time  to every VM and it soon adds up.

cheers

NageshPatil
Moderator
Moderator

We have not seen any concerns about slowness in VM backup after upgrade to Backup Exec 21.2. We suggest we check below things to possibly isolate the cause : 1. Does Backup Exec Remote Agent version of all VMs matches to the media server version i.e. 21.2 ? If not, I suggest we upgrade all remote agents and observe the speed. 2. In case you are observing the slowness for specific VM, I suggest you to disable all GRT options and give a try once. This would help to isolate any VM or workload specific issue. Thank you. Regards, Nagesh

Hi,

I've done some more testing today.

I exported a small VM (15GB) from our system to a new test server and installed 21.1

This server has backup exec and hvper-v installed so there is no network traffic as everything is local.

Ran the backup with debug logs enabled and catalog generation took 42 seconds on 21.1

[14052] 2021-06-25T10:02:12.989 [loops] - PDI backup: Starting catalog generation on server [TestHV]

[14052] 2021-06-25T10:02:54.208 [loops] - PDI backup: Catalog generation done

I upgraded straight to 21.2r2 and ran the backup with debug logs enabled and same catalog generation took 3 mins and 48 seconds.

[8408] 2021-06-25T11:02:11.799 [loops] - PDI backup: Starting catalog generation on server [TestHV]

[8408] 2021-06-25T11:05:59.772 [loops] - PDI backup: Catalog generation done

It's all seems to be pointing to the Instant GRT cataloging for VM's.

This particular VM was a domain controller, but i'm seeing the same slowness with SQL databases as well.

Disabling GRT on 21.2r2 does make it run faster, but still not as fast as 21.1 with instant GRT.

So i'm seeing exactly the same issue on different server with backup exec with backup exec 21.1 v 21.2.

Just an update to the post. Issue is under investigation.

https://www.veritas.com/content/support/en_US/article.100051020

 

Cheers