VM Backups over Transport Type = SAN is slow,better Transport Type = nbd

[sigitbys]

Hi guys 

please help me whether any problem from my configuration 

why backups vm with transport SAN is slow better transport ndb this the detail status backup :

Over SAN 

May 12, 2017 5:00:53 PM - Info nbjm (pid=10184) starting backup job (jobid=25320) for client 10.148.148.201, policy vm_fs1, schedule full
May 12, 2017 5:00:53 PM - estimated 0 kbytes needed
May 12, 2017 5:00:53 PM - Info nbjm (pid=10184) started backup (backupid=10.148.148.201_1494583253) job for client 10.148.148.201, policy vm_fs1, schedule full on storage unit Dedupe_STU using backup host mjtbackup.mjt.app.com
May 12, 2017 5:00:54 PM - Info bpbrm (pid=6048) 10.148.148.201 is the host to backup data from
May 12, 2017 5:00:54 PM - Info bpbrm (pid=6048) reading file list for client
May 12, 2017 5:00:54 PM - started process bpbrm (pid=6048)
May 12, 2017 5:00:54 PM - connecting
May 12, 2017 5:00:55 PM - Info bpbrm (pid=6048) starting bpbkar32 on client
May 12, 2017 5:00:55 PM - connected; connect time: 0:00:00
May 12, 2017 5:00:56 PM - Info bpbkar32 (pid=12240) Backup started
May 12, 2017 5:00:56 PM - Info bpbkar32 (pid=12240) archive bit processing:<enabled>
May 12, 2017 5:00:56 PM - Info bptm (pid=4784) start
May 12, 2017 5:00:56 PM - Info bptm (pid=4784) using 262144 data buffer size
May 12, 2017 5:00:56 PM - Info bptm (pid=4784) setting receive network buffer to 1049600 bytes
May 12, 2017 5:00:56 PM - Info bptm (pid=4784) using 16 data buffers
May 12, 2017 5:00:57 PM - Info bptm (pid=4784) start backup
May 12, 2017 5:00:58 PM - Info bpbkar32 (pid=12240) INF - Backing up vCenter server vcenter, ESX host 10.148.148.31, BIOS UUID 423b24a9-a027-8d7d-4a4f-c2e9a2dfe70a, Instance UUID 503bc590-9ef4-16ce-9cfa-b96c8ae0eb48, Display Name MJTFS1-3, Hostname 10.148.148.201
May 12, 2017 5:01:09 PM - begin writing
May 14, 2017 12:52:38 AM - Info bpbkar32 (pid=12240) INF - Transport Type = san
May 15, 2017 7:29:08 AM - Info bptm (pid=4784) waited for full buffer 840270 times, delayed 11476751 times
May 15, 2017 7:29:10 AM - Info bpbkar32 (pid=12240) bpbkar waited 593955 times for empty buffer, delayed 597878 times.
May 15, 2017 7:30:51 AM - Info bptm (pid=4784) EXITING with status 0 <----------
May 15, 2017 7:30:51 AM - Info mjtbackup.mjt.app.com (pid=4784) StorageServer=PureDisk:mjtbackup.mjt.app.com; Report=PDDO Stats for (mjtbackup.mjt.app.com): scanned: 1948006304 KB, CR sent: 1534887442 KB, CR sent over FC: 0 KB, dedup: 21.2%, cache hits: 575974 (2.3%), rebased: 233321 (0.9%)
May 15, 2017 7:30:51 AM - Info bpbrm (pid=6048) validating image for client 10.148.148.201
May 15, 2017 7:31:31 AM - Info bpbkar32 (pid=12240) done. status: 0: the requested operation was successfully completed
May 15, 2017 7:31:31 AM - end writing; write time: 62:30:22
the requested operation was successfully completed (0)

Over ndb

May 13, 2017 2:01:23 AM - Info nbjm (pid=10184) starting backup job (jobid=25326) for client 10.148.148.203, policy vm_fs3, schedule full
May 13, 2017 2:01:24 AM - estimated 2119934476 kbytes needed
May 13, 2017 2:01:24 AM - Info nbjm (pid=10184) started backup (backupid=10.148.148.203_1494615683) job for client 10.148.148.203, policy vm_fs3, schedule full on storage unit Dedupe_STU using backup host mjtbackup.mjt.app.com
May 13, 2017 2:01:24 AM - started process bpbrm (pid=684)
May 13, 2017 2:01:25 AM - Info bpbrm (pid=684) 10.148.148.203 is the host to backup data from
May 13, 2017 2:01:25 AM - Info bpbrm (pid=684) reading file list for client
May 13, 2017 2:01:25 AM - Info bpbrm (pid=684) starting bpbkar32 on client
May 13, 2017 2:01:25 AM - connecting
May 13, 2017 2:01:25 AM - connected; connect time: 0:00:00
May 13, 2017 2:01:26 AM - Info bpbkar32 (pid=12324) Backup started
May 13, 2017 2:01:26 AM - Info bpbkar32 (pid=12324) archive bit processing:<enabled>
May 13, 2017 2:01:26 AM - Info bptm (pid=12712) start
May 13, 2017 2:01:26 AM - Info bptm (pid=12712) using 262144 data buffer size
May 13, 2017 2:01:26 AM - Info bptm (pid=12712) setting receive network buffer to 1049600 bytes
May 13, 2017 2:01:26 AM - Info bptm (pid=12712) using 16 data buffers
May 13, 2017 2:01:28 AM - Info bptm (pid=12712) start backup
May 13, 2017 2:01:28 AM - Info bpbkar32 (pid=12324) INF - Backing up vCenter server vcenter, ESX host 10.148.148.31, BIOS UUID 423536a3-e76a-53d4-4168-f6d38fc09a6c, Instance UUID 50351418-66d6-11dd-cf96-d61edca9e060, Display Name MJTFS3, Hostname 10.148.148.203
May 13, 2017 2:01:38 AM - begin writing
May 13, 2017 3:44:35 AM - Info bpbkar32 (pid=12324) INF - Transport Type = nbd
May 14, 2017 4:17:42 AM - Info bptm (pid=12712) waited for full buffer 2866029 times, delayed 5064270 times
May 14, 2017 4:17:45 AM - Info bpbkar32 (pid=12324) bpbkar waited 121 times for empty buffer, delayed 8208 times.
May 14, 2017 4:21:01 AM - Info bptm (pid=12712) EXITING with status 0 <----------
May 14, 2017 4:21:01 AM - Info mjtbackup.mjt.app.com (pid=12712) StorageServer=PureDisk:mjtbackup.mjt.app.com; Report=PDDO Stats for (mjtbackup.mjt.app.com): scanned: 1776648238 KB, CR sent: 25803727 KB, CR sent over FC: 0 KB, dedup: 98.5%, cache hits: 267023 (0.5%), rebased: 73006 (0.1%)
May 14, 2017 4:21:02 AM - Info bpbrm (pid=684) validating image for client 10.148.148.203
May 14, 2017 4:21:32 AM - Info bpbkar32 (pid=12324) done. status: 0: the requested operation was successfully completed
May 14, 2017 4:21:32 AM - end writing; write time: 26:19:54
the requested operation was successfully completed (0)

Regards,

sigit 

[Marianne · Accepted solution]

Firstly - compare apples with apples.

The SAN backup was a lot bigger than ndb backup and much more unique data:

SAN:
scanned: 1776648238 KB, CR sent: 1534887442 KB

ndb:
scanned: 1948006304 KB, CR sent: 25803727 KB

From NBU point of view, there is not much that can be done other than increasing the number of data buffers:

Info bptm (pid=12712) using 16 data buffers

16 is way too low. Increase that to 64.

You may also want to consult with your SAN admin to monitor activity and throughput on switch ports while the backup is running.

[quebek]

Hi

If you can try to use NBU accelerator for VM based backup - have wmare admin to enable CBT.

Also as Marriane stated two different client... two different dedupe rates: 

scanned: 1948006304 KB, CR sent: 1534887442 KB, CR sent over FC: 0 KB, dedup: 21.2%, cache hits: 575974 (2.3%), rebased: 233321 (0.9%)

scanned: 1776648238 KB, CR sent: 25803727 KB, CR sent over FC: 0 KB, dedup: 98.5%, cache hits: 267023 (0.5%), rebased: 73006 (0.1%)

I would enabled NBU accelerator.... as it already looks like you do write to disk maybe to MSDP...

[sigitbys]

Hi Marianne,

Okay I have change buffering to 64 and now I rerun the job 

we will see later result 

[sigitbys]

 Hi quebek,

It is quite strange thought this in the same vcenter and policy just different my clients 

yes I have enable now in vm CBT is automatic 

will see the result