NBU, NDMP slow ~10MB/s only

[ERICsssss]

Hello everybody!

We are using NBU 7.1.0.1 to backup IBM N6 filer by NDMP to disk on media server with Giga LAN but got only max 10 - 11MB/s. It is very slow and need 3days to full backup our 2TB data in filer. Tested using Backupexec with NDMP on the same media server but got 80MB/s and used only 11hrs to complete the full backup. I think NBU could get the same speed or more faster. I asked for help with my support for a month already but no luck! Hope someone can help me here, thanks so much!

Here is our servers info:

Master Server
Dell PE2950 + 2GB + 1TB SAS + Win08R2 SP1 + NBU 7.1.0.1

Media Server
HP DL380 G4 + 4GB + 3TB SATA (Raid-5) + Win08R2SP1 + NBU 7.1.0.1  (BE12.5 was tested in this server)

Filer
IBM N6040 + 2TB data + 1GBx4 Network Virtual Interface
-DataOnTap 8.0
-about 1,300,000 files and 120,000 folders.
-we backup the snapshot folder: /vol/vol1/.snapshot/nightly.0

What we tested:
1. NUMBER_DATA_BUFFERS, NUMBER_DATA_BUFFERS_DISK, NUMBER_DATA_BUFFERS_NDMP - tried values 16, 32, 64, 128, 256.
2. SIZE_DATA_BUFFERS, SIZE_DATA_BUFFERS_DISK, SIZE_DATA_BUFFERS_NDMP - tried values 262144, 524288.
3. set HIST=n (But I do not want to use this since it will cause no folder and file history)
4. vxlogcfg -a -p 51216 -o Default  -s DiagnosticLevel=0 -s DebugLevel=0
5. Allow / Not Allow Multiple data streams
6. online folder or snapshot folder (i.e. /vol/vol1 or /vol/vol1/.snapshop/nightly.0)
7. Master Server Properties – Bandwidth / Throttle Bandwidth 0KB or 10000KB
8. office hour or non-office hour

Unfortunately, all of the changes above didn’t have any improvement, the backup keep only around 10-11MB.
I also tested NBU v7.0, v7.0.1, v7.1, same.

If anyone have any idea please help me =) thank you very much!
Eric

[jim_dalton]

Eric

What is your media server connected to network-wise and what path does the data take?

Your data have to go over the network - unless I misread your setup - so we need to see how that is set up.

Same deal for the master, but I would assume the media server will have greater opportunity to bottleneck.

Following the theme, have you tried a non NDMP backup using the same media server to see how that performs? You can use the data creating facilities or give it some nice (ie large multi gb files) so youre testing the xfer and not the other parts of the ensemble.

If it too is slow then (as you probably suspect) NDMP is being bottlenecked by something else, the net, the disk, the raid...

Jim

[jim_dalton]

I see you say with same media server you have got 80m/s so we assume the media server is unchanged so I would head to my last suggestion: run a non ndmp backup and see where it leads us.

Jim

[ERICsssss]

I performed some tests:

On Server:Media Server
Type:NBU 7.1.0.1 NDMP
Copied File Size: 13,663,993,856 bytes
Time Used: 00:19:52
Speed: 11MB/Sec

On Server:Media Server
Type: Windows Direct Copy
Copied File Size: 13,585,956,864 bytes
Time Used: 00:05:43
Speed: 38MB/Sec

On Server: Filer
Type: DUMP
Copied File Size: 13,653,608,448 bytes
Time Used: 00:01:07
Speed: 195MB/Sec

On Server:Media Server
Type: NBU 7.1.0.1 Map Network Drive
Copied File Size: 13,647,003,648 bytes
Time Used: 00:05:54
Speed: 36MB/Sec

[jim_dalton]

Let me see if this is correct:

Test 1 is the problem:ndmp backup across the network to media server,slow.

Test 2 is a windows copy to the media server: is this local->local or via network?.

Test 3 is local ndmp dump to self, rapid as you would expect.

Test 4 is ..what is test 4 .. a backup of a networked drive to disk? Also decent speed .

So we conclude:

-network is likely to run at 30 odd M/s

-nas can deliver data very rapidly (to itself)

-ndmp across network is slow.

A thought:do you happen to have another filer that you can target the dump to? This would exercise the teamed NIC which is only exercised in Test 1. My suspicion is the network: the media server can write at a fair speed, the nas can read data at a very high rate but combine the media server, the nas and the network in between and it slows right down.

I'd go to my network guy.

Jim

[ERICsssss]

Thank you for your comments Jim!

For Test 2, I map a share drive X: on the media server from the filer CIFS share, and copy from filer to media hard disk using Windows Explorer.

For Test 4, using media server NBU, backup the filer CIFS share \\filer\vol\vol1\ (path same with the above drive X:) (This is the test reply your first comment =) )

and, I have a Test 5, using BackupExec on the same media server above, backup the same path using NDMP option, i got 80MB/sec.

I am sorry I don't have another filer.

[jim_dalton]

Test 5 is interesting...same setup , same network, same data , different app :much more perfomant.

Are BackupExec and NB living together happily? Can you do the test with BE completely removed from the scene? Or have another media server (in a VM maybe) that hasnt had BE ever?

Intriguing indeed.

[ERICsssss]

Sorry for late reply!

Haha! Yes... same enviroment but difference performance..

Sure, I tested below before:

5a. Server1 + NBU
5b. Server1 + NBU + BE
5c. Server1(Reinstall) + BE
5d. Server2 + NBU + BE
5e. Server3 + NBU + BE

Win03, Win08R2, NBU7.0-7.1.0.1, BE12.5, 2-4GB RAM, SAS/SATA/SCSI....
all the results are same NBU Max 10-11MB/s, BE Max 80MB/s / .\

dizzy now

[Michael_G_Ander]

Don't know much about NDMP, but as it seems to be somtehing with how netbackup uses the network I would check if NET_BUFFER_SZ had an impact.

Maybe there something like NET_BUFFER_SZ_NDMP ?

Regards

Michael

[ERICsssss]

I am not lazy, I performed a network test and got a funny result.....

Environment	Network Connection	NBU w/NDMP	BE w/NDMP	Windows Explorer File Copy
Production	N6040 --[1GBx4]-- C4506  --[1GBx2]-- PE2850	10MB/Sec	80MB/Sec	50MB/Sec
Test1	N6040 --[1GBx2]-- C4506  --[1GBx2]-- PE2850	10MB/Sec	80MB/Sec	50MB/Sec
Test2	N6040 --1GB-- C4506  --[1GBx2]-- PE2850	10MB/Sec	80MB/Sec	50MB/Sec
Test3	N6040 --1GB-- C4506  --1GB-- PE2850	10MB/Sec	80MB/Sec	50MB/Sec
Test4	N6040 --1GB-- C2950  --1GB-- PE2850	100MB/Sec	80MB/Sec	50MB/Sec
Test5	N6040 --1GB-- PE2850	100MB/Sec	80MB/Sec	50MB/Sec

N6040 = IBM N6040 with DataOnTap 8.0
C4506 = Cisco 4506 Switch
C2950 = Cisco 2950 Switch
PE2850 = Dell PE2850

Finally I got a better speed by changing the network connection - 100MB/Sec!

Funny thing is....how come BE and Direct Copy has no affect? I am very confused....

Due to BE 80MB/Copy 50MB in all tests, I think the network in Production Environment is fine and I still want to fix it instead of adding a direct cable... Does any setting need to apply between Cisco switch and NBU?

help!

thank you!
Eric~

[jim_dalton]

Good tests, but I'm also confused. Accoridng to my lookup, 2950 only has 100mbit datalinks (with gbit uplink) so how did you get 100MB/secin test4? Thats faster than can be supported by the switch. I'm not a network guy, I add. I'm assuming MB=megabyte, not megabit, yes?

It would make more sense if that 2950 in test4 was a 4506 and the 4506 in test3 was a 2950...then we could say: 2950 will only go at 100mbit max (which is kind of about right for NB but not for other tests) and the 4506 nic teaming affects the NB perfomance of an otherwise sound single nic perfomance. All teaming tests give bad results with NB, only when theres a single nic does it give good results.

I think youre very close to the solution....

Jim

[ERICsssss]

LOL! you are right! sorry for my typo!
it sould be 2960 switch with two 1G ports.

Since I have only two 1G ports in 2960 switch, so that I cannot try the teaming affect if using 2960. But test3 already show not affected by teaming.

Eric

[jim_dalton]

Does look like a netb+4506 issue but you suspected that already.Got any network analysis tools? Is the nic connected at 1gbit when you run the specific test? If so then we need to look at the storage.Its a tricky one!

jim

[ERICsssss]

I think, the BE and file copy tests can show the network and storage are ok. and Yes, tests was is 1GB or 2GB teaming.