performance on Netbackup 6.5.4

[ITBOY]

Gentlemen,
I have problem to backup my data within the specified period.In fact ,I have about 6TB full data to run netbackup job every week. But according to our calcuation, it will take more 33 hours which is more than 24 hours ( 1 day ) . We are to backup via the dedicated 1Gbps ethernet link huge amount of data everyday.
In order to maximize the perfomance of netbackup ,which of the following ways is the best option to do ? ( Which one is the bottle neck of the performace ?)
( 1 ) Increase the size of Memory of the backup server ? ( Provided that memory slots are available )
( 2 ) Increase the CPU speed of the backup server?
( 3 ) Backup over the FC ( fibre channel ) link instead of gibabit ethernet ? ( in that case , which would be running the speed of the gibabit ethernet , 1Gbps , or only 80 % of 1 Gbps ?)
( 4 ) Increase the number of backup tape drives in the tape library?
( 5 ) Segregate the amount of data to backup and run the backup job concurrently?
        Is there any way to compute the backup window time frame ?
Thx.

[ITBOY]

Hi Nathan,
Thx. In fact binary files need to be backup. Is there limitation that the number of files that netbackup support? According to the Synthetic backup, need storage space. But  i m wondering how to backup the full amount of 5 TB data every week within 24 hours. This is my concern.

Schedules to include in a policy for synthetic backups

A policy for synthetic backups must contain at least three types of schedules:

■

At least one traditional, full backup must be run successfully to create a full image. The synthetic backup jobs fails if there is not at least one previous full image

[Mike_Gavrilov]

Yes , 3 NICs +3 VLANS=No trunks :)

[Mike_Gavrilov]

What speed you expect to get between client and media server? And what speed do you have now?

[sdo]

I agree with Garilov...
6 TB of data is quite a lot for any one client, no matter how large our NetBackup environments are.
If had to backup 6 TB every night, then I'd definitely be looking at adding some HBAs to the client, getting some NetBackup SSO licences and sharing with the normal media servers.

1 Gb/s
1000 Mb/s
125 MB/s
60% efficiency
75 MB/s max practical
 
6 TB
6,144 GB
6,291,456 MB
83,886 seconds
23 hours

Based on these numbers you'll be lucky to get the job completed in 35 hours, let alone 23 - as the above assumes that the disks + the client + the LAN + the media server + the SAN + the tape drives are all not busy doing other things - and it is highly unlikely that even just two of these components are idle at any given time.

Also consider, if it's 6 TB now, what will it be next year, and the year after that.

IMHO - a bigger box for the client, two HBA ports per fabric (e.g. two dual port cards), 4 x LTO4 (i.e. 2 x LTO4 per fabric), and very carefully balance the file systems and disk groups with your NetBackup policy path selections and/or database elections - you'll need to work very closely with your DBAs.

Pay careful attention to the "Backup Planning Guide" - and consider how busy the client hardware will be supporting all this additional work load - i.e. how much CPU, RAM and disk space you need just for NetBackup.

HTH,
Dave.

[sdo]

I would aim to complete the backup in under six hours, to allow for maintenance windows, down time, day to day problems, backup re-runs, restores, other works on either client, media server, LAN, SAN, library, tape drives, O/S upgrades, S/W patches, firmware updates, etc etc - i.e. someone else will always be delaying your backups!

90 MB/s per LTO4 practical
4 drives
360 MB/s
2880 Mb/s
2.81 Gb/s
 Therefore 2 x 4 Gb/s FC paths are required for redundancy
 
6 TB
6,144 GB
6,291,456 MB
360 MB/s
17,476 seconds
5 hours

Also, reading above, it still wasn't clear to me how many files are involved.  Someone else picked up on this too.  If this is one huge database server then it's probably not an issue, but if this is a file server with millions or even tens of millions of small files, then you will highly likely need to use the FlashBackup capabilities of the Snapshot Client option (which is within the "NetBackup Enterprise Client" licence).

HTH.

[sdo]

One final point - NetBackup does scale!  It just takes time to work it all out.

You will have to do your math for each stage of data producer (clients) and data consumers (media servers) - or in your case both in the same box (i.e. the client itself) - and think about the performance profile, capabilities and limititations of each single component/stage/hop along the entire data path end-to-end, e.g.:
disks - RAID - shelves/controllers/heads - storage HBAs - SAN path - switch - SAN path - client HBAs - CPU/memory - NetBackup buffers - client HBAs - SAN path - switch - SAN path - tape HBAs - tape drives - tapes.

A word of advice - do not be tempted to zone both your FC disk and your FC tape through the same HBAs.  I believe that the standard recommendations still are not to mix disk and tape traffic.  So, I recommend additional separate HBAs in the client specifically for backup traffic to tape.

An example of scaling, we have four media servers - each has:
NICs - Eight - in a 2 trunks of 4 Gb/s in IPMP failover config.
HBAs - Seven dual port HBAs - i.e. 14 FC connections per media server - thus 7 on each fabric.
Zoning - Carefully patched and zoned so that all backup SAN traffic always stays within the ASICs of the SAN switches - i.e. the backup FC traffic never enters the backplane of any SAN switch nor does the backup FC traffic traverse switch to switch.  We have disk storage arrays, VTLs and tape libraries connected to our media servers hence the number of FC connections in each.  There are four media servers as we determined that site resilience, site H/A and site DR were all important factors.

Your business requirements may well be different.
6 TB of data is not inconsequential.
Good luck.

[Mike_Gavrilov]

Ok.
Look, you have 3  X 1G  NICs (3 VLANs). What speed do you get for this NICs? Something about 3X50(MB)=150 MB/sec. Speed between Media-server and TapeLibrary is 360 MB/s. As you understand speed between clients and mediaserver isn't enough. You have to increase a  LAN speed  for example by adding new NICs (Adding Quad Ethernet card is a cheapest way) or use SAN-client/SAN-media server  (it's more expensive).

P.S. Have you check your Pramble size?

[Mike_Gavrilov]

Make a simple test  - transfer 1G file via ftp/sftp, NFS. check practical speed. I never see 1G LAN speed more then 65 MB/sec.

[Mike_Gavrilov]

The simples way to increase your perforance is sit down, draw your topology, mark every link (LAN/SAN) and write it's theoretically  speed. Most bottlenecks  will be filtered on this stage. Next step - begin test every link to understand is it new  bottlenecks? What is the way to make it better?
If you want us to do it draw the picture of your environment, mark the connection speeds and we'll look at it.
Now we are just cheating and do nothing :)

[sql_noob]

with LTO-4 it should be an achievable goal, you'll need to maximize I/O

we run netbackup on a Proliant DL380 G5 server. it's i/o is around 3gbps. g6 servers are double that. a cheapo G6 server will run $2500
we use 2 gigabit NIC's teamed together
for the clients teaming Gbps NIC's will also help a lot
we're an HP shop but the tech is pretty much all the same no matter who you buy from. G5 servers have much faster I/O than G4 or earlier generation. upgrading your clients to G6 servers which is what is out now will speed up your backup times.

[ITBOY]

Hi sql-noob,
Thx for your kind advise. I'm currently using the SUN FireX4150 SVR with quad core memory and 8GB or RAM. According to the Design the server spec has been approvd and i can't change unless i upgrade the memory . But I noticed that during the backup/archive time , the CPU and memory utilization is not that high.  I' m not very sure that my server is G5 or G6 . May I know what is the difference ?Will try the "bonding "on the NIC again. It's that significantly improved performance ? I ve tried the jambo frame but not that significantly improved .

[stu52]

"90 MB/s per LTO4 practical"

There's really no need to go to LTO-4 if LTO-3 will do for speed.  LTO-3 speed is 60MB/s native & 120MB/s compressed.

If there is a problem with data coming into the media server then you should NOT go to a faster tape drive, you will only end up starving it for data even more than a slower drive.

The best thing would be to backup to disk and from there to tape.  But, it sounds like there is not much budget for that possibility based on the comments by the OP.

If your media server can only ingest 150MB/s, then it can optimally only drive 2 LTO-3 drives at best.

[ITBOY]

Hi ,
Kindly refer the attached drawing.The performance is as follow.

Date

Client

Data Size ( A/ B )

CPU

CPU Utilization

Memory

Memory Utilization

Memory Free

MTU( NIC / Switch )

ISCSI VLAN

Throughput (Mbps)

Packets(Packet/Sec)

Time Elapsed

Speed

IN / OUT

IN/ OUT

23/11/2009

CLIENT

100.5 GB ( B )

1Quad Core

0.3 us

12306928k

12240324k

66604k

9000 NIC , Cisco 1500

1

7.64/362.27

13.88K /32.08K

41 minutes

2.5G/min

23/11/2009

CLIENT

100.5 GB ( B )

1Quad Core

0.3 us

12306928k

12239256k

67672k

1500 NIC , Cisco 1500

1

7.87/375.34

14.32K/33.28K

41 minutes

23/11/2009

CLIENT

100.5 GB ( B )

1Quad Core

0.3 us

12306928k

12242828k

64100k

1500 NIC , LinkSys1500

1

7.86/362.38

14.31K/32.15K

41 minutes

24/11/2009

CLIENT

100.5 GB ( B )

1Quad Core

0.3 us

12306928k

12240116k

66812k

9000 NIC , Cisco 9000

1

7.87/369.05

14.33K/32.72K

40 minutes

24/11/2009

CLIENT

100.5 GB ( B )

1Quad Core

0.3 us

12306928k

12237020k

69908k

9000 NIC , Cisco9000

2

7.90/369.73

14.37K/32.78K

41 minutes

24/11/2009

NBU

100.5 GB ( B )

1Quad Core

0.3 us

12306928k

12237020k

69908k

9001 NIC , Cisco9000

1

326.12/324.57

50.34k/46.60k

50minutes

24/11/2009

NBU

100.5 GB ( B, RCP)

1Quad Core

03. us

12306928k

12237020k

69908k

9002 NIC , Cisco9000

1

326.12/324.58

50.34k/46.60k

40 minutes

24/11/2009

NBU

100.3 GB ( B, RCP )

1Quad Core

03. us

12306928k

12237020k

69908k

9003 NIC , Cisco9000

1

326.12/324.59

50.34k/46.60k

44 minutes

24/11/2009

NBU

32.1 GB (B)

1Quad Core

03. us

12306928k

12237020k

69908k

9004 NIC , Cisco9000

1

13 minutes

2.5G/min

  RCP =  remote file copy command in Red Hat Linux ENT5.

[ITBOY]

Hi Gavrilov,
I'm sorry for late reply. Kindly advise base on the following info I've provided.