Media server hardware replacement leads to slower tape throughput

Are you able to share the backup policy config?

bppllist AUTO_ORA_APG -U

Nicolai, sdo, and I have given you some things to check, have you had a chance to check things like the new media server's connection to the network?  Since it sounds like you can back up the media server at a decent speed, there is nothing with the tape path that should be causing this.  So the logical place to look is your network infrastructure between your clients and the media server.

If you used to get 57MB/sec but you are now only getting 5MB/sec, that looks like something in your network path is misconfigured, likely at the 10G connection on your switch.  The difference is about 10:1, which is what would happen if the switch's port was defaulting to 100Mbit/sec.

Check the switch's port settings and make sure it is HARD CODED to use 10G, not AUTO, not 10/100, not 1GB.  Please let us know what you find.

1) You say the original server used 4 x 1 Gb bonding - but can I ask exactly what bonding type?

2) And you say new server is using 10Gb NIC - just the one?  Or bonded too?  If so, exactly which type of bonding?  And if bonded - on the same switch or across physically deiverse LAN switches - or across "stacked" LAN switches ("stacked" = "separate but not really separate") ?

3) Are any of the servers (master, media, or client) in this scenario virtualised ?  And did the media server change virtualise from physical to virtual?

Bonding on the old server was on the same switch as the new server and consisted of a port channel made up of 4 1GB links.  The new media server is using a single 10GB witthout bonding.  No virtualizing is involved on any clients or media server.  All servers are physical.

Your first email indicated: "many, if not all, backups are throughputting at 7 MB/s, which were previously at 30 MB/s".  So do you have some backups going through the new RHEL media server that are running just as fast as they did on the old SunOS media server?  Or are all backups that are going through the new RHEL media server slow?

I'm wondering...maybe there is a hard-coded IP (like in the "hosts" file) or a network path that is routing some of the backups through the wrong ethernet interface on the media server, but other clients maybe don't have that same configuration and are coming in through a different interface.  Can you check the output of the "ifconfig -a" command?  Is there more than one "eth" connection defined on the new media server?  And could one of them be a 100Mbit connection?

Yes some clients are running just as fast as before on the new RHEL media server.

There is another NIC on the new media server and this has a new IP and no backups are routed there.  Tcpdump on that NIC shows no traffic from backup clients.

Yes, I did some checks with network admin and looked at switch configs and the new media server is connected to a 10GB port, and lokking at backup speeds from several clients (before and after the new media server) and it does represent many parameters and data to analyse.  I can't seem to find anything striking execpt that the new media server seems to perform less than before with slower clients.

For example, two bad clients that used to perform 10X give the following iperf results

Client A:    139 Mbits/sec,  299 Mbits/sec,  486 Mbits/sec

Client B     795 Mbits/sec  199 Mbits/sec  146 Mbits/sec

Client C, which has not changed in speed since the upgrade gives:

Client C      610 Mbits/sec  846 Mbits/sec  844 Mbits/sec

 Clients A and B have become slow and they are on the same switch.  C is on another switch.   Perfaps  some interswitch link is saturating now that the new media server has a 10GB inetrface.

1) In this new configuration is it always the same selections/parts/databases of the same clients which now always run slow - or is the slowness intermittent and so appears to hop around different backup policies and/or different policy types and/or different backup clients?

2) Your iperf stats look quite variable.  Is it worth running an iperf test a couple of times a day, over a period of several days, and recording the numbers?  Any patterns emerge?

Only specific clients appear to be slower than before, independent of the policy type (filesystem or oracle database) and of storage unit ( we have both LTO4 tape and disk (Data Domain), and in all cases, backup jobs on thse clients always converge to a 5-7 MB/s.  Sometimes, but not always, backup speeds are inititally high, say 40MB/s, but quickly (within minutes) they continuously slow down to 7 MB/s.  However, occsionally, some jobs on these slow clients are eratically fast, as shown in logs above which show two jobs fromm the same client occuring 17 minutes apart on the same storage unit, tape dive, and tape: one at 52MB/s and the other at 3 MB/s, each several GBytes in size.

Thanks Jean.

Now looking for any correletions...

1) Are the clients which go slow... of the same NetBackup Client version level, or perhaps a sub-set of version levels/range, or a complete mix of different NetBackup Client versions?

2) Similar question, but this time re OS families... i.e. are all clients that go slow of the same OS version and patch level, or even are they all of the same OS family (e.g. Windows, RHEL, Solaris...) ?

3) And similar question yet again... are all the clients that go slow off the same hardware type?  And by this I guess I mean do they all use the same NIC model, and same NIC firmware and same (notwithstanding OS family differences) NIC driver revision?

4) Do any of the clients which go slow still have 'jumbo' frames left enabled (from another exercise in the past) ?

5) I know you've said before that the clients that go slow are spread across different LAN switches, but are all of the clients which go slow all on the same LAN switch 'blade' or LAN switch 'stack module'?  By this I mean are all of the clients which slow on just a few blades or modules - or are the clients which go slow spread all over the physical LAN infrastructure?

6) Are all of the clients that go slow in the same building/site/room but a different site/building/room to the NetBackup servers?

Tricky questions I know - but might be worth being able to answer.