Extremely slow backup of one particular Linux client

[Toddman214]

Master and media servers running Windows 2008r2, Netbackup 7.6.0.4

Hello all,

I have a particular Linux client out there that will not back up with any proper performance. I have an open ticket with Symantec, but assistance is very slow on the issue. I've already been asked twice to send in logs once the job completes, and have informed the tech that the job will not complete. The job is about 1tb, running at 6 KB/sec. Not MB/sec.....KB!

Ive run an appnet test out to the client, and sent those results in, but in the meantime, wanted to reach out to the comminuty. The performance is the same (5 to 9KB/sec) regardless of whether I send to tape or disk. For comparison, my Linux boxes back up at about 30MB/sec.

Thank you. I'll provide whatever information is needed.

Todd

[jim_dalton]

Have you run the null test? Ie bpbkar ?

This will determine if the issue is down to reading the filesystem/data or the network.

This will narrow your problem majorly.

If it is slow , then its disk...if its disk, whats the data profile?

How about a synthetic test so that it generates its own data via GEN_DATA directive in policy? That removes the disk from the scenario and tests the rest of the infra. Jim. 

[bbahnmiller · Accepted solution]

We have found, about 99 times out of a 100, that really slow clients usually have some sort of network mis-configuration, such as a speed/duplex issue with the network switch.

We have even seen, in more complicated networks where you use static routes, that you can have a particular route's gateway defined to a wrong gateway. If you had buildings A and B, both with backup networks, with your server and client in building A. Then if you have the client route pointing to a router in B, the backup data will go from A to B and back again. And if the inter-link is bandwidth limited...

Most significantly slower backups are network related. However, the other thing that can cause very slow backups is millions of tiny files, or very deep directory depth. Each of those factors can contribute to slow backups as well.

[revarooo]

Have you copied any files from client to media directly to see what transfer speed you are getting to determine if there is a network bottleneck?

[Nicolai]

Try a iperf test between backup server and client. This should quickly reveal IP problems.

iperf is available for both Windows and Linux/UNIX

http://www.symantec.com/docs/HOWTO64302

[Toddman214]

Are any of these suggestions able to be tested only from the Master/Media server side? Im a backup admin with no access to Linux clients, other than the limited access I have to the client properties via the master gui. Ive been working with our Unix engineers as well, but want to try everything I can from my end.

[sdo]

Yes - Jim's suggestion to use GEN_DATA is a a good suggestion - you do eveythign you need from NetBackup server side.

Be warned though, that if the box is powerful then NetBackup Client will easily fill the LAN/WAN link... so probably best to run such a test outside of core operating hours - when hopefully no one will notice.

[sdo]

Need an example of using GEN_DATA ?

[sdo]

How to use GEN_DATA:

http://www.symantec.com/business/support/index?page=content&id=TECH75213

[Toddman214]

Thank you, sdo. I'll try Jim's suggestion. We have quite a robust network here, but I'll certainly keep an eye on things.

[Toddman214]

This client has millions of tiny files on it, and constant change. Plus, its about to be moved onto other storage anyway, so lets close this one out. Spent way too much time on an issue that is admittedly caused by the team that owns that box. Thanks all, for your recommendations,

[Toddman214]

Problem was located. The switch ports that this server is plugged into is set to auto/auto and is negotiating to 100 and half duplex, instead of 1000 and full duplex. I'll mark bbhanmiller with solution, as thats what it turned out to be. Thanks all.