Forum Discussion

nairaj's avatar
nairaj
Level 3
10 years ago

Poor NDMP backup performance while backing up EMC isilon

We have a new EMC 3 node Isilon and we are using (3-way) NDMP. Isilon has multiple 10G network but our backup speed is 3MB/s.

We have NetApps with similar setup and it gives better backup speed.

Please advice if any of you has encountered similar issues.

Thanks.

7.5
Backing Up
Backup and Recovery
Linux
NetBackup
  • nairaj's avatar
    nairaj
    10 years ago

    This is resolved with the help of EMC support.

    Resolution was to change the gateway priority of the multiple 1G network (subnet0) to 2 so that the backup traffic will go through the multiple 10G network.

6 Replies

Sort By
  • mnolan's avatar
    mnolan
    Level 6
    10 years ago

    So in ALL performance troubleshooting you have to look at ALL theorecticals.

    Read Speed - Transfer Speed - Write Speed

     

    If you feel that transfer network and storage speed is ok, I would focus on the read speed (As perhaps your NetApp is a better tiered device.)

    From:

        How to troubleshoot NDMP Backups failures when status code 99 (or other NDMP backup failure) is reported. Includes logging instructions.

    Article: TECH56492

    Updated: February 24, 2015

    Article URL: http://www.symantec.com/docs/TECH56492

    7.  For performance issues, try backing up the path in the policy to a null device and note the timing:

    NetApp:  dump 0f null /vol/volname
    EMC Celerra:  server_archive <datamover name> -w -f /dev/null/ -J <file path>

     

    If the Celerra command is not applicible for the Isilon, please see EMC for the correct command to attempt the dump to null to test local device speeds.

  • elanmbx's avatar
    elanmbx
    Level 6
    10 years ago

    Does the data have to be rehydrated prior to being backed up?  I'd guess it does and perhaps this is contributing to the poor backup performance?

  • a_la_carte's avatar
    a_la_carte
    Level 5
    10 years ago

    NDMP local is the fastest way among 3-way NDMP backup and remote NDMP backup.

    However, seggregating the backup into different backup selection lists, depanding upon the Q-tree structure at the filer end (Guessing that EMC Isilon has the similar "Q-tree" concept as NetApp filers.) helps speeding up the overall backup.

    Rather than giving all the data into a single backup selection list such as /vol/vol0/xxx, if this can be split into multiple q-trees and hence different backup selections, that would definately help a bit such as /vol/vol0/xx, /vol/vol1/yy etc.

    Dump test would be the first step towards troubleshooting slower NDMP backup as discussed by mnolan above.

     

     

     

  • Nicolai's avatar
    Nicolai
    Moderator
    10 years ago

    Let EMC have a look at the issue - they may be able to help you. 3MB/sec is ridiculous low.

    Are you using the Isilon NDMP accelerator appliance ?

    What NDMP environment variables are used - if any ?

    Page 69 : https://www.emc.com/collateral/TechnicalDocument/docu50233.pdf

  • nairaj's avatar
    nairaj
    Level 3
    10 years ago

    This is resolved with the help of EMC support.

    Resolution was to change the gateway priority of the multiple 1G network (subnet0) to 2 so that the backup traffic will go through the multiple 10G network.

  • mnolan's avatar
    mnolan
    Level 6
    10 years ago

    So basically my previous comment at looking at all three areas was pretty spot on?