Forum Discussion

mohanl's avatar
mohanl
Level 3
13 years ago

client with millions of small files

A client with > 2 million files of average size only 370KB each, is taking 24 hours to backup during weekend full job. Is this becuase each file must be opened and closed? Is yes, how to overcome this problem?

12.x and Earlier
Backup and Recovery
Backup Exec
  • BanksyMJ's avatar
    BanksyMJ
    13 years ago

    Hi ZeRoCOOL

    When backing up millions of small files the only quick way of backing them up is to use a form of Snapshot. Symantec provide a couple of solutions that can perform snapshot backups:

    1. NetBackup - This is an backup product aimed at the Enterprise Market. It includes the functionality to perform Snapshot backups both to disk and tape...

    2...if NetBackup is to 'big' a solution then you could look at Symantec System Recovery Server (SSR). SSR ignores the files on a server and just backs up at a block level. The beauty though is that you can still restore individual files if required. The downside is it only supports backups to disk.

    Hope that helps

  • ZeRoC00L's avatar
    ZeRoC00L
    Level 6
    13 years ago

    Yes, small files are slow to backup.

    Depending on your environment you can create another backup.
    For example:

    - if the data is on a NAS (like EMC or NetApp), you can look into the NDMP option.

    - if the data is on a virtual machine, you can look into the Vmware agent to directly backup the VMDK.

  • AmolB's avatar
    AmolB
    Moderator
    13 years ago

    As per  http://www.symantec.com/docs/TECH49521

    The total number of files on a disk and the relative size of each file impacts backup performance.

    Fastest backups occur when the disk contains fewer large size files. Slowest backups occur when

    the disk contains thousands of small files. A large number of files located in the same directory

    path back up more efficiently than backing them up from multiple directory locations

    Also refer to BE2012 Performance Tunning Guide, quoted on Pg# 39

    http://www.symantec.com/docs/DOC5481

    Although the article is for BE2012 but the concept is same for all the versions of BE.

     

     

  • Sush---'s avatar
    Sush---
    Level 6
    13 years ago

    Hello Mohan,

    Check the following 

     

    6. Size and Number of Files

    The total number of files on a disk and the relative size of each file can either speed up backup or slow it down. The fastest backups occur when the disk contains a few large size files. The slowest backups occur when the disk contains thousands of small files. A large number of files located in the same directory path will back up more efficiently compared to backing them up from multiple locations.

     

    This is from the technote  " http://www.symantec.com/docs/TECH8326 " which is about 

    Reasons why the data throughput rate can be slower than the theoretical maximum when backing up to or restoring from tape media or disk (B2D) & how to troubleshoot or improve backup performance

     

    Thanks,

    -Sush...

     
  • BanksyMJ's avatar
    BanksyMJ
    Level 4
    13 years ago

    Hi ZeRoCOOL

    When backing up millions of small files the only quick way of backing them up is to use a form of Snapshot. Symantec provide a couple of solutions that can perform snapshot backups:

    1. NetBackup - This is an backup product aimed at the Enterprise Market. It includes the functionality to perform Snapshot backups both to disk and tape...

    2...if NetBackup is to 'big' a solution then you could look at Symantec System Recovery Server (SSR). SSR ignores the files on a server and just backs up at a block level. The beauty though is that you can still restore individual files if required. The downside is it only supports backups to disk.

    Hope that helps