Best way to backup millions of empty folders?

[zebo51]

We have a directory structure that a program creates for a index so to speak.  This has grown to over 2.6 million directories containing around 5.3 million files.  These are also all very small files and tons of folders are even empty.  Total data is only around 1GB.

Both the server running our backups and the server containing the data are hefty, so it is not a performance issue of hardware.  I get a rate of around 27mb/min and takes over 17hrs to complete.  This is only going to continue to grow.  When backing up these servers excluding these certain folders we get around 1650mb/min.

So, does anyone else out there have to deal with this and know of a good way to backup this info?

Any suggestions at all are appreciated.

Thanks

[Peter_Bevington]

I had a similar experience, but with not so many folders/files. The easiest way to resolve this which I found was to complete a full backup as opposed to an incremental, that way you will remove the time it would take to check for file changes which is probably the reason why they're taking so long.

Hope this helps.

[zebo51]

We do a full once a week and diffs the other 4 days, nothing on weekends.  The diffs take just as long.  The way I understand it is your top level folder has no idea if a file or folder below it has changed since a full backup, so your backup still has to check each folder and file for the archive bit.

Thanks though.

[Ken_Putnam]

Any time that BackupExec is not actually backing up data slows down throughput, sometimes drastically (DOH!)

1000 1K files will take a LOT longer to backup than a single 1MB file.  And yes, BackupExec needs to "read" each directory every time it does a backup.

Bottom line - you're screwed as far as throughput goes  .  Even B2D as opposed to tape wouldn't make much difference, since most of the time spent is all "overhead" anyway

[zebo51]

Would setting up and using CPA be any better or would it still take forever?

Thanks