Your device speed will drop off with that number of files and their size. It is because the drive doesn't spin at a constant speed for any sizeable amount of time.
Why not use NTbackup to backup those files nightly, and exclude the folder in question in Backup Exec. You can then put the *.bkf file into a folder and back that up instead.
That way you'll have a folder containing a solid file that's a lot larger in size.