It is not the size of the backup that matters (in this case), but the number of files involved. If you are doing a "standard backup" with millions of files, then NBU is retrieving a file at a time. If the volume is fragmented, then Windows is having to hunt all over the disk to get that 1 file. Once done, onto the next file, with the same consequences.
If you have a large number of files, you may want to consider the NBU FlashBackup facility. This performs a "raw disk" backup with the ability to do individual file restores. In our case, we have a 1.6TB volume with 4+ million files. Backup time was approximately 72 to 90 hours. With NBU Flashbackup, we are down to 10 hours.