Yes, that is exactly what I was talking about, notice the percentage of number of files over the total size, can impose a huge difference to the backup time. Try to cut down number of files (by placing them in zip files).
If you don't have control over the number of files (since this is Sharepoint backup), usually what I do is inform the site owner/user that their backup is taking too long, and ask them to reduce number of files. They will do so if you let them know how a long backup can impact their access speed and risk of having no backup at all.
^_^