I'd like to chime in with the response that I just received after about a week of working with a support technician.
We're running 10.0d, and also have some sizable drive arrays. While you're just working with ~5 tb, we're working with 13 tb on one box and having the same issue with not being able to complete a snap.
The final result of tests and such, it has come to be known that: "At this time Backup Exec is only able to snap a volume up to around 6 terabytes. Its not a bug, but a known limitation in the shipping product, which means that we do not have a fix at present."
This is a real problem, and will only keep growing since disk space keeps growing. Hopefully it'll be taken care of in a near future release.