Restore of files slows Windows server to a halt for users

Hello.  I have a client who does restores of files almost daily.  He is just restoring files (usually pictures, cad drawings, etc).  When he performs this restore, it runs fine, but basically brings the server itself to a halt.  Users cannot access files on the server while the restore is running.  When it is done, the server comes back to life.  He is running the job at medium priority.  Here are the basic specs of the server/drive:

 

Windows Server 2003

AD Domain Controller

HP ML350 G5

Intel Dual Core w/ 3GB of RAM

Backup Exec 11d

HP LTO Ultrium 400/800GB tape drive

Backs up about 450GB per night (all from the one server)

About 30 users

 

Any thoughts???