Even after increasing timeout if you find the same issue then check for the below,
on the client run the below command in the command promt and check whether all the VSS writers are stable or not . If you notice any failed components then system state backup will not work . Most of the time this failed components will come stable once after reboot.
"VSSadmin list writers" ---- run this command and check .
one more thing i noticed from your message is even after rebooting the server shadow copy components backup is not working . For this verify as below,
check for the sanpshot method, whether you are using VSS or VSP for snapshot . This you can see in master server host properties in 'client attributes' tab , if you are using VSP then please change it to VSS. This is the recommended snapshot menthod for win 2003 machines.
Test the NTbackup i.e. native windows backup for the system state and verify wheter you can able to backup over there or not . If its working fine then check for below
Then on the client server check for the shadow copy space , it may be limited to 300 MB or less increase it to 2000 MB or more .
