I had a similar issue yesterday. I have just installed BE 11d build 7170, fully patched on a client running W2k3 SBS sp1, but with that "other" product Arc**rve on it. The existing backup is part of a suite from CA that has caused all sorts of issues that even CA had major problems working around (don't know if it is all working yet 2 years later....) so I don't want to remove it until I can get BE working. Stopping all services seems to be ok there. When I try to set up a selection list if I select system state or shadow copy components the selection list hangs. Also if I try to test the resource credentials it hangs. The only way out is to kill BE.
The suite from CA includes pest patrol and AV, but I don't think they are causing the issues - I killed the pest patrol service and it still happened.
Any ideas please? The client is getting desperate to get rid of the Arc**rve software
Thanks