Backup Exec 12.5 intermittantly hangs on back ups on Snapshot Processing

[RobbieTheK]

2 jobs are running, set to go off 3 hours apart. 1 goes to tape, 1 to external USB. Currently the tape job is stuck on Snapshot Processing while the USB is stuck at 201,021,799,273 bytes. Running BE 12.5 rev 2213, all updates installed on Windows 2003 SP2, A reboot of the server usually fixes it. I restarted the BE services with ServicesMgr.exe and same thing happens. No media alerts to remove a tape (as per this thread). Tape driver is HP LTO Ultrium-2 drive, file version 1.0.6.3, this is a network and tape backup running 3 hours apart.

I made sure to push down all of the latest updates to the agent on the respective servers. After reboot the jobs ran ok twice. Both jobs seem to get stuck at the same time but this time for a different reason (last time jobs stopped at the exact same byte count). I scoured the Windows event logs for anything related to VSS but nothing pops up Here are some warnings/error, hoping for some clues here:

1)      The shadow copies of volume D: were deleted because the shadow copy storage could not grow in time.  Consider reducing the IO load on the system or choose a shadow copy storage volume that is not being shadow copied.

2)      Scope, 10.0.225.0, is 85 percent full with only 14 IP addresses remaining.

3)      Could not scan 1 files inside D:\Symantec Endpoint\Symantec_Endpoint_Protection_SBE_12.1_RU1_MP1_Part1_Installation_Software_EN\SEPM\Packages\SAVLegacy32.dat due to extraction errors encountered by the Decomposer Engines. Application has encountered an error. For more information, please go to: http://www.symantec.com/techsupp/servlet/ProductMessages?product=SAVCORP&version=12.1.2015.2015.seps...

4)      The DNS server has encountered numerous run-time events. To determine the initial cause of these run-time events, examine the DNS server event log entries that precede this event. To prevent the DNS server from filling the event log too quickly, subsequent events with Event IDs higher than 3000 will be suppressed until events are no longer being generated at a high rate.

These was a suggestion from another thread about the 2 jobs perhaps hitting the same file and locking up. Anything else?

[lmosla]

Hi, make sure and exclude all the following from any virus scans on the media server:

C:\Program Files\Symantec\Backup Exec\beremote.exe
C:\Program Files\Symantec\Backup Exec\beserver.exe
C:\Program Files\Symantec\Backup Exec\bengine.exe
C:\Program Files\Symantec\Backup Exec\benetns.exe
C:\Program Files\Symantec\Backup Exec\pvlsvr.exe
C:\Program Files\Symantec\Backup Exec\BkUpexec.exe

and on the remote servers make an exlusion for:
C:\Program Files\Symantec\Backup Exec\beremote.exe

Take a look at the following link and though refers to slow performance it is a good reference for making exclusions

[RobbieTheK]

The SEPM interface looks a little different we have 12.1.2015.2015 and there's a separate Exceptions tab. But this seems to be for slow backups not ones that stall.

[CraigV]

...open a command prompt on the affected server and type: vssadmin list writers. Do this when the job stalls...all writers must be stable and running.

If they're not, the reboot does fix them until the next time it happens.

If the server OS is Windows Server 2003, do a search for the latest VSS Rollup Patch. Download and run it on that server, and check the backups again.

Thanks!

[RobbieTheK]

OK I ran the command suggested here are the results. I also just installed the VSS rollup, rebooting over night hope it all helps.

vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool

Waiting for responses.
These may be delayed if a shadow copy is being prepared.

Writer name: 'FSRM Writer'
   Writer Id: {12ce4370-5bb7-4c58-a76a-e5d5097e3674}
   Writer Instance Id: {a5f961b4-c9a1-41a4-a4f4-519a2f0163eb}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {9dc6b453-5375-41e9-a120-bedec4044664}
   State: [1] Stable
   Last error: No error

Writer name: 'FRS Writer'
   Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
   Writer Instance Id: {ade18965-d21a-4e60-9aa4-1beb743f0bc0}
   State: [1] Stable
   Last error: No error

Writer name: 'MSDEWriter'
   Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
   Writer Instance Id: {c1f2c233-5008-443b-bea9-abd4774c90ed}
   State: [1] Stable
   Last error: No error

Writer name: 'Event Log Writer'
   Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
   Writer Instance Id: {1b4b454f-972f-4b2a-9b97-74894ee58b78}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {b89ede5e-f8ee-48ff-a9a6-56cda6cedd23}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {360b87a0-62ff-40e5-8ec1-6892fd249d7c}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {48ea3033-532f-4867-bb57-f3a77750f441}
   State: [1] Stable
   Last error: No error

Writer name: 'WINS Jet Writer'
   Writer Id: {f08c1483-8407-4a26-8c26-6c267a629741}
   Writer Instance Id: {b08a51cd-0fbc-48fd-a55e-0ec3370d765c}
   State: [1] Stable
   Last error: No error

Writer name: 'Dhcp Jet Writer'
   Writer Id: {be9ac81e-3619-421f-920f-4c6fea9e93ad}
   Writer Instance Id: {2c261d51-5e68-408d-a51a-111bee054e3a}
   State: [1] Stable
   Last error: No error

Writer name: 'NTDS'
   Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
   Writer Instance Id: {6978b2c3-8481-4c72-a70f-1256a8eda912}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {cad82874-7c4e-4129-b7a6-2120ccd0a92d}
   State: [1] Stable
   Last error: No error