Puredisk Backups Failing

[DLOW]

Puredisk worked great until I reached about 80% capacity.  Now all of my jobs fail no matter how large are small they are.  I run retention everday and have set retention as low as 1 day.  I still sit at 80% capacity after setting to 1 day and running all DB processes after.  I have changed all the workflows to run multiple times a day (i.e. Data Management, Storage Pool Policies etc...)  Here is my log.  Job fails on Store on Contenet Router.

Ideas? 

d during communication with a remote
system. Please use the following information, examine the system
or program log file to determine the exact cause of this error.

Connection Type : S
Remote Address : 10.15.172.117:10082
Local Address : 10.15.167.16:3178
Session ID : 696723099
Data Selection ID: 187
Event code : 5
Event Description: try again
Server response : data store failed: could not spool object: try again
Error: 5: \\?\C:\Documents and Settings\Default User\Local Settings\Application Data\Microsoft\Dr Watson\drwtsn32.log: could not be stored (try again)
Warning: 42: \\?\C:\Documents and Settings\Default User\Local Settings\Application Data\Microsoft\Dr Watson\drwtsn32.log: failed (aborted). Retrying in 10 seconds
Info: Server is Version 6.5.1.19343, Protocol Version 6.1
Error: 5: data store failed: could not spool object: try again
Error: 5: An unexpected error occurred during communication with a remote
system. Please use the following information, examine the system
or program log file to determine the exact cause of this error.

Connection Type : S
Remote Address : 10.15.172.117:10082
Local Address : 10.15.167.16:3181
Session ID : 2165411120
Data Selection ID: 187
Event code : 5
Event Description: try again
Server response : data store failed: could not spool object: try again
Error: 5: \\?\C:\Documents and Settings\Default User\Local Settings\Application Data\Microsoft\Dr Watson\drwtsn32.log: could not be stored (try again)
Warning: 42: \\?\C:\Documents and Settings\Default User\Local Settings\Application Data\Microsoft\Dr Watson\drwtsn32.log: failed (aborted). Retrying in 15 seconds
Info: Server is Version 6.5.1.19343, Protocol Version 6.1
Error: 5: data store failed: could not spool object: try again
Error: 5: An unexpected error occurred during communication with a remote
system. Please use the following information, examine the system
or program log file to determine the exact cause of this error.

Connection Type : S
Remote Address : 10.15.172.117:10082
Local Address : 10.15.167.16:3183
Session ID : 929171095
Data Selection ID: 187
Event code : 5
Event Description: try again
Server response : data store failed: could not spool object: try again
Error: 5: \\?\C:\Documents and Settings\Default User\Local Settings\Application Data\Microsoft\Dr Watson\drwtsn32.log: could not be stored (try again)
Warning: 42: \\?\C:\Documents and Settings\Default User\Local Settings\Application Data\Microsoft\Dr Watson\drwtsn32.log: failed (aborted). Retrying in 20 seconds
Info: Server is Version 6.5.1.19343, Protocol Version 6.1
Error: 5: data store failed: could not spool object: try again
Error: 5: An unexpected error occurred during communication with a remote
system. Please use the following information, examine the system
or program log file to determine the exact cause of this error.

Connection Type : S
Remote Address : 10.15.172.117:10082
Local Address : 10.15.167.16:3185
Session ID : 3668724610
Data Selection ID: 187
Event code : 5
Event Description: try again
Server response : data store failed: could not spool object: try again
Error: 5: \\?\C:\Documents and Settings\Default User\Local Settings\Application Data\Microsoft\Dr Watson\drwtsn32.log: could not be stored (try again)
Error: 42: \\?\C:\Documents and Settings\Default User\Local Settings\Application Data\Microsoft\Dr Watson\drwtsn32.log: did not succeed after 5 consecutive attempts, giving up (at line 7 in input).
Error: 42: A fatal error was encountered while processing (unknown) (at line 7 in input). Cannot continue.
Info: Backup completed successfully (0 PO processed, 0 bytes)
Info: Duration : 62.94 seconds
Info: Items Stored : 0 (0 bytes)
Info: Items transferred to Storage: 0 (0 bytes)
Info: Average Backup Speed : 0 bytes/sec
Info: Average Transfer Rate : 0
Info: Average Processing Speed : 0.00 PO/sec
Info: Files Stored : 0 of which 0 (0 bytes) new, 0 (0 bytes) changed and 0 (0 bytes) unchanged
Info: Directories Stored : 0
Info: Hard Links Stored : 0
Info: Symbolic Links Stored : 0
Info: Devices Stored : 0
*** Error Message ***

severity: 6
server: 24
source: PutFiles_PutFiles
description:
Command '""C:\PROGRA~1\Symantec\NetBackup PureDisk Agent\\bin\pdbackup" -I "C:\PROGRA~1\Symantec\NetBackup PureDisk Agent\var\srd\187\polist.po" -O "C:\Program Files\Symantec\NetBackup PureDisk Agent\tmp\wf_1CF.tmp" -S "C:\Program Files\Symantec\NetBackup PureDisk Agent\tmp\wf_1D0.tmp" --options compress=0~encrypt=0~servercrypt=0~checkifexists=0~acl=1~noarchivebit=0~segmentsize=131072~preserveaccesstime=1"' unexpectedly terminated with exit status 1
*** End ***

*** Supportability Summary ***
jobid = 59483
jobstepid = 206957
agentid = 24
hostname = DC0301.firm.mycompany.com
starttimejobstep = July 31, 2009, 9:26 pm
endtimejobstep = July 31, 2009, 9:27 pm
workflowstepname = PutFiles
status = ERROR


PureDisk truncated this log file. You can download the complete log file from the Job log tab.

[Abesama]

Are you running data removal policies and MB/CR garbage collection policies?

PureDisk keeps last copy of data even after the retention expires (sounds not right, but true) which means, in extreme situation, you configure it to expire everything after 1 day and PureDisk data still staying there using 90% of your storage.

The only time PureDisk really-really wipes out the last copy of the data off hard disk is when the associated dataselections are all removed, followed by running a dataselection removal policy.

Anyway, if your /Storage and /Storage/data are only 80% full on the content router, then check the metabase engine and it's capacity report, you might be over-running the MBE/CR capacity. (over-running capacity should not necessarily cause the backups to fail, but it's worth a look anyway)

You can also check if spoold is running fine on the content router, and check the spoold log under /Storage/log/spoold.

To get more responses from others, I suggest you explain some more detail about your setup - like, the SPA/MBE/CR nodes, their "df -h" output, capacity report, event log.

Good luck!

Abe

[Douglas_A]

you need to run the Garbage collection jobs.. per Support you have to run it over expired data 4 times before the space becomes available again.