Sharepoint Incremental backup finishing partially Status 1.

[Spartacus81]

Hello again guys,

Master W2K8 7.5.0.7

Media Linux SuSe 11 SP2 7.5.0.7

FEWS and BESQL Windows 2008 Standard R2

I have got everything working hunky dory till last week and now sharepoint incremental backups are finishing partially. 

Full, GRT and VMware Sharepoint backups all working fine. It's just the incrementals that reports status 1. I have checked bpbkar logs, bpresolvers logs on both FEWS and Backend SQL but couldn't find anything.. "getting old maybe" 

16/03/2015 11:34:57 - Warning bpbrm(pid=6103) from client mtsp02.ix.com: WRN - can't open object: Microsoft SharePoint Resources:\Services\State Service\Services-DB 1 (MTDB01/SQLDB/StateService_e2221bd916184a3e8817f10a9480c6c2)\StateService_e2221bd916184a3e8817f10a9480c6c2 (BEDS 0xE000FEA9: The Backup Exec data store encountered a problem during the operation. See the job log for details.)

any ideas?

[Marianne]

So, if nothing has changed in NBU, speak to Sharepoint admin and SQL dba. Look for errors in Event Viewer Application log on SQL server.

[Spartacus81]

Nothing prominent coming up in the event viewer that could lead to the root cause.. 

I have just ran a full and inc back to back and no partials. two hours later ran another inc and partial again?

any thoughts? thanks marianne..

[Marianne]

I would look on the SQL server for errors in Event Viewer logs.

I remember while troubleshooting some time ago that the errors found in Application log was more useful than NBU logs.

Sorry - no more ideas... Hopefully someone else does.
Or Symantec Support...

[Spartacus81]

Right enabled the full set of loggings including the database so here are the findings:

bpbkar log on SQL

16:22:16.062: [3264.4824] <2> ov_log::V_GlobalLogEx: WRN - sharepoint_v2_access::V_OpenForRead() FS_EXTENDED_ERROR on FS_OpenObj() Result:0xE0000363 Action:0x2 Error_id:0x2D4A Error_String:WSS_Search_MTSP02 
16:22:16.062: [3264.4824] <2> tar_base::V_vTarMsgW: WRN - can't open object: Microsoft SharePoint Resources:\SharePoint Foundation Help Search\Search instance\Search-DB 1 (MTDB01/SQLDB/WSS_Search_MTSP02)\WSS_Search_MTSP02 (BEDS 0xE000FEA9: The Backup Exec data store encountered a problem during the operation. See the job log for details.
)

Event Viewer log on SQL

SQL Server Scheduled Job 'Topn past day VIM_VCDB' (0x2F666A5EC636B04286113EA4E10B7F16) - Status: Failed - Invoked on: 2015-03-17 13:25:00 - Message: The job failed.  The Job was invoked by Schedule 81 (10 minutes schedule for topn past day).  The last step to run was step 1 (Control topn past day).

this particular error was coming up in SQL every 10mins am not sure what it does, but when i ran the full and incr back to back, they both ran and completed successfully within 10mins.. moment this SQL task ran again after 10mins incremental started to report partials. So assumingly i think this task is the culprit which for whatever reason resets the LSN on SQL databases hence the above warnings. Its just an assumption. 

I have asked customer to disable that SQL task (underlined above) so that i could test the backups. more updates to follow.

but with regards to the bpbkar log error, any thoughts on it marianne?

[Spartacus81]

Running daily full for now until further updates...

[Will_Restore]

VIM_VCDB sounds like VMware database.  That said, the task must be touching some SharePoint database(s) too.  Look for 0xE0000363 in your database server bpbkar log.

[Spartacus81]

Yap Will_Restore... it's there as i shared above..

16:22:16.062: [3264.4824] <2> ov_log::V_GlobalLogEx: WRN - sharepoint_v2_access::V_OpenForRead() FS_EXTENDED_ERROR on FS_OpenObj() Result:0xE0000363 Action:0x2 Error_id:0x2D4A Error_String:WSS_Search_MTSP02 
16:22:16.062: [3264.4824] <2> tar_base::V_vTarMsgW: WRN - can't open object: Microsoft SharePoint Resources:\SharePoint Foundation Help Search\Search instance\Search-DB 1 (MTDB01/SQLDB/WSS_Search_MTSP02)\WSS_Search_MTSP02 (BEDS 0xE000FEA9: The Backup Exec data store encountered a problem during the operation. See the job log for details.
)

I am running full for now, i will try disabling the task once my customer have confirmed and will share the result. There are number of VIM_VCDB tasks scheduled in SQL and interestingly none of them ever worked so not sure which one is touching the sharepoint databases... 

[Will_Restore · Accepted solution]

0xE0000363 basically says some other backup process touched the SharePoint database so Netbackup cannot perform an incremental backup.  Full backups complete successfully as you note. 

[Spartacus81]

Will_Restore i agree.... 

[punbs]

Please check if any transaction logs are getting tranckated from the SQL DB level for this database.