Deduplication - PDVFS errors

[TechyMr_B]

Hi All,

I'm struggling with an issue that has occured with deduplication storage (or thats what it appears to be to me) on one of our managed media servers.  All of a sudden most of the jobs  that are run on this server are not completing.  They start and go into the "Active: Running - Backup" state, but the average job rate for 75% of the jobs is only 1-2 MB/min or just stops at 0 MB/min.

I have checked the event logs today and can see that PDVFS errors have appeared as well.  Here is what we are getting (oldest first):

Log Name: Application
Source: PDVFS
Date: 15/08/2016 22:00:53
Event ID: 50000
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: RESBE03.RES.TSS.NHS.UK
Description:
An error occurred in the PDVFS driver: <ERR> PdvfsOpen: O_CREAT not set

Log Name: Application
Source: PDVFS
Date: 15/08/2016 22:00:53
Event ID: 50000
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: RESBE03.RES.TSS.NHS.UK
Description:
An error occurred in the PDVFS driver: <ERR> PdvfsOpen: failed for "/RESBE03/2/BEData/_BE101_.TMP": No such file or directory (2)

Log Name: Application
Source: PDVFS
Date: 15/08/2016 22:12:23
Event ID: 50000
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: RESBE03.RES.TSS.NHS.UK
Description:
An error occurred in the PDVFS driver: <ERR> PdvfsPread: pdvfs_io_pread_fake failed: No such file or directory (2)

Log Name: Application
Source: PDVFS
Date: 15/08/2016 22:39:17
Event ID: 50000
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: RESBE03.RES.TSS.NHS.UK
Description:
An error occurred in the PDVFS driver: <ERR> PdvfsPread: fd->dep or fp->dep->stat NULL

I have tried accessing the pdvfs file path and it is present, I have been able to navigate (very slowly) to the  directory mentioned in the alerts, \\.pdvfs\RESBE03\2\BEdata, I'm still waiting for it to open to see if the mentioned files are present as well.

While I'm confident this is pointing to a storage error, I',m confused as to he extent of the issue as we are still seing jobs completing without any issues??

The server itself is running Backup exec 2015 FP4, Windows 2012 R2.  Dedupe storage is a VNX5500.

Thanks in advance for any insight anyone can provide on this.

P.S. Other media servers we are running are not affected by this issue.

[CraigV]

When it comes to a dedupe folder I'd recommend opening up a support call with Veritas to get that resolved. If they do help you find a solution, then post back here and close it off. Thanks!

[TechyMr_B]

Hi Craig,

Thanks for the response to my post, I submitted a support case with Veritas along side posting this thread. I included some of the dedupe log files so just waiting on a response now.

I was hoping on the off-chance that someone may have experienced something similar and could get a few pointers on possible cause while I'm waiting on feedback.  But knowing our system it's ended up breaking in an all-new and imaginative way!

[TechyMr_B]

Well so far support have tried the following:

• Clearing the delstat_* files from the dedupe store.
• Recreating the affected backup jobs.

Still no joy on getting this sorted.  The frustarting thing is it is not affecting all jobs.  Some run while others hang part way through, this appears to be prevelant with VMWare backups.  We back up groups of machines at a time. The job appears to start ok, then will stall on no machine in particular (it takes the snapshot then sits there doing nothing!).  Then you have to reboot the entire managed media server just to get cancel the job!!!

This is becoming very frustrating, while I'm getting feedback from support we don't appear to be making any headway.  Why is it that Backup Exec has turned into a Jenga simulator, one false move and it all comes tumbling down...

[TechyMr_B]

Another update after the weekend.  The Recreated job worked after disabling GRT, however the subsequent run of the same job (along with a few other scheduled jobs) failed/hung/crashed due to a generic DLL error that took out the beremote service:

Faulting application name: beremote.exe, version: 14.2.1180.3142, time stamp: 0x5797d1b0

Faulting module name: MSVCR120.dll, version: 12.0.21005.1, time stamp: 0x524f83ff

Exception code: 0xc0000409

Fault offset: 0x0000000000074a30

Faulting process id: 0xed8

Faulting application start time: 0x01d204f76e5b5a2d

Faulting application path: C:\Program Files\Symantec\Backup Exec\beremote.exe

Faulting module path: C:\Windows\SYSTEM32\MSVCR120.dll

Report Id: 634c152b-72ed-11e6-80e3-3863bb44a45b

Faulting package full name:

Faulting package-relative application ID:

I have ran Windows Update to ensure that the operating system is up to date and ran a repair on the c++ 2012 redistributable package (all sources pointed to the problem DLL belonging to this).  Support say as a result the call will be passed to a core engineer.

While I'm waiting on a call I'm re-running one of the failed jobs to see what happens.

[TechyMr_B]

....AAAAnnnnddddd even more fun. 

Now it looks like the PDVFS service is throwing another tantrum, seen numerous errors this morning stating that it cannot access the drive and all running jobs have hung.  Had to restart the server again to resolve a stuck service (beremote playing games again) and now it appears to be running a few backups again.  Not holding out much hope that it will remain stable however.

Still waiting on support to get back to me with the two cases we have open for this server now.  Fingers crossed.

[TechyMr_B]

Fantastic, Backup Exec has fallen over once again for no good reason.  It is no longer processing jobs on two media servers,  jobs are saying running but the byte count stays at zero.  

Fortunately, we are already taking steps to move off this awful "solution" (using the term very losely here).  Anyone else who wants to use Backup Exec 2015 who reads this, do yourself a favour, save your sanity and don't touch it.