BE15 - B2D Verify hangs indefinitely, E_PVL_MEDIA_NOT_AVAILABLE

I have a specific job that has now become stuck on Job Status  "Active: Queued - Verify" while backing up to Disk a total of 5 seperate times with the same issue, although each has occured unpredictibly, on different days, weeks apart.

The job will hang in Verifty indefinitely until I manually cancel, and then the duplicate-to-tape job runs, but also hangs at the same Byte Count where the Verify job stopped (during the duplicate phase of the tape job, not the verify phase), and I have to manually cancel that as well.  Ultimately this means we do not have a complete backup of this job on tape.

Digging deeper into the Backup 'Job Activity' log:  the Backup phase includes a full set of B2D media names, and the Verify phase stops listing B2D media at about 3/4 of the Backup media list.  (The B2D media names are different for each of the job failures, so it's not an issue with a specific B2Dxxxxxx.bkf file.)

The job errors have occured with at least 2 weeks to a month between them; not consistent. I have also removed and re-presented this specific disk to BE in between the last two job errors, for unrelated reasons.)

Running the Debugger revealed that indeed the B2D file that would come next in the Verify sequence was somehow caught up in the PVLSVR process:

735    PVLSVR    2940    6/2/2016 9:09:04 AM    4448    AdammSession::MountSpecificMedia()
        Requested Media = {F39F96B9-A134-45F2-B819-12C99A497E4D}

736    PVLSVR    2940    6/2/2016 9:09:04 AM    4448    AdammSession::MountSpecificMedia() - currently mounted
        Media = {F39F96B9-A134-45F2-B819-12C99A497E4D}, "B2D321858"

737    PVLSVR    2940    6/2/2016 9:09:04 AM    4448    PvlSession::MountSpecificMedia() - ERROR = 0xE000810E (E_PVL_MEDIA_NOT_AVAILABLE) - 0000 seconds, 0000 (8364) SQL seconds

738    PVLSVR    2940    6/2/2016 9:09:04 AM    4448    AdammSession::Execute( ADAMM_SESSION_EXECUTE_MOVER_MOUNT_SPECIFIC )
        ERROR = 0xE000810E (E_PVL_MEDIA_NOT_AVAILABLE)

So the PVLSVR is unable to mount B2D321858 for some reason, perhaps because it thinks it is *already* mounted?

Looking at the details of B2D321858, there is nothing remarkable.   No read/write errors, 21 seeks, freshly created for the backup job.

The only vaguely related solution item I found was this article (https://www.veritas.com/support/en_US/article.000038222) ... but that discusses Tape media specifically, and there is apparently no option to Retire disk media in the same way.  Regardless, the problem is not related to a specific B2D file, as this job has hung in the same way on different B2D files each time.

And I don't think it's related to the content of the backup job either, because a different file is listed in the Activity Log each time the job has hung.

I'll restart the BE services, then do an Inventory/Catalog on the disk in question, but without any predictibility of the job failing, it is hard to isolate and experiment.  And interestingly, no other job of our 50+ has this problem:  This is backing up a specific directory on a physical server, but the job settings are very standard and similar if not identical to other physical server jobs.

Perhaps this is a database issue?

 

Any insights or suggestions would be most appreciated.

Thank you.

9 Replies
Highlighted

6/9 - Happened again. Of the

6/9 - Happened again. Of the six times this has occured the Wednesday differential job has hung 4 of them. Tues & Thurs once each. Never the full backup on Saturday (thus far).  No obvious correlation as to why it favors Wednesday.

I have scheduled the Verify portion of the Wednesday job to run about 6 hours after the job normally ends. We'll see in about 3-5 weeks if that has made any difference.

Highlighted

Re: BE15 - B2D Verify hangs indefinitely, E_PVL_MEDIA_NOT_AVAILABLE

6/15 - Occurred again, this time with a different job altogether.  Reviewing notes to see if it is the same destination disk.

Highlighted

Re: BE15 - B2D Verify hangs indefinitely, E_PVL_MEDIA_NOT_AVAILABLE

Make sure you exclude the backup to disk folder from any anti-virus scanning and if it still happens log a formal support case - basically no answer since 2nd of June on these community pages, probably means that there is no obvious suggestion as to what could be the cause of your problem and it therefore needs a more in depth look at by technical support

Highlighted

Re: BE15 - B2D Verify hangs indefinitely, E_PVL_MEDIA_NOT_AVAILABLE

Thanks Colin. We definitely don't have antivirus running on our backup server, so that isn't it.   I'll open a support case for this then.  Cheers.

Highlighted

Re: BE15 - B2D Verify hangs indefinitely, E_PVL_MEDIA_NOT_AVAILABLE

Hi,

I know this is an old thread but we are having exactly the problem. Did you ever get it resolved?

 

Highlighted

Re: BE15 - B2D Verify hangs indefinitely, E_PVL_MEDIA_NOT_AVAILABLE

First need to make sure the media where the backup went is present in the storage.
If verify complains about media not found then it could mean that media is not present and hence that msg. Check application and system event viewer, ensure no NTFS errors exist. These issues may happen if there are any disk issues. Chkdsk may need to be run.
Highlighted

Re: BE15 - B2D Verify hangs indefinitely, E_PVL_MEDIA_NOT_AVAILABLE

thanks,

Its a 13Tb SAN storage. I havent seen anything in event viewer related to disk errors. There are no error logs reported by the storage controller. Also I did run a DB repair.

Doing check disk sounds reasonable.

I'll try that.

Highlighted

Re: BE15 - B2D Verify hangs indefinitely, E_PVL_MEDIA_NOT_AVAILABLE

Make sure to check on the bkf media in the storage location as well. Is the media present in first place ? Backup may have used x, y, z media and during verify if the error is received during mount of a media then need to make sure that media is present in storage first.
Highlighted

Re: BE15 - B2D Verify hangs indefinitely, E_PVL_MEDIA_NOT_AVAILABLE

When you run into this error, run an inventory as well. Check if inventory is able to read that z media error