cancel
Showing results for 
Search instead for 
Did you mean: 

Database Management job brings disk folders offline

RichST71
Level 2

I have just upgraded from Backup Exec from 2010 to 2012 and have been finding my backup to disk folders have been going offline for no apparent reason.  On investigation, I am seeing that there are event logs that show the Virtual Disk service restarting at 4AM, which coincides with time the disks go offline.

It looks like the Backup Exec Database Maintenance routine runs at 4AM - I am guessing that this is somehow causing the problem.  I cannot see how to modify the Maintenance routine, or why it would stop the Virtual Disk service.

Has anyone come across this before and if so, is there a resolution to it?

 

Thanks

Richard

1 ACCEPTED SOLUTION

Accepted Solutions

Cullen
Not applicable

It turns out it is not the database management job after all. Running the database job at any other time does not cause the drives to go offline, but the drives still go offline at 4AM. It took me about 5 hours to figure this one out. First, some info about my setup:

2 identical Dell 2950 servers, Server 2008R2 (64-Bit), Backup Exec 2012 SP1a, 2 Identical Dell MD300 arrays, directly attached. The drives go offline on one server, but not the other

The drives go offline 5 seconds after the database job ends at 4AM (04:00:11). It is usually the same drive that fails.

The log files in C:\Program Files\Symantec\Backup Exec\Logs that are created when the management job runs or the services restart have a time stamp that matches the failure. The only OFFLINE error message is in the InitVDS log file, so it is the Virtual Disk Service Initialization job that causes the failure.

The only difference between the two arrays I have is that the "Access" Virtual disk that gives access to the host group was using LUN 0, and not LUN 31 as specified by Dell, and was using the same LUN as one of the virtual disks. The backup servers are not live yet so I removed all of the virtual disks, hosts and host groups from the array and configured it again using the correct LUN for the "Access" disk. The drive no longer goes offline when the services are restarted. I believe the InitVDS job saw the access disk at LUN 0 and tried to initialize it. Now that it is not the first LUN, the job does not cause an error.

Symantec needs to patch that discovery procedure to deal with the possibility that the access partition will be scanned and not crash if it is.

I hope this helps you all and that this truly is the answer. It worked for me. I have a Case open with Symantec and I will pass this info on to them and hopefully they will put out a KB article on it.

Maybe they will send me a check for the time I spent troubleshooting their product. Now they don't have to spend the time doing it! :)

We will see if it goes offline at 4AM tomorrow morning. Cross your fingers....

Good Luck!

View solution in original post

18 REPLIES 18

CraigV
Moderator
Moderator
Partner    VIP    Accredited

Hi Richard,

 

Open up BE and go to Configuration & Settings (hit the BE button within BE) --> Backup Exec Settings.

Click Database Maintenance --> Perform database maintenance at...

Change it to a time more suitable to allow your folders to stay online.

Thanks!

RichST71
Level 2

Thanks Craig

I will investigate this.  I was to be able to monitor the disks for going offline and briung back online as and when it detects this.  Looking at the powershell cmdlets, I don't see a way to bring storage back online.  Do ytou know if this is possible?

Ultimately it would be better to find out why it keeps going offline and fix this, but right now I am after a workaround to bring the disks back online.

Thanks again for your response.

Regards

Rich

MDCnet
Level 3

I am experiencing the exact same issue with iSCSI volumes.    Even disabling the database maintenance makes no difference.   Symantec when will a hotfix be released to resolve this problem?   Note I have applied BE 2012 SP1 already

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

 

We are aware of an issue where B2D Locations on SAN volumes (particularly where multipath might be involved) go offline during DB maintenance or service restart. I suspect your condition might be related. 

With regards the B2D issue, for some reason the TECH article for this issue is not yet public and it is currently still early in the investigations. If anyone does contact tech support formally about this issue then please quote internal TECH article TECH189931. Current workaround does appear to be change the DB maintenenace to a time when no jobs are running.

MDCnet
Level 3

Thanks Colin, if I log a support call will they be able to provide a hotfix yet or is it still being developed?

JamesBarber
Not applicable

Colin,

Do you have any kind of timescales as to when a fix for this may be available? Both SP1 and 1a made no difference to this issue for us and when I logged a call through technical support they were not aware of any issues and went through a whole range of things to attempt to fix it blaming backup jobs etc

We are going to have to revert back to using 2010 as we have been unable to perform backups using 2012 reliably using 2012.

Thanks

James

U645893
Level 3

I am experiencing this same problem (and many others) with Backup Exec 2012 SP1a. My disk based storage, a Data Domain DD140, goes off-line about once or twice a day. This only started happening after I upgraded the DD140 to the latest OS version, as I was instructed to do by Symantec in an attempt to solve other major problems. When this happens, all I have to do is go set it to on-line, no other measures are required.  However, any jobs directed to this disk will obviously fail unless I intervene.  I also have a Backup Exec 2010 R3 media server, using the same DD140, and it never experiences this issue.  I have even had a management session open to the DD140, and had Backup Exec 2012 open at the very instant that it went off-line, and there was nothing else going on, no database maintenance, no backup jobs running, nothing, and it just mysteriously goes off-line as far as Backup Exec 2012 is concerned, but is functioning properly in ever other way.  I have deleted the Changer.cfg and Folder.cfg files and ran an inventory, no problem.  Then at any random time, it will just go off-line.

Any suggestions from anyone is most appreciated.

Thanks,

Todd

HNeumann
Level 4

Hello at all,

plaese open a call with Symantec. My Case was fixed by an orphan fix which was sent from Symantec. After applying the hotfix the error is gone. Refer to my case number. I think Symantec can help now. The case numerb is Case 418-444-078.

Regards HNeumann

frengo
Not applicable
Partner

Thank you very much, this is what forums are for...

U645893
Level 3

I did open a case.  It took me almost an hour to persuade the technician to review Case 418-444-078, as recommended above.  Once he did, he said I had to be transferred to an advance level technician, and I would get a call back.  Later that day, I got a call back, after another long conversation, I was told that my case had to be handled by an advance level technician, and I was right back where I started.  I waited for two days for a call back, nothing.  I called back in this morning, was put on hold for 30 minutes, then the call was disconnected with no call back.  Very frustrating, and this is happening with regularity when dealing with Backup Exec 2012 support.

U645893
Level 3

Symantec did get back to me today, and told me that my issue is unrelated to Case 418-444-078, as my disk based storage goes off-line randomly, and is not associated with database maintenance activity.  Back to square one.

Nicolas_d2b
Level 4
Partner Accredited

Dear All

I have te same issue on SAN disk with 2 storage device, one with deduplication and one with classic backup disk , and tape library on Fiber Channel...

and another custumer with 2 NAS have the same issue.

The auto discovery is checked, but on deduplication disk you don't have this check box

So the solution, disable the database management, but it's not safe....

I hope we gonna have a fix because it's very embarrasing.

Regards

Justin2_0
Level 2
Employee Accredited

Please see the following: http://www.symantec.com/docs/TECH189831

 

If you are experienceing this issue please log a case with support and ask them to investigate your case using:

Source ETrack
Value 2800050
Description

CORE: B2D folders created on SAN drives (SAN with MultiPath) go offline when Backup Exec DB maintenance runs or if the Backup Exec services are restarted.

Please note that disabling DB maintenance won't correct the situation, but changing the time of when it occurs will allow you to controll when goes offline.

U645893
Level 3

My issue is different.  My B2D folder goes off line randomly, at any time, once or twice a day, completely unrelated to DB maintenance jobs, backup jobs, or anything else.

Nicolas_d2b
Level 4
Partner Accredited

Some news....

DB Mainetance scheduled to 8 PM. but Db maintenance disable, the disk gone off line a the time scheduled too...

Magic....

I Open a call now....

Regards

 

Nicolas_d2b
Level 4
Partner Accredited

Thanks Justin 2.0

So the resume about discussion with the support is :

Disable the checkbox "db maintenance" doesn't works (In fack the maintenance still alive), it's gonna be fixed soon by patch.

The Script you send works fine...but i don't know why I could not mark your post as solution ;o(

But Not for Déduplication disk......

Thanks a lot.

Best Regards.

Cullen
Not applicable

It turns out it is not the database management job after all. Running the database job at any other time does not cause the drives to go offline, but the drives still go offline at 4AM. It took me about 5 hours to figure this one out. First, some info about my setup:

2 identical Dell 2950 servers, Server 2008R2 (64-Bit), Backup Exec 2012 SP1a, 2 Identical Dell MD300 arrays, directly attached. The drives go offline on one server, but not the other

The drives go offline 5 seconds after the database job ends at 4AM (04:00:11). It is usually the same drive that fails.

The log files in C:\Program Files\Symantec\Backup Exec\Logs that are created when the management job runs or the services restart have a time stamp that matches the failure. The only OFFLINE error message is in the InitVDS log file, so it is the Virtual Disk Service Initialization job that causes the failure.

The only difference between the two arrays I have is that the "Access" Virtual disk that gives access to the host group was using LUN 0, and not LUN 31 as specified by Dell, and was using the same LUN as one of the virtual disks. The backup servers are not live yet so I removed all of the virtual disks, hosts and host groups from the array and configured it again using the correct LUN for the "Access" disk. The drive no longer goes offline when the services are restarted. I believe the InitVDS job saw the access disk at LUN 0 and tried to initialize it. Now that it is not the first LUN, the job does not cause an error.

Symantec needs to patch that discovery procedure to deal with the possibility that the access partition will be scanned and not crash if it is.

I hope this helps you all and that this truly is the answer. It worked for me. I have a Case open with Symantec and I will pass this info on to them and hopefully they will put out a KB article on it.

Maybe they will send me a check for the time I spent troubleshooting their product. Now they don't have to spend the time doing it! :)

We will see if it goes offline at 4AM tomorrow morning. Cross your fingers....

Good Luck!

strger2012
Level 3

I too have the problem with offline disks at 4AM.

Any patches from Symantec available?