RD1000 Unable to Eject after backups complete

[DNeuwir]

Hello all,

We're having troubles with Dell RD1000 cartridges using Backup Exec 12 for Windows Servers (Small Business Server).  In all 4 cases, this is on systems that had previously been running nightly backups using v11d successfully for months (or even a year) without a hitch.

We can back up properly, but we can never eject the cartridges once a backup has been completed.  We press the "eject" button on the drive, but the cartridge will not eject.  It's totally "locked".

This has been tested and confirmed on four unrelated and relatively dissimilar servers at four different locations.  We have tried this with both the Internal (SATA) RD1000 drive chassis and have also reproduced the issue on one of our servers using the external USB drive chassis.  In all cases, the Dell RD1000 "RDXMon" service (which supposedly aides in ejection) is installed and running fine.  In all 4 cases, these drives had been previously successfully used with Backup Exec 11d, and we had done an upgrade installation to v12.  The only change to the system was an upgrade to BE 12, along with all available LiveUpdates (3 at the time).

One of the servers is a clone (running an Asus server motherboard, external USB RD1000 drive).  The remaining 3 servers are Dell PowerEdge 2900 systems using the Internal SATA chassis.

At this point, as best I can tell, we have a serious regression in the product and the product is completely unusable on RD1000-equipped systems (again across 4 locations, we have not had a successful backup, through the point of cartridge ejection yet).  This is holding off a rollout to our remaining 25 servers (there are about 30 total).

I understand that BE cannot "automatically" eject the cartridge as it can with "true" tapes.  But I'm not talking about that.  I'm referring to being totally unable to manually eject the cartridge (using the eject button on the drive) unless I PHYSICALLY POWER OFF the server and power it back on.

Help anyone?  I've got a string of clients we're waiting to upgrade here and I cannot in good conscience bring them forward until this issue is resolved.

-Dan

[Layla]

I posted the same problem on a different section of the board.  We have a Dell PE840 with RD1000 drive that was working fine on v11 and now will not eject on v12.  I found the same problem reported regarding the Symantec System recovery on their forum.  The solution in that case was a patch from Symantec.  I referenced that in my notes but no one from Symantec has responded.  For now we are following the below steps to keep from having to reboot the server.  These are user level instructions as I had to send them to my client. 

Below are the steps to eject the tape.  For now the best course of action is to wait for a patch from Symantec to fix the problem.  If you want me to walk you through the steps I will be glad to.

Open Services
Locate the RDXmon service, Right click and stop the service
Open Windows Explorer Navigate to the RD1000 drive, right click select Eject
Click Continue on warning messages, sometimes it appear 2 or 3 times
Wait a few seconds,  if it doesn't eject, right click and select Eject again
Change tape
Go back to Services and right click and start the RDXmon service

This is a copy of what I posted in the other section of the forum.

Windows Small Business Server 2003 running Backup Exec which I just upgraded to V12.  Drive is a PowerVault RD1000 in a Dell PE 840.  The drive ejected fine using the button prior to version 12 now it will not eject.  I stop BEWS services and still nothing.  I stop the RDXmon service and try to manually eject and I get a warning that it is still in use, if I continue and ignore the message then it will eject.  This job is finished and nothing else is using that drive.  I found a similar problem listed for Symantec System Recovery v7 which was patched by Symantec in 7.02. Dell has a utility which I have installed and still does not work.

https://forums.symantec.com/syment/board/message?board.id=115&thread.id=18430

My customer is getting very frustrated as this is their first experience with Symantec. Can someone from Symantec tell me if this is being caused by the same problem Symantec Sys Rec had and if so can I get a patch?

[DNeuwir]

While that's an okay workaround for an administrator (me or my techs) that is not going to work for the managers that need to just swap the cartridges every day.  We set these servers up and continually monitor them and can administer them from remote locations (as we're not full-time onsite staff), but ultimately our clients will be the ones that swap the daily cartridges out and those folks won't have admin/console access to log in.

They expect, as they did with BE10d and 11d, to spend 10 seconds pushing the eject button, snapping in the next cartridge, and walking away without ever logging in.  In fact, many of these servers don't even have mice/keyboards because I use RDP/remote logins to manage them, so they couldn't log in locally anyway without bringing a mouse/keyboard over and plugging it in.

Any solution that requires more interaction than pushing the [Eject] button on the drive is not going to work, because any solutions beyond that make the (incorrect) assumption that the person changing the cartridge is a network administrator with console or RDP access to log in to the server.  In the vast majority of cases, the person changing the cartridge is just a non-administrative non-management office worker tasked with the responsibility of "pushing the button" each day.

-Dan

[DNeuwir]

Hello?  Does anyone at Symantec read these forums?

I would have thought with an issue of this magnitude ("product not working, can't complete a single backup on a commonly-used device that's on the Symantec HCL" ) that someone at Symantec would have at least posted a response by now.

We're in the process of starting to revert our four servers that we've rolled out on the product back to version 11d, which worked fine for months and months.  With no support or even response on this issue, we've now moved to telling our clients to hold off on any plans for upgrade to BE12 and to stick with the 11d for now.

Clearly BE 12 is still a beta-quality product and is not quite ready for prime time.  Neither does support appear to be ready for an issue of this gravity.

-Dan

Message Edited by DNeuwir on 04-09-2008 08:28 AM

[Ken_Putnam]

Hello?  Does anyone at Symantec read these forums?

The forums  are primarily Peer-to-Peer.  Symantec employees read/respond as their other duties permit

The only Symantec support you can "count" on is to open a paid incident or to have a maintenance agreement inplace  (if you want to catagorize any Symantec support as something you can count on   )

Message Edited by Ken Putnam on 04-09-2008 10:59 AM

[DNeuwir]

Ahh...  I see. 

So "forums" is another feature provided to us by Symantec so that we can "figure out the problems on our own" rather than have to call the company who caused them in the first place...

Nice "benefit".

By the way (as we have a support and maintenance contract) I called in this morning.  After waiting 45 minutes on hold to talk to (I presume by the accent) someone in India, I had to spend the first 5 minutes trying to explain to him what an RD1000 cartridge was.  This is the support I paid for?  (reiterating again: RD1000 is on the BE12 HCL, so in theory I'd expect their support to have at least a remote clue as to what I'm dealing with).  The best answer he was able to give me (from what I could understand through the thick accent and low-volume) was that I had to start/stop some service to get the cartridge to eject because you're not supposed to be able to "just eject" a removable storage device.

Ummm... worked fine for a year under BE10 and 11, so....?

Tier 2 support will be calling me tomorrow morning.  Perhaps they know what an RD1000 is.

-Dan

[Layla]

Well chancing getting my head ripped off again for trying to be helpful,  we have one client that use the RD1000 that is running v12 without a problem but this was a fresh install not an upgrade.

[DNeuwir]

Layla,

I apologize, if you're referring to my original response to you above I was not trying (nor intending in any way) to "attack" you or your posting.  I was just trying to keep a "factual" tone and saying that while the solution may work, it's not going to be practical for the clients we've set up.  That said, I appreciate your suggestion and I have nothing against you personally...

:)

-Dan

[Layla]

Good Morning Dan,

I was wondering if Tier 2 support ever called to resolve your issue with the RD1000?

Thanks,

Layla

[DNeuwir]

Yes they did, but unfortunately as an IT Tech/Implementer I'm out of the office most of the time and I'm difficult to get a hold of "whenever they happen to call".  To their credit, they always seem to call back, but it's usually 2-3 hours later when I've already left for client appointments for the day.

Calling them back and punching in the case # into their robo-dialer system is useless and never gets me to the tech--always to "your tech isn't answering, so please leave the tech a message and we'll call you back sometime" generic voice mail box.

It just surprises me that despite paying for a support contract along with the product, I've been unsuccessful in getting any support on this issue.  I don't want "support at some pre-arranged time".  I want support when I call, because that's when I'm at a server and I happen to be in the office and have 30-60 minutes available in my already-hectic schedule to look at this issue.  At this point, despite having case numbers and a supposedly-assigned tech rep, we have no solution and we're ready to start evaluating and possibly recommending other companies' backup solutions moving forward.

Abysmal.

Message Edited by DNeuwir on 04-14-2008 08:05 AM

[DNeuwir]

Okay, I have progress on the issue. Not necessarily good news, but progress anyway.

After 90 minutes on the phone this morning, I was told that this is a "known issue" in the product, and is currently in Engineering.  The plan, at current, is to issue a hotfix at some future time (no clues as to lead time).

The current recommended solution is to go back to Backup Exec 11d.

Related articles include:

http://seer.entsupport.symantec.com/docs/301226.htm

The article does not specifically mention the RD1000 eject issue, but the tech assured me that the underlying problem that causes the 301226 issue is identical, and that by resolving 301226 they will also be resolving the "Can't eject RD1000" issue.

It was suggested to me that I "subscribe" to article updates, which will include any notifications of a fix/patch being released.

So for the moment, "yeah, don't use the product" is the best solution I've been given, but at least now there's a glimmer of hope down the road.

-Dan

[Layla]

Dan,

I don't know if you have tried it but they have posted a hotfix for the issue that Tech 2 support said would fix the problem.  I downloaded it today but can't apply it until tomorrow.  Just wanted to see if you have had a chance to check it out and if it is working for you.

Thanks,

Layla

[DNeuwir]

Yes, I have tried it and it does not resolve the issue.  No change to the symptoms.

-Dan

[jtatum]

Hi Dan,

We reviewed this thread and performed testing in-house on this issue.

When a Backup-to-disk folder is configured on the RD1000 (or similar removable device), eject does not work when the backup completes.  When a Removable Backup-to-disk folder is used, eject works when the backup completes.  Eject is only supported by a Removable Backup-to-disk folder - this is as designed.

One thing we noticed is that hard ejects of the media worked all throughout testing once the RDXMon service was installed (and running).  In the absence of this service, then media could be ejected via a soft eject through Windows Explorer.

[Layla]

I'm not trying to have Symantec Backup Exec eject the drive after completion, I understand that this is not supported.  We I have experienced is that after upgrading from v11d to v12 the eject using the eject button on the front of the drive does not work.  I have the Rxmon service running, it does not seem to help. It worked fine in v11d then we upgraded to v12 (Key word here is upgraded). 

For a very temporary fix we have to stop the rxmon service, right click the drive in Explorer and click continue on the messages warning that the drive is still in use.  I have confirmed that nothing else is writing to the drive.  Change the cartridge and restart the service.

I posted that this exact situation was posted on an upgrade to Symantec System Recovery and was fixed in patch 7.02.  This is posted on their section of the board.

[DNeuwir]

Your test results do not reflect the "real world" results we're seeing, and frankly, conflict with what your tech support people are handing out.  I really wish you all would get on the same page and givev me a single, unified answer that works.

Across four different calls, I've been told:

1. Use a "removable backup to disk device".

2. Don't use a "removable backup to disk device, use it as a regular "backup to disk device"

3. I should use the "Eject media when job completes" function in the job setup.

4. Don't use the "eject media when job completes" as that's only designed for tapes, not RD1000's

5. ...

Are you starting to see a pattern here?  I am.  The pattern is that no one at Symantec seems to know what they're talking about, or at least, can't agree on whether a particular option should be "on" or "off".

The ONLY THING I AM INTERESTED IN is being able to walk up to the device (8 hours after a midnight backup), hit the hard "eject" button on the front of the device, and have the RD1000 pop out.  No logging in to the (usually monitor-less) server, no clicking "Eject" in Explorer, no running BE. 

In other words, exactly what BE 10d and 11d did just fine.  That, UNIVERSALLY, is not happening after BE 12 is installed.  WE HAVE NOT YET, IN ANY PRODUCTION SERVER ACROSS 4 DIFFERENT ORGANIZATIONS, EVER BEEN ABLE TO PUSH THE EJECT BUTTON AND HAVE THE CARTRIDGE EJECT.

IN ALL CASES, we have the RDXMon service installed and running--per Dell it is a required service and NO RD1000 MACHINE will work without it, so of course it's installed; it's the first thing we do before even installing BE on a new server.

We have now reverted one of the four servers back to BE11d and like clockwork, the Eject function again works perfectly.

Looks like your lab testing needs a bit of honing.

-Dan

[jtatum]

OK guys, I hear your frustration.  Sorry about the conflicting information you have received and the pain this issue is causing.

We are looking in to this.  I will update this thread as we have more information.

[Joshua_Kane]

Dan:

Jamie has been looking into this for you along with the folks in our Compatibility Lab (cLab)  They have run the following tests and come to the following conclusions.

On Windows 2003,  once the RDX services are installed,  the RD1000 ejects the drive properly (either by pressing the eject button or by using the Eject option in BE).   This was tested on a system running Backup Exec for Windows Servers version 11d and then upgraded to 12.  We then enabled, and subsequently disabled the  Removable Storage Management (RSM )service and encountered no problems.

The same test was then run on a Windows 2008 server, there we encountered different results.  Once we manually installed the Windows Removable Storage Management (RSM) service (not installed by default) the RD1000 stopped ejecting the drive. Even with the RSM service stopped the drive would not eject and the only way to get it to work was to uninstall this service.

Dan, can you please confirm that your system is a Windows 2008 server that has the Removable Storage Management (RSM) service installed?

-- Joshua

Message Edited by Joshua Kane on 04-24-2008 08:37 PM

Message Edited by Joshua Kane on 04-24-2008 08:37 PM

[DNeuwir]

Nope,

Windows 2003 SBS, as clearly stated in my original posting on this thread.

Again, we've seen this on four different 2003SBS servers, running the SBS edition of the product, and now on one we've gone back to 11d and the carts again pop out fine.  Tested on 3 PowerEdge 2900's with internal SATA RD1000 and a Asus (clone) server with an External USB RD1000.  Identical symptoms.  This is 4 out of 4 servers, all belonging to different companies.

-Dan

Message Edited by DNeuwir on 04-24-2008 05:44 PM

[R_H_2]

There is a person over here that might be on to something if someone else wants to try it...