Event ID 8390- The EnterpriseVault.DirectoryConnection object reported an error. The RPC server is unavailable

[S-E-E-D]

hi
Just upgraded from EV7 to EV8 SP2. all worked fine. done some test archiving and all seemed to be OK.

Rejigged some old provisioning groups to set up new groups with new AD groups. (previously our servers archiving has been turned off, due to lack of disk space. this is now resolved so im looking to archive old mailboxes from the last 4 months, as well as turning on a 60 day policy for live users)

Starting with the leavers, i enabled them all in AD, unhid etc, enabled then in EV, manually run the archive task against 50 mailboxes and went home last night. worked fine and managed to archive about 3GB or more, but lots of 8390 errors started coming in the event log at about 11pm.

ive looked at various forums etc, turned off delete orphaned shortcuts which seemed to sound like a fix but wasnt.
Waiting for support renewal to come through from symantec, and they've recommended i post here in the meantime.

I have reduced the number of mailboxes im trying to kick start to 1, and i still get the 8390 error. i know its quite generic, so need some assistance. dont really know/understand dtrace, so instructions on what to run and what to look for would be useful.

thanks!

Edward

[TonySterling]

Can you post the actual event?

What process is throwing the 8390? 

[Douglas_Snyder]

Hi Edward,

Most RPC errors are related to memory or system resource issues.  I suspect that making so many changes to the AD and then running the archive task might be the problem.  If the directory synchronization did not run or run to completion then the system is trying to archive a lot of mailboxes that are not updated (as far as EV is concerned).

You already turned off deleting orphaned shortcuts (noticed that in a previous post) so we can eliminate that.  Unfortunately once RPC errors accumulate the system generally needs restarted to clear them.

I suggest restarting the EV server when possible then running the directory synchronization. Once that is completed try the archiving task.  Also - please open a support case so if you have issues after trying this you can give this information to support and move your case along.  Thanks!

[S-E-E-D]

Sorry, see below.

Event Type: Error
Event Source: Enterprise Vault
Event Category: Archive Task
Event ID: 8390
Date:  10/09/2009
Time:  12:15:28
User:  N/A
Computer: HAL-SERVER9
Description:
The EnterpriseVault.DirectoryConnection object reported an error.
 
The RPC server is unavailable.
For more information, see Help and Support Center at http://evevent.symantec.com/rosetta/showevent.asp
 

[S-E-E-D]

Thanks for the info. I have been struggling with test archiving on remote exchange servers and have been forcing AD replication etc to get things moving. i guess what you are saying is true, that lots is changing and i need to step back, reboot, and then start again.

Let me do that and i'll get back to you, after trying ONE mailbox. I'll open a case as soon as i get an email from Symantec with my updated sypport contract info.

Thanks
Edward
 

[JolaineY · Accepted solution]

I've seen this error occur  during shortcut processing in EV 8.0.  Delete orphaned shortcuts is part of shortcut processing, but I am talking about the 'Retention Category' and 'Location' updates

Try turning off the 'Update location' & 'Update Retention' in the mailbox archiving policy (on the Moved Items tab) and  make sure to clear out the A6 MSMQ (http://support.veritas.com/docs/276082)

You can run the shortcut processing independently by Right Clicking on the Archiving task and choosing 'Run Now' and select Shortcut processing.

If that fixes the issue for you but you need to know whats causing the errors, you can enable retention and location updates, get a dtrace of the Archive Task while running shortcut processing.   In my experience shortcuts that are associated with an unreachable EV Site or Directory are the culprit. 

[S-E-E-D]

rebooted server
when i start the SYNC process, it get 8390 errors for ARCHIVING TASK in my event logs. so possibly something relating to that then?? 
tried on 2 different mailboxes so dont suspect it to be a particular users mailstore.

any further suggestions or am i stuck till i can call Symantec??

Thanks

Edward

[TonySterling]

Dtrace the archive task as Jolaine said.  That might give us some insight.

[S-E-E-D]

hi
just trying what Jolaine said so will get back after that.
For a DTrace novice, what commands would i need to get that tracing, and to turn it off again afterwards? did this once with Symantec on the phone but didnt take note.

Thanks
Edward

[S-E-E-D]

hi
Continuing to get 8390 errors.
Have disabled Update retention and Update location
purged queues (about 60 items)

Ran an archive only task, and seems to be working?!?!? the usage.asp page seems to slowly be incrementing. no errors yet. not holiding my breath yet!

Is this a fixable issue if related to moved items and orphaned shortcuts? or is this something that i wont need so can keep turned off? this setting was on and working i assume in v7 without problems? 

Will report back, and hopefully give some feedback if i speak with Symantec.

Thanks for the help today!

Edward

[TonySterling]

Update Retention and Update location are new to EV 8.

Those items in the queue would have been causing the errors to occurr even after you unchecked the previous options.

Hopefully you can keep going and then have symc support help you figure out the rest.  :)

Cheers

[JolaineY]

Glad to hear that helped to isolate the problem.

With EV 8.0 the location and retention category updates works by checking the shortcut properties for the retention category and folder id and matching against the current folder id and retention category.  I'm guessing there are some shortcuts in the mailbox that reference an old EV Site or Directory so the 'matching' fails and that is where you are seeing problems. (For all the EV experts, please forgive me for oversimplifying this)

The next step in troubleshooting is to turn the 'Moved Items' shortcut processing back on, enable Dtrace for ArchiveTask and run the Archive Task for shortcut processing. 
Instructions for enabling dtrace: http://support.veritas.com/docs/276120

Once you see the 8390 errors disable dtrace.  In the log have a look for lines similar to this:

CBaseDirectoryServiceWrapper::CreateDirectoryService() - Will try to connect to EV Directory Service on EVSITE

CArchivingAgent::GetParentArchiveId - COM Error The RPC server is unavailable

If the Directory Service computer that is referenced in the dtrace does not look right (it's not the current EV server/site alias or name) it would confirm that there are shortcuts in the mailbox that point to an invalid EV site.  If that's the case you do have a few options, the first is to leave the moved items processing off.  The second is to use HOST file entries to point the invalid Directory Service computer name to the current EV server.  The caveat with the second method is that the shortcuts will be treated as orphaned and will be deleted from the mailbox if you have Orphaned Shortcut deletion turned on.

If you're having troubles reading the dtrace we can help.

[HonkyTonkHero]

I have the same issue every few weeks. Every time I have run a dTrace it has mentioned the same user mailbox. It says that another process thread is working on that object and puts it back in the queue. My A6 queue gets items stuck in it. I stop the Task Controller services and purge the queue. THe issue happens again a few days later. I imagine its trying to update the retention becuase there is another error that refers to the moved items not being able to be updated.

Not sure what the problem is. I am forcing his mailbox to be archived with the Move options disabled. Hopefully this resolves the issue. I dont see any errors regarding the location of the directory server or any COM errors. I have a suspicion that the name of the folder could be an issue. It is a very long path and it has special characters in it.

I did open a ticket with Symantec support and they thought it had to do with the number of items and connections the task was using. I changed the values to lower values and for 3 days everything seemed ok so we closed the ticket. A few days later the same error and the same user. Since then I purge the queues and a few days later the error comes up.

Very annoying as it fills up my event log. I'll post back if I find out anything.

[HonkyTonkHero]

As I suspected, the error was being triggered by a users archive folder that contained a special character. In this case he was using a ">" and a "<" symbol in his folder name. I replaced the characters with "GT" and "LT" respectively, stopped EV, purged the A6 queue, restarted EV and forced an archive. I have not seen the error again.

In this case dTrace helped me realize that this was isolated to a single user which lead me to believe this was something wrong with his mailbox. I also noticed in the dTrace events that the folder path to the items in question had non standard symbols in the name and I have seen other programs have issues processing these special characters.