Journal Delete overloads Domain Controller

[BBenedeta]

I have a hand full of Journal archives I have to delete. However, every time I mark a Journal archive for deletion, our Domain Controllers get over loaded with directory queries and/or authentication requests. This does not seem to appear with mailbox deletions.In both cases ( mailbox and journal deletioln) their corresponding mailboxes are gone.

I figure a Dtrace may help. I am just not too sure what to Dtrace.

If anyone has any ideas, I'd apreciate it a lot.

[GertjanA]

Hello,

That is strange. As far as I know, an archive deletion only hits SQL and Storage and Index locations, not a DC. Can you advise the EV version you have? Are the queries/requests on the DC coming from the EV server?

I would start by doing a trace on StorageDelete, see what that shows.

[BBenedeta]

GertjanA,

Beyond weird, but the behaviour is consistent. The only time we see the issue on the DCs is when I am deleting a Journal archvie. As soon as I stop the deletion (a bit of SQL hackery is used), the errors goes away.  I've asked the team watching the DCs to see if they can pinpoint which EV server ( I have 2 storage servers) the requests are coming from without success.

I ran a couple Dtraces on StorageDelete. They don't show any connection request to any of the DCs.. I wonder if SD passes the request to another process to authenticate and that's why we see the problem.

My EV is on 12.2.2. Hoping to upgrade to 12.4.1 when it comes out.

[GertjanA]

Hello again,

Besides request coming from the EV servers, can you also check requests coming from the SQL server, and (if used) the storage device?

Additionally, you might want to use the resource monitor on the EV server to see which connections are made. As you are on 12, I suggest to involve support on this one. They'll now what to Dtrace, and possibly can provide a solution.

[ChrisLangevin]

That's really weird. I'm not sure a dtrace will be much help, because StorageDelete is the process to trace for expiry, and as you have seen, it's not making any connections to a DC to do its expiry. If I were troubleshooting this issue, I'd try a packet capture (Wireshark, NetMon, etc.) on the affected DC and try to figure out what machine the excess traffic is coming from. Gertjan's right that it might be from a non-EV machine that's nonetheless involved with the deletion, such as an AD-integrated storage device.

Hope that helps.

--Chris

[BBenedeta]

It turned out the issue was part with our storage and part with EV. We use Data Domain for our storage and it uses retention lock to prevent items from being deleted prior to their retention period. This is only applied to Journal archives. However, when we need to remove a Journal archive, we can run a script to change the retention and allow EV to delete those messages.  DD  has to make sure EV was authorized to delete every single message. Chris, I had one of the DC admins watch the network traffic. He did not see any authentication calls coming from our storage, but from the EV storage server I was deleting the archive from. I had already changed the WriteOnceMedia setting to 0, so I was confused as why EV was sending all of those requests to the DC ( that was when I posted my original question here). It wasn't until I tricked EV to think it was deleting from a different type of storage that those requests stopped and I was able to delete the archives / partitions / Vault stores etc...

Thank you all for the pointers. ;)