Forum Discussion

martin_macf's avatar
martin_macf
Level 1
5 years ago

*.EML files stuck in the SMTP Holding Folder

Good afternoon,

I have an issue.  The emails coming from our Exchange Server and being processed by our SMTP Archiving Task are all getting stuck in the SMTP Holding Folder.  It has enough space and has the correct permissions.  The server and service have been restarted.  There are no significant errors messages in the event viewer.  It started a couple of days ago during the afternoon without warning.  The SQL Server is working fine, no resource hungry processes in operation.

A sample of an error is Unable to archive eml file,24-06-2020,02:42:21 PM,..\24\13\48\1593006504.2016.12650.eml,<e475af80-2e3d-4ed9-90f7-edf9100f05df@journal.report.generator>,0x80040302,No action,"Recipient: ev_smtp_journal@evsmtp.journal. Enterprise Vault is currently busy or has insufficient resource to archive the file, or Enterprise Vault is currently being backed up and unable to archive files.".  This is from the report EVSMTP_ERROR_LOG_MACFEV02_20200624_142712_33.

I also did a dtrace against smtp archiving issues.  Here's a sample of the output.
2 16:02:23.495 [14376] (EvSMTPTask) <2076> EV:M {CMimeItem::LoadStream} Loaded item [S-14376-000000001D2C7BC0]. Elapsed time [0.01s]
3 16:02:23.511 [14376] (EvSMTPTask) <2076> EV:L {CMimeItem::FinalRelease} Unloaded item [S-14376-000000001D2C7BC0]
4 16:02:23.526 [14376] (EvSMTPTask) <SmtpItemProcessorThread13:2076> EV-M {SmtpArchivingErrorHandler} \\?\UNC\MACFEV02.macfarlanegroup.net\F$\SMTP Holding Folder\24\13\57\1593007035.2016.33772.eml file is skipped to archive. Reason, storage is either busy or unavailable.
5 16:02:23.526 [14376] (EvSMTPTask) <SmtpItemProcessorThread13:2076> EV-L {SmtpItemProcessor} TraceTimer [1379ms] Elapsed time to process item - \\?\UNC\MACFEV02.macfarlanegroup.net\F$\SMTP Holding Folder\24\13\57\1593007035.2016.33772.eml
6 16:02:23.526 [14376] (EvSMTPTask) <SmtpFolderEnumeratorThread:12952> EV-L {ThreadSafeQueue} Item [Symantec.EnterpriseVault.SmtpArchivingTask.SmtpWorkItem] will be added to the end of the queue.
7 16:02:23.526 [14376] (EvSMTPTask) <SmtpFolderEnumeratorThread:12952> EV-L {ThreadSafeQueue} Enqueuing 'Symantec.EnterpriseVault.SmtpArchivingTask.SmtpWorkItem' will be enqueued at the end of the queue - Force Enqueue: False - delaySeconds: 0
8 16:02:23.526 [6080] (DirectoryService) <13776> EV:M CEventLog::RegisterEventFilterForThread - registered thread filter callback [TID: 13776]
9 16:02:23.526 [6080] (DirectoryService) <13776> EV:L CEventBuffer::Register - registered EventBuffer filter callback [TID: 13776]
10 16:02:23.526 [6080] (DirectoryService) <13776> EV:L CADOContext::CreateConnection entry
11 16:02:23.558 [14376] (EvSMTPTask) <13668> EV:L {CMimeItem::LoadStream} Loading item [(null)].
12 16:02:23.558 [14376] (EvSMTPTask) <13668> EV:M {CMimeItem::LoadStream} Loaded item [S-14376-000000001D2B5F50]. Elapsed time [0.00s]
13 16:02:23.558 [6080] (DirectoryService) <13776> EV:L CADOContext::CreateConnection exit. source:Provider=SQLOLEDB;Server=MACFSQLAPP;Database=EnterpriseVaultDirectory;Trusted_Connection=yes hr=Success (0)
14 16:02:23.558 [6080] (DirectoryService) <13776> EV:M DirectoryService: ADO: GetArchiveFolderIncludingDeletedFolders |For more information, see Help and Support Center at http://entced.symantec.com/entt?product=ev&language=english&version=11.0.1.0&build=11.0.1.3683&error=V-437-8223
15 16:02:23.574 [6080] (DirectoryService) <13776> EV:M CEventLog::RegisterEventFilterForThread - registered thread filter callback [TID: 13776]
16 16:02:23.574 [6080] (DirectoryService) <13776> EV:L CEventBuffer::Unregister - unregistered EventBuffer filter callback [TID: 13776]
17 16:02:23.574 [6080] (DirectoryService) <13776> EV:M CEventLog::RegisterEventFilterForThread - registered thread filter callback [TID: 13776]
18 16:02:23.574 [6080] (DirectoryService) <13776> EV:L CEventBuffer::Unregister - unregistered EventBuffer filter callback [TID: 13776]
19 16:02:23.574 [14376] (EvSMTPTask) <13668> EV:L {CMimeItem::FinalRelease} Unloaded item [S-14376-000000001D2B5F50]

The version of EV is 11.0.1.3683.

Any thoughts, ideas, fixes, etc would be extremely appreciative.

Regards
Martin Godfrey

  • GertjanA's avatar
    GertjanA
    Moderator
    5 years ago

    Hello Martin,

    The obvious answer is to upgrade to 12.5.x, but I assume you have reasons not to. (I use 12.3 SMTP archiving on VM's, no issues, doing approx 4 mill. messages a day).

    The indication in the dtrace seems to point to resources. I know there as an issue way back with EV taking a lot of handles, but a server restart solves that (which you did). Anyway. If it started a few days ago, did anything happen? Microsoft patches? New software installed? Passwords changed? Anything that might have been done could have accidentally caused this.

    Check that first. Application and System eventlog would be my first start, then the EV log.

    What I would initially do is clear backup mode on the Site level. There was an issue where backup mode was not properlty cleared, cuasing a counter to go up to 2. That told EV Vault Stores are in backup mode, but they will show as 'not in backup' in the Console. Clearing BUmode on Site level will reset the value to 0.

    I would then try is to create a new archiving task. Sometimes that fixes issues.

    Then I would try to setup a different location for the Holding Folder. If possible a new disk, otherwise on a disk having sufficient space. When the holding folder location is switched, monitor if that location does get processed. That might indicate a corrupt item in the original holding folder somewhere. 

    If that does not help, I would try a re-install of EV. Sometimes, a re-install fixes strange issues.

    • SheldonDsouza's avatar
      SheldonDsouza
      Level 4
      5 years ago

      Hi,

      I agree with Gertjan. He seems to have spelled out in detail exactly what needs to be done.

      "Enterprise Vault is currently busy or has insufficient resource to archive the file, or Enterprise Vault is currently being backed up and unable to archive files" - Seeing this in the report, and as mentioned by Gertjan about Clearing the Backup Mode at the Site Level in the GUI, I would go a step further and actually run the Powershell commands to clear the backup mode of the Vault Stores and Index Locations. 

      Creating another location for the SMTP Holding Folder is a good way to start troubleshooting the issue.