Forum Discussion

Matthew_J's avatar
Matthew_J
Level 4
8 years ago

EV 10.0.4 CHF3 installed.... now DB logs are growing out of control?

We are still on EV 10.0.4.  I installed Cumulative Hotfix 3 tonight during a scheduled maintenance window. All seemed to go according to plan.  Shortly after I had wrapped it all up and called it good, I got a PRTG alert about the drive where the EV databases are held being close to full.  I checked it and it's consuming space in the EVVSMailStore_2LOG.ldf file, and is growing at a rate of about 11 gigs per hour.  I've added buffer space to hold off the inevitable, but I'm not able to determine what is going on here.  I've stopped a mailbox archiving run that was happening, and undid the schedule (set to never run) so it wouldn't restart, as I initially thought that was causing the space consumption.

I've set up a full backup to run tonight, i'm hoping that a log truncate will resolve the issue here but I'm not holding my breath.  In the morning I will open a high priority case with Veritas... in the mean time, does anyone have any thoughts on what could be ripping through space this fast? 

  • Matthew_J's avatar
    Matthew_J
    8 years ago

    Thanks Michal for the input, and you are absolutely right it turns out.

    For an update... I worked with Veritas support today and ultimately determined that yes, this amount of db changes is to be expected after the update, since it's pretty common for the database entries to be updated as part of a hotfix or update.  What I wasn't aware of, and what I now have squirrelled away in my brain for future updates, after a few harrowing hours of "what the heck is going on!?" is that it is a good idea to put the db into simple recovery mode before applying an update or fix, otherwise you may end up with a bunch of transactions that fill up a LOT of log space on the DB drive pretty quickly. 

    My understanding of simple recovery mode is that the log is "auto-truncated" at whatever size the current log container is, so it does not cause the space growth.  Since nobody in their right mind would re-wind the DB to a point halfway through a scheduled update (you would restore to your backup right before the update), having the DB in simple recovery mode during the upgrade is not that big of a deal... just remember to take it back out after all the db updates are made.

    The final piece here was that our full DB backup method does NOT truncate the DB logs, so this morning we were scratching our heads because we were expecting it to.  We use the Netbackup Enterprise Vault plugin to back up the vault DB's, and it only does a log truncate when doing a differential incremental backup.  So, after firing off a manual diff backup of the vault DB's with Netbackup, 99% of the 101 gig log space that vault was freaking out about was recovered.  Once this was done, I re-ran the health checks in the console are we are back to normal.

    Hope this helps some other poor soul some day who does an update and sees some unexpected behavior.

3 Replies

  • Hello,

    it is difficult to say what was the cause, however it is a good practise to switch off generating of db logs during EV upgrades (e.g. put EV databases into Simple Recovery mode).

    First it is good for upgrade performance, second the intention to have them for possible restore is usually not feasible here - in the case of disaster you will rather restore pre-upgrade state and repeat the upgrade by EV means again.

    Michal

    • Matthew_J's avatar
      Matthew_J
      Level 4

      Thanks Michal for the input, and you are absolutely right it turns out.

      For an update... I worked with Veritas support today and ultimately determined that yes, this amount of db changes is to be expected after the update, since it's pretty common for the database entries to be updated as part of a hotfix or update.  What I wasn't aware of, and what I now have squirrelled away in my brain for future updates, after a few harrowing hours of "what the heck is going on!?" is that it is a good idea to put the db into simple recovery mode before applying an update or fix, otherwise you may end up with a bunch of transactions that fill up a LOT of log space on the DB drive pretty quickly. 

      My understanding of simple recovery mode is that the log is "auto-truncated" at whatever size the current log container is, so it does not cause the space growth.  Since nobody in their right mind would re-wind the DB to a point halfway through a scheduled update (you would restore to your backup right before the update), having the DB in simple recovery mode during the upgrade is not that big of a deal... just remember to take it back out after all the db updates are made.

      The final piece here was that our full DB backup method does NOT truncate the DB logs, so this morning we were scratching our heads because we were expecting it to.  We use the Netbackup Enterprise Vault plugin to back up the vault DB's, and it only does a log truncate when doing a differential incremental backup.  So, after firing off a manual diff backup of the vault DB's with Netbackup, 99% of the 101 gig log space that vault was freaking out about was recovered.  Once this was done, I re-ran the health checks in the console are we are back to normal.

      Hope this helps some other poor soul some day who does an update and sees some unexpected behavior.

  • Hello Mathew,

    Thanks for the good explanation. I thought that EV set the databases in single usermode when upgrading, apparently that is not the case then.

    could you mark your answer as solution? That way others know the solution is in the post, and they will not open it to provide a solution (if they know of one)