Forum Discussion

asg2ki's avatar
asg2ki
Level 4
5 years ago

EMM_DATA.db - huge size can't be reduced

Dear All,

I recently got stuck with an issue where the size of my EMM_DATA.db has grown enormously and I can't find a way to reduce it at all. As a short description of my environment (LAB), I have 2 NBU master servers participating in AIR scenario where the first one (which is the source for AIR) has 3 separate MSDP based media servers connected to it. The second master server also has MSDP pool plus a tape library for offloading purposes. I'm replicating all backed up images between my two NBU master domains regularly as backup images gets produced by the clients.

The EMM_DATA.db file on the source master server somehow managed to grow to the enourmous size of 8.9GB while on the target master server it is not larger than 30MB. Although the infrastructure works well enough and I have no major failure out of this DB size except maybe for the increased size of the catalog backups, I wanted to optimize the DB to a somewhat normal size.

After trying to validate, reorganize and rebuild the DB section as per the tech. articles via the regular "nbdb_admin" and "nbdb_unload" commands nothing really has changed to the DB size therefore I started digging a bit further. I noticed one particular discrepancy through the NBDBAdmin utility which shown me visually that the EMM_DATA portion is indicating "-1 bytes" for both "Free DBspace" and "Total DBspace" metrics. Also the size of the DB is indicated to be as 4096MB while in reality it's twice as big which makes me think that there is some kind of a corruption throughout NBDB or perhaps just the EMM_DATA portion although all the validation activities I did via CLI indicated that there are no errors at all. Reorganizing the DB via CLI indicated very few fragments and it also didn't indicate any errors. The rebuild command didn't give me any output but just returned to command prompt, however I noticed slight modification to the NBDB log files (got truncated almost immediately) so I assume that it didn't fail but it was doing it's job.

On the second master server everything works absolutely fine and the sizes are definitely in order. Even though there is no relationship with this I would like to mention that all backup images in the catalogs for both master servers are exactly the same including the granularity factor for some of the backups where I use this feature (in my case Active Directory).

Strangely enough when I tried to do the run the "Reorganize All" command through the GUI utility it returned back "Operation Failed" status message. The result was the same also for the "Rebuild" command. On the contrary "Validation" went through and didn't indicate any issues.

My environment is based on Windows Server 2019 + NetBackup v8.2 for all NBU servers. One of my very old catalog backups that I could dig out (from exactly two years ago) clearly indicates that the EMM_DATA.db file at the time being was a little less than 30MB. If I remember right my servers were running NBU v8.1 binaries at the time being which I have upgraded to v8.1.1, v8.1.2 up till the current v8.2 through the years. I suppose this could be some sort of a bug with any of the intermediate or latest binaries, but right now I'm clueless on how this issue can be solved.

I would appreciate any thoughts from experts around and I hope to find a solution to this pesky little problem.

Cheers

8.1.2
8.2
EMM_DATA
EMM_DATA.db
Windows Server 2016
Windows Server 2019
  • Systems_Team's avatar
    Systems_Team
    Level 6
    5 years ago

    Hi Asg2ki,

    It seems this is a known issue under 8.2.  Here's the the KB doc for it:

    The EMM_DATA.db file can grow very large when running NetBackup 8.2  https://www.veritas.com/support/en_US/article.100047315 

    Hopefully you have current maintenance, which you would need if you want to get the EEB for it.  As the article states, if there are no negative symptoms then corrective action is not required.  It also states that applying the EEB and performing all the other required steps will not reduce the size of the database if it has already experienced this growth.

    Hope this helps,

    Steve

    • asg2ki's avatar
      asg2ki
      Level 4
      5 years ago

      Thanks Steve,

      I looked at the article you provided and it definitely seems to have some similarities with scope of my case but only as related to the DB growth. I'm not really sure it falls under exactly the same symptoms through since the article refers more to a situation where the system is experiencing huge stress. In my case this looks more like a problem with the structure / schema of the DB but as you already pointed out even if EEB is applied the growth is not likely to get reduced. Unfortunately I have no ways to get this EEB and respectively give it a try but thanks anyway for trying to help.

      BTW as a test I also managed to add additional free space (+25MB) via "NbDbAdmin.exe" utility to the EMM_DATA.db portion specifically but the GUI utility didn't reflect these changes at all and it is still showing me the "-1 bytes" usage as well as 4096MB of allocation. I definitely see the 25MB increase on the file itself from the OS point of view though, so it looks like a nasty bug with that file only.

      The negative side effect on my end is the execssive size of backup images produced by the NBU catalog policies since the incremental jobs are also producing 8GB of data during the staging operations. Although my Catalog backups are working absolutely fine I can't say the situation is ideal so hopefully somebody would come up with a way on how the EMM_DATA.db can be fixed or even re-created without having to rebuild the system from scratch.

      Cheers

      • mph999's avatar
        mph999
        Level 6
        5 years ago

        I would run nbdb_unload, and then look to see which tables are taking the most space - make a note of the xxx.dat file

        nbdb_unload /someoutput /dir

        If you then open reload.sql and search forward for 'INPUT INTO' and then from there search forward for xxx.dat, you will find the table name.

        For example:

        INPUT INTO "DBM_MAIN"."DBM_HoldImageMedia"
        FROM '/netbackup/db/3130.dat'

        If you know the Windows version of this, it's even easier ...

        grep -B1 '\.dat' reload.sql