Why suddenly increased /opt/openv/db/image frowth is increasing after 7.5 upgradation

[susindran_surul]

Hi Folks,

After we have upgraded 7.5.0.1 suddly database image growth is increaed and we have increase file system size 100GB last month but within one month again it reached 95%...Please let us know any other modifcation required on 7.5.0.1 datagrowth reduction..

Thanks,

Susi.S

[sksujeet]

The upgrade should not cause the space to be filled up in the images folder. The db/images directory has a list of directories for clients and if you have recently added new clients or changed the retention of your policies this might be causing the space to be filled up. Have you changed the policy to be full backups every day with long retention periods? You should have considered the space requirement while doing the upgrade.If your environment is very big you might need to do some analysis. You can either increase the space of your disk again after doing a good analysis or check the client in the images folder and find out which client is taking the space. Catalog Compression http://www.symantec.com/business/support/index?page=content&id=HOWTO34441 Catalog Archiving http://www.symantec.com/business/support/index?page=content&id=TECH36433 Also you can read NBU Admin Guide for Windows, Volume I catalog sizing If you can't increase the space then below is the link to move the images to different drive https://www-secure.symantec.com/connect/forums/tech-note-68160-moving-netbackupdbimages-different-path

[Marianne]

NBU 7.5 actually stores a bit less in images folder because the header files are moved into EMM database.

The biggest space consumers remain the '.f'' files - same as previous versions.
Factors that play a role (extract from NBU 7.5 Admin Guide I):

■ Number of files to be backed up

■ Frequency of full and incremental backups

■ Number of user backups and archives

■ Retention period of backups

■ Average length of full path of files

■ File information (such as owner permissions)

■ Average amount of error log information existing at any given time

■ Whether you have enabled the database compression option.

I have highlighted the biggest 'culprits':
lots of files, lots of backups (especially full) with long retention periods.

Please see the sample calculations in chapter 17 of Admin Guide I under this topic: Estimating catalog space requirements.

Also verify that Image Cleanup is running at least twice a day (every 12 hours and after successful catalog backup) and that cleanup job completes with status 0.

[mph999]

Adding to Mariannes excellent post, you could have some files left in the tmp directories eg:

/usr/openv/netbackup/db/images/<client>/<ctime>/tmp

Files go in there during backup, and get moved up a level if it completes successfully, and should be deleted if it fails.  If the system has an unclean shoutdown, files could be left behind, but, I wouldn't expect this to be a regular occurance, so it not likely to be the issue - however, worth checking.

As Marianne said, NBU 7.5.x does noy use more catalog space, in fact it uses slightly less - I'm not aware of any known issues with it, so sussect the most likely cause is one of those listed by Marianne.

Martin

[Mark_Solutions]

Can you tell us the Operating System you are using and if you have identified which area is suing up the disk space?

Is it the /images/ directory or somewhere else (tmp / temp etc.)?

Thanks

[susindran_surul]

Hi Marianne,

We found some of client incremental backup size is more than 300GB for daily basis.Those client images utilizing much more size...Is the any way to compress image size of those clients alone ???

[susindran_surul]

Its solaris and windows client...under /image/client only utilizing much more size....

[susindran_surul]

Hi Martin,

under /tmp is not utlizing much more size..I could see /usr/openv/netbackup/db/images/<client>/<ctime>/catstore utlizing much more size...Is there any way to compress it?

[sksujeet]

We found some of client incremental backup size is more than 300GB for daily basis.Those client images utilizing much more size...Is the any way to compress image size of those clients alone ???

That is normal the more the data the image would be bigger, we have incrementals more than 500 GB in our environment. You can do the archiving or compression on complete catalog as per my first post but not for some clients. Please consider again your retention If you have longer retention then this will happen. You have to plan accordingly as per you environment

[susindran_surul]

Actually we have expired most of the incremental backup of the particular client but still its not removing from /opt/openv/netbackup/db/images/client/time-stamp..Please let us know how we will remove those files under /opt/openv/netbackup/db/images/client/time-stamp..

[Mark_Solutions]

If they are on tape then use bpexpdate -deassignempty -force

If on disk use nbdelete -allvolumes

Followup using bpimage -cleanup -allclients

If you have already expired them then the last command above should do the trick for you

[susindran_surul]

Hi Mark,

We have catalog image on the disk only...I have did below steps to get free space..

1)Expired entire november month on one client(this client is utilized much space)

2)Run bpimage cleanup to cleanup images..

Please let us know nbdelete -allvolumes will delete only expired images???

[Mark_Solutions]

Yes - it is perfectly safe to run - from the guide:

Queries the image list in the EMMdatabase to obtain the list of volumes with
deleted fragments. It removes the fragments from those volumes and deletes
eligible imported snaps, unimported snaps, and tar images in that order.
-allvolumes calls bpdm on master server to delete imported snaps. It also
queries the storage servers with unimported snapshots but does not direct
bpdm to delete them.

This clears down the disk itself - the bpimage command cleans the catalog

Does it run with 100% success?

[susindran_surul]

Yes..bpimage cleanup completed 100% success...Please find the below for details..

01/07/2013 10:50:25 - Info bpdbm (pid=24813) deleted 430 expired records, compressed 66, tir removed 7, deleted 335 expired copies

the requested operation was successfully completed  (0)

My doubt is i have deleted entire november month all icremental images and i could see /opt/openv/netbackup/db/images/Client-name/time-stamp/catstore/ all november month images removed..Only full backup images is there..

My problem is i could see large number of file utlized on december moonth incremnetal backup only..Please let us know wheather we need to expire december month incemental backup aswel to get some more space???

Please find the  below for sample image details for december month.

bash-3.00$ sudo ls -lart catstore/

total 27819214

-rw-------   1 root     root           0 Dec  9 06:18 std-wv1s1-sa-0300_1355050947_INCR.f_imgNDMP0

-rw-------   1 root     root           0 Dec  9 06:18 std-wv1s1-sa-0300_1355050947_INCR.f_imgExtraObj0

-rw-------   1 root     root     981154524 Dec  9 23:11 std-wv1s1-sa-0300_1355050947_INCR.f_imgRecord0

-rw-------   1 root     root     184050304 Dec  9 23:11 std-wv1s1-sa-0300_1355050947_INCR.f_imgFile0

-rw-------   1 root     root     5672480 Dec  9 23:11 std-wv1s1-sa-0300_1355050947_INCR.f_imgDir0

-rw-------   1 root     root         519 Dec  9 23:13 std-wv1s1-sa-0300_1355050947_INCR.f_imgUserGroupNames0

-rw-------   1 root     root     253434806 Dec  9 23:13 std-wv1s1-sa-0300_1355050947_INCR.f_imgStrings0

-rw-------   1 root     root         212 Dec  9 23:13 std-wv1s1-sa-0300_1355050947_INCR.f_imgHeader0

-rw-------   1 root     root         924 Dec  9 23:13 std-wv1s1-sa-0300_1355050947_INCR.f-list

-rw-------   1 root     root           0 Dec 10 07:35 std-wv1s1-sa-0300_1355137371_INCR.f_imgNDMP0

-rw-------   1 root     root           0 Dec 10 07:35 std-wv1s1-sa-0300_1355137371_INCR.f_imgExtraObj0

-rw-------   1 root     root     981154860 Dec 11 01:00 std-wv1s1-sa-0300_1355137371_INCR.f_imgRecord0

-rw-------   1 root     root     184050368 Dec 11 01:00 std-wv1s1-sa-0300_1355137371_INCR.f_imgFile0

-rw-------   1 root     root     5672480 Dec 11 01:00 std-wv1s1-sa-0300_1355137371_INCR.f_imgDir0

-rw-------   1 root     root         516 Dec 11 01:02 std-wv1s1-sa-0300_1355137371_INCR.f_imgUserGroupNames0

-rw-------   1 root     root     253434905 Dec 11 01:02 std-wv1s1-sa-0300_1355137371_INCR.f_imgStrings0

-rw-------   1 root     root         212 Dec 11 01:02 std-wv1s1-sa-0300_1355137371_INCR.f_imgHeader0

-rw-------   1 root     root         924 Dec 11 01:02 std-wv1s1-sa-0300_1355137371_INCR.f-list

-rw-------   1 root     root           0 Dec 11 07:48 std-wv1s1-sa-0300_1355223745_FULL.f_imgNDMP0

-rw-------   1 root     root           0 Dec 11 07:48 std-wv1s1-sa-0300_1355223745_FULL.f_imgExtraObj0

-rw-------   1 root     root     253434621 Dec 12 07:46 std-wv1s1-sa-0300_1355223745_FULL.f_imgStrings0

-rw-------   1 root     root     981153768 Dec 12 07:46 std-wv1s1-sa-0300_1355223745_FULL.f_imgRecord0

-rw-------   1 root     root     184050192 Dec 12 07:46 std-wv1s1-sa-0300_1355223745_FULL.f_imgFile0

-rw-------   1 root     root     5672416 Dec 12 07:46 std-wv1s1-sa-0300_1355223745_FULL.f_imgDir0

-rw-------   1 root     root         572 Dec 12 07:48 std-wv1s1-sa-0300_1355223745_FULL.f_imgUserGroupNames0

-rw-------   1 root     root         212 Dec 12 07:48 std-wv1s1-sa-0300_1355223745_FULL.f_imgHeader0

-rw-------   1 root     root         924 Dec 12 07:48 std-wv1s1-sa-0300_1355223745_FULL.f-list

-rw-------   1 root     root           0 Dec 12 09:07 std-wv1s1-sa-0300_1355320366_INCR.f_imgNDMP0

-rw-------   1 root     root           0 Dec 12 09:07 std-wv1s1-sa-0300_1355320366_INCR.f_imgExtraObj0

-rw-------   1 root     root     981153936 Dec 13 02:51 std-wv1s1-sa-0300_1355320366_INCR.f_imgRecord0

-rw-------   1 root     root     184050224 Dec 13 02:51 std-wv1s1-sa-0300_1355320366_INCR.f_imgFile0

-rw-------   1 root     root     5672416 Dec 13 02:51 std-wv1s1-sa-0300_1355320366_INCR.f_imgDir0

-rw-------   1 root     root         511 Dec 13 02:56 std-wv1s1-sa-0300_1355320366_INCR.f_imgUserGroupNames0

-rw-------   1 root     root     253434727 Dec 13 02:56 std-wv1s1-sa-0300_1355320366_INCR.f_imgStrings0

-rw-------   1 root     root         212 Dec 13 02:56 std-wv1s1-sa-0300_1355320366_INCR.f_imgHeader0

-rw-------   1 root     root         924 Dec 13 02:56 std-wv1s1-sa-0300_1355320366_INCR.f-list

-rw-------   1 root     root           0 Dec 13 06:26 std-wv1s1-sa-0300_1355397102_INCR.f_imgNDMP0

-rw-------   1 root     root           0 Dec 13 06:26 std-wv1s1-sa-0300_1355397102_INCR.f_imgExtraObj0

-rw-------   1 root     root     981154188 Dec 13 21:15 std-wv1s1-sa-0300_1355397102_INCR.f_imgRecord0

-rw-------   1 root     root     184050272 Dec 13 21:15 std-wv1s1-sa-0300_1355397102_INCR.f_imgFile0

-rw-------   1 root     root     5672416 Dec 13 21:15 std-wv1s1-sa-0300_1355397102_INCR.f_imgDir0

-rw-------   1 root     root         511 Dec 13 21:17 std-wv1s1-sa-0300_1355397102_INCR.f_imgUserGroupNames0

-rw-------   1 root     root     253434812 Dec 13 21:17 std-wv1s1-sa-0300_1355397102_INCR.f_imgStrings0

-rw-------   1 root     root         212 Dec 13 21:17 std-wv1s1-sa-0300_1355397102_INCR.f_imgHeader0

-rw-------   1 root     root         924 Dec 13 21:17 std-wv1s1-sa-0300_1355397102_INCR.f-list

-rw-------   1 root     root           0 Dec 15 15:30 std-wv1s1-sa-0300_1355602411_INCR.f_imgNDMP0

-rw-------   1 root     root           0 Dec 15 15:30 std-wv1s1-sa-0300_1355602411_INCR.f_imgExtraObj0

-rw-------   1 root     root     981154860 Dec 16 07:46 std-wv1s1-sa-0300_1355602411_INCR.f_imgRecord0

-rw-------   1 root     root     184050400 Dec 16 07:46 std-wv1s1-sa-0300_1355602411_INCR.f_imgFile0

-rw-------   1 root     root     5672416 Dec 16 07:46 std-wv1s1-sa-0300_1355602411_INCR.f_imgDir0

-rw-------   1 root     root         514 Dec 16 07:48 std-wv1s1-sa-0300_1355602411_INCR.f_imgUserGroupNames0

-rw-------   1 root     root     253434949 Dec 16 07:48 std-wv1s1-sa-0300_1355602411_INCR.f_imgStrings0

-rw-------   1 root     root         212 Dec 16 07:48 std-wv1s1-sa-0300_1355602411_INCR.f_imgHeader0

-rw-------   1 root     root         924 Dec 16 07:48 std-wv1s1-sa-0300_1355602411_INCR.f-list

-rw-------   1 root     root           0 Dec 16 09:05 std-wv1s1-sa-0300_1355662293_INCR.f_imgNDMP0

-rw-------   1 root     root           0 Dec 16 09:05 std-wv1s1-sa-0300_1355662293_INCR.f_imgExtraObj0

-rw-------   1 root     root     981154860 Dec 17 01:39 std-wv1s1-sa-0300_1355662293_INCR.f_imgRecord0

-rw-------   1 root     root     184050400 Dec 17 01:39 std-wv1s1-sa-0300_1355662293_INCR.f_imgFile0

-rw-------   1 root     root     5672416 Dec 17 01:39 std-wv1s1-sa-0300_1355662293_INCR.f_imgDir0

-rw-------   1 root     root         514 Dec 17 01:46 std-wv1s1-sa-0300_1355662293_INCR.f_imgUserGroupNames0

-rw-------   1 root     root     253434966 Dec 17 01:46 std-wv1s1-sa-0300_1355662293_INCR.f_imgStrings0

-rw-------   1 root     root         212 Dec 17 01:46 std-wv1s1-sa-0300_1355662293_INCR.f_imgHeader0

-rw-------   1 root     root         924 Dec 17 01:46 std-wv1s1-sa-0300_1355662293_INCR.f-list

-rw-------   1 root     root           0 Dec 17 06:22 std-wv1s1-sa-0300_1355742628_INCR.f_imgNDMP0

-rw-------   1 root     root           0 Dec 17 06:22 std-wv1s1-sa-0300_1355742628_INCR.f_imgExtraObj0

-rw-------   1 root     root     981155028 Dec 17 22:45 std-wv1s1-sa-0300_1355742628_INCR.f_imgRecord0

-rw-------   1 root     root     184050432 Dec 17 22:45 std-wv1s1-sa-0300_1355742628_INCR.f_imgFile0

-rw-------   1 root     root     5672416 Dec 17 22:45 std-wv1s1-sa-0300_1355742628_INCR.f_imgDir0

-rw-------   1 root     root         511 Dec 17 22:47 std-wv1s1-sa-0300_1355742628_INCR.f_imgUserGroupNames0

-rw-------   1 root     root     253434982 Dec 17 22:47 std-wv1s1-sa-0300_1355742628_INCR.f_imgStrings0

-rw-------   1 root     root         212 Dec 17 22:47 std-wv1s1-sa-0300_1355742628_INCR.f_imgHeader0

-rw-------   1 root     root         924 Dec 17 22:47 std-wv1s1-sa-0300_1355742628_INCR.f-list

-rw-------   1 root     root           0 Dec 18 07:44 std-wv1s1-sa-0300_1355828782_FULL.f_imgNDMP0

-rw-------   1 root     root           0 Dec 18 07:44 std-wv1s1-sa-0300_1355828782_FULL.f_imgExtraObj0

-rw-------   1 root     root     253434953 Dec 19 07:53 std-wv1s1-sa-0300_135582

-rw-------   1 root     root     981155196 Dec 19 07:53 std-wv1s1-sa-0300_135582

-rw-------   1 root     root     184050464 Dec 19 07:53 std-wv1s1-sa-0300_135582

-rw-------   1 root     root     5672416 Dec 19 07:53 std-wv1s1-sa-0300_13558287

-rw-------   1 root     root         572 Dec 19 07:54 std-wv1s1-sa-0300_13558287

-rw-------   1 root     root         212 Dec 19 07:54 std-wv1s1-sa-0300_13558287

-rw-------   1 root     root         924 Dec 19 07:54 std-wv1s1-sa-0300_13558287

-rw-------   1 root     root           0 Dec 19 08:13 std-wv1s1-sa-0300_13559219

-rw-------   1 root     root           0 Dec 19 08:13 std-wv1s1-sa-0300_13559219

-rw-------   1 root     root     981155364 Dec 19 23:39 std-wv1s1-sa-0300_135592

-rw-------   1 root     root     184050496 Dec 19 23:39 std-wv1s1-sa-0300_135592

-rw-------   1 root     root     5672416 Dec 19 23:39 std-wv1s1-sa-0300_13559219

-rw-------   1 root     root         511 Dec 19 23:41 std-wv1s1-sa-0300_13559219

-rw-------   1 root     root     253435043 Dec 19 23:41 std-wv1s1-sa-0300_135592

-rw-------   1 root     root         212 Dec 19 23:41 std-wv1s1-sa-0300_13559219

-rw-------   1 root     root         924 Dec 19 23:41 std-wv1s1-sa-0300_13559219

drwx------   2 root     root        6144 Dec 19 23:41 .

drwxr-xr-x   4 root     root        2048 Dec 20 17:12 ..

[Mark_Solutions · Accepted solution]

It does seem extreme to be deleting backup images due to lack of space on the NBU server

Would it not be better to add disk space rather than deleting your backups?

[Will_Restore]

Incrementals running as Full?  Please provide backup policy details

  bppllist -L <policy_name>

[susindran_surul]

Actually problem is we have taken cold data base backup for particular server and removed exclude list for all database file systems,then we forgot to add in the exclude list after cold backup completes...Because of that reason its huge file system having that database server and image size increased due this issue.,,

[susindran_surul]

We have schedueld correct one but its database file system we forgot to add on exclude_list after cold backup completes...

 Schedule:              daily_incr

    Type:                Differential Incremental Backup

    Frequency:           every 1 day

    Maximum MPX:         5

    Synthetic:           0

    Checksum Change Detection: 0

    PFI Recovery:        0

    Retention Level:     6 (6 months)

    Number Copies:       1

    Fail on Error:       0

    Residence:           (specific storage unit not required)

    Volume Pool:         (same as policy volume pool)

    Server Group:        (same as specified for policy)

    Residence is Storage Lifecycle Policy:         0

    Schedule indexing:     0

    Daily Windows:

          Sunday     06:00:00  -->  Sunday     19:00:00

          Monday     06:00:00  -->  Monday     19:00:00

          Wednesday  06:00:00  -->  Wednesday  19:00:00

          Thursday   06:00:00  -->  Thursday   19:00:00

          Friday     06:00:00  -->  Friday     19:00:00

          Saturday   06:00:00  -->  Saturday   19:00:00

[Will_Restore]

Partial output?  Incremental schedule has to be in same policy as the Full schedule. 

[susindran_surul]

Policy Name:       std-wv1s1-sa-0300

  Policy Type:         Standard

  Active:              yes

  Effective date:      05/20/2008 00:00:00

  Client Compress:     no

  Follow NFS Mounts:   no

  Cross Mount Points:  no

  Collect TIR info:    yes, with move detection

  Block Incremental:   no

  Mult. Data Streams:  no

  Client Encrypt:      no

  Checkpoint:          no

  Policy Priority:     0

  Max Jobs/Policy:     Unlimited

  Disaster Recovery:   0

  Collect BMR info:    yes

  Residence:           wv1bkps1-hcart3-robot-tld-0

  Volume Pool:         NetBackup

  Server Group:        *ANY*

  Keyword:             (none specified)

  Data Classification:       -

  Residence is Storage Lifecycle Policy:    no

  Application Discovery:      no

  Discovery Lifetime:      0 seconds

ASC Application and attributes: (none defined)

  Granular Restore Info:  no

  Ignore Client Direct:  no

Enable Metadata Indexing:  no

Index server name:  NULL

  Use Accelerator:  no

  HW/OS/Client:  

                 Solaris       Solaris8      wv1b1-bkp

  Include:  ALL_LOCAL_DRIVES

  Schedule:              yearly_full

    Type:                Full Backup

    Frequency:           every 365 days

    Maximum MPX:         5

    Synthetic:           0

    Checksum Change Detection: 0

    PFI Recovery:        0

    Retention Level:     10 (7 years)

    Number Copies:       1

    Fail on Error:       0

    Residence:           (specific storage unit not required)

    Volume Pool:         (same as policy volume pool)

    Server Group:        (same as specified for policy)

    Residence is Storage Lifecycle Policy:         0

    Schedule indexing:     0

    Daily Windows:

          Tuesday    06:00:00  -->  Tuesday    19:00:00

  Schedule:              monthly_full

    Type:                Full Backup

    Frequency:           every 28 days

    Maximum MPX:         5

    Synthetic:           0

    Checksum Change Detection: 0

    PFI Recovery:        0

    Retention Level:     8 (1 year)

    Number Copies:       1

    Fail on Error:       0

    Residence:           (specific storage unit not required)

    Volume Pool:         (same as policy volume pool)

    Server Group:        (same as specified for policy)

    Residence is Storage Lifecycle Policy:         0

    Schedule indexing:     0

    Daily Windows:

          Tuesday    06:00:00  -->  Tuesday    19:00:00

  Schedule:              full_weekly

    Type:                Full Backup

    Frequency:           every 7 days

    Maximum MPX:         5

    Synthetic:           0

    Checksum Change Detection: 0

    PFI Recovery:        0

    Retention Level:     6 (6 months)

    Number Copies:       1

    Fail on Error:       0

    Residence:           (specific storage unit not required)

    Volume Pool:         (same as policy volume pool)

    Server Group:        (same as specified for policy)

    Residence is Storage Lifecycle Policy:         0

    Schedule indexing:     0

    Daily Windows:

          Tuesday    06:00:00  -->  Tuesday    19:00:00

  Schedule:              daily_incr

    Type:                Differential Incremental Backup

    Frequency:           every 1 day

    Maximum MPX:         5

    Synthetic:           0

    Checksum Change Detection: 0

    PFI Recovery:        0

    Retention Level:     6 (6 months)

    Number Copies:       1

    Fail on Error:       0

    Residence:           (specific storage unit not required)

    Volume Pool:         (same as policy volume pool)

    Server Group:        (same as specified for policy)

    Residence is Storage Lifecycle Policy:         0

    Schedule indexing:     0

    Daily Windows:

          Sunday     06:00:00  -->  Sunday     19:00:00

          Monday     06:00:00  -->  Monday     19:00:00

          Wednesday  06:00:00  -->  Wednesday  19:00:00

          Thursday   06:00:00  -->  Thursday   19:00:00

          Friday     06:00:00  -->  Friday     19:00:00

          Saturday   06:00:00  -->  Saturday   19:00:00