cancel
Showing results for 
Search instead for 
Did you mean: 

Diskstorage keeps on growing despite of overwrite period is exceeded.

Nidall
Level 3

Hi,

I'm running BE 2012 SP1a, on a virtual machine running win 2008R2 SP1, and have multiple backup jobs storing files on a NAS (B2D backupstorage).

All the backup jobs (full & inc) has a "Keep-Data" for a period of 8 weeks. The Diskstorage keeps on getting filled up with new .bkf files and the storage is keeping on shrinking. It doesn't seem that BE is retaining the media (.bkf) after the 8 weeks period is passed.

A quick look att the .bkf files stored on the NAS shows files dated 13 weeks back.

 

What am I missing here?

 

\nidall F

18 REPLIES 18

CraigV
Moderator
Moderator
Partner    VIP    Accredited

Are you appending or overwriting your media? Best practice for B2D backups is to overwrite...

Thanks!

Nidall
Level 3

Note::

 

look closer on the name and dates of the .bkf files on the NAS..

the nameing, whitch should indicates that in witch order BE has created the files rages from B2D002518 to B2D009548 while the oldes dated file is B2D004415-

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

Are you doing Incremental or differential backups as if yes then BE will keep the full backup until such time as all depending backups can also be overwritten.

Do you have any backuop sets where you were testign and then stopped performing the same backup (as last copy of any backup set on disk  retained for ever (by default)

Also as this is NAS storage are there any point in time where the NAS might go offline from the media server (as we are investigating an issue with unplgged USB disks not reclaiming old sets when the disk is disconnected and it is possible that a NAS could experience the same if not always online on the media server.)

Nidall
Level 3

@CraigV:

Where can I find this option?

the closest thing I came across is this option

Backup Exec Settings -> Storage
Under the "Tap/Disk cartridge Devices" section
Media overwrite options
1-Overwrite scratch media before overwriting recyclable media contained in the targeted media set. (Chosen)
2-Overwrite recyclable media contained in the targeted media set before overwriting scratch media.

 

@Colin Weaver:

I'm doing incremental and the backup jobs are weekly jobs where full backup is don on thu and inc sun-wed.

 

No test backup sets.. the same weekly backup is running week after week as intended.

 

the NAS has gone offline(in BE) once 2 or maybe 3 weeks ago. The incremental job that was supposed to run that nigh failed. But after setting the NAS back online in BE the following night job ran successfully

Nidall
Level 3

Where can I find this option?

the closest thing I came across is this option

Backup Exec Settings -> Storage
Under the "Tap/Disk cartridge Devices" section
Media overwrite options
1-Overwrite scratch media before overwriting recyclable media contained in the targeted media set. (Chosen)
2-Overwrite recyclable media contained in the targeted media set before overwriting scratch media.

David_Palmersto
Level 4

Nidall -

 

Here are a couple of places for you to look to see what's happening:

1) Check to see if Data Lifecycle Management (DLM) is running on your media server:

a) In BE2012, click the Backup Exec Jewel (upper left corner)

b) Click "Configuration and Settings", "Audit Log"

c) Select the category "Backup Set Retention" and make sure that you sort by Date at least once (I normally do it twice to get the most recent at the top).

d)The dates/times that DLM  has run are listed - DLM CAN run as often as every 4 hours - you CANNOT force DLM to run manually.

e) You should see 2 types of transactions: "Deleted expired image" AND "Deleted expired media" - deleting the image happens first and moves the BKF into the "Scratch" category while deleting the media happens afterward and is the process of removing the BKF from the disk storage.

f) If you do NOT see DLM running recently and deleting files, then you may have a problem with DLM and either need to log a case, delete some files manually (via BE2012 interface), or re-verify that your expectations for when backups are available for pruning are what BE expects, or all of the above...

 

2) Check to see if the existing backup sets are available for being pruned by DLM

a) In BE2012, Click Storage, <your storage device>, Backup Sets

b) This non-sortable list gives you ALL of the backup sets that are on the device in order from most recent Backup Time to the oldest

c) Scroll to the bottom - if you have more than 1000 backup sets on the device be very persistent at scrolling to the VERY bottom.

d) Select one of the backup sets that you believe is expired (and should be removed) right-click and select "Delete"

e) If there are no other backup sets that are dependent upon this one you will only receive a prompt to delete the set - DLM should be doing this to this set in the near future.

f) If there ARE other backup sets that are dependent upon this one, after you answer "Yes" to do you want to delete?, you will be given a list of the dependencies.

g) The list of dependencies will have all of the backup sets that are preventing this one backup set from being deleted.  Carefully review the dates on the dependent backup sets, if any are in the future, that is why DLM is not pruning that backup set.

h) Correct your expiration date and time settings and you may also need to delete backup sets manually to remove ones you believe should be expired, but which Backup Exec does not.

 

Good Luck.

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

Ok so thee incrementals running for a week and being retaing for 8 weeks will mean that the full will not be deleted for 9 weeks as the full cannot be reclaimed before the incrementals have been. (this is one of the rules of DLM there are others)

Nidall
Level 3

Hi Colin,

I'm aware of that rule. But as I mentioned before I have files on the NAS stretching 13 weeks back and BE keeps on writing new files instead of reusing the media(.bkf).

Nidall
Level 3

I mentioned in another reply that our NAS went Offline only once for 2-3 weeks ago.. Now it has done that (going offline)  2 times, 2 days in a row.

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

OK well if you have the issue where you NAS is going offline and DLM tries to operate against that offline drive, which we have usually seen against unplugged USB diskse offline device, then I believe the workaround is once the device is online, run an inventory and catalog job against the. (not sure how long this might take)  and then once thjs has finished, with 4 hours (as long as the device remains online it should reclaim the data.

 

EDIT: if above does not help suggest you log a formal support case as we may need etra logs etc as we may need a more in depth look into your problem.

Nidall
Level 3

Hi David,

I found the following informaation in the logs:::

**********************************************************************************************************************

The latest log from Device and Media:
Date/Time:
2012-10-26 13:20:30

User Name:

Category:
Device and Media

Message:
Media : B2D009587 has been overwritten by job VMFILExx.local Domain Backup Policy-6-Full Domain Backup Template new\


the latest log from Backup Set Retention:
Date/Time:
2012-10-11 02:24:06

User Name:
Data Lifecycle Management

Category:
Backup Set Retention

Message:
Deleted expired media: B2D005838,A735F1B1-4CD9-46BC-98E0-FB86290826D1,,0


Tracking down the deleted media B2D005838 in the Device and Media logs:::

Date/Time:
2012-10-11 02:24:05

Category:
Device and Media

Message:
Media : B2D005838 has been moved from media set Internal Disk Images to media set Retired Media

**********************************************************************************************************************

Not sure what that realy means. But It looks like DLM is running on the media server. All though the latest Backup Set Retention hasn't retained any media sins 2012-10-11

 

I select one of the backup sets (inc 14 weeks old) that I believe is expired (and should be removed) right-clicked and select "Delete" it found dependecies to 3 incremental backup sets dated a couple of days ago.. that is realy strange.

 

I'm running an Inventory job now ( been running for 6h now ) to see if that could provide other information and sort out the dependecies..

 

Nidall
Level 3

well it went offline once 3 weeks ago and once this sunday and once yesterday,,

Still think that backup sets that shouldn't exceed 2T for a weekly (full + inc) backup, that is retained after 8 weeks should fitt on a 26TB NAS.. today I'm down to 600GB of free space.

 

I'm running an Inventory job now ( been running for 6h now ) to see if that could provide other information and sort out the dependecies.. Sey didn't see you reply about runing the "Inventory and Catalog" job instead.

what is the difference betwee "Inventory" and "Inventory and Catalog"?

David_Palmersto
Level 4

You are making progress.  Please make sure that you are sorting the Audit Logs MULTIPLE TIMES by clicking in the column header for Date - until you are CERTAIN that the most recent transactions are at the top.

Some concerns:

1) DLM runs at least every 4 hours and your most recent "Backup Set Retention" is almost 20 DAYS ago either means that your backup sets have lots of dependencies (are FULL BACKUPS being done regularly?) or that the list is not sorted completely (see above).

2) The difference between the B2D file names from the 11th and the 26th shows that you ARE getting thousands of new backup sets over that time - THAT IS GOOD

3) The message from the 26th about the B2D being overwritten is a KNOWN SYMANTEC BUG that should not exist in the next release (this comes from your server being busy with high priority items and DLM not being able to delete the image (B2D file) quickly enough AFTER being moved to Retired Media - you can do nothing about this except let it happen now and hope that it goes away in the next update.

4) IF you see any TAPE dependencies when you try to manually delete a Backup Set (from within BE2012) it is because of #3 - just ignore the TAPE dependencies errors (there is a pre-release hotfix from symantec to remove these if they bother you - call support)...

What I'd recommend at this point:

 

1) Sort out those dependencies VERY CAREFULLY so that you understand what is happening to set dates out so far that your disk storage is filling up.

2) Pick a server or two and manually track how many copies of backups exist on the B2D storage device and identify why they are hanging around still - are they separate backups that are not related to each other? have the backups been changed so often that they are not part of the same dependency chain?

I hope your Inventory and Catalog (you did do BOTH, yes?) can help you move ahead in this matter.

 

I think that you are making progress (albeit slowly at this point)...

Colin_Weaver
Moderator
Moderator
Employee Accredited Certified

2012 has the option to run a catalog directly after an inventory.

 

An inventory records record of the media in the database, a catalog build a record of what files, folders, e-mail ec are held inside the media.

 

I had a feeling it might take a while to do it, it is only intended otbe a workaround and of course most USB disks probably do not hold as much data as a NAS. Really you need to sort out why the disks are going offline (and also review David's information on how DLM works as you may just have a bad config)

 

Also for reference 2012 will not let you append to a jobs sent to disk hence you cannot see the choice for overwrite or append.

Nidall
Level 3

I've done some progress here.

 

I've been deleting backup sets and there dependencies ( as long they are older than 9 week ) mainly taking in account the expiration date.. I have reclaimed 50% of the storage space.

The DLM has been running as it should from what I could see in the Audit logs.

I've reviewed my configuration and the retention time schedule mm are correct. Full backup been taken once a week Thursdays and incremental is taken 4 days a week Sunday to Wednesday. Retention time 8 weeks.

 

Got some Problems still though::

I'm having a great deal of problems scrolling there the Backup Sets on the Storage.

1-It keeps on jumping up n down there the dates.

2-Hitting "end" will take me down but not all the way some times it takes me further back in time than others. Hitting "End" again or scrolling / page down might take me down a notch or as the majority of times i takes me back up all the way to the latest backup set.. It even sometimes give me the error Query failed.. Hitting F5 will refresh, but I get to start all over..

3-When I finally get my hands on backup sets that are expired but not retained.. I select and delete. Sometimes I get this wonderful error " Could not Delete {0}"

4-While deleting & scrolling through backup sets (to get to the bottom) the following service BackupExecManagementService stops and I get to start all over.

5-I ran an "Inventory and Catalog" job yesterday. The inventory took 6h to complete successfully. I forgot though to set the incremental jobs "On Hold" so they kicked in and ran successfully after that the "Catalog" job started and ran for about 3h before it failed with this error code E0000900.

a).Could this be because of the incremental job ran in between the inventory and the Catalog job?

b).Should I rerun the Catalog job again?

c).The "Byte Count" of the "Catalog" job is 17GB, Is this just a counter or is this job creating files that are 17GB large or more every time I run a "Catalog" job?

 

Cheers

Nidall
Level 3

Here is more discription regading the error with the Catalo job..

******************************************

Final error: 0xe0000900 - The requested media is not listed in the media index and could not be mounted. To add the media's catalog information to the disk-based catalogs, run an inventory operation on the media and resubmit the Catalog operation. Final error category: Backup Media Errors For additional information regarding this error refer to link V-79-57344-2304

******************************************

This is the solution recomended

http://www.symantec.com/business/support/index?page=content&id=TECH153770&actp=search&viewlocale=en_US&searchid=1351773207286

 

Should I go ahead and folow the instructions above?

 

Nidall
Level 3

Here is some more information about that catalgo job error i recieved yesterday.

***************************************

Completed status: Failed Final error: 0xe0000900 - The requested media is not listed in the media index and could not be mounted. To add the media's catalog information to the disk-based catalogs, run an inventory operation on the media and resubmit the Catalog operation. Final error category: Backup Media Errors For additional information regarding this error refer to link V-79-57344-2304

***************************************

 

here is the sugested solution

http://www.symantec.com/business/support/index?page=content&id=TECH153770&actp=search&viewlocale=en_US&searchid=1351773207286

 

should I or not? is this oslution för BE 2012 SP1a??

I'm a litle uncleare about step 2&3

  • Go to start/ symantec/ back up exec folder/catalogs
  • change the name on folder "catalogs" to "catalogs_old"

I can't seem to fined start/ symantec/ back up exec folder/catalogs  :)

David_Palmersto
Level 4

Nidal -

(I was hoping a symantec employee would reply as I'd like to know their current information on this)

What I can tell you from my own personal experience is that we have experienced all of the items you describe in #1-4 on your first 01-Nov post regularly. I opened support cases on all of the items and the scrolling, jumping, and Management service crashes are on the Symantec list to fix in a future (hopefully next) release.

You item #5 is also very familiar to us, having had the same situation and questions as you. 

For this one, I have less definitive results, but here is my theory at this time:

1) The Inventory and Catalog will be more likely to complete successfully with each of these items:

a) A quiet system that is NOT running any backups or restores

b) A system that has NO reused B2D files (this is also scheduled for a future release but tends to happen less on systems that are not pushed to their limits) - A reused B2D file comes when DLM process a backup set for deletion by moving the backup to the retired media set but the actual media has not been deleted from the storage drive - if a backup to disk is then run, it can reused this retired media before it actually is deleted (this is why the Audit Log for Backup Set Retention must show the deletion of BOTH the image and the media).

c) The less work that the inventory and catalog needs to accomplish.

2) I have also gone the route of renaming and rebuilding the catalogs AND also tried simply reusing the existing catalog files. I was finally able to get successful Inventory and Catalog jobs with a combination of both techniques.

a) If you rename the catalogs folder, then the catrebuild will attempt to do a complete Catalog of the entire B2D storage and all of their contents.

b) If you simply re-run the Inventory and Catalog from within BE, any existing Catalogs in the catalog folder will be found, but their contents will NOT be re-cataloged.  You will get a message in the log of the Catalog job that the file(s) existed and may not contain backup sets (this is because it doesn't even bother looking into them) Message from log: "This backup set has already been cataloged. This backup set may not contain any data."

c) If you have concerns about the contents of each of the catalog files, I'd suggest doing the rename, if you are just concerned that the DLM is working correctly, the Inventory and Catalog from within BE will probably be enough.

3) Step 2 about going to Start... is really, just find the catalogs folder on your media server and rename it to something that BE will not reuse (my folder is in "C:\Program Files\Symantec\Backup Exec\Catalogs") yours is probably similar. The basic idea is to completely remove the existing catalog folder files and remake them from existing media on the B2D storage device.

Good Luck.