catalog backup finishes with status 1

[phoenix24x1]

Hey All,

i run 7.5.0.7 on win2008 enterprise.

My catalog backup is finsihing with a status 1. In viewing the job details it is stating a log file is not there to be captured. Job Details pasted below. The log file is indeed not there. I have the "cleanup" settings (host properties > master > cleanup) set to 10 days for log file retention. The unix timestamp for the log file is just outside 10 days. 

Does this cleanup operation include the log file in the catalog referenced below?

If so, I should just be able to increase the retention time and should not receive this error.

4/28/2014 6:17:30 PM - positioning C103L4 to file 2
4/28/2014 6:17:32 PM - positioned C103L4; position time: 00:00:02
4/28/2014 6:17:32 PM - begin writing
4/29/2014 2:25:32 PM - Info bpbrm(pid=8656) from client vsadm52: TRV - object not found for file system backup: E:\NetBackup\db\error\log_1397880000
4/29/2014 2:26:08 PM - Info bptm(pid=9788) waited for full buffer 390403 times, delayed 4526586 times    
4/29/2014 2:26:31 PM - Info bptm(pid=9788) EXITING with status 0 <----------        
4/29/2014 2:26:31 PM - Info bpbrm(pid=8656) validating image for client vsadm52        
4/29/2014 2:26:34 PM - end writing; write time: 20:09:02
the requested operation was partially successful(1)

[Marianne]

WOW! How big is you catalog? Seems your backup ran for about 20 hours?

Yes, db/error logs are included in Days to keep Logs settings.

The above means that the log file was enumerated at the start of the backup but got deleted as part of normal Clean-Up while the catalog backup was running.

I would worry more about catalog backup running this long, rather than worry about a log file that got cleaned out during this period.

[phoenix24x1]

my catalog backup is ~800gb, a solution to the size is on my to-do list. I thought it prudent to address the status 1 first.

[Marianne]

I cannot see a way to solve the status 1 backup other than addressing the 20 hour backup duration.
Each time the backup goes past 12 hours and logs are getting cleaned up, this will happen.

~800 gb should not take that long to backup. Should not take more than 6 or 7 hours.
We have a customer with similar catalog size on a Windows server and catalog backup takes about 5 hours.

Where are you backing up to?
What kind of disk storage is used for NBU catalogs?
When last was file system checked for fragmentation?

[phoenix24x1]

- Catalog backup is writing to tape

- san storage

- 94% fragmented     =/

I have vault jobs writing to tape with much better speeds, so I suspect the severe fragmentation is at the very least contributing to this problem. 

[mph999]

There are 4.5 million delays, that is a lot of time ...  18 odd hours 

So for 18 hours, bptm was doing nothing ... lots and lots of 15ms delays.

I would guess it's waiting on a Full buffer which would suggest a disk read issu, or network if client is remote,

.

[Marianne · Accepted solution]

Local tape or across network to a media server?

This is most certainly a big problem:

- 94% fragmented 

Try to find a defrag tool that can run online.

In summary - I do not see status 1 as an issue, rather the catalog backup runing more than 20 hours.

[phoenix24x1]

Across a network to media server.

This may actually be an issue with reading the from the disk presented by the SAN. We have similiar issues in the environment with SAN. Additionally, my vault jobs (Symantec Appliance -> tape) that run do not experience the buffer problems that is seen in the catalog backup.

I will defrag the volume but that will not happen for 10 days or so. I will update the thread with my results thereafter.

[Michael_G_Ander]

If you havn't already it might be worth to look into the disk layout technote, sorry don't remember the number

In general more disks gives more disk queues/buffers in the OS which gives better performance of cause that does not help if SAN issues are caused by load by the storage controllers

Think a catalog on windows should be defragmented something like at least once a week

[phoenix24x1]

Our storage Admins have done some tweeking on the SAN. The catalog is now completing with a status 0. Its still taking too long to complete and I am still going to get this volume defragged.

Ideally I would like to defrag this volume when NBU is down, but due to the size I worry that it will take way too long. Does anyone have any recommendations for a tool (preferrably free) that will defrag while online?
 

[rookie11]

pheonix : as ur catalog backup is aroind 800 GB. you need to check with symantec whether sybase database is getting truncated or not. it can be checked from nbdb logs. log trucncation happens only on full catalog backup but not on incremental catalog backup. Ask symantec they can provide a setting which can truncate DB logs even after incremental catalog backup.