Unable resume the backups failed with 50,in case of check points enabled

[Sandeepkumar]

Hi Team

 We are unable start activity monitor from jnbsa due to disk space was filled up,we have cleared and we have restarted netbackup services all the backups went into error 50(client process aborted) in INCOMPLETE state.When i try to resume it is not possible to do so.Backups are not resuming.We are able to trigger new backups backups.As few backups have already written up around 1TB we dont want to cancel instead resume them.

Can anyone please explain or refer any document to see details about CHECKPOINT RESTART,where check points are taken and where they will be located,is it possible to view check points that are taken,any troubleshooting procedure for this will be appreciated.

Regards

Sandeep

[Michal_Mikulik1]

Hello,

the subject does not correspond to your question very well, your question is not about AM opening.

As for Checkpoint Restart, it is well documented in Admin Guide and also googable. Remember that generally you can use it only for MS-Windows and Standard policies.

Checkpoints are low level mechanism, I doubt they are reasonably viewable.

Michal

[Sandeepkumar]

Hi Michal

Changed the subject line,

Is there any possibility that i can resume with error 50.I have restarted services on master media and client.but still not able to resume.

How can we know where are check points are created.I have seen admin guide,it is well documented but iam curious about how to troubleshoot if the backup doesnot resume even check point enabled

Regards

Sandeep

[Marianne]

I would personally not trust anything in Incompete state after 'disk full' situation on the master.
NBU was busing adding to Image Catalog when this happened. Those images are probably corrupt with checkpoints unable to be written.

Restart those backups and put this one behind you as a valuable lesson learned.
Put filesystem monitoring and notification in place...

[Sandeepkumar]

Hi Marianne

Thanks, "openv" partition has been filled up,will this affect netbackup EMM Database as we were unable to ping the NBDB and bpjobd daemon was not running as a result we are not able to open ACTIVITY MONITOR.

Also kindly tell when we expire catalog images will the images  from /usr/openv/netbackup/db/images directory will be removed automatically...???

What all the unwanted files that can be removed from /usr/openv directory as this is occupying more space.

Regards

Sandeep

[Marianne · Accepted solution]

Yes - NBU is installed in /usr/openv along with all of its databases.

EMM is the relational database.
NBU will shutdown EMM when disk full situation is approached (cannot remember at which percentage).  

images folder is where information is stored about each file and folder that has been backed up where expiration date has not been reached.
Size is dictated by amount of files backed up at retention level/period.

NBU will automatically cleanup image files as and when backups expire naturally - this folder or any subfolder under images must NEVER be cleaned up manually.
Only files underneath .../db/images are cleaned up when images expire - not the image or client sub-folders.

PLEASE read up in NBU Admin Guide I about planning for catalog/image space and how this space can be increased or moved to new storage.

You can look for log folders - if they take up lots of space, read up on decreasing logging levels for unified logs and about removing unneccessary legacy logs or to decrease the logging levels.

[vtas_chas · Accepted solution]

Marianne raises the one thing that might help you the most immediately -- the logs.  Check what your logging level is and adjust it down if it's too high.  

Please also heed her advice on not adjusting the /usr/openv directory other than the logs.  I've had customers destroy NBU environments just removing things with old dates on them or inadvertently removing something from the directory structure that looked "unimportant".

Is this a completely separate partition that filled up or did your root filesystem fill for other reasons and that caused the full disk situation?  If this is a separate partition, did you add a lot of clients suddenly to the environment and your catalog grew dramatically?  

Managing disk space in an NBU environment can be quite an important activity if you're going to leave the partitions at bare minimum sizes or right on the edge of the minimum requirements, which I don't recommend.  

[Sandeepkumar]

Hi Marianne and Charles

Thanks for your excellent posts.

It is not the '/' partition that is filled it is a separate partition which got filled up.

As of now we dont have enough space in that partition to extend its size.So is it recommended to create soft link and so it can pointed to new partition after copying all the data from /usr/openv directory and then create soft link between these partitions..??

All loggings are at minimum level and are not occupying minimal space.only /netbackup/db/images is filled up and catalog filled up.We have managed to get some space as work aroung but for longterm we are creating the above said procedure.

Kindly confim if it can be done..?

Regards

Sandeep

[Marianne · Accepted solution]

Yes - it can be done.

Detailed steps in NetBackup Administrator's Guide, Volume I   under this topic: 
Moving the image catalog -> To move the image catalog on UNIX