cancel
Showing results for 
Search instead for 
Did you mean: 

Catalog Backup partially sucessful

sagar666
Level 5

Hi,

I have moved the db images to another filesystem and Created the symbolic link. Below is the process what i have done.

Stoped all netbackup services

Step1: cp -pr /usr/openv/netbackup/db/images /db/netbackup/db/images/

Step2: mv /usr/openv/netbackup/db/images /db/netbackup/db/images_bak

step3: ln -s /db/netbackup/db/images/images /usr/openv/netbackup/db/images

started all netbackup services

Started  few scheduled backups and later Catalog backup started but getting error code 1 that is Partially sucessful 

Below is the details of Job description

01/18/2017 10:07:13 - Error bpbrm (pid=8610) from client inbmaster.asicdesigners.com: ERR - Unable to process CATALOG_BACKUP inbmaster.asicdesigners.com hw_home_1479997800_INCR.f
01/18/2017 10:07:16 - Warning bpbrm (pid=8610) from client inbmaster.asicdesigners.com: WRN - /usr/openv/netbackup/db/error/daily_messages.log is only being backed up as a symbolic link
01/18/2017 10:07:16 - Info bpbkar (pid=8614) bpbkar waited 18266 times for empty buffer, delayed 23409 times
01/18/2017 10:07:17 - Info bptm (pid=8615) waited for full buffer 33267 times, delayed 34091 times
01/18/2017 10:07:39 - Info bptm (pid=8615) EXITING with status 0 <----------
01/18/2017 10:07:39 - Info bpbrm (pid=8610) validating image for client inbmaster.asicdesigners.com
01/18/2017 10:07:39 - Info bpbkar (pid=8614) done. status: 1: the requested operation was partially successful
01/18/2017 10:07:39 - end writing; write time: 0:13:27
the requested operation was partially successful (1)

 

hw_home is the policy that is disabled.

Please help to fix

 

Thanks,

 

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

"I digged more info from team got to know incidently they have deleted the few files. "

I don't have words!

Do they understand the implication of what they did?
Do they know about 'image retention'?

You should be able to restore the missing files from a previous catalog backup.

View solution in original post

14 REPLIES 14

sdo
Moderator
Moderator
Partner    VIP    Certified

I think you might need to open a support case to get your environment properly inspected and possibly fixed, if it's fixable.

.

Perhaps a procedure like this for anyone else coming across this post in future.

1) de-activate all SLPs

2) de-activate all backup policies (except the catalog backup)

3) de-activate all DSSU schedules

4) take a catalog backup

5) take an image list count:  bpimagelist -idonly -d 01/01/1970 00:00:00 | wc -l

6) netbackup stop

7) check it really is stopped:   bpps -x

8) do the move

....but are you sure those commands did the right thing?  why no -v switch and tee capture of what was actually moved?  why the double trailing "/images/images" on the link target, looks weird.

9) inspect the target structure:   ls -lash  MYNEWPATH

10) netbackup start

11) has it all started ok:   bpps -x

12) check the image list count:  bpimagelist -idonly -d 01/01/1970 00:00:00 | wc -l

13) enable one test policy, and run one small backup

14) check the image list count: bpimagelist -idonly -d 01/01/1970 00:00:00 | wc -l

15) run a catalog backup

16) If you are happy, then re-activate:  SLPs, policies, DSSU.

.

With something like this - my advice is to build a test system and prove the procedure before implementing in production.  But then that's easy for me to say after the horse has bolted.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

What was the purpose of step 2?

To rename images folder under /usr/openv or to really move entire folder and contents to /db? And thus having 2 copies of images in /db?

The paths in /db look very long and 'messy'. And images folder within another images folder?

Even so - the test is to cd to /usr/openv/netbackup/db/images.
What does 'ls -l' show?

The image file that is showing the error has nothing to do with deactivated policy. 
It is a .f file that was created during a backup that was done on 24 Nov 2016.

Can you locate the file by using 'normal' path from /usr/openv?

 

Hi Marianne

What was the purpose of step 2?

Root file system got filled up. So I moved to another file system that is under /db as backup

Yes it is messy i created one images directory so its looks odd

[root@inbmaster ~]# ls -l /usr/openv/netbackup/db/images
lrwxrwxrwx 1 root root 23 Jan 18 06:13 /usr/openv/netbackup/db/images -> /dbimages/images/images

Can you locate the file by using 'normal' path from /usr/openv?

I did not get your question, You mean move back images directory to same location?

 

 

 

tunix2k
Level 5
Partner Accredited

Hi,

 

in this case a better solution would be to migrate the complete masterserversoftware from /usr/openv to another volumen/disk and mount them under /usr/openv

 

ciao

tunix2k

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

What I meant - if you change directory to /usr/openv/netbackup/db/images:

# cd /usr/openv/netbackup/db/images

and then run 'ls -l' , what do you see?

I did not mean that you should move images back.
Can you locate the .f file by navigating through default /usr/openv path?
e.g.
# cd /usr/openv/netbackup/db/images/<client-name>/1479000000

Have you tried to run bpdbm -consistency 2 ?

Hi Marianne,

[root@inbmaster ~]# cd /usr/openv/netbackup/db/images
[root@inbmaster images]# ls -l
total 16
-r--r--r-- 1 root bin 263 Feb 8 2012 db_marker.txt
drwxr-xr-x 27 root root 4096 Jan 10 20:05 inbmaster
drwxr-xr-x 11 root root 4096 Jan 1 02:00 Inbmaster
drwxr-xr-x 28 root root 4096 Jan 17 21:39 inbmaster.asicdesigners.com

[root@inbmaster 1470000000]# ls -l
total 3300
drwx------ 2 root root 4096 Nov 9 20:54 catstore
-rw-rw-rw- 1 root root 0 Aug 1 08:23 inbmaster-catalog_1470018600_FULL.lck
-rw------- 1 root root 4887 Aug 1 08:00 inbmaster-catalog_1470018609_FULL.f
-rw-rw-rw- 1 root root 0 Aug 1 08:03 inbmaster-catalog_1470018609_FULL.lck
-rw------- 1 root root 322841 Aug 1 08:23 inbmaster-catalog_1470018952_FULL.f
-rw-rw-rw- 1 root root 0 Aug 1 08:23 inbmaster-catalog_1470018952_FULL.lck
-rw-rw-rw- 1 root root 0 Aug 2 08:20 inbmaster-catalog_1470105000_FULL.lck
-rw------- 1 root root 4887 Aug 2 08:00 inbmaster-catalog_1470105010_FULL.f
-rw-rw-rw- 1 root root 0 Aug 2 08:02 inbmaster-catalog_1470105010_FULL.lck
-rw------- 1 root root 323860 Aug 2 08:20 inbmaster-catalog_1470105231_FULL.f
-rw-rw-rw- 1 root root 0 Aug 2 08:20 inbmaster-catalog_1470105231_FULL.lck
-rw-rw-rw- 1 root root 0 Aug 3 08:24 inbmaster-catalog_1470191400_FULL.lck
-rw------- 1 root root 4887 Aug 3 08:00 inbmaster-catalog_1470191413_FULL.f
-rw-rw-rw- 1 root root 0 Aug 3 08:03 inbmaster-catalog_1470191413_FULL.lck
-rw------- 1 root root 328270 Aug 3 08:24 inbmaster-catalog_1470191780_FULL.f
-rw-rw-rw- 1 root root 0 Aug 3 08:23 inbmaster-catalog_1470191780_FULL.lck
-rw-rw-rw- 1 root root 0 Aug 4 08:20 inbmaster-catalog_1470277800_FULL.lck
-rw------- 1 root root 4887 Aug 4 08:00 inbmaster-catalog_1470277809_FULL.f
-rw-rw-rw- 1 root root 0 Aug 4 08:02 inbmaster-catalog_1470277809_FULL.lck
-rw------- 1 root root 327653 Aug 4 08:20 inbmaster-catalog_1470278018_FULL.f
-rw-rw-rw- 1 root root 0 Aug 4 08:19 inbmaster-catalog_1470278018_FULL.lck
-rw-rw-rw- 1 root root 0 Aug 5 08:24 inbmaster-catalog_1470364200_FULL.lck
-rw------- 1 root root 4887 Aug 5 08:00 inbmaster-catalog_1470364209_FULL.f
-rw-rw-rw- 1 root root 0 Aug 5 08:03 inbmaster-catalog_1470364209_FULL.lck
-rw------- 1 root root 326753 Aug 5 08:24 inbmaster-catalog_1470364589_FULL.f
-rw-rw-rw- 1 root root 0 Aug 5 08:24 inbmaster-catalog_1470364589_FULL.lck
-rw-rw-rw- 1 root root 0 Aug 8 08:20 inbmaster-catalog_1470623400_FULL.lck
-rw------- 1 root root 4887 Aug 8 08:00 inbmaster-catalog_1470623409_FULL.f
-rw-rw-rw- 1 root root 0 Aug 8 08:02 inbmaster-catalog_1470623409_FULL.lck
-rw------- 1 root root 324606 Aug 8 08:20 inbmaster-catalog_1470623610_FULL.f
-rw-rw-rw- 1 root root 0 Aug 8 08:19 inbmaster-catalog_1470623610_FULL.lck
-rw-rw-rw- 1 root root 0 Aug 9 08:24 inbmaster-catalog_1470709800_FULL.lck
-rw------- 1 root root 4887 Aug 9 08:00 inbmaster-catalog_1470709812_FULL.f
-rw-rw-rw- 1 root root 0 Aug 9 08:03 inbmaster-catalog_1470709812_FULL.lck
-rw------- 1 root root 328037 Aug 9 08:24 inbmaster-catalog_1470710195_FULL.f
-rw-rw-rw- 1 root root 0 Aug 9 08:24 inbmaster-catalog_1470710195_FULL.lck
-rw-rw-rw- 1 root root 0 Aug 10 08:20 inbmaster-catalog_1470796200_FULL.lck
-rw------- 1 root root 4887 Aug 10 08:00 inbmaster-catalog_1470796210_FULL.f
-rw-rw-rw- 1 root root 0 Aug 10 08:02 inbmaster-catalog_1470796210_FULL.lck
-rw------- 1 root root 326620 Aug 10 08:20 inbmaster-catalog_1470796431_FULL.f
-rw-rw-rw- 1 root root 0 Aug 10 08:20 inbmaster-catalog_1470796431_FULL.lck
-rw-rw-rw- 1 root root 0 Aug 11 08:23 inbmaster-catalog_1470882600_FULL.lck
-rw------- 1 root root 4887 Aug 11 08:00 inbmaster-catalog_1470882610_FULL.f
-rw-rw-rw- 1 root root 0 Aug 11 08:03 inbmaster-catalog_1470882610_FULL.lck
-rw------- 1 root root 328035 Aug 11 08:23 inbmaster-catalog_1470882954_FULL.f
-rw-rw-rw- 1 root root 0 Aug 11 08:23 inbmaster-catalog_1470882954_FULL.lck
-rw-rw-rw- 1 root root 0 Aug 12 08:20 inbmaster-catalog_1470969000_FULL.lck
-rw------- 1 root root 4887 Aug 12 08:00 inbmaster-catalog_1470969012_FULL.f
-rw-rw-rw- 1 root root 0 Aug 12 08:02 inbmaster-catalog_1470969012_FULL.lck
-rw------- 1 root root 329575 Aug 12 08:20 inbmaster-catalog_1470969249_FULL.f
-rw-rw-rw- 1 root root 0 Aug 12 08:20 inbmaster-catalog_1470969249_FULL.lck
drwxr-xr-x 3 root root 4096 Aug 12 08:20 tmp
[root@inbmaster 1470000000]# pwd
/usr/openv/netbackup/db/images/inbmaster.asicdesigners.com/1470000000

bpdbm -consistency 2  Few of .f files saying does not exits and attached file for reference

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Only those 3 folders? No client folders?

This means that there is something wrong with your symbolic links and actual location of images that were moved.

Have a look at your previous posts:

Here you showed where the images were copied to: 

cp -pr /usr/openv/netbackup/db/images /db/netbackup/db/images/

But this is where the symbolic link is pointing to: 

rwxrwxrwx 1 root root 23 Jan 18 06:13 /usr/openv/netbackup/db/images -> /dbimages/images/images

So, where exactly are the client folders located? 

When we know this we can help you to fix the symlink.

Hi Marriane,

Only one client  inbmaster.asicdesigners.com

and /db/netbackup/db/images/ is wrong I just mentioned steps exact path is /dbimages/images/images

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Okay... I have honestly never see a NetBackup installation where the master server is the only client....
You seem to be using 3 different hostnames for the master in backup policies:
inbmaster, Inbmaster, inbmaster.asicdesigners.com.

So, the problem seems to be with missing .f files.

The one in your opening post and the others mentioned in consistency check.

Any idea why they are missing? Not just one or two - quite a large number!

Can you locate them using find command? 
e.g.
find /dbimages/images/images -name hw_home_1479997800_INCR.f -print

Search images_bak folder as well to see if the .f files were present before they were copied/moved. 


[root@inbmaster 1470000000]# find /dbimages/images/images -name hw_home_1479997800_INCR.f -print
/dbimages/images/images/inbmaster.asicdesigners.com/1479000000/hw_home_1479997800_INCR.f

Any idea why they are missing? Not just one or two - quite a large number!

It was working fine after images migration it starts giving me problem.

Search images_bak folder as well to see if the .f files were present before they were copied/moved.

I digged more info from team got to know incidently they have deleted the few files.

Is there way we can restore to last full catalog backup ?

we dont have backup for the deleted files

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

"I digged more info from team got to know incidently they have deleted the few files. "

I don't have words!

Do they understand the implication of what they did?
Do they know about 'image retention'?

You should be able to restore the missing files from a previous catalog backup.

Thanks a lot marriane for your setup. I have restored Catalog backup

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Hopefully you have only restored the missing files (via BAR) and not the entire catalog!

Hi Marriene,

I have restored all the catalog for 4 days back. We have snapshots and another bakup for same clients.

 

Thanks,

Vidya Sagar