Forum Discussion

virankumar's avatar
virankumar
Level 4
10 years ago

BMR Backup Hang-Not moving

Hello Team,

I ran one BMR backup , it been queued from 16hours . There is no Child jods triggered

When i was trying to kill the master server process , its picking the backups (child jobs are triggering )

Any one please help me in this situvation

Master server -Solaris (Netbackup 6.5.6)

Client - Linux (Netbackup 6.5.6)

 

Thanks & Regards

Virankumar S

  • The client logs show that the "bmrsavecfg" process ran to completion without errors and did so very quickly.  Issue is outbound of the client.

    Try running the manual import of the client configuration file to see how long it takes.  That should be fairly fast as well.

20 Replies

  • Hi Jim,

     

    in which command i need to give the truss and -f specification in which part i need to Do that

  • Hi Maria,

     

    I have ran command again in client and Master

    In master the  Manually import the configuration into the BMRDB , its hanging here is no movement at all

     

    Detail discription of job from GUI

    -----------------------------------------------

    Sep 19, 2014 2:30:59 PM - collecting BMR information
    Sep 19, 2014 2:30:59 PM - connecting
    Sep 19, 2014 2:30:59 PM - connected; connect time: 0:00:00
    Sep 19, 2014 2:30:59 PM - transfering BMR information to the master server
    Sep 19, 2014 2:30:59 PM - connecting
    Sep 19, 2014 2:30:59 PM - connected; connect time: 0:00:00
    Sep 19, 2014 2:31:13 PM - requesting resource tb-mast-01-disk
    Sep 19, 2014 2:31:13 PM - requesting resource tb-mast-01.NBU_CLIENT.MAXJOBS.tb0dbs0-bsn
    Sep 19, 2014 2:31:13 PM - requesting resource tb-mast-01.NBU_POLICY.MAXJOBS.ct1_ivr_ccps_fs_policy
    Sep 19, 2014 2:31:14 PM - granted resource  tb-mast-01.NBU_CLIENT.MAXJOBS.tb0dbs0-bsn
    Sep 19, 2014 2:31:14 PM - granted resource  tb-mast-01.NBU_POLICY.MAXJOBS.ct1_ivr_ccps_fs_policy
    Sep 19, 2014 2:31:14 PM - granted resource  MediaID=@aaaab;Path=/staging;MediaServer=bnct1-dl180g6-04
    Sep 19, 2014 2:31:14 PM - granted resource  tb-mast-01-disk
    Sep 19, 2014 2:31:14 PM - estimated 0 kbytes needed
    Sep 19, 2014 2:31:14 PM - begin Parent Job
    Sep 19, 2014 2:31:14 PM - begin Stream Discovery: Start Notify Script
    Sep 19, 2014 2:31:14 PM - started process RUNCMD (pid=12483)
    Sep 19, 2014 2:31:14 PM - ended process 0 (pid=12483)
    Operation Status: 0
    Sep 19, 2014 2:31:14 PM - end Stream Discovery: Start Notify Script; elapsed time 0:00:00
    Sep 19, 2014 2:31:14 PM - begin Stream Discovery: Stream Discovery
    Sep 19, 2014 2:31:14 PM - started process bpmount (pid=8339)
    Operation Status: 0
    Sep 19, 2014 2:31:14 PM - end Stream Discovery: Stream Discovery; elapsed time 0:00:00
    Sep 19, 2014 2:31:14 PM - begin Stream Discovery: Bare Metal Restore Save
    Sep 19, 2014 2:31:15 PM - started process bpbrm (pid=27132)

     

     

    From client side
    -------------------


    [root@Client ~]# cd /usr/openv/netbackup/bin
    [root@Client bin]# ./bmrsavecfg -infoonly
    [root@Client bin]# echo $?
    0
    [root@Client bin]#

    [root@Client bin]# cd /usr/openv/netbackup/baremetal/client/data
    [root@Client data]# ls -l bun*
    -rw-r----- 1 root root 129536 Sep 19 14:32 bundle.dat
    [root@Client data]# scp -rp bundle.dat root@10.0.4.10:/tmp
    Password:
    bundle.dat                                                                                 100%  127KB 126.5KB/s   00:00
    [root@Client data]#


    From master
    --------------------

    root@Master # bmrs -o import -res config -path /tmp/bundle.dat

     

     

     

    Wat else need to do Can you please suggest

     

  • So run that step agan that hangs prefixed with the "truss" command. This will show you the system calls being executed and should show you the syscall it makes prior to it hanging.That should then lead you to a fix. I recommend using the -f option to the command.

    The output will be potentially large, so you could choose to put it in a file and then examine the file

    Jim

  • Each step james suggested i have ran successfully

    But when the step in master server

    root@masterserver # bmrs -o import -res config -path /tmp/bundle.dat

     

    its been hang here

    Thanks & Regards

    Virankumar

  • Please show us detailed output of each step suggested by Symantec BMR Expert Jaime's post on 28 August.

    You are going to battle to find better assistance than this - NBU 6.x ran out of support 2 years ago.

    If this has worked before, you need to find out what exactly has changed, since it seems that NBU has not changed in some years in this environment...

     

  • HI Team,

     

    Can any one please give me update on this

    previusly BMR backup works on this server but now its hanging in between

     

  • I'd be seriously tempted to upgrade to latest and greatest: there have been no end of fixes from 6.5 to 7.6 and plenty of those are BMR. Besides, if you dont then Symantec wont be helping you either. Maybe you are no longer under a support contract?

    One obvious question: you've been through the bmrsetup steps yes? And all was good?

    Jim 

  • Hello Jaime,

     

    Client Linux

    -----------------------

    Red Hat Enterprise Linux Server release 5.5 (Tikanga)

     

    Master Solaris :

    -------------------

    Solaris 5.10

    Detaisl JOb details

    ------------------------------

    Aug 26, 2014 6:00:00 PM - requesting resource tb-mast-01-disk
    Aug 26, 2014 6:00:00 PM - requesting resource tb-mast-01.NBU_CLIENT.MAXJOBS.tb0dbs0-bsn
    Aug 26, 2014 6:00:00 PM - requesting resource tb-mast-01.NBU_POLICY.MAXJOBS.ct1_ivr_ccps_fs_policy
    Aug 26, 2014 5:59:47 PM - collecting BMR information
    Aug 26, 2014 5:59:47 PM - connecting
    Aug 26, 2014 5:59:48 PM - connected; connect time: 0:00:00
    Aug 26, 2014 5:59:48 PM - transfering BMR information to the master server
    Aug 26, 2014 5:59:48 PM - connecting
    Aug 26, 2014 5:59:48 PM - connected; connect time: 0:00:00
    Aug 26, 2014 6:00:02 PM - granted resource  tb-mast-01.NBU_CLIENT.MAXJOBS.tb0dbs0-bsn
    Aug 26, 2014 6:00:02 PM - granted resource  tb-mast-01.NBU_POLICY.MAXJOBS.ct1_ivr_ccps_fs_policy
    Aug 26, 2014 6:00:02 PM - granted resource  MediaID=@aaaab;Path=/staging;MediaServer=bnct1-dl180g6-04
    Aug 26, 2014 6:00:02 PM - granted resource  tb-mast-01-disk
    Aug 26, 2014 6:00:03 PM - estimated 36698 kbytes needed
    Aug 26, 2014 6:00:03 PM - begin Parent Job
    Aug 26, 2014 6:00:03 PM - begin Stream Discovery: Start Notify Script
    Aug 26, 2014 6:00:03 PM - started process RUNCMD (pid=4060)
    Aug 26, 2014 6:00:03 PM - ended process 0 (pid=4060)
    Operation Status: 0
    Aug 26, 2014 6:00:03 PM - end Stream Discovery: Start Notify Script; elapsed time 0:00:00
    Aug 26, 2014 6:00:03 PM - begin Stream Discovery: Stream Discovery
    Aug 26, 2014 6:00:03 PM - started process bpmount (pid=23048)
    Operation Status: 0
    Aug 26, 2014 6:00:03 PM - end Stream Discovery: Stream Discovery; elapsed time 0:00:00
    Aug 26, 2014 6:00:03 PM - begin Stream Discovery: Bare Metal Restore Save
    Aug 26, 2014 6:00:04 PM - started process bpbrm (pid=14381)

     

     

    Its been hang , here not connecting

    root@tb-mast-01 # bmrs -o import -res config -path /tmp/bundle.dat

     

     

     

  • OK, there are several tings in here that are problematic For starts, the environment is 6.5.6, which has been EOSL for quite a while now. 

    Need some more details for the Master/Client environment. What version of Solaris is the Master?  What version/release is the Linux client?

    When performing an NBU backup with the BMR feature enabled, you should get an initial job on the Admin Console that shows the BMR portion of the  process.  Do you see that?  If so can you share the job details as shown for it?

    The BMR portion is there to create, capture and insert the client configuration into the BMRDB.  The NBU job manager performs two functions for this:

    1.It sends a START_NOTIFY to the client with appropriate option information which causes the client system to run "bmrsavecfg" locally.  This process creates the client information in a file called "bundle.dat". 
    2. It creates a bpbrm process which starts up and waits for the client to send it the client configuration file. This becomes the "master job" and stays resident until the entire NBU back is completed.
    3.  When the client process completes, it sends the bundle file to bpbrm for insertion into the BMRDB.
    4. The bpbrm process forwards the file to the bmrd process on the Master for DB insertion and waits for a response when complete.
    5. The bmrd process, using database services, insert the information into the BMRDB and when complete, returns either a zero or a one to bpbrm as the status code.
    6. The bpbrm gets the response and then starts firing off the normal NBU backup streams noted in the policy.  Those become their own jobs and get their own job id.
    7. Normal NBU backup occurs and when the last stream completes, the "master job" captures the highest status code of all of the job and terminates with that status code.

    With all of this, please tell me where you are seeing the "hang"? 

    You can emulate the BMR portion of this by doing the following (all commands are in /usr/openv/netbackup/bin):

    On the Linux client, run the command "/bmrsavecfg -infoonly". This creates the file "/usr/openv/netbackup/baremetal/client/data/bundle.dat". Run "echo $?" immediately after that to get the return code.
    If the return code on the client was zero, copy this file to the Master at any directory.  As an example, create the file "/tmp/bundle.dat".
    Manually import the configuration into the BMRDB by running "bmrs -o import -res config -path /tmp//bundle.dat". Again run "echo $?" to get its return code.

    If all of this worked with no errors (rc=0) then BMR is not the culprit. Try the backup with the BMR option disabled but with the "True Image" and "With move detection" enabled. Setting the BMR option on automatically enables both.

    Please let us know your results for all of this.