Forum Discussion

etmsreec's avatar
etmsreec
Level 3
7 months ago

Netbackup 5.1 bpsched issue on HP-UX

Hi,

Newbie here, so apologies for any errors in formatting or etiquette.

We have a Netbackup installation running v5.1MP6 on HP-UX.  Yes, it's old, yes we'd like to retire it or upgrade.

It seems to have stopped scheduling jobs all for itself?  It's been reliable and happy up to just before the holidays, but now the scheduling of backups seems to have started to fail.

We have restarted NBU a couple of times, which doesn't seem to have made any difference.

We are seeing the following message in the bpsched log file (once I'd created the directory for it to log bpsched to):

10:21:30.620 [16340] <2> bpsched: INITIATING (verbose=0) ...
10:21:30.621 [16340] <2> bpsched: BPRD PPID = 20591
10:21:30.621 [16340] <2> logparams: /usr/openv/netbackup/bin/bpsched -ppid 20591
10:21:30.621 [16340] <2> bpsched_main: wait_on_que=0, timeout_in_que=36000, reread_interval=300,queue_on_error=0, bptm_query_timeout=480, s_m 0
10:21:30.871 [16340] <2> LOCAL CLASS_ATT_DEFS: Product ID = 6
10:21:31.109 [16340] <8> bpsched_main: another regular bpsched is already examining the policy configuration
10:21:31.110 [16340] <4> bpsched: scheduler exiting - regular bpsched is already running (214)
10:31:30.249 [18497] <2> bpsched: INITIATING (verbose=0) ...
10:31:30.249 [18497] <2> bpsched: BPRD PPID = 20591
10:31:30.249 [18497] <2> logparams: /usr/openv/netbackup/bin/bpsched -ppid 20591
10:31:30.249 [18497] <2> bpsched_main: wait_on_que=0, timeout_in_que=36000, reread_interval=300,queue_on_error=0, bptm_query_timeout=480, s_m 0
10:31:30.482 [18497] <2> LOCAL CLASS_ATT_DEFS: Product ID = 6
10:31:30.714 [18497] <8> bpsched_main: another regular bpsched is already examining the policy configuration
10:31:30.715 [18497] <4> bpsched: scheduler exiting - regular bpsched is already running (214)
10:41:30.389 [20828] <2> bpsched: INITIATING (verbose=0) ...
10:41:30.389 [20828] <2> bpsched: BPRD PPID = 20591
10:41:30.389 [20828] <2> logparams: /usr/openv/netbackup/bin/bpsched -ppid 20591

Even without the "another regular bpsched is already examining..." it still didn't seem to want to schedule backups for itself?

Any thoughts please?  We haven't just rebooted the server, as why would we have to and why would that make any difference?  (Typical sys admin response! :) )

Thanks

Other

23 Replies

Sort By
  • Nicolai's avatar
    Nicolai
    Moderator
    7 months ago

    hi etmsreec 

     

    Put VERBOSE = 5 in bp.conf and restart Netbackup. This will increase the debugging level of bpsched and maybe explain why a previous bpsched is still examine the NBU policies. 

    Check for :

    • any zombie processes
    • use /usr/openv/netbackup/bin/bp.kill_all to stop all Netbackup processes and check with bpps -a they are ALL gone. Kill -9 on the NBU process that doesn't die.
    • host names with non ASCII names or hostnames with a extra dot e.g srv1.acme. <-- dot in the end of host name.
    • davidmoline's avatar
      davidmoline
      Level 6
      7 months ago

      Hi etmsreec 

      In addition to the suggestions from Nicolai have a look for stale lock files. 

      These will most likely be in the /usr/openv/netbackup/bin directory and for bpsched the lock file may be bpsched.lock or bpsched.lck. Shutdown NetBackup before removing any lock files. 

      The final step would be to reboot the server to ensure there are no zombie processes etc. as per Nicolai . 

      Cheers
      David

       

       

      • Nicolai's avatar
        Nicolai
        Moderator
        7 months ago

        bpsched.lock or bpsched.lck - good memory davidmoline 

        I had a faint memory of lock files, but time had erased the name of the files in my memory :)

  • StoneRam-Simon's avatar
    StoneRam-Simon
    Level 6
    7 months ago

    bpsched issues.... those were the days.....  

    Some things you can check...


    Has anyone done any manual modification of any files or folder under /usr/openv/netbackup/db/class

    Check files / folder permission, check if there are files in the toplevel of the class folder (this should just contain more folders not files..  there should be one per policy...
    Each policy folder should have 3 files (info, includes and clients) and one folder schedule, in schedule there will be one folder by schedule with 3 files (days, info and calendar)

    Is the filesystem full or has it filled up and left any files corrupt.

    Check the contents and permissions of the /usr/openv/netbackup/db/jobs too.

    I assume you checked if there were multiple bpsched processes running...
    bpps -a | grep bpsched

    How many jobs does it try to launch? (are you seeing any failed jobs or that nothing is scheduled).
    What do you have the "tries" set to in the host properties general attributes.


     

     

     

     

     

  • Nicolai's avatar
    Nicolai
    Moderator
    6 months ago

    etmsreec 

    Strange - Let's me ask in Veritas country why you don't have the option to attach files - I thought it was a default option for everyone.

    /Nicolai

  • benspickard's avatar
    benspickard
    Community Manager
    6 months ago

    We have turned on the ability to attach files again! Give it another try. 

  • etmsreec's avatar
    etmsreec
    Level 3
    5 months ago

    Hi,

    Apologies for the delay.

    To close this out, I've put a kill script (file attached) into cron which just checks for a bpsched once an hour and kills it off if it's there.  It seems to work, but may not be the cleanest.  Give me a DCL script and I'd be much happier!

    Thanks for the help and guidance on this one.

    Steve

  • etmsreec's avatar
    etmsreec
    Level 3
    6 months ago

    Hi Nicolai ,

    I did try to reply earlier, but the reply got marked as spam for some reason. :(

    We did a variety of things last week, including :

    - killing off all of the NBU processes, setting the verbosity, and restarting;

    - performing a reboot of the server;

    - stopping NBU, moving the .lock files to a different directory and then restarting NBU.

    The step of moving the lock files seemed to have allowed jobs to kick off on Friday night which I thought was going to be job done, but the jobs over the rest of the weekend failed to trigger.  We get the error:

    14:14:48.461 [21711] <8> bpsched_main: another regular bpsched is already examining the policy configuration
    14:14:48.462 [21711] <4> bpsched: scheduler exiting - regular bpsched is already running (214)

    As the song says, "...this is just where I came in..."

     

    • Nicolai's avatar
      Nicolai
      Moderator
      6 months ago

      Hi etmsreec 

      Posting debug text, usually get you in trouble, because your post will be marked as SPAM. To avoid this attach debug text as a file to a post.

      In the bpsched debug text look for <8> (warning) & <16> (catastrophic). <1> <2> <4> can be ignored.

      The "another regular bpsched is already examining the policy configuration" is quite normal for the old Netbackup versions. The scheduler wakes up every 20 minutes to examine the policy configuration to see if work is due. On larger installation this could easily take more than 20 minutes. But in your case is seems like bpsched never finishes it's work or crashes during the process

      Has any new clients been added to the policy configuration lately , potential sneaking a non-ascii character in the host name ?

      My recommendation :

      • Put VERBOSE = 5 in bp.cond
      • Clear out old debug logs in /usr/openv/netbackup/logs/bpsched
      • Restart Netbackup
      • Capture failed backup schedule
      • Capture bpsched log
      • Attach log file(s) to a post in this forum

      Pls note bpsched log may contain hostnames, you don't want to share with the internet.

      Also check DNS name resolution - both forward and reverse name lookup must be authorized.  Non-authorized answered will fail the backup.

       

  • etmsreec's avatar
    etmsreec
    Level 3
    6 months ago

    Hiya,

    I think I have caught all of the hostnames, policy names, and IP addresses that needed to be changed in my log file, before posting!  I didn't think posting the whole of the file was relevant, as it came out to be quite big, I have an extract.

    The first message relating to another bpsched already running is at 21:18:37.979, and it is then repeated every ten minutes thereafter.

    There have been no new clients added to the configuration in more than ten years.

    I'm trying to add the file to the post, but it keeps telling me that the file type is not supported if I drag and drop, and there doesn't appear to be an Attach File option?

    Steve

  • Nicolai's avatar
    Nicolai
    Moderator
    6 months ago

    Hi etmsreec 

    When creating a forum post, I get the option to attach files. A red cloud upload image. File size is only 71 MB. If file size is larger than 71 MB then consider to share from dropbox, G-Drive / OneDrive.

    Best Regards

    Nicolai

  • etmsreec's avatar
    etmsreec
    Level 3
    6 months ago

    Hi Nicolai ,

    Nope, I don't get that, either with a reply or a new posting.

     

    benspickarddid say that it was probably due to the ability to add text files and attachments to posts in order to reduce spam.

    We don't have a dropbox or public drive that I could present a link to, unfortunately. :o(