NB process nbsvcmon, nbrmms, nbsl does not start on media server.

[Titus_Bandugula]

Hi All,

I have commissioned a new media server namely hbaltaprodap1 in my environment; the server is able to communicate to my master.

However, the policy for the server fails with status 25(cannot connect on socket).

When checking the same have observed that the below mentioned daemons were not running on this server (hbaltaprodap1); while these daemons are running on all the other media servers.

nbsvcmon, nbrmms, nbsl

MEDIA SERVER O.S : - Linux hbaltaprodap1.localdomain 2.6.18-274.el5 #1 SMP Fri Jul 8 17:36:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

MASTER O.S : - Sun Solaris 10

NBU Version : - 6.5.4

[root@dakcmstnetbkp /]\ # bptestbpcd -client hbaltaprodap1
1 1 1
10.226.7.187:49796 -> 10.226.19.232:13724
10.226.7.187:49813 -> 10.226.19.232:13724
10.226.7.187:49855 -> 10.226.19.232:13724
 

[root@dakcmstnetbkp /]\ # nbemmcmd -listh | grep -i hbaltaprodap1
media           hbaltaprodap1
 

[Marianne]

Please show us output of:

bptestbpcd -host hbaltaprodap1 
(this will test media server comms, not client)

nbemmcmd -listhosts -verbose

[revarooo]

6.5.4 is no longer supported. You should look to upgrade to 7.x (preferably 7.5.0.4)

[Titus_Bandugula]

Hi Marianne,

Kindly find the outputs of the given command.

[root@dakcmstnetbkp /]\ # bptestbpcd -host hbaltaprodap1
1 1 1
10.226.7.187:53443 -> 10.226.19.232:13724
10.226.7.187:53444 -> 10.226.19.232:13724
 

hbaltaprodap1
        ClusterName = ""
        MachineName = "hbaltaprodap1"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x61
        MachineNbuType = media (1)
        MachineState = active for disk jobs (12)
        MasterServerName = "dakcmstnetbkp"
        NetBackupVersion = 6.5.4.0 (654000)
        OperatingSystem = linux (16)
        ScanAbility = 5
 

[bharatgangawane]

Hi Titus ,

Have u created a Storage unit for this client, if yes then please upload the bptm & bpcd logs.

Thanks & Regards

Bharat

[Marianne]

 MachineState = active for disk jobs (12)

Have you created a Disk Storage Unit?

Please show us output of your policy config and STU:

bppllist <policy-name> -U

bpstulist -label <STU-name> -U

and all processes running on the Media server:

bpps -x

Have you tried to stop NBU on the media server and start it again?
Any error messages during startup?

Can you see if the processes are started and then terminate?

I have once seen a similar issue at a customer when a NBU patch was copied to a media server using ssh. We asked the customer to copy the actual downloaded patch (not extracted patch) to the media server and reinstall.
All daemons started up successfully.

PS: Are you aware of the fact that NBU 6.x ran out of support on 3 October?
 

[Titus_Bandugula]

Hello Marianne,

[root@dakcmstnetbkp /]\ # bppllist DAKC_D_W_M_hbaltaprodap1 -U
------------------------------------------------------------

Policy Name:       DAKC_D_W_M_hbaltaprodap1

  Policy Type:         Standard
  Active:              yes
  Effective date:      11/20/2012 14:18:25
  Client Compress:     no
  Follow NFS Mounts:   no
  Cross Mount Points:  no
  Collect TIR info:    no
  Block Incremental:   no
  Mult. Data Streams:  no
  Client Encrypt:      no
  Checkpoint:          yes
       Interval:       15
  Policy Priority:     0
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           dakcmstnetbkp-hcart-robot-tld-0
  Volume Pool:         DAKC_daily_pool
  Server Group:        Daily
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no

Granular Restore Info:  no
  HW/OS/Client:  Linux         IBMpSeriesRed hbaltaprodap1

  Include:  /CMB/SDSV3
            /CASA/CASAStatement/log/CASAStatement.db
            /CASA/NcbCasaSt/log/NcbCasaSt.db
            /CC/CCStatement/log/CCStatement.db
            /CC/LOS/log/LOS.db
            /CC/FasLas/log/FasLas.db
            /CC/DPStatement/log/DPStatement.db
            /CC/NCB/log/NCB.db
            /CC/BalanceStatement/log/Balance Statement.db

  Schedule:          M_hbaltaprodap1
    Type:            Full Backup
    Maximum MPX:     10
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 10 (12 years)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
    Calendar sched: Enabled
      Last day of month
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Sunday     01:00:00  -->  Sunday     07:00:00
          Monday     01:00:00  -->  Monday     07:00:00
          Tuesday    01:00:00  -->  Tuesday    07:00:00
          Wednesday  01:00:00  -->  Wednesday  07:00:00
          Thursday   01:00:00  -->  Thursday   07:00:00
          Friday     01:00:00  -->  Friday     07:00:00
          Saturday   01:00:00  -->  Saturday   07:00:00

  Schedule:          W_hbaltaprodap1
    Type:            Full Backup
    Frequency:       every 7 days
    Maximum MPX:     10
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 3 (1 month)
    Number Copies:   1
    Fail on Error:   0
    Residence:       (specific storage unit not required)
    Volume Pool:     (same as policy volume pool)
    Server Group:    (same as specified for policy)
      EXCLUDE DATE 0 - 08/31/2013
      EXCLUDE DATE 1 - 11/30/2013
      EXCLUDE DATE 2 - 05/31/2014
      EXCLUDE DATE 3 - 01/31/2015
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Saturday   01:00:00  -->  Saturday   06:00:00

  Schedule:          D_hbaltaprodap1
    Type:            Full Backup
    Frequency:       every 8 hours
    Maximum MPX:     10
    Synthetic:       0
    PFI Recovery:    0
    Retention Level: 0 (1 week)
    Number Copies:   1
    Fail on Error:   0
    Residence:       dakcmstnetbkp-hcart3-robot-tld-1_MPX
    Volume Pool:     (same as policy volume pool)
    Server Group:    Daily
      EXCLUDE DATE 0 - 11/30/2012
      EXCLUDE DATE 1 - 12/31/2012
      EXCLUDE DATE 2 - 01/31/2013
      EXCLUDE DATE 3 - 02/28/2013
      EXCLUDE DATE 4 - 03/31/2013
      EXCLUDE DATE 5 - 04/30/2013
      EXCLUDE DATE 6 - 05/31/2013
      EXCLUDE DATE 7 - 06/30/2013
      EXCLUDE DATE 8 - 09/30/2013
      EXCLUDE DATE 9 - 10/31/2013
      EXCLUDE DATE 10 - 12/31/2013
      EXCLUDE DATE 11 - 01/31/2014
    Residence is Storage Lifecycle Policy:     0
    Daily Windows:
          Sunday     01:00:00  -->  Sunday     06:00:00
          Monday     01:00:00  -->  Monday     06:00:00
          Tuesday    01:00:00  -->  Tuesday    06:00:00
          Wednesday  01:00:00  -->  Wednesday  06:00:00
          Thursday   01:00:00  -->  Thursday   06:00:00
          Friday     01:00:00  -->  Friday     06:00:00

######################################################

[root@dakcmstnetbkp /]\ # bpstulist -label hbaltaprodap1-hcart-robot-tld-0 -U

Label:                hbaltaprodap1-hcart-robot-tld-0
Storage Unit Type:    Media Manager
Host Connection:      hbaltaprodap1.localdomain
Number of Drives:     8
On Demand Only:       yes
Max MPX/drive:        1
Density:              hcart - 1/2 Inch Cartridge
Robot Type/Number:    TLD / 0
Max Fragment Size:    1048576 MB
[root@dakcmstnetbkp /]\ # bpstulist -label hbaltaprodap1-hcart2-robot-tld-2 -U

Label:                hbaltaprodap1-hcart2-robot-tld-2
Storage Unit Type:    Media Manager
Host Connection:      hbaltaprodap1.localdomain
Number of Drives:     7
On Demand Only:       yes
Max MPX/drive:        1
Density:              hcart2 - 1/2 Inch Cartridge 2
Robot Type/Number:    TLD / 2
Max Fragment Size:    1048576 MB

[Marianne]

Storage Unit in policy attributes: 
 Residence:           dakcmstnetbkp-hcart-robot-tld-0

Storage unit in  Schedule D_hbaltaprodap1:
 Residence:       dakcmstnetbkp-hcart3-robot-tld-1_MPX

Neither of these two STU's are selected in the policy:
hbaltaprodap1-hcart-robot-tld-0
hbaltaprodap1-hcart2-robot-tld-2 

How did you manage to create Media Manager devices and STUs for this media server when  "MachineState = active for disk jobs (12)"?

We are still missing answers to the following questions in my previous post:

Please show us output of .... all processes running on the Media server:
bpps -x

Have you tried to stop NBU on the media server and start it again?
Any error messages during startup?

Can you see if the processes are started and then terminate?

[bharatgangawane]

Hi Titus ,

It shows the Machine State is Active for disk.

Kindly delete the Storage unit related to this media server from EMM database & Bounce the Netbackup services on Media Server and then try to configure the Storage unit for this Media Server.

As per Marianne shows the out-put of bpps -x.

Bharat