Netbackup volume manager\request daemon\device manager services are not coming up

[Amarnath_Sathis]

Hi

Upon login to admin console I got database is down error message and I tried to restart the service.

Netbackup volume manager\request daemon\device manager services are not coming up. We tried rebooting the server also.

Now I couldnot see data\logs\bin folders under Program Files\Veritas\NetBackup\db. nbdb_ping gives sometimes as available and sometimes as not available.

Product and Version: VRTS NETBACKUP ENTERPRISE 7.1.0.3

Master server operating system:  Windows server 2008 R2 Standard

Please help.

[Marianne]

Sounds like EMM is not running. Have you checked disk space where NBU is installed?
If disk fills up to 99%, NBU will shutdown EMM to prevent corruption of the nbdb database

Please try to stop NBU from cmd and then restart:

bpdown -f -v

(Wait for everything to stop and check screen for messages. Use 'bpps' to confirm that everything is stopped.)

Use 'Services' to restart Symantec Private Branch Exchange.

bpup -f -v

(Check screen for messages)

bpps

If nbemm does not start or starts up and then terminate, please show us emm logs:

vxlogview -o 111 -t 00:10:00

Above command will show emm logs for last 10 minutes. Please adjust -t minutes to include last startup attempt.

Please also check server.log for errors:
File is in <install-path>\Veritas\NetBackupDB\log\

[Amarnath_Sathis]

Drive where netbackup is installed has 10 % free space.

Starting services
> NetBackup Legacy Network Service
> NetBackup Legacy Network Service -- STARTED
> NetBackup Legacy Client Service
> NetBackup Legacy Client Service -- STARTED
> NetBackup Client Service
> NetBackup Client Service -- STARTED
> NetBackup SAN Client Fibre Transport Service
> NetBackup SAN Client Fibre Transport Service -- NOT STARTED
> NetBackup Event Manager
> NetBackup Event Manager -- STARTED
> NetBackup Relational Database Manager
> NetBackup Relational Database Manager -- STARTED
> NetBackup Enterprise Media Manager
> NetBackup Enterprise Media Manager -- STARTED
> NetBackup Resource Broker
> NetBackup Resource Broker -- STARTED
> NetBackup Volume Manager
> NetBackup Volume Manager -- NOT STARTED
> NetBackup Device Manager
> NetBackup Device Manager -- NOT STARTED
> NetBackup Remote Manager and Monitor Service
> NetBackup Remote Manager and Monitor Service -- STARTED
> NetBackup Deduplication Manager
> NetBackup Deduplication Manager -- NOT STARTED
> NetBackup Deduplication Engine
> NetBackup Deduplication Engine -- NOT STARTED
> NetBackup Authentication
> NetBackup Authentication -- DISABLED
> NetBackup Authorization
> NetBackup Authorization -- DISABLED
> NetBackup Audit Manager
> NetBackup Audit Manager -- STARTED
> NetBackup Database Manager
> NetBackup Database Manager -- NOT STARTED
> NetBackup Compatibility Service
> NetBackup Compatibility Service -- STARTED
> NetBackup Request Daemon
> NetBackup Request Daemon -- NOT STARTED
> NetBackup Job Manager
> NetBackup Job Manager -- NOT STARTED
> NetBackup Policy Execution Manager
> NetBackup Policy Execution Manager -- NOT STARTED
> NetBackup Service Layer
> NetBackup Service Layer -- STARTED
> NetBackup Vault Manager
> NetBackup Vault Manager -- NOT STARTED
> NetBackup Key Management Service
> NetBackup Key Management Service -- STARTED
> NetBackup Storage Lifecycle Manager
> NetBackup Storage Lifecycle Manager -- STARTED
> NetBackup Agent Request Server
> NetBackup Agent Request Server -- STARTED
> NetBackup Service Monitor
> NetBackup Service Monitor -- STARTED
> NetBackup Bare Metal Restore Master Server
> NetBackup Bare Metal Restore Master Server -- STARTED
> NetBackup Bare Metal Restore Boot Server
> NetBackup Bare Metal Restore Boot Server -- STARTED
Start up of one or more services failed.

D:\Program Files\Veritas\NetBackup\bin>bpps
* EUAYLNBU01                                             4/23/14 09:52:03.584
COMMAND           PID      LOAD             TIME   MEM                  START
vnetd            6924    0.000%            0.062  9.7M   4/23/14 09:38:39.841
bpinetd          1752    0.000%            0.062   11M   4/23/14 09:38:39.970
bpcd             6684    0.000%            0.046   10M   4/23/14 09:38:40.098
nbevtmgr         4476    0.000%            1.232   39M   4/23/14 09:38:40.908
dbsrv11          5280    0.000%            6.567   33M   4/23/14 09:38:41.617
nbemm            1804    0.000%            3.182   45M   4/23/14 09:38:43.154
nbrb             5888    0.000%            1.107   32M   4/23/14 09:38:44.313
nbrmms           2036    0.000%            0.982   31M   4/23/14 09:38:45.341
nbaudit          3564    0.000%            2.121   28M   4/23/14 09:38:46.627
bpcompatd        5656    0.000%            0.062   14M   4/23/14 09:38:47.664
nbsl             1492    0.000%            0.920   29M   4/23/14 09:38:47.807
nbstserv         5008    0.000%            2.979   35M   4/23/14 09:38:50.645
nbars            5772    0.000%            0.920   30M   4/23/14 09:39:16.867
nbsvcmon         4964    0.000%            0.218   20M   4/23/14 09:39:18.128
bmrd             6932    0.000%            0.124   20M   4/23/14 09:39:19.144
bpps             1744    0.000%            0.078  8.4M   4/23/14 09:52:02.505

D:\Program Files\Veritas\NetBackup\bin>

Please find the log file.

[Amarnath_Sathis]

-

[Marianne]

So, EMM is not the problem and dbsrv11 and nbemm stay running.

Any chance you were running demo license and this has expired in the meantime?
Have you checked Windows Event Viewer Application log for errors?

We see other importand services not starting, such as Request Daemon, Database Manager and others.

I should have asked you to create log folders before restarting NBU.

We need to see bprd and bpdbm logs.

If these log folders do not exist under NetBackup\logs, create them and try to start NetBackup Request Service (If it starts up, it will also start Database Manager, Job Manager, Policy Execution Manager.

Check bprd and bpdbm for errors.

[Marianne]

What is the date in bprd log file name?

All I can see in bprd log is this:

10:06:44.733 [10844.8840] <4> bprd: service failed to start, probably because it is already running

The rest of the processes are not running because bprd is not starting.

Please stop all NBU processes again, then check in Windows Task Manager for any 'hung' bprd processes.

When you are sure that no bprd (or any other NetBackup processes (bp... or nb...) processes are running, try to start NBU request service. What happens?

Is anything logged in bprd log?

I am also curious about the IP addresses seen in bprd log:

69.254.104.181 
10.1.10.2 
10.1.10.1 
10.1.10.4

...etc...

Does the master have more than one NIC?
If so, is each IP address bound to different hostname?

Was anything changed at OS level recently as far as network config is concerned?

[Amarnath_Sathis]

stopped all services and tried to start netbackup request daemo. I got the above error.

Yes we have 4 NIC and no changes had been done in OS level recently

[Amarnath_Sathis]

and no bprd logs created

[Amarnath_Sathis]

And I do have one more system error

BXOIS: Login request failed. The login response packet is given in the dump data.

[Marianne]

It is difficult to try and help if you do not answer all questions..........................

Is each IP address bound to different hostname in server's hosts file?

Have you checked Windows Task Manager for any 'hung' bprd processes?
Or any other hung NBU processes?

Is anything logged in bprd log after the startup attempt above? Or in Event Viewer Application log?

[Amarnath_Sathis]

Just now I noticed that license expired but all these days it was running fine. Does this is the reason for bprd not coming up.

And Under D:\Program Files\Veritas\NetBackup\db - data,logs,bin folders are not available. Do we need to create it?

[Marianne]

Maybe time to log a Support call.

The fact that nothing is updated in bprd log file could mean that the bprd binary somehow got damaged or corrupted.
(We still don't know what the date of last bprd log file is...)

Symantec engineer will be able to check file size and checksum with you.

The Event Viewer errors do not seem to be related to NBU.

PS: 

All IP addresses for each of the 4 NICs on the master server should have hosts entries in the master's hosts file, each binding to a different hostname.

Only one IP address should resolve to master server hostname:

euaylnbu01.accoeu.local
EUAYLNBU01.accoeu.local

[Amarnath_Sathis]

Last date of bprd file is 22/04/14 i.e. yesterday.

yes x.x.10.1 is alone pointing master server.

Does license is the reason for bprd starting issue?

[Marianne]

It could be. 

I have seen a similar post here on Connect a couple of months ago with similar symptoms, but here we could clearly see expired license message in bprd log.

Will see if I can find the post.

Related post over here: https://www-secure.symantec.com/connect/forums/error-connection-rejected-pbx-exchange

Oh! Another similar post where the OP never got back to us:

https://www-secure.symantec.com/connect/forums/unable-start-bprd

(Maybe revaroo can check if the Symantec case was ever resolved?)

[revarooo]

If your license has run out I am not surprised NetBackup won't start. You need a new license.

Time to call Customer Services.

[Amarnath_Sathis]

May I know the exact log. So I can go to my client to get it renewed.

[Marianne]

Simply check 'bpminlicense -list_keys -verbose' on the master.

As I have said above - on the other installation we could see licensing error in bprd log.
You may want to check logs for previous days.

[revarooo]

Yes - not all daemons will come up if the license has expired.
 

[revarooo]

Marianne, yes that case was fixed:

Solution: USE_VXSS = Automatic -  deleted the key

[Marianne]

Wonder if this is a similar issue? 
Symptoms seem to match without anything logged in bprd...

Someone tried to enable VxSS/NBAC and 'forgot' to tell anyone?