cancel
Showing results for 
Search instead for 
Did you mean: 

Netbackup volume manager\request daemon\device manager services are not coming up

Amarnath_Sathis
Level 5

Hi

Upon login to admin console I got database is down error message and I tried to restart the service.

Netbackup volume manager\request daemon\device manager services are not coming up. We tried rebooting the server also.

Now I couldnot see data\logs\bin folders under Program Files\Veritas\NetBackup\db. nbdb_ping gives sometimes as available and sometimes as not available.

Product and Version: VRTS NETBACKUP ENTERPRISE 7.1.0.3

Master server operating system:  Windows server 2008 R2 Standard

Please help.

39 REPLIES 39

Amarnath_Sathis
Level 5

May I know where to check VxSS/NBAC is enabled or not?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

See http://www.symantec.com/docs/TECH29205 

Check the registry for USE_VXSS key in   HKEY_Local_Machine\Software\VERITAS\NetBackup\CurrentVersion\Config

Amarnath_Sathis
Level 5

Untitled1.jpg

Amarnath_Sathis
Level 5

No current version folder is under HKEY_Local_Machine\Software\VERITAS\NetBackup\

revarooo
Level 6
Employee

Please print up the output from bpminlicense -list_keys -verbose

Amarnath_Sathis
Level 5

D:\Program Files\Veritas\NetBackup\bin>bpminlicense -list_keys -verbose
xxxxxx-xxxx-xxxx
  file version    = 0x05000000
  time added      = 0x4f5e0123 03/12/12 13:58:59
  hostname        = euaylnbu01.accoeu.local
  product ID      = 6 NetBackup Enterprise Server
  serial number   = 566858
  key version     = 0x05000000
  count           = 0
  server platform = 0 Any platform
  client platform = 0 Any platform
  server tier     = 10 NetBackup Enterprise Server
  client tier     = 0 No tier
  license type    = 1 Permanent
  Site ID         = 2006 Unknown site: 2006
  Feature ID      = 84 SAN Client +
  Feature ID      = 83 PureDisk MS Exchange Agent +
  Feature ID      = 81 Enterprise Vault Agent +
  Feature ID      = 80 PureDisk MS SQL Server Agent +
  Feature ID      = 79 Flexible Disk Option +
  Feature ID      = 78 OpenStorage Disk Option +
  Feature ID      = 76 Virtual Tape Option +
  Feature ID      = 75 Bare Metal Restore +
  Feature ID      = 74 Encryption +
  Feature ID      = 73 Open File Backup +
  Feature ID      = 71 Remote Client Support +
  Feature ID      = 70 Robotic Library Sharing Support +
  Feature ID      = 69 Remote Media Server Support +
  Feature ID      = 68 Microsoft RSM Robotic Libraries +
  Feature ID      = 67 ADIC DAS/SDLC Robotic Libraries +
  Feature ID      = 66 IBM ATL Robotic Libraries +
  Feature ID      = 65 Fujitsu LMF Robotic Libraries +
  Feature ID      = 64 StorageTek ACS Robotic Libraries +
  Feature ID      = 62 Snapshot Client +
  Feature ID      = 61 MS SharePoint Agent +
  Feature ID      = 58 Inline Tape Copy +
  Feature ID      = 54 Vault +
  Feature ID      = 49 Library Based Tape Drives +
  Feature ID      = 46 DataStore +
  Feature ID      = 43 Encryption (Legacy DES 56-bit) +
  Feature ID      = 42 Encryption (Legacy DES 40-bit) +
  Feature ID      = 40 SAP extension +
  Feature ID      = 38 Sybase extension +
  Feature ID      = 37 Informix extension +
  Feature ID      = 36 Oracle extension +
  Feature ID      = 35 Lotus Notes extension +
  Feature ID      = 34 DB2 extension +
  Feature ID      = 33 MS SQL Server extension +
  Feature ID      = 32 MS Exchange extension +
  Feature ID      = 30 Shared Storage Option +
  Feature ID      = 27 NDMP +
  Feature ID      = 22 Additional clients +
  Feature ID      = 20 Base NetBackup +
xxxx-xxxxx-xxxx
  file version    = 0x05000000
  time added      = 0x4c0e2c6a 06/08/10 12:41:30
  hostname        = euaylnbu01.accoeu.local
  product ID      = 6 NetBackup Enterprise Server
  serial number   = 305217
  key version     = 0x05000000
  count           = 0
  server platform = 0 Any platform
  client platform = 0 Any platform
  server tier     = 10 NetBackup Enterprise Server
  client tier     = 0 No tier
  license type    = 1 Permanent
  Site ID         = 2006 Unknown site: 2006
  Feature ID      = 85 PureDisk Option +
  Feature ID      = 81 Enterprise Vault Agent +
  Feature ID      = 79 Flexible Disk Option +
  Feature ID      = 78 OpenStorage Disk Option +
  Feature ID      = 77 NAS SnapVault Option +
  Feature ID      = 76 Virtual Tape Option +
  Feature ID      = 75 Bare Metal Restore +
  Feature ID      = 74 Encryption +
  Feature ID      = 73 Open File Backup +
  Feature ID      = 71 Remote Client Support +
  Feature ID      = 70 Robotic Library Sharing Support +
  Feature ID      = 69 Remote Media Server Support +
  Feature ID      = 68 Microsoft RSM Robotic Libraries +
  Feature ID      = 67 ADIC DAS/SDLC Robotic Libraries +
  Feature ID      = 66 IBM ATL Robotic Libraries +
  Feature ID      = 65 Fujitsu LMF Robotic Libraries +
  Feature ID      = 64 StorageTek ACS Robotic Libraries +
  Feature ID      = 62 Snapshot Client +
  Feature ID      = 61 MS SharePoint Agent +
  Feature ID      = 58 Inline Tape Copy +
  Feature ID      = 54 Vault +
  Feature ID      = 49 Library Based Tape Drives +
  Feature ID      = 48 Additional Drives/Robotics +
  Feature ID      = 46 DataStore +
  Feature ID      = 43 Encryption (Legacy DES 56-bit) +
  Feature ID      = 42 Encryption (Legacy DES 40-bit) +
  Feature ID      = 40 SAP extension +
  Feature ID      = 39 SQL-Backtrack extension +
  Feature ID      = 38 Sybase extension +
  Feature ID      = 37 Informix extension +
  Feature ID      = 36 Oracle extension +
  Feature ID      = 35 Lotus Notes extension +
  Feature ID      = 34 DB2 extension +
  Feature ID      = 33 MS SQL Server extension +
  Feature ID      = 32 MS Exchange extension +
  Feature ID      = 31 Advanced Reporting Option +
  Feature ID      = 30 Shared Storage Option +
  Feature ID      = 27 NDMP +
  Feature ID      = 22 Additional clients +
  Feature ID      = 20 Base NetBackup +

D:\Program Files\Veritas\NetBackup\bin>

 

**** Post edited to remove actual license keys ****

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

No current version folder is under HKEY_Local_Machine\Software\VERITAS\NetBackup\

 

Simply not possible... Unless something is VERY WRONG with this installation!

Have a look at this screenshot:

 

PS:

How did you manage to open the NBU GUI without Request Service running?

Amarnath_Sathis
Level 5

Is each IP address bound to different hostname in server's hosts file?

these IPs are related to servers for different servers

Server's host file is not having entry of below ip

 

No hung bprd process.Have you checked Windows Task Manager for any 'hung' bprd processes?


Or any other hung NBU processes?

no other hung NBU process

Is anything logged in bprd log after the startup attempt above? 

bprd logs is not getting updated

Or in Event Viewer Application log? I got only the below error in 

Failed extract of third-party root list from auto update cab at: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab> with error: A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Nothing wrong with licenses.

 license type    = 1 Permanent

Have you logged a Support call yet?

Amarnath_Sathis
Level 5

Kindly delete server IPs.

Amarnath_Sathis
Level 5

I tried to call Symantec support and they informed me that my contract has been expired.

Amarnath_Sathis
Level 5

Please find server.log

Amarnath_Sathis
Level 5

Hello

Drive has 10% free space.

 

D:\Program Files\Veritas\NetBackup\bin>bpdown -f -v

NetBackup 7.1 -- Shutdown Utility

Shutting down services
> NetBackup Bare Metal Restore Boot Server
> NetBackup Bare Metal Restore Boot Server -- STOPPED
> NetBackup Bare Metal Restore Master Server
> NetBackup Bare Metal Restore Master Server -- STOPPED
> NetBackup Service Monitor
> NetBackup Service Monitor -- STOPPED
> NetBackup Agent Request Server
> NetBackup Agent Request Server -- STOPPED
> NetBackup Storage Lifecycle Manager
> NetBackup Storage Lifecycle Manager -- STOPPED
> NetBackup Key Management Service
> NetBackup Key Management Service -- STOPPED
> NetBackup Vault Manager
> NetBackup Vault Manager -- STOPPED
> NetBackup Service Layer
> NetBackup Service Layer -- STOPPED
> NetBackup Policy Execution Manager
> NetBackup Policy Execution Manager -- STOPPED
> NetBackup Job Manager
> NetBackup Job Manager -- STOPPED
> NetBackup Request Daemon
> NetBackup Request Daemon -- STOPPED
> NetBackup Compatibility Service
> NetBackup Compatibility Service -- STOPPED
> NetBackup Database Manager
> NetBackup Database Manager -- STOPPED
> NetBackup Audit Manager
> NetBackup Audit Manager -- STOPPED
> NetBackup Authorization
> NetBackup Authorization -- DISABLED
> NetBackup Authentication
> NetBackup Authentication -- DISABLED
> NetBackup Deduplication Engine
> NetBackup Deduplication Engine -- STOPPED
> NetBackup Deduplication Manager
> NetBackup Deduplication Manager -- STOPPED
> NetBackup Remote Manager and Monitor Service
> NetBackup Remote Manager and Monitor Service -- STOPPED
> NetBackup Device Manager
> NetBackup Device Manager -- STOPPED
> NetBackup Volume Manager
> NetBackup Volume Manager -- STOPPED
> NetBackup Resource Broker
> NetBackup Resource Broker -- STOPPED
> NetBackup Enterprise Media Manager
> NetBackup Enterprise Media Manager -- STOPPED
> NetBackup Relational Database Manager
> NetBackup Relational Database Manager -- STOPPED
> NetBackup Event Manager
> NetBackup Event Manager -- STOPPED
> NetBackup SAN Client Fibre Transport Service
> NetBackup SAN Client Fibre Transport Service -- STOPPED
> NetBackup Client Service
> NetBackup Client Service -- STOPPED
> NetBackup Legacy Client Service
> NetBackup Legacy Client Service -- STOPPED
> NetBackup Legacy Network Service
> NetBackup Legacy Network Service -- STOPPED

Shutdown completed successfully.

D:\Program Files\Veritas\NetBackup\bin>bpps
* EUAYLNBU01                                             4/23/14 09:37:05.313
COMMAND           PID      LOAD             TIME   MEM                  START
bpps             6324    0.000%            0.078  8.4M   4/23/14 09:37:04.234

D:\Program Files\Veritas\NetBackup\bin>bpup -f -v

NetBackup 7.1 -- Startup Utility

Starting services
> NetBackup Legacy Network Service
> NetBackup Legacy Network Service -- STARTED
> NetBackup Legacy Client Service
> NetBackup Legacy Client Service -- STARTED
> NetBackup Client Service
> NetBackup Client Service -- STARTED
> NetBackup SAN Client Fibre Transport Service
> NetBackup SAN Client Fibre Transport Service -- STARTED
> NetBackup Event Manager
> NetBackup Event Manager -- STARTED
> NetBackup Relational Database Manager
> NetBackup Relational Database Manager -- STARTED
> NetBackup Enterprise Media Manager
> NetBackup Enterprise Media Manager -- STARTED
> NetBackup Resource Broker
> NetBackup Resource Broker -- STARTED
> NetBackup Volume Manager
> NetBackup Volume Manager -- NOT STARTED
> NetBackup Device Manager
> NetBackup Device Manager -- NOT STARTED
> NetBackup Remote Manager and Monitor Service
> NetBackup Remote Manager and Monitor Service -- STARTED
> NetBackup Deduplication Manager
> NetBackup Deduplication Manager -- NOT STARTED
> NetBackup Deduplication Engine
> NetBackup Deduplication Engine -- NOT STARTED
> NetBackup Authentication
> NetBackup Authentication -- DISABLED
> NetBackup Authorization
> NetBackup Authorization -- DISABLED
> NetBackup Audit Manager
> NetBackup Audit Manager -- STARTED
> NetBackup Database Manager
> NetBackup Database Manager -- NOT STARTED
> NetBackup Compatibility Service
> NetBackup Compatibility Service -- STARTED
> NetBackup Request Daemon
> NetBackup Request Daemon -- NOT STARTED
> NetBackup Job Manager
> NetBackup Job Manager -- NOT STARTED
> NetBackup Policy Execution Manager
> NetBackup Policy Execution Manager -- NOT STARTED
> NetBackup Service Layer
> NetBackup Service Layer -- STARTED
> NetBackup Vault Manager
> NetBackup Vault Manager -- NOT STARTED
> NetBackup Key Management Service
> NetBackup Key Management Service -- STARTED
> NetBackup Storage Lifecycle Manager
> NetBackup Storage Lifecycle Manager -- STARTED
> NetBackup Agent Request Server
> NetBackup Agent Request Server -- STARTED
> NetBackup Service Monitor
> NetBackup Service Monitor -- STARTED
> NetBackup Bare Metal Restore Master Server
> NetBackup Bare Metal Restore Master Server -- STARTED
> NetBackup Bare Metal Restore Boot Server
> NetBackup Bare Metal Restore Boot Server -- STARTED
Start up of one or more services failed.

Attached EMM log

Amarnath_Sathis
Level 5

Already the logs are enabled and it is not getting updated now. attached recent logs

I could see only two errors 

1.

Volume Shadow Copy Service error: Unexpected error calling routine ConvertStringSidToSid(S-1-5-21-3525326957-3164864679-108986488-16036.bak).  hr = 0x80070539, The security ID structure is invalid.

Operation:
   OnIdentify event
   Gathering Writer Data

Context:
   Execution Context: Shadow Copy Optimization Writer
   Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Name: Shadow Copy Optimization Writer
   Writer Instance ID: {bb2975f4-a047-4ae3-bc34-73a6cc19c514}

2. 

Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards.  

 DETAIL - 
 1 user registry handles leaked from \Registry\User\S-1-5-21-2528180303-2856184433-1496139042-23208:
Process 460 (\Device\HarddiskVolume2\Windows\System32\svchost.exe) has opened key \REGISTRY\USER\S-1-5-21-2528180303-2856184433-1496139042-23208\Printers\DevModePerUser

Amarnath_Sathis
Level 5

Since License is permanent. May I know what will be the next step to make bprd up.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

This post is getting more and more confusing.

I have just noticed the following in a post above about 2 hours ago:

Just now I noticed that license expired but all these days it was running fine. Does this is the reason for bprd not coming up.

And Under D:\Program Files\Veritas\NetBackup\db - data,logs,bin folders are not available. Do we need to create it?

 

Where are the permanent license keys coming from in bpminlicense output? 

I think the files and folders you are looking for should be under D:\Program Files\Veritas\NetBackupDB\ folder.

Pity your customer never renewed their NBU support contract.

Although licenses are permanent, Support must be purchased/renewed every year.

 

All I can think right now is that something in this installation got corrupted.

Hopefully your customer is doing daily catalog backups?

If so, probably time to remove installation,  re-install to same path and patch to same level as before and recover catalogs....

 

Amarnath_Sathis
Level 5

Yes, We got confused support contract with license which made this confusion. 

Even after we have license, bprd is not coming up.

Amarnath_Sathis
Level 5

Thanks a lot for your updates Marianne. Will check for reinstallation and post the updates.

Amarnath_Sathis
Level 5

Hello

Issue was fixed after renaming a "PROGRAM file" named "program" to "renamed.program" in drive where the netbackup is installed.

Thanks Marianne for your inputs.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I don't understand... Did someone manually try to move NBU installation somewhere else? Or renamed installation path?