04-23-2014 01:12 AM
Hi
Upon login to admin console I got database is down error message and I tried to restart the service.
Netbackup volume manager\request daemon\device manager services are not coming up. We tried rebooting the server also.
Now I couldnot see data\logs\bin folders under Program Files\Veritas\NetBackup\db. nbdb_ping gives sometimes as available and sometimes as not available.
Product and Version: VRTS NETBACKUP ENTERPRISE 7.1.0.3
Master server operating system: Windows server 2008 R2 Standard
Please help.
Solved! Go to Solution.
04-25-2014 12:35 AM
Hello
Issue was fixed after renaming a "PROGRAM file" named "program" to "renamed.program" in drive where the netbackup is installed.
Thanks Marianne for your inputs.
04-23-2014 01:31 AM
Sounds like EMM is not running. Have you checked disk space where NBU is installed?
If disk fills up to 99%, NBU will shutdown EMM to prevent corruption of the nbdb database
Please try to stop NBU from cmd and then restart:
bpdown -f -v
(Wait for everything to stop and check screen for messages. Use 'bpps' to confirm that everything is stopped.)
Use 'Services' to restart Symantec Private Branch Exchange.
bpup -f -v
(Check screen for messages)
bpps
If nbemm does not start or starts up and then terminate, please show us emm logs:
vxlogview -o 111 -t 00:10:00
Above command will show emm logs for last 10 minutes. Please adjust -t minutes to include last startup attempt.
Please also check server.log for errors:
File is in <install-path>\Veritas\NetBackupDB\log\
04-23-2014 01:57 AM
Drive where netbackup is installed has 10 % free space.
Starting services
> NetBackup Legacy Network Service
> NetBackup Legacy Network Service -- STARTED
> NetBackup Legacy Client Service
> NetBackup Legacy Client Service -- STARTED
> NetBackup Client Service
> NetBackup Client Service -- STARTED
> NetBackup SAN Client Fibre Transport Service
> NetBackup SAN Client Fibre Transport Service -- NOT STARTED
> NetBackup Event Manager
> NetBackup Event Manager -- STARTED
> NetBackup Relational Database Manager
> NetBackup Relational Database Manager -- STARTED
> NetBackup Enterprise Media Manager
> NetBackup Enterprise Media Manager -- STARTED
> NetBackup Resource Broker
> NetBackup Resource Broker -- STARTED
> NetBackup Volume Manager
> NetBackup Volume Manager -- NOT STARTED
> NetBackup Device Manager
> NetBackup Device Manager -- NOT STARTED
> NetBackup Remote Manager and Monitor Service
> NetBackup Remote Manager and Monitor Service -- STARTED
> NetBackup Deduplication Manager
> NetBackup Deduplication Manager -- NOT STARTED
> NetBackup Deduplication Engine
> NetBackup Deduplication Engine -- NOT STARTED
> NetBackup Authentication
> NetBackup Authentication -- DISABLED
> NetBackup Authorization
> NetBackup Authorization -- DISABLED
> NetBackup Audit Manager
> NetBackup Audit Manager -- STARTED
> NetBackup Database Manager
> NetBackup Database Manager -- NOT STARTED
> NetBackup Compatibility Service
> NetBackup Compatibility Service -- STARTED
> NetBackup Request Daemon
> NetBackup Request Daemon -- NOT STARTED
> NetBackup Job Manager
> NetBackup Job Manager -- NOT STARTED
> NetBackup Policy Execution Manager
> NetBackup Policy Execution Manager -- NOT STARTED
> NetBackup Service Layer
> NetBackup Service Layer -- STARTED
> NetBackup Vault Manager
> NetBackup Vault Manager -- NOT STARTED
> NetBackup Key Management Service
> NetBackup Key Management Service -- STARTED
> NetBackup Storage Lifecycle Manager
> NetBackup Storage Lifecycle Manager -- STARTED
> NetBackup Agent Request Server
> NetBackup Agent Request Server -- STARTED
> NetBackup Service Monitor
> NetBackup Service Monitor -- STARTED
> NetBackup Bare Metal Restore Master Server
> NetBackup Bare Metal Restore Master Server -- STARTED
> NetBackup Bare Metal Restore Boot Server
> NetBackup Bare Metal Restore Boot Server -- STARTED
Start up of one or more services failed.
D:\Program Files\Veritas\NetBackup\bin>bpps
* EUAYLNBU01 4/23/14 09:52:03.584
COMMAND PID LOAD TIME MEM START
vnetd 6924 0.000% 0.062 9.7M 4/23/14 09:38:39.841
bpinetd 1752 0.000% 0.062 11M 4/23/14 09:38:39.970
bpcd 6684 0.000% 0.046 10M 4/23/14 09:38:40.098
nbevtmgr 4476 0.000% 1.232 39M 4/23/14 09:38:40.908
dbsrv11 5280 0.000% 6.567 33M 4/23/14 09:38:41.617
nbemm 1804 0.000% 3.182 45M 4/23/14 09:38:43.154
nbrb 5888 0.000% 1.107 32M 4/23/14 09:38:44.313
nbrmms 2036 0.000% 0.982 31M 4/23/14 09:38:45.341
nbaudit 3564 0.000% 2.121 28M 4/23/14 09:38:46.627
bpcompatd 5656 0.000% 0.062 14M 4/23/14 09:38:47.664
nbsl 1492 0.000% 0.920 29M 4/23/14 09:38:47.807
nbstserv 5008 0.000% 2.979 35M 4/23/14 09:38:50.645
nbars 5772 0.000% 0.920 30M 4/23/14 09:39:16.867
nbsvcmon 4964 0.000% 0.218 20M 4/23/14 09:39:18.128
bmrd 6932 0.000% 0.124 20M 4/23/14 09:39:19.144
bpps 1744 0.000% 0.078 8.4M 4/23/14 09:52:02.505
D:\Program Files\Veritas\NetBackup\bin>
Please find the log file.
04-23-2014 02:20 AM
-
04-23-2014 02:26 AM
So, EMM is not the problem and dbsrv11 and nbemm stay running.
Any chance you were running demo license and this has expired in the meantime?
Have you checked Windows Event Viewer Application log for errors?
We see other importand services not starting, such as Request Daemon, Database Manager and others.
I should have asked you to create log folders before restarting NBU.
We need to see bprd and bpdbm logs.
If these log folders do not exist under NetBackup\logs, create them and try to start NetBackup Request Service (If it starts up, it will also start Database Manager, Job Manager, Policy Execution Manager.
Check bprd and bpdbm for errors.
04-23-2014 03:22 AM
What is the date in bprd log file name?
All I can see in bprd log is this:
10:06:44.733 [10844.8840] <4> bprd: service failed to start, probably because it is already running
The rest of the processes are not running because bprd is not starting.
Please stop all NBU processes again, then check in Windows Task Manager for any 'hung' bprd processes.
When you are sure that no bprd (or any other NetBackup processes (bp... or nb...) processes are running, try to start NBU request service. What happens?
Is anything logged in bprd log?
I am also curious about the IP addresses seen in bprd log:
69.254.104.181 10.1.10.2 10.1.10.1 10.1.10.4
...etc...
Does the master have more than one NIC?
If so, is each IP address bound to different hostname?
Was anything changed at OS level recently as far as network config is concerned?
04-23-2014 03:46 AM
stopped all services and tried to start netbackup request daemo. I got the above error.
Yes we have 4 NIC and no changes had been done in OS level recently
04-23-2014 03:50 AM
and no bprd logs created
04-23-2014 03:54 AM
And I do have one more system error
BXOIS: Login request failed. The login response packet is given in the dump data.
04-23-2014 03:55 AM
It is difficult to try and help if you do not answer all questions..........................
Is each IP address bound to different hostname in server's hosts file?
Have you checked Windows Task Manager for any 'hung' bprd processes?
Or any other hung NBU processes?
Is anything logged in bprd log after the startup attempt above? Or in Event Viewer Application log?
04-23-2014 04:27 AM
Just now I noticed that license expired but all these days it was running fine. Does this is the reason for bprd not coming up.
And Under D:\Program Files\Veritas\NetBackup\db - data,logs,bin folders are not available. Do we need to create it?
04-23-2014 04:38 AM
Maybe time to log a Support call.
The fact that nothing is updated in bprd log file could mean that the bprd binary somehow got damaged or corrupted.
(We still don't know what the date of last bprd log file is...)
Symantec engineer will be able to check file size and checksum with you.
The Event Viewer errors do not seem to be related to NBU.
PS:
All IP addresses for each of the 4 NICs on the master server should have hosts entries in the master's hosts file, each binding to a different hostname.
Only one IP address should resolve to master server hostname:
euaylnbu01.accoeu.local
EUAYLNBU01.accoeu.local
04-23-2014 04:53 AM
Last date of bprd file is 22/04/14 i.e. yesterday.
yes x.x.10.1 is alone pointing master server.
Does license is the reason for bprd starting issue?
04-23-2014 04:59 AM
It could be.
I have seen a similar post here on Connect a couple of months ago with similar symptoms, but here we could clearly see expired license message in bprd log.
Will see if I can find the post.
Related post over here: https://www-secure.symantec.com/connect/forums/error-connection-rejected-pbx-exchange
Oh! Another similar post where the OP never got back to us:
https://www-secure.symantec.com/connect/forums/unable-start-bprd
(Maybe revaroo can check if the Symantec case was ever resolved?)
04-23-2014 04:59 AM
If your license has run out I am not surprised NetBackup won't start. You need a new license.
Time to call Customer Services.
04-23-2014 05:09 AM
May I know the exact log. So I can go to my client to get it renewed.
04-23-2014 05:12 AM
Simply check 'bpminlicense -list_keys -verbose' on the master.
As I have said above - on the other installation we could see licensing error in bprd log.
You may want to check logs for previous days.
04-23-2014 05:20 AM
Yes - not all daemons will come up if the license has expired.
04-23-2014 05:23 AM
Marianne, yes that case was fixed:
Solution: USE_VXSS = Automatic - deleted the key
04-23-2014 05:36 AM
Wonder if this is a similar issue?
Symptoms seem to match without anything logged in bprd...
Someone tried to enable VxSS/NBAC and 'forgot' to tell anyone?