cancel
Showing results for 
Search instead for 
Did you mean: 

Netbackup 7.x Console Freeze and Timeout Error

aviltauro
Level 4

I have Netbackup running on Windows 2012 and the console is freezing as shown in the attachment, please advise.1-Error.JPG

We performed the following steps but still the error persists

 

On the backup server edit the install_path\VERITAS\Java\setconf.bat as follows:

Modify this default setting.

SET NBJAVA_CORBA_DEFAULT_TIMEOUT=60

Change it to:

SET NBJAVA_CORBA_DEFAULT_TIMEOUT=300

17 REPLIES 17

Nicolai
Moderator
Moderator
Partner    VIP   

It is not very likely you can recover this situation - it may be the daemon not responding, and not a java issue.

The fastest method to get back in normal operation may be a re-start of the Netbackup services on the master server.

Bessam
Level 4

Hello,

I think it's better to restart Netbackup services in the master server.

iVaibhav
Level 4

Well, This could be due to a number of issues.

Bouncing of NBU Services generally fixes this . Unless its something else.

Bouncing of NBU Services means what do I need to do?

quebek
Moderator
Moderator
   VIP    Certified

This mean you have to stop NBU

bpdown -f -v

and start it back again

bpup -f -v

that's all.

Please find attached the command execution results and advice.

quebek
Moderator
Moderator
   VIP    Certified

Hey

Something is NOK within your NBU domain... These four are really needed processes:

> NetBackup Database Manager -- NOT STARTED
> NetBackup Request Daemon
> NetBackup Request Daemon -- NOT STARTED
> NetBackup Job Manager
> NetBackup Job Manager -- NOT STARTED
> NetBackup Policy Execution Manager
> NetBackup Policy Execution Manager -- NOT STARTED

 

Check event viewer to figure out why these were having trouble to start. Also create logging for these processes in 

install_dir\Netbackup\logs\ for

bpdbm, bprd, nbjm and nbpem - set verbose logging for whole NBU services... and search for any clue out there...

here is some link for nbu logging reference guide

https://www.veritas.com/support/en_US/doc/86063237-127664549-0/v41116163-127664549?profileURL=https%...

 

 

I see masive applciation events as below

EMM interface initialization failed, status = 334

Systems Logs:

The IPMI device driver attempted to communicate with the IPMI BMC device during normal operation. However the communication failed due to a timeout. You can increase the timeouts associated with the IPMI device driver.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@aviltauro 

Please have another look at @quebek 's last post. We need to see logs to understand what is happening here and why services are not starting. 

Has anyone perhaps change the master server's hostname? 
Or changed SERVER entries in Host Properties or Registry so that master server name is no longer at the top of the server list? 

The logs that I am interested in are: bpdbm, bprd and nbemm.

Please do the following: 
Check and confirm that bpdbm and bprd folders exist in <install_dir>Veritas\Netbackup\logs\. 
If not, create these folders. 
Stop NBU. Check with bpps that all processes/services have stopped. 
Start NetBackup again. 

If the same NOT STARTED messages are seen, please collect logs and upload here. 

To collect EMM log, please run the following command (which will collect emm events for the last 10 minutes - assuming you started NBU within the last 10 minutes):

vxlogview -o 111 -t 00:10:00 > C:\Temp\emm.txt
(Assuming C:\Temp exists. Create the folder or use another temp directory that exists.)

Copy newest log file in bprd and bpdbm folders to bprd.txt and bpdbm.txt.

Please upload .txt files. 

We have not changed the hostname.

"changed SERVER entries in Host Properties or Registry so that master server name is no longer at the top of the server list?" 

Please explain how to trace the server list order.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

If Server entries are updated via Host Properties, it will update the Registry. 
From Host Properties, it is more difficult to make a mistake, because the Master server entry is clearly marked. 

When Server entries are added by editing the Registry, the Server entries are all in the same line. Only difference is that the Master name must be first. 

Please collect the logs that I have listed? 
We should get a good indication of what is wrong. 

 

Please find attached the requried logs

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I do not see anything in bprd or bpdbm logs that looks remotely useful. They do not even appear to be legacy logs like I'm expecting... 

In EMM log we can see that there was no hostname change, but something seems to be wrong with NBU databases.
EMM starts and then terminates again. This is probably preventing bpdbm and bprd to start. 

Could you please locate the server.log file in <install_path>\NetBackupDB\log\ folder? 
Please copy to server.txt and upload here. 

Please find attached server log file

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

@aviltauro 

I do not see any errors in server log. There are only warnings about fragmentation, but no real issues. 

I am confused about the logs files that you have in bprd and bpdbm folders - the bprd and bpdbm files look like copies of the emm log.  Please have another look in these folders on the master? 
In bpdbm folder, we see ALL_ADMINS log that looks like a legacy log. It simply repeats these 3 lines over and over:

17:25:31.648 [4980.4016] <2> read_legacy_touch_file: Found C:\Program Files\Veritas\NetBackupDB\data\vxdbms.conf; requested from (vxdbms_conf.cpp.1529).
17:25:31.662 [4980.4016] <16> DbmOdbcConnect::initialize_odbc: Odbc connection failed ErrMsg [Sybase][ODBC Driver][SQL Anywhere]Invalid user ID or password, ErrCode -1, Sqlstate 28000, ConnStr DRIVER=NB SQL Anywhere;PROWS=1000;PBUF=8m;CS=utf8;UID=DBM_MAIN;PWD=***************;LINKS=shmem,tcpip{PORT=13785};ServerName=NB_BRS001.MYDOMAIN.NET;DBN=NBDB;CON=DBM_4980_3
17:25:31.668 [4980.4016] <32> UpgradeHelper::perform_upgrade: Cannot connect to the database, please ensure services are properly started.

I cannot tell you what is causing this or how to fix it. 
What you really need is to log a call with Veritas Support and have a knowledgeable support engineer use webex to look around, but I guess you do not have active maintenance and that is why you are still on version 7.x? 
Do you perhaps have a support agreement with a partner whou could possibly try to assist? 

You can try to follow steps in this TN to rebuild  vxdbms.conf:  
https://www.veritas.com/content/support/en_US/article.100021292
but I cannot guarantee that this will fix your issue. 
Still worth a try IMHO.

I have decided to re-install the software. Shall I re-install (in place install)  or perform a complete uninstall and re-install.

Do I need to re-deploy the agents if I have to perform a complete fresh install of Netbackup?

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

That seems to be very extreme.
I would personally try to fix it so as not to lose backup history.
For a re-install, you will completely uninstall NBU, patch to same level as before, then perform NBU catalog recovery. 

For how long has NBU be running in this environment? 
Do you have a recent good catalog backup?
Without a catalog backup to recover, you will need to start from scratch and start to import backup images from tapes or disk.