cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted

Netbackup 7.x Console Freeze and Timeout Error

I have Netbackup running on Windows 2012 and the console is freezing as shown in the attachment, please advise.1-Error.JPG

We performed the following steps but still the error persists

 

On the backup server edit the install_path\VERITAS\Java\setconf.bat as follows:

Modify this default setting.

SET NBJAVA_CORBA_DEFAULT_TIMEOUT=60

Change it to:

SET NBJAVA_CORBA_DEFAULT_TIMEOUT=300

Tags (2)
17 Replies
Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

It is not very likely you can recover this situation - it may be the daemon not responding, and not a java issue.

The fastest method to get back in normal operation may be a re-start of the Netbackup services on the master server.

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

Hello,

I think it's better to restart Netbackup services in the master server.

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

Well, This could be due to a number of issues.

Bouncing of NBU Services generally fixes this . Unless its something else.

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

Bouncing of NBU Services means what do I need to do?

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

This mean you have to stop NBU

bpdown -f -v

and start it back again

bpup -f -v

that's all.

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

Please find attached the command execution results and advice.

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

Hey

Something is NOK within your NBU domain... These four are really needed processes:

> NetBackup Database Manager -- NOT STARTED
> NetBackup Request Daemon
> NetBackup Request Daemon -- NOT STARTED
> NetBackup Job Manager
> NetBackup Job Manager -- NOT STARTED
> NetBackup Policy Execution Manager
> NetBackup Policy Execution Manager -- NOT STARTED

 

Check event viewer to figure out why these were having trouble to start. Also create logging for these processes in 

install_dir\Netbackup\logs\ for

bpdbm, bprd, nbjm and nbpem - set verbose logging for whole NBU services... and search for any clue out there...

here is some link for nbu logging reference guide

https://www.veritas.com/support/en_US/doc/86063237-127664549-0/v41116163-127664549?profileURL=https%...

 

 

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

I see masive applciation events as below

EMM interface initialization failed, status = 334

Systems Logs:

The IPMI device driver attempted to communicate with the IPMI BMC device during normal operation. However the communication failed due to a timeout. You can increase the timeouts associated with the IPMI device driver.

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

@aviltauro 

Please have another look at @quebek 's last post. We need to see logs to understand what is happening here and why services are not starting. 

Has anyone perhaps change the master server's hostname? 
Or changed SERVER entries in Host Properties or Registry so that master server name is no longer at the top of the server list? 

The logs that I am interested in are: bpdbm, bprd and nbemm.

Please do the following: 
Check and confirm that bpdbm and bprd folders exist in <install_dir>Veritas\Netbackup\logs\. 
If not, create these folders. 
Stop NBU. Check with bpps that all processes/services have stopped. 
Start NetBackup again. 

If the same NOT STARTED messages are seen, please collect logs and upload here. 

To collect EMM log, please run the following command (which will collect emm events for the last 10 minutes - assuming you started NBU within the last 10 minutes):

vxlogview -o 111 -t 00:10:00 > C:\Temp\emm.txt
(Assuming C:\Temp exists. Create the folder or use another temp directory that exists.)

Copy newest log file in bprd and bpdbm folders to bprd.txt and bpdbm.txt.

Please upload .txt files. 

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

We have not changed the hostname.

"changed SERVER entries in Host Properties or Registry so that master server name is no longer at the top of the server list?" 

Please explain how to trace the server list order.

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

If Server entries are updated via Host Properties, it will update the Registry. 
From Host Properties, it is more difficult to make a mistake, because the Master server entry is clearly marked. 

When Server entries are added by editing the Registry, the Server entries are all in the same line. Only difference is that the Master name must be first. 

Please collect the logs that I have listed? 
We should get a good indication of what is wrong. 

 

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

Please find attached the requried logs

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

I do not see anything in bprd or bpdbm logs that looks remotely useful. They do not even appear to be legacy logs like I'm expecting... 

In EMM log we can see that there was no hostname change, but something seems to be wrong with NBU databases.
EMM starts and then terminates again. This is probably preventing bpdbm and bprd to start. 

Could you please locate the server.log file in <install_path>\NetBackupDB\log\ folder? 
Please copy to server.txt and upload here. 

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

Please find attached server log file

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

@aviltauro 

I do not see any errors in server log. There are only warnings about fragmentation, but no real issues. 

I am confused about the logs files that you have in bprd and bpdbm folders - the bprd and bpdbm files look like copies of the emm log.  Please have another look in these folders on the master? 
In bpdbm folder, we see ALL_ADMINS log that looks like a legacy log. It simply repeats these 3 lines over and over:

17:25:31.648 [4980.4016] <2> read_legacy_touch_file: Found C:\Program Files\Veritas\NetBackupDB\data\vxdbms.conf; requested from (vxdbms_conf.cpp.1529).
17:25:31.662 [4980.4016] <16> DbmOdbcConnect::initialize_odbc: Odbc connection failed ErrMsg [Sybase][ODBC Driver][SQL Anywhere]Invalid user ID or password, ErrCode -1, Sqlstate 28000, ConnStr DRIVER=NB SQL Anywhere;PROWS=1000;PBUF=8m;CS=utf8;UID=DBM_MAIN;PWD=***************;LINKS=shmem,tcpip{PORT=13785};ServerName=NB_BRS001.MYDOMAIN.NET;DBN=NBDB;CON=DBM_4980_3
17:25:31.668 [4980.4016] <32> UpgradeHelper:Smiley Tongueerform_upgrade: Cannot connect to the database, please ensure services are properly started.

I cannot tell you what is causing this or how to fix it. 
What you really need is to log a call with Veritas Support and have a knowledgeable support engineer use webex to look around, but I guess you do not have active maintenance and that is why you are still on version 7.x? 
Do you perhaps have a support agreement with a partner whou could possibly try to assist? 

You can try to follow steps in this TN to rebuild  vxdbms.conf:  
https://www.veritas.com/content/support/en_US/article.100021292
but I cannot guarantee that this will fix your issue. 
Still worth a try IMHO.

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

I have decided to re-install the software. Shall I re-install (in place install)  or perform a complete uninstall and re-install.

Do I need to re-deploy the agents if I have to perform a complete fresh install of Netbackup?

Highlighted

Re: Netbackup 7.x Console Freeze and Timeout Error

That seems to be very extreme.
I would personally try to fix it so as not to lose backup history.
For a re-install, you will completely uninstall NBU, patch to same level as before, then perform NBU catalog recovery. 

For how long has NBU be running in this environment? 
Do you have a recent good catalog backup?
Without a catalog backup to recover, you will need to start from scratch and start to import backup images from tapes or disk.