NetBackup Windows admin console very slow

[Niall_Porter]

Hi all,

We have NetBackup 6.5.4 Enterprise on a SPARC/Solaris 10 server here.  Personally I use the Java client (jnbSA) to admin the server from my Linux workstation but some of our guys use the native Windows admin gui (i.e. not Java gui) from their PCs to do the same.

This all worked fine until last monday.  I had to rearrange the network interfaces on the server in advance of creating some trunked ports for extra bandwidth to the server so I moved the main NIC for the machine (i.e. the one that the backup and admin clients talk to it on) to a different physical port - in Solaris terms I renamed the "/etc/hostname.ce0" file to "/etc/hostname.ce2" and rebooted the machine.  Prior to that the ce2 interface had been used for something else but that was moved to ce0 so in effect, I've swapped the two interfaces around (and obviously the cables too).

Now, the windows admin gui runs very slow at certain things, in particular accessing the media on the server - the windows guys use this for doing daily tape ejects and imports, the windows gui sits there for ages saying "Opening EMM database".  It does eventually get there after a couple of minutes or so.  The Java gui that I use still works fine, however.

Any ideas?

Thanks in advance,
Niall

[J_H_Is_gone]

As the windows console is so easy to remove and install I would do just that.

un-install from the PC and reinstall -- it may have something to do with something in it having issues finding the server again.

I had the same issue once, and I just un-installed it and reinstalled - worked OK for me then.

[Niall_Porter]

OK I tried removing and reinstalling the Windows client on the affected machine but still no joy - the native Windows admin console app still takes several minutes saying "Opening EMM database" before allowing media-related functions and sometimes when doing ejects, times out saying eject aborted due to errors.  Looking at policies seems to take a long time to get started too.  The Java-based client I use on my workstation is still fine, however, and backup performance is still good.

Is there a known issue using Sun Trunking with NetBackup? 

[Niall_Porter]

I hate to do this but - BUMP - this is still a big issue for us.  The Windows NBU admin console (i.e. native Windows one, not Java version) is *very* slow doing media-related operations.  In particular, when clicking on "Media" in the tree view on the left the hourglass appears with the message "Opening EMM database" and this stays there for a long time (approx 5 minutes).  Doing tape ejects takes a very long time also and sometimes fails with no real explanation.

Can anyone please give me some idea as to how to go about troubleshooting this, or where to look?

[Andy_Welburn]

that have been removed from your environment or switched off that are still referenced by NetBackup?

Check in ALL listings that you can find for references to the above e.g. Master Server properties(inc firewall & servers tab), Media servers etc etc etc.

Name resolution issues for media servers? IP lookup could be slowing things? May seem obvious, but a lot of the slow down you experience could 'simply' be NB looking for resources it can't find (i.e. other hosts).

Couple of T/N's, but these mainly focussing on "missing" media servers:
DOCUMENTATION: The NetBackup Windows Administration console is very slow when attempting to view the...

The NetBackup Administration Console responds very slowly if server names from the Enterprise Media ...

What made you query Sun Trunking? Is this set up correctly? Is there some n/w or resolution issue here maybe?

PS: don't worry about bumping!

***EDIT***
One day, just one day I'm going to read things thru'!! If the slow down happened after your n/w changes then I would have to point the finger there.
We implemented Sun Trunking many moons ago with no issues.

[Niall_Porter]

Hey, thanks for the quick response.

We haven't removed any media servers, the only media services are run on the same machine as the master.  Running "nbemmcmd -listhosts" on the master server showed an old NDMP host that doesn't exist any more so I took that out with nbemmcmd -deletehost, but that hasn't helped.

I tried nbemmcmd.exe -listhosts on the Windows machine running the admin console and got the following error:

NBEMMCMD, Version:6.5.4
Failed to initialize EMM connection.  Verify that network access to the EMM serv
er is available and that the services nbemm and pbx_exchange are running on the
EMM server. (195)
Command did not complete successfully.

Did check those processes and both are running normally.

I also noticed that it does take a while to open the Activity Monitor (about 20-30 seconds) on the Windows admin console, again this displays the "Opening EMM database" message during the waiting time.  Additionally, when I attempt to look inside the "Media" branch of the left tree view under "Media and Device Management" the NBConsole.exe process maxes out the CPU on the machine.  Even after five minutes or so and the media is being displayed in the main panel on the right, that process continues to consume all available CPU.  CPU usage only returns to normal on closing the admin console altogether.

I have enabled the NBConsole logging on the Windows admin box but don't see any gethostbyname type errors, nothing much that gives me any clues at all but I'll provide a sample log file if that's any help.  This afternoon I'm going to try returning the server to only a single NIC with no trunking or similar and see if that fixes the problem, then do the reconfiguration back to where we are now, one step at a time testing each change, to see if I can work out exactly what caused the problem to occur in the first place.  Unless the above gives any ideas of anything further I can try first?

Thanks
Niall

[Andy_Welburn]

GENERAL ERROR: Admin Console hangs after reboot. Errors connecting to Activity Monitor and Storage U...

[marekkedzierski]

I had the same issue .. Windows remote console wanted to resolve all IP addresses defined on Master Server.. I know it's strange but after adding few lines to hosts file on remote console machie with all IP addresses and dns name of Master Server it works fine.

[Niall_Porter]

OK tried that - copied the entire hosts file from the NBU server to the hosts file on the windows box.  Still no improvement.

[marekkedzierski · Accepted solution]

Open java console on the same machine. Find logs of java console. It will show you info about ip addresses that can't be resolved.
My customers had the same issue and I've found that it was a problem with ip addresses configured on Master Server but not defined in dns. Windows console hanged for few minutes but java worked fine. Java console showed error with resolving of ip address but there was no timeout.

[Niall_Porter]

OK I can try that, but where do I find Java console logs?  Or how do I switch this on?  Also, does the java console still exist on a 6.5.4 installation with the native Windows console?

Sorry, not very familiar with the Windows stuff, I'm normally 100% Unix at work! :)

[marekkedzierski]

Java console for windows is available on NB windows installation CD.. (additional products).. Then you have to install maintenance pack for java on windows.
Log files are located at <install path>/veritas/netbackup/logs/user_ops/nbjlogs

[Niall_Porter]

Thankyou so much!  You were right - the java console logs mentioned a hostname of another interface which we use with crossover cable to connect to a DataDomain de-dupe disk backup unit.  It was saying it couldn't resolve this hostname so I put it into the Windows hosts file and bingo, now the Windows console works again!

No idea why the windows client feels the need to try and resolve hostnames like this, any way to switch this off?

Thanks again :)

[marekkedzierski]

:)

I don't know how to switch it off. I didn't have time to open a case on support ;)

You mentioned DataDomain.. Could you tell me more about performance, % of deduplication, deduplication of each type of data (vmware, oracle, file system), disadvantages and why not puredisk ?

marekk

[Niall_Porter]

OK I might see about opening a support case then, it seems the Windows client was trying to resolve all hostnames belonging to the master server.  Strange.

The Data Domain units are great, they do block-level, inline de-duplication.  Our DD565's support up to 60 (I think) concurrent data streams, can't remember the data ingest rate but in NBU terms they back up data at about 2x the speed of LTO4, altho bearing in mind we don't have a dedicated backup network and our NBU server has only PCI NICs so with better master/media servers and more careful architecting they could probably go even faster.  There are also newer, faster models available now than we have.

In terms of compression, we see about 15x compression - currently we have about 83TB of data occupying 5.5TB of space post-dedupe on one unit, about the same on the other.  We have one unit at each site and they do block-level replication to each other outwith the backup window so we always have offsite copies too.

In general, they're very easy to use and I'm very happy with them.  I've never looked at PureDisk, what do you mean by that?