After upgrade to 7.7.2 only BAR is available - 5230 Appliance

[HoldTheLine]

Yep it's me again - this 7.7.2 upgrade across all of our environments is turning out to be challenging.  Here is a new one:

After upgrading our 5230 from 7.6.0.2 to 7.7.2 with no apparent issues, logging in with the new Java console only shows the BAR.  I did some searching on this and none of the fixes apply, i.e. auth.conf entries and such - none of the 'common' reasons for this have lead anywhere.  There is a case open with Veritas that has been escalated, they are also stumped and are currently reviewing logs.  Interesting to note is that we can use the same Java console to connect to other environments with no issues so we are ruling out the possibility of the console application itself, for better or worse. 

We seem to have a knack for finding the odd situations don't we!

[Mark_Solutions]

I have noticed that the Java console must be run "as administrator" otherwise you only get the BAR GUI part - i assume it uses some Java components that use local admin right that reflect what is sent to the master?

[HoldTheLine]

RE: "Java console must be run "as administrator" otherwise you only get the BAR GUI part "

There are two  responses to this:

Running the Java console as administrator does not make a difference - still only see the BAR.

Running the Java console as administrator OR local user works fine connecting to two other 7.7.2 environments - one of them is a Windows 2008 R2 master and the other is identicial to the environment we have the BAR issues on, another 5230.

[Mark_Solutions]

Ah - so just when connecting to the one site that has a 5230 Master ... pretty specific then .... anything different between the two appliances?

Wondering about the usual stuff ... one has had its password updated to the new requriements and the other hasn't (depending on when upgraded early on) .... connecting on a different interface so detects it as a client rather than a master.... just thoughts

[HoldTheLine]

Right, only one of the 5230s that has been upgraded has this issue.   I have been looking for differences and there is one pretty major difference; the test system (that works) is a simple environment, just one appliance that is master/media.  The prod system (that doesnt work) is more complicated, several media servers on different networks so the bp.conf has quite a bit more entries i.e. things like preferred networks and of course all the SERVER entries for the media servers.  Although I will say that it has one entry that confuses me:

REVERSE_NAME_LOOKUP = PROHIBITED

I don't see this entry on any other appliance, and it doesn't even make sense to me; my understanding is NBU depends on DNS pretty heavilly so why restrict it?  Either way I tried commenting this entry out and it doesn't help.  But still seems odd to me.

[Mark_Solutions]

My though on the interfaces was if the Java console was connecting via an interface which didn't see the Master as a Master .... so name resolution somewhere.

So by connecting to the right interface ... which would need perhaps a little hosts file entry work on the PC you run the console from and the master so that they talk to each other over a single interface

But it does seem odd that this is only after the upgrade ... or did you use the Windows admin console before then?

Are you using the appliance admin account when you login to the appliance via the Java console?

[HoldTheLine]

"

So by connecting to the right interface ... which would need perhaps a little hosts file entry work on the PC you run the console from and the master so that they talk to each other over a single interface

But it does seem odd that this is only after the upgrade ... or did you use the Windows admin console before then?"

These are both sort of related - we did use the Java console before the upgrade.  And have tried with all of the accounts that are configured - the admin account as well as each of our seperate accounts which also worked before the upgrade.

[Mark_Solutions]

OK - maybe seeing the BAR GUI is a red herring as that is just a client GUI with no authentication as such

You may have a similar issue to part of another thread on here: https://www.veritas.com/community/forums/772-upgrade-you-must-deploy-netbackup-machine-certificate-host

[HoldTheLine]

The link you posted was an issue with another environment that we were having problems with and is currently working :)

I am also thinking the BAR Gui is a red herring, there is obviously something not configured correctly on only this appliance but nobody seems to be able to figure out what it is.

[HoldTheLine]

I am seeing some interesting things in the log file /usr/openv/netbackup/logs/bpjava-msvc but since it contains internal system names and IP addresses I am not able to post them :(

I can show these parts of it because they do not contain any internal information:

 Line 1: 15:02:19.572 [261897] <16> session_dispatch: Request count = 0 tag = 510
 Line 8: 15:02:19.769 [261897] <16> session_dispatch: Request count = 1 tag = 118
 Line 9: 15:02:19.827 [261897] <16> session_dispatch: Request count = 2 tag = 101
 Line 12: 15:02:20.870 [261897] <16> command_LOGON_TO_MSERVER: putenv(BPJAVA_MASTER_IPC_STRING=) failed
 Line 29: 15:02:20.946 [261964] <16> isVxssActive: authentication determination failed, assume none required: (193) VxSS authentication is requested but not allowed
 Line 31: 15:02:21.011 [261897] <16> readCharByChar: read failed(?), sockFd = 0, count = 0 (want = 3072),  errno = 104 = Connection reset by peer
 Line 32: 15:02:21.011 [261897] <16> session_dispatch: recv failed, sockFd = 0, errno = 104 = Connection reset by peer
 Line 37: 15:02:21.015 [261896] <16> poll_listen: can't find file descriptor in polling table

[HoldTheLine]

New development: apparently this appliance thinks it's hostname is <ServerName>.local

Thing is, .local is a domain that is not part of this environment.  It's the wrong domain name, and we can't seem to find where it is coming from.  /etc/sysconfig/network  shows this name, the command 'hostname' returns this name but nobody knows WHY.  If I add an alias to /etc/hosts for this (incorrect) FQDN the console works, but that is not an acceptable solution.

It's like we have gremlins or something.

[Mark_Solutions]

That rings bells to other things i have seen about how the Appliance is actually built .. i will try and dig out the tech note .. an old one is here: http://www.veritas.com/docs/000015210

But it was something that is built in