Remote Media Server Timeout

[JustN]

Good Morning-

All of my media servers exist at different customer locations (mostly in the US, a few overseas) and are all part of the same WAN link, which has its moments. Sometimes things are great, sometimes, well, not so much.

What I am finding is, when this link is a little slow, even for a single media server, the entire Java console becomes very lagged and takes forever to open an element. Example: Today, when I click on Media, while one of the media servers is running a little slow, it takes forever to expand the items below it. I can watch in the command prompt window behind the Java console and see it sitting on a particular server for a long time, then finally says it was unsuccessful.

Is there a timeout settings somewhere I can change? It almost seems like NBU is waiting 5 minutes to hear back from a server, then moving on to the next one.

I believe this also brings up the topic of an offline media server -- what is the typical practice when a media server goes offline? Let's say the power goes out at a facility and the server goes down, this also makes the console operate extremely slow while NBU attempts to contact the host. This can happen at any time, depending on the site -- the only solution I could come up with so far is to ping the host via a crontab and when it fails, mark the host admin offline. Does this make any sense?

Thoughts?

JustN

[Marianne]

I thought only Dark Africa had problems with bad network connections.... I do not have answers on how to make it better. Other than configuring remote sites with problematic connectivity as independent masters. If there is dedupe in the environment, configure AIR to replicate to main site.

[sdo]

Pure guess this... Java Console, View, Options..., Devices (tab), and select "Show connections only for selected devices".    Dunno if it will help at the media server level.   Worth a try?

.

Re:

"I believe this also brings up the topic of an offline media server -- what is the typical practice when a media server goes offline? Let's say the power goes out at a facility and the server goes down, this also makes the console operate extremely slow while NBU attempts to contact the host. This can happen at any time, depending on the site -- the only solution I could come up with so far is to ping the host via a crontab and when it fails, mark the host admin offline. Does this make any sense?"

...sounds like you're doing the right thing.  But, what does "mark the host admin offline" mean?

[sdo]

What the java console setting of:

FORCE_IPADDR_LOOKUP=?                <<< BTW - I've put a question mark in here :)

...for your host that you are running the java admin console on?

[JustN]

Configuring independent masters... Interesting thought... But, that would require me to manage each one of the masters independently, no? I would loose the ability to manage policies, etc... all directly from within a single Java console?

[JustN]

> ...sounds like you're doing the right thing.  But, what does "mark the host admin offline" mean?

The administration guide talks about a process through NBEMMCMD to mark a media server as administratively down, such that if a media server were going to be offline for long enough, you could tell NBU to ignore it. At least, that's the way the guide talks.

My thought was, to ping the remote host every 5 minutes or so and, if the host was unreachable, mark it as administratively down... And if was up, issue the command to mark it online.

I haven't implemented this, but it was a thought.

[JustN]

I am not sure how this would help? The NBU server itself has all of the IP's in a host file (not using DNS) and the lookups seem to be OK, it's just a timeout trying to talk to the remote media server. I'm interested, if you can explain more?

[sdo]

Ok, let's ignore the IP lookup angle.

[sdo]

That's either exactly what I would have suggested, or very similar, let me check commands... brb.

[JustN]

This is what I was thinking: ./nbemmcmd -updatehost -machinename MEDIA SERVER NAME -machinestateop set_admin_pause -machinetype media -masterserver MASTER SERVER

[sdo]

Found it.

I used to use this in scripts to de-activate media servers when doing maintenance:

vmoprcmd -hoststatus       -h med99-local
vmoprcmd -deactivate_host  -h med99-local
vmoprcmd -hoststatus       -h med99-local

...and in DOS/batch test %errorlevel% after each command...

...or in shell scripts test $? after each command.

[sdo]

Maybe:

a) either will work

b) neither will work

c) both required

?

[sdo]

Your command looks a bit more serious though.   ;)

.

I think the vpoprcmd just stops new work being assigned to the media server.

Whereas your command looks like it do what you want.

If it was me, I'd take your advice, and try the admin pause.

[nbutech]

Check the technote for more explaination

https://www.veritas.com/support/en_US/article.TECH77811

[areznik]

Fairly sure both of your commands do the same thing, I think this is also the same as right clicking on the media server in the GUI and selecting Deactivate. 

[sdo]

Yeah, I'm sure the vmoprcmd command is the same as you say, but I think that the nbemmcmd with -machinestateop set_admin_pause goes a step further in forcing admin consoles to ignore a server.  I need to test it, if I get time - as I wasn't aware of that command option before.

[sdo · Accepted solution]

I've noticed two time out variables used by the Java Admin Console.

See NBJAVA_CORBA_DEFAULT_TIMEOUT and NBJAVA_CORBA_LONG_TIMEOUT - I'm looking at the v7.7.2 Admin Guide I right now, but I guess they'll also be documented in the admin guide for your version.  Maybe reducing these will result in a quicker drop/release of element sthat get stuck/hung/waiting in your instance of admin console when devices are no longer reachable.