Clients not reporting in to central server

[Scooby_Doo]

Hi,

Hope I'm posting this in the right place!

I am working for a client who has approx 250 Windows Server 2003 machines at remote sites, all with their own BE 11d (build 7170) installations backing up locally at each remote site. At the central site, we have another 2003 Server running Reporter 6.5. In the last few days, only about 40 of the clients have been appearing in the reports. I'm not overly familiar with the product and can't seem to make head nor tail of the various logs. There is nothing recent displayed in the Alerts section of the console either.

Thanks in advance!

[CRZ]

Do you still have the correct number of modules?

[ Settings » Global Settings » Agents » (click on Agent hostname under "Configured Agents" to list its modules, or click on "Show Details" under "Agent Status" to look for more trouble/alerts) ]

[Scooby_Doo]

Thanks for the reply.

It appears so. All of the modules are listed, but if I then click 'Show Agent Status' I get the following error:

There was an error while getting complete module status for agent. 

 Error Message : Error Communicating with agent.

I have reinstalled the VBR software on the server but to no avail, so I'm going to start checking with individual servers now.

[Scooby_Doo]

FOun this error in the core-0.log on the servers:

2008/07/30 20:06:58 BST agent pool-1-thread-6 FINER ACT.transmitData THROW
com.veritas.gom.agent.shared.AgentException: A219: The module has thrown an error: java.lang.OutOfMemoryError: Java heap space
 at com.veritas.gom.agent.shared.AgentException.createAgentException(AgentException.java:281)
 at com.veritas.gom.agent.core.comm.AgentCommunicationThread.transmitData(AgentCommunicationThread.java:297)
 at com.veritas.gom.agent.core.comm.AgentCommunicationThread.run(AgentCommunicationThread.java:353)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space

Sadly, java errors don't mean much to me!! Looking at the admin guide there are instructions for checking the heap size in Solaris, but not in Windows, so I can't tell what the heap size, although the default is 1GB.

[Scooby_Doo]

Thought I'd update from the situation to make it a little clearer as the original post wasn't clear.

Until a week or so ago, we ran a scheduled tabular report which reported the status of the previous day's backups on all the 250 or so remote servers. This e-mail function is still working, however it is now sending blank reports. Previously my colleague had stopped the VBR Agent service and cleared out C:\Program Files\Symantec\Veritas Backup Reporter\Agent\Data before restarting the service, and after polling again, the reports would work. However, this approach (or rebooting) has so far failed to resolve the issue. After the last reboot however, I can now get the agent status. Whereas previously most were green, the vast majority are yellow now, and display that they last reported about a week ago. Looking under Settings -> Global Settings -> Agents -> Agent Status there are a lot of agents with events ready, but very few completed.

I logged a call with Symantec yesterday but after 30 minutes on hold I had to give up. I called again today but requested a call back so I'll wait and see what happens there.

[Scooby_Doo]

More info! l've looked through the err-0.log and found the following recurring error:

A4012: Error while Job data collection: Return Code: -100 in native code indicates a BE server connection issue

Now, I've followed the instructions listed at http://seer.entsupport.symantec.com/docs/297527.htm, however the settings were already correct, so I merely changed them to *Veritas* and then back again (by copying the install location directly from a server) to see if it would kick into life. So far, at least some are indicating a data load date of today under Agent Details, but most are still yellow and not showing up in the tabular report.

I'll let it continue to poll and see if it's fixed itself in the morning (I'm in the UK, so it's 3:39PM here).

[CRZ]

That's pretty odd.  If you continue to see problems, you will probably want to open a support case (mentioning "A4012"), run the VBR support script and email in the results, so they can look a little deeper into your logs.

On continued faulty clients, it wouldn't hurt to double check and ensure the path didn't mysteriously change as well.  (What version(s) of BE are on the clients?  Working AND not working)

[Scooby_Doo]

You're not kidding! Nothing was changed with the configuration of the agents at all, yet some still seem to report in (now approx 30) whilst others won't. All clients are BE 11d 7170 SP3 with the latest patches. Actually, that's a thought - they might not all have SP3. I'll double check that in the morning.

I'll run the support script first thing in the morning and phone up (again). I've logged a call, but was left on hold for 30 minutes the first time, and requested a call back this morning, which never came, so I'll persevere tomorrow morning.

[Scooby_Doo]

It gets weirder! After making the change to the install path, all the clients reported in for Policy, but Job and Tape Drive were grey and set to 'Not Queued'. Decided to clear out the Data folder and restart again, but this time they all reverted back to yellow and stated that they hadn't reported in since the 23/07!!

I've had to give up logging a support call - I've been trying for 3 days now. First day I was put into the technicians queue but gave up after 45 minutes on hold. The next day, I got the option of a call back, which never came. Today I called to be told that my case ID had been assigned to 2 companies at the same time, (neither of which were me), and was put back to Customer Care to try and retrieve the case. Got through there to be cold transferred through to licensing, who were as bemused as me. Finally got through to someone who then told me that I couldn't log the call as the licence and contract were bought by the outsourcing company who support some of our servers and I couldn't log a call for VBR, only Backup Exec and NetBackup!!

Apologies for the wee rant, but it got it off my chest.

[KEELIN_HART_2]

I have been having similar issues. This was the advice that tech support gave to me -

1. Increase the agent heap size

Windows:

1. Stop the agent service.

2. Run regedit

Go to >

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VBRAgent\Parameters

3. Change value of the "JVM Option Number 2" key from -Xmx256m to

-Xmx1024m

4. Restart the agent.

5. Verify that the agent heart beats are getting updated in the web UI.

Solaris:

Increase Heap size for VBR agent (restart agent after the change)

/opt/VRTSccsva/bin/vbragent
/opt/VRTSccsva/bin/vxccsvcagent

Change “-Xmx256m” to “-Xmx512m”

2. Increase maxheap for Web GUI

Windows:

C:\program files\Veritas\vrtsweb\bin\webgui maxheap

* This will return a value in megabytes the output of which is similar to the example below:

C:\program files\Veritas\vrtsweb\bin>webgui maxheap

1024 MB

To modify the value, issue the command webgui maxheap new_heap_size_in_megs. For example:

C:\program files\Veritas\vrtsweb\bin>webgui maxheap 2048

And then run webgui restart for changes.

Solaris:

Run /opt/VRTSweb/bin/webgui maxheap 2048

Restart VBR server and web server after the change.

[Scooby_Doo]

Did it do the trick? I had tried to reset the size previously but I only found the registry entry (the documentation only refers to Solaris).

I'll try the webgui command anyway.

[Andrew_K]

Hi Keelan

I am having the same problem on VBR6.5.1 for Job collection only.

How exaclty do you increase the agent heap size, as I have tried the webgui heap size but still get the same error.

Thanks

Andrew

[TonyDavids]

If the VBR Agent is on Windows, the heap size is in a registry entry, HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VBRAgent\Parameters\JVM Opton Number 2

-Xmx256m

If your VBR Agent is running on Solaris, you need to change the VBR agent start script in /opt/VRTSccsva/bin.

Tony

[Steven_Forlini]

In case anybody else still has this problem as I have, this is what I did:

1. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VBRAgent\Parameters - change value of the "JVM Option Number 2" key from -Xmx256m to -Xmx1024m (as above)
2. C:\program files\Veritas\vrtsweb\bin>webgui maxheap 1024 (not 2048 as mentioned).
3. Restart webserver (webgui restart) and it worked.