NetBackup 7.5.0.5 Admin Console Error 10 Allocation Error

[blcsenosc]

Everything I have read points to a memory problem, but I have 96GB of RAM on this server and it is used for NOTHING but NetBackup.  The Media  Server is the same configuration and throws the same error.  Need to determine if this error could be caused by something else.  Only other two possible changes to the system were UAC related and the installation of Enterporise Endpoint Server A/V protection.  Any help would be great.  Can't see or edit my policies, just when I had them all perfectly in place.

[Marianne]

You are not giving us any details of what your actual issue is?

You really need to tell us more.

When exactly are you seeing this error? 
I have seen all sorts of weird and wonderful things causing this error, but only solved within context with all relevant info available to troubleshoot.

If you think problem might be UAC related, try the following:

Disable UAC on the server.
If not possible, get into the habit of opening the GUI with right-click, run as Administrator each time you open it.
Verify that your user has local admin rights.

If you think the problem may be AV related, exclude NBU binaries from AV scanning.
See http://www.symantec.com/docs/TECH152328
 

If the problem is only seen when you try to access policies, it is possible that someone may have moved or removed something at filesystem level.

Tell us what you see under <install-path>\veritas\netbackup\db\class.

How many subfolders? Do they correspond with the policy names you expect?
If you go into subfolders, is the structure the same for all the policy names?

[blcsenosc]

Sorry for not being clear.  The error occurs when I open the NetBackup Administration Console and click to expand the policies.

I have a Master Server and 2 Media Servers. The Master Server throws this error when I try to open the Policies.  The Media Server gives me the same error.  The Other Media Server gives me no error, just sits and spins when I try to open the policies.

I tried to backtrace anything that may have changed causing this error and thats what pointed to Endpoint AV, but it was only loaded on the master Server and not the others.

No significant changes have been made to the Master Server, but I'm pretty confident it is not a page file or memory error as suggested.

When I navigate to c:\program files\veritas\netbackup\db\class   All of my policy names are there. The subfolders are all identical so they all look good.

Most importantly, no one else administers this software.

Thanks so much!

[Marianne]

Have you seen my recommendations w.r.t. UAC and AV?

Please double-check....

I have too many times seen how some sort of file corruption or file/folder/link in the wrong place has caused this kind of error.

If UAC and/or AV have been excluded as possible reasons, the only thing that remains is a slow process to determine what is wrong at file level.

This is how I would go about troubleshooting:

Close GUI.
Rename Class folder.
Create new Class folder.
Open GUI. See if you can open Policies. There will obviously be no policies.
If no error, close GUI.
Copy one policy from renamed folder to empty Class folder.
Open GUI. See if you can open Policies.

Carry on like this until you find the culprit that breaks the GUI.

You will obviously have to do this when no backups are due.

Let us know result!

[Yasuhisa_Ishika]

Don't exclude Anti-Virus from candidates since it is possible that AV affect the activity of master server and GUI on each host have trouble by illegal master responce. Try to disable AV on your master server once.

[blcsenosc]

Thanks to each of you for the comments.  In digging deeper, I went back at looked at the application, security and system logs.  There are hundreds of errors in there, but the primary repeated error is the EMM inyterface initialization failed.  Initially, I though of extended memory manager but fear it is referring to the Enterprise Media Manager.  I also looked up the error code 334 and may have found the cuulprit in the sense that Private branch Exchange (PBX) service is not loaded on my server, so I am going to load that from the original NBU master files.

Will keep you updated.  Thanks for your time.

[Marianne]

... Private branch Exchange (PBX) service is not loaded on my server, so I am going to load that from the original NBU master files.

HUH?!!!

Any NBU 7.5 installation automatically installs PBX.

What makes you think it is not installed?

[blcsenosc]

Hi Marianne,

    Read a post that said sometimes it gets deinstalled or uninstalled and I guess thats what happened here.  It wasn't listed in my services, so I reinstalled and I no longer have that error.  What I do have is a faulting application nbproxy.exe (event id 1000) and a faulting application bpdbm.exe.

My NetBackup Policy Execution is now also started but still get the memory allocation error.

[Marianne]

Seems that your installation is seriously broken. 

nbproxy is needed for comms between legacy and new processes.
nbpem (policy execution manager) needs bpdbm to access policy database.

Do you have a recent catalog backup?

I guess it is time to reinstall NBU and recover catalog....

[blcsenosc]

Ideas on reinstall -  Do I need to start with 7.5 then re-install the iterations 7.5.0.4 and 7.5.0.5?

Yes I have a recent catalog backup.

Thanks -  Will tackle again on Monday!

Have a great weekend!

[Marianne]

Just 7.5 followed by 7.5.0.5.  Patches are cummulative - all previous patches are included.

I guess with all these problems no backups are currently running?

[blcsenosc]

Some of the backups are running, however this morning I come in to Licensing Issues (Error 159) .  And this is brand new software!

I have confirmed all my policies and all the systems that write to these policies.  I have a separate issue with an ISCSI initiator that became disconnected so that portion of the backups is broken.

So my question now is, do I need to go to control panel snd uninsall NBU completely or just reinstall the software?

Thanks so much

[Dyneshia]

It seems you are just missing a license or maybe one is expired.  You should always export your licenses before a migration or an uninstall reinstall.  You should be able to go to the licenseing portal and look up you licnese.http://www.symantec.com/en/us/global/licensing/

[blcsenosc]

Thanks Dyneshia,

Know my licenses are current and up to date. If I can't do anything else with this **#$*$#*$*# program, I can add, delete and list keys from the command line.  Working for the gov't, we had an initial license, then 3 extender keys until the product was finally purchased.  That's why, of all things, I would never expect this error code on top of everything else.

[Mark_Solutions]

If you have re-installed pbx then you may have a binary mis-match between that and NetBackup

I would try re-applying the 7.5.0.5 patch first to see if that sorts you out and gets it stable again before doing the uninstall / re-install.

If the licenses are not registering properly then try re-registering them (or removing and re-adding them) using NBLicenseApp.exe on the Master (in the netbackup\bin directory)

Hope this helps

[blcsenosc]

Thanks mark,

There could be a binary mismatch, that makes a lot of sense.  Its ironic though, the Media Server, a completely different server has the same allocation errors.  

Before starting the reinstall, I took all the services down with bpdown -v -f and then brought them back up.  In comparing services from the command line  and the services.msc on the system, most are consistent.  I could be off base, but isn't it important for the NetBackup Authentication and the Netbackup Authorization Services to be running.  They take a minute but DO NOT start from either approach. The only other service that doesn't start is the Deduplication manager.

Are these vital and if so, what could be preventing them from starting as well as what is not allowing me to start them now?

Thanks,

Bill

[Marianne]

When you use the GUI on the media server, it is actually issuing commands on the master server. Only the master server holds databases and configuration.

NetBackup Authentication and the Netbackup Authorization Services will only start if you have configured it (not enabled/configured by default). 

Same with Deduplication manager - it needs separate license key and configuration.

Are all services now started on the master server? Including bpdbm?

Can you now access policies?

See http://www.symantec.com/docs/TECH135186 
and http://www.symantec.com/docs/TECH142421

[blcsenosc]

Working with tech support to completely uninstall and reinstall 7.5.0.0 then 7.5.0.5.  The answer was no, still could not access the policies.

[Mark_Solutions]

A couple of things then, though it sounds like you are re-installing so that may well fix it.

PBX gets installed to "C:\Program Files (x86)\VERITAS\VxPBX\bin\pbx_exchange.exe" so do ensure that path and process is excluded along with all other AV exclusions in the usual NetBackup installation lcoation.

Even though you have a huge amount of RAM in the server doesn't mean it has usable memory - the old restrictions carry through from the early days of computing so it may actually need tuning.

I would reccomend the following:

1. Do not use the /3GB and /PAE switch in the boot.ini (or if you do test it with and without these values)

2. Tune the memory of the server for paging and desktop heap settings - this needs registry entries editing or creating and needs a reboot afterwards:

HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\

New DWORD - PoolUsageMaximum  - Decimal value of 40

Edit DWORD - PagedPoolSize Hex value of FFFFFFFF (this is 8 x F)

HKLM\System\CurrentControlSet\Control\Session Manager\SubSystems\

Edit the Windows SZ key and change the values after "Windows Shared Section =" editing the third value to the next one up (so if it is 1024 make it 2048 etc.)

3. Make sure you are not running out of TCPIP ports. Edit the registry (needs a reboot so do it at the same time as 2. above) :

 HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\
New DWORD – TcpTimedWaitDelay  - Decimal Value of 30

From an Administrative command line run:

netsh int ipv4 set dynamicport tcp start=10000 num=50000

All of this will tune the server to a good point for NetBackup to run. These can also be done on your Windows Media Servers.

I would have expected something to have shown up in either the NetBackup logs or in the Windows Event Viewer to be honest that could point to what is wrong on this server.

Hope this helps

[blcsenosc]

Uninstall  /reinstall is completely unsuccessful.  Removed a lot of reference to Symantec, NetBackup and Veritas in the registry.  Also was sent a technote on PostgreSQL and removed all the references in registry for that. The uninstall was successful, but reinstall hung up several times and led to a blank desktop because of a corrupt local account. Have tried reinstall 3 times and it gets almost through then rolls itself back.

Looking for original disks, gonna shut down this server and reload the Master elsewhere.