my master server info are as follow:
- OS version AIX 7.1
- netbackup agent version 8.1.1
Recently upgraded the master and media servers to the version 8.1.1.
Not long after the upgrade we start having the errors when performing backups as per mentioned above.
My clients are a combination of version 6.5, 7.1, 7.7 and above.
Have logged the case to the technical support and their respond was to upgrade the clients to the same level as master server.
It will be quite hard for us as we're having more than 500 clients, and by upgrade each of the client, will it be able to solve the current happening issue?
Hoping for guidance for this
With clients on NBU 6.5 and 7.1 you need to understand that there is NO support for these versions for a number of years now. You are lucky if anything is working with these old versions.
Even NBU 7.7 clients should be upgraded as well as this version is EOSS phase and 'extented support' is needed if you need support. (See https://sort.veritas.com/eosl )
Error 7643 is about Security Certificates.
What have you done to exclude the back-level clients from certificate-based communications?
Have you read in NetBackup Read This First for Secure Communications about “How NetBackup 8.1 hosts communicate with NetBackup 8.0 and earlier hosts” ?
NetBackup status code 8500 is related to web service on the master server.
From Status Code manual:
Message: Connection with the web service was not established
Explanation: The connection with the web service was not established.
Recommended Action: Check that the NetBackup Web Management Console is up and running. If it is not running, start it with the nbwmc -start command.
When my backup went to "incomplete" status especially when hitting errors like 8500 and 7643, I tried to resume it back for the 2nd attempt, the backup were able to run and complete without any errors.
Furthermore I'm having a lot of intermittent response from the netbackup console lately since the netbackup upgrade.
Is it somehow related with communication (network) issue?
As Marianne Said, you need to pretty much disable security on your master to make it work with these older clients.
I found several issues with any media server below 8.1 - please please please upgrade as soon as you can.
Unix can upgrade from the master, without even touching the clients - NOTE - you need 4GB in tmp on the clients or it will fail!
Unfortunately version upgrade is not an option due to environment dependencies.
There are other servers which are using the same OS and NBU version but did not encounter such issue.
Any other option besides of the netbackup version upgrade?
Regarding this issue, have you checked this :
Failure during communication with 8.0 or earlier hosts
If insecure communication is not allowed in NetBackup, communication with 8.0 and earlier hosts fails.
For successful communication with 8.0 and earlier NetBackup hosts, use one of the following methods:
• In the NetBackup Administration Console on the master server host, select the Security
Management > Global Security > Hosts > Enable insecure communication with NetBackup
8.0 and earlier hosts option.
• On the master server host, run the following
I know that you have other servers for which backup is successful, we just need to confirm that the insecure communication is enabled.
You said that when the job is in an incomplete state and you rerun it, it works, I'm assuming that the failed job was during the night, have you checked if there is an FW rule or GPO that closes some needed ports during the night? or anything like that?
I also experienced status 8500 after upgrading my master server (Win2008 R2) and some media servers (Solaris 10 and AIX 7.2) to 8.1. I ONLY got errors 8500 with media servers in version 8.1 (Solaris 10 and AIX 7.2) for Solaris clients in version 7.6.x, 7.7.x and 8.1 and for AIX clients in version 8.1. I NEVER got errors 8500 with media servers in version 7.7.x.
I opened a case with Veritas support: they asked to install EEB 3957949 and to enable WAL (Write-Ahead Logging) mode in the whitelist cache database but this did not solve the pb.
Of course the option "Enable insecure communication with Netbackup 8.0 and earlier..." was checked.
I upgraded the master server to 8.1.2 and the errors 8500 disappeared. This pb was clearly due to the secure communication between master server and media servers in version 8.1. When the communication between master server and media server was insecure (media servers in 7.7.x) there was no error 8500.
Currently I'm still backing up many clients in version 7.6.x and 7.7.x. I know these versions are not supported anymore but it works. But you should upgrade your clients in 6.5 and 7.1: these are really very old NBU version.
I know that upgrading your master server is not an option for you (as it's an AIX and last compatible NBU master/media version is 8.1.1 for AIX) but anyway you should move your master server to another platform/OS (Linux, Solaris) and upgrade to 8.1.2 or 8.2.
On the contrary @Marianne The issue seems to have calmed down, for now.
Suddenly the issue stop occurring and so far no such errors arising anyomre.
I found out that it has something to do with the network condition. Although I am not sure why it was behaving so.
I stumbled upon a technote saying that is has something to do incoming connections being dropped (running the command netstat -s | grep queue showed a lot of incrementing count value. (http://www.veritas.com/support/en_US/article.100043398)
FYI, I was having a lot of difficulties in operatng the netbackup admin console as well where I kept getting network intermittent erros and other stuffs (daemon not running, connect to port failed, etc)
That's why I have asked whether this could also be related to network issue or not? I did asked the network guys but no abnormalities showed from their end.
Hello @azmie_ramly ,
As long as you are noticing intermittent network problems, it must be from that from where you are getting the errors, especially when you say that when you rerun the job it works.
The master server is all the time communicating with its servers (especially media servers) so if there is a network issue between them, you will "suffer" ( I used this word because I used to work on a similar infra with network issues, and it wasn't really easy to get it work or even troubleshoot, because sometimes it works and sometimes doesn't)