The backup of Exchange 2007 Cluster on Windows 2008 R2 Servers with NetBackup Client 7.1 is failing. Netbackup Master Server 188.8.131.52
Error bpbrm(pid=8684) socket read failed, An existing connection was forcibly closed by the remote host. (10054)
9/24/2020 5:34:29 PM - Info nbjm(pid=9948) starting backup job (jobid=1580014) for client riysvmbx-800, policy RIYSVMBX-800_IS_FAILED_online_DW_Backup, schedule Weekly
9/24/2020 5:34:30 PM - estimated 504641431 Kbytes needed
9/24/2020 5:34:30 PM - Info nbjm(pid=9948) started backup (backupid=riysvmbx-800_1600958070) job for client riysvmbx-800, policy RIYSVMBX-800_IS_FAILED_online_DW_Backup, schedule Weekly on storage unit riysvsan-001-hcart2-robot-tld-0
9/24/2020 5:34:32 PM - Info bpbrm(pid=9448) riysvmbx-800 is the host to backup data from
9/24/2020 5:34:32 PM - Info bpbrm(pid=9448) telling media manager to start backup on client
9/24/2020 5:34:32 PM - Info bptm(pid=5032) using 262144 data buffer size
9/24/2020 5:34:32 PM - Info bptm(pid=5032) using 12 data buffers
9/24/2020 5:34:39 PM - Info bpbrm(pid=8684) sending bpsched msg: CONNECTING TO CLIENT FOR riysvmbx-800_1600958070
9/24/2020 5:34:39 PM - connecting
9/24/2020 5:34:41 PM - Info bpbrm(pid=8684) start bpbkar32 on client
9/24/2020 5:34:41 PM - connected; connect time: 0:00:02
9/24/2020 5:34:41 PM - begin writing
9/24/2020 5:47:40 PM - Error bpbrm(pid=8684) socket read failed, An existing connection was forcibly closed by the remote host. (10054)
9/24/2020 5:47:40 PM - Error bptm(pid=8672) socket operation failed - 10054 (at child.c.1304)
9/24/2020 5:47:40 PM - Error bptm(pid=8672) unable to perform read from client socket, connection may have been broken
9/24/2020 5:47:40 PM - Info bpbrm(pid=9448) child done, status 13
9/24/2020 5:47:40 PM - Info bpbrm(pid=9448) sending message to media manager: STOP BACKUP riysvmbx-800_1600958070
9/24/2020 5:47:41 PM - Info bpbrm(pid=9448) media manager for backup id riysvmbx-800_1600958070 exited with status 150: termination requested by administrator
9/24/2020 5:47:41 PM - end writing; write time: 0:13:00
file read failed (13)
Files system backups are fine but Exchange Backup is failing.
bpbkar log on client stops with following last message always:
"Active Directory Application Mode"
When we try to browse the Backup Selection on Exchange policy, Information store is not shown at all.
bpbkar log from client with enhanced verbose debug level.
Have you tried to check Application log on the Exchange server for errors?
Or checked for Service logon that may need to be changed or expired password?
With literally everything in this environment no longer supported from Microsoft or Veritas, you are going to battle to find support.
Any chance that the environment can be upgraded/migrated to supported OS, Exchange and NBU?
Client's Application => Netbackup client log or Exchange logs from client ?
We did not find any event in system event logs related to Application or Application Error. BPBKAR is crashing and we did not observe any event. Please guide if we can some how force NB Client to generate event.
We are already suffering without support so I posted it here to get some experts opinion. We are planning to upgrade and migrate but for that purpose backup is even more important now.
We did not see any Application events in Event viewer. How can we verify Service Logon? any technote to refere.
It looks like NB Client is unable to talk to Exchange to get the data set TAR file created for backup. BPBKAR crashes just before it created the TAR for backup. Secondly, in the policy when we browse for storage groups and information store, it shows nothing, just the exchange server name.
We tried upgrade and reinstallation of client on second node of exchange cluster but "Socket error 13" remains same. Only difference was that we could observe BPBKAR crash event logged on this node:
Faulting application bpbkar32.exe, version 7.1.2011.203, time stamp 0x4d4b1fda, faulting module MSVCR80.dll, version 8.0.50727.6195, time stamp 0x4dcdd833, exception code 0xc0000005, fault offset 0x000000000001e38c, process id 0x3d4,
application start time 0x01d6925e67c54c7e.
We verified the library MSVCR80.dll on both nodes and other working clusters. It was leading us to other errors but the situation was same, bpbkar was crashing exactly after 10-13 minutes of "Begin writing" where it should create the TAR to write. so we failed over to the node where old client was available. These clients and configurations were working fine a week before. There is no update, patching or configuration changes in NetBackup.
Since the socket failure error was appearing after exactly 10 minutes so we changed and doubled timeout values also but no success.
We can update NB Client to 7.6 to test it but did it support Exchange 2007 on Windoes Server 2008 R2?
Thanks Marianne, Really appreciated your quick response.
We will try upgrade and test.
Do we have any way to test connection between NB Client and Exchange ? We want to verify that there is no libraries, permissions or authorization issues on client and application by passing Master Server, Network and Policies.
The filesystem backups are working fine with same client and we have issues with only Exchange backup which was also working fine one week before. So we suspect some connection issues betweek NB Client and Exchange.
Top question: Has this ever worked? If yes, what changed?
Before NetBackup 7.6, you needed to execute 3 services on the client as a domain admin:
- NetBackup Client Service
- NetBackup Legacy Client Service
- NetBackup Legacy Network Service
7.6 introduced the option to run these services as LocalSystem and configure the Exchange credentials in the client host properties. (Beginning with Exchange 2013, you must configure the credentials in the client host properties.)
If you can't browse for the backup selections in the policy, how are you setting them? What is your backup selection?
With Exchange 2007, you could do snapshot backups, or you could use the Exchange streaming backup feature. Which are you doing?
When you say you have Exchange 2007 cluster, which flavor of cluster is it? I vaguely remember letters like CCR and LCR.
We have successfully installed NB Client 7.6 and tested successful file system backup. We are facing problem with Exchange backup only.
The services are running as LocalSystem and no credentials in Exchange credentials in client host properties. This was working fine with this cluster and similar configurations are working with another cluster.
We used to browse and selected storage groups and they are still in the policy in backup selection but if we try to browse it now, we see nothing except the name of the exchange.
It is snapshot backup and CCR Cluster.
Suggestions will be appreciated.
How can we see and log the activity of NB Client connections in Exchange Cluster logs ? any ideas?
Any changes on Exchange cluster may impact backups ?
Exchange CCR was just Exchange 2007 on two Windows servers replicating their data to each other. As far as NetBackup connections were concerned, they were two distinct clients. You should find logs on both Exchange servers.
For CCR, one server was active and the other passive. Exchange could fail over from one server to the other. In later Exchange versions individual databases have an active copy on one server and passive copies on zero or more other servers.
NetBackup snapshot Exchange backups involve two process, bpfis.exe for the snapshot and bpbkar32.exe to back up the snapshot to the media server. CCR added a complication if the policy was configured to back up the passive copy. Bpbrm would start bpfis on the active CCR node. Bpfis would detect that it should be running on the other node and return to bpbrm, which would then execute bpfis on the passive node. (There was a bug in that dance move at one point. I think I fixed it in 7.6.0.x, but I don't remember x.)
That's my next point. NetBackup releases through 7.6 had companion maintenance releases. Even when leaving the master server on the GA release, you should always have installed the highest triple-dot maintenance release on the clients (184.108.40.206, 220.127.116.11, 18.104.22.168). This gets you lots of fixes, and in the case of 22.214.171.124 and later you get an upgrade to the BEDS backend database agents.
And yes, you should configure Exchange credentials in 7.6 even if you hadn't done so previously.
It would be easier to give more specific answers if you were on a supported version of NetBackup. NetBackup support for Exchange 2007 ran until version 8.1.