Flash Backup failing with error code 13

[sksujeet]

NBU Master/Media - 7.1 with ws 2008*64

Clients - Running on 7.0.1.4 with ws 2008*64

Backing up the E drive on the ws 2008 and it is faling with the error code 13. The bpfis logs shows that snapshot was successfull.

The E drive is around 1230 GB with 210 GB free.

The normal backup works fine.

Tried moving the shadow copy to different drive thought it might be extensive i/o but still same error message. The disk is recently defragmented seeing this issue but still it fails with the error message. The flash backup never worked for this drive though normal backup works. There are no error whatsoever in the event viewer, the snapshot method used is vss and all the writers on the server are stable and no error.

Any help will be much appreciated.

I am attaching BPBRM,BPTM and BPBKAR.

The bpbrm shows below error message

13:57:27.797 [7024.6768] <16> bpbrm readline: socket read failed, An existing connection was forcibly closed by the remote host.  (10054)
13:57:27.797 [7024.6768] <2> set_job_details: Tfile (147997): LOG 1380347847 16 bpbrm 7024 socket read failed, An existing connection was forcibly closed by the remote host.  (10054)

13:57:39.746 [7024.6768] <2> inform_client_of_status: INF - Server status = 13
13:57:39.746 [7024.6768] <2> put_string: cannot write data to network:  An existing connection was forcibly closed by the remote host.
13:57:39.746 [7024.6768] <16> inform_client_of_status: could not send server status message

[RamNagalla]

bpbkar log does not have the log till the failure.. could you post the entire log.. and also the bpfis log..

[sksujeet]

That is what was generated in bpbkar, the timing is different as that is in different time zone. I will post the bpfis if required but that was successful.

[sksujeet]

Please find the attached bpfis log.

[quebek]

Please update the NICs drivers on clients and servers. When I was facing this error the NIC driver plus its firmware helped to get rid of this error...

[sksujeet]

Thanks for the reply, we had a call going on with Symantec and they are analyzing the logs. Flashbackup is working for other servers so I guess media servers are fine.Also the normal backup for the server itself is working, its only flash backup that is not working.

[huanglao2002]

If normal backup was support,you can try to use it.

[sksujeet]

We have a case going on with Symantec. The appcritical results shows fine and no issues shows with the n/w or the n/w adapter.

Disabled all the firewall and antivirus on the server but no avail.

The server was not rebooted for > 200 days and thought of rebooting it but that didn't resolve the issue.

We compared the nic drivers with the other working server and planning to upgrade the nic drivers. Will keep you posted once done

[Marianne]

Any reason why Master is unpatched?

Although different 'point' versions are supported, my experience with features such as Snapshot Client/FlashBackup that versions should match.

[sksujeet]

I asked them the same :).They have more than 20 Master server in whole environment all running the same version with all client on 7.1.0.4 to get support for 2012.

As per audit they have to keep the version same on all the master servers and will be upgrading all of them to 7.6 in near future. As all the other flash backups working and this one is failing it is hard for me to defend myself.

[Marianne]

So, they won't even considering patching this master if it may fix the FlashBackup issue?

I guess the FB backups stopped working when clients were upgraded to 7.1.0.4?

[sksujeet]

Ya Marianne I understand what you mean!Seeing the fact that other flash backups are working in the same environment and other environments they are not able to buy this suggestion. This is the new client which they wanted to add and flash backup never worked for this.

The n/w drivers are updated now for the client but still it fails with exactly same error message. This is escalated to backline now

[sksujeet · Accepted solution]

Symantec created EEB 3405297 for it.

Engineering have told me the following about the issue:

NetBackup creates a bitmap of the disk blocks to identify which blocks are allocated and which are not.  So it allocates memory to store this bitmap at runtime.

The problem here was that the allocated bitmap size was smaller than the size that we were actually using for operations related to the bitmap.  We derive the allocation size value from bitmap data from the MFT record on the disk.

It looks like there is some problem in the MFT record on that specific disk related to the bitmap data which could have caused the problem here.

Generally we expect that the allocated size be greater than or equal to the size of the actual bitmap data that we use for operations.

We have hit this issue for the very first time, so we are not totally sure what kind of scenarios it can happen in, where we see discrepancies within the bitmap data on the MFT record of the disk.

In the EEB change, we corrected this behavior, so that we actually allocate the size larger of both the values always for bitmap.