error 87 using Netbackup appliance 2.6.0.2

[W5]

We just upgraded our Netbackup appliances from 2.5.3 to the latest 2.6.0.2, we noticed quite a few status code 87 errors which we did not see before the upgrade. Anybody has seen simliar issues or has any ideas?

An example failed job detail is like this:

05/23/2014 11:35:14 - Info bpbrm (pid=20806) from client daa21267app016: TRV - last message being suppressed after 10 occurrences
05/23/2014 11:39:04 - Info bpbkar (pid=6800) 30000 entries sent to bpdbm
05/23/2014 11:45:15 - Info bpbrm (pid=20806) from client daa21267app016: TRV - object not found for file system backup: D:
05/23/2014 11:45:15 - Info bpbkar (pid=6800) bpbkar waited 11523 times for empty buffer, delayed 81235 times.
05/23/2014 11:45:25 - Critical bptm (pid=20881) sts_close_handle failed: 2060017 system call failed
05/23/2014 11:45:25 - Critical bptm (pid=20881) cannot write image to disk, media close failed with status 2060017
05/23/2014 11:45:25 - Info ldcsymapp01 (pid=20881) StorageServer=PureDisk:ldcsymapp01; Report=PDDO Stats for (ldcsymapp01): scanned: 14276287 KB, CR sent: 9755 KB, CR sent over FC: 0 KB, dedup: 99.9%, cache hits: 126699 (95.3%)
05/23/2014 11:45:25 - Critical bptm (pid=20881) sts_close_server failed: error 2060005 object is busy, cannot be closed
05/23/2014 11:45:28 - Info bptm (pid=20881) EXITING with status 87 <----------
05/23/2014 11:45:28 - Info bpbkar (pid=6800) done. status: 87: media close error
05/23/2014 11:45:28 - end writing; write time: 0:44:59
05/23/2014 12:39:28 - job 129499 was restarted as job 129519
media close error  (87)

Thanks,

Jason

[SymTerry]

Just to verify, are you using client side Deduplication?

Also what version is the clients on? There have been some status 87 issues that were resolved by getting the client on the same version as the appliance.

[W5]

Some are using client side dedup, some are not. Some clients are Windows, some are Linux/Unix. Most of the client version are 7.5.0.X.

Thanks,

Jason

[Beavisrulz]

Does anyone have a solution for this yet? We are having the exact same problem with the exact same errors in the job details. Re-running the failed jobs does not help, they fail again.

We upgraded from 7.5.0.7 to 7.6.0.3 on the master server, media servers, and appliances.

Currently, all of our policies and clients are set to use client-side dedupe. I've upgraded the client on a few and the jobs still fail.

Any help would be greatly appreciated! Thank you!

Larry

[watsons]

Does this technote help at all? 

http://www.symantec.com/docs/TECH206337

[Beavisrulz]

No, I don't believe so. I've had jobs that run for as little as 4 minutes and fail, and one of the jobs from last night failed within 12 minutes with a transfer rate of 73000 KB/sec. The clients are local to the appliances also, and most are on a 1G Ethernet connection.

All of my backup jobs ran perfectly fine until I upgraded the appliances a few days ago from 2.5.4 to 2.6.0.3. Most of the clients are running NBU 7.5.0.4, a few at 7.5.0.7. I have not had a chance to upgrade any of them to 7.6.0.3. I just don't see why upgrading the master server and disk appliance would cause these issues. They are supposed to be backwards compatible.

Today I am seeing not only the error 87, but now some error 14 and 24. I tried turning off client-side dedupe and got the same errors. I will upgrade the clients and see if that helps.

Thanks,

Larry

[Beavisrulz]

OK, so I went ahead and upgraded the client software to 7.6.0.3 on 10 clients, re-ran the backups, and 8 out of 10 were successful. 2 jobs still failed with Error 13:

8/28/2014 12:54:20 PM - Info bpbkar(pid=8680) accelerator sent 115614870528 bytes out of 194726662656 bytes to server, optimization 40.6%
8/28/2014 1:05:09 PM - Error bptm(pid=12717) system call failed - Connection reset by peer (at child.c.1306)  
8/28/2014 1:05:09 PM - Error bptm(pid=12717) unable to perform read from client socket, connection may have been broken
8/28/2014 1:05:09 PM - Error bptm(pid=12700) media manager terminated by parent process      
8/28/2014 1:07:05 PM - Info khapbak011(pid=12700) StorageServer=PureDisk:khapbak011; Report=PDDO Stats for (khapbak011): scanned: 199669095 KB, CR sent: 16144300 KB, CR sent over FC: 0 KB, dedup: 91.9%, cache disabled
8/28/2014 1:07:05 PM - Error bpbrm(pid=12652) could not send server status message      
8/28/2014 1:07:05 PM - end writing; write time: 1:12:45
8/28/2014 1:07:06 PM - Info bpbkar(pid=8680) done. status: 13: file read failed      
file read failed(13)

I should NOT have to upgrade the client software just to get the backups to run. Everything worked fine until I upgraded the disk appliances to 2.6.0.3.

Can anyone from Symantec chime in please?

Thanks,

Larry

[D_Flood]

Since you were also getting 14's and 24's at one point, have you taken a look at this discussion (and a "fix" at the end)

https://www-secure.symantec.com/connect/forums/after-upgrading-appliance-2602-backups-fail-intermittantly-error-24-and-14

[Beavisrulz]

Thanks for responding D.Flood.

Well, I made the changes as specified in the TID, waited for the backup jobs to run last night, but still had failures. I'm still seeing errors 14, 13, and 24. Today, all butI believe upgrading the clients did help with some of the jobs, especially the ones with error 87, as long as I also disabled client-side dedup at the policy level.

Still looking, but haven't found a solution yet.

[Mark_Solutions]

I see you have :

8/28/2014 12:54:20 PM - Info bpbkar(pid=8680) accelerator sent 115614870528 bytes out of 194726662656 bytes to server, optimization 40.6%
8/28/2014 1:05:09 PM - Error bptm(pid=12717) system call failed - Connection reset by peer (at child.c.1306)  

That is 10 minutes (roughly) which does still sound like a timeout .. check your appliances timeouts (host properties - timeouts) and set client read and file browse to 1800 and see if that helps... but do read that thread linked above too.