Restores from ops center failing with status 2850

[Eric_Engberg]

I'm having a weird issue with restores started from Ops Center that are failing on a fairly regular basis with status 2850.  If I do the restore from the Java Admin GUI it completes without issue.  Incase it matters the restore is to a different client than the source using a different media server to perform the restore.  It seems like it could possibly be related to file size as the file is always partially restored with the exact file size of '1924538880' or 1.8G.  I enabled logging for tar but it didn't provide any insight into why the restore is failing.  It just says tar failed to restore the file with no reason why and that tar actually exits with a status of 0.  The tar log entry for one of the problem restores is below.  The master server, ops center and all of the media servers and clients for this particular backup and restore are all version 7.6.0.2.

10:26:54 (91496.001) INF - TAR STARTED 17874
10:26:54 (91496.001) **LOCALE ERROR** locale <en_US.iso88591> not found in file </usr/openv/msg/.conf>
10:26:54 (91496.001) Setting network receive buffer size to 32032 bytes
10:28:48 (91496.001) CKP - 91496.001 1409758128 0 0 1 0 0 0 16 /foobar/file
10:29:07 (91496.001) INF - TAR EXITING WITH STATUS = 0
10:29:07 (91496.001) INF - TAR RESTORED 0 OF 1 FILES SUCCESSFULLY
10:29:07 (91496.001) INF - TAR KEPT 0 EXISTING FILES
10:29:07 (91496.001) INF - TAR PARTIALLY RESTORED 0 FILES

[Mark_Solutions]

Do all of your clients and media servers have the OpsCenter server added as a valid server in their servers list?

I have seen something very similar with an error message that meant nothing but once i worked through the logs it was simply an access denied which if i remember correctly was either the client or the media server telling the OpsCenter server that it had no right to request such a restore.

[Eric_Engberg]

I do not currently have the ops center server listed in the configs for any media servers or clients.  It's currently only in the master server's bp.conf.  So I need the 'OPS_CENTER_SERVER_NAME' in bp.conf for media servers and possibly clients as well?

[RiaanBadenhorst]

No, you need it as a SERVER entry as its going entry

[Eric_Engberg]

I added the server entries and it had no change.  I'm fairly confident that isn't the problem since the restore is starting and always stopping at exactly 1924538880 bytes for a file.  If the restore does not contain a file that is over 1924538880 bytes it completes without issue.  This only started occuring after upgrading from 7.5.0.3 to 7.6.0.2.  This backup is new enough that it was backed up using the 7.6.0.2 client so I don't think it is an issue of restoring a 7.5.0.3 backup to a 7.6.0.2 client.  Maybe there is an issue with the version of tar that comes in 7.6.0.2 with files over this particular file size?  I've found no other evidence of that online though.

[Eric_Engberg]

Adding some additonal info

I can restore to the media server itself that is performing the restore just fine using ops center however restoring to the client the restore stops when the file hits the previous mentioned file size.

I performed a stack trace on the tar on the client and found this that may help point somewhere.

write(2, "Unexpected EOF on archive file", 30) = 30

I also tried to see if there was a difference in invoking tar between a restore from Ops Center vs the Java Admin client.  The only difference I can see really is en_US.UTF-8 vs en_US for some of the parameters.

tar -x -v -Y -p -P -I 1409770926 -U 0 -E /usr/openv/netbackup/.rename.28586    -Q -J clnt_lc_messages=en_US.UTF-8 -J clnt_lc_time=en_US.UTF-8 -J clnt_lc_ctype=en_US.UTF-8 -J clnt_lc_collate=en_US.UTF-8 -J clnt_lc_numeric=en_US.UTF-8 -J restoreid=91571.001 -J job_total=1 -J client=restore-client -J requesting_client=masterserver -J browse_client=backup-client -J backup_time=1409749211                                                                                              -f - -J verbose=5 -J disallow_server_file_writes=0
tar -x -v -Y -p -P -I 1409771509 -U 0 -E /usr/openv/netbackup/.rename.30040 -j -Q -J clnt_lc_messages=en_US       -J clnt_lc_time=en_US       -J clnt_lc_ctype=en_US       -J clnt_lc_collate=en_US       -J clnt_lc_numeric=en_US       -J restoreid=91572.001 -J job_total=1 -J client=restore-client -J requesting_client=masterserver -J browse_client=backup-client -J backup_time=1409749211 -L /usr/openv/netbackup/logs/user_ops/root/logs/jbp-24274409771508901319000000409-DvKORJ.log -f - -J verbose=5 -J disallow_server_file_writes=0

[RiaanBadenhorst]

Eliminate OpsCenter from the picture and restore using the Admin Console. Then you'll know where the problem lies, client or OpsCenter config.

[Eric_Engberg]

I've already done that and the problem is definitely with restores from Ops Center.  I have no idea why a restore from ops center would fail at the exact same number of bytes on a file if a file is larger than 1.8gig though.

[Marianne]

2 months later and no solution as yet.

It seems as if you have discovered a bug with OpsCenter.

Hopefully you have logged a Support call with Symantec by now?

[Eric_Engberg]

I meant to reply back to this thread I think.  I did log a support ticket and was told that there is a known bug with restoring large files from Ops Center.  I think the tech article I was given the link to said files larger than 1GB though the problem for me seemed to be files over 1.8GB not 1GB.  I was assured it was the same problem.

Unfortunately I was told this would not be fixed until 7.6.0.4 which is still not out and there were no plans for an EEB to fix this problem.  I will be deploying 7.6.0.4 as soon as it comes out and pray that it doesn't cause any other severe problems.

[CRZ]

Was it this one?

Restores initiated via OpsCenter Simplified File Restore (SFR) do not correctly restore files larger than 1 GB.
 http://symantec.com/docs/TECH222720

The Etrack listed in this doc IS resolved in 7.6.0.4 (and we should probably update the TechNote).

[Eric_Engberg]

Yup pretty sure that was the article I was sent.  Though my restores didn't complete with a status of zero obviously.  I questioned the fact that both the result of the restore as well as the size difference but was assured it was the same problem despite those differences.  Hopefully I don't find out otherwise after upgrading to 7.6.0.4 and hopefully it comes out soon.

[Eric_Engberg]

Well what I feared happened.  I just finished updating to 7.6.0.4 and my issue is not solved.  I figured this would happen when there were significant differences in my issue compared to the TECH article.  Calling support now to reopen this issue.

[Eric_Engberg]

So it seems things have changed slightly.  It's still broken but the results are different than before.  It still fails with status 2850 but it is restoring more of the file than previously.

Source backup:

-rw-rw---- 1 user group 2998280192 Nov 18 00:00 filename

Restore:

-rw-rw---- 1 user group 2435200000 Nov 18 00:00 filename

Log from tar:

15:30:49 (159691.001) INF - TAR STARTED 17274
15:30:49 (159691.001) Setting network receive buffer size to 32032 bytes
15:32:33 (159691.001) CKP - 159691.001 1416346353 0 0 1 0 0 0 16 /dir/filename
15:32:57 (159691.001) INF - TAR EXITING WITH STATUS = 0
15:32:57 (159691.001) INF - TAR RESTORED 1 OF 1 FILES SUCCESSFULLY
15:32:57 (159691.001) INF - TAR KEPT 0 EXISTING FILES
15:32:57 (159691.001) INF - TAR PARTIALLY RESTORED 0 FILES