Restore to Solaris client in a Zone does complete but errors out with 'Standard Policy Restore Error' with missing files

[Les_Denniss]

Hello

We are trying to restore to Solaris 10 Client zone. 

When we restore the root path in the Windows GUI it doesn't restore everything (root file system )and errors out with 'Standard Policy Restore Error'

When we do a restore through the java GUI on the client (Host) to the zone on the host, the files more that 10 files have not been restored and seems to hang.

We also get Errno: 18 : Cross-Device 

Does anyone have any suggestions as the best way to restore a Solaris zone?

Thanks in advance

[Marianne]

We need logs to troubleshoot.

On client: tar

On media server: bptm and bpbrm

And the user_ops progress log on the server/admin console where restore was kicked off.

Please also let us know what the restore selection was - just data? Or OS files as well?

Whole root local zone or sparse root local zone ?

The 'Cross-Device' error seems to indicate that an attempt is made to restore device or filesystem that may be shared with the global zone?

[Les_Denniss]

Hi Marianne - Thanks for your response and suggestions, much appreciated.

We have been working on this for a few days now at a disaster recovery test! . The Solaris client we are restoring to has not been changed in any way since last years test which was successfull using NetBackup 7.1 from netbackup master server using Windows admin console GUI

This year we patched the NetBackup client (Host) for the Solaris zone to 7.6.0.1 and also the disaster recovery Master and Media servers to 7.6.0.1. The client zone host is at os level Solaris 10 - 9/10. The backup we used was also taken using 7.6.0.1 so netbackup agents and servers are at the same patch level. Unfortunatly i cannot post entire logs as they contain IP addresses and server names below are extracts.

I have replaced the actual server names to 'clientname' and used << media server >> and << master server >> where aplicable.

As a test we are restoring the  ' vi ' Solaris os executable from /clientname/root/usr/bin to the solaris zone host /tmp just to see if we can get it to restore!

tar file from client (Solaris Host for Zone '/tmp')

08:53:27 (769194.001) INF - TAR STARTED 8732
08:53:27 (769194.001) **LOCALE ERROR** locale <en_GB.ISO8859-15> not found in file </usr/openv/msg/.conf>
08:53:27 (769194.001) Setting network receive buffer size to 32032 bytes
08:53:31 (769194.001) INF - TAR EXITING WITH STATUS = 0
08:53:31 (769194.001) INF - TAR RESTORED 0 OF 1 FILES SUCCESSFULLY
08:53:31 (769194.001) INF - TAR KEPT 0 EXISTING FILES
08:53:31 (769194.001) INF - TAR PARTIALLY RESTORED 0 FILES

Media server  bptm

08:53:30.055 [3444.9512] <16> 769194:bptm:3444:<<media server >>: [ERROR] PDVFS: pdvfs_readcache_lookup: BufferFill failed with error: Input/output error (5)
08:53:30.804 [8840.10848] <2> filter_image: [3444] Min_records before frag switch is 90000, locate available = 1

Media server bpbrm

08:53:20.898 [11192.7948] <2> verify_client: ../bpbrm.c.41690: db_getCLIENT failed for CLIENT_HOSTNAME: clientname
08:53:20.898 [11192.7948] <2> verify_client: ../bpbrm.c.41839: db_getCLIENT failed: 227 227 0x000000e308:53:20.914 [11192.7948] <2> vnet_pbxConnect: pbxConnectEx Succeeded

08:53:32.411 [11192.7948] <2> bpbrm check_for_terminate: exit_status = 0
08:53:32.411 [11192.7948] <2> bpbrm check_for_terminate: unexpected normal child exit
08:53:32.411 [11192.7948] <2> bpbrm write_msg_to_progress_file: (769194.001) Changed /usr/bin/vi to /tmp/vi

08:53:32.411 [11192.7948] <2> bpbrm check_for_terminate: exit_status = 0
08:53:32.411 [11192.7948] <2> bpbrm check_for_terminate: unexpected normal child exit
08:53:32.411 [11192.7948] <2> bpbrm write_msg_to_progress_file: (769194.001) Changed /usr/bin/vi to /tmp/vi

08:53:32.411 [11192.7948] <2> bpbrm check_for_terminate: exit_status = 0
08:53:32.411 [11192.7948] <2> bpbrm check_for_terminate: unexpected normal child exit
08:53:32.411 [11192.7948] <2> bpbrm write_msg_to_progress_file: (769194.001) Could not link /tmp/vi -> /usr/bin/edit. Errno = 18: Cross-device link

08:53:32.411 [11192.7948] <2> bpbrm check_for_terminate: exit_status = 0
08:53:32.411 [11192.7948] <2> bpbrm check_for_terminate: unexpected normal child exit
08:53:32.411 [11192.7948] <2> bpbrm write_msg_to_progress_file: (769194.001) INF - TAR EXITING WITH STATUS = 0

08:53:32.411 [11192.7948] <2> bpbrm main: from client clientname: INF - TAR RESTORED 0 OF 1 FILES SUCCESSFULLY
08:53:32.629 [11192.7948] <2> job_monitoring_exex: ACK disconnect
08:53:32.629 [11192.7948] <2> job_disconnect: Disconnected
08:53:32.629 [11192.7948] <2> bpbrm write_msg_to_progress_file: (769194.001) INF - TAR RESTORED 0 OF 1 FILES SUCCESSFULLY

08:53:32.629 [11192.7948] <2> bpbrm main: from client clientname: INF - TAR KEPT 0 EXISTING FILES
08:53:32.629 [11192.7948] <2> bpbrm write_msg_to_progress_file: (769194.001) INF - TAR KEPT 0 EXISTING FILES

08:53:32.629 [11192.7948] <2> bpbrm main: from client clientname: INF - TAR PARTIALLY RESTORED 0 FILES
08:53:32.629 [11192.7948] <2> bpbrm write_msg_to_progress_file: (769194.001) INF - TAR PARTIALLY RESTORED 0 FILES

08:53:32.629 [11192.7948] <2> bpbrm main: from client clientname: INF - TAR EXITING WITH STATUS = 5 - the restore failed to recover the requested files
08:53:32.629 [11192.7948] <2> vnet_pbxConnect: pbxConnectEx Succeeded 08:53:32.629 [11192.7948] <2> bpbrm main: from client clientname: INF - TAR EXITING WITH STATUS = 5 - the restore failed to recover the requested files

Under user_ops - nothing logged.

I hope this helps? - thanks again

[Les_Denniss]

Just to add we have also used at the test the java GUI from the client, restoring with hardlinks and softlinks checked which failed. We tried again with just hardlinks checked which failed and also just soft links which also failed.

[Les_Denniss]

Our solaris client zone host OS level:

Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
  

[Les_Denniss]

Hello we still have a problem with this has anyone else had this problem restoring a Solaris zone? - Thanks

[Marianne]

Probably best to log a Support call with Symantec.

They will need all of these level 5 logs:

Master: bprd

Media server: bptm and bpbrm

Client: tar

Seems level 5 tar log will be most important as this is where we see:

 INF - TAR RESTORED 0 OF 1 FILES SUCCESSFULLY 

[Les_Denniss]

Hi Marianne. I have logged a support call but thanks for you assistance

[Marianne]

I would love to see what the outcome of the Support call is...

Please remember to update this post with the findings as this will help other users with similar issues.

[Les_Denniss]

Yes I will do that I raised a support call end of last week so hope to hear some news soon. I will keep you posted! - Thanks

[jim_dalton]

vi and edit are hardlinks : they are essentially the same object...they are the same inode when nb restores the hardlink it also makes efforts to manage the links, and cant.

/usr/bin/ex
/usr/bin/vi
/usr/bin/vedit
/usr/bin/view
/usr/bin/edit
 

are all the same object.

This doc is probably useful

http://www.symantec.com/business/support/index?page=content&id=TECH10897&profileURL=https%3A%2F%2Fsy...

[Les_Denniss]

Hi Jim - thanks for your input and tech note link! Are you suggesting that we ensure that hardlinks are checked? We did this from the java GUI on the client and the restore still failed, does this still apply for 7.6?

Thanks

[jim_dalton]

I'm not suggesting that Les...Symantec are and who am I to argue? Sounds to me like theres some doubt about it , so I think a few experiments are in order (for you!). It could be that there has been a regression.  It wouldnt be the first time.

The error is from Solaris: its saying its making a link across filesystems: that violates the hardlink rule, which have to be in the same filesys. That document shows a good example of whats happening. Restoring just that one file and to alternate directory is just a bad selection really, if you were doing it for real the situation probably wouldnt happen as you'd be doing it to the source.  Or equally you might just relink vi to any of its pals.

As the log states NB knows its a link and it then tries to manage the link by leveraging "ln"  and that violates the rule : it cant link from tmp to /usr/bin or /usr or indeed anywhere except /tmp in this case.

Jim

[jim_dalton]

I dont recall any issues with zone restore personally. These are whole root zones...what about yours?

I think the vi issue is a distraction to the main event. And we need to see more of the main event!

So give us some detail as to how you are going about wth the restore, especially the solaris side of things and the filesys management, the zone management.

Jim 

[Les_Denniss · Accepted solution]

Hi Marianne / Jim

With your advice and input I have persevered and managed to get the restore to complete successfully and continued to restore two more additional zones/ containers successfully also. Jim is correct that the vi issue is a distraction as we still cant restore vi to /tmp - but who would do that anyway!!

So what I did! - I will use client_name for the zone being restored.

1. Install the Windows NB Java Admin GUI on the media server which has access (did the backup or stores the replicated data if using AIR ) Ensure the console is at the same patch level as media / master and client.

2. On the zone host create the root directory - cd /client_name then mkdir root

3. Then changed permissions on the ' root ' directory just to do the restore ( changed back after! ) chmod 777 /client_name/root and made sure the was nothing open in root ie other sessions.

4. Open Java NB Console on the media server

Restore Server: < Master Server >

Source: Client_name

Dentination: < Solaris Zone Host >

Type: Standard

5. Selected the date then select alternative location and enter /client_name/root

6. Then in 7.6.0.2 gives you the option to choose media server. Please leave as default !! when i chose the media server it failed!! which is odd!

7. uncheck soft links and ensure you check hard links

8. Then run the restore! - I found the java console can hang. You can monitor it from the Windows GUI on the master server. The restore seems to read all the files before restoring so be patient. I have restored 3 zones/ containers using this method without any issues. The root we restored was around 19GB in size which took around 20 minuites to restore.

Hope this helps someone - Thanks again!

[Les_Denniss]

Hello

Just to add - I have done some tests and found that the issue seems to be where I run the restore from. If I use the java GUI on the client if fails. If I run the restore from the Java GUI on the master server I get errors and then fails.

WRN - Can't create file (WIN32 3: Unknown error) - thought this may be permissions to the root directory I am restoring to hence chmod 777.

Also from the master server the whole restore then errors out with 'socket write error'.

However if I run the restore from the Java GUI on the media server which has the replicated data on its puredisk storage area, I dont need to change permissions on the root directory and there are no errors while restoring.

It is as if the master server cannot access the files (backups) on the media server?? Or has permission somewhere?

So doing the restore, do it from the media server which has the data!

The files for the java NBconsole can be found in the Windows NB install files under 'Addons' / JavaInstallFiles