Need Path to Tape help urgently...Help!

[Hanzo581]

Alright, we've been dealing with this problem for over a week now.  Need help urgently.

We just recently installed a Quantum DXi8500 (configured as VTL) to compliment our Quantum i2000 physical library.  We can write data to the DXi fine, it's when we go to duplicate the data off to physical tape where we run into problems.  We get HUGE spikes in network traffic that basically shuts down our network.  We have our master server set up as NDMP host, when we dupe from it it goes over fiber as it should and we get no spike, but any other media server and it goes over ethernet....the way we were sold this was the media server would just coordinate the data being pushed from the VTL to physical, and it wouldn't have to perform any read/writes....but it isn't happening that way.

We've contacted support numerous times and are basically getting nowhere.  Is there some way to make the media servers facilitate path to tape like our master server does without using ethernet?

Thanks!

[Nicolai]

The media server that did the backup will also perform the duplication.

Have you set any servers as "alternate read server" in the SLP configuration ?

[Hanzo581]

We've tried selecting the master server/NDMP host as alternate read server in both SLP and Vault and that didn't seem to work.

[mph999]

Do you mean you wish to use optimised duplication, so the data goes straight from VTL > Physical tape. This is infact called Direct Copy to Tape, but is very similar to optimised duplication. If so, lookup Direct Copy to Tape in the guides: Openstorage Solutions Guide for Disk (ver. 7.5) NDMP Amin Guide (it uses NDMP protocol, hence why NDMP guide is required, search for NDMP Direct Copy) There may be settings on the Quantum Devices also, I don't have confirmation of this. One catch with this, which will stop it working is explained in TECH206370 Logs for issues would be bptm (from both read and write media servers) and admin from master. The admin log will show if direct copy will be used.

[Hanzo581]

Yes, we are looking to go straight from VTL to Physical tape, it works that way with our master/NDMP host, but not with any other images written with other media servers.

[Marianne]

You need to config the media servers the same way as the master server.  

Covered over here:
Chapter 5 of NetBackup for NDMP Administrator's Guide http://www.symantec.com/docs/DOC5164  

About NDMP DirectCopy

NDMP DirectCopy operates in the following environments:

■ A NetBackup media server that is connected to a VTL that has access to a

physical tape library. The steps for configuring NDMP DirectCopy are described

in this topic.

Also see NDMP DirectCopy notes and limitations on p.33.

Configuration also in this series of HOWTO docs:

http://www.symantec.com/docs/HOWTO89213 
Configuring OpenStorage direct to tape

See this forum post that could be similar to your environment:  https://www-secure.symantec.com/connect/forums/quantum-dxi-path-tape#comment-5908331

[Hanzo581]

So each media server needs drives shared from the NDMP host?

[mph999]

No, the drives only need to be attached to the VTL, although they 'could' be shared with media server(s) if you wish. If you read the chapters in the guides I (and Marianne) posted it explains how to set it up.

[Hanzo581]

Ok, we've got the path to tape thing figured out.  We can dupe to physical tape fine...but now, just when we think we're good something else breaks.  Now our drives for master are flip flopping between PEND-TLD and SCAN-TLD and just our windows jobs are having trouble mounting virtual tapes from our VTL.

10/24/2013 12:42:02 PM - Info nbjm(pid=4360) starting backup job (jobid=9353) for client finms-nbmaster, policy z-test_finms-nbmaster, schedule Onetime 
10/24/2013 12:42:02 PM - Info nbjm(pid=4360) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=9353, request id:{EE9110F2-55F3-4AA9-9425-7BF5691C85CE}) 
10/24/2013 12:42:02 PM - requesting resource nbmaster_tld0
10/24/2013 12:42:02 PM - requesting resource finms-nbmaster.NBU_CLIENT.MAXJOBS.finms-nbmaster
10/24/2013 12:42:02 PM - requesting resource finms-nbmaster.NBU_POLICY.MAXJOBS.z-test_finms-nbmaster
10/24/2013 12:42:03 PM - granted resource finms-nbmaster.NBU_CLIENT.MAXJOBS.finms-nbmaster
10/24/2013 12:42:03 PM - granted resource finms-nbmaster.NBU_POLICY.MAXJOBS.z-test_finms-nbmaster
10/24/2013 12:42:03 PM - granted resource AA1479
10/24/2013 12:42:03 PM - granted resource IBM.ULTRIUM-TD4.056
10/24/2013 12:42:03 PM - granted resource nbmaster_tld0
10/24/2013 12:42:03 PM - estimated 241781869 Kbytes needed
10/24/2013 12:42:03 PM - Info nbjm(pid=4360) started backup (backupid=finms-nbmaster_1382632923) job for client finms-nbmaster, policy z-test_finms-nbmaster, schedule Onetime on storage unit nbmaster_tld0
10/24/2013 12:42:03 PM - started
10/24/2013 12:42:04 PM - Info bpbrm(pid=8412) finms-nbmaster is the host to backup data from    
10/24/2013 12:42:04 PM - Info bpbrm(pid=8412) reading file list from client       
10/24/2013 12:42:04 PM - started process bpbrm (8412)
10/24/2013 12:42:04 PM - connecting
10/24/2013 12:42:05 PM - Info bpbrm(pid=8412) starting bpbkar32 on client        
10/24/2013 12:42:05 PM - connected; connect time: 00:00:01
10/24/2013 12:42:07 PM - Info bpbkar32(pid=6644) Backup started          
10/24/2013 12:42:07 PM - Info bptm(pid=5072) start           
10/24/2013 12:42:07 PM - Info bptm(pid=5072) using 262144 data buffer size       
10/24/2013 12:42:07 PM - Info bptm(pid=5072) setting receive network buffer to 262144 bytes     
10/24/2013 12:42:07 PM - Info bptm(pid=5072) using 32 data buffers        
10/24/2013 12:42:08 PM - Info bptm(pid=5072) start backup          
10/24/2013 12:42:08 PM - Info bptm(pid=5072) Waiting for mount of media id AA1479 (copy 1) on server finms-nbmaster.
10/24/2013 12:42:08 PM - mounting AA1479
10/24/2013 12:42:44 PM - Info bpbkar32(pid=6644) change journal NOT enabled for <D:\From_Win2K3_NBU_D\backup_host_file>      
10/24/2013 12:42:52 PM - Info bpbkar32(pid=6644) bpbkar waited 0 times for empty buffer, delayed 0 times.  
10/24/2013 1:00:34 PM - Error bptm(pid=5072) error requesting media, TpErrno = Robot operation failed    
10/24/2013 1:00:34 PM - Warning bptm(pid=5072) media id AA1479 load operation reported an error    
10/24/2013 1:00:34 PM - current media AA1479 complete, requesting next resource Any
10/24/2013 1:04:06 PM - Error bptm(pid=5072) NBJM returned an extended error status: All compatible drive paths are down but media is available (2009)
10/24/2013 1:04:06 PM - Info bptm(pid=5072) EXITING with status 252 <----------       
10/24/2013 1:04:10 PM - Error bpbrm(pid=8412) cannot send mail to XXXXXX     
10/24/2013 1:04:11 PM - Info bpbkar32(pid=6644) done. status: 252: extended error status has been encountered, check logs 
10/24/2013 1:04:11 PM - end writing
10/24/2013 1:04:12 PM - Info bpbrm(pid=9336) Starting delete snapshot processing        
10/24/2013 1:04:12 PM - Info bpfis(pid=0) Snapshot will not be deleted       
10/24/2013 1:04:15 PM - Info bpfis(pid=1120) Backup started          
10/24/2013 1:04:15 PM - Critical bpbrm(pid=9336) from client finms-nbmaster: cannot open D:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.finms-nbmaster_1382632923.1.0     
10/24/2013 1:04:15 PM - Info bpfis(pid=1120) done. status: 1542         
10/24/2013 1:04:15 PM - end operation
10/24/2013 1:04:15 PM - Info bpfis(pid=0) done. status: 1542: An existing snapshot is no longer valid and cannot be mounted for subsequent operations
extended error status has been encountered, check logs(252)

[Will_Restore]

possible solution from Article URL http://www.symantec.com/docs/TECH124872

SOLUTION/WORKAROUND:
-Customer is backing up to a VTL then duplicating to tape
-Trying to lock a file/drive that was already locked on the VTL seems to be the source of the issue
-We downed and upped the drives again
-This broke the deadlock and backups started writing again

[mph999]

If all the devices attached to the drives support it, use sci persistant mode, for reservation Master host properties > media This will allow NBU to break the reservation if we made it. If not, it 'might' sugest something outside NBU is creating the reservation.

[mnolan]

Just wanted to pipe in here.  Haven't see a mention on what version you are, but I believe Direct Copy was broken again in 7.5 until 7.5.0.3

Had a customer set this up recently and through testing and with the fact that the original media server is going to run the duplication, we ensured that it had ndmp credentials for the quantum and configured ndmp tape paths for the vtl.  This qualifies as the read server and while we were at it setup ndmp path tapes for the physical library as well so it could also be the write server, effectively isolating the duplication to that server.