cancel
Showing results for 
Search instead for 
Did you mean: 

SAN Backup failed - backup active with error robot operation failed

Imra_backup
Level 4

Hi fellow friends;

I have issue with SAN backup.

Component:

  1. Master Server: Windows 2008R2 Server
  2. Media Server: IBM AIX 7.1
  3. Tape Library: Quantum i80, 4 units tape drives (LTO5)
  4. SAN Switch (Brocade)
  5. Veritas Netbackup v7.1

Issue:

When run backup 'policy' for Media server (IBM AIX 7.1), media keep mounting forever then got robot operation failed (status from activity monitor). This policy configure to use storage unit define to backup through SAN network.

If run backup for other client & using different storage unit, backup running OK and normally ended up status=0,  completed.

In summary: can conclude backup over SAN not OK while backup through LAN network is OK.

Appreciated advice/comments/recommendation. Thank you in Advance..

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

imra2013
Level 4

Hi Everyone,

Manage to solve this issue after server been force to shut down (power outage) and rescan tape drives when reconfigure storage devices. Basically OS have some configuration mismatch with tape drives and manage to re-detect tape drives after server boot-up.

Hi Marianne,

I could not find 'mark solution' link

View solution in original post

43 REPLIES 43

RamNagalla
Moderator
Moderator
Partner    VIP    Certified

hi,

I just want to make sure

are you trying to use the FT media server or SAN media server?

 

Imra_backup
Level 4

Hi Nagalla,

Thanks for quick reply.

Not sure how to answer this (sorry), but I guess I have Media Server. How to verify whether FT or SAN media server ?

 

Imra_backup
Level 4

sorry I guess I have SAN Media Server configured.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Are tape drives shared? Do you have SSO license added on master and media server? Did you use the Device Config wizard on the master to config devices for master and media server? Please post output from media server: /usr/openv/volmgr/bin/scan /usr/openv/volmgr/bin/tpconfig -l /usr/openv/volmgr/bin/vmoprcmd -d Ensure logging on media server is enabled as follows: bptm log folder exists in /usr/openv/netbackup/logs VERBOSE entry in /usr/openv/volmgr/vm.conf (followed by NBU restart) Ensure syslog is enabled at OS level.

Imra_backup
Level 4

Hi Marianne,

Tape drives are shared. SSO license both install on  master & media.

Let me check on the device config. Below media server output as requested.

scan

oot@ibmdb1:>./scan
************************************************************
*********************** SDT_TAPE    ************************
*********************** SDT_CHANGER ************************
************************************************************

------------------------------------------------------------
Device Name  : "/dev/rmt0.1"
Passthru Name: "/dev/rmt0.1"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "IBM     DDS Gen7        VS11"
Vendor ID  : "IBM     "
Product ID : "DDS Gen7        "
Product Rev: "VS11"
Serial Number: "20150146"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: Not Found(5)
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x0
Reason: 0x0

 

tpconfig -l

root@ibmdb1:>./tpconfig -l
Device Robot Drive       Robot                    Drive                 Device
Type     Num Index  Type DrNum Status  Comment    Name                  Path
robot      0    -    TLD    -       -  -          -                     backupserver
  drive    -    1 hcart2    4      UP  -          HP.ULTRIUM5-SCSI.002  /dev/rmt1.1
  drive    -    2 hcart2    3      UP  -          HP.ULTRIUM5-SCSI.003  /dev/rmt2.1
  drive    -    3 hcart2    2      UP  -          HP.ULTRIUM5-SCSI.000  /dev/rmt3.1
  drive    -    4 hcart2    1      UP  -          HP.ULTRIUM5-SCSI.001  /dev/rmt4.1
drive      -    0    pcd    -  DISABL  -          IBM.DDSGEN7.000       /dev/rmt0.1

 

vmoprcmd

root@ibmdb1:>./vmoprcmd -d

                                PENDING REQUESTS

                                     <NONE>

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  1 hcart2   TLD                -                     No       -         0
  2 hcart2   TLD                -                     No       -         0
  3 hcart2   TLD                -                     No       -         0
  4 hcart2   TLD                -                     No       -         0

                             ADDITIONAL DRIVE STATUS

Drv DriveName            Shared    Assigned        Comment
  1 HP.ULTRIUM5-SCSI.002  Yes      -
  2 HP.ULTRIUM5-SCSI.003  Yes      -
  3 HP.ULTRIUM5-SCSI.000  Yes      -
  4 HP.ULTRIUM5-SCSI.001  Yes      -
root@ibmdb1:>

 

Ok vm.conf have already add VERBOSE.

Over to you Marianne..thanks.

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Seems 'scan' can only see the internal tape drive?

 

Device Name  : "/dev/rmt0.1"
Passthru Name: "/dev/rmt0.1"
...
Inquiry    : "IBM     DDS Gen7        VS11"

drive      -    0    pcd    -  DISABL  -          IBM.DDSGEN7.000       /dev/rmt0.1

Please show us output of:

lsdev -C -c tape

Imra_backup
Level 4

Hi Marianne,

yes indeed, OS (media Server, IBM AIX) only detect internal tape drive which is not use for backup.

lsdev -Cc tape output;

root@ibmdb1:>lsdev -Cc tape
rmt0 Available 00-08-00 SAS 4mm Tape Drive
rmt1 Available 04-01-02 Other FC SCSI Tape Drive
rmt2 Available 04-01-02 Other FC SCSI Tape Drive
rmt3 Available 04-01-02 Other FC SCSI Tape Drive
rmt4 Available 04-01-02 Other FC SCSI Tape Drive

Over to you Marianne

Possible
Level 6
Accredited Certified
If run backup for other client & using different storage unit... different stu under same media server ? Can you try to move media from slot to drive via robtest..if no...what error? Can you post /usr/openv/volmgr/debug/tpreq logs o/p... Thanks, Giri.

imra2013
Level 4

Hi Giri,

Thanks for reply. What I mean is I have define 2 STU.

STU1 = for other client (for normal LAN backup, means client must go through Master Server for backup)

STU2 = for media server (which is for SAN backup - direct backup to Quantum i80 tape library)

Ok for robtest just to let you know, robtest command can nonly run from Master Server (which also as Media Server)

I can't do robtest from my Media Server (the IBM AIX Server)

There is no /tpreq folder in /volmgr/debug in Master but I can run robtest utility, and just now able to ove tape from slot to drive - with no error.

Hope this helps, let me know if you need additional info. Thanks.

 

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

I am curious to know how devices were configured if 'scan' does not pick up the drives.

We need to be able to somehow double-check that  /dev/rmt1.1 is indeed position 4 in the robot ,  /dev/rmt4.1 is indeed position 1 in the robot, etc.

Normally this can be easily confirmed with output from scan or 'tpautoconf -t' combined with 'scan -changer' output from the robot control host.

 

PS: Why exactly do you need 2 user id's on Connect? 
Once again you start a discussion as one user and then respond with your other id?

imra2013
Level 4

Hi Marianne,

Sorry I should logout from imra2013 & log as Imra_backup. Can I delete my user ID (either one will do..)

I'm also puzzle how to explain this (scan does not pickup drive) when I assigned to this project. It was OK for Master Server but not for Media Server, maybe something was wrong and affect setting to media Server. Because previously we can do backup via SAN for Media Server. Now we can't even do robtest in Media Server (IBM AIX).

Hope this provide some background information (of the issue)

 

 

 

imra2013
Level 4

Hi Marianne,

I would like to use imra2013 ID in Connect from now on.

Somehow can't use Imra_backup ID

Thanks for your reminder.

Sorry for inconvenienced caused.

 

imra2013
Level 4

Hi,

Any helps ?

Many thanks in advance.

 

imra2013
Level 4

Hi Marianne,

Sorry below output scan -changer (from Master Server)

C:\Program Files\Veritas\Volmgr\bin>scan -changer
************************************************************
*********************** SDT_CHANGER ************************
************************************************************
------------------------------------------------------------
Device Name  : ""
Passthru Name: ""
Volume Header: ""
Port: 3; Bus: 0; Target: 6; LUN: 1
Inquiry    : "QUANTUM Scalar i40-i80  140G"
Vendor ID  : "QUANTUM "
Product ID : "Scalar i40-i80  "
Product Rev: "140G"
Serial Number: "QUANTUMD1H0131319_LLA"
WWN          : ""
WWN Id Type  : 0
Device Identifier: "QUANTUM D1H0131319_LLA          "
Device Type    : SDT_CHANGER
NetBackup Robot Type: 8
Removable      : Yes
Device Supports: SCSI-3
Number of Drives : 4
Number of Slots  : 49
Number of Media Access Ports: 5
Drive 1 Serial Number      : "C38C1BE000"
Drive 2 Serial Number      : "C38C1BE004"
Drive 3 Serial Number      : "C38C1BE008"
Drive 4 Serial Number      : "C38C1BE00C"
Flags : 0x0
Reason: 0x0

 

and tpautoconf -t output

C:\Program Files\Veritas\Volmgr\bin>tpautoconf -t
TPAC60 HP      Ultrium 5-SCSI  Y5AZ C38C1BE004 3 0 4 0 Tape0 -
TPAC60 HP      Ultrium 5-SCSI  Y5AZ C38C1BE000 3 0 6 0 Tape2 -
TPAC60 HP      Ultrium 5-SCSI  Y5AZ C38C1BE00C 3 0 7 0 Tape3 -

Hope this helps.

 

 

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Maybe I was not clear enough in my previous post:

 

We need to be able to somehow double-check that  /dev/rmt1.1 is indeed position 4 in the robot ,  /dev/rmt4.1 is indeed position 1 in the robot, etc.

Normally this can be easily confirmed with output from scan or 'tpautoconf -t' combined with 'scan -changer' output from the robot control host.

In the last statement I meant 'tpautoconf -t' on the media server.
This output will give us similar output as tpautoconf on the master and we will be able to see serial numbers for device names (rmt1.1, rmt2.1, etc).

We can then compare this with drive position and serial no's in scan output:
Drive 1 Serial Number      : "C38C1BE000"
Drive 2 Serial Number      : "C38C1BE004"
Drive 3 Serial Number      : "C38C1BE008"
Drive 4 Serial Number      : "C38C1BE00C"

which will finally enable us to compare with 'tpconfig -l' output to ensure there are no device mapping mismatches.

imra2013
Level 4

Hi Marianne,

tpautoconf output from my Media Server.

root@ibmdb1:>./tpautoconf -t
TPAC60 IBM     DDS Gen7        VS11 20150146 -1 -1 -1 -1 /dev/rmt0.1 - -

Looks Media Server does not detect LTO5 tape drives ..

 

imra2013
Level 4

Hi Marianne,

Just to add, have been reading SAN client & Fibre Transport Guide and I think all the while existing Media Server was not utilizing backup through SAN Network. My understanding we need to configure the machine as SAN client then would be able to utilize backup using fibre connectivity which suppose to be more faster than LAN connection. No wonder 900 GB data backup more than 9 hours whereas our LTO tape drives configured point to point at 8GB/s. Because all the while when backup can only wee 'Transport Type' as LAN.

Please correct me if I'm wrong.

Thanks in advance

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

You clearly have media server software installed with devices configured for AIX server.

There is a big difference between SAN Media server and SAN Client.

see: http://www.symantec.com/docs/TECH135896 and http://www.symantec.com/docs/TECH53815

Your current issue is with device access on AIX media server. 
You need to get that fixed.

imra2013
Level 4

Hi Marianne,

Agree.

My current setup is SAN Media Server (since media server software installed in AIX Server)

I have read both articles, looks SAN client require extra setting to configure. Maybe thats why our previous vendor setup our AIX media server as SAN Media Server. Anyway you are write, I need to solve device access on AIX Media Server.

Do you think I should refer to AIX support how to access the devices or Quantum support to provide drivers etc. Please advice.

Thanks in advance.sad