10-23-2013 01:42 AM
Hi All,
Our Netbackup system is version 7.5.0.4. We have 2 tape library (HP ESL322e) connect using SAN. We already had up running drives running as SSO (setup by our System Integrator). Two of drives on the TL being shared between 2 server , one running Solaris 10 the other one AIX 6.1. in site A. There is another pair with the similar configuration on site B.
One day our HP did firmware upgrade to TL on site A. The upgrade was not smooth, I had to reconfigure again the connection on Solaris 10 (but the AIX was fine).
The problem if I setup individual tape drive to the Solaris or AIX, the backup could run fine. Then I did the SSO with configuration like 1 only for solaris, another 1 only for AIX, but 2 other dirves I config as SSO. Then I test the backup on Solaris , it supposed can see 2 drives (1 single, 1SSO). I fired first backup it run on the Solaris, but when I run the second drive, it won' go that SSO drive, it keep on waiting for resources.
10/23/2013 16:17:16 - awaiting resource ESL025_EBS6_MPX. Waiting for resources.
Reason: Drives are in use, Media server: ebs6-bck,
Robot Type(Number): TLD(1), Media ID: N/A, Drive Name: N/A,
Volume Pool: test1, Storage Unit: ESL025_EBS6_MPX, Drive Scan Host: N/A,
Disk Pool: N/A, Disk Volume: N/A
notes: ebs6 is the Solaris media server.
What i did wrong? Why it cannot use that SSO drive?
Thank you and regards,
Iwan
Solved! Go to Solution.
11-05-2013 03:25 AM
Thanks for the output.
Firstly, we see that stu ESL025_EBS6_MPX is configured for 4 drives:
Label: ESL025_EBS6_MPX Storage Unit Type: Media Manager Host Connection: ebs6-bck Number of Drives: 4 On Demand Only: no Max MPX/drive: 12 Density: hcart2 - 1/2 Inch Cartridge 2 Robot Type/Number: TLD / 1 Max Fragment Size: 1048575 MB
Why is that when only 2 drives are assigned to this media server on this host?
As I have said before, this will result in over-allocation of jobs for this STU.
Can you confirm that SSO license key was added to both of these media servers? (Check output of get_license_key on both media servers).
vmoprcmd tells us that NBU was probably not restarted on media server ebs7 after drives were configured as SSO?
ESL025_Drive12 No No No hcart2 ebs6-bck /dev/rmt/1cbn SCAN-TLD ebs7-bck /dev/rmt1.1 RESTART
NBU needs to be restarted on both media servers.
Please correct the STU config on the master server and run 'bprdreq -rereadconfig'.
If above steps were taken and drive 12 is still not used, try to down drive 11 on ebs6 and see if drive 12 gets used.
Let us know the result.
10-23-2013 02:05 AM
what is the robot number for SSO Drive..?
when you have 2 tape Libraries.. you need to have 2 robots too.. one robot for tape library one and another for tape library 2.
each robot needs to have the seperate storage unit created and used..
so do you have 2 robots?
does it have 2 diffent storage units created?
10-23-2013 02:22 AM
How many drives configured in the Storage Unit?
Please show us STU config:
/usr/openv/netbackup/bin/admincmd/bpstulist -U
These commands as well on Solaris and AIX servers:
/usr/openv/volmgr/bin/tpconfig -l
/usr/openv/volmgr/bin/vmoprcmd -d
10-24-2013 12:23 AM
Hi all,
Thanks for the responses,
Nagalla: The two tape libraries are separated, they only shared the same master server, so actually I mentioned we have 2 tape library only for reference, the other tape library the SSO is still working fine. I am sorry to confuse yu.
Marianne:
It should go to the storage unit ESL025_EBS6_MPX (Solaris) and ESL025_EBS7_MPX (AIX)
Currently there is single drive (11) for EBS6, single drive (14) and SSO drive on EBS6/7 (12) which is not working, 1 drive (13) is not configured yet. The robot connect to EBS3
ebs1moc1 ~]# /usr/openv/netbackup/bin/admincmd/bpstulist -U
Label: ESL025_EBS6_MPX
Storage Unit Type: Media Manager
Host Connection: ebs6-bck
Number of Drives: 4
On Demand Only: no
Max MPX/drive: 12
Density: hcart2 - 1/2 Inch Cartridge 2
Robot Type/Number: TLD / 1
Max Fragment Size: 1048575 MB
Label: ESL125_EBS13_MPX
Storage Unit Type: Media Manager
Host Connection: ebs13-bck
Number of Drives: 4
On Demand Only: no
Max MPX/drive: 4
Density: hcart2 - 1/2 Inch Cartridge 2
Robot Type/Number: TLD / 3
Max Fragment Size: 1048575 MB
Label: ESL12_EBS11_MPX
Storage Unit Type: Media Manager
Host Connection: ebs11bck
Number of Drives: 4
On Demand Only: no
Max MPX/drive: 10
Density: hcart - 1/2 Inch Cartridge
Robot Type/Number: TLD / 3
Max Fragment Size: 1048576 MB
Label: ESL02_EBS3_MPX
Storage Unit Type: Media Manager
Host Connection: ebs3-bck
Number of Drives: 4
On Demand Only: no
Max MPX/drive: 10
Density: hcart - 1/2 Inch Cartridge
Robot Type/Number: TLD / 1
Max Fragment Size: 1048576 MB
Label: ESL02_EBS2_MPX
Storage Unit Type: Media Manager
Host Connection: ebs2-bck
Number of Drives: 4
On Demand Only: no
Max MPX/drive: 12
Density: hcart - 1/2 Inch Cartridge
Robot Type/Number: TLD / 1
Max Fragment Size: 1048575 MB
Label: ESL02_EBS5_MPX
Storage Unit Type: Media Manager
Host Connection: ebs5-bck
Number of Drives: 4
On Demand Only: no
Max MPX/drive: 12
Density: hcart - 1/2 Inch Cartridge
Robot Type/Number: TLD / 1
Max Fragment Size: 1048576 MB
Label: ebs11bck-4mm
Storage Unit Type: Media Manager
Host Connection: ebs11bck
Number of Drives: 1
On Demand Only: no
Max MPX/drive: 1
Density: 4mm - 4mm Cartridge
Robot Type: (not robotic)
Max Fragment Size: 1048576 MB
Label: ebs3-bck-4mm
Storage Unit Type: Media Manager
Host Connection: ebs3-bck
Number of Drives: 1
On Demand Only: no
Max MPX/drive: 1
Density: 4mm - 4mm Cartridge
Robot Type: (not robotic)
Max Fragment Size: 1048576 MB
Label: ESL025_EBS7_MPX
Storage Unit Type: Media Manager
Host Connection: ebs7-bck
Number of Drives: 4
On Demand Only: no
Max MPX/drive: 12
Density: hcart2 - 1/2 Inch Cartridge 2
Robot Type/Number: TLD / 1
Max Fragment Size: 1048575 MB
Label: ESL125_EBS12_MPX
Storage Unit Type: Media Manager
Host Connection: ebs12-bck
Number of Drives: 4
On Demand Only: no
Max MPX/drive: 12
Density: hcart2 - 1/2 Inch Cartridge 2
Robot Type/Number: TLD / 3
Max Fragment Size: 1048575 MB
Label: TEST_DD_VM
Storage Unit Type: Disk
Storage Unit Subtype: DiskPool (6)
Host Connection: ebs14bck
Concurrent Jobs: 80
On Demand Only: yes
Max MPX: 1
Max Fragment Size: 524288 MB
Block Sharing: no
Ok On Root: no
Disk Pool: ROC_DDBoost_DP
Snapshots: no
Replication Primary: no
Replication Source: no
Replication Target: no
Mirror : no
Independent : no
Label: ROC_DDBoost_SU
Storage Unit Type: Disk
Storage Unit Subtype: DiskPool (6)
Host Connection: ebs14bck
Concurrent Jobs: 80
On Demand Only: yes
Max MPX: 1
Max Fragment Size: 524288 MB
Block Sharing: no
Ok On Root: no
Disk Pool: ROC_DDBoost_DP
Snapshots: no
Replication Primary: no
Replication Source: no
Replication Target: no
Mirror : no
Independent : no
Label: REPL_DDMOC_SU
Storage Unit Type: Disk
Storage Unit Subtype: DiskPool (6)
Host Connection: _STU_NO_DEV_HOST_
Concurrent Jobs: 80
On Demand Only: yes
Max MPX: 1
Max Fragment Size: 524288 MB
Block Sharing: no
Ok On Root: no
Disk Pool: REPL_DDMOC_DP
Snapshots: no
Replication Primary: no
Replication Source: no
Replication Target: no
Mirror : no
Independent : no
Label: EBS2_DISK_2
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: ebs2-bck
Concurrent Jobs: 1
On Demand Only: yes
Max MPX: 1
Path: "/usr/openv/netbackup/db/stu_disk_2"
Max Fragment Size: 1000 MB
Stage data: no
Block Sharing: no
High Water Mark: 98
Low Water Mark: 80
Ok On Root: yes
Label: EBS5_Disk
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: ebs5-bck
Concurrent Jobs: 1
On Demand Only: yes
Max MPX: 1
Path: "G:\Netbackup\DB\stu_disk"
Max Fragment Size: 4000 MB
Stage data: no
Block Sharing: no
High Water Mark: 98
Low Water Mark: 80
Ok On Root: yes
Label: EBS3_DISK
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: ebs3-bck
Concurrent Jobs: 1
On Demand Only: yes
Max MPX: 1
Path: "/usr/openv/netbackup/db/stu_disk"
Max Fragment Size: 2000 MB
Stage data: no
Block Sharing: no
High Water Mark: 100
Low Water Mark: 80
Ok On Root: yes
Label: EBS2_DISK
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: ebs2-bck
Concurrent Jobs: 1
On Demand Only: yes
Max MPX: 1
Path: "/usr/openv/netbackup/db/stu_disk"
Max Fragment Size: 1000 MB
Stage data: no
Block Sharing: no
High Water Mark: 100
Low Water Mark: 80
Ok On Root: yes
Label: EBS11_DISK
Storage Unit Type: Disk
Storage Unit Subtype: Basic (1)
Host Connection: ebs11bck
Concurrent Jobs: 1
On Demand Only: yes
Max MPX: 1
Path: "/usr/openv/netbackup/db/stu_disk"
Max Fragment Size: 2000 MB
Stage data: no
Block Sharing: no
High Water Mark: 100
Low Water Mark: 80
Ok On Root: yes
Label: MOC_DDBoost_SU
Storage Unit Type: Disk
Storage Unit Subtype: DiskPool (6)
Host Connection: ebs5-bck
Concurrent Jobs: 80
On Demand Only: yes
Max MPX: 1
Max Fragment Size: 524288 MB
Block Sharing: no
Ok On Root: no
Disk Pool: MOC_DDBoost_DP
Snapshots: no
Replication Primary: no
Replication Source: no
Replication Target: no
Mirror : no
Independent : no
Solaris 10
root@ebs6-bck # /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive Robot Drive Device Second
Type Num Index Type DrNum Status Comment Name Path Device Path
robot 1 - TLD - - - - ebs3-bck
drive - 0 hcart2 11 UP - ESL025_Drive11 /dev/rmt/0cbn
robot 3 - TLD - - - - ebs11bck
drive - 1 hcart2 12 UP - ESL025_Drive12 /dev/rmt/1cbn
root@ebs6-bck # /usr/openv/volmgr/bin/vmoprcmd -d
PENDING REQUESTS
<NONE>
DRIVE STATUS
Drv Type Control User Label RecMID ExtMID Ready Wr.Enbl. ReqId
0 hcart2 TLD - No - 0
1 hcart2 TLD - No - 0
ADDITIONAL DRIVE STATUS
Drv DriveName Shared Assigned Comment
0 ESL025_Drive11 No -
1 ESL025_Drive12 Yes -
AIX
ebs7:/ # /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive Robot Drive Device
Type Num Index Type DrNum Status Comment Name Path
robot 1 - TLD - - - - ebs3-bck
drive - 0 hcart2 14 UP - ESL025_Drive14 /dev/rmt3.1
robot 3 - TLD - - - - ebs11bck
drive - 1 hcart2 12 UP - ESL025_Drive12 /dev/rmt1.1
ebs7:/ # /usr/openv/volmgr/bin/vmoprcmd -d
PENDING REQUESTS
<NONE>
DRIVE STATUS
Drv Type Control User Label RecMID ExtMID Ready Wr.Enbl. ReqId
0 hcart2 TLD - No - 0
1 hcart2 TLD - No - 0
ADDITIONAL DRIVE STATUS
Drv DriveName Shared Assigned Comment
0 ESL025_Drive14 No -
1 ESL025_Drive12 Yes -
ebs7:/ #
Regards,
Iwan Tamimi
10-24-2013 12:56 AM
Storage Unit is configured for 4 drives in robot 1:
Label: ESL025_EBS6_MPX
Host Connection: ebs6-bck
Number of Drives: 4
Robot Type/Number: TLD / 1
While there are only 2 drives attached - 1 drive in robot 1 and another drive in robot 3:
root@ebs6-bck # /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive Robot Drive Device Second
Type Num Index Type DrNum Status Comment Name Path Device Path
robot 1 - TLD - - - - ebs3-bck
drive - 0 hcart2 11 UP - ESL025_Drive11 /dev/rmt/0cbn
robot 3 - TLD - - - - ebs11bck
drive - 1 hcart2 12 UP - ESL025_Drive12 /dev/rmt/1cbn
So, NBU will over-assign jobs for STU ESL025_EBS6_MPX that has in fact only one drive configured.
Is Drive12 in a different robot? There is no STU for this media server for robot 3.
If so, you need to create another STU and drop the number of drives in ESL025_EBS6_MPX to 1. Possibly a good idea to then group them in a STU group and select policies in the STUG.
Same for AIX server - one drive in robot 1, one drive in robot 3.
Hope this helps.
10-24-2013 02:02 AM
Marianne,
Aha, looks like that is the problem , the Drive 12 supposed to be in the robot 1 too. Hmm never notice that, thank you. I remember I pointed to the robot 1, but let me check again.
Let me try to reconfigure it. Thanks for the early finding.
Regards,
Iwan Tamimi
10-24-2013 02:13 AM
Weird that you only have robot 1 and 3 configured. No 0 and 2?
Always best to use the Device Config wizard - it will read all drive serial numbers in the robot on the robot control host, and then config all drives in correct robot based on the serial numbers with correct drive position number.
10-24-2013 02:51 AM
Marianne,
BTW I have changed it, but still cannot used the tape (I was testing on ebs6 and run 2 policies with different pull the firest one use the 11 but the second one just wait, not using the 12
Thu Oct 24 17:34:23 SGT 2013
root@ebs6-bck # /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive Robot Drive Device Second
Type Num Index Type DrNum Status Comment Name Path Device Path
robot 1 - TLD - - - - ebs3-bck
drive - 0 hcart2 11 UP - ESL025_Drive11 /dev/rmt/0cbn
drive - 1 hcart2 12 UP - ESL025_Drive12 /dev/rmt/1cbn
robot 3 - TLD - - - - ebs11bck
ebs7:/home/narahariak # /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive Robot Drive Device
Type Num Index Type DrNum Status Comment Name Path
robot 1 - TLD - - - - ebs3-bck
drive - 0 hcart2 14 UP - ESL025_Drive14 /dev/rmt3.1
drive - 1 hcart2 12 UP - ESL025_Drive12 /dev/rmt1.1
robot 3 - TLD - - - - ebs11bck
ebs7:/home/narahariak #
Regards,
Iwan
10-24-2013 03:07 AM
Was NBU restarted on the media server?
It may also be a good idea to refresh config and the resource broker on the master server:
bprdreq -rereadconfig
nbrbutil -resetAll
(only run last command when no backups are running).
11-03-2013 10:23 PM
I am so sorry for the very late reply due to some circumstances.
I have rebooted the server and the media server but still failed for SSO but OK for single connection.
Regards,
Iwan
11-03-2013 10:41 PM
Please show us output for the following:
On master:
/usr/openv/netbackup/bin/admincmd/bpstulist -U
/usr/openv/netbackup/bin/admincmd/nbrbutil -dump
(Just copy the MDS Allocation section at the bottom)
/usr/openv/volmgr/bin/vmdareq
/usr/openv/volmgr/bin/vmoprcmd
Can I assume that 'tpconfig -l' on both servers are still as per your last post?
Is reason for queued jobs still 'Drives are in use'?
11-04-2013 06:16 PM
Marrianne,
Thanks again for helping with the case.
I attached all the information you requested.
> Is reason for queued jobs still 'Drives are in use'?
Yes. on storage unit ESL025_EBS6_MPX consist of 1 single drive and 1 SSO drive, when the single drive empty then the job continue to that drive.
11-05-2013 03:25 AM
Thanks for the output.
Firstly, we see that stu ESL025_EBS6_MPX is configured for 4 drives:
Label: ESL025_EBS6_MPX Storage Unit Type: Media Manager Host Connection: ebs6-bck Number of Drives: 4 On Demand Only: no Max MPX/drive: 12 Density: hcart2 - 1/2 Inch Cartridge 2 Robot Type/Number: TLD / 1 Max Fragment Size: 1048575 MB
Why is that when only 2 drives are assigned to this media server on this host?
As I have said before, this will result in over-allocation of jobs for this STU.
Can you confirm that SSO license key was added to both of these media servers? (Check output of get_license_key on both media servers).
vmoprcmd tells us that NBU was probably not restarted on media server ebs7 after drives were configured as SSO?
ESL025_Drive12 No No No hcart2 ebs6-bck /dev/rmt/1cbn SCAN-TLD ebs7-bck /dev/rmt1.1 RESTART
NBU needs to be restarted on both media servers.
Please correct the STU config on the master server and run 'bprdreq -rereadconfig'.
If above steps were taken and drive 12 is still not used, try to down drive 11 on ebs6 and see if drive 12 gets used.
Let us know the result.
11-05-2013 02:29 PM
Try to play with SCSI reservation on tape drives. it can be configured globally and the each drive is using "Default" or you can change it per tape drive. SCSI-2 reservation doesn't work everytime with SSO but SCSI-3 reservation is not supported by all devices.
Play with both versions of config, check config of tape library. Many times firware upgrade change some values to defaults. Ask HP which SCSI reservation is supported by TL (SCSI-3 should be supported by new devices).
11-05-2013 09:51 PM
Marrianne,
> Why is that when only 2 drives are assigned to this media server on this host?
Actually I wasn't sure why it was 4, now I undertand a bit, I will change accordingly.
>Can you confirm that SSO license key was added to both of these media servers?
I think it should not be a problem, the configuration for both server have been in production for several month until the TapeLibrary upgrade problem. We didn't change any license config in both servers.
I will restart the NBU and give the ressutl.
Radovan,
The drives connected through fiber through the Fiber switch. The system actauly has been run for sometimes.
Regards,
iwan
11-05-2013 10:03 PM
PLEASE double-check SSO licenses on both media servers.
We saw a similar issue here about a week ago when drive control remained in RESTART.
Issue was solved when SSO licenses were added to media servers.
Media servers should have these 3 licenses:
NBU Enterprise server (or Enterprise Client if SAN media server)
Library based tape drives
Shared Storage Option
11-05-2013 10:34 PM
Marianne,
Actually I also asked Symantec to help , and the support use the "Configure Storage Devices" wizard on the Java GUI and after that seems ok. I didn't use the wizard when I configured it, I still don't know what I did wrong. Using the wizard , the support actually configure all the for drives to be SSO. After that I took out 1 drives from each media server (so each media server will get 3 drives which 2 of them are in SSO)then the problem come again.
Then I see your finding about my STU config, I corrected to 3 then looks the backup running fine.
Thank you very much.
Iwan
11-05-2013 10:36 PM
The problem solved.
Regards,
Iwan
11-05-2013 10:43 PM
11-19-2013 06:22 PM
Thank you so much Marianne!
Iwan