Tape media server is not active,

[Guruh_Sulistiyo]

Hi All,

I am using Windows Server 2003 R2 Standard Edition SP2, NetBackup 7.1.0.1, and Dell PowerVault 124T as the Tape Library. 

I have a weird issue setting up Media Server. When I configure storage devices from home. I am able to detect the Robot and Media Server which is connected to the Media Server. But after sometime, when I go to Device Monitor, I will see the drive showing <Retrieving status..>. Also when I check in the Media Server, the status is Active for Disk only. 

I tried to run the following command in Master Server. sudo /usr/openv/netbackup/bin/admincmd/nbemmcmd -updatehost -machinename <media_server_name> -machinetype media -machinestateop set_tape_active -masterserver <master_server_name>. Tape will showing in Device Monitor and Media Servers status for a while and then missing again. 

Attached is the bptm log from the media server. Hope someone can help me to give solution for this issue. 

[Marianne]

Is this a newly added media server or has it been working fine before?

If new - firstly why W2003 when Microsoft has dropped support for W2003?
Have you added Library Based tape drive license on the media server?

Can you confirm comms between master and media server?
Please show us output of these commands on the master server:

nbemmcmd -listhosts -verbose
nbemmcmd -getemmserver

Oh... bptm shows SCSI Reservation errors - are tape drives shared?
If so, have you added SSO license on this and other media servers?

[Guruh_Sulistiyo]

We use ARCserve before. When we were using ARCserve, we never get this problem. We just migrated to NetBackup couple weeks ago and start getting this issue right after the migration.

We want to use the existing server (which is W2003) since we don't have capacity to build a W2008.

Yes, I look into the License of the Media Server, I can see the Library Based Tape Drives in the license.

No, it's not being shared with other media server.

attached is the output for those 2 commands :

[Marianne]

I cannot open .log file on mobile device.
Any chance you can rename file to .txt and upload again?

Have you uninstalled Arcserve on this media server and rebooted before installing NBU?
Each software has its own device management software and will clash.

[Guruh_Sulistiyo]

No I didn't. Let me try and get back to you the result.

Attached is the txt version

[Marianne]

Which one is the problematic media server?
(I did not realize that you had a bunch of media servers and there are 2 media servers that shows 'Active for disk' only.)

Let us know once Arcserver is removed.

[Guruh_Sulistiyo]

Hi Marianne,

I completely removed the ARCserve, Reboot, Completely remove NetBackup, Reboot, Reinstall NetBackup, Reboot. But I am still having the same problem till now. 

FRSMP002 is the problematic server. 

[Marianne]

Please double check licenses on this media servers.
I see MachineFlags in nbemmcmd looks different for this media server.
Normally indication of difference in registered license keys.
Use nbminlicense on the media server.

The other reason for this that I've seen was comms issues between the master and media server where emm heartbeats from the master is missed.

[Guruh_Sulistiyo]

How to fix the comms issue between Master and Media Server? As I didn't see any issue on the media server

Here is the output from bminlicense on the media server

DJXJ-OJ3J-3UCT-6XPP-PPNP-XPP3-????-????-? DJXJ-OJ3J-3UCT-6XPP-PPNP-????-????-???
file version = 0x05000000
time added = 0x57e4f2b7 09/23/16 11:15:35
hostname = frsmp002.specialties.int.huntsman.com
product ID = 6 NetBackup
serial number = 22932
key version = 0x05000000
count = 0
server platform = 0 Any platform
client platform = 2 Any Windows platform
server tier = 5 NetBackup Server
client tier = 0 No tier
license type = 1 Permanent
Site ID = 2006 Unknown site: 2006
Feature ID = 22 Additional clients +

DJX8-CT6O-NZCD-6CXP-PPP3-C6PP-????-????-??? DJX8-CT6O-NZCD-6CXP-PPP3-????-????-???
file version = 0x05000000
time added = 0x57e4f2ae 09/23/16 11:15:26
hostname = frsmp002.specialties.int.huntsman.com
product ID = 6 NetBackup
serial number = 22933
key version = 0x05000000
count = 0
server platform = 0 Any platform
client platform = 0 Any platform
server tier = 5 NetBackup Server
client tier = 0 No tier
license type = 1 Permanent
Site ID = 2006 Unknown site: 2006
Feature ID = 49 Library Based Tape Drives +

OEX2-6PON-WYPR-W8P6-CPDI-P4F4-KXFF-????-????-??? OEX2-6PON-WYPR-W8P6-????-????-???
file version = 0x05000000
time added = 0x57e4f1ed 09/23/16 11:12:13
hostname = frsmp002.specialties.int.huntsman.com
product ID = 6 NetBackup Enterprise Server
serial number = 806155
key version = 0x05000000
count = 0
server platform = 0 Any platform
client platform = 0 Any platform
server tier = 10 NetBackup Enterprise Server
client tier = 0 No tier
license type = 1 Permanent
Site ID = 2006 Unknown site: 2006
Feature ID = 84 SAN Client +
Feature ID = 83 PureDisk MS Exchange Agent +
Feature ID = 81 Enterprise Vault Agent +
Feature ID = 80 PureDisk MS SQL Server Agent +
Feature ID = 79 Flexible Disk Option +
Feature ID = 78 OpenStorage Disk Option +
Feature ID = 76 Virtual Tape Option +
Feature ID = 75 Bare Metal Restore +
Feature ID = 74 Encryption +
Feature ID = 73 Open File Backup +
Feature ID = 71 Remote Client Support +
Feature ID = 70 Robotic Library Sharing Support +
Feature ID = 69 Remote Media Server Support +
Feature ID = 68 Microsoft RSM Robotic Libraries +
Feature ID = 67 ADIC DAS/SDLC Robotic Libraries +
Feature ID = 66 IBM ATL Robotic Libraries +
Feature ID = 65 Fujitsu LMF Robotic Libraries +
Feature ID = 64 StorageTek ACS Robotic Libraries +
Feature ID = 62 Snapshot Client +
Feature ID = 61 MS SharePoint Agent +
Feature ID = 58 Inline Tape Copy +
Feature ID = 54 Vault +
Feature ID = 49 Library Based Tape Drives +
Feature ID = 46 DataStore +
Feature ID = 43 Encryption (Legacy DES 56-bit) +
Feature ID = 42 Encryption (Legacy DES 40-bit) +
Feature ID = 40 SAP extension +
Feature ID = 38 Sybase extension +
Feature ID = 37 Informix extension +
Feature ID = 36 Oracle extension +
Feature ID = 35 Lotus Notes extension +
Feature ID = 34 DB2 extension +
Feature ID = 33 MS SQL Server extension +
Feature ID = 32 MS Exchange extension +
Feature ID = 30 Shared Storage Option +
Feature ID = 27 NDMP +
Feature ID = 22 Additional clients +
Feature ID = 20 Base NetBackup +

[Marianne]

"How to fix the comms issue between Master and Media Server? "

I cannot tell you. What is different about this media server? Is it at remote location? Same or different subnet?

Licenses look fine. So, not a license issue.

[Marianne]

See this TN:

http://www.veritas.com/docs/000038655

[Guruh_Sulistiyo]

I notice the TLD in Media and Device Management, Devices, Robot for this robot showing Enabled = No. Is there a reason for this?

When I tried to do the direct D2T. I will get below message in the Activity Monitor

09/26/2016 16:25:23 - Info nbjm (pid=6311) starting backup job (jobid=2000846) for client FRSMP002, policy Z_RnD_Policy, schedule Full
09/26/2016 16:25:23 - Info nbjm (pid=6311) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=2000846, request id:{0FE7DA0E-83F5-11E6-A694-4C7B107C2627})
09/26/2016 16:25:23 - requesting resource frsmp002-hcart3-robot-tld-3
09/26/2016 16:25:23 - requesting resource bebeb245.NBU_CLIENT.MAXJOBS.FRSMP002
09/26/2016 16:25:23 - requesting resource bebeb245.NBU_POLICY.MAXJOBS.Z_RnD_Policy
09/26/2016 16:25:24 - awaiting resource frsmp002-hcart3-robot-tld-3. Waiting for resources.
Reason: Robotic library is down on server, Media server: frsmp002,
Robot Type(Number): TLD(3), Media ID: N/A, Drive Name: N/A,
Volume Pool: FRSM_VOLUME_POOL, Storage Unit: frsmp002-hcart3-robot-tld-3, Drive Scan Host: N/A,
Disk Pool: N/A, Disk Volume: N/A

Attached is the bptm log

[Marianne]

Please add VERBOSE entry in vm.conf on the media server and restart Device Management Service.

Device-related issues will now be logged to Event Viewer System and Application logs.

Please also check and confirm that Removable Storage Manager service is stopped and disabled.
(Basic requirement for NBU on W2003 server. Because hardly anyone is still using W2003, we don't always remember to check.)

*** EDIT *** 

Quite an old Article, but still very much applicable to W2003 and this old version of NBU:

https://vox.veritas.com/t5/Articles/Tuning-Windows-2003-and-2008-for-Symantec-NetBackup/ta-p/812401

[Guruh]

Removable Storage service is stopped and disabled.

After I enabled the VERBOSE. I found the following errors :

in the Application Log :
Unable to send scsi command, ScsiStatus = 0x2
TLD(3) key = 0x2, asc = 0x4, ascq = 0x1, LOGICAL UNIT IS IN PROCESS OF BECOMING READY
TLD(3) drive read_element_status error
TLD(3) going to DOWN state, status: Unable to sense robotic device
Request for media ID 0037L3 is being rejected because mount requests are disabled (reason = robotic daemon going to DOWN state)
TLD(3) key = 0x2, asc = 0x4, ascq = 0x1, LOGICAL UNIT IS IN PROCESS OF BECOMING READY
TLD(3) drive read_element_status error
TLD(3) going to DOWN state, status: Unable to sense robotic device

Any idea?

[Marianne]

Those Sense Key errors looks like a hardware issue.

Either faulty tape library or someone has opened the robot and it is busy initializing. 

If it has a web console, check and see if any errors are reported.
Else, physical inspection is needed to check display panel.

[watsons]

Are you experiencing drives or robots going up and down again after some time, that caused the "active for tape" to disappear?

Check this out if you haven't done so - known to cause many device issue:

http://www.veritas.com/docs/000042456

Also, version 7.1.0.1 is pretty old, and even as 7.1 it's not the latest, go to 7.1.0.4 or higher if possible.

[Guruh_Sulistiyo]

Hi Marianne,

 Either in Web Console or on the Front Panel showing Ok. No alert or error showing

Hi Watsons,

I follow the guide mentioned in the link. It's true I can see few ghost driver in driver manager. I removed the ghost drivers. Reboot the server. But still problem exist. We also thought maybe the higher version will help us. Last time we upgrade the version to 7.1.0.4. But still it's not helping us. Any other advice?

[Marianne]

These errors are coming directly from the tape library:

Unable to send scsi command, ScsiStatus = 0x2
key = 0x2, asc = 0x4, ascq = 0x1, LOGICAL UNIT IS IN PROCESS OF BECOMING READY

NBU is merely reporting the errors.

Please send this to your Tape Library support vendor and see what they say.

Updating NBU will not fix those errors. 

[Guruh_Sulistiyo]

One more thing. Everytime I restarted the Media Manager Device Daemon. The TLD will showing Enabled = Yes. But after 3 seconds. It will be back to No again.

[Marianne]

TLD is going down because of those Sense key errors.

Any possibility you could save text version of Event Viewer System and Application logs and upload here?