cancel
Showing results forΒ 
Search instead forΒ 
Did you mean:Β 

Media server bounce back to Active for disk in SSO

Shas
Level 4
Partner

Hi All,

We have a newly setup media servers which is showing Active for Disk only instead of Active for Disk & Tape . SSO licence is active on the Media server

While configuration it shows the drive path and drives are visible from the command "vmoprcmd -d " for few seconds and then the drives go invisible as given below :

D:\Veritas\Volmgr\bin>vmoprcmd -d


                                PENDING REQUESTS

                                     <NONE>

                                  DRIVE STATUS

Drv Type Control User Label RecMID ExtMID Ready Wr.Enbl. ReqId

                             ADDITIONAL DRIVE STATUS

Drv DriveName Shared Assigned Comment

D:\Veritas\Volmgr\bin>

 

We tried to force it from the command , but no use :

nbemmcmd -updatehost -machinename client_name -machinestateop set_tape_active -machinetype media -masterserver <Master server>

NBU Version: 7.5.0.6

OS : win2008 R2

I am attaching the LTID debug log.

1 ACCEPTED SOLUTION

Accepted Solutions

Deb_Wilmot
Level 6
Employee Accredited Certified

There are connectivity errors.

Media servers can go offline for tape (active for disk only) for many reasons including the following:

   - Master failed to receive the hearbeats.

    -  Drive polling fails.

If ltid is exiting it's because it can't detect any devices on the system. 

 

In this case ltid starts up, but detects a change in machine state:
17:36:15.121 [904.1212] <4> avrd: INITIATING
17:36:15.121 [6968.4420] <4> SendEmmHeartbeat: Detected change in MachineState...
17:36:15.121 [6968.4420] <4> SendEmmHeartbeat: ...clearing LOCAL_CONTROL bits
 

So apparently the heartbeats failed.

Every 5 minutes the media server will attempt to send the heart beat again...  Every time after 17:36 - it Detects a change in machine state and tries to clear local bits and up the libraries.

 

At this point all updates fail:

18:06:15.442 [6968.4420] <16> emmlib_SetRobotStatus: (0) UpdateLibrary failed, emmError = 2007031, nbError = 0
18:06:15.442 [6968.4420] <16> ROBOT_CONNECT: (-) Translating EMM_ERROR_SQLNoDataFound(2007031) to 328 in the device management context
18:06:15.442 [6968.4420] <16> ROBOT_CONNECT: emmlib_SetRobotStatus() failed for robot 3 with error 328
18:06:15.442 [6968.4420] <4> SendEmmHeartbeat: Going Active failed - reset machine state.
 

I would suggest you check the following:

WINS or DNS server issues

A NIC going bad.

You might want to schedule a bptestnetconn every 10 minutes or so to see when the media sever is not available.

I woul suggest enabling the folloiwng logigng levels on the media server:

vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -a -p 51216 -o 156 -s DebugLevel=6 -s DiagnosticLevel=6

 

Make sure that the daemon directory is also available under volmgr\debug

On the maser - if space allows - run the following:
 

vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -a -p 51216 -o 156 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -a -p 51216 -o 137-s DebugLevel=6 -s DiagnosticLevel=6

 

Check the resulting log files for issues connecting.

An example is in http://www.symantec.com/docs/TECH69625

 

*** Don't forget to remove logging when done.

Media server:

vxlogcfg -r -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -r -p 51216 -o 156 -s DebugLevel=6 -s DiagnosticLevel=6

 

Master:
 

vxlogcfg -r -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -r -p 51216 -o 156 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -r -p 51216 -o 137-s DebugLevel=6 -s DiagnosticLevel=6

 

Good luck

 

 

 

 

 

After that happens this entity can no longer make any updates to the database and

 

 

 

View solution in original post

11 REPLIES 11

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

SSO licence is active on the Media server.

What about Library Based Tape Drives license?

Any chance a demo license key has expired and the above license is not installed?

Have you checked bpminlicense output on the media server?

So, you need 3 licenses on a media server:

Enterprise Server or Enterprise Client (depending on 'full' or SAN media server)

Library Based Tape Drives

Shared Storage Option

Shas
Level 4
Partner

Hi Marianne

Apology for delayed respone.

Below are the licences which are active on the new media server  ( hostname02 ).

xxXx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xx xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xx

  file version = 0x05000000
  time added = 0x53175ef4 03/05/14 17:29:24
  hostname = hostname02   (edited)
  product ID = 6 NetBackup Enterprise Server
  serial number = 228302
  key version = 0x05000000
  count = 0
  server platform = 0 Any platform
  client platform = 0 Any platform
  server tier = 10 NetBackup Enterprise Server
  client tier = 0 No tier
  license type = 1 Permanent
  Site ID = 2006 Unknown site: 2006
  Feature ID = 89 Replication Director +
  Feature ID = 84 SAN Client +
  Feature ID = 83 PureDisk MS Exchange Agent +
  Feature ID = 81 Enterprise Vault Agent +
  Feature ID = 80 PureDisk MS SQL Server Agent +
  Feature ID = 79 Flexible Disk Option +
  Feature ID = 78 OpenStorage Disk Option +
  Feature ID = 76 Virtual Tape Option +
  Feature ID = 75 Bare Metal Restore +
  Feature ID = 74 Encryption +
  Feature ID = 73 Open File Backup +
  Feature ID = 71 Remote Client Support +
  Feature ID = 70 Robotic Library Sharing Support +
  Feature ID = 69 Remote Media Server Support +
  Feature ID = 68 Microsoft RSM Robotic Libraries +
  Feature ID = 67 ADIC DAS/SDLC Robotic Libraries +
  Feature ID = 66 IBM ATL Robotic Libraries +
  Feature ID = 65 Fujitsu LMF Robotic Libraries +
  Feature ID = 64 StorageTek ACS Robotic Libraries +
  Feature ID = 62 Snapshot Client +
  Feature ID = 61 MS SharePoint Agent +
  Feature ID = 58 Inline Tape Copy +
  Feature ID = 54 Vault +
  Feature ID = 49 Library Based Tape Drives +
  Feature ID = 46 DataStore +
  Feature ID = 43 Encryption (Legacy DES 56-bit) +
  Feature ID = 42 Encryption (Legacy DES 40-bit) +
  Feature ID = 40 SAP extension +
  Feature ID = 38 Sybase extension +
  Feature ID = 37 Informix extension +
  Feature ID = 36 Oracle extension +
  Feature ID = 35 Lotus Notes extension +
  Feature ID = 34 DB2 extension +
  Feature ID = 33 MS SQL Server extension +
  Feature ID = 32 MS Exchange extension +
  Feature ID = 30 Shared Storage Option +
  Feature ID = 27 NDMP +
  Feature ID = 22 Additional clients +
  Feature ID = 20 Base NetBackup +
xxXx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xx xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xx
  file version = 0x05000000
  time added = 0x53174e0f 03/05/14 16:17:19
  hostname = hostname02  (edited)
  product ID = 6 NetBackup Enterprise Server
  serial number = 806254
  key version = 0x05000000
  count = 0
  server platform = 0 Any platform
  client platform = 0 Any platform
  server tier = 10 NetBackup Enterprise Server
  client tier = 0 No tier
  license type = 1 Permanent
  Site ID = 2006 Unknown site: 2006
  Feature ID = 86 PureDisk Remote Office +
  Feature ID = 85 PureDisk Option +
  Feature ID = 84 SAN Client +
  Feature ID = 83 PureDisk MS Exchange Agent +
  Feature ID = 81 Enterprise Vault Agent +
  Feature ID = 80 PureDisk MS SQL Server Agent +
  Feature ID = 79 Flexible Disk Option +
  Feature ID = 78 OpenStorage Disk Option +
  Feature ID = 77 NAS SnapVault Option +
  Feature ID = 76 Virtual Tape Option +
  Feature ID = 75 Bare Metal Restore +
  Feature ID = 74 Encryption +
  Feature ID = 73 Open File Backup +
  Feature ID = 71 Remote Client Support +
  Feature ID = 70 Robotic Library Sharing Support +
  Feature ID = 69 Remote Media Server Support +
  Feature ID = 68 Microsoft RSM Robotic Libraries +
  Feature ID = 67 ADIC DAS/SDLC Robotic Libraries +
  Feature ID = 66 IBM ATL Robotic Libraries +
  Feature ID = 65 Fujitsu LMF Robotic Libraries +
  Feature ID = 64 StorageTek ACS Robotic Libraries +
  Feature ID = 61 MS SharePoint Agent +
  Feature ID = 58 Inline Tape Copy +
  Feature ID = 54 Vault +
  Feature ID = 49 Library Based Tape Drives +
  Feature ID = 48 Additional Drives/Robotics +
  Feature ID = 46 DataStore +
  Feature ID = 43 Encryption (Legacy DES 56-bit) +
  Feature ID = 42 Encryption (Legacy DES 40-bit) +
  Feature ID = 40 SAP extension +
  Feature ID = 39 SQL-Backtrack extension +
  Feature ID = 38 Sybase extension +
  Feature ID = 37 Informix extension +
  Feature ID = 36 Oracle extension +
  Feature ID = 35 Lotus Notes extension +
  Feature ID = 34 DB2 extension +
  Feature ID = 33 MS SQL Server extension +
  Feature ID = 32 MS Exchange extension +
  Feature ID = 31 Advanced Reporting Option +
  Feature ID = 30 Shared Storage Option +
  Feature ID = 27 NDMP +
  Feature ID = 22 Additional clients +
  Feature ID = 20 Base NetBackup +

Shas
Level 4
Partner

Hello Marianne 

Here is the details of license keys active on the new media server ..

xxXx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xx xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xx
 
  file version = 0x05000000
  time added = 0x53175ef4 03/05/14 17:29:24
  hostname = hostname02                                      #(edited)
  product ID = 6 NetBackup Enterprise Server
  serial number = 228302
  key version = 0x05000000
  count = 0
  server platform = 0 Any platform
  client platform = 0 Any platform
  server tier = 10 NetBackup Enterprise Server
  client tier = 0 No tier
  license type = 1 Permanent
  Site ID = 2006 Unknown site: 2006
  Feature ID = 89 Replication Director +
  Feature ID = 84 SAN Client +
  Feature ID = 83 PureDisk MS Exchange Agent +
  Feature ID = 81 Enterprise Vault Agent +
  Feature ID = 80 PureDisk MS SQL Server Agent +
  Feature ID = 79 Flexible Disk Option +
  Feature ID = 78 OpenStorage Disk Option +
  Feature ID = 76 Virtual Tape Option +
  Feature ID = 75 Bare Metal Restore +
  Feature ID = 74 Encryption +
  Feature ID = 73 Open File Backup +
  Feature ID = 71 Remote Client Support +
  Feature ID = 70 Robotic Library Sharing Support +
  Feature ID = 69 Remote Media Server Support +
  Feature ID = 68 Microsoft RSM Robotic Libraries +
  Feature ID = 67 ADIC DAS/SDLC Robotic Libraries +
  Feature ID = 66 IBM ATL Robotic Libraries +
  Feature ID = 65 Fujitsu LMF Robotic Libraries +
  Feature ID = 64 StorageTek ACS Robotic Libraries +
  Feature ID = 62 Snapshot Client +
  Feature ID = 61 MS SharePoint Agent +
  Feature ID = 58 Inline Tape Copy +
  Feature ID = 54 Vault +
  Feature ID = 49 Library Based Tape Drives +
  Feature ID = 46 DataStore +
  Feature ID = 43 Encryption (Legacy DES 56-bit) +
  Feature ID = 42 Encryption (Legacy DES 40-bit) +
  Feature ID = 40 SAP extension +
  Feature ID = 38 Sybase extension +
  Feature ID = 37 Informix extension +
  Feature ID = 36 Oracle extension +
  Feature ID = 35 Lotus Notes extension +
  Feature ID = 34 DB2 extension +
  Feature ID = 33 MS SQL Server extension +
  Feature ID = 32 MS Exchange extension +
  Feature ID = 30 Shared Storage Option +
  Feature ID = 27 NDMP +
  Feature ID = 22 Additional clients +
  Feature ID = 20 Base NetBackup +
xxXx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xx xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xx
  file version = 0x05000000
  time added = 0x53174e0f 03/05/14 16:17:19
  hostname = hostname02                                       #(edited)
  product ID = 6 NetBackup Enterprise Server
  serial number = 806254
  key version = 0x05000000
  count = 0
  server platform = 0 Any platform
  client platform = 0 Any platform
  server tier = 10 NetBackup Enterprise Server
  client tier = 0 No tier
  license type = 1 Permanent
  Site ID = 2006 Unknown site: 2006
  Feature ID = 86 PureDisk Remote Office +
  Feature ID = 85 PureDisk Option +
  Feature ID = 84 SAN Client +
  Feature ID = 83 PureDisk MS Exchange Agent +
  Feature ID = 81 Enterprise Vault Agent +
  Feature ID = 80 PureDisk MS SQL Server Agent +
  Feature ID = 79 Flexible Disk Option +
  Feature ID = 78 OpenStorage Disk Option +
  Feature ID = 77 NAS SnapVault Option +
  Feature ID = 76 Virtual Tape Option +
  Feature ID = 75 Bare Metal Restore +
  Feature ID = 74 Encryption +
  Feature ID = 73 Open File Backup +
  Feature ID = 71 Remote Client Support +
  Feature ID = 70 Robotic Library Sharing Support +
  Feature ID = 69 Remote Media Server Support +
  Feature ID = 68 Microsoft RSM Robotic Libraries +
  Feature ID = 67 ADIC DAS/SDLC Robotic Libraries +
  Feature ID = 66 IBM ATL Robotic Libraries +
  Feature ID = 65 Fujitsu LMF Robotic Libraries +
  Feature ID = 64 StorageTek ACS Robotic Libraries +
  Feature ID = 61 MS SharePoint Agent +
  Feature ID = 58 Inline Tape Copy +
  Feature ID = 54 Vault +
  Feature ID = 49 Library Based Tape Drives +
  Feature ID = 48 Additional Drives/Robotics +
  Feature ID = 46 DataStore +
  Feature ID = 43 Encryption (Legacy DES 56-bit) +
  Feature ID = 42 Encryption (Legacy DES 40-bit) +
  Feature ID = 40 SAP extension +
  Feature ID = 39 SQL-Backtrack extension +
  Feature ID = 38 Sybase extension +
  Feature ID = 37 Informix extension +
  Feature ID = 36 Oracle extension +
  Feature ID = 35 Lotus Notes extension +
  Feature ID = 34 DB2 extension +
  Feature ID = 33 MS SQL Server extension +
  Feature ID = 32 MS Exchange extension +
  Feature ID = 31 Advanced Reporting Option +
  Feature ID = 30 Shared Storage Option +
  Feature ID = 27 NDMP +
  Feature ID = 22 Additional clients +
  Feature ID = 20 Base NetBackup +
 

Shas
Level 4
Partner

 

Hi Marianne,

I have attached a file for the active licenses on the media server. 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Okay, it's all there:

NetBackup Enterprise Server
Library Based Tape Drives
Shared Storage Option

The issue you are experiencing looks like a comms issue between EMM on the master and the media server(s) using PBX (port 1556).

Can you confirm that this port is open between master and media servers in both directions?
Windows firewall disabled on all Windows servers?

Is forward and reverse lookup working in all directions?

Please post output of the following commands on the master server:

nbemmcmd -listhosts -verbose

nbemmcmd -getemmserver

Mark_Solutions
Level 6
Partner Accredited Certified

Check you can connect to it from the Master

Open the admin console - go to Media and Device Management - Devices - Media Servers

Right click on the Media Server and select Stop / Re-Start Device Manager Service

If it connects then you should be Ok for comms, if not there is an issue

When you do this it is worth clicking Apply anyway as that usually kicks a new media server into life and sets it active for tape too

Shas
Level 4
Partner

Marianne's  Answer : 

Yes , Port 1556 is open bidirectionally Host file entries is been on both the servers.

Firewall is disabled .

All the media servers are listed in the output of "nbemmcmd -getemmserver" including the new one

MEDIA 7.5 Mediaserver02 Masterserver02                       # server name has been edited

Output of "nbemmcmd -listhosts -verbose"

Mediaserver02
  ClusterName = ""
  MachineName = "Mediaserver02"
  FQName = "Mediaserver02"
  LocalDriveSeed = ""
  MachineDescription = ""
  MachineFlags = 0x17
  MachineNbuType = media (1)
  MachineState = active for disk jobs (12)
  MasterServerName = "Masterserver02"
  NetBackupVersion = 7.5.0.6 (750600)
  OperatingSystem = windows (11)
 

 

And yesterday's troubshooting

We deleted drives including Ghost drives,Uninstalled IBM drivers,Removed target entries from registry and rebooted,
Now 12 drives were showing in Device manager and in scan output
Installed IBM drivers with install_nonexclusive.exe -t
only after installation I can see 12 drives in device manager but in scan output tapes count is dangling
smtime is shows 7 ,8 and 5 
we checked from SAN zoning no. of drive is 12
 

 

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

License keys seems fine and comms also seems to be fine (at the time that you issued nbemmcmd commands).

Are these media server in a remote site? Or different vlan?
Anything that could cause intermittent comms issues?
 

EMM sends out regular 'heartbeat' comms to media servers. If media servers are slow or fail to respond in time, EMM will mark them as 'Active for disk'.

EMM heartbeat polling to media servers can be configured with "SCAN_HOST_STATUS_INTERVAL" .

See http://www.symantec.com/docs/TECH70827

The real solution will be to troubleshoot the intermittent network issues between the master and media server.

A complete restart of NBU (incl PBX) on master and media servers may also be a good idea.

Shas
Level 4
Partner
Hi Mark, I tried your option but it doesnt work :(

Deb_Wilmot
Level 6
Employee Accredited Certified

There are connectivity errors.

Media servers can go offline for tape (active for disk only) for many reasons including the following:

   - Master failed to receive the hearbeats.

    -  Drive polling fails.

If ltid is exiting it's because it can't detect any devices on the system. 

 

In this case ltid starts up, but detects a change in machine state:
17:36:15.121 [904.1212] <4> avrd: INITIATING
17:36:15.121 [6968.4420] <4> SendEmmHeartbeat: Detected change in MachineState...
17:36:15.121 [6968.4420] <4> SendEmmHeartbeat: ...clearing LOCAL_CONTROL bits
 

So apparently the heartbeats failed.

Every 5 minutes the media server will attempt to send the heart beat again...  Every time after 17:36 - it Detects a change in machine state and tries to clear local bits and up the libraries.

 

At this point all updates fail:

18:06:15.442 [6968.4420] <16> emmlib_SetRobotStatus: (0) UpdateLibrary failed, emmError = 2007031, nbError = 0
18:06:15.442 [6968.4420] <16> ROBOT_CONNECT: (-) Translating EMM_ERROR_SQLNoDataFound(2007031) to 328 in the device management context
18:06:15.442 [6968.4420] <16> ROBOT_CONNECT: emmlib_SetRobotStatus() failed for robot 3 with error 328
18:06:15.442 [6968.4420] <4> SendEmmHeartbeat: Going Active failed - reset machine state.
 

I would suggest you check the following:

WINS or DNS server issues

A NIC going bad.

You might want to schedule a bptestnetconn every 10 minutes or so to see when the media sever is not available.

I woul suggest enabling the folloiwng logigng levels on the media server:

vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -a -p 51216 -o 156 -s DebugLevel=6 -s DiagnosticLevel=6

 

Make sure that the daemon directory is also available under volmgr\debug

On the maser - if space allows - run the following:
 

vxlogcfg -a -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -a -p 51216 -o 156 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -a -p 51216 -o 137-s DebugLevel=6 -s DiagnosticLevel=6

 

Check the resulting log files for issues connecting.

An example is in http://www.symantec.com/docs/TECH69625

 

*** Don't forget to remove logging when done.

Media server:

vxlogcfg -r -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -r -p 51216 -o 156 -s DebugLevel=6 -s DiagnosticLevel=6

 

Master:
 

vxlogcfg -r -p 51216 -o 111 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -r -p 51216 -o 156 -s DebugLevel=6 -s DiagnosticLevel=6

vxlogcfg -r -p 51216 -o 137-s DebugLevel=6 -s DiagnosticLevel=6

 

Good luck

 

 

 

 

 

After that happens this entity can no longer make any updates to the database and

 

 

 

Shas
Level 4
Partner

Hi All ,

Thanks for your valuable contributions.

This issue has been resolved after OS rebuild and did persistent binding of the drive.

Problem : Drives were attached to the Library which has some issues.