cancel
Showing results for 
Search instead for 
Did you mean: 

Backups failing with 2027 error code

Varma_Chiluvuri
Level 5
Certified

Some of the backups in our environment are failing with 2027 error code.

Netbackup media server is available

Currently backups are runing using the drives in this media server

Please find the log for the backup failure

12/03/2014 09:05:06 - Info nbjm (pid=23658736) starting backup job (jobid=3044394) for client sapbwsv-nb, policy Ora_RMAN_BWS, schedule Arch_BWS
12/03/2014 09:05:06 - Info nbjm (pid=23658736) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=3044394, request id:{62938BE6-7AF5-11E4-A27D-665E5F7F0000})
12/03/2014 09:05:06 - requesting resource media-server-nb-hcart3-robot-tld-3
12/03/2014 09:05:06 - requesting resource sun107-nb.NBU_CLIENT.MAXJOBS.sapbwsv-nb
12/03/2014 09:05:06 - requesting resource sun107-nb.NBU_POLICY.MAXJOBS.Ora_RMAN_BWS
12/03/2014 09:05:06 - Waiting for scan drive stop STK.T10000A.013, Media server: media-server-nb
12/03/2014 09:07:06 - Info nbjm (pid=23658736) Waiting in NetBackup scheduler work queue on server sun107-nb
12/03/2014 09:07:13 - Error nbjm (pid=23658736) NBU status: 2027, EMM status: Media server is not active
Media server is not active  (2027)

13 REPLIES 13

revarooo
Level 6
Employee

Let's see output of:

vmoprcmd

Is the media server down?

Varma_Chiluvuri
Level 5
Certified

vmoprcmd output

 

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart3   TLD                -                     No       -         0
  0 hcart3   TLD                -                     No       -         0
  1 hcart3   TLD                -                     No       -         0
  1 hcart3   TLD                -                     No       -         0
  2 hcart3   TLD                -                     No       -         0
  2 hcart3   TLD                -                     No       -         0
  3 hcart3   TLD               Yes   I00876  I00876   Yes     Yes        0
  3 hcart3   TLD               Yes   I00876  I00876   Yes     Yes        0
  4 hcart3   TLD                -                     No       -         0
  4 hcart3   TLD                -                     No       -         0
  5 hcart3   TLD               Yes   I00745  I00745   Yes     Yes        0
  5 hcart3   TLD               Yes   I00745  I00745   Yes     Yes        0
  6 hcart3   TLD               Yes   I00765  I00765   Yes     Yes        0
  6 hcart3   TLD               Yes   I00765  I00765   Yes     Yes        0
  7 hcart3   TLD                -                     No       -         0
  7 hcart3   TLD                -                     No       -         0
  8 hcart3   TLD                -                     No       -         0
  8 hcart3   TLD                -                     No       -         0
  9 hcart3   TLD                -                     No       -         0
  9 hcart3   TLD                -                     No       -         0
 10 hcart2   TLD                -                     No       -         0
 10 hcart2   TLD                -                     No       -         0
 11 hcart3   TLD               Yes   I00883  I00883   Yes     Yes        0
 11 hcart3   TLD               Yes   I00883  I00883   Yes     Yes        0

                             ADDITIONAL DRIVE STATUS

Drv DriveName            Shared    Assigned        Comment
  0 STK.T10000A.011       Yes      -
  0 STK.T10000A.011       Yes      -
  1 STK.T10000A.013       Yes      -
  1 STK.T10000A.013       Yes      -
  2 STK.T10000A.014       Yes      -
  2 STK.T10000A.014       Yes      -
  3 STK.T10000A.006       Yes      media-server
  3 STK.T10000A.006       Yes      media-server
  4 STK.T10000A.010       Yes      -
  4 STK.T10000A.010       Yes      -
  5 GRV_T10000_02         Yes      -
  5 GRV_T10000_02         Yes      -
  6 GRV_T10000_03         Yes      media-server
  6 GRV_T10000_03         Yes      media-server
  7 STK.T10000A.004       Yes      -
  7 STK.T10000A.004       Yes      -
  8 STK.T10000A.015       Yes      -
  8 STK.T10000A.015       Yes      -
  9 STK.T10000A.073       Yes      -
  9 STK.T10000A.073       Yes      -
 10 GRV_LTO3_05           Yes      -
 10 GRV_LTO3_05           Yes      -
 11 STK.T10000A.009       Yes      media-server
 11 STK.T10000A.009       Yes      media-server

Varma_Chiluvuri
Level 5
Certified

attached output of vmoprcmd

J_MCCOLL
Level 4

You need to run the vmoprcmd command without any switches...  This will provide the details of the shared drives, server connections and status. 

 

Varma_Chiluvuri
Level 5
Certified

Sorry for my late reply.

Please find the output without any switches

watsons
Level 6

Use "nbemmcmd -listhosts -verbose" to check the status of your media server. An active media server looks like this:

mediaserver1
        ClusterName = ""
        MachineName = "mediaserver1"
        FQName = "mediaserver1"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0x77
        MachineNbuType = media (1)
        MachineState = active for tape and disk jobs (14)
        MasterServerName = "masterserver"
        NetBackupVersion = 7.6.0.3 (760300)
        OperatingSystem = linux (16)
        ScanAbility = 5

If in the MachineState it's not active for tape, it's a problem. Use this to change the state:

nbemmcmd -updatehost -machinestateop set_tape_active  -machinename mediaserver1 -machinetype media  -masterserver masterserver

If the updatehost command does not work either, you need to make sure your master server can connect (ping, nslookup, bptestbpcd) to the media server.

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Multiple interfaces on the media server?

vmoprcmd output shows drives in use on media-server.

Job details shows a problem with media-server-nb.

Curious to see output of 'nbemmcmd -listhosts -verbose'.
This will show us how '
media-server' is configured in EMM.

Kanta1990
Level 2

Jobs are still failing with 2027:

02/17/2015 04:10:38 - Error nbjm (pid=23658736) NBU status: 2027, EMM status: Media server is not active
Media server is not active  (2027)

 

Checked status of media server by nbemmcmd -listhosts -verbose:

MachineState = active for tape and disk jobs (14)

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Are you working with the person who posted back in Dec?
Or are you having the same/similar issue?

Kanta1990
Level 2

Yes, Marianne...This issue still persists. Backup is getting successful but few channels are getting failed with below error

 

02/17/2015 04:10:38 - Error nbjm (pid=23658736) NBU status: 2027, EMM status: Media server is not active
Media server is not active  (2027)
 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Show us output of nbemmcmd -listhosts at the time when error is seen.
It is possible that there is a network glitch at the time when the error occurs.

See this TN regarding EMM heartbeat timeout: http://www.symantec.com/business/support/index?page=content&id=TECH70827 
(There seems to be a problem with Symantec Support site at the moment.)

My questions of 14 Jan were not answered:

Multiple interfaces on the media server?

vmoprcmd output shows drives in use on media-server.

Job details shows a problem with media-server-nb.

Curious to see output of 'nbemmcmd -listhosts -verbose'.
This will show us how 'media-server' is configured in EMM.

 

 

Kanta1990
Level 2

output of nbemmcmd -listhosts at present: But how to get the output for that perticular time when the issue occured?

XYZ-nb
        ClusterName = ""
        MachineName = "XYZ-nb"
        FQName = "XYZ-nb"
        LocalDriveSeed = ""
        MachineDescription = ""
        MachineFlags = 0xf7
        MachineNbuType = media (1)
        MachineState = active for tape and disk jobs (14)
        MasterServerName = "ABC-nb"
        NetBackupVersion = 7.5.0.4 (750400)
        OperatingSystem = solaris (2)
        ScanAbility = 5

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

My question again: 

Multiple interfaces on the media server?

vmoprcmd output shows drives in use on media-server.

Job details shows a problem with media-server-nb.

 

How are media servers actually configured in NBU? With which name?

 

But how to get the output for that perticular time when the issue occured?

Pay someone to monitor backups... 
Or do it yourself.
I have often in my level 1 and 2 Support days sat at a customer's site overnight to see what is happening at OS and network level when things go wrong in NBU.

Please read though the TN that I have posted to see which logs to check for EMM heartbeat errors.