Hello,We are constantly but seemingly randomly getting this popup message during backups which results in jobs failing with status 13,23,24, etc. We have troubleshooted the network side and no changes have helped. We have tried some media server timeout setting changes to no avail. Any suggetions would be much appreciated.Thanks,Jamie

The fact that the errors are intermittent says to me that there were no 'recent network changes', right? Tell us more about master and media servers - same location and subnet or media server(s) at remote location?Are you using hosts files or DNS for comms?Are all servers (master and media) W2008 or any other OS's as well? Are errors seen specifically during high load?Are OS updates up to date? (There have been specific OS-related issues with W2008 under high I/O) What kind of network troubleshooting has been done?What about intermittent DNS issues?Any continuous monitoring during backup time? Something like a continuous ping?What about resource monitoring on the master? (memory, cpu, network) Have you enabled any kind of logs that may indicate 'emm heartbeat' timeouts?See this TN for logs that are needed to troubleshoot emm comms:https://www.veritas.com/support/en_US/article.100021074

One more thing - status 23 and 24 are 100% network-related issues. Herewith good explanation of these errors: https://vox.veritas.com/t5/NetBackup/Error-23/m-p/738836#M201891 Has anyone EVER encountered a network team who were prepared to admit that the problem might be network-related? Oh! And hopefully you are aware that your NBU version ran out of support about a year ago?Not that the NBU version is causing the problems, it just means that you cannot log a support call with Veritas.Support has tools to assist with network troubleshooting.

Thanks for the additional information : .... Both NICS on the master server/media sever .... Master and media server are same box All very important info that we did not know about. The 2 NICs - can we assume that they are not bonded and installed for different purposes?Do they have IP addresses on different subnets/VLans?And linked to different hostnames? This is important as the master/media also communicates between different processes via TCP/IP ports. Even though it's on the same server.If the IP's for the 2 NICs are on the same subnet and linked to the same hostname, then TCP/IP is going to 'round robin' between the NICs/IPs, causing major issues, such as request going out on one IP with response back from the other IP. The outgoing request is in the meantime expecting response on same IP. Please confirm config on the master/media with the following commands: ipconfig /all blclntcmd -self (in ...\netbackup\bin)nbemmcmd -listhosts -all (in ...\netbackup\bin\admincmd)nbemmcmd -getemmserver The logs will be handy to trace internal comms.We once picked up an error with NIC config on a media server via admin log on the master server (request going to one IP on the media server and response coming back from different IP). I also remember some time back where RiaanBadenhorst picked up an issue with 2 NICs on a master and PBX comms... will see if I can find it. **** Found it **** https://vox.veritas.com/t5/NetBackup/PBX/m-p/418924

Technical Solution: Unable to connect to the EMM server (77) after network changes To resolve the issue: 1. Flush the cache by running: UNIX platform: /usr/openv/netbackup/bin/bpclntcmd -clear_host_cache Windows platform: <install dir>\Veritas\NetBackup\bin\bpclntcmd -clear_host_cache 2. Restart NetBackup daemons on server:- For Windows:- <install dir>\Veritas\NetBackup\bin\bpdown -v -f Restart PBX:- services.msc-->Symantec Private Branch Exchange->Stop/Start or Restart service <install dir>\Veritas\NetBackup\bin\bpup -v -f For Linux/Unix:- /usr/openv/netbackup/bin/goodies/netbackup stop /opt/VRTSpbx/bin/vxpbx_exchanged stop /opt/VRTSpbx/bin/vxpbx_exchanged start /usr/openv/netbackup/bin/goodies/netbackup start

Has anyone EVER encountered a network team who were prepared to admit that the problem might be network-related?Nope, I don't think so ....

"Unable to obtain the device list from the EMM server. Unable to connect to the EMM server (77)"

Marianne
Level 6
7 years ago
The fact that the errors are intermittent says to me that there were no 'recent network changes', right?

Tell us more about master and media servers - same location and subnet or media server(s) at remote location?
Are you using hosts files or DNS for comms?
Are all servers (master and media) W2008 or any other OS's as well?

Are errors seen specifically during high load?
Are OS updates up to date? (There have been specific OS-related issues with W2008 under high I/O)

What kind of network troubleshooting has been done?
What about intermittent DNS issues?
Any continuous monitoring during backup time?
Something like a continuous ping?
What about resource monitoring on the master? (memory, cpu, network)

Have you enabled any kind of logs that may indicate 'emm heartbeat' timeouts?
See this TN for logs that are needed to troubleshoot emm comms:
https://www.veritas.com/support/en_US/article.100021074
- Marianne
  Level 6
  7 years ago
  One more thing - status 23 and 24 are 100% network-related issues.
  
  Herewith good explanation of these errors:
  
  https://vox.veritas.com/t5/NetBackup/Error-23/m-p/738836#M201891
  
  Has anyone EVER encountered a network team who were prepared to admit that the problem might be network-related?
  
  Oh! And hopefully you are aware that your NBU version ran out of support about a year ago?
  Not that the NBU version is causing the problems, it just means that you cannot log a support call with Veritas.
  Support has tools to assist with network troubleshooting.
  - mph999
    Level 6
    7 years ago
    Has anyone EVER encountered a network team who were prepared to admit that the problem might be network-related?
    Nope, I don't think so ....
- weigojmi
  Level 3
  7 years ago
  Hi all,
  Thanks for the replies. Both NICS on the master server/media sever are now changed to 1000 FULL on server and switch side (from auto). Since failures are with so many clients we will check those as a future test. I also cleared the cache as advised as well.
  Regarding the other questions:
  - Master and media server are same box
  - Using DNS and it appears to be working fine
  - We have 2008 and 2012 in our environment
  - I've noticed no load issues during the failures or as backups are running.
  - OS updates are current.
  - Regarding network troubleshooting just verifying if there are any port errors and we've tried some different teaming settings. I mentioned current status on that above. We have changed master timout settings (I know probably il advised). Here are current settings:
  Client connect: 3600
  Client read: 3600
  Backup start...: 300
  Backup end...: 300
  File browse: 1800
  Media server connect: 30
  Use OS dependent: "not checked"
  
  - No continuous monitoring yet. Partly because of how intermittent this issue is, I am often looking at the console as they fail and have checked connectivity. There never seems to be any visible issues at the time.
  - No logs enabled yet, is your "heartbeat" suggestion still valid if master/media is same box?
  - Marianne
    Level 6
    7 years ago
    Thanks for the additional information :
    
    .... Both NICS on the master server/media sever ....
    
    Master and media server are same box
    
    All very important info that we did not know about.
    
    The 2 NICs - can we assume that they are not bonded and installed for different purposes?
    Do they have IP addresses on different subnets/VLans?
    And linked to different hostnames?
    
    This is important as the master/media also communicates between different processes via TCP/IP ports. Even though it's on the same server.
    If the IP's for the 2 NICs are on the same subnet and linked to the same hostname, then TCP/IP is going to 'round robin' between the NICs/IPs, causing major issues, such as request going out on one IP with response back from the other IP. The outgoing request is in the meantime expecting response on same IP.
    
    Please confirm config on the master/media with the following commands:
    
    ipconfig /all
    
    blclntcmd -self (in ...\netbackup\bin)
    nbemmcmd -listhosts -all (in ...\netbackup\bin\admincmd)
    nbemmcmd -getemmserver
    
    The logs will be handy to trace internal comms.
    We once picked up an error with NIC config on a media server via admin log on the master server (request going to one IP on the media server and response coming back from different IP).
    
    I also remember some time back where RiaanBadenhorst picked up an issue with 2 NICs on a master and PBX comms... will see if I can find it.
    
    **** Found it ****
    
    https://vox.veritas.com/t5/NetBackup/PBX/m-p/418924