cancel
Showing results for 
Search instead for 
Did you mean: 
smurphy
Level 4
Employee Certified

Permanent link

 

Problem:  How to troubleshoot NetBackup for NDMP Backup failures when status code: 99 (NDMP backup failure) is reported.  Includes logging instructions.
 

Error:  STATUS CODE: 99
 

Cause:  Status 99 in NDMP backups is generic. Further troubleshooting is required.
 

Solution:

 
 

During regular (standard) NDMP backups, avoid potential NDMP communications failures between NetBackup media servers and the Network Attached Storage (NAS) host.

The following troubleshooting steps may help isolate the root cause of NDMP backup issues. If they do not, proceed with logging instructions in Step 8.

NOTE:  Terminology for the types of NDMP backups can be confusing:

  • Remote NDMP:  The backup is performed on the media server, using the device paths defined to the media server.
  • Local:  The backup is performed using the device paths attached/zoned to the NAS host.
  • 3-Way:  The backup of a NAS host is performed using the device paths attached/zoned to another NAS host

To isolate the root cause of a NDMP backup failure:

1. On the media server, test the connection to the NAS device on the NDMP port (10000) by typing the telnet command.  Try both the hostname and its IP address.  For example: telnet ndmp_host 10000

2. To test connectivity and credentials of the NDMP host and its devices:

  •  Type tpautoconf -verify [NAS hostname]
  •  Type tpautoconf -probe [NAS hostname] to  probe the scsi bus for devices (local or 3-Way only).
  •  On another master or media server, type the above two commands to check for another route or network path.
  • Compare the probe results with the native NAS commands:

          NetApp:  sysconfig -t
          EMC Celerra:  server_devconfig [datamover name] -l -s -n  (list, scsi, non-disk)
         Other:  See system documentation

3. On the NAS, and type the command ndmpd status to verify that the NDMP daemon is running.  If it is not, enable the ndmp daemon and verify again by typing the command ndmpd status.  For information on starting the ndmp daemon, see the NAS system documentation.

4. For EMC Celerra devices, ensure that the SnapSure feature is enabled.  See TECH170310 for information on SnapSure and other considerations.

5. On the media or master server, type the telnet and tpautoconf command tests again from Step 1 to test connectivity, credentials, and devices.

6.  If Local or 3-Way backup, do the following to verify that the volume can be accessed and backed up:

  • First, mount a scratch tape in the drive using robtest or some other means and perform a backup.
  • On the NAS device, type the appropriate native command:

Network Appliance (NetApp): dump 0uf nrst0a /<file system> (Substitute the proper non-rewind device path if it is not nrst0a as in this example).

This command will write data from the specified file system on the Network Appliance server directly to tape. If this dump does not complete successfully, then the problem is most likely a NetWork Appliance problem. To resolve the issue, open a call with Network Appliance technical support. If the dump does complete successfully, open a call with NetBackup support.

For EMC Celerra, refer to the server_archive command in the system documentation.

NOTE:  The Backup Path specified in the NetBackup policy must be the exact case-sensitive path as the filer sees it.

7.  For performance issues, try backing up the path in the policy to a null device and note the timing:

NetApp:  dump 0f null /vol/volname
EMC Celerra:  server_archive <datamover name> -w -f /dev/null/ -J <file path>

8. Based on your findings, proceed with any required configuration changes or, if there are no changes, begin backups or restores.

9.  If none of the above troublesooting steps resolve the issue, enable debug logging on the NetBackup media server and on the NAS device.

To enable debug logging:

1. Depending on your media server platform, do the following:

For Windows media server: 

  • On the NetBackup console, go to Host Properties --> Media Server and double-click the media server that services this backup to open the Properties dialog box. If this media server is the same as the master server, go to Master Server instead.
  • Go to the Logging section and change the global logging level to 5 (maximum).
  • Click OK to exit.

For UNIX media server: 

  • Add VERBOSE to the /usr/openv/netbackup/bp.conf file in the following format:  VERBOSE = 5

2. Make sure that the following legacy log directories exist on the media server that services the backup:

For Windows media server:
\NetBackup\logs\bpbrm
\NetBackup\logs\bptm
\NetBackup\logs\ndmpagent

For Unix media server:
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bptm
/usr/openv/netbackup/logs/ndmpagent

3. Run the following commands on the media server that services this backup to increase the unified logging levels:

For Windows media server:
\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DebugLevel=6
\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DebugLevel=6
\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=6
\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=6

For Unix media server:
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DebugLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DebugLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=6

4. Enable ndmpd debug logging on the NAS device:

  • For Network Appliance NAS, type the following to enable ndmpd debug logging:  ndmpd debug 70
  • EMC Celerra:  Enable debug logging on the datamover as described in Article TECH150646
  • For other NAS devices, see the vendor documentation.

5. Reproduce the error.

6.  Disable logging:

  • For Network Appliance NAS, type the following to disable ndmpd debug logging:  ndmpd debug 0
  • EMC Celerra, see the disable commands in TECH150646.
  • For other NAS devices, see the vendor documentation.

7. Retrieve the bpbrm, bptm, ndmpagent logs from the NetBackup media server for today's date. 

8. Do one of the following to retrieve logs from the NAS device:

  • Pull the datamover server_log from the EMC Celerra:  server_log server_2 -a -s > /output.file  (assumes ‘server_2’ is the datamover name)
  • Pull the system log and the ndmpdlog from the Network Appliance NAS:
    • /vol/etc/messages
    • <volume>/etc/log/ndmpdlog (/vol/etc/log/mlog in DataOnTap 8)

9. Type the commands on the media server to disable logging:

For Windows media server:

<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DebugLevel=0
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DebugLevel=0
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=0
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=0

For Unix media server:

/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DebugLevel=0
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DebugLevel=0
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=0
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=0

10. Create a temp directory.
 For example: C:\temp\vxul or /tmp/vxul

11. Run the following command to retrieve all unified logs for today's date:

For Windows media server: <install path>\NetBackup\bin\vxlogmgr -c -t 1 -f c:\temp\vxul

For Unix meda server: /usr/openv/netbackup/bin/vxlogmgr -c -t 1 -f /tmp/vxul

Where the '-t 1' value will retrieve the unified logs from the last hour.  Adjust the value as appropriate to go back past the beginning of the job.

12. Compress the temp directory (Zip for Windows, tar for Unix) and send to Symantec NetBackup Support along with the legacy logs and the NAS logs.

13. To create a long listing of the policy, run the following comand on the NetBackup master server: 

Windows master server: <install path>\NetBackup\bin\admincmd\bppllist <name of policy> -L

Unix master server: /usr/openv/netbackup/bin/admincmd/bppllist <name of policy> -L

14. On the media server, on the command line, run the nbsu tool as follows to create the output file:

Windows media server: 

  • In a command window, change the directory to <install path>\NetBackup\bin\support
  •  Type the following command:  nbsu -c -t -nozip

Unix media server: 

  • Type the following command: /usr/openv/netbackup/bin/support/nbsu -c -t

The script will create an output file and tell you where it was stored.

15. Cut and paste the Job Details for the job in the Activity Monitor during the backup attempt.  Send that to Symantec Support along with the nbsu output file and the bppllist output.


Related Articles:


 

 

Comments
Yasuhisa_Ishika
Level 6
Partner Accredited Certified

Quite nice!

Please add logging of Oracle ZFS Appliance if you have any information.

monty_222
Level 4
Certified

Nice to have all NDMP related information at one spot . 

rk1074
Level 6
Partner

good information

Deb_Wilmot
Level 6
Employee Accredited Certified

Please always check the information on the permanent link as the technote may change (http://www.symantec.com/docs/TECH56492 ).  An example is that the technote has recently been updated to include better disabling of logging by using the vxlogcfg command with the "-r" flag to remove the lines added when using "vxlogcfg -a"

 

i.e.

For Windows media server:

<install path>\NetBackup\bin\vxlogcfg -r -p 51216 -o 134 -s DebugLevel=6 -s DiagnosticLevel=6
<install path>\NetBackup\bin\vxlogcfg -r -p 51216 -o 151 -s DebugLevel=6 -s DiagnosticLevel=6 

For Unix media server:

/usr/openv/netbackup/bin/vxlogcfg -r -p 51216 -o 134 -s DebugLevel=6 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -r -p 51216 -o 151 -s DebugLevel=6 -s DiagnosticLevel=6

 

 *** PS.  By default DiagnosticLevel=6 is set.  Unfortunately, sometimes the default levels don't get reset properly.  Adding the diagnostic level in the command ensures that regardless of the default settings, the appropriate process logging will be enabled.

 

 

 

Rehman
Level 2

Can someone pl advise for below error:

media_server>tpautoconf -verify netapp_filer
Connecting to host "netapp_filer" as user "nbackup"...
Waiting for connect notification message...
Opening session--attempting with NDMP protocol version 4...
Opening session--successful with NDMP protocol version 4
  host supports MD5 authentication
Getting MD5 challenge from host...
Logging in using MD5 method...
ndmp_connect_client_auth failed
: host "netapp_filer" failed
NDMP failed to verify host
 

rk1074
Level 6
Partner

its the authentication error. Check for the correct credentials entry of the NDMP user in the NBU java console under the Media and device management>>credentials.

smwoodcrafts
Level 6

Thank you so much for this post. I've been having intermittent issues with this and this troubleshooting tip should come in handy.

Version history
Last update:
‎08-12-2013 12:40 PM
Updated by: