Forum Discussion

Mehul_Pal's avatar
Mehul_Pal
Level 5
9 years ago

I wanted to understand if the backups are getting hung because of tape drive issue or any other issu

I wanted to understand if the backups are getting hung because of tape drive issue or any other issue in my environment

we are using NBU 7.5.0.4 RHEL 

 

NetBackup-RedHat2.6.18 7.5.0.4

we are continously facing issues from our L1 team that the backups are getting hung however when i see them i dont see hung but they take longer time then normal I checked the os logs found some tape alert & bptm also shows that. i wanted to understand if the hung thing because of that tape drive or not

 

OS logs

 

 

8/10/2016 3:13bptm[9264]: TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive IBM.ULT3580-TD5.000 (index 1), Media Id
8/10/2016 3:13bptm[9264]: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
8/10/2016 3:33bptm[6963]: TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
8/10/2016 3:33bptm[6963]: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
8/10/2016 3:33bptm[6963]: TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
8/10/2016 3:33bptm[6963]: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
8/10/2016 3:55bptm[6963]: TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
8/10/2016 3:55bptm[6963]: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
8/10/2016 3:55bptm[6963]: TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
8/10/2016 3:55bptm[6963]: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
8/10/2016 4:01bptm[28352]: TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive  (index -2), Media Id
8/10/2016 4:01bptm[28352]: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive  (index -2), Media Id
8/10/2016 4:01bptm[28352]: TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive  (index -2), Media Id
8/10/2016 4:01bptm[28352]: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive  (index -2), Media Id
8/10/2016 4:23bptm[28352]: TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive  (index -2), Media Id
8/10/2016 4:23bptm[28352]: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive  (index -2), Media Id
8/10/2016 4:23bptm[28352]: TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive  (index -2), Media Id
8/10/2016 4:23bptm[28352]: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive  (index -2), Media Id
8/10/2016 7:18bptm[6947]: TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
8/10/2016 7:18bptm[6947]: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
8/10/2016 7:18bptm[6947]: TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
8/10/2016 7:18bptm[6947]: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
8/10/2016 7:41bptm[6947]: TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
8/10/2016 7:41bptm[6947]: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
8/10/2016 7:41bptm[6947]: TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
8/10/2016 7:41bptm[6947]: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
8/10/2016 8:21bptm[6418]: TapeAlert Code: 0x03, Type: Warning, Flag: HARD ERROR, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
8/10/2016 8:21bptm[6418]: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
8/10/2016 8:21bptm[6418]: TapeAlert Code: 0x27, Type: Warning, Flag: DIAGNOSTICS REQ., from drive IBM.ULT3580-TD5.000 (index 1), Media Id
8/10/2016 8:21

bptm[6418]: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id

 

 

removed the master name from it but logs are intact

 

bptm logs

 

cat bptm/log.081016 | grep -i "<16>"
00:00:42.339 [13713] <16> really_tpunmount: error unloading media, TpErrno = Robot operation failed
00:00:42.376 [13713] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0514
00:00:42.379 [13713] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0514
00:01:20.043 [19136] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
00:01:20.045 [19136] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
00:23:48.412 [19136] <16> really_tpunmount: error unloading media, TpErrno = Robot operation failed
00:23:48.422 [19136] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
00:23:48.424 [19136] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
01:39:01.830 [13107] <16> open_ndmp_device: cannot open ndmp device nrst5a, error code 16 (NDMP_NO_DEVICE_ERR)
01:49:04.121 [13107] <16> mount_open_media: error requesting media, TpErrno = Robot operation failed
01:49:07.176 [13107] <16> open_ndmp_device: cannot open ndmp device nrst5a, error code 16 (NDMP_NO_DEVICE_ERR)
01:49:10.986 [17540] <16> open_ndmp_device: cannot open ndmp device nrst5a, error code 16 (NDMP_NO_DEVICE_ERR)
02:17:46.863 [17540] <16> really_tpunmount: error unloading media, TpErrno = Robot operation failed
02:17:49.925 [17540] <16> open_ndmp_device: cannot open ndmp device nrst5a, error code 16 (NDMP_NO_DEVICE_ERR)
02:25:48.099 [9859] <16> mount_open_media: error requesting media, TpErrno = Robot operation failed
02:25:49.010 [28304] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0593
02:25:49.012 [28304] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0593
02:48:15.656 [28304] <16> really_tpunmount: error unloading media, TpErrno = Robot operation failed
02:48:15.686 [28304] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0593
02:48:15.689 [28304] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0593
02:51:20.070 [9264] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
02:51:20.073 [9264] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
03:13:48.283 [9264] <16> really_tpunmount: error unloading media, TpErrno = Robot operation failed
03:13:48.314 [9264] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
03:13:48.316 [9264] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
03:33:04.699 [10104] <16> mount_open_media: error requesting media, TpErrno = Robot operation failed
03:33:05.390 [6963] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
03:33:05.392 [6963] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
03:55:32.124 [6963] <16> really_tpunmount: error unloading media, TpErrno = Robot operation failed
03:55:32.138 [6963] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
03:55:32.140 [6963] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0594
04:01:20.082 [28352] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive (index -2), Media Id
04:01:20.084 [28352] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive (index -2), Media Id
04:23:46.996 [28352] <16> really_tpunmount: error unloading media, TpErrno = Robot operation failed
04:23:47.030 [28352] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive (index -2), Media Id
04:23:47.032 [28352] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive (index -2), Media Id
07:18:53.712 [22589] <16> mount_open_media: error requesting media, TpErrno = Robot operation failed
07:18:54.332 [6947] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
07:18:54.334 [6947] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
07:32:40.294 [13386] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 2 (NDMP_DEVICE_BUSY_ERR)
07:33:31.664 [13386] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 2 (NDMP_DEVICE_BUSY_ERR)
07:34:22.771 [13386] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 2 (NDMP_DEVICE_BUSY_ERR)
07:35:14.546 [21099] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 2 (NDMP_DEVICE_BUSY_ERR)
07:41:25.192 [6947] <16> really_tpunmount: error unloading media, TpErrno = Robot operation failed
07:41:25.202 [6947] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
07:41:25.204 [6947] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id CA0041
08:05:20.853 [21099] <16> really_tpunmount: error unloading media, TpErrno = Request terminated by tpunmount call from another process
08:06:11.919 [21099] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 2 (NDMP_DEVICE_BUSY_ERR)
08:06:12.060 [13386] <16> RequestSpanResources: MultiResReq.cpp:2754 resource request failed [98]
08:06:12.068 [13386] <16> send_MDS_msg: Error from emmlib_handleMessage, Master cangac-bkp02, type 12, returned error 2005023
08:21:37.939 [6418] <16> really_tpunmount: error unloading media, TpErrno = Request terminated by tpunmount call from another process
08:21:37.956 [6418] <16> process_tapealert: TapeAlert Code: 0x04, Type: Critical, Flag: MEDIA, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
08:21:37.958 [6418] <16> process_tapealert: TapeAlert Code: 0x37, Type: Critical, Flag: LOADING FAILURE, from drive IBM.ULT3580-TD5.000 (index 1), Media Id
09:56:12.496 [18904] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 16 (NDMP_NO_DEVICE_ERR)
09:56:15.864 [18904] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 16 (NDMP_NO_DEVICE_ERR)
09:56:18.892 [18904] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 16 (NDMP_NO_DEVICE_ERR)
09:56:22.755 [18970] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 16 (NDMP_NO_DEVICE_ERR)
10:24:52.974 [18970] <16> really_tpunmount: error unloading media, TpErrno = Robot operation failed
10:24:56.001 [18970] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 16 (NDMP_NO_DEVICE_ERR)
11:32:52.051 [11800] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 2 (NDMP_DEVICE_BUSY_ERR)
11:48:35.388 [11800] <16> mount_open_media: error requesting media, TpErrno = Robot operation failed
11:49:26.523 [11800] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 2 (NDMP_DEVICE_BUSY_ERR)
11:50:18.380 [29436] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 2 (NDMP_DEVICE_BUSY_ERR)
12:20:36.543 [29436] <16> really_tpunmount: error unloading media, TpErrno = Request terminated by tpunmount call from another process
12:21:27.668 [29436] <16> open_ndmp_device: cannot open ndmp device nrst2a, error code 2 (NDMP_DEVICE_BUSY_ERR)
12:21:27.788 [11800] <16> RequestSpanResources: MultiResReq.cpp:2754 resource request failed [98]
12:21:27.795 [11800] <16> send_MDS_msg: Error from emmlib_handleMessage, Master cangac-bkp02, type 12, returned error 2005023
13:06:18.899 [7297] <16> mount_open_media: error requesting media, TpErrno = Robot operation failed

NBU 7.5- backup issue
  • Marianne's avatar
    Marianne
    Level 6
    9 years ago
    These errors should result in all sorts of device-related backup failures. Not hung backups.
    What do you see in Job Details for jobs that match these timestamps?

  • avinash59's avatar
    avinash59
    Level 4
    9 years ago

    Hi,

    1) Have you checked how the status of the drives are in the IBM library console. Maybe there is some issues in the      drive or the tape.

    2) And as Marrianne said, is it like only a few jobd are getting hung or is it like all tyhe jobs which are going to the tape drive are getting hung.

    Regards,

    Avinash