External request already in progress on TLD(0)

[Chronos]

Hi guys,

We've recently had some issues with a non-functional tape drive on an STK L180, which we've replaced. However, the alerts as the ones seen below have not stopped:

Aug 23 13:28:43 backup5 tldcd[28736]: [ID 200886 daemon.error] External request already in progress on TLD(0), from host backup4
Aug 23 13:29:00 backup5 tldcd[28736]: [ID 200886 daemon.error] External request already in progress on TLD(0), from host backup4
Aug 23 13:29:18 backup5 tldcd[28736]: [ID 200886 daemon.error] External request already in progress on TLD(0), from host backup4
Aug 23 13:29:35 backup5 tldcd[28736]: [ID 200886 daemon.error] External request already in progress on TLD(0), from host backup4

My best guess is that it's somehow linked to the Vault jobs that keep hanging, which I gotta manually cancel in order to have (at least one or two) running normally afterwards:

root@backup5 # bpdbjobs | grep Vault | more
1123015          Vault Active                                           vault-daily-eject                               daily-eject        backup34_1      backup34_1 08/26/11 11:39:00 002:52:59         
                   
1123010          Vault   Done      0                                  vault-daily-reports                             daily-reports        backup34_1      backup34_1 08/26/11 10:39:03 000:05:59 08/26/11
 10:45:02          
1123001          Vault   Done    275                                      vault-daily-itc                        daily-inlinecopies        backup34_1      backup34_1 08/26/11 08:09:11 000:00:03 08/26/11
 08:09:14          
1122988          Vault   Done      0                                vault-daily-duplicate                           daily-duplicate        backup34_1      backup34_1 08/26/11 06:19:11 001:56:51 08/26/11
 08:16:02          
1122547          Vault   Done    275                                  vault-daily-reports                             daily-reports        backup34_1      backup34_1 08/25/11 10:39:16 000:00:03 08/25/11
 10:39:19          
1122539          Vault   Done    275                                      vault-daily-itc                        daily-inlinecopies        backup34_1      backup34_1 08/25/11 08:09:20 000:00:03 08/25/11
 08:09:23          
1122131          Vault   Done    150                                    vault-daily-eject                               daily-eject        backup34_1      backup34_1 08/24/11 11:39:07 030:34:44 08/25/11
 18:13:51          
1122129          Vault   Done    275                                  vault-daily-reports                             daily-reports        backup34_1      backup34_1 08/24/11 10:39:07 000:00:02 08/24/11
 10:39:09          
1122119          Vault   Done    275                                      vault-daily-itc                        daily-inlinecopies        backup34_1      backup34_1 08/24/11 08:09:10 000:00:03 08/24/11
 08:09:13          
1121803          Vault   Done     63                                vault-daily-duplicate                           daily-duplicate        backup34_1      backup34_1 08/24/11 04:08:58 038:04:39 08/25/11
 18:13:37          
1121692          Vault   Done    150                                    vault-daily-eject                               daily-eject        backup34_1      backup34_1 08/23/11 11:39:02 006:57:40 08/23/11
 18:36:42          
1121687          Vault   Done      0                                  vault-daily-reports                             daily-reports        backup34_1      backup34_1 08/23/11 10:38:53 000:05:54 08/23/11
 10:44:47          
1121675          Vault   Done      0                                      vault-daily-itc                        daily-inlinecopies        backup34_1      backup34_1 08/23/11 08:10:09 000:00:32 08/23/11
 08:10:41          
1121220          Vault   Done    275                                  vault-daily-reports                             daily-reports        backup34_1      backup34_1 08/22/11 10:39:13 000:00:03 08/22/11
 10:39:16          
1121209          Vault   Done    275                                      vault-daily-itc                        daily-inlinecopies        backup34_1      backup34_1 08/22/11 08:09:17 000:00:03 08/22/11
 08:09:20          
1120802          Vault   Done    275                                  vault-daily-reports                             daily-reports        backup34_1      backup34_1 08/21/11 10:39:02 000:00:03 08/21/11
 10:39:05          
1120796          Vault   Done    275                                      vault-daily-itc                        daily-inlinecopies        backup34_1      backup34_1 08/21/11 08:09:06 000:00:03 08/21/11
 08:09:09          
1120483          Vault   Done     63                                vault-daily-duplicate                           daily-duplicate        backup34_1      backup34_1 08/21/11 04:08:51 041:36:12 08/22/11
 21:45:03          
1120398          Vault   Done    150                                    vault-daily-eject                               daily-eject        backup34_1      backup34_1 08/20/11 11:38:53 058:06:11 08/22/11
 21:45:04          
1120395          Vault   Done      0                                  vault-daily-reports                             daily-reports        backup34_1      backup34_1 08/20/11 10:30:21 000:08:31 08/20/11
 10:38:52          
1120390          Vault   Done    275                                      vault-daily-itc                        daily-inlinecopies        backup34_1      backup34_1 08/20/11 08:09:26 000:00:03 08/20/11
 08:09:29          
1119968          Vault   Done    296                                    vault-daily-eject                               daily-eject        backup34_1      backup34_1 08/19/11 11:33:03 000:00:03 08/19/11
 11:33:06          
1119957          Vault   Done    275                                  vault-daily-reports                             daily-reports        backup34_1      backup34_1 08/19/11 10:39:09 000:00:03 08/19/11
 10:39:12          
1119950          Vault   Done    275                                      vault-daily-itc                        daily-inlinecopies        backup34_1      backup34_1 08/19/11 08:09:13 000:00:03 08/19/11
 08:09:16          
1119522          Vault   Done    296                                    vault-daily-eject                               daily-eject        backup34_1      backup34_1 08/18/11 11:38:57 000:00:03 08/18/11
 11:39:00          
1119520          Vault   Done    275                                  vault-daily-reports                             daily-reports        backup34_1      backup34_1 08/18/11 10:38:57 000:00:03 08/18/11
 10:39:00          
1119512          Vault   Done    275                                      vault-daily-itc                        daily-inlinecopies        backup34_1      backup34_1 08/18/11 08:09:01 000:00:03 08/18/11
 08:09:04          
1119195          Vault   Done     63                                vault-daily-duplicate                           daily-duplicate        backup34_1      backup34_1 08/18/11 04:11:32 053:54:28 08/20/11
 10:06:00          
1119082          Vault   Done      0                                    vault-daily-eject                               daily-eject        backup34_1      backup34_1 08/17/11 11:38:50 000:29:12 08/17/11
 12:08:02       

Notice that it's the vault-daily-eject that craps out and hangs in most cases, however I cannot 100% link the errors in the logs to the hung jobs... Any ideas as to where I might start my investigations?

NBU 5.0 (unpatched...don't ask...), Solaris 5.9, STK L180. The newly replaced TD has a newer FW version than the others, could that be an issue or should I look for something that blocked?

Thanks in advance for your englightning replies,

Dan

[Marianne]

Have you tried to reboot the whole lot? Robot as well as robot control host.

[Chronos · Accepted solution]

Thanks Marianne, actually this is exactly what I've done last Monday (how unfortunate though that I had not seen your response sooner... twould've spared me a lotta troubles....). I did not however reboot the machines themselves, I've just restarted NBU on all the media servers on which the new tape drive had been reconfigured and the errors have stopped (moreover, a whole bunch of other errors stopped too...).

So, if anyone has similar issues and stumbles upon this thread, do this: restart NBU on all the media servers.