EUB
8 years agoLevel 4
Communication errors
Hello I got: Win2008R2 Master server Tape Library Quantum Scalar i500 Oracle(RHEL) client machine 3 defferent Windows 2008R2 client machine All 3 Windows client backups fails with error 98: ...
You have uploaded 3 logs.
When did the error occur and on which day?
Please post the job details of the failed backup so that we know where in the logs to search.
Screenshot 98 error's:
and logs the job:
KZWLOT08_LOTUS
30.07.2016 4:46:39 - Info nbjm(pid=6096) starting backup job (jobid=495701) for client KZWLOT08.EUB.KZ, policy KZWLOT08_LOTUS, schedule Differential-Inc
30.07.2016 4:46:39 - Info nbjm(pid=6096) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=495701, request id:{B17564B7-CF7A-4DCC-99AC-E21DA789D7B4})
30.07.2016 4:46:39 - requesting resource Any
30.07.2016 4:46:39 - requesting resource kzwnb01.eub.kz.NBU_CLIENT.MAXJOBS.KZWLOT08.EUB.KZ
30.07.2016 4:46:39 - requesting resource kzwnb01.eub.kz.NBU_POLICY.MAXJOBS.KZWLOT08_LOTUS
30.07.2016 4:46:40 - awaiting resource Any Reason: Drives are in use, Media Server: kzwnb01.eub.kz,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: Lotus, Storage Unit: kzwnb01-hcart3-robot-tld-0, Drive Scan Host: N/A
30.07.2016 4:49:16 - awaiting resource Any - No drives are available
30.07.2016 4:49:18 - awaiting resource Any Reason: Drives are in use, Media Server: kzwnb01.eub.kz,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: Lotus, Storage Unit: kzwnb01-hcart3-robot-tld-0, Drive Scan Host: N/A
30.07.2016 5:19:29 - granted resource kzwnb01.eub.kz.NBU_CLIENT.MAXJOBS.KZWLOT08.EUB.KZ
30.07.2016 5:19:29 - granted resource kzwnb01.eub.kz.NBU_POLICY.MAXJOBS.KZWLOT08_LOTUS
30.07.2016 5:19:29 - granted resource 000069
30.07.2016 5:19:29 - granted resource HP.ULTRIUM6-SCSI.000
30.07.2016 5:19:29 - granted resource kzwnb01-hcart3-robot-tld-0
30.07.2016 5:19:29 - estimated 4204336 Kbytes needed
30.07.2016 5:19:29 - Info nbjm(pid=6096) started backup (backupid=KZWLOT08.EUB.KZ_1469834369) job for client KZWLOT08.EUB.KZ, policy KZWLOT08_LOTUS, schedule Differential-Inc on storage unit kzwnb01-hcart3-robot-tld-0
30.07.2016 5:19:30 - started process bpbrm (6924)
30.07.2016 5:19:32 - Info bpbrm(pid=6924) KZWLOT08.EUB.KZ is the host to backup data from
30.07.2016 5:19:32 - Info bpbrm(pid=6924) reading file list for client
30.07.2016 5:19:32 - connecting
30.07.2016 5:19:36 - Info bpbrm(pid=6924) starting bpbkar32 on client
30.07.2016 5:19:36 - connected; connect time: 0:00:04
30.07.2016 5:19:39 - Info bpbkar32(pid=3376) Backup started
30.07.2016 5:19:39 - Info bpbkar32(pid=3376) change time comparison:<disabled>
30.07.2016 5:19:39 - Info bpbkar32(pid=3376) archive bit processing:<enabled>
30.07.2016 5:19:39 - Info bptm(pid=6340) start
30.07.2016 5:19:39 - Info bptm(pid=6340) using 65536 data buffer size
30.07.2016 5:19:39 - Info bptm(pid=6340) setting receive network buffer to 263168 bytes
30.07.2016 5:19:39 - Info bptm(pid=6340) using 30 data buffers
30.07.2016 5:19:39 - Info bptm(pid=6340) start backup
30.07.2016 5:19:39 - Info bptm(pid=6340) backup child process is pid 9152.2424
30.07.2016 5:19:39 - Info bptm(pid=6340) Waiting for mount of media id 000069 (copy 1) on server kzwnb01.eub.kz.
30.07.2016 5:19:39 - Info bptm(pid=9152) start
30.07.2016 5:19:39 - mounting 000069
30.07.2016 5:20:45 - Error bptm(pid=6340) error requesting media, TpErrno = No robot daemon or robotics are unavailable
30.07.2016 5:20:45 - Warning bptm(pid=6340) media id 000069 load operation reported an error
30.07.2016 5:20:45 - current media 000069 complete, requesting next resource Any
30.07.2016 5:50:49 - Info bptm(pid=6340) EXITING with status 98 <----------
30.07.2016 5:50:53 - Info bpbkar32(pid=3376) done. status: 98: error requesting media (tpreq)
30.07.2016 5:50:53 - end writing
error requesting media (tpreq)(98)
Backup_SQL_Detwig
31.07.2016 15:01:31 - Info nbjm(pid=6096) starting backup job (jobid=497798) for client kzwbackup01.eub.kz, policy Backup_SQL_Detwig, schedule Full
31.07.2016 15:01:31 - Info nbjm(pid=6096) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=497798, request id:{1B60CFDE-E54C-432D-84AC-0FFE0FC6132F})
31.07.2016 15:01:31 - requesting resource kzwnb01-hcart3-robot-tld-0
31.07.2016 15:01:31 - requesting resource kzwnb01.eub.kz.NBU_CLIENT.MAXJOBS.kzwbackup01.eub.kz
31.07.2016 15:01:31 - requesting resource kzwnb01.eub.kz.NBU_POLICY.MAXJOBS.Backup_SQL_Detwig
31.07.2016 15:01:31 - granted resource kzwnb01.eub.kz.NBU_CLIENT.MAXJOBS.kzwbackup01.eub.kz
31.07.2016 15:01:31 - granted resource kzwnb01.eub.kz.NBU_POLICY.MAXJOBS.Backup_SQL_Detwig
31.07.2016 15:01:31 - granted resource 000026
31.07.2016 15:01:31 - granted resource HP.ULTRIUM6-SCSI.000
31.07.2016 15:01:31 - granted resource kzwnb01-hcart3-robot-tld-0
31.07.2016 15:01:31 - estimated 168409357 Kbytes needed
31.07.2016 15:01:31 - Info nbjm(pid=6096) started backup (backupid=kzwbackup01.eub.kz_1469955691) job for client kzwbackup01.eub.kz, policy Backup_SQL_Detwig, schedule Full on storage unit kzwnb01-hcart3-robot-tld-0
31.07.2016 15:01:31 - started
31.07.2016 15:01:32 - started process bpbrm (8988)
31.07.2016 15:01:36 - Info bpbrm(pid=8988) kzwbackup01.eub.kz is the host to backup data from
31.07.2016 15:01:36 - Info bpbrm(pid=8988) reading file list for client
31.07.2016 15:01:37 - connecting
31.07.2016 15:01:40 - Info bpbrm(pid=8988) starting bpbkar32 on client
31.07.2016 15:01:40 - connected; connect time: 0:00:03
31.07.2016 15:01:42 - Info bpbkar32(pid=8128) Backup started
31.07.2016 15:01:42 - Info bptm(pid=8888) start
31.07.2016 15:01:42 - Info bptm(pid=8888) using 65536 data buffer size
31.07.2016 15:01:42 - Info bptm(pid=8888) setting receive network buffer to 263168 bytes
31.07.2016 15:01:42 - Info bptm(pid=8888) using 30 data buffers
31.07.2016 15:01:42 - Info bptm(pid=8888) start backup
31.07.2016 15:01:42 - Info bptm(pid=8888) backup child process is pid 916.10360
31.07.2016 15:01:42 - Info bptm(pid=8888) Waiting for mount of media id 000026 (copy 1) on server kzwnb01.eub.kz.
31.07.2016 15:01:42 - Info bptm(pid=916) start
31.07.2016 15:01:42 - mounting 000026
31.07.2016 15:02:50 - Info bpbkar32(pid=8128) change journal NOT enabled for <D:\sql_backup\Detwig>
31.07.2016 15:29:28 - Error bptm(pid=8888) error requesting media, TpErrno = No robot daemon or robotics are unavailable
31.07.2016 15:29:28 - Warning bptm(pid=8888) media id 000026 load operation reported an error
31.07.2016 15:29:28 - current media 000026 complete, requesting next resource Any
31.07.2016 15:59:54 - Info bptm(pid=8888) EXITING with status 98 <----------
31.07.2016 15:59:59 - Info bpbkar32(pid=8128) done. status: 98: error requesting media (tpreq)
31.07.2016 15:59:59 - end writing
error requesting media (tpreq)(98)
31.07.2016 16:00:03 - Info bpbrm(pid=7708) Starting delete snapshot processing
31.07.2016 16:00:08 - Info bpfis(pid=8112) Backup started
31.07.2016 16:00:08 - Critical bpbrm(pid=7708) from client kzwbackup01.eub.kz: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.kzwbackup01.eub.kz_1469955691.1.0
31.07.2016 16:00:08 - Info bpfis(pid=8112) done. status: 1542
31.07.2016 16:00:08 - end operation
31.07.2016 16:00:08 - Info bpfis(pid=8112) done. status: 1542: An existing snapshot is no longer valid and cannot be mounted for subsequent operations
Backup_SQL_MATM
30.07.2016 4:46:39 - Info nbjm(pid=6096) starting backup job (jobid=495699) for client kzwbackup01.eub.kz, policy Backup_SQL_MATM, schedule Differential-Inc
30.07.2016 4:46:39 - Info nbjm(pid=6096) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=495699, request id:{05140F21-D3AE-4103-8C7C-A86366320E59})
30.07.2016 4:46:39 - requesting resource kzwnb01-hcart3-robot-tld-0
30.07.2016 4:46:39 - requesting resource kzwnb01.eub.kz.NBU_CLIENT.MAXJOBS.kzwbackup01.eub.kz
30.07.2016 4:46:39 - requesting resource kzwnb01.eub.kz.NBU_POLICY.MAXJOBS.Backup_SQL_MATM
30.07.2016 4:46:40 - awaiting resource kzwnb01-hcart3-robot-tld-0 Reason: Drives are in use, Media Server: kzwnb01.eub.kz,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: SQL_DB, Storage Unit: kzwnb01-hcart3-robot-tld-0, Drive Scan Host: N/A
30.07.2016 4:49:16 - awaiting resource kzwnb01-hcart3-robot-tld-0 - No drives are available
30.07.2016 5:19:59 - granted resource kzwnb01.eub.kz.NBU_CLIENT.MAXJOBS.kzwbackup01.eub.kz
30.07.2016 5:19:59 - granted resource kzwnb01.eub.kz.NBU_POLICY.MAXJOBS.Backup_SQL_MATM
30.07.2016 5:19:59 - granted resource 000105
30.07.2016 5:19:59 - granted resource HP.ULTRIUM6-SCSI.005
30.07.2016 5:19:59 - granted resource kzwnb01-hcart3-robot-tld-0
30.07.2016 5:19:59 - estimated 29094294 Kbytes needed
30.07.2016 5:19:59 - Info nbjm(pid=6096) started backup (backupid=kzwbackup01.eub.kz_1469834399) job for client kzwbackup01.eub.kz, policy Backup_SQL_MATM, schedule Differential-Inc on storage unit kzwnb01-hcart3-robot-tld-0
30.07.2016 5:19:59 - started process bpbrm (6024)
30.07.2016 5:19:59 - started
30.07.2016 5:20:04 - Info bpbrm(pid=6024) kzwbackup01.eub.kz is the host to backup data from
30.07.2016 5:20:04 - Info bpbrm(pid=6024) reading file list for client
30.07.2016 5:20:04 - connecting
30.07.2016 5:20:06 - Info bpbrm(pid=6024) starting bpbkar32 on client
30.07.2016 5:20:06 - connected; connect time: 0:00:02
30.07.2016 5:20:08 - Info bpbkar32(pid=6456) Backup started
30.07.2016 5:20:08 - Info bptm(pid=7876) start
30.07.2016 5:20:08 - Info bptm(pid=7876) using 65536 data buffer size
30.07.2016 5:20:08 - Info bptm(pid=7876) setting receive network buffer to 263168 bytes
30.07.2016 5:20:08 - Info bptm(pid=7876) using 30 data buffers
30.07.2016 5:20:09 - Info bptm(pid=7876) start backup
30.07.2016 5:20:09 - Info bptm(pid=7876) backup child process is pid 8812.820
30.07.2016 5:20:09 - Info bptm(pid=7876) Waiting for mount of media id 000105 (copy 1) on server kzwnb01.eub.kz.
30.07.2016 5:20:09 - mounting 000105
30.07.2016 5:20:09 - Info bptm(pid=8812) start
30.07.2016 5:20:50 - Error bptm(pid=7876) error requesting media, TpErrno = No robot daemon or robotics are unavailable
30.07.2016 5:21:00 - Warning bptm(pid=7876) media id 000105 load operation reported an error
30.07.2016 5:21:00 - current media 000105 complete, requesting next resource Any
30.07.2016 5:21:47 - current media -- complete, awaiting next media Any Reason: Robotic library is down on server, Media Server: kzwnb01.eub.kz,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: SQL_DB, Storage Unit: kzwnb01-hcart3-robot-tld-0, Drive Scan Host: N/A
30.07.2016 5:21:57 - Info bpbkar32(pid=6456) change journal NOT enabled for <D:\sql_backup\MATM>
30.07.2016 5:27:27 - current media -- complete, awaiting next media Any Reason: Drives are in use, Media Server: kzwnb01.eub.kz,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: SQL_DB, Storage Unit: kzwnb01-hcart3-robot-tld-0, Drive Scan Host: N/A
30.07.2016 5:59:28 - current media -- complete, awaiting next media Any Reason: Robotic library is down on server, Media Server: kzwnb01.eub.kz,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: SQL_DB, Storage Unit: kzwnb01-hcart3-robot-tld-0, Drive Scan Host: N/A
30.07.2016 6:04:15 - Info bptm(pid=7876) Waiting for mount of media id 000105 (copy 1) on server kzwnb01.eub.kz.
30.07.2016 6:04:15 - granted resource 000105
30.07.2016 6:04:15 - granted resource HP.ULTRIUM6-SCSI.005
30.07.2016 6:04:15 - granted resource kzwnb01-hcart3-robot-tld-0
30.07.2016 6:04:15 - mounting 000105
30.07.2016 6:34:59 - Error bptm(pid=7876) error requesting media, TpErrno = No robot daemon or robotics are unavailable
30.07.2016 6:35:04 - Warning bptm(pid=7876) media id 000105 load operation reported an error
30.07.2016 6:35:04 - current media 000105 complete, requesting next resource Any
30.07.2016 7:05:31 - Info bptm(pid=7876) EXITING with status 98 <----------
30.07.2016 7:05:37 - Info bpbkar32(pid=6456) done. status: 98: error requesting media (tpreq)
30.07.2016 7:05:37 - end writing
error requesting media (tpreq)(98)
30.07.2016 7:05:40 - Info bpbrm(pid=7336) Starting delete snapshot processing
30.07.2016 7:05:48 - Info bpfis(pid=484) Backup started
30.07.2016 7:05:48 - Critical bpbrm(pid=7336) from client kzwbackup01.eub.kz: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.kzwbackup01.eub.kz_1469834399.1.0
30.07.2016 7:05:48 - Info bpfis(pid=484) done. status: 1542
30.07.2016 7:05:48 - end operation
30.07.2016 7:05:48 - Info bpfis(pid=484) done. status: 1542: An existing snapshot is no longer valid and cannot be mounted for subsequent operations
KZWLOT01_Lotus
01.08.2016 2:51:19 - Info nbjm(pid=6096) starting backup job (jobid=498127) for client kzwlot01.eub.kz, policy KZWLOT01_Lotus, schedule Differential-Inc
01.08.2016 2:51:19 - Info nbjm(pid=6096) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=498127, request id:{F694E5B4-54EA-45AF-9415-8BD1EC04BE0A})
01.08.2016 2:51:19 - requesting resource Any
01.08.2016 2:51:19 - requesting resource kzwnb01.eub.kz.NBU_CLIENT.MAXJOBS.kzwlot01.eub.kz
01.08.2016 2:51:19 - requesting resource kzwnb01.eub.kz.NBU_POLICY.MAXJOBS.KZWLOT01_Lotus
01.08.2016 2:51:19 - awaiting resource Any Reason: Drives are in use, Media Server: kzwnb01.eub.kz,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: Lotus, Storage Unit: kzwnb01-hcart3-robot-tld-0, Drive Scan Host: N/A
01.08.2016 3:03:32 - awaiting resource Any - No drives are available
01.08.2016 3:03:51 - awaiting resource Any Reason: Drives are in use, Media Server: kzwnb01.eub.kz,
Robot Number: 0, Robot Type: TLD, Media ID: N/A, Drive Name: N/A,
Volume Pool: Lotus, Storage Unit: kzwnb01-hcart3-robot-tld-0, Drive Scan Host: N/A
01.08.2016 3:19:36 - granted resource kzwnb01.eub.kz.NBU_CLIENT.MAXJOBS.kzwlot01.eub.kz
01.08.2016 3:19:36 - granted resource kzwnb01.eub.kz.NBU_POLICY.MAXJOBS.KZWLOT01_Lotus
01.08.2016 3:19:36 - granted resource 000005
01.08.2016 3:19:36 - granted resource HP.ULTRIUM6-SCSI.000
01.08.2016 3:19:36 - granted resource kzwnb01-hcart3-robot-tld-0
01.08.2016 3:19:36 - estimated 63788092 Kbytes needed
01.08.2016 3:19:36 - Info nbjm(pid=6096) started backup (backupid=kzwlot01.eub.kz_1469999976) job for client kzwlot01.eub.kz, policy KZWLOT01_Lotus, schedule Differential-Inc on storage unit kzwnb01-hcart3-robot-tld-0
01.08.2016 3:19:36 - started process bpbrm (8052)
01.08.2016 3:19:38 - Info bpbrm(pid=8052) kzwlot01.eub.kz is the host to backup data from
01.08.2016 3:19:38 - Info bpbrm(pid=8052) reading file list for client
01.08.2016 3:19:38 - connecting
01.08.2016 3:19:41 - Info bpbrm(pid=8052) starting bpbkar32 on client
01.08.2016 3:19:41 - connected; connect time: 0:00:03
01.08.2016 3:19:43 - Info bpbkar32(pid=3544) Backup started
01.08.2016 3:19:43 - Info bptm(pid=1396) start
01.08.2016 3:19:43 - Info bpbkar32(pid=3544) change time comparison:<disabled>
01.08.2016 3:19:43 - Info bpbkar32(pid=3544) archive bit processing:<enabled>
01.08.2016 3:19:43 - Info bptm(pid=1396) using 65536 data buffer size
01.08.2016 3:19:43 - Info bptm(pid=1396) setting receive network buffer to 263168 bytes
01.08.2016 3:19:43 - Info bptm(pid=1396) using 30 data buffers
01.08.2016 3:19:43 - Info bptm(pid=1396) start backup
01.08.2016 3:19:43 - Info bptm(pid=1396) backup child process is pid 10964.7812
01.08.2016 3:19:43 - Info bptm(pid=1396) Waiting for mount of media id 000005 (copy 1) on server kzwnb01.eub.kz.
01.08.2016 3:19:43 - Info bptm(pid=10964) start
01.08.2016 3:19:43 - mounting 000005
01.08.2016 3:20:24 - Error bptm(pid=1396) error requesting media, TpErrno = No robot daemon or robotics are unavailable
01.08.2016 3:20:24 - Warning bptm(pid=1396) media id 000005 load operation reported an error
01.08.2016 3:20:24 - current media 000005 complete, requesting next resource Any
01.08.2016 3:50:40 - Info bptm(pid=1396) EXITING with status 98 <----------
01.08.2016 3:50:45 - Info bpbkar32(pid=3544) done. status: 98: error requesting media (tpreq)
01.08.2016 3:50:45 - end writing
error requesting media (tpreq)(98)
Thanks for the job details.
The error is different to the previous status 98.
Before:
Warning bptm(pid=4040) media id .... load operation reported an error
We now see the following:
Waiting for mount of media id .... on server kzwnb01.eub.kz.
error requesting media, TpErrno = No robot daemon or robotics are unavailable
Which media server is robot control host for robot 0?
Can you check and verify that tldcd is running?
And tldd on media server kzwnb01?
Hopefully nobody is trying to use robtest (or forgot to exit the utility) while backups are running?
Yes, the media server kzwnb01 controls the robot 0
Tell me how to check the status of tldcd and tldd ?
From cmd, check output of 'bpps' on Windows master/media server.
(Command is in ...netbackup\bin)
There should be one tldcd process (robot control) and one tldd process for each robot-controlled tape drive.
Also check output of 'vmoprcmd -d'
(in ....volmgr\bin) to ensure that all drives are in TLD control (not AVR).
This post is getting a bit lengthy and I now have to constantly page between previous and current posts...
Not sure if I have asked the following:
Add VERBOSE entry to vm.conf on master/media server
Create debug logs on master/media such as ltid and robots (see http://www.veritas.com/docs/000109772 )
and restart Device Manager service.
Look for errors in Event Viewer System and Application logs and debug logs when status 98 is seen.
Thank you!
Here is a screenshot's of the teams "vmoprcmd -d" and "bpps"
"VERBOSE" was added earlier.
You can see that there is no tldcd process and that tape drives are in AVR (non-robotic) control.
Check Windows Event Viewer System and Application log to see when and why tldcd went down.
Please enable VERBOSE logging and create log folders, then restart NBU Device Management service.
The service restart should hopefully start tldcd again.
Verbose logging should help with future troubleshooting.
PS:
6 tape drives attached to a single master/media server seems a bit much.
Have you confirmed sufficient physical resources (memory, cpu, hba's, pci slots, bus speed, etc) to ensure that a single server can manage and stream 6 x LTO6 tape drives simultaneously?
VERBOSE included. Service were restarted few times.
Environment is old, and in that configuration works well
The master server is powerful enough (CPU 2xIntel Xeon E5-2620 2.00GHz 24 Core, 32Gb RAM, 250Gb HDD, 10Gbps LAN) OS Windows Server 2008R2
We have a robotic library Quantum Scalar i500 with 6 LTO drives in it
Why robotic library must be non-robotic?
Also we got another backup spot(media server attached to kzwnb01) with name kzwnb03 and the same hardware configuration as described above only it is installed OS Windows Server 2012R2 and no backup fails with error code-98 on it
Here is screenshot from media server kzwnb03 where all tape drives are not in AVR(and works fine).
Service were restarted few times.
So, by the looks of vmoprcmd ouput, drives are back in TLD control.
This means that the service restart has started tldcd (robot control daemon).
It also seems as if 2 tape drives are loaded - meaning backups are running that previously ended in status 98?
Next time you see status 98, please check Windows System and Application logs for possible reasons why tldcd is going down.
Services restart seems to be a short-term solution.
You need to find out why tldcd is going down as this quite abnormal.
The last time I have seen tldcd going down for no apparent reason was about 12 years ago when that particular NBU version had a problem with AUTO_UPDATE_ROBOT entry in vm.conf. tldcd went down each time the MAP was opened instead of updating the robot inventory.
Windows System and/or Application logs will tell you if the server has lost connectivity to the robot or if robot is not responding as a result of robot door being opened, etc... etc...
Other helpful logs will be debug logs under volgmr. Hopefully you have created log folders such as 'robots'?
Why robotic library must be non-robotic?
Seems you misunderstood what I was trying to say.
The tape drives were in AVR /non-robotic mode because tldcd was not running.
tldcd not running was the reason for the most recent status 98.