(2074) Disk volume is down - Can't get a status 'UP' to stick

[BackupDan]

Bare with me as I am picking up someone else's NetBackup responsibilities and my knowledge is light at best.

We are using a Symantec NetBackup 5230 appliance and it is not completing Snapshots and backups after a reboot of the appliance. The error received is (2074) Disk volume is down. When performing a 'nbdevquery -listdv' I am seeing a status 'DOWN'. I performed a '-state UP' command and it looked like it took successfully. This did not work so I tried shutting down and restarting the NetBackup service. This too did not work. I rebooted the appliance and this also did not help. After some more research, I went to the web console and 'Appliance Diagnostics Center' where I ran a 'Check Disk Configuration'. Received 'AdvancedDisk storage check complete' in the first section and 'Following MSDP disk volumes are down:' with mine listed. I perform the requested reset and it tells me they have been reset successfully. I confirm by immediately doing 'nbdevquery -listdv' and it shows as 'UP' for just about 10 seconds before it once again begins showing as 'DOWN'. Can someone provide a little insight here as I think I am getting very close to a resolution. Haven't captured the site backups  in over a week at this point.

Thanks for any help!

[Marianne]

How full is the MSDP pool? 

Please check 'Free Space' and 'Use%' in nbdevquery -listdv -stype PureDisk -U -dp disk_pool_name output.

Also check spoold.log and storaged.log in storage_path/log/spoold for errors.

[BackupDan]

Thanks for the reply Marianne!

The Use% is 54 and the total free space is nearing 6TB. I am searching for the logs you mentioned and will provide some additional information soon. Thanks again!

[BackupDan]

I feel quite helpless and a bit embarrassed as I cannot figure out how to generate those logs from the Maintenance shell. I will continue to poke around but some specifics may be helpful.. Sorry

[BackupDan]

Added last 100 lines for each of the logs requested (spoold.log & storaged.log) in the attachment. Please let me know if this is helpful.

[Marianne]

Hi @BackupDan

I could only access your log file today - I can only view .txt extentions on my mobile phone.

We see this in your spoold.log:
Failed to get startup CR modes from SPA after xxxxx attempts

I would have a look at spad.log next (/storage_path/log/spad/spad.log)

I found this TN, but do not see the connection errors mentioned here:
https://www.veritas.com/support/en_US/article.000013199

If spad.log does not show anything helpful, then best to log a Support call with Veritas.

[BackupDan]

So as I have gotten deeper into troubleshooting, I am finding that my tech had changed IP addressing around on the appliance which is when the connectivity issues began to present themsleves. The main BOND0 was changed from one IP address to another and a second fiber connection was created to use the original IP address.

Since then, I have gone into the system and recreated the original BOND0 consisting of the 4 ethernet ports and have assigned the original IP address to this bond. I have also removed the newly added Fiber connection. I went in to clean up the Host files and am now having some other issues. When I go to delete the host file for the valid IP address, it tells me the following: 

Invalid hostname 'My Server name'. the hostname resolves to IP address '127.0.0.1' which is not configured. Reconfigure network with correct IP address.

Invalid hostname 'My Server name'. the hostname resolves to IP address 'MY IP ADDRESS' which is not configured. Reconfigure network with correct IP address.

So I ran a 'hostname show' command and received 'Failed to get Host Name'. Should I try to re-set the hostname? That makes me nervous.. Any thoughts

[Marianne]

Best to log a call with Veritas Support and share all of the info in your latest post.