01-27-2021 02:51 AM
Hello,
Can someone help me ?
I need help..suddenly I have issued with our customer NBU appliance version 3.0/netbackup 8.0. When I opened the Netbackup Java Console for the Appliance, i see these options greyed out
1) Configure Disk Storage Servers
2) Configure Cloud Storage Servers
3) Configure Disk Pool
When I click on the Policy, I got the below error:
NBARSMgmt_i::queryApplicationMetadata: Error detected during
execution of query. Please check the logs for more
details (2)
(Status 2)
When I click on Activity Monitor, i get this error message :
The Activity Monitor failed because of
Connection to bpjob deamon on server failed.
For more details, refer to the log files
When i click on Device Monitor, I get the below message :
Device Monitor,
Database system error(220)
Database server is down. check for low disk space
(Don't know why i see this message, because the NBU appliance is very free : 300 GB used out of 13000 GB
When i click on the Login Activity shortcut, i see System Error 130.
This appliance we have stopped all the backup but customer is keeping it for restoration purposes.
Attached together with this is the nbu processes currently running, issued via command line.
I also received email notification on this
Your appliance has encountered the following error(s):
• The MSDP is configured, however the system failed to retrieve MSDP getmode or tlog status information. Contact Veritas Technical Support for more assistance.
o Time of event: 2021-01-27 18:41:32 (+08:00)
o UMI Event code: V-475-109-1000
o Component Type: MSDP
o Component: MSDP Partition
o Status: Not Available
o State: CRITICAL
o Additional information about this error is available at following link:
V-475-109-1000
01-27-2021 03:51 AM
[Info] Performing sanity check on disks and partitions... (5 mins approx)
----------------------------------------------------------------------------------
Partition | Total | Available | Used | %Used | Status
----------------------------------------------------------------------------------
AdvancedDisk | 130 GB | 113.92 GB | 16.08 GB | 13 | Optimal
Configuration | 100 GB | 87.90 GB | 12.10 GB | 13 | Optimal
MSDP | 13.08 TB | 12.80 TB | 285.48 GB | 3 | Optimal
MSDP Catalog | 47 GB | 46.55 GB | 465.27 MB | 1 | Optimal
NetBackup Catalog | 250 GB | 245.53 GB | 4.47 GB | 2 | Optimal
Share | 0 GB | 0 GB | 0 GB | 0 | Not Configured
Unallocated | 49.8 GB | - | - | - | -
AdvancedDisk
-------------------------------------------------------------------
Partition | Total | Available | Used | %Used | Status
-------------------------------------------------------------------
0 | 130 GB | 113.92 GB | 16.08 GB | 13 | Mounted
Check shows MDSP partition is very free
01-27-2021 04:51 AM
01-27-2021 04:51 AM
i ran a software test from the appliance
.Support> Test software
Running Validation tests on Master Appliance tuaap02dmzbkup.
Checking whether serial number is present ... [OK]
Checking whether installation is successful ... [OK]
Checking whether critical kernel drivers exist and can work ... [OK]
Checking whether SSH is enabled ... [OK]
Checking whether hostname is set ... [OK]
Checking whether IP address is set ... [OK]
Checking whether appliance user accounts exist ... [OK]
Checking whether required packages are installed ... [OK]
Checking whether BIOS firmware is compatible with the software ... [WARNING]
Checking whether registration information is properly set ... [OK]
Checking whether NetBackup is configured and running ... - [Error] Unable to get list of disk pools from master server 't'.
- [Error] Unable to get list of disk pools from master server 'tuaap02dmzbkup'.
[OK]
Checking whether database service is running ... [OK]
Checking whether message queue service is running ... [OK]
Checking whether web server service is running ... [OK]
Checking whether storage units and diskpools have been created ... - [Error] Unable to get list of disk pools from master server 'tuaap02dmzbkup'.
- [Error] Unable to get list of disk pools from master server '.
- [Error] Unable to get list of disk pools from master server '.
- [Error] Unable to get list of disk pools from master server .
- [Error] Unable to get list of disk pools from master server ''.
[NA]
Checking test backup operations ... [FAIL]
Checking test restore operations ... [NA]
Expiring test backup images and deleting test policy ... [OK]
Completed running tests on the appliance.
SELF TEST RESULT : [FAIL]
01-27-2021 05:51 AM
If I look at your attachment, it seems that 'Processes NetBackup Show
' returned nothing?
That means that NetBackup is not running.
Have you tried to start NBU with Processes NetBackup Start
?
PS:
not sure if you know this, but there is a separate NBU Appliance forum: https://vox.veritas.com/t5/NetBackup-Appliance/bd-p/netbackup-appliance
01-27-2021 06:22 AM
My apologies..not sure why previous attachment has nothing..reattaching the processes..yes the netbackup services are running
01-27-2021 07:19 AM - edited 01-27-2021 10:45 AM
If this were my nbu server? I'd shutdown Netbackup, shutdown pbx_exchange, make sure that all processes for nbu were down (bpps -x would show no active processes) and then I'd start vxpbx_exchanged, and then I'd run bp.start_all.
Check all processes via bpps -x (at an elevated prompt). Sometimes a clean restart of netbackup will fix the issues we wrestle with. IMHO, a clean start of netbackup is a great place to start. Also, if your appliance hasn't been rebooted in a long time (more than 90 days), I'd look at rebooting it.
Also, there are many fixes and enhancements in 3.2/8.2 that helps with things like this and your firmware may also need updating. Just some suggestions is all. :)
01-27-2021 07:24 AM
Hi
In posted output I cant see bpjobd deamon... IMHO restart NBU or reboot whole appliance.
01-27-2021 03:03 PM
Hi @cc1702004
Have you checked the NetBackup license to make sure it is okay - that is often a reason why things become greyed out in the GUI when a license expires.
Also I noticed in the process listing that you seem to have a NAS device mounted on the appliance as a backup target "/mnt/remote/SynNasArchive/Jan" and there are many bpdm processes attempting to run a volume_cleanup on this (and are presumably stuck). What is this volume and how is it mounted (and is it a recent change)?
Also - the process listing shown is indicating the appliance hostname (short) is nbu02p while the validation tests output is from the master server tuaap02dmzbkup. Are these the same system?
I agree with the others though that a reboot of the appliance is not going to hurt matters and may well resolve the issue if it is transient.
Cheers
David
01-27-2021 11:12 PM
I rebooted the nbu appliance..
Some of the things seem to come up..for example when i click on Device Manager, I can see the tape library, then i can activate the disabled policy, but still could not see Storage unit..
After 10 minutes, there is a message pop-up, saying there is a database system error, complaining low disk space..i don;t understand this NBU appliance we disabled all backups, 10 TB of free space...
01-27-2021 11:54 PM
Where do you see the 10TB free space? The Dedupe pool?
Please check all filesystems - I cannot remember off-hand where NBU EMM database resides on an appliance.
EMM and the relational database will shut down when file system is low on available space to prevent database corruption.
01-28-2021 09:04 AM
Have you elevated access on the appliance and looked at the individual file systems?