'Error: can't query catalog' NBU5230
Hello,
This should maybe go in the appliance forum but I don't know how many experts look there so we'll keep it here for now.
Master W2K8 - 7.5.0.6
Media - NBU 5230 App - 2.5.3
Client - Redhat - Linux 7.5.0.6
Started getting this error yesterday on 1 client only. All other clients work fine.
Status 84 and then in details 'impl_image_handle: impl_get_imh_image_prop: unexpected error (2060029:authorization failure)'
Will post the full details but its not really relevant as when I check the spoold logs on the appliance I noticed that it was having trouble accessing the "database" kept in /disk/database/catalog/2
I then found that if I put the specific client in a policy by itself it run fine again. This indicates to me that there is nothing wrong with the client. It creates a new folder structure in /disk/database/catalog/2/CLIENT and goes about its business.
So the question is what exactly is going on in this folder structure (/disk/database/catalog/2/) and how do we clean it?
Errors from the spad session logs
severity: error
server:
source: spad
description: Error: can't query catalog
***DONE***
January 22 09:29:11 INFO [1094347072]: [_handle_find] filter [1|0||-1|-1|/CLIENT/LINUX_BACKUP1|*|*|*|*|*|*|*|*|*|-1|-1|-1|-1|-1|-1|-1|-1|-1|-1|*]
January 22 09:29:11 ERR [1094347072]: 25004: can't open FilePO file /disk/databases/catalog/2/CLIENT/LINUX_BACKUP1/CLIENT_1390338067_C1_F1.img
January 22 09:29:11 ERR [1094347072]: 25004: Could not load po /disk/databases/catalog/2/CLIENT/LINUX_BACKUP1/CLIENT_1390338067_C1_F1.img
January 22 09:29:11 ERR [1094347072]: 25004: can't get po /disk/databases/catalog/2/CLIENT/LINUX_BACKUP1/CLIENT_1390338067_C1_F1.img
January 22 09:29:11 ERR [1094347072]: -1: spad request failed:
***ERROR***
Any body come accross this before.
Hi,
Support managed to identify and resolve the issue.
There appears to be a corrupt image in the directory for this client and particular policy. We see the following in the logs.
January 22 09:29:11 ERR [1094347072]: 25004: can't open FilePO file /disk/databases/catalog/2/CLIENT/LINUX_BACKUP1/CLIENT_1390338067_C1_F1.img
January 22 09:29:11 ERR [1094347072]: 25004: Could not load po /disk/databases/catalog/2/CLIENT/LINUX_BACKUP1/CLIENT_1390338067_C1_F1.img
January 22 09:29:11 ERR [1094347072]: 25004: can't get po /disk/databases/catalog/2/CLIENT/LINUX_BACKUP1/CLIENT_1390338067_C1_F1.img
January 22 09:29:11 ERR [1094347072]: -1: spad request failed:
***ERROR***
4
severity: error
server:
source: spad
description: Error: can't query catalog
***DONE***
If we look at the files for the policy we can see corrupt files for this
image, one is 0 bytes and another has some temporary name:-
hct-nbu-app-02:/home/maintenance # ls -la
/disk/databases/catalog/2/CLIENT/LINUX_BACKUP1/
total 4400
drwx------ 2 root root 32768 Jan 29 04:15 .
drwx------ 6 root root 8192 Jan 22 22:06 ..
-rw-r----- 1 root root 76 Dec 26 15:37 __dirpo__
<snip>
<snip>
-rw-r----- 1 root root 0 Jan 22 01:02 CLIENT_1390338067_C1_F1.img
-rw-r----- 1 root root 161 Jan 22 01:02 CLIENT_1390338067_C1_F1.info[R_2]
-rw-r----- 1 root root 162 Jan 22 01:02 CLIENT_1390338067_C1_HDR.imgI removed the files relating to that specific image (CLIENT_1390338067) from the /disk/databases/catalog/2/CLIENT/LINUX_BACKUP1/ folder and the backups are now working again within the original policy.
We don't really know what happened for the file to be created with 0 size but it must have happened in the beginning of the job so it never accepted any data, and there was nothing in the NBU catalog. So I dindn't have to expire this image. No dataloss.