catalog backup is failaing with error code 227 in NB 7.1

[RajkumarBaniwal]

catalog backup is failing with 227 error(No entity found)

PFB logs

licy            = Catalog_Backup
Num copies        = 1
Include_file#     = 1
resourceCount     = 1
Stu               = gnnbkp81-hcart-robot-tld-0
Pool              = CatalogBackup
Retention         = 6
UseLocal          = false
(../GenBackupJob.cpp:569)
10/21/12 22:19:38.240 [Info] V-117-32 Map jobtype to column=7, column name=CATALOG_BACKUP:
10/21/12 22:19:38.240 [Info] V-117-88 FSM JobTable[CATALOG_BACKUP:StartNotifyScript] m_JobStep=1, Jobid=123296
10/21/12 22:19:38.251 [Info] V-117-122 started backup job for client gnnbkp81, policy Catalog_Backup, schedule Full on storage unit
10/21/12 22:19:38.379 [Info] V-117-94 FSM JobTable[CATALOG_BACKUP:StartNotifyScript] m_JobStep=1, Jobid=123296, returns=0
10/21/12 22:19:38.379 [Info] V-117-88 FSM JobTable[CATALOG_BACKUP:DbmQuery] m_JobStep=2, Jobid=123296
10/21/12 22:23:34.356 [JobInstance::state] (1012c6c10) state=4(../JobInstance.cpp:316)
10/21/12 22:23:34.356 [Info] V-117-94 FSM JobTable[CATALOG_BACKUP:DbmQuery] m_JobStep=2, Jobid=123296, returns=227
10/21/12 22:23:34.357 [Info] V-117-88 FSM JobTable[CATALOG_BACKUP:StopOnError] m_JobStep=-5, Jobid=123296
10/21/12 22:23:34.357 [Info] V-117-94 FSM JobTable[CATALOG_BACKUP:StopOnError] m_JobStep=3, Jobid=123296, returns=0
10/21/12 22:23:34.357 [Info] V-117-88 FSM JobTable[CATALOG_BACKUP:EndNotifyScript] m_JobStep=4, Jobid=123296
10/21/12 22:23:34.502 [Info] V-117-94 FSM JobTable[CATALOG_BACKUP:EndNotifyScript] m_JobStep=4, Jobid=123296, returns=0
10/21/12 22:23:34.502 [Info] V-117-107 *********** Jobid 123296 status summary for CATALOG_BACKUP: *********
10/21/12 22:23:34.502 [Info] V-117-114 Step 1 StartNotifyScript status 0 jobstatus -
10/21/12 22:23:34.503 [Info] V-117-114 Step 2 DbmQuery status 227 jobstatus 227
10/21/12 22:23:34.503 [Info] V-117-114 Step 3 ValidateImage status - jobstatus 227
10/21/12 22:23:34.503 [Info] V-117-114 Step 4 EndNotifyScript status 0 jobstatus 227
10/21/12 22:23:34.503 [Info] V-117-114 Step -5 StopOnError status 0 jobstatus 227
10/21/12 22:23:34.503 [Info] V-117-111 ********** job status is 227 **********
10/21/12 22:23:34.503 [Info] V-117-52 GenBackupJob Terminating status=227
10/21/12 22:23:34.504 [JobInstance::state] (1012c6c10) state=4(../JobInstance.cpp:316)
10/21/12 22:23:34.504 [JobInstance::state] (1012c6c10) state=4(../JobInstance.cpp:316)
10/21/12 22:23:34.505 [BackupJob::sendNotifications] (1012c6c10) disable resume=false, brm started=false(../BackupJob.cpp:2116)
10/21/12 22:23:34.505 V-117-239 [BackupJob::terminateThisJob] terminated job, jobid=123296, status=227
10/21/12 22:23:34.506 [BackupJob::~BackupJob] (dump Backup Job):
**************************
*BackupJob               *
(jobid=123296 jobtype=6)
**************************
isStdBackup       = false
backupType        = 6
subtype           = 35
mpxid             = {00000000-0000-0000-0000-000000000000}
maxmpx            = 1
Birthtime         = 1350838178
backupid          = gnnbkp81_1350838178
Active pid        = 0
Job Count         = 1
IPaddr            =
currentThrottle   = -1
estimatedSize     = 0
estimatedFileCount= 0
suspendable       = 0
progressLogTimerID= -1
localBackup       = 0
policy            = Catalog_Backup
Num copies        = 1
Include_file#     = 1
resourceCount     = 1
Stu               = gnnbkp81-hcart-robot-tld-0
Pool              = CatalogBackup
Retention         = 6
UseLocal          = false
(../BackupJob.cpp:289)
10/21/12 22:23:34.507 V-116-216 [RecoverableJob::doJobCompletion] jobid=123296 completed with status 227
10/21/12 22:23:34.562 V-116-222 [RecoverableJob::registerForRetryTime] retry of jobid=123296 schedule for 10/21/12 10:33 PM
10/21/12 22:23:34.575 V-118-102 [ResBroker_i::releaseOne] releasing allocation ID {4D1478E6-1B9F-11E2-96F5-0021283F909A}
10/21/12 22:23:34.585 [Info] V-116-14 CLIENT gnnbkp81  POLICY Catalog_Backup  SCHED Full  EXIT STATUS 227 (no entity was found)
10/21/12 22:23:34.622 [Error] V-116-305 catalog backup exited with status 227 (no entity was found)
10/21/12 22:23:34.636 [RecoverableJob::writeBooleans] (ID:1016f6d18 CTX:10174e8d0) flags=775(../RecoverableJob.cpp:102)
10/21/12 22:33:34.000 [RecoverableJob::writeBooleans] (ID:1016f6d18 CTX:10174e8d0) flags=775(../RecoverableJob.cpp:102)
10/21/12 22:33:34.001 V-116-215 [RecoverableJob::run_SUBMIT_RETRY] jobid=123296 submitted to nbjm for processing
10/21/12 22:33:34.066 [JobInstance::state] (1012c6c10) state=0(../JobInstance.cpp:316)
10/21/12 22:33:34.066 V-117-56 [BackupJob::sendRequestToRB] requesting resources from RB for backup job (jobid=123296)
10/21/12 22:33:34.069 V-118-227 [ResBroker_i::requestResources] received resource request ID {3F53A2D4-1BA1-11E2-8EE9-0021283F909A}, priority 99,999, secondary priority 32,767, description THE_BACKUP_JOB-123296-{3F53A2D4-1BA1-11E2-8EE9-0021283F909A}
10/21/12 22:33:34.069 [RecoverableJob::writeBooleans] (ID:1016f6d18 CTX:10174e8d0) flags=775(../RecoverableJob.cpp:102)
10/21/12 22:33:34.070 [ResBroker_i::requestResources] RequestSeq:
Request=0 provider=NamedResourceProvider resourcename=gnnbkp81.NBU_CATALOG.MAXJOBS CountedResourceRequest { max=1 }
10/21/12 22:33:34.176 V-118-226 [ResBroker_i::evaluateOne] Evaluating request ID {3F53A2D4-1BA1-11E2-8EE9-0021283F909A}
10/21/12 22:33:34.176 V-118-146 [ProviderManager::allocate] NamedResourceProvider returned Allocation Granted for request ID {3F53A2D4-1BA1-11E2-8EE9-0021283F909A}
10/21/12 22:33:34.205 [ResBroker_i::evaluateOne] resources found for request, requestId={3F53A2D4-1BA1-11E2-8EE9-0021283F909A}, userId=jobid=123296, birthTime=1350839014, timeToFirstEval=0 sec, timeToSuccessOrFailure=0 sec, numberOfTimesEvaluated=1
10/21/12 22:33:34.205 V-118-109 [ResBroker_i::passOne] resources found for request ID {3F53A2D4-1BA1-11E2-8EE9-0021283F909A}, following allocations are granted:
10/21/12 22:33:34.205 V-118-228 [ResBroker_i::passOne] allocation ID {3F646A10-1BA1-11E2-8169-0021283F909A}, group ID {00000000-0000-0000-0000-000000000000}, provider NamedResourceProvider, resource gnnbkp81.NBU_CATALOG.MAXJOBS
10/21/12 22:33:34.205 [ResBroker_i::passOne] allocation: (Allocation: id={3F646A10-1BA1-11E2-8169-0021283F909A} provider=NamedResourceProvider resourcename=gnnbkp81.NBU_CATALOG.MAXJOBS masterserver=gnnbkp81 groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=123296"  firstuserid="jobid=123296" named resource allocation)
10/21/12 22:33:34.211 V-118-249 [CallbackQueue::resourcesAvailable] added allocations to the callback queue for request ID {3F53A2D4-1BA1-11E2-8EE9-0021283F909A}
10/21/12 22:33:34.211 V-118-250 [CallbackQueue::notifyClientAvailable] queueing notification of availability of resources for request ID {3F53A2D4-1BA1-11E2-8EE9-0021283F909A}
10/21/12 22:33:34.211 V-118-256 [CorbaCall_notifyClientAvailable::execute] sending availability of resources to nbjm for request ID {3F53A2D4-1BA1-11E2-8EE9-0021283F909A}
10/21/12 22:33:34.213 [JobInstance::state] (1012c6c10) state=1(../JobInstance.cpp:316)
10/21/12 22:33:34.213 V-117-296 [BackupJob::extractResourceAttributes] unable to get storage unit resource attribute
10/21/12 22:33:34.213 V-117-296 [BackupJob::extractResourceAttributes] unable to get media server resource attribute
10/21/12 22:33:34.213 V-117-296 [BackupJob::extractResourceAttributes] unable to get stu name resource attribute
10/21/12 22:33:34.214 V-117-296 [BackupJob::extractResourceAttributes] unable to get mpxid resource attribute
10/21/12 22:33:34.214 [BackupJob::extractResourceAttributes] (1012c6c10) unable to get FAT pipe transport(../BackupJob.cpp:1208)
10/21/12 22:33:34.215 [BackupJob::processProxyResponse] (1012c6c10) Done: flush proxy response, next:start job, jobid=123296(../BackupJob.cpp:3347)
10/21/12 22:33:34.215 [JobInstance::state] (1012c6c10) state=1(../JobInstance.cpp:316)
10/21/12 22:33:34.215 [JobInstance::state] (1012c6c10) state=1(../JobInstance.cpp:316)
10/21/12 22:33:34.217 [BackupJob::birthtimeArrived] (1012c6c10) birth time has arrived(../BackupJob.cpp:1649)
10/21/12 22:33:34.261 [GenBackupJob::processProxyResponse] (1012c6c10) Done: create image, Next: start genjob, proxy status=0(../GenBackupJob.cpp:678)
10/21/12 22:33:34.261 [JobInstance::state] (1012c6c10) state=3(../JobInstance.cpp:316)
10/21/12 22:33:34.261 [GenBackupJob::spawnGenericJob] (1012c6c10) start CommandActions(../GenBackupJob.cpp:496)
10/21/12 22:33:34.263 [GenBackupJob::spawnGenericJob] GenBackupJob Commandline Options=
**************************
* GenBackupJob           *
**************************
Jobid            = 123296
GroupId          = 123296
SubType          = 35
MediaServer      =
MasterServer     = gnnbkp81
PolicyName       = Catalog_Backup
SchedName        = Full
SchedType        = 0
ClientName       = gnnbkp81
ClientType       = 35
Stream Num       = -1
Birthtime        = 1350839014
BackupCopy       = 0
BackupId         = gnnbkp81_1350839014
SnapshotId       =
User             = root
BMRSave          = no
FilerDiscovery   = no
StreamDiscovery  = no
UseProxy         = no
Resume           = no
Checkpoint       = no
UseInternalJobid = no
OpaqueString     =

**************************
*BackupJob               *
(jobid=123296 jobtype=6)
**************************
isStdBackup       = false
backupType        = 6
subtype           = 35
mpxid             = {00000000-0000-0000-0000-000000000000}
maxmpx            = 1
Birthtime         = 1350839014
backupid          = gnnbkp81_1350839014
Active pid        = 0
Job Count         = 1
IPaddr            =
currentThrottle   = -1
estimatedSize     = 0
estimatedFileCount= 0
suspendable       = 0
progressLogTimerID= -1
localBackup       = 0
policy            = Catalog_Backup
Num copies        = 1
Include_file#     = 1
resourceCount     = 1
Stu               = gnnbkp81-hcart-robot-tld-0
Pool              = CatalogBackup
Retention         = 6
UseLocal          = false
(../GenBackupJob.cpp:569)
10/21/12 22:33:34.264 [Info] V-117-32 Map jobtype to column=7, column name=CATALOG_BACKUP:
10/21/12 22:33:34.264 [Info] V-117-88 FSM JobTable[CATALOG_BACKUP:StartNotifyScript] m_JobStep=1, Jobid=123296
10/21/12 22:33:34.287 [Info] V-117-122 started backup job for client gnnbkp81, policy Catalog_Backup, schedule Full on storage unit
10/21/12 22:33:34.399 [Info] V-117-94 FSM JobTable[CATALOG_BACKUP:StartNotifyScript] m_JobStep=1, Jobid=123296, returns=0
10/21/12 22:33:34.399 [Info] V-117-88 FSM JobTable[CATALOG_BACKUP:DbmQuery] m_JobStep=2, Jobid=123296
10/21/12 22:37:40.037 [JobInstance::state] (1012c6c10) state=4(../JobInstance.cpp:316)
10/21/12 22:37:40.038 [Info] V-117-94 FSM JobTable[CATALOG_BACKUP:DbmQuery] m_JobStep=2, Jobid=123296, returns=227
10/21/12 22:37:40.038 [Info] V-117-88 FSM JobTable[CATALOG_BACKUP:StopOnError] m_JobStep=-5, Jobid=123296
10/21/12 22:37:40.038 [Info] V-117-94 FSM JobTable[CATALOG_BACKUP:StopOnError] m_JobStep=3, Jobid=123296, returns=0
10/21/12 22:37:40.038 [Info] V-117-88 FSM JobTable[CATALOG_BACKUP:EndNotifyScript] m_JobStep=4, Jobid=123296
10/21/12 22:37:40.232 [Info] V-117-94 FSM JobTable[CATALOG_BACKUP:EndNotifyScript] m_JobStep=4, Jobid=123296, returns=0
10/21/12 22:37:40.233 [Info] V-117-107 *********** Jobid 123296 status summary for CATALOG_BACKUP: *********
10/21/12 22:37:40.233 [Info] V-117-114 Step 1 StartNotifyScript status 0 jobstatus -
10/21/12 22:37:40.233 [Info] V-117-114 Step 2 DbmQuery status 227 jobstatus 227
10/21/12 22:37:40.233 [Info] V-117-114 Step 3 ValidateImage status - jobstatus 227
10/21/12 22:37:40.233 [Info] V-117-114 Step 4 EndNotifyScript status 0 jobstatus 227
10/21/12 22:37:40.233 [Info] V-117-114 Step -5 StopOnError status 0 jobstatus 227
10/21/12 22:37:40.234 [Info] V-117-111 ********** job status is 227 **********
10/21/12 22:37:40.234 [Info] V-117-52 GenBackupJob Terminating status=227
10/21/12 22:37:40.234 [JobInstance::state] (1012c6c10) state=4(../JobInstance.cpp:316)
10/21/12 22:37:40.235 [JobInstance::state] (1012c6c10) state=4(../JobInstance.cpp:316)
10/21/12 22:37:40.235 [BackupJob::sendNotifications] (1012c6c10) disable resume=false, brm started=false(../BackupJob.cpp:2116)
10/21/12 22:37:40.235 V-117-239 [BackupJob::terminateThisJob] terminated job, jobid=123296, status=227
10/21/12 22:37:40.236 [BackupJob::~BackupJob] (dump Backup Job):
**************************
*BackupJob               *
(jobid=123296 jobtype=6)
**************************
isStdBackup       = false
backupType        = 6
subtype           = 35
mpxid             = {00000000-0000-0000-0000-000000000000}
maxmpx            = 1
Birthtime         = 1350839014
backupid          = gnnbkp81_1350839014
Active pid        = 0
Job Count         = 1
IPaddr            =
currentThrottle   = -1
estimatedSize     = 0
estimatedFileCount= 0
suspendable       = 0
progressLogTimerID= -1
localBackup       = 0
policy            = Catalog_Backup
Num copies        = 1
Include_file#     = 1
resourceCount     = 1
Stu               = gnnbkp81-hcart-robot-tld-0
Pool              = CatalogBackup
Retention         = 6
UseLocal          = false
(../BackupJob.cpp:289)
10/21/12 22:37:40.238 V-116-216 [RecoverableJob::doJobCompletion] jobid=123296 completed with status 227
10/21/12 22:37:40.259 V-116-221 [RecoverableJob::run_PROCESS_STATUS] retry of jobid=123296 exceeds failure history, ending retries
10/21/12 22:37:40.260 [RecoverableJob::writeBooleans] (ID:1016f6d18 CTX:10174e8d0) flags=775(../RecoverableJob.cpp:102)
10/21/12 22:37:40.268 [PemTask::taskComplete] (ID:1016f6d18 CTX:10174e8d0) Active subtask count=0(../PemTask.cpp:529)
10/21/12 22:37:40.287 V-118-102 [ResBroker_i::releaseOne] releasing allocation ID {3F646A10-1BA1-11E2-8169-0021283F909A}
10/21/12 22:37:40.292 [Info] V-116-14 CLIENT gnnbkp81  POLICY Catalog_Backup  SCHED Full  EXIT STATUS 227 (no entity was found)
10/21/12 22:37:40.344 [Error] V-116-305 catalog backup exited with status 227 (no entity was found)
 

[Dyneshia]

Please confrim on the excact version of Netbackup you are running.  There was an issue with catalog backups after upgrading to 7.1.x

Please also reply back with your policy information : bin\admincmd\bppllist <policy name> -L

One last question, when was the last sucessful full catalog backup ?

[Anshu_Pathak]

Catalog backup creates - 1 parent job, 1 staging job, 1 Relational database backup job, 1 Image database backup job so total 4 jobs and incase of NBAC there is one more job (5 jobs).

From the logs pasted above I am assuming that the staging job is failing.

Staging is done by BPDBM process hence we will need BPDBM log snippet. 

Could you please confirm which job is failing.

[Mark_Solutions]

The log indicates that it cannot locate the media server and / or storage unit listed in the policy

Can you clarify that gnnbkp81 is up and running and has the tape library specified still exists and also that the Catalog_Backup pool exists

Thanks

[RajkumarBaniwal]

@Dyneshia: We are using NB 7.1.0.3.catalog was running fine even after upgrade also.We have last successful catalog backup is of this year july month.

root@gnnbkp81 # ./bppllist Catalog_Backup -L

Policy Name:       Catalog_Backup
Options:           0x0
template:          FALSE
audit_reason:         ?
Names:             (none)
Policy Type:       NBU-Catalog (35)
Active:            yes
Effective date:    10/21/2012 22:02:23
Mult. Data Stream: no
Perform Snapshot Backup:   no
Snapshot Method:           (none)
Snapshot Method Arguments: (none)
Perform Offhost Backup:    no
Backup Copy:               0
Use Data Mover:            no
Data Mover Type:           -1
Use Alternate Client:      no
Alternate Client Name:     (none)
Use Virtual Machine:      0
Hyper-V Server Name:     (none)
Enable Instant Recovery:   no
Policy Priority:   99999
Max Jobs/Policy:   1
Disaster Recovery: 0
Collect BMR Info:  no
Keyword:           (none specified)
Data Classification:       -
Residence is Storage Lifecycle Policy:    no
Client Encrypt:    no
Checkpoint:        no
Residence:         gnnbkp81-hcart-robot-tld-0
Volume Pool:       CatalogBackup
Server Group:      *ANY*
Granular Restore Info:  no
Exchange Source attributes:              no
Exchange 2010 Preferred Server: (none defined)
Application Discovery:      no
Discovery Lifetime:      0 seconds
Generation:      8
Ignore Client Direct:  no
Client/HW/OS/Pri:  gnnbkp81 Solaris SunOS 0 0 0 0 ?
Include:           CATALOG_DRIVEN_BACKUP
Schedule:          Full
  Type:            FULL (0)
  Frequency:       7 day(s) (604800 seconds)
  Maximum MPX:     1
  Synthetic:       0
  PFI Recovery:    0
  Retention Level: 6 (6 months)
  u-wind/o/d:      0 0
  Incr Type:       DELTA (0)
  Alt Read Host:   (none defined)
  Max Frag Size:   0 MB
  Number Copies:   1
  Fail on Error:   0
  Residence:       (specific storage unit not required)
  Volume Pool:     (same as policy volume pool)
  Server Group:    (same as specified for policy)
  Residence is Storage Lifecycle Policy:     0
  Daily Windows:
   Day         Open       Close       W-Open     W-Close
   Sunday      000:00:00  024:00:00   000:00:00  024:00:00
   Monday      000:00:00  024:00:00   024:00:00  048:00:00
   Tuesday     000:00:00  024:00:00   048:00:00  072:00:00
   Wednesday   000:00:00  024:00:00   072:00:00  096:00:00
   Thursday    000:00:00  024:00:00   096:00:00  120:00:00
   Friday      000:00:00  024:00:00   120:00:00  144:00:00
   Saturday    000:00:00  024:00:00   144:00:00  168:00:00
Schedule:          Differential-Inc
  Type:            INCR (1)
  Frequency:       1 day(s) (86400 seconds)
  Maximum MPX:     1
  Synthetic:       0
  PFI Recovery:    0
  Retention Level: 2 (3 weeks)
  u-wind/o/d:      0 0
  Incr Type:       DELTA (0)
  Alt Read Host:   (none defined)
  Max Frag Size:   0 MB
  Number Copies:   1
  Fail on Error:   0
  Residence:       (specific storage unit not required)
  Volume Pool:     (same as policy volume pool)
  Server Group:    (same as specified for policy)
  Residence is Storage Lifecycle Policy:     0
  Daily Windows:
   Day         Open       Close       W-Open     W-Close
   Sunday      000:00:00  024:00:00   000:00:00  024:00:00
   Monday      000:00:00  024:00:00   024:00:00  048:00:00
   Tuesday     000:00:00  024:00:00   048:00:00  072:00:00
   Wednesday   000:00:00  024:00:00   072:00:00  096:00:00
   Thursday    000:00:00  024:00:00   096:00:00  120:00:00
   Friday      000:00:00  024:00:00   120:00:00  144:00:00
   Saturday    000:00:00  024:00:00   144:00:00  168:00:00
Catalog Disaster Recovery Configuration:
  Email Address:   raj.kumar13@uninor.in
  Disk Path:       /var/backup/catalog_dont_delete
  User Name:       root
  Pass Word:       xxxx
  Critical policy: (none specified)
 

@Anshu: only Catalog backup parent job is failing, remaining all jobs are running fine.

@mark _solutions:  The server on which this backup is running is named as gnnbkp81 and library is directly connected with it. All backups are running fine .Only catalog backup is failing.

[Mark_Solutions]

What changes have you made to this policy?

The reason that I ask is that the policy listing says:

Effective date: 10/21/2012 22:02:23
 

and the failed job ran at 10/21/12 22:19

Is this the first time that job has run?

If only the parent job fails perhaps it is failing to do something else like save the DR file -  is there proper access to /var/backup/catalog_dont_delete and is mail setup correctly to send mail to your address?

[RajkumarBaniwal]

the backup with old policy was also failing with same error. So I deleted the same and created new policy with different schedule so that I can check the same in my work time.

server can easily access its local path /var/backup/catalo_dont-delete.

Main thing is that mail setup is not configured on the server.So I disabled the same(send Dr file to mail address) and checked but still getting same error. i beleive its is getting error out while making DR file....   Please suggest

[Dyneshia]

Please upload the bpdbm and bptm log

[Dyneshia]

Please upload the bpdbm and bptm log

[Marianne]

Please remove this from Disaster Recovery tab:

User Name:       root
  Pass Word:       xxxx

There is no need for user name and password if DR-file is written to same server.

[RajkumarBaniwal]

Hi Marriane

Still it is failing with same error

[Marianne]

What is the actual dr folder? 

Your post above says:  /var/backup/catalo_dont-delete

Policy output:  /var/backup/catalog_dont_delete

Please upload bpdbm log as File attachment as per Dyneshia's request?

[RajkumarBaniwal]

that was typo mistake .

policy output is /var/backup/catalog_dont_delete.

Ans sharing bpdbm and bptm logs soon

[RajkumarBaniwal]

PFA logs bpdbm and bptm

[Mark_Solutions]

In you logs are these entries:

02:01:02.645 [23648] <16> db_get_image_info: gnndwh_1278625579_INCR doesn't contain valid image info, the file may be corrupted

02:01:02.663 [23648] <16> list_client_images: Bad image header: gnndwh_1278625579_INCR, error: events out of sequence - image inconsistency (229)

May be worth running a catalog consistency check before going any further with this one as it may just be corruption causing the issues

[RajkumarBaniwal · Accepted solution]

Issue is as master server was previously running on 7.1.0.3 version and someone uninstalled patch version 3 from testing master server.

After upgrading the same to 7.1.0.3 it is working fine.

thanks Dyneshia for your support

[Marianne]

 someone uninstalled patch version 3

I am surprised that failed catalog was the ONLY problem after uninstalling the patch... No  other symptoms?