ORA-19511: non RMAN, but media manager or vendor specific failure, error text: Failed to process backup file <c-2057956423-20150516-0c>
Hi All, I'm trying to restore Oracle 12c database backup, but unable to restore it, script using for restoring controlfile :- run { set until time "to_date('2015/05/15 18:40:00','YYYY/MM/DD HH24:MI:SS')"; allocate channel ch1 type 'sbt_tape' parms="ENV=(NB_ORA_CLASS=AIR_ABCDB12_DB_DSTGEDV_ONLINE, NB_ORA_SERV=abcbkp02, NB_ORA_CLIENT=abcdb12)"; restore controlfile; alter database mount; } Error : - channel ch1: ORA-19870: error while restoring backup piece c-2057956423-20150516-0d ORA-19507: failed to retrieve sequential file, handle="c-2057956423-20150516-0d", parms="" ORA-27029: skgfrtrv: sbtrestore returned error ORA-19511: non RMAN, but media manager or vendor specific failure, error text: Failed to process backup file <c-2057956423-20150516-0d> Detailed log :- abcbkp02:> rman target / catalog rman/xxx@rmancat Recovery Manager: Release 12.1.0.2.0 - Production on Fri May 15 20:10:23 2015 Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved. connected to target database: DSTGEDV (not mounted) connected to recovery catalog database run { set until time "to_date('2015/05/15 18:40:00','YYYY/MM/DD HH24:MI:SS')"; allocate channel ch1 type 'sbt_tape' parms="ENV=(NB_ORA_CLASS=AIR_ABCDB12_DB_DSTGEDV_ONLINE, NB_ORA_SERV=abcbkp02, NB_ORA_CLIENT=abcdb12)"; restore controlfile; alter database mount; } executing command: SET until clause allocated channel: ch1 channel ch1: SID=122 device type=SBT_TAPE channel ch1: Veritas NetBackup for Oracle - Release 7.6 (2015021210) Starting restore at 15-MAY-15 new media label is "@aaaah" for piece "c-2057956423-20150515-2b" channel ch1: starting datafile backup set restore channel ch1: restoring control file channel ch1: reading from backup piece c-2057956423-20150515-2b channel ch1: ORA-19870: error while restoring backup piece c-2057956423-20150515-2b ORA-19507: failed to retrieve sequential file, handle="c-2057956423-20150515-2b", parms="" ORA-27029: skgfrtrv: sbtrestore returned error ORA-19511: non RMAN, but media manager or vendor specific failure, error text: Failed to process backup file <c-2057956423-20150515-2b> failover to previous backup new media label is "@aaaah" for piece "c-2057956423-20150515-2a" channel ch1: starting datafile backup set restore channel ch1: restoring control file channel ch1: reading from backup piece c-2057956423-20150515-2a channel ch1: ORA-19870: error while restoring backup piece c-2057956423-20150515-2a ORA-19507: failed to retrieve sequential file, handle="c-2057956423-20150515-2a", parms="" ORA-27029: skgfrtrv: sbtrestore returned error ORA-19511: non RMAN, but media manager or vendor specific failure, error text: Failed to process backup file <c-2057956423-20150515-2a> failover to previous backup ^C user interrupt received PSDRPC returns significant error 1013. released channel: ch1 RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of restore command at 05/15/2015 20:10:33 RMAN-03099: job cancelled at user request ************************** Please let me know incase if any other info needed for this. Thanks in advance for your suggestions/recommendations/solution.!Solved67KViews1like12Commentssnapshot failed quiesce the virtual machine
Hi All, Vm backup failling with snapshot error (156)An error occurred while saving the snapshot: Failed to quiesce the virtual machine. below is the error message: 4-03-2016 23:17:20 - Critical bpbrm(pid=13524) from client : FTL - VMware_freeze: VIXAPI freeze (VMware snapshot) failed with -1: Unrecognized error 14-03-2016 23:17:20 - Critical bpbrm(pid=13524) from client : FTL - VMware error received: An error occurred while saving the snapshot: Failed to quiesce the virtual machine. 14-03-2016 23:17:20 - Critical bpbrm(pid=13524) from client : FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze 14-03-2016 23:17:24 - Critical bpbrm(pid=13524) from client : FTL - 14-03-2016 23:17:24 - Critical bpbrm(pid=13524) from client : FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze 14-03-2016 23:17:24 - Critical bpbrm(pid=13524) from client : FTL - 14-03-2016 23:17:24 - Critical bpbrm(pid=13524) from client : FTL - snapshot processing failed, status 156 14-03-2016 23:17:24 - Critical bpbrm(pid=13524) from client : FTL - snapshot creation failed, status 156 Please help on this.Solved22KViews0likes4Comments
How to kill a hung job that wont cancel
I have run nbrbutil -dump and I can see the below lines in reference to a job I have hung in the queue that just wont cancel out. index=6 (Allocation: id={7968DB34-F37A-11E4-8E5C-002128A6F97C} provider=NamedResourceProvider resourcename=xp53tape001.NBU_POLICY.MAXJOBS.Omaha_VMware2 masterserver=xp53tape001 groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=3036644" firstuserid="jobid=3036644" named resource allocation) index=7 (Allocation: id={7968DB5C-F37A-11E4-AEB0-002128A6F97C} provider=NamedResourceProvider resourcename=xp53tape001.NBU_CLIENT.MAXJOBS.53tape002.botw.ad.bankofthewest.com masterserver=xp53tape001 groupid={00000000-0000-0000-0000-000000000000} userSequence=-2 userid="jobid=3036644" firstuserid="jobid=3036644" named resource allocation) index=8 (Allocation: id={7968D8E6-F37A-11E4-B17F-002128A6F97C} provider=DriveOperationProvider resourcename= masterserver=xp53tape001 groupid={00000000-0000-0000-0000-000000000000} userSequence=0 userid="jobid=3036644" firstuserid="jobid=3036644" (Media_Drive_Allocation_Record: allocationKey=0 (Media_Drive_Record: MediaKey=0 MediaId= MediaServer=53tape002 DriveKey=0 DriveName= PrimaryPath= PoolName= RobotNum=0 RobotType=0 MediaTypeName= DriveTypeName= NdmpControlHost= RetentionLevel=3 PolicyType=2 JobType=7 MasterServer=xp53tape001) (Storage_Unit_Record: STU= STUType=0 MasterServer=xp53tape001 MediaServer=53tape002 RobotType=0 RobotNumber=0 Density=0 OnDemandOnly=0 ConcurrentJobs=0 ActiveJobs=0 MaxMultiplexing=0 NdmpAttachHost= AbsolutePath=) (Bptm_Strings_Record: ) TpReqFileName=)) Is it possible to do something with nbrbutil -releaseMDS to kill this job? I wasnt sure if I could use the jobid or possibly the allocation ID. Also it looks to have several different allocation ID's so I figured I'd ask.Solved19KViews0likes8Comments(4207) Could not fetch snapshot metadata or state files
hi all please i have netbackup 7.6 on linux wend i tried to backup exchanche i have this error : ²20 avr. 2015 15:47:58 - Info nbjm (pid=4200) starting backup job (jobid=50468) for client mailboxeserver, policy nbuexchgdb, schedule exchbckweek_full 20 avr. 2015 15:47:58 - Info nbjm (pid=4200) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=50468, request id:{9E269824-E774-11E4-A3F7-99E7FA481C67}) 20 avr. 2015 15:47:58 - requesting resource studskfile 20 avr. 2015 15:47:58 - requesting resource netbserver.NBU_CLIENT.MAXJOBS.mailboxeserver 20 avr. 2015 15:47:58 - requesting resource netbserver.NBU_POLICY.MAXJOBS.nbuexchgdb 20 avr. 2015 15:47:58 - granted resource netbserver.NBU_CLIENT.MAXJOBS.mailboxeserver 20 avr. 2015 15:47:58 - granted resource netbserver.NBU_POLICY.MAXJOBS.nbuexchgdb 20 avr. 2015 15:47:58 - granted resource MediaID=@aaaay;DiskVolume=PureDiskVolume;DiskPool=MSDP01;Path=PureDiskVolume;StorageServer=netbserver;MediaServer=netbserver 20 avr. 2015 15:47:58 - granted resource studskfile 20 avr. 2015 15:47:58 - estimated 298453854 kbytes needed 20 avr. 2015 15:47:58 - begin Parent Job 20 avr. 2015 15:47:58 - begin Snapshot: Start Notify Script 20 avr. 2015 15:47:58 - Info RUNCMD (pid=3052) started 20 avr. 2015 15:47:58 - Info RUNCMD (pid=3052) exiting with status: 0 Operation Status: 0 20 avr. 2015 15:47:58 - end Snapshot: Start Notify Script; elapsed time 0:00:00 20 avr. 2015 15:47:58 - begin Snapshot: Step By Condition Operation Status: 0 20 avr. 2015 15:47:58 - end Snapshot: Step By Condition; elapsed time 0:00:00 20 avr. 2015 15:47:58 - begin Snapshot: Stream Discovery Operation Status: 0 20 avr. 2015 15:47:58 - end Snapshot: Stream Discovery; elapsed time 0:00:00 20 avr. 2015 15:47:58 - begin Snapshot: Read File List Operation Status: 0 20 avr. 2015 15:47:58 - end Snapshot: Read File List; elapsed time 0:00:00 20 avr. 2015 15:47:58 - begin Snapshot: Create Snapshot 20 avr. 2015 15:47:59 - started process bpbrm (pid=3058) 20 avr. 2015 15:48:03 - Info bpbrm (pid=3058) mailboxeserver is the host to backup data from 20 avr. 2015 15:48:03 - Info bpbrm (pid=3058) reading file list for client 20 avr. 2015 15:48:03 - Info bpbrm (pid=3058) start bpfis on client 20 avr. 2015 15:48:03 - Info bpbrm (pid=3058) Starting create snapshot processing 20 avr. 2015 15:48:04 - Info bpfis (pid=9244) Backup started 20 avr. 2015 15:48:31 - Critical bpbrm (pid=3058) from client mailboxeserver: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit 20 avr. 2015 15:48:31 - Critical bpbrm (pid=3058) from client mailboxeserver: FTL - 20 avr. 2015 15:48:31 - Critical bpbrm (pid=3058) from client mailboxeserver: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit 20 avr. 2015 15:48:31 - Critical bpbrm (pid=3058) from client mailboxeserver: FTL - 20 avr. 2015 15:48:31 - Critical bpbrm (pid=3058) from client mailboxeserver: FTL - snapshot processing failed, status 156 20 avr. 2015 15:48:31 - Critical bpbrm (pid=3058) from client mailboxeserver: FTL - snapshot creation failed - SNAPSHOT_NOTIFICATION::PostSnapshot failed., status 130 20 avr. 2015 15:48:31 - Warning bpbrm (pid=3058) from client mailboxeserver: WRN - NEW_STREAM0 is not frozen 20 avr. 2015 15:48:31 - Warning bpbrm (pid=3058) from client mailboxeserver: WRN - Microsoft Information Store:\mtnmailbox1\ is not frozen 20 avr. 2015 15:48:31 - Warning bpbrm (pid=3058) from client mailboxeserver: WRN - NEW_STREAM1 is not frozen 20 avr. 2015 15:48:31 - Warning bpbrm (pid=3058) from client mailboxeserver: WRN - Microsoft Information Store:\mtnmailbox2\ is not frozen 20 avr. 2015 15:48:31 - Warning bpbrm (pid=3058) from client mailboxeserver: WRN - NEW_STREAM2 is not frozen 20 avr. 2015 15:48:31 - Warning bpbrm (pid=3058) from client mailboxeserver: WRN - Microsoft Information Store:\mtnmailbox3\ is not frozen 20 avr. 2015 15:48:31 - Warning bpbrm (pid=3058) from client mailboxeserver: WRN - NEW_STREAM3 is not frozen 20 avr. 2015 15:48:31 - Warning bpbrm (pid=3058) from client mailboxeserver: WRN - Microsoft Information Store:\mtnmailbox3bis\ is not frozen 20 avr. 2015 15:48:31 - Warning bpbrm (pid=3058) from client mailboxeserver: WRN - NEW_STREAM4 is not frozen 20 avr. 2015 15:48:31 - Warning bpbrm (pid=3058) from client mailboxeserver: WRN - Microsoft Information Store:\mtnmailbox5\ is not frozen 20 avr. 2015 15:48:31 - Info bpfis (pid=9244) done. status: 130 20 avr. 2015 15:48:31 - end Snapshot: Create Snapshot; elapsed time 0:00:33 20 avr. 2015 15:48:31 - Info bpfis (pid=9244) done. status: 130: system error occurred 20 avr. 2015 15:48:31 - end writing Operation Status: 130 20 avr. 2015 15:48:31 - end Parent Job; elapsed time 0:00:33 20 avr. 2015 15:48:31 - begin Snapshot: Stop On Error Operation Status: 0 20 avr. 2015 15:48:31 - end Snapshot: Stop On Error; elapsed time 0:00:00 20 avr. 2015 15:48:31 - begin Snapshot: Delete Snapshot 20 avr. 2015 15:48:31 - started process bpbrm (pid=3261) 20 avr. 2015 15:48:32 - Info bpbrm (pid=3261) Starting delete snapshot processing 20 avr. 2015 15:48:34 - Info bpfis (pid=9560) Backup started 20 avr. 2015 15:48:34 - Critical bpbrm (pid=3261) from client mailboxeserver: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.mailboxeserver_1429544878.1.0 20 avr. 2015 15:48:35 - Info bpfis (pid=9560) done. status: 4207 20 avr. 2015 15:48:35 - end Snapshot: Delete Snapshot; elapsed time 0:00:04 20 avr. 2015 15:48:35 - Info bpfis (pid=9560) done. status: 4207: Could not fetch snapshot metadata or state files 20 avr. 2015 15:48:35 - end writing Operation Status: 4207 20 avr. 2015 15:48:35 - begin Snapshot: End Notify Script 20 avr. 2015 15:48:35 - Info RUNCMD (pid=3266) started 20 avr. 2015 15:48:35 - Info RUNCMD (pid=3266) exiting with status: 0 Operation Status: 0 20 avr. 2015 15:48:35 - end Snapshot: End Notify Script; elapsed time 0:00:00 Operation Status: 4207 Could not fetch snapshot metadata or state files (4207) please i need help !!!Solved17KViews0likes1Comment
activity monitor failed because of connection to bpjobd daemon on the server failed
activity monitor failed because of connection to bpjobd daemon on the master server failed . Master server is running on Oracle Solaris 10 8/11 s10x_u10wos_17b X86 , Netbackup version is 7.6.0.4Solved17KViews1like9Comments
VMware VM "needs consolidation" after backup failure
Has anyone else run into this issue? VM backup fails with a "Status 13" or "Status 156" or "Status 40" - I have several hundred VMs and only a few end up in this state during a backup window. No snapshots show up in Snapshot Manager on the VMs that have this error. When consolidation is attempted VMware throws a "Unable to access file <unspecified filename> since it is locked" error. The "fix" for us has been to run "services.sh restart" on the ESXi server - this unlocks the file - then manually consolidating the VM. Sounds a lot like this issue: https://www-secure.symantec.com/connect/forums/problem-backup-virtual-machine-when-vm-convert-status-consolidate Master: 7.6.0.2 on Solaris 10 Media: 2.6.0.2 on 5220 Appliances17KViews3likes11CommentsVMWare snapshots being locked by NetBackup since 7.6.0.2 upgrade.
Since upgradingfrom 7.5.0.6 to 7.6.0.2 and our appliances from 2.5.3 to 2.6.2we have been experiencing sporadic status 156 errors on someVM based backups. The issue seems to be that NetBackup locks thesnapshot down and does not come back and deletes it.Since the snapshot isn't being delete the netbackup doesn't release the lock and thereforethe disk cannot be consolidated. This causes the next day's backups to fail with status 156. Our VMWare backup policies goto a single 5230 appliance and run early evening and finish early morning. However,hours after the jobs finish andzero jobs are running to the appliance there are still bpbkar processes running that correlate to the previous night's backup jobs . If I don't kill these process then the disk wont consolidate and causes some of the next night'sVMware backups to fail. The only way to VM admin's to consolidate the disks is for me to kill the pids associated with the hung bpbkar process. Below are the job details of one of the failed jobs: 7/21/2014 10:41:56 AM - Info nbjm(pid=30146) starting backup job (jobid=1798854) for client mp1cucm-sub03, policy VMQUERY_CISCO, schedule DIFF-2WK 7/21/2014 10:41:56 AM - Info nbjm(pid=30146) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1798854, request id:{8B81D818-10ED-11E4-B49E-B151A9591D42}) 7/21/2014 10:41:56 AM - requesting resource stu_disk_mpnbuapp02 7/21/2014 10:41:56 AM - requesting resource mpnbu01.NBU_CLIENT.MAXJOBS.mp1cucm-sub03 7/21/2014 10:41:56 AM - requesting resource mpnbu01.VMware.Datastore.chivmvc03/Cisco/Application Datastore 7/21/2014 10:41:56 AM - granted resource mpnbu01.NBU_CLIENT.MAXJOBS.mp1cucm-sub03 7/21/2014 10:41:56 AM - granted resource mpnbu01.VMware.Datastore.chivmvc03/Cisco/Application 7/21/2014 10:41:56 AM - granted resource MediaID=@aaaah;DiskVolume=PureDiskVolume;DiskPool=dp_disk_mpnbuapp02;Path=PureDiskVolume;StorageServer=mpnbuapp02;MediaServer=mpnbuapp02 7/21/2014 10:41:56 AM - granted resource stu_disk_mpnbuapp02 7/21/2014 10:41:56 AM - estimated 13991947 Kbytes needed 7/21/2014 10:41:56 AM - begin Parent Job 7/21/2014 10:41:56 AM - begin Application Snapshot, Step By Condition Status 0 7/21/2014 10:41:56 AM - end Application Snapshot, Step By Condition; elapsed time: 0:00:00 7/21/2014 10:41:56 AM - begin Application Snapshot, Read File List Status 0 7/21/2014 10:41:56 AM - end Application Snapshot, Read File List; elapsed time: 0:00:00 7/21/2014 10:41:56 AM - begin Application Snapshot, Create Snapshot 7/21/2014 10:41:56 AM - started process bpbrm (21073) 7/21/2014 10:41:56 AM - started 7/21/2014 10:41:57 AM - Info bpbrm(pid=21073) mp1cucm-sub03 is the host to backup data from 7/21/2014 10:41:57 AM - Info bpbrm(pid=21073) reading file list for client 7/21/2014 10:41:57 AM - Info bpbrm(pid=21073) start bpfis on client 7/21/2014 10:41:57 AM - Info bpbrm(pid=21073) Starting create snapshot processing 7/21/2014 10:41:57 AM - Info bpfis(pid=21082) Backup started 7/21/2014 10:41:57 AM - snapshot backup of client mp1cucm-sub03 using method VMware_v2 7/21/2014 10:42:14 AM - Info bpbrm(pid=21073) INF - vmwareLogger: WaitForTaskCompleteEx: Unable to access file <unspecified filename> since it is locked <232> 7/21/2014 10:42:14 AM - Info bpbrm(pid=21073) INF - vmwareLogger: WaitForTaskCompleteEx: SYM_VMC_ERROR: TASK_REACHED_ERROR_STATE 7/21/2014 10:42:14 AM - Info bpbrm(pid=21073) INF - vmwareLogger: ConsolidateVMDisks: SYM_VMC_ERROR: TASK_REACHED_ERROR_STATE 7/21/2014 10:42:14 AM - Info bpbrm(pid=21073) INF - vmwareLogger: ConsolidateVMDisksAPI: SYM_VMC_ERROR: TASK_REACHED_ERROR_STATE 7/21/2014 10:42:15 AM - Critical bpbrm(pid=21073) from client mp1cucm-sub03: FTL - VMware_freeze: VIXAPI freeze (VMware snapshot) failed with 36: SYM_VMC_TASK_REACHED_ERROR_STATE 7/21/2014 10:42:15 AM - Critical bpbrm(pid=21073) from client mp1cucm-sub03: FTL - VMware error received: Unable to access file <unspecified filename> since it is locked 7/21/2014 10:42:15 AM - Critical bpbrm(pid=21073) from client mp1cucm-sub03: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze 7/21/2014 10:42:15 AM - Critical bpbrm(pid=21073) from client mp1cucm-sub03: FTL - 7/21/2014 10:42:15 AM - Critical bpbrm(pid=21073) from client mp1cucm-sub03: FTL - vfm_freeze: method: VMware_v2, type: FIM, function: VMware_v2_freeze 7/21/2014 10:42:15 AM - Critical bpbrm(pid=21073) from client mp1cucm-sub03: FTL - 7/21/2014 10:42:15 AM - Critical bpbrm(pid=21073) from client mp1cucm-sub03: FTL - snapshot processing failed, status 156 7/21/2014 10:42:15 AM - Critical bpbrm(pid=21073) from client mp1cucm-sub03: FTL - snapshot creation failed, status 156 7/21/2014 10:42:15 AM - Warning bpbrm(pid=21073) from client mp1cucm-sub03: WRN - ALL_LOCAL_DRIVES is not frozen 7/21/2014 10:42:15 AM - Info bpfis(pid=21082) done. status: 156 7/21/2014 10:42:15 AM - end Application Snapshot, Create Snapshot; elapsed time: 0:00:19 7/21/2014 10:42:15 AM - Info bpfis(pid=21082) done. status: 156: snapshot error encountered 7/21/2014 10:42:15 AM - end writing Status 156 7/21/2014 10:42:15 AM - end Parent Job; elapsed time: 0:00:19 7/21/2014 10:42:15 AM - begin Application Snapshot, Stop On Error Status 0 7/21/2014 10:42:15 AM - end Application Snapshot, Stop On Error; elapsed time: 0:00:00 7/21/2014 10:42:15 AM - begin Application Snapshot, Resources For Cleanup Status -9999 7/21/2014 10:42:15 AM - end Application Snapshot, Resources For Cleanup; elapsed time: 0:00:00 7/21/2014 10:42:15 AM - begin Application Snapshot, Delete Snapshot 7/21/2014 10:42:15 AM - started process bpbrm (21152) 7/21/2014 10:42:16 AM - Info bpbrm(pid=21152) Starting delete snapshot processing 7/21/2014 10:42:16 AM - Info bpfis(pid=21161) Backup started 7/21/2014 10:42:16 AM - Critical bpbrm(pid=21152) from client mp1cucm-sub03: FTL - cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.mp1cucm-sub03_1405957316.1.0 7/21/2014 10:42:16 AM - Info bpfis(pid=21161) done. status: 4207 7/21/2014 10:42:16 AM - end Application Snapshot, Delete Snapshot; elapsed time: 0:00:01 7/21/2014 10:42:16 AM - Info bpfis(pid=21161) done. status: 4207: Could not fetch snapshot metadata or state files 7/21/2014 10:42:16 AM - end writing Status 4207 7/21/2014 10:42:16 AM - end operation Status 156 7/21/2014 10:42:16 AM - end operation snapshot error encountered(156) Has anyone come across thisbefore? Anyhelp would be greatly appreciated. Thanks, ChrisSolved17KViews1like11CommentsVMware - Snapshot error encountered (156)
I have a VMware policy configured using annotation query to select clients. Yesterday one of the clients failed with a status code of 156. Last night the same client completed successfully. I noticed in the detailed status, I have error's relating to resource. 23/09/2014 18:00:52 - Info nbrb(pid=2888) Limit has been reached for the logical resource pd-dc2-nbumst.VMware.snapshot.vCenter.pd-dc2-vcenter.pdports.co.uk What do I need to check/change to stop this from re-occuring? Thanks,Solved16KViews2likes5Comments
