12-31-2012 04:24 AM
Hello All,
Facing an issue with automatic vault job.It runs fine to certain point but at 76% it hungs.I have tried to cancel and refire it but its again stopping at 76%
OS-Solaris 10,NBU-6.5
Dec 31, 2012 5:36:30 PM - vault waiting for session ID lock
Dec 31, 2012 5:37:03 PM - vault session ID lock acquired
Dec 31, 2012 5:37:03 PM - vault session ID lock released
Dec 31, 2012 5:36:03 PM - requesting resource asprd222.aldc.att.com.NBVAULT.MAXJOBS
Dec 31, 2012 5:36:03 PM - requesting resource asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Eject_Production
Dec 31, 2012 5:36:26 PM - granted resource asprd222.aldc.att.com.NBVAULT.MAXJOBS
Dec 31, 2012 5:36:26 PM - granted resource asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Eject_Production
Dec 31, 2012 5:36:26 PM - estimated 0 kbytes needed
Dec 31, 2012 5:36:26 PM - begin Parent Job
Dec 31, 2012 5:36:26 PM - begin Vault: Start Notify Script
Dec 31, 2012 5:36:26 PM - started process RUNCMD (pid=22600)
Dec 31, 2012 5:36:26 PM - ended process 0 (pid=22600)
Operation Status: 0
Dec 31, 2012 5:36:26 PM - end Vault: Start Notify Script; elapsed time 0:00:00
Dec 31, 2012 5:36:26 PM - begin Vault: Execute Script
Dec 31, 2012 5:36:27 PM - started process bpbrm (pid=22614)
Dec 31, 2012 5:36:31 PM - requesting resource asprd222.aldc.att.com.VAULT_CREATE_SESSION_ID.LOCK_TLD(0)_Production
Dec 31, 2012 5:37:02 PM - granted resource asprd222.aldc.att.com.VAULT_CREATE_SESSION_ID.LOCK_TLD(0)_Production
Dec 31, 2012 5:40:58 PM - Duplication skipped
Dec 31, 2012 5:40:59 PM - vault waiting for assign slot lock
Dec 31, 2012 5:40:59 PM - requesting resource asprd222.aldc.att.com.VAULT_ASSIGN_SLOT.LOCK_TLD(0)_Production
Dec 31, 2012 5:41:07 PM - vault assign slot lock acquired
Dec 31, 2012 5:41:05 PM - granted resource asprd222.aldc.att.com.VAULT_ASSIGN_SLOT.LOCK_TLD(0)_Production
Dec 31, 2012 5:41:42 PM - vault assign slot lock released
Dec 31, 2012 5:41:42 PM - Catalog Backup skipped
Dec 31, 2012 5:41:42 PM - vault waiting for assign slot lock
Dec 31, 2012 5:41:43 PM - requesting resource asprd222.aldc.att.com.VAULT_ASSIGN_SLOT.LOCK_TLD(0)_Production
Dec 31, 2012 5:42:15 PM - vault assign slot lock acquired
Dec 31, 2012 5:42:17 PM - vault assign slot lock released
Dec 31, 2012 5:42:20 PM - before eject, waiting for media to be unmounted; sleeping for 380 seconds
Dec 31, 2012 5:42:14 PM - granted resource asprd222.aldc.att.com.VAULT_ASSIGN_SLOT.LOCK_TLD(0)_Production
Dec 31, 2012 5:48:40 PM - starting eject operation
Dec 31, 2012 5:48:40 PM - begin Eject and Report
Dec 31, 2012 5:48:40 PM - connecting
Dec 31, 2012 5:48:40 PM - connected; connect time: 0:00:00
Dec 31, 2012 5:48:42 PM - vault waiting for eject lock
Dec 31, 2012 5:48:42 PM - requesting resource asprd222.aldc.att.com.VAULT_EJECT.LOCK_0
Dec 31, 2012 5:49:08 PM - Info nbrb (pid=7943) Limit has been reached for the logical resource asprd222.aldc.att.com.VAULT_EJECT.LOCK_0
Tried to delete the lock files but seems nothing changed.
Please suggest.
Solved! Go to Solution.
12-31-2012 07:04 AM
so you got to know what you need to do...
just disable/delete the policy which is triggering this vault job. and also you need to check with your team, about the chages on this vault profiles to make sure everything is fine..
Good luck..
12-31-2012 04:49 AM
hi,
it looks like you have another Vault job running...
For each vault name only one vault profile can be run at a time. This is by design in Netbackup vault feature and cannot be changed.
if you are not seeing another vault job for the same profile..
could you expline from where/how did you try to delte the lock files?
12-31-2012 04:59 AM
12-31-2012 05:14 AM
@ Nagalla
I could see files in misc folder under volmgr.As of now i have these files
lmfcd.lock robotic_db vmd.lock
No other job for the same profile is running.
12-31-2012 05:58 AM
I just want to make sure you are seeing misc folder in Robot control host.. it may be your master server or may not be.. .. but you are seeing in Robot control host.
could you check if you are seeing old jobs stuck in EMM.
nbrbuitl -dump | grep -i Eject
nbrbutil -dump | grep -i vault
and also provide the output of below command.
vxlogview -p 51216 -o 118 -b "12/31/12 05:00:00 PM" -e "12/31/12 6:32:00 PM" -d p
12-31-2012 06:01 AM
12-31-2012 06:21 AM
@Nagalla
index=0 (Request provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 userSequence=0 (CountedResourceRequest resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 max=1)))
index=0 (Request provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 userSequence=0 (CountedResourceRequest resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 max=1)))
index=47 (Allocation: id={8027C028-1DD2-11B2-A310-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Eject_Production masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425658" named resource allocation)
index=170 (Allocation: id={CF1DCF76-1DD1-11B2-844F-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=0 userid="jobid=2403050" named resource allocation)
index=0 (Request provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 userSequence=0 (CountedResourceRequest resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 max=1)))
index=0 (Request provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 userSequence=0 (CountedResourceRequest resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 max=1)))
index=40 (Allocation: id={95B04D10-1DD1-11B2-9742-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_DUPLICATION.LOCK_TLD(0)_Vault_orca_pisces_toll masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=0 userid="jobid=2378694" named resource allocation)
index=46 (Allocation: id={5035E592-1DD2-11B2-AA16-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Vault_LTO3_orca_pisces_toll masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2378694" named resource allocation)
index=77 (Allocation: id={5035E51A-1DD2-11B2-A3BB-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBVAULT.MAXJOBS masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2378694" named resource allocation)
index=78 (Allocation: id={76938746-1DD2-11B2-9476-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBVAULT.MAXJOBS masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425161" named resource allocation)
index=79 (Allocation: id={76D6E252-1DD2-11B2-B5C2-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBVAULT.MAXJOBS masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425159" named resource allocation)
index=81 (Allocation: id={7950D684-1DD2-11B2-87F5-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBVAULT.MAXJOBS masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2403050" named resource allocation)
index=86 (Allocation: id={7950D710-1DD2-11B2-89C9-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Vault_LTO3_orca_pisces masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2403050" named resource allocation)
index=125 (Allocation: id={00C7E7B8-1DD2-11B2-9E64-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_DUPLICATION.LOCK_TLD(0)_Production masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=0 userid="jobid=2425159" named resource allocation)
index=128 (Allocation: id={769387BE-1DD2-11B2-9904-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Vault_LTO3_orca_2_7_10 masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425161" named resource allocation)
index=129 (Allocation: id={76D6E2D4-1DD2-11B2-A112-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBU_POLICY.MAXJOBS.Vault_Production masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425159" named resource allocation)
index=153 (Allocation: id={8027BFB0-1DD2-11B2-8BA9-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.NBVAULT.MAXJOBS masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=-1 userid="jobid=2425658" named resource allocation)
index=166 (Allocation: id={CF1DCF76-1DD1-11B2-844F-00212817E080} provider=NamedResourceProvider resourcename=asprd222.aldc.att.com.VAULT_EJECT.LOCK_0 masterserver=asprd222.aldc.att.com groupid={00000000-0000-0000-0000-000000000000} userSequence=0 userid="jobid=2403050" named resource allocation)
vxlogview -p 51216 -o 118 -b "12/31/12 05:00:00 PM" -e "12/31/12 6:32:00 PM" -d p
V-1-45 No log files found.
@Mark
We have already bounced the services on this server.But issue seems to be there.
12-31-2012 06:30 AM
12-31-2012 06:38 AM
2425658 | Vault | Active | Limit has been reached for requested resource (asprd222.aldc.att.com.VAULT_EJECT.LOCK_0) | Eject_Production | Production_Eject | asprd222.aldc.att.com | Mon Dec 31 17:36:03 IST 2012 | 8956 | 1 | Eject and Report | 76 | 22617 | root | 2425658 | Mon Dec 31 17:36:26 IST 2012 | 8933 | TLD(0) | Production | Production_Eject | 540 | Standard | asprd222.aldc.att.com | 0 |
2403050 | Vault | Active | Vault_LTO3_orca_pisces | Vault_LTO2_rsprd319 | asprd222.aldc.att.com | Tue Dec 25 14:30:00 IST 2012 | 537318 | 1 | Eject and Report | 76 | 24677 | root | 2403050 | Tue Dec 25 14:30:05 IST 2012 | 537313 | TLD(0) | Vault_LTO3_orca_pisces | Vault_LTO3_orca_pisces | 55 | 2 | Standard | asprd222.aldc.att.com | 0 |
current one which is stuck at 76%
STATE DETAILS POLICY SCHEDULE
2425658 | Vault | Active | Limit has been reached for requested resource (asprd222.aldc.att.com.VAULT_EJECT.LOCK_0) | Eject_Production | Production_Eject | asprd222.aldc.att.com | Mon Dec 31 17:36:03 IST 2012 | 8956 | 1 | Eject and Report | 76 | 22617 | root | 2425658 | Mon Dec 31 17:36:26 IST 2012 | 8933 | TLD(0) | Production | Production_Eject | 540 | Standard | asprd222.aldc.att.com | 0 |
12-31-2012 06:48 AM
both these jobs are for the same robot?
12-31-2012 06:59 AM
I think some profile config issue has been encontered..coz both these jobs were for same robot but now only
profile for Vault_LTO2_rsprod319 exists and i cant find Eject_Production one..:(
12-31-2012 07:04 AM
so you got to know what you need to do...
just disable/delete the policy which is triggering this vault job. and also you need to check with your team, about the chages on this vault profiles to make sure everything is fine..
Good luck..