12-31-2012 08:26 AM
Hello,
We have been using Netbackup with SLP duplications for over a year with no issues. Suddenly within the last two weeks I am having issues with duplications of SLP duplication from Advanced disk to tape of Exchange GRT backups. This has happened with a regular exchange database and also now a DAG setup. It does not happen for all images though, for the DAG the backup makes mulitple images and others duplicate fine but one seems to get stuck. In the activity monitor all I see is the SLP job with the two duplication jobs (we are copying to two tapes) and they show the operation as mounting. I have left them there for over a day and the state never changes.
This is Netbackup 7.1 running on Windows 2008 R2. We only have one master/media server.
Backing up from Exchange 2010 running on Windows 2008 R2
The job log shows:
30/12/2012 11:45:09 PM - requesting resource LCM_nayul-netbackup-hcart2-robot-tld-0
30/12/2012 11:45:09 PM - granted resource LCM_nayul-netbackup-hcart2-robot-tld-0
30/12/2012 11:45:10 PM - begin Duplicate
30/12/2012 11:45:10 PM - started process RUNCMD (1236)
30/12/2012 11:45:10 PM - ended process 0 (1236)
30/12/2012 11:45:10 PM - requesting resource nayul-netbackup-hcart2-robot-tld-0
30/12/2012 11:45:10 PM - requesting resource nayul-netbackup-hcart2-robot-tld-0
30/12/2012 11:45:10 PM - requesting resource @aaaad
30/12/2012 11:45:10 PM - reserving resource @aaaad
30/12/2012 11:45:10 PM - reserved resource @aaaad
30/12/2012 11:45:10 PM - granted resource 7997L5
30/12/2012 11:45:10 PM - granted resource HP.ULTRIUM5-SCSI.005
30/12/2012 11:45:10 PM - granted resource nayul-netbackup-hcart2-robot-tld-0
30/12/2012 11:45:10 PM - granted resource 7770L5
30/12/2012 11:45:10 PM - granted resource HP.ULTRIUM5-SCSI.003
30/12/2012 11:45:10 PM - granted resource nayul-netbackup-hcart2-robot-tld-0
30/12/2012 11:45:10 PM - granted resource MediaID=@aaaad;DiskVolume=D:\;DiskPool=AdvanceDisk;Path=D:\;StorageServer=nayul-netbackup.na.manwin.local;MediaServer=nayul-netbackup.na.manwin.local
The duplication jobs just show:
30/12/2012 11:45:10 PM - begin Duplicate
This job only started last night since I paused SLP, killed the service so the job stopped and froze the tape it was using because I thought maybe it was an issue with the tape. I had left it run or over a day before that and there was no change. Always in state "mounting"
I am not sure where to look to see what the issue is since I don't know to much about the netbackup log structure. What I do find odd is that the file list is empty (but I read this is normal sometimes until the duplication reaches a certain stage). I do need to find a solution though as the backup is filling up our disk and I can't run others. Also when the image cleanup tries to run it hangs when it reaches this client. I am guessing because it wants the duplication to be done first??
I really need a solution as I need to get these jobs done.
Thank you,
Brent
12-31-2012 09:11 AM
Always in state "mounting"
i dont think it has a relation with the exchage images..
its seems its a issue with the tape drives or with the image..but as your saying its happening only with exchage images.. we would need some testing to make sure this.
1)make a note of tape drives and tapes being allocated for the exhcange image duplications. and check if those tape drives aer being allocated to any othe jbos and getting successfull or not.
you can directly check those functionlaty by sending the test backup jobs also..
2)try to chage the source location of the exchage backup image and try duplications form new location to tape drive.
PS:-do you aware that the Exchage GRT restore are not supported from tapes.?
12-31-2012 09:44 AM
Hello,
The tape drives that the image is using was used just before for another backup sucessfully, as was the tape for another backup image in the same set from the DAG. Other images from the DAG set duplicate fine. Just this one is not. I can't really tell why it is stuck since it just says mounting.
I don't have another location to test from as we have a small Netbackup install. Only have one advanced disk.
I know GRT doesn't work from tape, if we need to restore we duplicate back from tape and then do the restore. We don't have a large enough advanced disk to keep the backups there.
Brent
12-31-2012 10:05 AM
Check bptm log on master/media server.
If the log folder does not exist, create it in ...veritas\\netbackup\logs.
Also add VERBOSE entry to vm.conf (in ...veritas\volmgr) and restart NBU Device Management service.
Robot mount actions will be logged in Event Viewer Application log and bptm log.
12-31-2012 10:18 AM
Other images from the DAG set duplicate fine. Just this one is not.
Does this image is having any issues?
did you try verifying the image?
GUI--->catalog -->search the image--->right click on it and select Verify.. --
see the activity moniter for results.
12-31-2012 12:50 PM
Hello,
I verified the other image and it is fine. I created the bptm filder and added the verbose and restarted the service. The duplication restarted and I don't see anything in the log file about it or anything in the event manager? I only see this which seems unrelated:
15:25:52.553 [4752.5112] <2> bptm: INITIATING (VERBOSE = 0): -rptdrv -jobid -1356983144 -jm
15:25:52.553 [4752.5112] <2> main: Sending [EXIT STATUS 0] to NBJM
15:25:52.553 [4752.5112] <2> bptm: EXITING with status 0 <----------
15:26:37.266 [1180.5336] <2> bptm: INITIATING (VERBOSE = 0): -delete_all_expired
15:26:37.266 [1180.5336] <2> vnet_same_host: ../../libvlibs/vnet_addrinfo.c.2915: 0: name2 is empty: 0 0x00000000
15:26:37.266 [1180.5336] <4> bptm: emmserver_name = nayul-netbackup.na.manwin.local
15:26:37.266 [1180.5336] <4> bptm: emmserver_port = 1556
15:26:37.297 [1180.5336] <2> Orb::init: initializing ORB EMMlib_Orb with: dbstunitq -ORBSvcConfDirective "-ORBDottedDecimalAddresses 0" -ORBSvcConfDirective "static PBXIOP_Factory '-enable_keepalive'" -ORBSvcConfDirective "static EndpointSelectorFactory ''" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory PBXIOP_Factory'" -ORBSvcConfDirective "static Resource_Factory '-ORBProtocolFactory IIOP_Factory'" -ORBDefaultInitRef '' -ORBSvcConfDirective "static PBXIOP_Evaluator_Factory '-orb EMMlib_Orb'" -ORBSvcConfDirective "static Resource_Factory '-ORBConnectionCacheMax 1024 '" -ORBSvcConf nul -ORBSvcConfDirective "static Server_Strategy_Factory '-ORBMaxRecvGIOPPayloadSize 268435456'"(../Orb.cpp:824)
15:26:37.297 [1180.5336] <2> Orb::init: caching EndpointSelectorFactory(../Orb.cpp:839)
15:26:37.360 [1180.5336] <2> bptm: EXITING with status 0 <----------
Is it possible it isn't even asking for the tape? When I ran the verify it did show me the tape mount. Activity monitor does show the drive as mounting though.
Brent
12-31-2012 01:14 PM
hi could you provide the detail status of the duplicate job?
12-31-2012 01:27 PM
Hello,
Is is the same as before
31/12/2012 3:22:53 PM - begin Duplicate
31/12/2012 3:22:53 PM - requesting resource LCM_nayul-netbackup-hcart2-robot-tld-0
31/12/2012 3:22:53 PM - granted resource LCM_nayul-netbackup-hcart2-robot-tld-0
31/12/2012 3:22:53 PM - started process RUNCMD (5276)
31/12/2012 3:22:53 PM - ended process 0 (5276)
31/12/2012 3:22:54 PM - requesting resource nayul-netbackup-hcart2-robot-tld-0
31/12/2012 3:22:54 PM - requesting resource nayul-netbackup-hcart2-robot-tld-0
31/12/2012 3:22:54 PM - requesting resource @aaaad
31/12/2012 3:22:54 PM - reserving resource @aaaad
31/12/2012 3:22:54 PM - reserved resource @aaaad
31/12/2012 3:22:54 PM - granted resource 7997L5
31/12/2012 3:22:54 PM - granted resource HP.ULTRIUM5-SCSI.003
31/12/2012 3:22:54 PM - granted resource nayul-netbackup-hcart2-robot-tld-0
31/12/2012 3:22:54 PM - granted resource 7770L5
31/12/2012 3:22:54 PM - granted resource HP.ULTRIUM5-SCSI.001
31/12/2012 3:22:54 PM - granted resource nayul-netbackup-hcart2-robot-tld-0
31/12/2012 3:22:54 PM - granted resource MediaID=@aaaad;DiskVolume=D:\;
Brent
12-31-2012 10:04 PM
Where exactly do you see 'mounting'?
Seems to me that the resource broker got stuck between identifying and activating resources.
nbrb does not pass job to nbjm which will start bpbrm and bptm on the media server. Only in bptm will we see 'mounting'....
You did not mention your NBU 7.1 patch level?
I seem to remember that basically every 7.1 patch contained nbrb fixes.
You should be on 7.1.0.4 as a minimum.
01-01-2013 05:09 AM
05-27-2013 07:26 PM
Hello,
I'm trying staging using "multiple copies" feature on Exchange 2010 DAG (GRT + Dedup by MSDP) schedule.
I got:
"27/05/2013 22:36:56 - Error bpbrm(pid=1788) Granular backups must go to a disk based storage unit before moving to a removable media based storage unit
27/05/2013 22:37:00 - end writing
storage unit characteristics mismatched to request(154)"
The source copy was msdp storage unit; the secondary one was tape.
I'll try the reverse.