11-21-2012 03:34 AM
I am having issues when using multiplexing to MSL drives with my VMWare backups, at present I am backing up approx 80 VM clients using the NBU VSphere API to my VLS, this means that all allocated drives are used and each individual clinet backup is using one single virtual cartridge, I wanted to move the VM backups onto out LTO5 MSL and use multiplexing to sream several clients onto a drive therefore using less tapes.
If I run the backups using media multiplexing set to 4 in the policy (we have 4 LTO5 drives in the MSL) and leave the maximum concurrent write drives to 4 on the Storage Unit the backups work fine with a single VM Client backing up through a single drive; hiowever, if I enable multiplexing and set the maximum streams per drive on the storage unit then the backups fail with an error 71, it does not matter how many stream I set on the storage unit this still fails the backups.
Just wondering if this is due to the policy being a VM VSphere API policy?
Thanks in advance
Kev
Solved! Go to Solution.
11-27-2012 07:33 AM
Looks like I may have got it working now, I checked the configuration in the VMWare tab of the policy and changed the VMWare backup host from Backup Media Server to the FQDN of the backup host and this has allowed me to multiplex to the MSL attached on the Master Server.........
11-21-2012 08:55 AM
What version of NetBackup are you using?
There was a bug in 6.5.1 for a similar issue but not with a Status 71
If you are right up to date I would log a call with Symantec as it may be a new issue or they may have an EEB
What buffer numbers and sizes do you use? Just wondering if that may be causing things to go wrong
Could we also see a log - details staus of job and bpbrm / bptm logs from the media server
Thanks
11-22-2012 02:12 AM
Hi Mark,
I am using 7.5.0.4 on all my Master & Media Servers, not sure if this worked in previous 7.x versions as this is the first time I have tried this for the VM clients.
NUMBER_DATA_BUFFERS = 16
SIZE_DATA_BUFFERS = 262144
Will reset and capture the data from the bpbrm / bptm logs and post these in a mo
Kev
11-22-2012 02:35 AM
Detailed Status:
11-22-2012 05:21 AM
Interesting .. looks like an app crash (anything in the system and application logs?)
But i also see this:
10:22:01.845 [8712] <16> catch_signal: media manager terminated by media mount timeout
and this:
10:22:01.846 [8712] <2> send_MDS_msg: Error from sendIrmMsg, Master bfbackup.ipcmedia.com, type 11, returned error 805
10:22:01.846 [8712] <2> send_MDS_msg: NBJM returned an extended error status: invalid jobid (805)
Wondering if your backup host cannot cope with doing a multiplexed backup of this type (or at least the 4 simultaneous streams) or if it does crash.
I think the timeout is more of a disconect than a timeout
Check the system and application logs on the vmware backup host.
I see it is a Win 2003 32 bit server, how much RAM does it have?
11-22-2012 05:26 AM
The file that the Detailed Status looks like it is failing on does exist in the /usr/openv/netbackup/online_util/fi_cntl
11-22-2012 05:38 AM
The file will have gone after the job failed and most likely the slashes are just a NBU way of writing things
Let me know what you find relating to my earlier post
11-22-2012 06:02 AM
The vmbackup host (lonbfbnbumedia1) is a Win2008 R2 with an Intel Xeon 3.07GHz processor and 24Gb RAM, not sure where you got the Win2003 from?? the MSL library I am trying to Multiplex to is attached to the RHEL Master Server which is a DL580 G7 with 120Gb RAM
Kev
11-22-2012 06:35 AM
OK - so vmbackup host is a client passing data to media server (master)
Anything in the system or application event logs for either?
One other thing from the VMWare admin guide is a note about a limit on the number of connections per vCenter being 27 (NFC)
This is on a per disk basis so if the four clients being backed up between them have more than 27 disks it could cause a hang and i guess as it is supposed to be multiplexed backup this could cause it all to fail
How many disk would you have on the 4 you are backing up?
May also need to see the bpfis log from the vmbackup host - it would usually be bpfis that crashes although you log suggests it is something else
11-22-2012 06:55 AM
Hi Mark,
Just checked the logs from the servers and I cannot see any errors at all, the 4 servers only have a C drive on them and we configure the VMBackup to discount datadisks, I have attached the bpfis from the vmbackup host.
Kev
11-22-2012 07:26 AM
That looks OK - the answer i believe is in the disconnect between the Master and VMBackup Host when you use multiplexing but if there are no app crashes then i cannot quite put my finger on it.
Check all events at the time of the failure on the backup host and the Master (10:21am) in case anything is cropping up such as desktop heap, anti virus, access protection etc. etc. to try and pin this one down
On the point of anti virus is all of NetBackup excluded?
11-22-2012 07:40 AM
Thanks for taking a look Mark, what I might try doing is to zone in the MSL library directly to the VMBackup Host so it is not hopping through the Linux master server and see if that results in any change, will have to wait for our SAN guy to do this change for me tomorrow.
I have not excluded anything on the VM backups as I am not too sure how you do this as the option is not available on the policy and none of the VM clients have NBU installed as we use the VSphere API.
Going to trawl through the logs and see if anything springs to mind
Kev
11-22-2012 07:49 AM
Sorry - i meant anti viris exclusions on the vmware backup host incase AV was attacking the snapshots etc or blocking processes - something along the route kills a process somewhere
11-22-2012 07:53 AM
Ah I see now, we do use Symnatec Endpoint Protection so I will need to disable this and re-test it again.
11-22-2012 08:11 AM
Hi Mark,
Just disabled the SEP and re-ran the same backups using multiplexing and they fail with the same error, I will continue to try and trace anything in the logs and see if I can get a resolution.
11-27-2012 06:47 AM
I have checked through all the log files from the VM Backup Host and there is nothing out of the ordinary, I have tried the multiplexing on drives that are attached to the VM backup Host and still have the same issue, does anyone have Multiplexing working on VM backups using 7.5.0.4 using the VSphere API method of Full VM Mapped backups?
11-27-2012 07:07 AM
Just a thought .. do you have any VMNWare Resource limits set (Master Server Host Properties) - just wondering if they are conflicting .. i.e. job want to run 10 jobs, mpx wants to run 10 jobs, limit says you can only run 6 - so conflict makes it all fall over.
I do have one customer that does use mpx to tape with VMWare policies - as well as inline copy - and they work fine - they have made some recent changes so cannot guarantee they are VMWare policies and not Flashbackup-Windows policies though
11-27-2012 07:33 AM
Looks like I may have got it working now, I checked the configuration in the VMWare tab of the policy and changed the VMWare backup host from Backup Media Server to the FQDN of the backup host and this has allowed me to multiplex to the MSL attached on the Master Server.........