Hello everyone - Having an issue with a brand new Exchange 2010 policy that I'm hoping someone out there can shed some light on. Here is the situation: The policy runs just fine backing up the entire DAG with multiplexing set at 1, but loads tapes into both drives on the master/media server and writes to both. This is kind of silly, since all of the data will easily fit into one LTO3 tape. So, we changed the multiplexing value to 2 in hopes to get everything onto one tape, and many of the child jobs fail with error 25s. With this being said, we do multiplexing on many other policies without issue, so I'm inclined to think this isn't a network/connectivity issue as what the description for error 25 is. Well, we know it's not connectivity since the policy works perfectly with multiplexing set to 1. Vitals on our setup and the policy: We're running Netbackup v7.1.01 on a common master/media server with Windows 2003 Standard R2 w/ SP2 as the OS. The policy type is (obviously) MS-Exchange-Server, the Exchange attribute is set to 'Passive copy and if not available the active copy', and the single backup selection is 'Microsoft Exchange Database Availability Groups:\'. 'Perform snapshot backups' is also checked. Under the schedule options, 'Media Multiplexing' is set to 1. Type of backup is Full. The DAG has two servers (mailbox role only) in it -- both Windows 2008 R2 Enterprise w/ SP1. Exchange is version 2010 SP2 w/ RU2. I did open a ticket with Symantec on this issue and unfortunately, was given little assistence. The person we worked with just said it must be a network problem, but also had to look at the documentation to see what multiplexing even WAS. Sigh. This seems like it should be a lay-up to do. Does anyone have any ideas or suggestions as to what may be going on here? Thanks a ton in advance. -Craig

Everyone - I'm happy to share that this issue has been resolved. The fix, ultimately, was simple: Port 1556 needed to be opened on the Windows firewall on both of our DAG nodes. I didn't see in the NBU Exchange Admin guide, but maybe I missed it. :) A huge thanks goes out to Steve @ Symantec for working through this issue with me! Hopefully this helps someone else out there! -Craig

you want to go to only one tape. you can have many many storage units set up - and the total number of drives in all of your storage units can be more than the number of tape drives you have. Just create a storage unit with 1 tape drive and set the exchange policy to use it. -------------------------- as to the other issue - you said you have 2 exchange servers - are there passive copies on each one? what I have observed parent job dag - goes and sees what it is suppose to back up child job for serverA - sees that there are mailstores on serverA it needs to backup so it makes a parent job just for serverA grandchild job - for mailstore A grandchild job - for mailstore B child job for serverB = sees that there are mailstores on serverB it needs to backup so it makes a parent job just for serverB grandchild job - for mailstore C grandchild job - for mailstore D so this could be part of what is causing 2 tapes or contention when multistreaming. try putting each mailstore in the backup selection it could also be if you try to run two many streams at once the exchange server just cannot handle it. has anybody monitored the exchange servers when backups are running to see if you are killing it. are both exchange servers listed in someother policy? Meaning if you go to the client list do you see both exchange physical server names listed? and do you have both servers in the master server properties under distributed application restore mapping?

Hi there - First of all, thank you for taking the time to reply. Creating the new storage unit with only one drive is an interesting idea. I will give that a shot and report back. Yes, there are passive copies of on each of the DAGs. The machines are, more or less, doing absolutely nothing when the backups run. They aren't even in production yet and thus only have a very small handful of mailboxes at this time. Performance utilization is very, very low during the backups. We know they aren't being crushed. Per the documentation, we have only the DAG name set up as the Client. The person at Symantec we worked with said this was correct, but as I stated above, he also didn't know what multiplexing was before he looked it up. The cient list ONLY shows the DAG name. Negative on both servers being listed under the 'Distributed Application Restore Mapping' option. I will have to look into this one. -Craig

you need to have that info in the DARM - I am not to sure how will it works without it. second - I found when it goes to enumerate that dag to get check the mailstores - it checks both the passive and active copies to see what the state is. I had issues because the physical server name was not in my client list - so it was like the master did not know it was allowed to talk to them. so as a test create a policy that only backups the C drive and the system state of the physical exchange servers. This will get the server names into the client list. But I don't see how this would effect multiplexing.

I took your suggestion and made a Storage Unit with only one drive. While it only did use one drive, it still picked up two different tapes from the scratch pool. The backup IS finishing succesfully which is obviously a good thing, but the fact that it's using 2 tapes to store ~5GB of data is frustrating. Every other policy we have writes to one tape, and when it fills up, it grabs a new one. It's just this one that seems to have a difficult time with it. Ugh. -Craig

did it take two tapes from the very start? in the gui run the images on tape report for each tape do they have different images on the tape or is it the same backup on both tapes. I am wondering if by chance you have Multiply copies checked on the policy? can you please post the output of the job details.

Backing up Exchange 2010 DAG - Multiplexing Issue

22 Replies

J_H_Is_gone
Level 6
13 years ago
you want to go to only one tape.

you can have many many storage units set up - and the total number of drives in all of your storage units can be more than the number of tape drives you have.

Just create a storage unit with 1 tape drive and set the exchange policy to use it.

--------------------------

as to the other issue - you said you have 2 exchange servers - are there passive copies on each one?

what I have observed

parent job dag - goes and sees what it is suppose to back up

    child job for serverA - sees that there are mailstores on serverA it needs to backup so it makes a parent job just for serverA

              grandchild job - for mailstore A

              grandchild job - for mailstore B

    child job for serverB = sees that there are mailstores on serverB it needs to backup so it makes a parent job just for serverB

               grandchild job - for mailstore C

               grandchild job - for mailstore D

so this could be part of what is causing 2 tapes or contention when multistreaming.

try putting each mailstore in the backup selection

it could also be if you try to run two many streams at once the exchange server just cannot handle it.

has anybody monitored the exchange servers when backups are running to see if you are killing it.

are both exchange servers listed in someother policy? Meaning if you go to the client list do you see both exchange physical server names listed?

and do you have both servers in the master server properties under distributed application restore mapping?
CraigL
Level 4
13 years ago
Hi there -

First of all, thank you for taking the time to reply.

Creating the new storage unit with only one drive is an interesting idea. I will give that a shot and report back.

Yes, there are passive copies of on each of the DAGs. The machines are, more or less, doing absolutely nothing when the backups run. They aren't even in production yet and thus only have a very small handful of mailboxes at this time.

Performance utilization is very, very low during the backups. We know they aren't being crushed.

Per the documentation, we have only the DAG name set up as the Client. The person at Symantec we worked with said this was correct, but as I stated above, he also didn't know what multiplexing was before he looked it up.

The cient list ONLY shows the DAG name.

Negative on both servers being listed under the 'Distributed Application Restore Mapping' option. I will have to look into this one.

-Craig
J_H_Is_gone
Level 6
13 years ago
you need to have that info in the DARM - I am not to sure how will it works without it.

second - I found when it goes to enumerate that dag to get check the mailstores - it checks both the passive and active copies to see what the state is.

I had issues because the physical server name was not in my client list - so it was like the master did not know it was allowed to talk to them.

so as a test create a policy that only backups the C drive and the system state of the physical exchange servers. This will get the server names into the client list. But I don't see how this would effect multiplexing.
CraigL
Level 4
13 years ago
I took your suggestion and made a Storage Unit with only one drive. While it only did use one drive, it still picked up two different tapes from the scratch pool.

The backup IS finishing succesfully which is obviously a good thing, but the fact that it's using 2 tapes to store ~5GB of data is frustrating.

Every other policy we have writes to one tape, and when it fills up, it grabs a new one. It's just this one that seems to have a difficult time with it. Ugh.

-Craig
J_H_Is_gone
Level 6
13 years ago
did it take two tapes from the very start?

in the gui run the images on tape report for each tape

do they have different images on the tape or is it the same backup on both tapes.

I am wondering if by chance you have Multiply copies checked on the policy?

can you please post the output of the job details.
CraigL
Level 4
13 years ago
OK.. I think we may be dealing with multiple issues here. I spent some time today testing out lots of scanarios and have found the following:

Multiple copies is not set for the policy. That is confirmed :)

If I run the job with no multiplexing at all, it puts all the data from one server onto one tape, and data from the other server on another. I have a feeling this is by design.

If I move all active database copies to one server, everything backs up fine to a single tape without issue.

If I move any database over to the other DAG member and enable multiplexing, the second child job always fails with the error 25. The first one finishes without issue.

Backing up individual mail databases on each machine on their own works fine, so I know there are no connectivity issues.

Here's the detail status from the job that bombs on 25.

5/9/2012 3:46:51 PM - Info nbjm(pid=6380) starting backup job (jobid=21333) for client server-01, policy Test_ExchangeDAG, schedule TestFull
5/9/2012 3:46:51 PM - estimated 345676 Kbytes needed
5/9/2012 3:46:51 PM - Info nbjm(pid=6380) started backup job for client server-01, policy Test_ExchangeDAG, schedule TestFull on storage unit Local_Tape_Drives
5/9/2012 3:48:53 PM - mounting CEN508
5/9/2012 3:49:39 PM - Info bpbrm(pid=6792) telling media manager to start backup on client
5/9/2012 3:49:39 PM - Info bptm(pid=6724) using 131072 data buffer size
5/9/2012 3:49:39 PM - Info bptm(pid=6724) using 32 data buffers
5/9/2012 3:49:39 PM - mounted; mount time: 00:00:46
5/9/2012 3:49:39 PM - positioning CEN508 to file 10
5/9/2012 3:51:45 PM - positioned CEN508; position time: 00:02:06
5/9/2012 3:51:46 PM - Info bpbrm(pid=6792) child done, status 0
5/9/2012 3:52:07 PM - Info bpbrm(pid=4788) sending bpsched msg: CONNECTING TO CLIENT FOR DAG-01_1336592811
5/9/2012 3:52:07 PM - connecting
5/9/2012 3:52:49 PM - Error bpbrm(pid=4788) cannot create data socket, The operation completed successfully. (0)
5/9/2012 3:52:49 PM - Info bpbrm(pid=6792) child done, status 25
5/9/2012 3:52:49 PM - Info bpbrm(pid=6792) sending media manager msg: STOP BACKUP DAG-01_1336592811
5/9/2012 3:52:55 PM - Info bpbrm(pid=6792) media manager for backup id DAG-01_1336592811 exited with status 150: termination requested by administrator
5/9/2012 3:52:55 PM - end writing
cannot connect on socket(25)
5/9/2012 3:53:18 PM - Info bpbrm(pid=6792) sending media manager msg: TERMINATE

Yes, that's me cancelling it after I see it hit error 25 :)

The selections for this job look like this:

NEW_STREAM

DB1 (active on node 1)

NEW_STREAM

DB2 (active on node 2)

I also came across a scanario where I can actually crash the Netbackup Job Manager service with a certain combo of streams and mail databases. If I can continue to reproduce this at will, I'll open another ticket w/ Symantec.

What do ya think?

-Craig
rawilde
Level 4
13 years ago
Multiplexing with a DAG backup certainly should work. I would try the support case route again.

Also, that Job Manager crash is a known issue and should be addrssed in 7.5.0.3
J_H_Is_gone
Level 6
13 years ago
by chance are your exchange servers also media servers?

Look that the jobs that each ran and look for who the media server was that did the backup.
CraigL
Level 4
13 years ago
Yeah, I'm going to try and open another case and hope to get someone better.

Thanks for the tip on the Job Manager crash!

-Craig
CraigL
Level 4
13 years ago
Negative, we simply have one master/media server, and it absolutely does not run Exchange :)

Forum Discussion

Backing up Exchange 2010 DAG - Multiplexing Issue

22 Replies

Related Content

Re: Multiplexing and Multistreaming

media multiplexing

Backing up Microsoft Exchange 2016

Media Multiplexing

Media Multiplexing

Recent Discussions

command: bperror

MS-SharePoint policy restore error (2804) .

How to restore a backup

How to configure RBAC

10 years old netbackup appliance database service down, ssl certification out date