Slow SAP Backups (Oracle)

[Seth_E]

Hi there,

I'm experiencing a strange issue, but I don't think it has anything to do with Netbackup. However, I'd like to put this out there to see if any of you have a solution.

We have two VM's. Both are SAP and both run Oracle 11g.Both DB's are roughly the same size. Approx: 640GB

Both VM's are in the same SAP backup policy.

VM1 has 2 vCPU's and 16 GB of RAM

VM2 has 4 vCPU's and 16 GB of RAM

VM1's backup is is 3 hours.

VM2's backup is 11 hours.

Both are running Windows 2008 Datacentre and are both backing up to the same storage unit. VM1 runs at about 60 MB / sec, and VM2 is about 15 MB/sec.

Each VM is running on a different ESX host, however, vmotion does not solve the problem.

Both VM's storage is on the same SAN (XIV)

Therefore, is there any oracle parameters (outside of Netbackup) that could affect backups? Could Oracle be causing this issue?

We have this very same issue on a few other VM's. Some backup very fast, some slow (That's in our TSM environment)

If anybody can help, it would be appreciated.

We are running Netbackup 7.6 on Windows 2008 R2. We have one media / master server.

[jim_dalton]

Do the data reside on the same datastores/aggregates?...thats looking like the only difference.

What else are the VMs doing when you back up?

Jim

[Seth_E]

The D:\ (Where Oracle data files reside) of both VM's are RDM's to XIV.  the VM's are mostly dorment while backing up (Dev boxes - hardly used)

[Mark_Solutions]

Could there be anything in the network setup of these boxes that is different?

I am assuming you are backing them up over the LAN as clients so just looking at the usual causes of slow network backups, especially as the vmotion makes no difference.

There are also two bugs in 7.6 (I assume you are actually on 7.6.0.1?) which cause Incremental backups and DFSR backups to run slowly - these will be addressed in 7.6.0.2 but maybe they can be affecting your backups too.

It would be interesting to see what they actually look like when they start to run - the MB/s figures you have are an average during the entire backup but it would be good to see if the one gets averaged down due to a long pre-processing phase before data actually starts to flow which could indicate an issue somewhere.

[Will_Restore]

SAP always runs a bit slower.  Spring is just around the corner. 

(sorry, could not resist  ) 

[Nicolai]

Maybe related - somwhat - review it :

http://www.symantec.com/docs/TECH213360

[Seth_E]

Network setings are exactly the same, and both machines are on the same subnet. However, I do look forward to 7.6.0.2 as we are on .01 now. 

[Seth_E]

Lulz!

[Seth_E]

Excellent technote. Both VM's are using E1000, so, I doubt that is the issue. However, I will attempt to change the slower one to VMXNET3 and post back results.

[Seth_E]

One thing I notice when using RAMMAP to help diagnose this, is that "Mapped Memory" is extremely high when backups are running. I'm almost at the point of blaming the OS or Orace / SAP because we have similar issues in TSM. 

[Marianne]

Seems you are right about problem being with OS or SAP/Oracle.

Have you made any progress with troubleshooting?

[Seth_E · Accepted solution]

It seems that I may have found a solution. It's the drives parameter in the initsid.utl.

We backup our oracle databases to a disk staging unit. We don't have enough tape drives to facilitate direct to tape backups. I had thought that the drives parameter would only be effective with regards to tape backups, but, this is not the case. 

With drives = 1, I was getting about 10MB / sec.

When I set drives = 3, I had three streams and one of those streams was transfering at 30 MB / sec!

Now, suppose I wanted to still use drives = 3, but , have those three steams go directly to tape, is this possible? 

Thank you all for your help. I had originaly thought that this was an OS issue, and it may still be but, the drives parameter resolved the problem.

Seth

[Nicolai]

Yes - "drives" is  SAP word for streams. Thease strems can go to what ever device you have avaiable in Netbackup.

e.g Our larger SAP installation are running with DRIVES = 5 direct to disk for production and to tape for the rest.

Best Regards

Nicolai

[Seth_E]

Awesome!

Is there a way to direct all 5 of those streams to one tape drive?

[Marianne]

Yes - with MPX setting in STU properties and MPX setting in the Schedule.

[Seth_E]

Do you mean MPX setting in the policy? MPX is grayed out for SAP policy types.  I do not see an MPX setting in either the default-application or automatic schedule.