Master server overloade, memory full because of bpbkar process

[Thiago_Ribeiro]

Hi

I have a problem in my environment, where the Kernel from S.O is finishing bpbkar  process. I contacted S.O Team and they told me that kernel is finishing the bpbkar because of high cpu utilization.

Recently I did the upgrade from NBU version 7.6.1.1 to 7.6.1.2. My environment is totaly for Vmware backups, almost one hundred VMs, and these runs on weekends.

When the backup windows starts, some backups completes with success, but most of backups fail, after the crash of bpbkar process

My Environment

Master Server Type: Vmware
Windows Server 2008 R2 Enterprise SP1
Processor 4 processors 16 cores
Memory 32GB
NBU Version 7.6.1.2

Attached some logs, below follow messages from event viewer. Can you help me?

Faulting application name: bpbkar32.exe, version: 7.601.215.522, time stamp: 0x555fa395
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18409, time stamp: 0x5315a05a
Exception code: 0x80000003
Fault offset: 0x0000000000033ca2
Faulting process id: 0x1b48
Faulting application start time: 0x01d0b1ffee496440
Faulting application path: D:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: e2989e8c-1df4-11e5-86b6-005056a410ea

[Nicolai]

Consider to open a support case with Veritas.

[revarooo]

Are you using the master server as your backup host? If so, I would highly recommend you have a dedicated backup host and leave your master to just perform Master server duties.

How many backups are you running on the Backup host at any one time?

[Thiago_Ribeiro]

Hi Nicolai,

I opened a case with support Veritas and Im waiting they returned.

[Thiago_Ribeiro]

Hello revaroo

Yes, Im using the master server as my backup host.Currently I am running on the weekends almost a thousand and five hundred jobs, on whole weekend.

[revarooo]

1500 jobs? All from the same Esxi and same Datastore? Need to know more....

[mph999 · Accepted solution]

Why not split the jobs over the week, by this I mean instead of having 1500 fulls runing on Fri, have 215 running fulls on Fri, 215 running fulls Sat, 215 running fulls Sun, 215 running fulls Mon and so on.  For the days a group of clients is not running full, then run incremental.

I never understand why people run all the fulls at the weekend, it doesn't matter, as long as there is a full once a week, and incrementals the other days, then which day the full is is irrelvant.

This prevents hammering a system at a weekend and spreads the load out evenly.

[Michael_G_Ander]

Have had something similar on physical windows master server, in our case another program used almost all the memory. Back in Netbackup 7.0, exhaustion of the nointeractive desktop heap could be an issue. As your Master is a vm, have you looked at the underlying infrastructure performance when the issue occurs ? ESX host CPU/RAM, datastore response time which should be below 20ms and preferably 10ms or less for the Ntebackup Catalog and NBDB.

Have you limited the resources in Master server properties ?

[mcrigg]

Most companies run full backups over the weekend because unless you are running synthetic backups, it takes a load off of the client during peak working hours.  I agree with you that it does not matter what day fulls backups are run as long as it is done once a week and incremental backups the rest of the week. However, if backups are running while end users are hitting those servers, they will start to see performance degredation.

[mcrigg]

What you could do is create a couple of media servers to take the load off of your master.  Obviously you will need to incur the cost of two licenses, but this will definitely help.  Also, I don't know if you are using disk or tape but if you have tape libraries, you should also look into purchasing a share storage license so that the media servers can both write to your tape libraies at the same time.  You will have to do some performance testing so that you can match the throughput of the drives with the amount of streams, including multiplexing and multistreaming.  Just my two cents...................................... 

[mph999]

Sure, very true if you have employees in overnight.

If however, everyone, or the vast majority go home in the evening, it is a good approach to take.

[Nicolai]

This is a abandon thread. Marking it Read/Only.