Jobs never stops

[Jan-Olof_Erikss]

Hi,

I got problem with jobs that never stops, no error, doing nothing, but still in running mode.

What I have figured out so far is:

1. The clients (windows) with this problem is allways (as far as I´ve seen so far) virtuell clients, I have NBU client installed in this virtuell servers. 

2. All clients could run ok for weeks, and suddenly this happend.

3. Looking NBU servers logg, no problem.

4. Clients logg nothing, but cpu is running in 100%, and bpkar32.exe is the bad guy.

Have NBU 6.0 MP5 all windows and VMWARE ESX 3.2

Anyone seen this.

[adjon]

hi,

check that

1 – stop services netbackup

                 Netbackup stop

                /usr/openv/netbackup/bin/goodies/bp.kill_all

2 – run

                 Bpps –a

 3 – run  /usr/openv/netbackup/db/jobs

                 Rename file bpjobd.act in bpjobd.act.old

                 Cd /usr/openv/netbackup/db/jobs

                Mv bpjobd.act bpjobd.act.old

--> delete active job never stop 

 4 – Restart services

                 Netbackup start

or

Another way to cancel a job is to use the bpdbjobs command from the following path:

install_path\NetBackup\bin\admincmd

To cancel a single job_id, enter the following command:

bpdbjobs -cancel <job_id>  

Here is an example of the command line canceling of job_id 11328:

bpdbjobs -cancel 11328

To cancel multiple job_id's, enter the following command:

bpdbjobs -cancel job_id,job_id,job_id    (The job_id is listed in the activity monitor's display.)

The following is an example of canceling multiple jobs 11328, 11329, 11330 from the command line (do not type a space after the comma):

bpdbjobs -cancel 11328,11329,11330

[Jan-Olof_Erikss]

Thank,s

I have looked before after commands that can kill jobs, thank´s for that. But these jobs is no problem to just cancel, the problem is why do these jobs hang, without any errors at all. These jobs take all resurces for others ending up with a lot of jobs which fails.

/joe

[Robert_Minser]

You might want to check load ballancing of the backups.

If I run too many backups against a single ESX box at one time all kinds of strainge things start happening including a portion of the jobs going into limbo.

[Jan-Olof_Erikss]

I,m just moving my vmw clients to it´s own policy,s I have 7 clients in a ploicy for DMZ and the rest, 50 clients in another policy, before they was in the same ploicy as all others, the clients are sharing 5 ESX servers.

Well, I`ll see if things running better now.

/Thanks

[Ron_Cohn]

---

Robert Minser wrote:

You might want to check load ballancing of the backups.

If I run too many backups against a single ESX box at one time all kinds of strainge things start happening including a portion of the jobs going into limbo.

---

Robert,

Is there any documentation stating what that value could be?

[Jan-Olof_Erikss]

Hi,

Now I know where the problem is, as I wrote before we have 5 ESX boxes, and all the virtuell servers keeps moving around between these 5 ESX servers, and if a virtuell server is doing a switch during backup, the job hangs. Any solution to this? The switch is done by VMWARE, load balancing and so on.

/Joe

[Ron_Cohn]

Jan-Olof,

Did this take care of your problem?

[Jan-Olof_Erikss]

Hi,

Well, last weekend all jobs was running ok, I must see if there has been a switch for any vmw clients before I can tell for shure if this problem is related is or not to the switching, but all clients so far that has "hang" has been a during switching. I must do some more testing before I can tell for shure.

/Joe

[Ron_Cohn]

In our case, most times it occurs during a migration.  However, we had 1 guest OS migrate during the backup without incident.  From the ones it is occuring on, a communication break occurs.  We are also using NIC Teaming on the Master/Media server.  However, instead of hardcoding the information, it is allowed to auto-negotiate - a No No.  We are going to hardcode the NIC information first to see what happens.

I would hate to think that a VMware migration could cause this much heartburn.  If it does, we have bigger issues.

[Jan-Olof_Erikss]

Hi,

I have checked now and there have not been any migration for any vmw clients during the backup window from last weekend and up to now, and I have not had any problem at all. I just have to wait for another failure (hang) to see if migrating was going on.

I have all NIC,s harcoded no auto neg.

I will soon test VCB togehter with NBU and snapshot client, vmware proxy server and so on....

Does anybody have experience of that?

Looks nice to get rid of NBU clients on virtuell machines, IF now individuell file restore works, and what happend if a migrate take place?

Let,s say a backup was taken and I want to do a restore, between the backup and restore there had been a migration, how is that working?

/Joe  

[Ron_Cohn]

You bring up another point - What happens if DRS decides to move your VM during a VCB process?  I have posted another thread to address what we are currently discussing, but no one has replied.  Anyword from Symantec on this topic?

[Jan-Olof_Erikss]

Hi,

There is a parameter in vmware where you can adjust how often migration should take place, in our case maigration was going on now and then even during night time, so the vmw guys adjusted that a littel bit, and since then I never had a job hang, and I checked a ferw nights and there had never been a migrate during backup. I know that this is not sorted out yet, of course a backup job should not hang during a migration, but in my case it seemed to.

/regards

[Ron_Cohn]

Jan-Olaf,

I opened an incident with Symantec and it is an issue.  The only solution if DRS is set to automatic is to upgrade to 6.5.1 and use VCB.  For us, DRS is set to manual and we have not had any issues.  The technician did submit this to the engineers to see what other magic they can perform.

[Jan-Olof_Erikss]

Hi Ron,

Ok, I suspected that it is bug or what ever in NBU, other applications can "take" a migration without errors, so Symantec have work to do.

/regards

[Ron_Cohn]

---

Jan-Olof Eriksson wrote:

I will soon test VCB togehter with NBU and snapshot client, vmware proxy server and so on....

Does anybody have experience of that?

Looks nice to get rid of NBU clients on virtuell machines, IF now individuell file restore works, and what happend if a migrate take place?

Let,s say a backup was taken and I want to do a restore, between the backup and restore there had been a migration, how is that working?

/Joe  

---

Jan-Olof,

From my understanding, even while using VCB, to do individual file restores, a Netbackup client has to be installed.

[Jan-Olof_Erikss]

Ron,

Read the document http://ensupport.symantec.com/docs/293350

From this info, can you or anybody else for shure tell that NBU client need to be installed or not to do individual file restores?

/Regards

[Ron_Cohn]

Jan-Olof,

Here is the document I am referring to:

http://eval.symantec.com/mktginfo/enterprise/white_papers/ent-whitepaper_veritas_netBackup_6.5_vmwar...

[Jan-Olof_Erikss]

Hi,

Yes, of course this is the document to read.

In table 1 one can see that it,s possible to do single file restore with NBU and VCB working together, reading the table, at least I understand that it should be possible to do single file restore without the client in VM,sm using config 3.

But further down in the document its say, for config 3.

Restoration procedure

Single file restores—

NetBackup for VMware does not require that a NetBackup client be installed

inside either the VM or inside the VMware Service Console. For this reason, direct restores to

the VM are not possible unless a NetBackup client is installed inside the VM. Alternatively, an

alternate client restore can be performed to a Windows share, and the restored files may be

accessed and transferred to the VM through this share.

Confusing to me, I would hope that a single file restore WAS possible with help of VCB, and no need for client NBU in VM,s but seems tha it´s not the case.

But is also say in the document.

Enhanced Granular File Restore from a vmdk backup is planned for release in NetBackup 6.5.1.

Is it 6.5.1 who fixes single file restore, my english is not so good as I don´t understand "enhanced granular file restore"

/Joe

[Ron_Cohn]

Jan-Olof,

The other consideration is that VCB uses "FlashBackup" technology.  I do not know if you use this (we do), but it backs up the entire disk (full's only), not what is used.  For example, we define 30GB boot drives.  Even though we may use 12GB, FlashBackup will do the entire 30GB.

Just more food for thought...