Veritas Net backup not backing up a one particular production VM

[ramkr2020]

Hi All,

Veritas Net backup not backing up a one particular production VM, There is no sign that the backup is getting started on this VM either on Activity monitor, i even tried to initiate a manual backup but nothing happens.. its a production and needs a backup daily. 

The last backup was successful on 28th July but it failed on 29th and 30th July with status : 96, error Unable to allocate new media for backup,storage and since then there are no logs of the backup for this server either its failed or completed, how to check this out. please help. 

[Anonymous]

Error Code/Status 96 normally means there a no more media in the volume pool your policy specifies that includes this VM as a client.

Have you confirmed there is unallocated in that pool or alternately media available in scratch pool?

Run the available_media script also.

There is a wizard to troubleshoot status codes:

Start here and click on 96

http://www.symantec.com/business/support/index?page=content&id=TECH58686&key=15143

[ramkr2020]

Thanks Stuart, Now the problem is even when i try to manually initiate the backup on this particular VM, its not showing up in Activity monitor that backup started like other vm's and no logs since July 30th.. not sure how to find whether backup is getting triggered on this particular vm. 

[Anonymous]

Remove and then add back in the problem VM.

Are you manually adding VM's into the policy or using a query or folder based addition?

If it is a folder or query has someone moved the VM within the Virtual Infrastructure.?

[ramkr2020]

After the veritas netbackup master server reboot , the vm backup initiated and i can see that now in Activity monitor the backup started and got failed just a minute ago with  error : 41 

8/6/2012 6:06:50 PM - Error bpbrm(pid=5632) could not write KEEPALIVE to COMM_SOCK       

8/6/2012 6:07:05 PM - begin writing

8/6/2012 6:07:12 PM - positioned L283L4; position time: 00:00:45

8/6/2012 6:13:50 PM - Error bpbrm(pid=5632) could not send server status message       

8/6/2012 6:13:53 PM - end writing; write time: 00:06:48

network connection timed out(41)

This is happening randomly, whenever when the backup starts the client VM goes to hung state with error code 6.13 pm 

[Anonymous · Accepted solution]

1. Have you presented the LUN for the datastore that the problem VM is on to the NetBackup Vmware Backup Host. Confirm this.

2. Check that the VM conforms to supportability for your version of NetBackup. ie VM hardware version etc.

vmx-8 vs vmx-7.

3. If you have the space move the VM onto the same storage as other VM's that are successfully being backed up and retry.

4. Confirm you have the latest update release for NetBackup installed on your master and media server that are involved. (however, you do say that other VM's backup OK so only resort to this at the later stages of troubleshooting)

[ramkr2020]

the below thing is exactly happening when the backup starts, looks like delete snapshot throws error: 58 then error:41 and gives network connection timed out. we have also opened a case with Microsoft regrading this error. Any reason for this to happen?

8/6/2012 6:03:34 PM - requesting resource Any

8/6/2012 6:03:34 PM - requesting resource apsydbu01.NBU_CLIENT.MAXJOBS.apsydfs01

8/6/2012 6:03:34 PM - requesting resource apsydbu01.NBU_POLICY.MAXJOBS.APSYDFS01

8/6/2012 6:04:05 PM - granted resource apsydbu01.NBU_CLIENT.MAXJOBS.apsydfs01

8/6/2012 6:04:05 PM - granted resource apsydbu01.NBU_POLICY.MAXJOBS.APSYDFS01

8/6/2012 6:04:05 PM - granted resource L283L4

8/6/2012 6:04:05 PM - granted resource HP.ULTRIUM4-SCSI.000

8/6/2012 6:04:05 PM - granted resource apsydbu01-hcart-robot-tld-1

8/6/2012 6:04:06 PM - begin Parent Job

8/6/2012 6:04:06 PM - begin Snapshot , Step By Condition

Status 0

8/6/2012 6:04:06 PM - end Snapshot , Step By Condition; elapsed time: 00:00:00

8/6/2012 6:04:06 PM - begin Snapshot , Stream Discovery

Status 0

8/6/2012 6:04:06 PM - end Snapshot , Stream Discovery; elapsed time: 00:00:00

8/6/2012 6:04:06 PM - begin Snapshot , Read File List

Status 0

8/6/2012 6:04:06 PM - end Snapshot , Read File List; elapsed time: 00:00:00

8/6/2012 6:04:06 PM - begin Snapshot , Create Snapshot

8/6/2012 6:04:14 PM - begin Create Snapshot

8/6/2012 6:04:14 PM - started process bpbrm (5032)

8/6/2012 6:05:00 PM - end Create Snapshot; elapsed time: 00:00:46

8/6/2012 6:05:03 PM - end writing

Status 0

8/6/2012 6:05:03 PM - end Snapshot , Create Snapshot; elapsed time: 00:00:57

8/6/2012 6:05:03 PM - begin Snapshot , Policy Execution Manager Preprocessed

Status 41

8/6/2012 6:14:13 PM - end Snapshot , Policy Execution Manager Preprocessed; elapsed time: 00:09:10

8/6/2012 6:14:13 PM - begin Snapshot , Delete Snapshot

8/6/2012 6:17:20 PM - end writing

Status 58

8/6/2012 6:17:22 PM - end Snapshot , Delete Snapshot; elapsed time: 00:03:09

Status 41

8/6/2012 6:17:22 PM - end Parent Job; elapsed time: 00:13:16

network connection timed out(41)

[ramkr2020]

Now after the VM restart , the veritas netbackup started find and running without any issues. 

please find the log for snapshot creation, now its writing files on the tapes, not sure why it needs this particular VM to get restarted for the backup to run.. Any idea on this?

8/6/2012 6:42:25 PM - begin Create Snapshot

8/6/2012 6:42:18 PM - requesting resource Any

8/6/2012 6:42:18 PM - requesting resource apsydbu01.NBU_CLIENT.MAXJOBS.apsydfs01

8/6/2012 6:42:18 PM - requesting resource apsydbu01.NBU_POLICY.MAXJOBS.APSYDFS01

8/6/2012 6:42:18 PM - granted resource apsydbu01.NBU_CLIENT.MAXJOBS.apsydfs01

8/6/2012 6:42:18 PM - granted resource apsydbu01.NBU_POLICY.MAXJOBS.APSYDFS01

8/6/2012 6:42:18 PM - granted resource L283L4

8/6/2012 6:42:18 PM - granted resource HP.ULTRIUM4-SCSI.001

8/6/2012 6:42:18 PM - granted resource apsydbu01-hcart-robot-tld-1

8/6/2012 6:42:19 PM - begin Parent Job

8/6/2012 6:42:19 PM - begin Snapshot , Step By Condition

Status 0

8/6/2012 6:42:19 PM - end Snapshot , Step By Condition; elapsed time: 00:00:00

8/6/2012 6:42:19 PM - begin Snapshot , Stream Discovery

Status 0

8/6/2012 6:42:19 PM - end Snapshot , Stream Discovery; elapsed time: 00:00:00

8/6/2012 6:42:19 PM - begin Snapshot , Read File List

Status 0

8/6/2012 6:42:19 PM - end Snapshot , Read File List; elapsed time: 00:00:00

8/6/2012 6:42:19 PM - begin Snapshot , Create Snapshot

8/6/2012 6:42:25 PM - started process bpbrm (5444)

8/6/2012 6:43:16 PM - end Snapshot , Create Snapshot; elapsed time: 00:00:57

8/6/2012 6:43:18 PM - end writing

Status 0

8/6/2012 6:43:19 PM - end Parent Job; elapsed time: 00:01:00

8/6/2012 6:43:19 PM - begin Snapshot , Policy Execution Manager Preprocessed

[Anonymous]

Difficult to say why you had to restart the VM. Was there OS patching pending a reboot, or VMware Tools pending a reboot perhaps. Think what might have changed since the last working backup.
Good to hear its working.

If I help in any way then please mark any as a solution or vote. Thanks.

[ramkr2020]

This happens often and restart VM is only solution fixing the issue. Not sure whether the problem lies on netback side or microsoft end. Is there a way to analyze this kind of issue with any logs enabled why back up failed and what causing theVM to hung when backup starts. 

[Anonymous]

With problem VM's can you consistently take a successful snapshot within Virtual Infrastructure?

This removed NetBackup from equation. If fails then its the VI you need to analyze.

If this is VMware, then you can analyze the VM's 'vmware.log' file which is available on the datastore folder where the VM's configuration is also located. Browse the datastore and download using the VI Client and analyze the point of at time of snapshot. Could be some errors recorded for troubleshooting in there.

If clean then, I would compare working VM's with non-working VM's - are they all configured similarly.

VM Hardware version. Running on the same version hypervisor. Tools up to date.

You could reinstall the VMware Tools.

You could see if any other events recorded in against the VM in the VI Client.

Please also state some versions of software in your environment when posting in Forums. Can help a lot.

Also, confirm set the policy to cleanup old snapshots ( this is a 7.1 and greater feature)
Existing snapshot handling: Remove NBU