cancel
Showing results for 
Search instead for 
Did you mean: 

NBU 7.5.0.3 backup job fails with error 130

grigori
Level 5

hi All,

i have a dag incremental backup that is failing with error 130 folowed by DB_BACKUP_STATUS 103 and "snapshot preparation failed - Error attempting to find volumes to snap., status 130"

my NBU server is a solaris 10

i have 2 Exchange servers 2010 with dag

below are the job details of the failing job

2/29/2016 10:57:18 AM - Info nbjm(pid=22955) starting backup job (jobid=100168) for client MAILBOX01, policy XY-DAG, schedule XY-DAG-Daily  
2/29/2016 10:57:18 AM - Info nbjm(pid=22955) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=100168, request id:{0DD2FDC8-DEB1-11E5-8EE3-00269E9C1838})  
2/29/2016 10:57:18 AM - requesting resource Any
2/29/2016 10:57:18 AM - requesting resource SL500-Bserver.NBU_CLIENT.MAXJOBS.MAILBOX01
2/29/2016 10:57:18 AM - requesting resource SL500-Bserver.NBU_POLICY.MAXJOBS.XY-DAG
2/29/2016 10:57:19 AM - granted resource SL500-Bserver.NBU_CLIENT.MAXJOBS.MAILBOX01
2/29/2016 10:57:19 AM - granted resource SL500-Bserver.NBU_POLICY.MAXJOBS.XY-DAG
2/29/2016 10:57:19 AM - granted resource COM381
2/29/2016 10:57:19 AM - granted resource HP.ULTRIUM4-SCSI.000
2/29/2016 10:57:19 AM - granted resource mailbox01-hcart-robot-tld-0
2/29/2016 10:57:20 AM - estimated 43598370 Kbytes needed
2/29/2016 10:57:20 AM - begin Parent Job
2/29/2016 10:57:20 AM - begin Exchange 14 Snapshot, Step By Condition
Status 0
2/29/2016 10:57:20 AM - end Exchange 14 Snapshot, Step By Condition; elapsed time: 00:00:00
2/29/2016 10:57:20 AM - begin Exchange 14 Snapshot, Read File List
Status 0
2/29/2016 10:57:20 AM - end Exchange 14 Snapshot, Read File List; elapsed time: 00:00:00
2/29/2016 10:57:20 AM - begin Exchange 14 Snapshot, Create Snapshot
2/29/2016 10:57:20 AM - started
2/29/2016 10:57:21 AM - started process bpbrm (3068)
2/29/2016 10:58:14 AM - end writing
Status 130
2/29/2016 10:58:14 AM - end Exchange 14 Snapshot, Create Snapshot; elapsed time: 00:00:54
2/29/2016 10:58:14 AM - begin Exchange 14 Snapshot, Stop On Error
Status 0
2/29/2016 10:58:14 AM - end Exchange 14 Snapshot, Stop On Error; elapsed time: 00:00:00
2/29/2016 10:58:14 AM - begin Exchange 14 Snapshot, Delete Snapshot
2/29/2016 10:58:14 AM - started process bpbrm (1900)
2/29/2016 10:58:17 AM - end writing
Status 1542
2/29/2016 10:58:17 AM - end Exchange 14 Snapshot, Delete Snapshot; elapsed time: 00:00:03
Status 130
2/29/2016 10:58:17 AM - end Parent Job; elapsed time: 00:00:57
2/29/2016 12:01:47 PM - Info bpbrm(pid=3068) MAILBOX01 is the host to backup data from     
2/29/2016 12:01:47 PM - Info bpbrm(pid=3068) reading file list from client        
2/29/2016 12:01:47 PM - Info bpbrm(pid=3068) start bpfis on client         
2/29/2016 12:01:47 PM - Info bpbrm(pid=3068) Starting create snapshot processing         
2/29/2016 12:01:48 PM - Info bpfis(pid=8684) Backup started           
2/29/2016 12:02:34 PM - Critical bpbrm(pid=3068) from client MAILBOX01: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
2/29/2016 12:02:34 PM - Info bpbrm(pid=3068) DB_BACKUP_STATUS is 103          
2/29/2016 12:02:40 PM - Info bpbrm(pid=3068) DB_BACKUP_STATUS is 103          
2/29/2016 12:02:40 PM - Critical bpbrm(pid=3068) from client MAILBOX01: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
2/29/2016 12:02:41 PM - Info bpbrm(pid=1900) Starting delete snapshot processing         
2/29/2016 12:02:41 PM - Info bpfis(pid=0) Snapshot will not be deleted        
2/29/2016 12:02:42 PM - Info bpfis(pid=9092) Backup started           
2/29/2016 12:02:42 PM - Critical bpbrm(pid=1900) from client MAILBOX01: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.XY-dag_1456728774.1.0      
system error occurred(130)
 

 

regards,

 

23 REPLIES 23

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Is circular logging enabled on the DAG? Incr backups are not supported.

Michal_Mikulik1
Level 6
Partner    VIP    Accredited Certified

Hello,

onl\y incr backups are failing, not full. I would say that full backups must fail , too. It looks like VSS related issue.

Check health of VSS by OS command: vssadmin list writers.

I think there will be some problem, follow VSS steps then (sometimes node restart is required).

Michal

grigori
Level 5

hi Michal and Marianne.

thanks for replying.

@Marianne

the policy is of type inc differential backup. backup was running fine before and circular logging isnt enabled

@Michal

i have actually restarted the mailbox01 server, but the issue is still there also the writers are all stable on both mailbox servers.

 

thanks

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

.... the policy is of type inc differential backup.....

Full and Incr schedules need to be in the same policy.
Please confirm?

Also confirm that DAG virtual hostname is in Policy clients tab and not DAG node(s).

The next step is to enable logs - create bpfis log folder on all DAG nodes to troubleshoot snapshot failure.
Any VSS errors in Windows Event Viewer?

Mark_Solutions
Level 6
Partner Accredited Certified

There is quite a big gap here .. what are the other jobs showing during that period?

2/29/2016 10:58:17 AM - end Parent Job; elapsed time: 00:00:57
2/29/2016 12:01:47 PM - Info bpbrm(pid=3068) MAILBOX01 is the host to backup data from     

It may just be hitting a 1 hour timeout (Client read timeout of 3600 on media server?)

cruisen
Level 6
Partner Accredited

I do not remember quite well, but if your dag is on a virtual machines, you need to intstall the snaphhot provider from veritas, because the incremental flushes the logs.

https://www.veritas.com/support/en_US/article.HOWTO85812

I looked for you the technote, i hope it helps:

https://www.veritas.com/support/en_US/article.HOWTO85812

The Symantec VSS provider is recommended instead of the VMware VSS provider in the following cases:

  • You want VMware backups to truncate the logs on Exchange Server virtual machines. The Symantec VSS provider truncates logs for Exchange Server, by means of full VSS backups.

  • The virtual machine you want to back up is a node in an Exchange 2010 DAG or an Exchange 2007 CCR. In this case, only the active copies of the database are cataloged and the log files for only those same databases are truncated.

  • You want to use an exclude file list for Exchange. See the following for details on how to configure an exclude file list for Exchange.

 

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
Seems snapshot attempt took this long before giving up?

grigori
Level 5

hi,

sorry it has been a while

@Cruisen;

 our environment is based on physical server

@Mariane;

we are using virtual name in the clients tab selection also Full and Incr schedules are in the same policy.

i have enabled the bpfis logs on the master server and the mailbox servers.

it seems that the administrator has 2 public database folder that he says he is not using them but looking at them from the exchange console, these databases are mounted on mailbox01. i think the issue resides here.

i will try to exclude these database from backup selection and give the job a try.

i will post the bpfis logs later since i forgot to get them.

cruisen
Level 6
Partner Accredited

I remember something about claering in the public folders, let me check.

cruisen
Level 6
Partner Accredited

I am sorry, I did not get the information. Please try the folowing, try to trace exchange logs and find out which erreur message on the public folders happen. Aftre that with simple exchange commands you should e able to fix you issue. Please post your logs from the exchange. 

 

I remember that items were not visible, or 0 KB or not mounted but , i can not exactly tell you whiic erreur message i got. Try to check with your Exchange admin, and often they have heard about the error message before, but never fixed it because actually, it makes no difference on an Exchange perpective, but it will importent for a clean the backup.

grigori
Level 5

Hi Cruisen,

I am attaching the BPFIS file generated at the time we tried the DAG daily incremental backup.

please let me know if it gives a clear view about what should be treated.

 

Mark_Solutions
Level 6
Partner Accredited Certified

I dont see a log file attached - please try again - do the backups still fail after 1 hour (give or take a minute) as i pointed out early in this thread?

grigori
Level 5

while reviewing the bpfis file, it seems the issue is in "Microsoft Information Store:\Mailbox Database 1217415373" or/and "Microsoft Information Store:\Mailbox Database 0687444641" but I still dont have the clue in where the issue is exactly and what should be done to clear it.

grigori
Level 5

hi Mark,

thanks for replying,

actually when i run the dag daily job manually, the backup start 3 or 4 snapshot jobs, the first snapshot on mailbox01 directly fail after 2-3 minutes, later this job is retried and fails again.

 

 

Mark_Solutions
Level 6
Partner Accredited Certified

I do see tyhis in the logs:

32.9388] <4> bpfis Exit: INF - EXIT STATUS 1542: An existing snapshot is no longer valid and cannot be mounted for subsequent operations

It also seems to reference hyper-v in the logs as if it has a hyper-v snapshot provider available which seems odd on a physical.

Check "vssadmin list providers" to see what it has.

Also try "vssadmin list shadows" and "bpfis query" to see if something has been left behind that is causing your issues.

grigori
Level 5

hi Marianne,

it seems that one of the mailbox databases "Microsoft Information Store:\Mailbox Database 1217415373" has circular login enabled.

is the issue caused by this?

 

Mark_Solutions
Level 6
Partner Accredited Certified

That would cause that database backup to fail as you cannot do incremental backups for that one .. but i wouldnt expect it to make all of them fail .. or is it just the one?

grigori
Level 5

herer are the outputs of the vssadmin and bpfis

C:\Program Files\Veritas\NetBackup\bin>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

C:\Program Files\Veritas\NetBackup\bin>vssadmin list shadows
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Contents of shadow copy set ID: {7767a0fa-9ce4-4ada-a16c-97eb96c27359}
   Contained 5 shadow copies at creation time: 3/9/2016 6:35:51 PM
      Shadow Copy ID: {6dc540e1-4065-47ce-86e9-1c0c7c9a76b2}
         Original Volume: (D:)\\?\Volume{20ad3741-fcae-11df-9ccb-806e6f6e6963}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy88
         Originating Machine: Mailbox01.mtn.sd
         Service Machine: Mailbox01.mtn.sd
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: Backup
         Attributes: Differential

      Shadow Copy ID: {52e89c23-37da-4d7b-8983-a900e417bd7f}
         Original Volume: (E:)\\?\Volume{6138d341-0c1e-11e0-9a13-002128774711}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy84
         Originating Machine: Mailbox01.mtn.sd
         Service Machine: Mailbox01.mtn.sd
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: Backup
         Attributes: Differential

      Shadow Copy ID: {1714b862-38bf-496b-98d0-d9f2c9f4741c}
         Original Volume: (H:)\\?\Volume{6138d35f-0c1e-11e0-9a13-002128774711}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy85
         Originating Machine: Mailbox01.mtn.sd
         Service Machine: Mailbox01.mtn.sd
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: Backup
         Attributes: Differential

      Shadow Copy ID: {3dcbd1cc-6e4b-45f2-b601-d6246889fb61}
         Original Volume: (Q:)\\?\Volume{34635002-7098-11e2-9715-002128774711}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy87
         Originating Machine: Mailbox01.mtn.sd
         Service Machine: Mailbox01.mtn.sd
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: Backup
         Attributes: Differential

      Shadow Copy ID: {09d66822-e488-4fdf-a6f1-24f1c2473371}
         Original Volume: (T:)\\?\Volume{fbc0ea68-305c-11e4-a938-002128774710}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy86
         Originating Machine: Mailbox01.mtn.sd
         Service Machine: Mailbox01.mtn.sd
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: Backup
         Attributes: Differential

 

C:\Program Files\Veritas\NetBackup\bin>bpfis query
INF - BACKUP START 5504
INF - FIS IDs: mtn-dag_1457533869#1
INF - EXIT STATUS 0: the requested operation was successfully completed

C:\Program Files\Veritas\NetBackup\bin>

cruisen
Level 6
Partner Accredited

Hello Grigori,

please post the output of vssadmin list writers we need to know if all writers  are stable.