cancel
Showing results for 
Search instead for 
Did you mean: 

System error occured (130)

AjayKumarN
Level 4

Hi guys,

i am receiving 130 error in exchange backups. i have no clue ho eto trouble shoot this. Please help.

 

i am posting detailed job status. i cheked vssadmin list writers in MBX04 client machine . all are stable.

 

details status:

Feb 22, 2018 12:50:27 AM - Info nbjm (pid=12744) starting backup job (jobid=666154) for client EVM-EMXMBX004, policy CEC_EVM-EMXMBX004, schedule Full
Feb 22, 2018 12:50:27 AM - Info nbjm (pid=12744) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=666154, request id:{0AD38C09-D750-4F7A-A7BB-6F38FF4CDD1F})
Feb 22, 2018 12:50:27 AM - requesting resource LTO5_Media_Manager
Feb 22, 2018 12:50:27 AM - requesting resource edc-bkp01.NBU_CLIENT.MAXJOBS.EVM-EMXMBX004
Feb 22, 2018 12:50:27 AM - requesting resource edc-bkp01.NBU_POLICY.MAXJOBS.CEC_EVM-EMXMBX004
Feb 22, 2018 12:50:27 AM - granted resource  edc-bkp01.NBU_CLIENT.MAXJOBS.EVM-EMXMBX004
Feb 22, 2018 12:50:27 AM - granted resource  edc-bkp01.NBU_POLICY.MAXJOBS.CEC_EVM-EMXMBX004
Feb 22, 2018 12:50:27 AM - granted resource  000591
Feb 22, 2018 12:50:27 AM - granted resource  HP.ULTRIUM5-SCSI.001
Feb 22, 2018 12:50:27 AM - granted resource  edc-bkp01-hcart-robot-tld-0
Feb 22, 2018 12:50:27 AM - estimated 720458272 kbytes needed
Feb 22, 2018 12:50:27 AM - begin Parent Job
Operation Status: 0
Feb 22, 2018 12:50:27 AM - end Parent Job; elapsed time 0:00:00
Operation Status: 0
Feb 22, 2018 12:50:27 AM - started process bpbrm (pid=8800)
Feb 22, 2018 12:50:34 AM - Info bpbrm (pid=8800) EVM-EMXMBX004 is the host to backup data from
Feb 22, 2018 12:50:34 AM - Info bpbrm (pid=8800) reading file list for client
Feb 22, 2018 12:50:35 AM - Info bpbrm (pid=8800) start bpfis on client
Feb 22, 2018 12:50:35 AM - Info bpbrm (pid=8800) Starting create snapshot processing
Feb 22, 2018 12:50:37 AM - Info bpfis (pid=15120) Backup started
Feb 22, 2018 12:50:50 AM - Critical bpbrm (pid=8800) from client EVM-EMXMBX004: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
Feb 22, 2018 12:50:50 AM - Info bpbrm (pid=8800) DB_BACKUP_STATUS is 103
Feb 22, 2018 12:50:50 AM - Info bpbrm (pid=8800) DB_BACKUP_STATUS is 103
Feb 22, 2018 12:50:50 AM - Info bpbrm (pid=8800) DB_BACKUP_STATUS is 103
Feb 22, 2018 12:50:50 AM - Info bpbrm (pid=8800) DB_BACKUP_STATUS is 103
Feb 22, 2018 12:50:51 AM - Info bpbrm (pid=8800) DB_BACKUP_STATUS is 103
Feb 22, 2018 12:50:51 AM - Info bpbrm (pid=8800) DB_BACKUP_STATUS is 103
Feb 22, 2018 12:50:51 AM - Info bpbrm (pid=8800) DB_BACKUP_STATUS is 103
Feb 22, 2018 12:50:51 AM - Info bpbrm (pid=8800) DB_BACKUP_STATUS is 103
Feb 22, 2018 12:50:51 AM - Info bpbrm (pid=8800) DB_BACKUP_STATUS is 103
Feb 22, 2018 12:50:51 AM - Critical bpbrm (pid=8800) from client EVM-EMXMBX004: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
Feb 22, 2018 12:50:51 AM - Critical bpbrm (pid=8800) from client EVM-EMXMBX004: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
Feb 22, 2018 12:50:51 AM - Critical bpbrm (pid=8800) from client EVM-EMXMBX004: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
Feb 22, 2018 12:50:51 AM - Critical bpbrm (pid=8800) from client EVM-EMXMBX004: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
Feb 22, 2018 12:50:51 AM - Critical bpbrm (pid=8800) from client EVM-EMXMBX004: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
Feb 22, 2018 12:50:51 AM - Critical bpbrm (pid=8800) from client EVM-EMXMBX004: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
Feb 22, 2018 12:50:51 AM - Critical bpbrm (pid=8800) from client EVM-EMXMBX004: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
Feb 22, 2018 12:50:51 AM - Critical bpbrm (pid=8800) from client EVM-EMXMBX004: FTL - snapshot preparation failed - Error attempting to find volumes to snap., status 130
Feb 22, 2018 12:50:51 AM - Info bpfis (pid=15120) done. status: 130
Feb 22, 2018 12:50:51 AM - Info bpfis (pid=15120) done. status: 130: system error occurred
Feb 22, 2018 12:50:51 AM - end writing
Operation Status: 130
Operation Status: 0
Feb 22, 2018 12:50:51 AM - started process bpbrm (pid=6700)
Feb 22, 2018 12:50:58 AM - Info bpbrm (pid=6700) Starting delete snapshot processing
Feb 22, 2018 12:51:03 AM - Info bpfis (pid=32980) Backup started
Feb 22, 2018 12:51:03 AM - Critical bpbrm (pid=6700) from client EVM-EMXMBX004: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.cecdag.cec.root.careered.com_1519282227.1.0
Feb 22, 2018 12:51:04 AM - Info bpfis (pid=32980) done. status: 4207
Feb 22, 2018 12:51:04 AM - Info bpfis (pid=32980) done. status: 4207: Could not fetch snapshot metadata or state files
Feb 22, 2018 12:51:04 AM - end writing
Operation Status: 4207
Operation Status: 130
system error occurred  (130)
10 REPLIES 10

we are using 2010 and DAG also same ver. windows 2008 os

 

 


backup server vertion:8.0 and client ertion 7.6.0.2

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Is this a backup that worked fine before and now failing? 

Tell us about client EVM-EMXMBX004 - is that the DAG virtual name or a node name?

Please show us your policy config: 
On master, run this command from <install-path>\veritas\netbackup\bin\admincmd :
bppllist CEC_EVM-EMXMBX004 -U

Copy the text output and post here.

PS:
Any reason why NBU on Exchange is such old (EOSL) version? 

 

it failed only now. we run this policy weekly once. Its completed fine last week but not today. Per day we take backup for 1 exchange server.

 

i did test backup for another exchange server  mbx03 today and its completed successfully. i hope everything fine from backup server end.

 

Here is the policy info.

 

C:\Program Files\Veritas\NetBackup\bin\admincmd>bppllist CEC_EVM-EMXMBX004 -U
------------------------------------------------------------

Policy Name: CEC_EVM-EMXMBX004

Policy Type: MS-Exchange-Server
Active: yes
Effective date: 09/18/2013 09:39:07
Mult. Data Streams: yes
Client Encrypt: no
Checkpoint: no
Policy Priority: 0
Max Jobs/Policy: Unlimited
Disaster Recovery: 0
Collect BMR info: no
Residence: LTO5_Media_Manager
Volume Pool: Exchange2010
Server Group: *ANY*
Keyword: (none specified)
Data Classification: -
Residence is Storage Lifecycle Policy: no
Exchange Source active db: yes
Exchange DAG Preferred Server: EVM-EMXMBX004 Application Discovery: no
Discovery Lifetime: 28800 seconds
ASC Application and attributes: (none defined)

Granular Restore Info: no
Ignore Client Direct: no
Use Accelerator: no
HW/OS/Client: Windows-x64 Windows2008 cecdag.cec.root.careered.com

Include: Microsoft Exchange Database Availability Groups:\CECDAG\Microsoft Information Store\MDB22
Microsoft Exchange Database Availability Groups:\CECDAG\Microsoft Information Store\MDB32
Microsoft Exchange Database Availability Groups:\CECDAG\Microsoft Information Store\MDB20
Microsoft Exchange Database Availability Groups:\CECDAG\Microsoft Information Store\MDB01
Microsoft Exchange Database Availability Groups:\CECDAG\Microsoft Information Store\MDB07
Microsoft Exchange Database Availability Groups:\CECDAG\Microsoft Information Store\EAMDB01
Microsoft Exchange Database Availability Groups:\CECDAG\Microsoft Information Store\MDB31
Microsoft Exchange Database Availability Groups:\CECDAG\Microsoft Information Store\MDB16
Microsoft Exchange Database Availability Groups:\CECDAG\Microsoft Information Store\MDB30

Schedule: Full
Type: Automatic Backup
Calendar sched: Enabled
Included Dates-----------
SPECIFIC DATE 0 - 09/06/2015
SPECIFIC DATE 1 - 07/10/2016
Thursday, Week 1
Thursday, Week 2
Thursday, Week 3
Thursday, Week 4
Thursday, Week 5
Excluded Dates----------
EXCLUDE DATE 0 - 04/02/2015
No exclude days of week entered
PFI Recovery: 0
Maximum MPX: 3
Retention Level: 7 (4 months)
Number Copies: 1
Fail on Error: 0
Residence: (specific storage unit not required)
Volume Pool: (same as policy volume pool)
Server Group: (same as specified for policy)
Residence is Storage Lifecycle Policy: 0
Daily Windows:
Sunday 00:00:00 --> Sunday 08:00:00
Thursday 00:00:00 --> Thursday 08:00:00

Whenever you see "Critical bpbrm (pid=xxxx) from client..." look at the client log to find out why it failed. In this case, the client process is bpfis. This bpbrm / bpfis combination is the snapshot job.

There are a lot of different reasons for bpfis to produce 130 or 156 error statuses. The first error in your job details says "error finding volumes to snap." This is a result. The bpfis log almost surely will tell you more.

(Speculation: It may be that it worked last week because the snaphot job ran bpfis on a different Exchange server.)

As @Marianne says, get off 7.6.0.2. We had a major BEDS upgrade between 7.5.0.5 and 7.6.0.3 to support Windows 2012 and Exchange 2013. Anything in between was a journey along the way. BEDS does the back end processing for many Windows applications. The "error finding volumes to snap" points to BEDS.

Okay,

i will plan for agent upgrade. is it only option to resolve this, it will take time for me to upgrade vertion.

I am adding client bpfis log here. can you check once and let me know any action i can take to resolve this error.

is <4> bpfis: INF - Unsupported parameter: -ruroot  is the reason for conflict.

 

 

 

Parameter -ruroot should be -ru root. The lack of the -ru parameter should not matter for Windows. I have not noticed it before, but I will look for it in the current product.

Otherwise, the logging level is too low to tell what's wrong. For each database, you have the following sequence:

Preparing freeze of Microsoft Information Store:\MDB22 using snapshot method VSS_Writer.
Created mount point C:\...\_vrts_frzn_img_Microsoft Information Store_MDB22_32416_1
Deleted mount point C:\...\_vrts_frzn_img_Microsoft Information Store_MDB22_32416_1
VfMS error 10; see following messages:
Non-fatal method error was reported
vfm_freeze: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze
VfMS method error 0; see following message:

There is no "following message."  I think what's probably happening is that NetBackup is not finding the database on this host, therefore it doesn't return a volume to snap. I suspect NetBackup isn't finding Exchange at all on this server. I can't guess why without more detailed logging.

Well i checked logging level under host properties of client. Gloal logging level is kept as minimum level. I changed to maximum level.

Will this helps in more detaild log collection/generation?

I will restart job and get BPfis log agian.

Thank you for quick response.

Will any chance for server  reboot might fix issue, as i can see vss event logs on client end exactly when backup started.

I will do reboot generally when vss issue occured when i used to work with Microsoft DPM2012r2

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified
The Preferred host parameter seems to be forcing the backup onto this node (EVM-EMXMBX004).

Have you (or Exch admin) checked and confirmed that the databases are 'alive and well' on this server?

There are two logging levels to set for a Windows client. Set the global level to 5 and the general level under Windows client to 2. You can reduce them after your test.