Need help as there is unknown error which i never see before

HI below is the log.

Please advised me how to solve the backup... TQ

 

8/13/2014 3:25:32 AM - Info nbjm(pid=6356) starting backup job (jobid=118263) for client TGMEXCHMBOX01, policy TGMEXCHMBOX01_NEW, schedule daily-full  
8/13/2014 3:25:32 AM - Info nbjm(pid=6356) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=118263, request id:{9A4A8399-0092-4E49-AE42-EFA545716698})  
8/13/2014 3:25:32 AM - requesting resource HP-VTL-Storage
8/13/2014 3:25:32 AM - requesting resource tg-symbackup01.NBU_CLIENT.MAXJOBS.TGMEXCHMBOX01
8/13/2014 3:25:32 AM - requesting resource tg-symbackup01.NBU_POLICY.MAXJOBS.TGMEXCHMBOX01_NEW
8/13/2014 3:25:32 AM - granted resource tg-symbackup01.NBU_CLIENT.MAXJOBS.TGMEXCHMBOX01
8/13/2014 3:25:32 AM - granted resource tg-symbackup01.NBU_POLICY.MAXJOBS.TGMEXCHMBOX01_NEW
8/13/2014 3:25:32 AM - granted resource 022VTL
8/13/2014 3:25:32 AM - granted resource HP.ULTRIUM5-SCSI.006
8/13/2014 3:25:32 AM - granted resource HP-VTL-Storage
8/13/2014 3:25:32 AM - estimated 0 Kbytes needed
8/13/2014 3:25:32 AM - begin Parent Job
8/13/2014 3:25:32 AM - begin Exchange 14 Snapshot, Step By Condition
Status 0
8/13/2014 3:25:32 AM - end Exchange 14 Snapshot, Step By Condition; elapsed time: 0:00:00
8/13/2014 3:25:32 AM - begin Exchange 14 Snapshot, Read File List
Status 0
8/13/2014 3:25:32 AM - end Exchange 14 Snapshot, Read File List; elapsed time: 0:00:00
8/13/2014 3:25:32 AM - begin Exchange 14 Snapshot, Create Snapshot
8/13/2014 3:25:32 AM - started process bpbrm (7360)
8/13/2014 3:25:32 AM - started
8/13/2014 3:25:43 AM - Info bpbrm(pid=7360) TGMEXCHMBOX01 is the host to backup data from     
8/13/2014 3:25:43 AM - Info bpbrm(pid=7360) reading file list for client        
8/13/2014 3:25:43 AM - Info bpbrm(pid=7360) start bpfis on client         
8/13/2014 3:25:44 AM - Info bpbrm(pid=7360) Starting create snapshot processing         
8/13/2014 3:26:02 AM - Info bpfis(pid=20952) Backup started           
8/13/2014 3:26:56 AM - Critical bpbrm(pid=7360) from client TGMEXCHMBOX01: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit 
8/13/2014 3:26:56 AM - Critical bpbrm(pid=7360) from client TGMEXCHMBOX01: FTL -        
8/13/2014 3:26:56 AM - Critical bpbrm(pid=7360) from client TGMEXCHMBOX01: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit 
8/13/2014 3:26:56 AM - Critical bpbrm(pid=7360) from client TGMEXCHMBOX01: FTL -        
8/13/2014 3:26:56 AM - Critical bpbrm(pid=7360) from client TGMEXCHMBOX01: FTL - snapshot processing failed, status 156   
8/13/2014 3:26:56 AM - Critical bpbrm(pid=7360) from client TGMEXCHMBOX01: FTL - snapshot creation failed - SNAPSHOT_NOTIFICATION:Smiley TongueostSnapshot failed., status 130
8/13/2014 3:26:56 AM - Info bpbrm(pid=7360) DB_BACKUP_STATUS is 156          
8/13/2014 3:26:56 AM - Info bpbrm(pid=7360) DB_BACKUP_STATUS is 156          
8/13/2014 3:26:56 AM - Warning bpbrm(pid=7360) from client TGMEXCHMBOX01: WRN - NEW_STREAM0 is not frozen    
8/13/2014 3:26:56 AM - Warning bpbrm(pid=7360) from client TGMEXCHMBOX01: WRN - Microsoft Information Store:\DB-Factory-01 is not frozen  
8/13/2014 3:26:56 AM - Warning bpbrm(pid=7360) from client TGMEXCHMBOX01: WRN - Microsoft Information Store:\DB-Factory-02 is not frozen  
8/13/2014 3:26:56 AM - Info bpfis(pid=20952) done. status: 130          
8/13/2014 3:26:56 AM - end Exchange 14 Snapshot, Create Snapshot; elapsed time: 0:01:24
8/13/2014 3:26:56 AM - Info bpfis(pid=20952) done. status: 130: system error occurred       
8/13/2014 3:26:56 AM - end writing
Status 130
8/13/2014 3:26:56 AM - end Parent Job; elapsed time: 0:01:24
8/13/2014 3:26:56 AM - begin Exchange 14 Snapshot, Stop On Error
Status 0
8/13/2014 3:26:56 AM - end Exchange 14 Snapshot, Stop On Error; elapsed time: 0:00:00
8/13/2014 3:26:56 AM - begin Exchange 14 Snapshot, Delete Snapshot
8/13/2014 3:26:56 AM - started process bpbrm (8932)
8/13/2014 3:27:12 AM - Info bpbrm(pid=8932) Starting delete snapshot processing         
8/13/2014 3:27:39 AM - Info bpfis(pid=16884) Backup started           
8/13/2014 3:27:39 AM - Critical bpbrm(pid=8932) from client TGMEXCHMBOX01: cannot open C:\Program Files\Veritas\NetBackup\online_util\fi_cntl\bpfis.fim.TGMEXCHMBOX01_1407871532.1.0      
8/13/2014 3:27:40 AM - Info bpfis(pid=16884) done. status: 4207          
8/13/2014 3:27:40 AM - end Exchange 14 Snapshot, Delete Snapshot; elapsed time: 0:00:44
8/13/2014 3:27:40 AM - Info bpfis(pid=16884) done. status: 4207: Could not fetch snapshot metadata or state files  
8/13/2014 3:27:40 AM - end writing
Status 4207
8/13/2014 3:27:40 AM - end operation
Status 130
8/13/2014 3:27:40 AM - end operation
system error occurred(130)

1 Solution

Accepted Solutions
Accepted Solution!

Exchange writer is in waiting

Exchange writer is in waiting for completion state. It should be in stable state . Kincly coordinate with your windows adminstrator for troubleshooting vss writers

View solution in original post

11 Replies

The first error in the chain

The first error in the chain is:

"8/13/2014 3:26:56 AM - Critical bpbrm(pid=7360) from client TGMEXCHMBOX01: FTL - vfm_freeze_commit: method: VSS_Writer, type: FIM, function: VSS_Writer_freeze_commit 
8/13/2014 3:26:56 AM - Critical bpbrm(pid=7360) from client TGMEXCHMBOX01: FTL -        
8/13/2014 3:26:56 AM - Critical bpbrm(pid=7360) from client TGMEXCHMBOX01: FTL - snapshot processing failed, status 156 "

This is a common Exchange snapshot error. You should see something in the event viewer on "TGMEXCHMBOX01" that lines up with the time stamps here. The problem is either with VSS or Exchange itself. 

The common causes of VSS failure are a hung writer or insufficient space to hold the snapshot. If the writer is hung then a reboot is the only way to recover. If VSS runs out of space you have to allow more disk space to be allocated to VSS.

If VSS checks out there are a number of problems with Exchange that can cause this error. It could be a problem with the state of a database. If you are running a DAG it becomes more complex.

Is this server part of a DAG cluster?

Hi, Yea we are using DAG. I

Hi,

Yea we are using DAG. I already check on the storage in Exchange local and VSS seem like it is ok. Can I know normal practise how many % of storage that we need either in Exchange local or VSS of storage that need to reserved?

Please advised.

TQ

Seems the actual status is

Seems the actual status is 130.

See  http://www.symantec.com/docs/TECH181190 

Create bpfis and bpresolver log folders on all DAG nodes.
Check bpfis log and status of VSS writers as per above TN.

Highlighted

How are you backing up your

How are you backing up your Exchange DBs, are you taking backup of active copy(Not recommanded) or passive copy(Recommanded)

 

Please run vssadmin list writers in command prompt and paste output.

HI Marianne, I just run the

HI Marianne,

I just run the vssadmin list writers

and this is the result...

1.jpg2.jpg

Please advise what is my next step to solve this problem.

TQ

HI Amit, See below

HI Amit,

See below picture.:

 

3.JPG

vssadmin list writers already been posted.

Please advise me the next step to solve this problem.

TQ

Can we take a step back and

Can we take a step back and review DAG backup config?

Which NBU version (and patch level) and Exchange version (and patch level) on Exchange nodes?
Which Windows version and patch level?

Which client name is specified in the policy?
Virtual hostname or the node name?

Have you added Distributed Application Restore Mapping?

Have you confirmed correct service logon account on all nodes?

HI marianne, NBU: Symantec

HI marianne,

NBU: Symantec Netbackup
Version: 7.6.0.1

Version.jpg

 NBU Policy

mkt1.jpg

 

below is the things that i not sure... Please guide me. TQ

Virtual hostname or the node name?

Have you added Distributed Application Restore Mapping?

Have you confirmed correct service logon account on all nodes?

Still unanswered

Still unanswered questions....

Please work with your Exchange/Cluster admins for answers:

 Exchange version (and patch level) on Exchange nodes?
Exch2010? Exch2013? 
And patch/SP/RP level?
(showing us Mailbox version does not help)


Which Windows version and patch level on Exchange nodes?
W2008? W2012? 
patch/SP/RP level?


Virtual hostname or the node name?
Check Client name in Clients tab of the policy.
Find out from Exchange Admin what are the node names and the Virtual hostname.

 Distributed Application Restore Mapping is explained in NetBackup for Microsoft Exchange Server Administrator's Guide   on page 51.

Service logon account is explained on page 88.

For a DAG, have a look at Backup Selection on page 93.
 

No. It depends on the size of

No. It depends on the size of your database. You must first determine if that is the problem before you try to fix it. Did you find anything in the Exchange error logs?

Accepted Solution!

Exchange writer is in waiting

Exchange writer is in waiting for completion state. It should be in stable state . Kincly coordinate with your windows adminstrator for troubleshooting vss writers

View solution in original post