cancel
Showing results for 
Search instead for 
Did you mean: 

nblbc.exe crashes on GRT proxy server

ShelleyK
Level 4

Our backup server is NBU Enterprise Server, 7.6.1.1 running on OEL 5.10 (x86), on a Dell R620 server and Dell PVTL 2000 (2 x LTO5 drives), each drive connected via 6Gps SAS interface to NBU server. 

We have two Exchange 2010 servers, each DB over 420GB. I put in place a Windows 2008 R2 proxy server to handle the duplication of images from disk (BasicDisk) to tape. We need the GRT option as we need to recover items at a message level. 

What has been erratically happening is that duplication jobs fail with a 191 status and the dreaded:
04/23/2015 09:08:17 - Error bpduplicate (pid=828) db_IMAGE() failed: database system error (220)
04/23/2015 09:08:17 - Error bpduplicate (pid=828) Status = no images were successfully processed.
04/23/2015 09:08:17 - Error bpduplicate (pid=828) Duplicate of backupid exch2_1429750827 failed, database system error (220).
04/23/2015 09:08:17 - Error bpduplicate (pid=828) Status = no images were successfully processed.

 

So... been going back and forth with support and for some reason, today I looked at the Application Log on the proxy server and found:

Error      4/23/2015 9:08:16 AM    Application Error              1005       (100)

Error      4/23/2015 9:08:16 AM    Application Error              1000       (100)

Error      4/23/2015 9:08:16 AM    Application Error              1000       (100)

The detail of the Event ID 1000 is:

Log Name:      Application
Source:        Application Error
Date:          4/23/2015 9:08:16 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      GRTPROXY
Description:
Faulting application name: nblbc.exe, version: 7.601.115.212, time stamp: 0x54dcea59
Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521eaf24
Exception code: 0xc0000006
Fault offset: 0x0000000000029c42
Faulting process id: 0x5a0
Faulting application start time: 0x01d07dbf25e6bbf8
Faulting application path: C:\Program Files\Veritas\NetBackup\bin\nblbc.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: cd8dd4eb-e9b9-11e4-8564-0050568d7352
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-04-23T13:08:16.000000000Z" />
    <EventRecordID>4001</EventRecordID>
    <Channel>Application</Channel>
    <Computer>GRTPROXY</Computer>
    <Security />
  </System>
  <EventData>
    <Data>nblbc.exe</Data>
    <Data>7.601.115.212</Data>
    <Data>54dcea59</Data>
    <Data>ntdll.dll</Data>
    <Data>6.1.7601.18247</Data>
    <Data>521eaf24</Data>
    <Data>c0000006</Data>
    <Data>0000000000029c42</Data>
    <Data>5a0</Data>
    <Data>01d07dbf25e6bbf8</Data>
    <Data>C:\Program Files\Veritas\NetBackup\bin\nblbc.exe</Data>
    <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
    <Data>cd8dd4eb-e9b9-11e4-8564-0050568d7352</Data>
  </EventData>
</Event>

And the detail of EventID 1005 is as follows:

Log Name:      Application
Source:        Application Error
Date:          4/23/2015 9:08:16 AM
Event ID:      1005
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      GRTPROXY
Description:
Windows cannot access the file  for one of the following reasons: there is a problem with the network connection, the disk that the file is stored on, or the storage drivers installed on this computer; or the disk is missing. Windows closed the program NetBackup because of this error.

Program: NetBackup
File: 

The error value is listed in the Additional Data section.
User Action
1. Open the file again. This situation might be a temporary problem that corrects itself when the program runs again.
2. If the file still cannot be accessed and
    - It is on the network, your network administrator should verify that there is not a problem with the network and that the server can be contacted.
    - It is on a removable disk, for example, a floppy disk or CD-ROM, verify that the disk is fully inserted into the computer.
3. Check and repair the file system by running CHKDSK. To run CHKDSK, click Start, click Run, type CMD, and then click OK. At the command prompt, type CHKDSK /F, and then press ENTER.
4. If the problem persists, restore the file from a backup copy.
5. Determine whether other files on the same disk can be opened. If not, the disk might be damaged. If it is a hard disk, contact your administrator or computer hardware vendor for further assistance.

Additional Data
Error value: C00000C4
Disk type: 0
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="49152">1005</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-04-23T13:08:16.000000000Z" />
    <EventRecordID>4003</EventRecordID>
    <Channel>Application</Channel>
    <Computer>GRTPROXY</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>NetBackup</Data>
    <Data>C00000C4</Data>
    <Data>0</Data>
  </EventData>
</Event>

Any ideas as to why nblbc would crash?

5 REPLIES 5

watsons
Level 6

An application fault is best pursued with Netbackup support, specially yours is already at latest verison 7.6.1.1.

What you can enable is ncflbc logs (OID=351) on the proxy server.

One way to narrow down the cause is to use another proxy server to see if it has the same issue, or use a different type of disk storage.

ShelleyK
Level 4

Marianne has asked me to provide feedback, so here we go!

I have been in touch with support. I have notified them of the above crashing, and have enabled the oid=351 logs. (A side note: I have become way more comfortable with vxlogcfg and vxlogview commands lol.) The case is still under investigation, and according to my case, they will be calling me any day now.

The disk storage is local to the backup server - we bought disks just for this purpose. That's an interesting idea about another proxy server, though. I'll see what support has to say...

 

thanks all!
Michelle

mikeybinthed
Level 2
Partner Certified

We are having a similar issue with 7.6.0.1 and two Exchange GRT backups. The backup works properly but the duplication fails a number of times ending in a 220 database error. Eventually it does work and the image duplicates. I have a case open with symantec and they had me make a new STU with only 1 writer operation but this did not resolve the issue. I would love to hear if you come up with a solution.

Marianne
Moderator
Moderator
Partner    VIP    Accredited Certified

Hopefully you have tried  to use another proxy server by now? Or asked Support?

mikeybinthed
Level 2
Partner Certified

Hi Marianne, can you tell me how to determine if I have another proxy? I'm not sure home to check this.