Netbackup 5.1 unable to backup Lotus Domino 8.5.3 - Error 71

[Sunlord]

Hi Forumites,

First up, a confession - I'm a bit of a NetBackup lightweight.  I've quite a bit of experience swapping the tapes and restoring 'essential' files for users - however my job is now seeing me taking on more responsibility these days.  This is good - but this is also challenging....

So I'd be grateful for some advice, if you think that you might be able to assist (thank you).

We have a number of separate Active Directory domains running 32-bit Server 2003.  Each domain has a Netbackup 5.1 server and a Domino 8.5.3 server.

The domino servers were formerly at version 6.5 - but have now all been upgraded.

We're in the process of upgrading all of the 5.1 servers, but upgrade takes time. 

Unfortunately, during that time, a number of the Netbackup Domino policy in some of our domains has stopped working.  Emphasis on *some* here - others continue to work fine.

I've spend a couple of (very educational) days poring over every article I could get a keyword hit on, discounting them one by one.

One advised that Netbackup 5.1 *would not* backup Domino 8.5.3 - however I have seen for certain that it does (in some instances).  Others predominately focused on capitalisation, syntax, stray punctuation and trailing spaces, sharing and mappings, Domino passwords and a mismatch between the information in the policies and the information in the notes.ini file on the Domino server.

From what I see, the issue is with Notes penetrating the Domino database.  If I run the policy as a 'Windows-NT' policy, the files back-up fine.  However when I return it to a 'Notes' policy, it reports a status code 71: none of the files in the files list exist.

Following the advice in a word document which offers in depth testing for status 71, I created a few log files on both the Netbackup server and the Domino server.

I believe that the issue occurs in the line which bolded in the BPBKAR log file from the Domino server below.  I hope you don't mind, but I've xxxxx'd out the server names (but case is preserved).

BONUS QUESTION (if I may)

I've also noticed a reference in the BPCD I created on the Domino server at the same time, which states:

15:37:48.895 [20576.20668] <2> ParseConfigExA: Unknown configuration option on line 50: RenameIfExists = 0

If anyone could advise me where (if) the script/file/other which contains the 'configuration option on line 50' exists in Windows Netbackup I'd be grateful (or if you have a more general link to a 'decoding the log file' type reference that'd be great - learning is good).

Thanks in advance to anyone who can offer any assistance - I hope I've provided enough info.

-------------------------------------------------------------------

3:37:47.160 PM: [20240.20636] <4> ov_log::OVInit: INF - Starting log file: C:\Program Files\VERITAS\NetBackup\logs\BPBKAR\050714.LOG

3:37:47.160 PM: [20240.20636] <4> ov_log::OVInit: GENERAL Log Level: 0

3:37:47.160 PM: [20240.20636] <4> ov_log::OVInit: TCP Log Level: 0

3:37:47.160 PM: [20240.20636] <4> ov_log::OVInit: INF - the log mutex: 1908

BPBKAR  NetBackup Backup/Archive  5.0GA  [Apr  5 2006]

Copyright 1993 - 2004 VERITAS Software Corporation

All Rights Reserved.

3:37:47.160 PM: [20240.20636] <2> WinMain: DAT - _pgmptr = 'C:\Program Files\VERITAS\NetBackup\bin\bpbkar32.exe'

3:37:47.160 PM: [20240.20636] <2> WinMain: DAT - lpCmdLine = '-r 604800 -ru root -dt 0 -to 0 -clnt xxxxxx01 -class XXXXXX01 -sched Daily -st INCR -bpstart_to 300 -bpend_to 300 -read_to 1200 -use_otm -fso -b xxxxxx01_1399441066 -kl 28 -ct 25 -use_ofb '

3:37:47.160 PM: [20240.20636] <2> date_debug: DAT - timezone: AUS Eastern Standard Time, offset=-36000, dst: AUS Eastern Daylight Time

3:37:47.160 PM: [20240.20636] <2> date_debug: DAT - current time: 1399441067, 7/05/2014 3:37:47 PM

3:37:47.160 PM: [20240.20636] <2> date_debug: DAT - 01/01/94 UCT:  757382400, 1/01/1994 10:00:00 AM

3:37:47.160 PM: [20240.20636] <2> date_debug: DAT - 07/01/94 UCT:  773020800, 1/07/1994 10:00:00 AM

3:37:47.160 PM: [20240.20636] <2> WinMain: DAT - standard input handle = 324

3:37:47.160 PM: [20240.20636] <2> WinMain: DAT - standard output handle = 1904

3:37:47.160 PM: [20240.20636] <2> WinMain: DAT - standard error handle = 1864

3:37:47.176 PM: [20240.20636] <4> tar_backup::V_SetupJobData: INF - dwJobData: 00000000

3:37:47.176 PM: [20240.20636] <4> tar_backup::V_SetupJobData: INF -     dwJob: 00000003

3:37:47.176 PM: [20240.20636] <4> dos_backup::V_PreProcessing: INF - backup privileges enabled, previous = 0

3:37:47.176 PM: [20240.20636] <4> dos_backup::V_PreProcessing: INF - restore privileges enabled, previous = 0

3:37:47.176 PM: [20240.20636] <4> dos_backup::V_PreProcessing: INF - security privileges enabled, previous = 0

3:37:47.176 PM: [20240.20636] <4> dos_backup::V_PreProcessing: INF - tcb privileges enabled, previous = 0

3:37:47.176 PM: [20240.20636] <4> dos_backup::V_PreProcessing: INF - create token privileges enabled, previous = 0

3:37:47.176 PM: [20240.20636] <4> dos_backup::V_PreProcessing: INF - user name: root

3:37:47.176 PM: [20240.20636] <4> dos_backup::V_PreProcessing: INF - no access token: scheduled backup

3:37:47.379 PM: [20240.20636] <2> tar_backup::V_SetupProcessContinue: TAR - CONTINUE BACKUP received

3:37:47.379 PM: [20240.20636] <2> tar_backup::V_SetupFileDirectives: TAR - backup filename = C:\Lotus\Domino\notes.ini

3:37:47.379 PM: [20240.20636] <2> tar_backup::V_SetupFileDirectives: TAR - backup filename = C:\Lotus\Domino\Data\mail

3:37:47.379 PM: [20240.20636] <2> tar_backup::V_SetupFileDirectives: TAR - backup filename = C:\Lotus\Domino\Data\Apps\RSF

3:37:47.379 PM: [20240.20636] <2> tar_backup::V_SetupFileDirectives: TAR - backup filename = C:\Lotus\Domino\Data\admin4.nsf

3:37:47.379 PM: [20240.20636] <2> tar_backup::V_SetupFileDirectives: TAR - backup filename = C:\Lotus\Domino\Data\dircat.nsf

3:37:47.379 PM: [20240.20636] <2> tar_backup::V_SetupFileDirectives: TAR - backup filename = C:\Lotus\Domino\Data\mail.box

3:37:47.379 PM: [20240.20636] <2> tar_backup::V_SetupFileDirectives: TAR - backup filename = C:\Lotus\Domino\Data\names.nsf

3:37:48.442 PM: [20240.20636] <4> nbex_DebugLog: INF - UpdatePath() Path updated with Lotus Program Dir 'C:\Lotus\Domino\'

3:37:48.442 PM: [20240.20636] <2> ov_log::V_GlobalLog: INF - BEDS_Init() Enter InitFlags:0x0

3:37:48.457 PM: [20240.20636] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: C:

3:37:48.457 PM: [20240.20636] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Microsoft Terminal Services

3:37:48.457 PM: [20240.20636] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Microsoft Windows Network

3:37:48.457 PM: [20240.20636] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: VMware Shared Folders

3:37:48.457 PM: [20240.20636] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Web Client Network

3:37:48.457 PM: [20240.20636] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Shadow?Copy?Components

3:37:48.457 PM: [20240.20636] <4> dos_backup::V_VerifyLotus: INF - Replaced: C:\Lotus\Domino\notes.ini with Lotus Notes:\C:\Lotus\Domino\notes.ini

3:37:48.457 PM: [20240.20636] <4> dos_backup::V_VerifyLotus: INF - Replaced: C:\Lotus\Domino\Data\mail with Lotus Notes:\C:\Lotus\Domino\Data\mail

3:37:48.457 PM: [20240.20636] <4> dos_backup::V_VerifyLotus: INF - Replaced: C:\Lotus\Domino\Data\Apps\RSF with Lotus Notes:\C:\Lotus\Domino\Data\Apps\RSF

3:37:48.457 PM: [20240.20636] <4> dos_backup::V_VerifyLotus: INF - Replaced: C:\Lotus\Domino\Data\admin4.nsf with Lotus Notes:\C:\Lotus\Domino\Data\admin4.nsf

3:37:48.457 PM: [20240.20636] <4> dos_backup::V_VerifyLotus: INF - Replaced: C:\Lotus\Domino\Data\dircat.nsf with Lotus Notes:\C:\Lotus\Domino\Data\dircat.nsf

3:37:48.457 PM: [20240.20636] <4> dos_backup::V_VerifyLotus: INF - Replaced: C:\Lotus\Domino\Data\mail.box with Lotus Notes:\C:\Lotus\Domino\Data\mail.box

3:37:48.457 PM: [20240.20636] <4> dos_backup::V_VerifyLotus: INF - Replaced: C:\Lotus\Domino\Data\names.nsf with Lotus Notes:\C:\Lotus\Domino\Data\names.nsf

3:37:48.473 PM: [20240.20636] <20240> nbex_DebugLog: ERR - NBLN_Connect() NotesInitExtended() Failed (ErrorCode: 0x1007)

3:37:48.473 PM: [20240.20636] <4> backup_create: INF - NetBackup Temp Directory: 'C:\Program Files\VERITAS\\NetBackup\Temp'

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF - ================================================================================

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF - job tracking estimate: start

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\notes.ini

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\mail

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\Apps\RSF

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\admin4.nsf

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\dircat.nsf

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\mail.box

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\names.nsf

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF - job tracking estimate: stop

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF -     job tracking time: 0 secs

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF - ================================================================================

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF - ================================================================================

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF - Job Tracking Estimates

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF - --------------------------------------------------------------------------------

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF -           Files: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF -         Folders: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF -      Bytes Data: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF -  Gigabytes Data: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF -     Bytes Image: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF - Gigabytes Image: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::V_DetermineEstimate: INF - ================================================================================

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\notes.ini

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\mail

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\Apps\RSF

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\admin4.nsf

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\dircat.nsf

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\mail.box

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: Lotus Notes:\C:\Lotus\Domino\Data\names.nsf

3:37:48.473 PM: [20240.20636] <2> tar_base::backup_finish: TAR - backup:                              0 files

3:37:48.473 PM: [20240.20636] <2> tar_base::backup_finish: TAR - backup:          file data:          0 bytes

3:37:48.473 PM: [20240.20636] <2> tar_base::backup_finish: TAR - backup:         image data:       1024 bytes

3:37:48.473 PM: [20240.20636] <2> tar_base::backup_finish: TAR - backup:       elapsed time:          1 secs         1024 bps

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 71: none of the files in the file list exist

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF - ================================================================================

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF - Job Tracking Statistics

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF - --------------------------------------------------------------------------------

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF -           Files: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF -         Folders: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF -      Bytes Data: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF -  Gigabytes Data: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF -     Bytes Image: 1024

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF - Gigabytes Image: 0

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF - ================================================================================

3:37:48.473 PM: [20240.20636] <4> tar_backup::backup_done_state: INF - Not waiting for server status

3:37:48.473 PM: [20240.20636] <2> tar_base::V_vTarMsgW: WRN - Abnormal backup status, Lotus Transaction Logs will not be recycled

3:37:48.473 PM: [20240.20636] <20240> nbex_DebugLog: INF - NBLN_ArchiveLogEnd() Transaction log archiving has not previously been started.

3:37:48.473 PM: [20240.20636] <4> OVStopCmd: INF - EXIT - status = 0

3:37:48.473 PM: [20240.20636] <2> ov_log::V_GlobalLog: INF - BEDS_Term() Enter InitFlags:0x1

3:37:49.473 PM: [20240.20636] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

3:37:50.473 PM: [20240.20636] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

3:37:51.473 PM: [20240.20636] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

3:37:52.473 PM: [20240.20636] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

3:37:53.473 PM: [20240.20636] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

3:37:54.473 PM: [20240.20636] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

3:37:55.473 PM: [20240.20636] <16> dtcp_read: TCP - failure: recv socket (324) (TCP 10053: Software caused connection abort)

3:37:55.473 PM: [20240.20636] <4> OVShutdown: INF - Finished process

[Mark_Solutions]

Glad you appreciate the need to upgrade - even NetBackup 6.x is no longer supported - 5.x went End Of Life a LONG time ago!!

Could we see the output if the following where policyname is the name of one of you lotus policies which fails:

\netbackup\bin\admincmd\bppllist policyname -U

Just to get a feel for what the selection lists look like for the policy

Also a screen shot of the client host properties of the lotus server that fails - the Windows client - Lotus tab (or whatever it was called in V 5.1)

Thanks

[Sunlord]

Hi Mark,

Thank you for taking the time to respond.

I have attached both the output of the \netbackup\bin\admincmd\bppllist policyname -U output and a screen shot of the lotus notes tab of the client host properties of the lotus server that fails to this message.

In each case I've xxxx'd out the server names (as I consider it appropriate - I'm sure you'll understand) - however I have preserved the number of characters and the capitalisation (lower case in the screen shot BTW).

Thanks again for your assistance.

Best regards,

-Dan

[Mark_Solutions]

OK - as you have specified the paths in the host properties then the selections list in 5.1 should normally just be th epath to the lotus databases - so just C:\Lotus\Domino\Data\

The single ones can be selected as you have but it is unusual

Try with just this one path and see how it goes

[Marianne]

The Backup Selection in the policy certainly seems odd. You may want to have a look at NBU 6.5 Lotus Notes for UNIX, Linux, and Windows http://www.symantec.com/docs/TECH52809 The rules for specifying the Backup selection should be similar across versions. The next thing you want to check is compatibility. No idea where you are going to find Compatibility list for NBU 5.1. You can find Compatibility lists as from NBU 6.5 in Handy NBU links in my signature.

[Sunlord]

Hi Mark (and Marianne),

I've removed the backup selections, replaced them with just C:\Lotus\Domino\Data\ (browsing to the locations) - same result (a '71' error occurs).

I don't believe that compatability is an issue Marianne, as we have "identical" (as far as I can tell) backups working in several other domains (same versions, same policies, same selections, same schedule).

I will review the document you've sent the link to Marianne (thanks).

Regards,

-Dan

[Marianne]

I am always amazed when companies upgrade OS, databases, applications, but fail to keep the backup software up to date.

You really need to upgrade NBU as a matter of urgency.

The NBU 6.x Database Agent CL is very clear - Lotus/Domino 8.5.x is only supported as from NBU 6.5.3.
Even NBU 6.5.x ran out of support in 2012.

To get back to status 71: 
Please check NBU Client Service logon account - ensure that this account has sufficient permissions.
Best to start this account with same domain service account that is used to start Lotus Notes services.

[Sunlord]

Hi Marianne,

I've just double checked the permissions and am confident that the AD account which starts the NetBackup Client Service service has sufficient permissions to access Domino - it has both local system admin and domain admin rights...

[Marianne]

You will need bpbkar log to troubleshoot.

And PLEASE upgrade! NBU 5.1 reached EOSL ages ago: https://sort.symantec.com/eosl/show_matrix
 

[Mark_Solutions]

Can you double check that all of your locations are correct...

On the client make sure that the databases do reside at C:\Lotus\Domino\Data\ and that the lotus ini file is at C:\Lotus\Domino\notes.ini in case they have made any changes or are using a different active database location on that server

[Sunlord]

Marianne - bpbkar log from the domino server (xxxxxx01) attached.

You'll note a line which reads "3:37:48.473 PM: [20240.20636] <20240> nbex_DebugLog: ERR - NBLN_Connect() NotesInitExtended() Failed (ErrorCode: 0x1007)" - this is where it appears to all unravel.

In a working domain it reads (there are two lines before the scripts re-align):

12:16:09.172 PM: [3720.2664] <3720> nbex_DebugLog: INF - Lotus Notes Version R8.5.3.4
12:16:09.188 PM: [3720.2664] <4> dos_backup::V_Initialize: INF - Lotus Notes Backup - Disabling snapshot, Archive Bit processing, TIR, and Excludes.

Mark - the locations are definately correct (checked and double checked).

In refererence to my "bonus question" above - can either of you tell me where "line 50" exists (if it exists anywhere other then on 'on-the-fly' process?  I may be 'barking up the wrong tree' - however if there is a script or some such I'd like to check file permissions, etc.

Thanks and regards,

-Dan

[Marianne · Accepted solution]

I think you have summarised it well in your opening post:

From what I see, the issue is with Notes penetrating the Domino database.

PLEASE do not ignore compatibilty advice. Even if it works on another server, you have no guarantee that a restore will work.

As per my post above - Lotus/Domino 8.5.x is only supported as from NBU 6.5.3.

PLEASE upgrade your NBU environment as a matter of urgency.
Backups and maintenance contract must be seen as insurance policy.
Where do you go if Notes server crash and you cannot restore?