cancel
Showing results for 
Search instead for 
Did you mean: 

Exchange 2007 backups fail after upgrade to NBU 7

bjameslanford
Level 3
Partner

Hello!

I've had successful backups for some time now against my Exchange 2007 clustered environment.  After upgrading to NBU 7, I am now getting error code 69 (invalid filelist specification).  I am also not able to enumerate ANY resources when selecting items to back up in the policy options.  Also, when I log in to the client and attempt to enumerate items available for backup, I do not see any Exchange resources, whereas before the upgrade I did see these resources.

Here is the contents of the bpbkar log in after a recent unsuccessful backup attempt:

8:06:26.368 AM: [5128.5016] <4> ov_log::OVInit: INF - Starting log file: C:\Program Files\Veritas\NetBackup\logs\BPBKAR\070210.LOG

8:06:26.368 AM: [5128.5016] <4> ov_log::OVInit: GENERAL Log Level: 0
8:06:26.368 AM: [5128.5016] <4> ov_log::OVInit: TCP Log Level: 0
8:06:26.368 AM: [5128.5016] <4> ov_log::OVInit: INF - the log mutex: 536
BPBKAR  NetBackup Backup/Archive  6.5GA  [Jan  4 2010]
Copyright 1993 - 2007 VERITAS Software Corporation
All Rights Reserved.

8:06:26.368 AM: [5128.5016] <2> WinMain: DAT - _pgmptr = 'C:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe'
8:06:26.368 AM: [5128.5016] <2> WinMain: DAT - lpCmdLine = '-r 2678400 -ru root -dt 0 -to 0 -clnt cairexmbx-exch -class mailboxes_a-l -sched WeeklyFullA-L -st FULL -bpstart_to 300 -bpend_to 300 -read_to 300 -blks_per_buffer 512 -pdi -granular_backup -use_otm -fso -b cairexmbx-exch_1278083166 -kl 5 -ct 16 -use_ofb '
8:06:26.368 AM: [5128.5016] <2> date_debug: DAT - timezone: Pacific Standard Time, offset=28800, dst: Pacific Daylight Time
8:06:26.368 AM: [5128.5016] <2> date_debug: DAT - current time: 1278083186, 7/2/2010 8:06:26 AM
8:06:26.368 AM: [5128.5016] <2> date_debug: DAT - 01/01/94 UCT:  757382400, 12/31/1993 5:00:00 PM
8:06:26.368 AM: [5128.5016] <2> date_debug: DAT - 07/01/94 UCT:  773020800, 6/30/1994 5:00:00 PM
8:06:26.368 AM: [5128.5016] <2> WinMain: DAT - standard input handle = 608
8:06:26.368 AM: [5128.5016] <2> WinMain: DAT - standard output handle = 380
8:06:26.368 AM: [5128.5016] <2> WinMain: DAT - standard error handle = 500
8:06:26.368 AM: [5128.5016] <4> tar_backup::V_SetupJobData: INF - dwJobData: ffffffff
8:06:26.368 AM: [5128.5016] <4> tar_backup::V_SetupJobData: INF -     dwJob: ffffffff
8:06:26.384 AM: [5128.5016] <4> dos_backup::V_PreProcessing: INF - backup privileges enabled, previous = 0
8:06:26.384 AM: [5128.5016] <4> dos_backup::V_PreProcessing: INF - restore privileges enabled, previous = 0
8:06:26.384 AM: [5128.5016] <4> dos_backup::V_PreProcessing: INF - security privileges enabled, previous = 0
8:06:26.384 AM: [5128.5016] <4> dos_backup::V_PreProcessing: INF - tcb privileges enabled, previous = 0
8:06:26.384 AM: [5128.5016] <4> dos_backup::V_PreProcessing: INF - create token privileges enabled, previous = 0
8:06:26.384 AM: [5128.5016] <4> dos_backup::V_PreProcessing: INF - user name: root
8:06:26.384 AM: [5128.5016] <4> dos_backup::V_PreProcessing: INF - no access token: scheduled backup
8:06:26.384 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - BEDS_Init() Enter InitFlags:0x0
8:06:26.431 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - BEDS_Init() BEDS debgging will be enabled - dwDebugLevel = 0.
8:06:28.958 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: C:
8:06:28.958 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: D:
8:06:28.958 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Microsoft Terminal Services
8:06:28.958 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Microsoft Windows Network
8:06:28.958 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: NFS Network
8:06:28.958 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Shadow?Copy?Components
8:06:28.958 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: System?State
8:06:28.958 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Active Directory Application Mode
8:07:20.938 AM: [5128.5016] <2> ov_log::V_GlobalLog: WRN - BEDS_Init() FS_BlowOutMachine() Failed! (0x35:The network path was not found.
)
8:07:21.297 AM: [5128.5016] <2> tar_backup::V_SetupProcessContinue: TAR - CONTINUE BACKUP received
8:07:21.500 AM: [5128.5016] <2> tar_backup_tfi::setupFileDirectives: TAR - backup filename = Microsoft Exchange Mailboxes:\[A-L]*
8:07:21.500 AM: [5128.5016] <2> tar_backup::setupNBFSD_Credentials: INF - Processing nbfsd credentials and port number
8:07:22.077 AM: [5128.6188] <4> tar_base::keepaliveThread: INF - keepalive thread is active with an interval of 60 seconds
8:07:22.108 AM: [5128.5016] <2> tar_backup_vxbsa::add: INF - called with 'Microsoft Exchange Mailboxes:\[A-L]*'
8:07:22.108 AM: [5128.5016] <4> dos_backup::V_VerifyFileList: INF - unable to determine UBS type for:_BACKUP_SPECIAL_OBJECTS BEFORE Microsoft Exchange Mailboxes:
8:07:22.108 AM: [5128.5016] <4> dos_backup::V_VerifyFileList: INF - UBS Local Type for '_BACKUP_SPECIAL_OBJECTS BEFORE Microsoft Exchange Mailboxes:' --> 00000000
8:07:22.108 AM: [5128.5016] <4> dos_backup::V_VerifyFileList: INF - unable to determine UBS type for:Microsoft Exchange Mailboxes:
8:07:22.108 AM: [5128.5016] <4> dos_backup::V_VerifyFileList: INF - UBS Local Type for 'Microsoft Exchange Mailboxes:\[A-L]*' --> 00000000
8:07:22.108 AM: [5128.5016] <4> dos_backup::V_VerifyFileList: INF - unable to determine UBS type for:_BACKUP_SPECIAL_OBJECTS AFTER Microsoft Exchange Mailboxes:
8:07:22.108 AM: [5128.5016] <4> dos_backup::V_VerifyFileList: INF - UBS Local Type for '_BACKUP_SPECIAL_OBJECTS AFTER Microsoft Exchange Mailboxes:' --> 00000000
8:07:22.108 AM: [5128.5016] <4> dos_backup::V_Initialize: INF - Exchange Backup - Disabling Open File Backups, Archive Bit processing, TIR, Job Estimating and Excludes.
8:07:22.108 AM: [5128.5016] <4> backup_create: INF - NetBackup Temp Directory: 'C:\Program Files\Veritas\\NetBackup\Temp'
8:07:22.108 AM: [5128.5016] <4> dos_backup::tfs_scanstart: INF - Backup Special Objects request BEFORE Microsoft Exchange Mailboxes:
8:07:22.108 AM: [5128.5016] <2> tar_base::V_vTarMsgW: ERR - object not found for file system backup: Microsoft Exchange Mailboxes:
8:07:22.108 AM: [5128.5016] <4> tar_backup::backup_done_state: INF - number of file directives not found: 1
8:07:22.108 AM: [5128.5016] <4> tar_backup::backup_done_state: INF -     number of file directives found: 0
8:07:22.108 AM: [5128.6188] <4> tar_base::keepaliveThread: INF - keepalive thread terminating (reason: WAIT_OBJECT_0)
8:07:22.108 AM: [5128.5016] <4> tar_base::stopKeepaliveThread: INF - keepalive thread has exited. (reason: WAIT_OBJECT_0)
8:07:22.108 AM: [5128.5016] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 69: invalid filelist specification
8:07:22.108 AM: [5128.5016] <4> tar_backup::backup_done_state: INF - Not waiting for server status
8:07:22.108 AM: [5128.5016] <4> dos_backup::tfs_reset: INF - Snapshot deletion start
8:07:22.108 AM: [5128.5016] <4> OVStopCmd: INF - EXIT - status = 0
8:07:22.108 AM: [5128.5016] <2> tar_base::V_Close: closing...
8:07:22.108 AM: [5128.5016] <4> dos_backup::tfs_reset: INF - Snapshot deletion start
8:07:22.108 AM: [5128.5016] <2> ov_log::V_GlobalLog: INF - BEDS_Term(): enter - InitFlags:0x00000001
8:07:22.124 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:23.138 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:24.152 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:25.166 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:26.180 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:27.194 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:28.208 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:29.222 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:30.236 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:31.250 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:32.264 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:33.278 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:34.292 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:35.306 AM: [5128.5016] <16> dtcp_read: TCP - failure: recv socket (608) (TCP 10053: Software caused connection abort)
8:07:35.306 AM: [5128.5016] <4> OVShutdown: INF - Finished process
8:07:35.306 AM: [5128.5016] <4> WinMain: INF - Exiting C:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe
8:07:37.334 AM: [5128.5016] <4> ov_log::OVClose: INF - Closing log file: C:\Program Files\Veritas\NetBackup\logs\BPBKAR\070210.LOG

I would appreciate any pointers... I've been troubleshooting this for two weeks and have tried everything I can find on Symantec's support site (hosts file cleanup, verifying permissions on the NetBackup service account, rebooting the clients, etc...) and haven't been able to make any progress whatsoever.  I'm coming up on two weeks of not having the Exchange store backed up.

Thank you!

-Brandon
8 REPLIES 8

MattS
Level 6
What do you have for backup selections in your exchange policy?

I believe the recommended backup selection is:
Microsoft Information Store:\*

bjameslanford
Level 3
Partner
The backup selection is indicated in the bpbkar log:

Microsoft Exchange Mailboxes:\[A-L]*, as this is a mailbox level GRT backup as opposed to an info store-level backup.

Thank you.

MattS
Level 6
I found this on pg 63 of the NB Exchange Admin guide for 7.0:



What i get from that, is that your not able to perform GRT backups when backing up with the Mailboxes selection.

Some other things to try:
  • In the documentation the backup selection is always lower case for 'Microsfot Exchange Mailboxes:\[a-l]*  ... Maybe give that a shot?
  • Can you attempt an information store backup and see if that works?
  • If GRT (not sure if your calling Mailbox backups GRT or if you're refering to the actual GRT option in the policy[aka Brick Level backup]) is selected in the policy, maybe try just unchecking it?

On a side note, i see that you said you can no longer enumerate resources in the backup policy.  If you're using the Java admin console i dont think you'll be able to.  The windows console does this... but i've only see it populate items once and that was a small mail store.  It simply doesnt work for our 100-300GB mail stores.

Let us know if you get this resolved, I plan on upgrading to 7.0 in a few weeks.

bjameslanford
Level 3
Partner
I've tried unchecking the option to no avail.  What I mean when I can't enumerate anythin under the client is this:


It doesn't enumerate ANYTHING under the client.  I should at least see something, no?

I have a case open with Symantec and will for sure post the solution here (and everywhere else).

With regard to the manual page, I read that section (as well as the rest... a few times) and it's not clear from the manual whether or not you have to have 'Enable granular recovery' checked in order to use the Microsoft Exchange Mailboxes directive to backup and ultimately restore individual emails (though I would assume I would need it checked, and I have in the past).  Keep in mind, this policy worked flawlessly BEFORE the upgrade.  This only happened after.

I just tried unchecking the box and changing the backup selection name to lowercase vice uppercase.  Same result.

Thanks!

MattS
Level 6
Oh i see what you're talking about.  I at least see Information Store and Public Folders, i just cant drill past the mail stores.  Ya thats not good.... Can you perform a standard or windows file system backup of the clustered host name?

Sounds like there might be an issue with hostname resolution.

bjameslanford
Level 3
Partner
I can enumerate resources on the client if I indicate a policy type other than MS-Exchange-Server.  Only if I choose that policy type am I then not able to enumerate the Exchange resources.

CRZ
Level 6
Employee Accredited Certified

Although I don't see it definitively in your bpbkar, I have a hunch that you may be experiencing a bug where we're truncating your client's longish hostname down to 12 characters and then later getting confused that what we're looking for doesn't exist, and throwing that status 69. 

Open a case with us and reference ET 1988899 to request a replacement library to see if that clears you up.

(A TechNote is forthcoming but not available yet)

Workaround is to downgrade your client back to 6.5.x, I'm afraid.

CRZ
Level 6
Employee Accredited Certified

Actually, your bpbkar log indicates a 6.5 client.  Is this right?

8:06:26.368 AM: [5128.5016] <4> ov_log::OVInit: INF - the log mutex: 536
BPBKAR  NetBackup Backup/Archive  6.5GA  [Jan  4 2010]
Copyright 1993 - 2007 VERITAS Software Corporation
All Rights Reserved.

That may be why I wasn't seeing the logs I was expecting.

The TechNote for hostname truncation is now available:

STATUS CODE 69: Exchange hostnames greater than twelve (12) characters may be truncated in NetBackup 7.0, causing backup failures as well as failures enumerating backups taken prior to an upgrade to 7.0.
 http://support.veritas.com/docs/357299

I'm not sure how this issue would manifest itself in a mixed environment, but I still think that given you are getting the same status code AND you have a hostname longer than 12 characters, this may still be the way to attack this.