Backup of Mounted Exchange .edb files are completing with partial backups

[dvinayku]

Hello team,

we are using symantec netbackup 7.5.03 for taking backups of Exchange 2007 mailbox database, here is what we are doing.

1. Take snapshot (point in time copy) of the production exchange databases using our Snapshot application (its an VSS application consistent snapshot).

2. We mount the snapshot to Media server (NBU) and start the media backup of these files.

3. Once the backup is started we are hitting these below issues, 

8:32:33.111 AM: [5580.4416] <2> tar_base::V_vTarMsgW: ERR - failure reading file: C:\3parSnapshot\092413073700\f5a0eab0-24a6-11e3-b540-0022196d93d0\mailboxdatabase5\mdb53.edb (WIN32 21: The device is not ready. )

the backups are completing with partial error.

Is that we are doing something wrong, need urgent help on this issue.

Thanks,

Vinay

[Mark_Solutions]

I would suggest using the Exchange Client to do your exchange backups

For a decent recovery you need the database and its log files and all in a consistent state, as well as having them truncated once the backup has finished

The error sounds to me like there is an issue with the VSS writers somewhere but it may be the way the snapshot is mounted - it may be better to mount the snapshot as a mount point if you must do it that way but worth checking the writers anyway (vssadmin list writers)

What does your policy look like - or is this all user backup initiated?

[dvinayku]

hi,

 Yes its an user initiated backups, for more clarity on the issue, here are steps we do before taking NBU backups 

1.       We have Exchange database configured on one of our HP Storage arrays.

2.       We are taking snapshot (point in time) of those Production Exchange databases using our customized HP VSS Provider.

3.       We then mount this snapshot on the NBU Media server in the form of file system.

4.       We then initiate the NBU Media backup to take backup of these mounted file system and files under it, (The files include exchange database file .edb file and Exchanges log files .log)

5.       When NBU is trying to take backup of these files, we are hitting the partial backup issue, where its coming when NBU is trying to take backup of .edb file.

so when NBU starts taking backup we are hitting the partial backup issue, this partial backup are not continous, its frequently ocurring, for example, out of 10 backups 7 are partial and 3 are full backups.

here i am attaching the bpkar log file where u can see the partial backup error.

[Mark_Solutions]

Could you show me you policy or the script that you use when running the backup

As the log file is huge could you cover a start and end time of a job that has an issue - I do see a connection closed message so am wondering if you hit a keep alive timeout but could do with some start and end times and exactly how this actually works to dig deeper into it

Thanks

[Marianne]

Any VSS errors in Event Viewer?

Seems like HP snapshot issue to me...

[dvinayku]

hello,

 I dont have much insight on logs of NBU, but still i am posting the error message seen as part of our backup operation

The highlighted text below is the exact issue we are seeing,

5:16:06.477 AM: [2920.4684] <4> WinMain: INF - Exiting C:\Program Files\Veritas\NetBackup\bin\bpbkar32.exe

5:16:08.505 AM: [2920.4684] <4> ov_log::OVClose: INF - Closing log file: C:\Program Files\Veritas\NetBackup\logs\BPBKAR\092413.LOG

5:26:19.186 AM: [6024.5744] <2> tar_base::V_vTarMsgW: ERR - failure reading file: C:\3parSnapshot\092413042718\f5a0e8cb-24a6-11e3-b540-0022196d93d0\mdb6\mdb6.edb (WIN32 21: The device is not ready. )

5:26:19.186 AM: [6024.5744] <4> dos_backup::V_AddToExcludeAfterList: INF - file will be skipped during cleanup processing:<C:\3parSnapshot\092413042718\f5a0e8cb-24a6-11e3-b540-0022196d93d0\mdb6\mdb6.edb>

5:26:19.342 AM: [6024.5744] <8> file_system_access::getAttributes: WRN - unable to get file attributes: '\\?\C:\3parSnapshot\092413042718\f5a0e8cb-24a6-11e3-b540-0022196d93d0\mdb6\mdb6.edb' (WIN32 5: Access is denied. )

5:26:19.358 AM: [6024.5744] <4> dos_backup::tfs_scannext: INF - no more path list entries

5:26:19.358 AM: [6024.5744] <4> tar_backup::backup_endarg_state: INF - finish error count: 1

5:26:19.358 AM: [6024.5744] <4> tar_backup::backup_endarg_state: INF - finish fatal count: 0

5:26:19.358 AM: [6024.5744] <4> tar_backup_tfi::backup_arg_state: INF - remember start error count: 1

5:26:19.358 AM: [6024.5744] <4> tar_backup_tfi::backup_arg_state: INF - remember start fatal count: 0

5:26:19.358 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - starting scan for file directive:<C:\3parSnapshot\092413042718\f5a0e8cb-24a6-11e3-b540-0022196d93d0\sysfile\tmp.edb>

5:26:19.358 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - starting scan using name:<C:\3parSnapshot\092413042718\f5a0e8cb-24a6-11e3-b540-0022196d93d0\sysfile\tmp.edb>

5:26:19.358 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - in 'backup' mode

5:26:24.818 AM: [6024.5744] <2> tar_base::V_vTarMsgW: TRV - object not found for file system backup: C:\3parSnapshot\092413042718\f5a0e8cb-24a6-11e3-b540-0022196d93d0\sysfile\tmp.edb

5:26:24.818 AM: [6024.5744] <4> dos_backup::tfs_scannext: INF - no more path list entries

5:26:24.818 AM: [6024.5744] <4> tar_backup::backup_endarg_state: INF - finish error count: 1

5:26:24.818 AM: [6024.5744] <4> tar_backup::backup_endarg_state: INF - finish fatal count: 0

5:26:24.818 AM: [6024.5744] <4> tar_backup_tfi::backup_arg_state: INF - remember start error count: 1

5:26:24.818 AM: [6024.5744] <4> tar_backup_tfi::backup_arg_state: INF - remember start fatal count: 0

5:26:24.818 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - starting scan for file directive:<C:\3parSnapshot\092413042718\f5a0e8ca-24a6-11e3-b540-0022196d93d0\logs\E06000185B5.log>

5:26:24.818 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - starting scan using name:<C:\3parSnapshot\092413042718\f5a0e8ca-24a6-11e3-b540-0022196d93d0\logs\E06000185B5.log>

5:26:24.818 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - in 'backup' mode

5:26:24.942 AM: [6024.5744] <2> tar_base::V_vTarMsgW: JBD - change journal NOT enabled for <C:\3parSnapshot\092413042718\f5a0e8ca-24a6-11e3-b540-0022196d93d0\logs\E06000185B5.log>

5:26:25.098 AM: [6024.5744] <4> dos_backup::tfs_scannext: INF - no more path list entries

5:26:25.098 AM: [6024.5744] <4> tar_backup::backup_endarg_state: INF - finish error count: 1

5:26:25.098 AM: [6024.5744] <4> tar_backup::backup_endarg_state: INF - finish fatal count: 0

5:26:25.114 AM: [6024.5744] <4> tar_backup_tfi::backup_arg_state: INF - remember start error count: 1

5:26:25.114 AM: [6024.5744] <4> tar_backup_tfi::backup_arg_state: INF - remember start fatal count: 0

5:26:25.114 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - starting scan for file directive:<C:\3parSnapshot\092413042718\f5a0e8ca-24a6-11e3-b540-0022196d93d0\logs\E06000185B6.log>

5:26:25.114 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - starting scan using name:<C:\3parSnapshot\092413042718\f5a0e8ca-24a6-11e3-b540-0022196d93d0\logs\E06000185B6.log>

5:26:25.114 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - in 'backup' mode

5:26:25.114 AM: [6024.5744] <2> tar_base::V_vTarMsgW: JBD - change journal NOT enabled for <C:\3parSnapshot\092413042718\f5a0e8ca-24a6-11e3-b540-0022196d93d0\logs\E06000185B6.log>

5:26:25.192 AM: [6024.5744] <4> dos_backup::tfs_scannext: INF - no more path list entries

5:26:25.192 AM: [6024.5744] <4> tar_backup::backup_endarg_state: INF - finish error count: 1

5:26:25.192 AM: [6024.5744] <4> tar_backup::backup_endarg_state: INF - finish fatal count: 0

5:26:25.192 AM: [6024.5744] <4> tar_backup_tfi::backup_arg_state: INF - remember start error count: 1

5:26:25.192 AM: [6024.5744] <4> tar_backup_tfi::backup_arg_state: INF - remember start fatal count: 0

5:26:25.192 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - starting scan for file directive:<C:\3parSnapshot\092413042718\f5a0e8ca-24a6-11e3-b540-0022196d93d0\logs\E06000185B7.log>

5:26:25.192 AM: [6024.5744] <4> dos_backup::tfs_scanstart: INF - starting scan using name:<C:\3parSnapshot\092413042718\f5a0e8ca-24a6-11e3-b540-0022196d93d0\logs\E06000185B7.log>

Thanks,

Vinay

[dvinayku]

Nope we are not seeing any VSS errors in Event viewer, How do we come to know its an HP snapshot issue...Any idea, or clue

Thanks,

Vinay