Backup to NAS works, Catalog fails, Catalog job devices list tape?

[Hooda_Thunkit]

Hi,

Backing up an exchange information store on Backup Exec 2014.  The backup completes, then an automatic catalog job starts.

The catalog job fails with

"Completed status: Failed Final error: 0xe0001211 - The volume that is used for the staging location must be a writable NTFS volume that is local to the Backup Exec server. Also, the volume must have the same sector size as the volume from which the Exchange log files were backed up. See the job log for details. Final error category: Resource Errors "

These conditions are true.

None of the other jobs on this media server have this problem.

When I look in the Jobs list, I see the Catalog job.  When I open the Catalog job, the Device section is only listing the tape devices that the media server has attached to it - the backup went to a NAS.

Any ideas? 

[CraigV]

...not sure why you want to automatically catalog the backup that has just run, and created a catalog file already?

Disable this and try again...

Thanks!

[Hooda_Thunkit]

This is the default setting after the GRT backup finishes - the Catalog job then starts.

Check the Backup Options on an Information Store backup - Delayed Catalog section.

"Catalog GRT data:

Immediately after a backup finishes, as a separate job."

It's this job that fails. 

[pkh]

Check that your NAS is not using 4K sector size and that the sector size of the NAS matches the sector size of the staging disk on the media server and the disk on on which Exchange resides.

[VJware]

ON the BE server, browse to HKLM\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\Exchange
Create a new string value "OnHostTemp" and set the value as C:\Temp or any local, NTFS location which has sufficient free space for the staging. Restart the BE services on both servers and rerun the backup.

[Hooda_Thunkit]

Sector size on Exchange mailstore disk (Virtual) obtained with fsutil:

Bytes Per Sector  :               512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
 

Sector size on BEX2014 Media server disk obtained with fsutil:

NTFS Volume Serial Number :       0x10a40c26a40c10c0

Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024

NAS sector size (via download of logs, NAS is ReadyNas3220):

Disk /dev/sdb: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): C21DAE61-07B5-4088-9641-F99E624D99F7
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 64-sector boundaries
Total free space is 4092 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64         8388671   4.0 GiB     FD00  
   2         8388672         9437247   512.0 MiB   FD00  
   3         9437248      5860529072   2.7 TiB     FD00  
Disk /dev/sdc: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 94A56E1F-4EFF-4E08-A568-7838BBA736F4
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 64-sector boundaries
Total free space is 4092 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64         8388671   4.0 GiB     FD00  
   2         8388672         9437247   512.0 MiB   FD00  
   3         9437248      5860529072   2.7 TiB     FD00  
Disk /dev/sde: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 7B751D38-FB08-4CF4-965A-6B459A8011E7
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 64-sector boundaries
Total free space is 4092 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64         8388671   4.0 GiB     FD00  
   2         8388672         9437247   512.0 MiB   FD00  
   3         9437248      5860529072   2.7 TiB     FD00  
Disk /dev/sdf: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 84A0192A-3752-442D-9AB4-5BB9E2DE8864
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 64-sector boundaries
Total free space is 4092 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64         8388671   4.0 GiB     FD00  
   2         8388672         9437247   512.0 MiB   FD00  
   3         9437248      5860529072   2.7 TiB     FD00  
Disk /dev/sda: 5860533168 sectors, 2.7 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 0096F1C2-5E61-480C-BA11-83DB87693D4E
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 64-sector boundaries
Total free space is 4092 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              64         8388671   4.0 GiB     FD00  
   2         8388672         9437247   512.0 MiB   FD00  
   3         9437248      5860529072   2.7 TiB     FD00  

Wasn't (until this morning) aware about the Great 512/4K Disk Sector Scandal™, seems then that the NAS is NOT setup for 4K sectors. So, the NAS/BEX/EXCHANGE sector sizes match.

Have put the registry setting in as recommended by VJware, (the directory is the one used in Configuration and Settings > Backup Exec Settings > Granular Recovery Technology) and restarted BEX services on media server and exchange server, will report back if the issue still presents itself after tonight's incremental.

Thanks for the advice so far.

[Hooda_Thunkit]

Registry fix in, Services restarted, Catalog job still failed, with a different error this time (log of catalog job):

Catalog- \\(my server name here)\Microsoft Information Store\First Storage Group
V-79-57344-918 - Unable to complete the operation. The following error was returned when opening the Exchange Database file: '-546 The log file sector size does not match the sector size of the current volumn. '

Will google this error but if anyone can throw some light on it please do so here.

BTW What's a "Volumn"?

[Hooda_Thunkit]

All seems to point back to sector sizes, which according to the best information available (posted above) all seem to match.

This is holding up a lot of work as the business MUST have GRT working - otherwise this backup solution is not appropriate for future use.

[Hooda_Thunkit]

Running out of places to look. 

FWIW, Exchange server is Virtual on WS2008R2

BEX Media Server is Virtual on WS2012R2

Any more places to look?

[VJware]

Would first recommend to patch both BE media server and the remote agent on the Exchange server with SP1.

Secondly, to verify the sector size on the NAS (as the above output lists only the logical sector size and not the physical one), run a command similar to fdisk. (The NAS vendor will be able to help you with the appropriate command for this model). Alternatively, run a backup to tape and check if these errors occur or not.

[Hooda_Thunkit]

Thanks for the reply - Patching the servers now, will keep you updated

[Hooda_Thunkit]

Patch applied (currently seeing this issue on multiple Backup Exec media servers), one did its incremental mailstore backup sucessfully without generating a catalog job (?), the other did its incremental backup, generated the catalog job, then the catalog job failed as usual with the e0001211 error.

[Hooda_Thunkit]

The server still that did not run the catalogue job after the incremental, did attempt a catalog job on last nights incremental.  Different error: e0002f7.  Log shows:

Job Completion Status:

Job ended: 12 November 2014 at 22:26:41 Completed status: Failed Final error: 0xe00002f7 - Cannot extract mailbox messages from the Exchange backup. Review the job log for more information. Final error category: Resource Errors For additional information regarding this error refer to link V-79-57344-759

Errors:

Catalog- \\PFSQDEX07\Microsoft Information Store\First Storage Group
V-79-57344-759 - Unable to complete the operation for the following reason: VFF Open Failure. This c...

TECH127758 explains this and says this might need to be done on a x64 bit server.

This is on an x64 bit virtual server, WS2012R2, 8GB Ram.

Have applied the registry fixes mentioned in TECH127758, steps 1 and 2.

Rebooted exchange server this morning.

Ran the incremental backup again (using Run Now) to see if the catalog job completes.

Incremental completed in 7 minutes.  That was 2 hours ago.

The Catalog Job has not happened at all.

?

[pkh]

What is the exact version of your Exchange, including the update level. Check the SCL below to make sure it is supported. http://www.symantec.com/docs/TECH217478

[pkh]

Using a VM for a media server is only supported for a limited number of configurations. For example, using pass-thru to a real device is not supported. What are you backing up to and how is it connected to the media server?

[Hooda_Thunkit]

PKH - Thanks for your reply.  The setup is as follows (we have multiple sites, but these are the two I am currently addressing):

SITE 1 (reporting 0xe00002f7 on catalog jobs after Information Store backups)

---

Exchange Server:

Virtual WS2008 R2 SP1 64-bit OS 3GB RAM

Exchange Server 2007

Microsoft Corporation

Version: 8.3.83.6

FSUTIL on mail volume of Exchange Server :

NTFS Volume Serial Number :       0x7c80a49980a45c00
Version :                         3.1
Number Sectors :                  0x000000003e7fe7ff
Total Clusters :                  0x0000000007cffcff
Free Clusters  :                  0x0000000003187bdf
Total Reserved :                  0x0000000000000780
Bytes Per Sector  :               512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000001140000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c1140
Mft Zone End   :                  0x00000000000cc820
RM Identifier:        58BED3B4-286C-11E4-8B1D-00155D0A1B09

Media Server:

Physical WS2012 R2 8GB RAM 64-bit OS

Backup Exec 2014

FSUTIL on mail volume of MEDIA Server :

NTFS Volume Serial Number :       0x10a40c26a40c10c0
NTFS Version   :                  3.1
LFS Version    :                  2.0
Number Sectors :                  0x000000000887ffff
Total Clusters :                  0x000000000110ffff
Free Clusters  :                  0x00000000007eb542
Total Reserved :                  0x0000000000000f80
Bytes Per Sector  :               512
Bytes Per Physical Sector :       512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x000000000c740000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000cc740
Mft Zone End   :                  0x00000000000cc820
Resource Manager Identifier :     E219B3FB-EDA6-11E3-81DD-99D36235A287

NAS

Netgear ReadyNAS 3220 (disk diags  in my post above). 5 x SATA 3TB disks

Excange server has 2 jobs - 1 for information store, 1 for the rest. AOFO off.  GRT on.

Storage is defined as \\ip address of nas1\FullBackupShare, standard disk backup options and \\ip address of nas1\IncrementalBackupShare
 

SITE 2 (reporting 0xe0001211 on catalog jobs after Information Store backups)

---

Exchange Server:

Virtual WS2008 R2 SP1 64-bit OS 3GB RAM

Exchange Server 2007

Microsoft Corporation

Version: 8.3.83.6

FSUTIL on mail volume of Exchange Server :

NTFS Volume Serial Number :       0xc6bc92f1bc92daef
Version :                         3.1
Number Sectors :                  0x000000003e7fe7ff
Total Clusters :                  0x0000000007cffcff
Free Clusters  :                  0x00000000059e0241
Total Reserved :                  0x0000000000002600
Bytes Per Sector  :               512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x0000000000040000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000c0040
Mft Zone End   :                  0x00000000000cc840
RM Identifier:        1C947CDF-286B-11E4-AD05-00155D327A00

Media Server:

Virtual WS2008 R2 3GB RAM 64-bit OS

Backup Exec 2014

FSUTIL on mail volume of MEDIA Server :

NTFS Volume Serial Number :       0xf45094235093eb1e
Version :                         3.1
Number Sectors :                  0x000000000fdccfff
Total Clusters :                  0x0000000001fb99ff
Free Clusters  :                  0x00000000019b17bb
Total Reserved :                  0x0000000000000040
Bytes Per Sector  :               512
Bytes Per Cluster :               4096
Bytes Per FileRecord Segment    : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length :           0x000000000bb80000
Mft Start Lcn  :                  0x00000000000c0000
Mft2 Start Lcn :                  0x0000000000000002
Mft Zone Start :                  0x00000000000cb380
Mft Zone End   :                  0x00000000000cc820
RM Identifier:        4BB45CC5-4045-19E4-AE11-A5E269615BBF

NAS

Netgear ReadyNAS 2120 (disk diags confirm same sectors 4 x SATA 2GB disks).

Excange server has 2 jobs - 1 for information store, 1 for the rest. AOFO off.  GRT on.

Storage is defined as \\ip address of nas2\FullBackupShare, standard disk backup options and \\ip address of nas2\IncrementalBackupShare

Thank in advance for any input.

[GROBSON]

This did not work for me.

[sdiedrich]

Hello,

I also did a fresh install and noticed this.  There seems to be a setting that by default is unchecked.  If you click on the Menu icon, then go to 'Configuration and Settings', then 'Backup Exec Settings', then 'Storage' and scroll all the way down, there is a 'Data management' section that has an unchecked setting that says 'Allow Backup Exec to delete all expired backup sets'.