This is a troubleshooting article showing some of the Issues that one might encounter when working with the Deduplication storage and how to go about solving them or collect info that would help Veritas Technical Support to solve it quicker for you.
It is important to start your checks by making sure the disk is healthy.
Deduplication folder in Backup Exec may show offline due to any of the following:
Ensure all Dedupe services can be started. Deduplication Engine, Deduplication Manager, Postgres, are all run under local system account. If any one does not start, then following can be checked:
1. PostgreSQL or any other deduplication service startup issues due to registry set incorrectly. Confirm ImagePath is accurate. This can be compared with another working system if available:
2. Logs that can be referred to apart from error in Windows Event Viewer for each Dedupe service startup issue
3. If the services starts up, it is also important to note that the queue processing in dedupe is working. If tlogs (inside dedupe\queue) are older than a couple of days and are not going away then this could mean there may be something wrong with the internal queue process.
They should be committing automatically and under no circumstances be deleted by anyone (manually deleting of tlog files can badly affect the dedupe folder and backup sets).
Note: Any errors for queue process will be recorded in Dedupe\log\spoold\storaged.log
See http://www.veritas.com/docs/000087645 - A known case where queue processing does not run.
Queue processing can be manually triggered by running crcontrol.exe --processqueue from command prompt (run from BE Install path. Run the command 2 times) to see if the tlogs inside the queue folder get cleared.
Note: Read the log from the bottom as the latest entries are added at the very end.
If errors are still being reported in stroraged.log then contact Veritas Technical Support.
4. If the dedupe folder is still offline i.e. all Deduplication Services are running, then restart the Backup Exec services. If the deduplication folder is still offline refer to the adamm.log present in Backup Exec Install Path\logs folder. Review the adamm.log from the bottom.
Example: See adamm log snippet below. In this case, the dedupe user's password was not correctly updated.
See, How to change password for logon account used for deduple storage folder
Here's how to look for a similar section in your adamm.log.
 02/24/16 03:09:24.300 Read OST Server Records - start
 02/24/16 03:09:24.345 OST Server: PureDisk:BE-CASureDiskVolume
 02/24/16 03:09:24.345 Read OST Server Records - end
 02/24/16 03:09:24.345 DeviceIo Discovery - start
 02/24/16 03:09:26.431 DeviceIo: STS: Critical: (Storage server: PureDisk:BE-CAS) PdvfsRegisterOST: Failed to register with SPA on storage server BE-CAS. Check to make sure the server is on and that the services are running. (Permission denied) V-454-25
 02/24/16 03:09:26.432 DeviceIo: sts_open_server PureDisk:BE-CAS dedup 2060029
 02/24/16 03:09:26.432 DeviceIo: ostaspi: sts_open_server PureDisk:BE-CAS as dedup error 2060029
 02/24/16 03:09:26.432 DeviceIo: ostaspi: authorization with server PureDisk:BE-CAS has failed
SGMON.exe can also be used to debug a Deduplication Folder Offline issue. If all deduplication related services are starting without any problem then shut down only the Backup Exec services, launch SGMON with Device and Media verbose (verbose can be enabled from SGMON settings) and start all Backup Exec services.
Note: SGMON.exe is present in Backup Exec Install Path
Filter SGMON or any other log with string “ERR” as shown below. The SGMON log file may be named differently, therefore check the log location to confirm the name of the log file.
C:\Program Files\Symantec\Backup Exec\Logs>findstr /C:"ERR" BE-CAS-SGMon.log > SGMON_ERR.log
PVLSVR: [02/24/16 03:18:32]  DeviceIo: STS: Error: [ERROR] PDSTS: pd_register: PdvfsRegisterOST(BE-CAS) failed (13ermission denied)
PVLSVR: [02/24/16 03:18:32]  DeviceIo: STS: Error: [ERROR] PDSTS: add_mount: PdvfsMount() failed for mount point:<BE-CAS#1> (13ermission denied)
PVLSVR: [02/24/16 03:18:32]  DeviceIo: STS: Error: [ERROR] PDSTS: open_server: pd_mount() failed (2060029:authorization failure)
PVLSVR: [02/24/16 03:18:32]  DeviceIo: STS: Error: [ERROR] PDSTS: impl_open_server: open_server(PureDisk:BE-CAS) failed (2060029:authorization failure)
PVLSVR: [02/24/16 03:18:32]  DeviceIo: STS: Error: [ERROR] PDSTS: pi_open_server_v7: impl_open_server(PureDisk:BE-CAS) failed (2060029:authorization failure)
There could be multiple reasons for dedupe being offline even if the services are started. This was just one example. But one thing to note is that deletion of the dedupe folder from UI and re-importing would not help in bringing the Dedupe folder online for the reasons discussed above.
Note: While creating or re-importing a Deduplication Storage Folder in Backup Exec, pdde-config.log needs to be referred if an error is seen while creation/re-import. The log file is present in DedupeFolder\log.
See Deduplication Folder creation or recreation fails with "An Error Occurred while Creating the dedupli....
Sometimes it may be required to delete the EtcPath and ConfigFilePath String Value from the following Windows Registry location:
Note: This only applies to re-import cases.
Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.
5. After upgrading to Backup Exec 2014, the Deduplication Manager (SPAD) is seen to crash on startup, See http://www.veritas.com/docs/000022713.
B. Backup fails when targeted to Deduplication folder and works when targeted to normal disk storage
It is important to narrow down the issue if the backup problem only occurs when directed to a deduplication folder. It may caused by the kind of resource we are backing up or something else outside of deduplication storage but affecting the backup and it could also be something within Dedupe.
There could also be other discrepancies, refer another Issue http://www.veritas.com/docs/000088005
Note: Nothing should be deleted or touched in the above path. Just view the content and attempt to restart Backup Exec services to dismount the logical drives if there are any. The server name in the path should be replaced by the hostname of the server where deduplication folder is located.
On the Backup Exec Server, KeepAliveTime registry dword can be created and set to decimal 5000 (Note: server needs to be rebooted) to test if it helps.
Refer http://www.veritas.com/docs/000076435 for more information.
If client side backup still fails an SGMON log with Job Engine, Backup Exec Server, Device and Media (from settings, select verbose for device and media) selected can be captured when backup is running. Also get the SGMON log from Client Server. More details about Logging can be found here - http://www.veritas.com/docs/000005927.
There are some additional logs created which can also be collected for analysis by Veritas Technical Support staff DedupeLocation\log\spoold\remoteclientserver\beremote\store
The above log location can be interpreted as spoold connecting to remote server and to its process i.e. Beremote process and that Beremote has made a store connection i.e. backup with Dedupe Server. The logs inside this can be looked for connection reset errors just in case the connection is being aborted. Antivirus software can sometimes cause these aborts so in order to isolate this issue, if possible, uninstall the Antivirus Software and see how it goes. It is sometimes observed that disabling the Antivirus application does not help and hence it is recommended to test by uninstalling the Antivirus Software (Note: This is not a solution but doing this will help us isolate the problem).
SGMON.exe is a utility which helps to debug many issues in Backup Exec. We can use SGMON to debug this Issue when the optimized duplication is running. This will need to be done on the Backup Exec server which running the optimized duplication operation.
Additionally, also look at the replication.log located at DedupeFolderlocation\log\spad on the primary deduplication backup (i.e 1st Copy). On the target side i.e. where the optimized duplication is targeted to, check DedupeDrive:\log\spoold\primarydedupeBEservername\spad.exe\Store\ and review the logs with the maximum size for clues.
Also review Windows Event Viewer and ADAMM log from the Backup Exec server for more information.
The Opt Dupe between the Hardware OST Appliance will need the SGMON and vendor OST plugin logs during the opt dupe failure to narrow down the Issue.
a. Check if the deduplication device is nearing full capacity:
The capacity column available in the Storage tab of Backup Exec console shows usage of a disk storage. If it is RED in color, it is an alarm that the Deduplication device may be nearing its full capacity.
Note: 95% usage is the highest we have seen a deduplication folder get to. This is the time to reclaim space within Deduplication Storage so that newer backups can run. It is recommended that "The percentage of disk space to reserve for non-Backup Exec operation" value should never be lowered below 5%.
b. Check for deduplication statictics:
Following command can be run from command prompt -> BE Install Path to check the real deduplication statistics:
c. Manual space reclamation:
If space need to be reclaimed follow this Technical Article Manual space reclaimation for Deduplication Storage Folder in Backup Exec 2012 and above.
Note: It is recommended to stop backups (i.e. backup to another storage) while attempting to manually reclaim the space within dedupe folder, since it may be difficult to identify how much space has been reclaimed due to a constant addition of data if Backups continue to run while reclaim process is being carried out.
Points to Remember Reclaiming Space manually from Deduplication Storage :
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.