07-18-2012 07:34 PM
Background:
Our BE2012 connected with physical and vitual tape library (8 drives), we backup the data to VTL daily and export the monthly backup set to physical tape.
We've updated the BE2012 with Service 1a two weeks ago, everything goes fine.
OS: Windows 2008 R2
Physical Library: Hewlett Packard MSL G3 Series Library. 2 slot
Virtual Tape Library: DataDomain DD670, 8 drives
No specific drives selected in all backup jobs, we checked all the drives in VTL.
Issue:
Last week, we encountered an issue about BE2012. the BE2012 only uses 2 drives only in VTL. Hence some of backup jobs keep queued and wait until the one of first two backup jobs completed. And the job status report "Ready no idle devices are available...."
We've called the support last Friday, and they finally replied me yesterday, what they did is try to remove the Windows Driver "Hewlett Packard MSL G3 Series Library" and use "Unknown Medium Changer" under "Device Manager" AND send me an email about close this case!
After checking backup job run on yesterday, the same issue happened.
This morning, I've tried to remove that VTL, restart server again. BE2012 recognized the VTL and we configurated it again.
When we try to inventory the tapes in VTL, the BE2012 inventory process still use 2 drives in VTL.
Anyone have idea about this?
Now we have to move two backup jobs which takes over 12 hour backup to BE2010 server, however, this is only a short term solution for us.
BE2012 gives me a lot of time on troubleshooting and calling Symantec support. I miss BE2010!
Solved! Go to Solution.
07-19-2012 07:25 PM
Yesterday, the Symantec support called me and tried to figure out what was happening.
As troubleshoot with Symantec support, we finally deleted the VTL in data domain, restarted the services and tried to backup again. Sadly, the result still the same. Then he gathered the log for further investigation.
After disconnected the line with Symantec support, I found that he did not cancel the testing backup jobs before deleting the VTL in data domain. Hence, BE2012 reported an error message for some active jobs in the tape drives which were not existing.
And none of backup jobs can be started, all in "Queued" states. (I guessed the last 2 available tapes drives were occupied by the testing backup job.)
Unfortunately, that was after office hour!!!! I stayed in office over time for troubleshoot by myself...
And finally I had to logon to Database level to delete the orphan entries in "Device" table of BEDB database, and restarted the services.
Finally BE2012 now can use all the tape drives in VTL now. I hope this is the last time I call for Symantec....
07-18-2012 07:50 PM
Have you checked which all licenses are installed?
07-18-2012 07:55 PM
Sure! All licenses are confirmed with Symantec
07-18-2012 08:51 PM
1) If unlimited drive option is not working then BE should only use one drive within VTL.
2) You have LEO option, for your HP tape library.
I doubt BE is messingup both these options and seems like it is applying LEO to VTL, because all the drives are reported as online but only two are being used.
If i were you, i will try removing and readding VTL unlimited drive license. Remove license and see how many drives BE will use before adding it back. If it uses two drives, then remove LEO license and add VTL unlimited drive option and check.
07-18-2012 08:51 PM
I tried to create a new backup job and config this backup job use specific drive.
However, it result still as same as before, the job state is "Active" but status is "Queued". The tapes in storage are Scratch Media. After checking the Backup Exec Debug Monitor, I find the following
PVLSVR: [07/19/12 11:39:12] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_INFO )Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}PVLSVR: [07/19/12 11:39:12] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_DEVICE_GUID )Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}TargetDevice = {B1F45157-B63E-445A-B635-F29CD2B099EE}, "Robotic library 0001 [0460..0521]" (Device Pool)TargetDevice = {C09ACD21-FAB3-4DAC-88E6-F4182F85DE29}, "Tape drive 0001" (Drive)TargetDevice = {325E172C-0B98-49B3-9C0E-8D2B5E5C5C03}, "Tape drive 0002" (Drive)TargetDevice = {CE04895D-73C5-4D29-B53A-0EB9E51A9E71}, "Tape drive 0003" (Drive)TargetDevice = {67C0AEFF-CD62-49D6-B9FE-CA2473516DBD}, "Tape drive 0004" (Drive)PVLSVR: [07/19/12 11:39:12] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_MEDIA_GUID )Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}TargetMedia = {00000009-03E9-0000-0000-000000000000}, "Keep Data for 4 Weeks" (Media Set)PVLSVR: [07/19/12 11:39:12] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_EXCLUDE_MEDIA_GUIDS )Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}PVLSVR: [07/19/12 11:39:12] [7356] AdammSession::CheckTargetDeviceStatus()Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}Job = {04D7CFF8-7808-4EC2-A5D0-EC73BBB52883}, "HKDC-BACKUP01 Backup 00218-Full"TargetDevice = {C09ACD21-FAB3-4DAC-88E6-F4182F85DE29}, "Tape drive 0001"State = 0003, ReservationID = 0000, AvailableMovers = 0001TargetDevice = {CE04895D-73C5-4D29-B53A-0EB9E51A9E71}, "Tape drive 0003"State = 0003, ReservationID = 0000, AvailableMovers = 0001TargetDevice = {325E172C-0B98-49B3-9C0E-8D2B5E5C5C03}, "Tape drive 0002"State = 0003, ReservationID = 0000, AvailableMovers = 0001TargetDevice = {67C0AEFF-CD62-49D6-B9FE-CA2473516DBD}, "Tape drive 0004"State = 0003, ReservationID = 0000, AvailableMovers = 0001PVLSVR: [07/19/12 11:39:12] [7356] AdammSession::MountOverwriteMedia() - 0004 drivesPVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - 0260 media view overwritable mediaPVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - 0046 initial priority overwritable mediaPVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - 0046 priority overwriteable media after blank/unknownPVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - 0046 priority overwritable media after standalonePVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - qualified drive Tape drive 0003 slot 0467PVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - reserve error ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)PVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - qualified drive Tape drive 0002 slot 0467BECAT: [07/19/12 11:39:12] [6012] SQL connection: 790fdb0 select * from ControlInfoBECAT: [07/19/12 11:39:12] [6012] SELECT * FROM CatImageFileDeleteTableBECAT: [07/19/12 11:39:12] [6012] SQL connection: 790fdb0 SELECT * FROM CatImageFileDeleteTableBECAT: [07/19/12 11:39:12] [6012] result(0) numRetry(300)PVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - reserve error ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)PVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - qualified drive Tape drive 0004 slot 0467BECAT: [07/19/12 11:39:12] [6012] Number of records returned: 0BECAT: [07/19/12 11:39:12] [6012] Cleanup images : 0PVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - reserve error ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)PVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - qualified drive Tape drive 0001 slot 0467PVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - reserve error ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)PVLSVR: [07/19/12 11:39:12] [7356] PvlSession::MountOverwriteMedia() - ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE) - 0000 secondsPVLSVR: [07/19/12 11:39:12] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_MOVER_MOUNT_OVERWRITE )Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.720 [server] + storagedevice.cpp (345):BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.720 [server] | DoMount Execute returned: 0xe000810cBENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.720 [server] + backupjob.cpp (6013):BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.720 [server] | BackupJob: unable to scratch mount, will retry append later.BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.720 [server] + backupjob.cpp (6126):BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.720 [server] | BackupJob: unable to mount, error 0xE000810C.BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.721 [server] + storagedevice.cpp (516):BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.721 [server] | Closing out mover session(0000000009CAF760): {253da68d-c69c-43ca-b71e-5df5c41703d9}BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.721 [server] | device id: {c09acd21-fab3-4dac-88e6-f4182f85de29}PVLSVR: [07/19/12 11:39:12] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_MOVER_DISMOUNT )Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}Flags = 0x00000020BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.721 [server] + storagedevice.cpp (540):BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.721 [server] | Dismounted mover(0000000009CAF760). result - 0x00000000BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.721 [server] + storagedevice.cpp (553):BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:12.721 [server] | Releasing mover: 0000000009CAF760PVLSVR: [07/19/12 11:39:12] [7356] AdammSession::Release()Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:12.722 [server] - DeviceManager: incoming event firedBENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:12.722 [server] - DeviceManager: Executing RequestDeviceCall from thread 1520.BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:12.722 [server] - DeviceManager: stallingBENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:12.722 [server] - DeviceManager: going to sleep for 29000 msecsBESERVER: [07/19/12 11:39:16] [2232] -1 SecurityBO: Successfully setup RAWS certs.BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.724 [server] - DeviceManager: timeout event firedBENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.724 [server] - DeviceManager: processing pending requestsBENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.724 [server] - DeviceManager: Processing device request for thread 1520.BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.724 [server] - DeviceManager: query cache missPVLSVR: [07/19/12 11:39:41] [7356] PvlEntityDatabase::QueryEntity()ParentEntityType = DRIVE_POOL_ENTITY_TYPEParentEntityGuid = {B1F45157-B63E-445A-B635-F29CD2B099EE}EntityType = NULL_ENTITY_TYPEEntityGuid = {00000000-0000-0000-0000-000000000000}EntityName = ""EntityFlags = 0x00000000BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Available and held drives from the poolBENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0001 {c09acd21-fab3-4dac-88e6-f4182f85de29}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0002 {325e172c-0b98-49b3-9c0e-8d2b5e5c5c03}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0003 {ce04895d-73c5-4d29-b53a-0eb9e51a9e71}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0004 {67c0aeff-cd62-49d6-b9fe-ca2473516dbd}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Drives not in use by the EngineBENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0001 {c09acd21-fab3-4dac-88e6-f4182f85de29}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0002 {325e172c-0b98-49b3-9c0e-8d2b5e5c5c03}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0003 {ce04895d-73c5-4d29-b53a-0eb9e51a9e71}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0004 {67c0aeff-cd62-49d6-b9fe-ca2473516dbd}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Drives available (not in use and not held)BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0001 {c09acd21-fab3-4dac-88e6-f4182f85de29}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0002 {325e172c-0b98-49b3-9c0e-8d2b5e5c5c03}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.730 [server] - Tape drive 0003 {ce04895d-73c5-4d29-b53a-0eb9e51a9e71}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.731 [server] - Tape drive 0004 {67c0aeff-cd62-49d6-b9fe-ca2473516dbd}PVLSVR: [07/19/12 11:39:41] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_DEVICE_GUID )Session = {18606A09-966C-4049-884D-FCF602B38D5D}TargetDevice = {B1F45157-B63E-445A-B635-F29CD2B099EE}, "Robotic library 0001 [0460..0521]" (Device Pool)TargetDevice = {C09ACD21-FAB3-4DAC-88E6-F4182F85DE29}, "Tape drive 0001" (Drive)TargetDevice = {325E172C-0B98-49B3-9C0E-8D2B5E5C5C03}, "Tape drive 0002" (Drive)TargetDevice = {CE04895D-73C5-4D29-B53A-0EB9E51A9E71}, "Tape drive 0003" (Drive)TargetDevice = {67C0AEFF-CD62-49D6-B9FE-CA2473516DBD}, "Tape drive 0004" (Drive)PVLSVR: [07/19/12 11:39:41] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_MEDIA_GUID )Session = {18606A09-966C-4049-884D-FCF602B38D5D}TargetMedia = {00000009-03E9-0000-0000-000000000000}, "Keep Data for 4 Weeks" (Media Set)PVLSVR: [07/19/12 11:39:41] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_EXCLUDE_MEDIA_GUIDS )Session = {18606A09-966C-4049-884D-FCF602B38D5D}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.739 [server] - DeviceManager: Handing out session {18606a09-966c-4049-884d-fcf602b38d5d} based on available drive:BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.739 [server] - Tape drive 0001 {c09acd21-fab3-4dac-88e6-f4182f85de29}BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.739 [server] - DeviceManager: returned result to caller 0x00000000BENGINE: [07/19/12 11:40:35] [1328] 2012-07-19T11:39:41.739 [server] - DeviceManager: going to sleep for 900000 msecsPVLSVR: [07/19/12 11:39:41] [7356] PvlEntityDatabase::QueryEntity()ParentEntityType = DRIVE_POOL_ENTITY_TYPEParentEntityGuid = {B1F45157-B63E-445A-B635-F29CD2B099EE}EntityType = NULL_ENTITY_TYPEEntityGuid = {00000000-0000-0000-0000-000000000000}EntityName = ""EntityFlags = 0x00000000PVLSVR: [07/19/12 11:39:41] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_JOB_INFO )Session = {18606A09-966C-4049-884D-FCF602B38D5D}Job = {04D7CFF8-7808-4EC2-A5D0-EC73BBB52883}, "HKDC-BACKUP01 Backup 00218-Full"PVLSVR: [07/19/12 11:39:41] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_MORE_JOB_INFO )Session = {18606A09-966C-4049-884D-FCF602B38D5D}Job = {04D7CFF8-7808-4EC2-A5D0-EC73BBB52883}, "HKDC-BACKUP01 Backup 00218-Full"SessionFlags = 0x00000002JobFamilyID = {8A2C78AE-231F-41BA-8476-69AA3CDCB6DD}BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:41.745 [server] + backupjob.cpp (5926):BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:41.745 [server] | BackupJob::MountThread : Device path hint : m_deviceHost.empty() is TRUEPVLSVR: [07/19/12 11:39:41] [7356] AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_DEVICE_HOST )Session = {18606A09-966C-4049-884D-FCF602B38D5D}Device Host = ""BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:41.745 [server] + backupjob.cpp (5943):BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:41.745 [server] | BackupJob: acquired device 'Tape drive 0001'.BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:41.745 [server] + backupjob.cpp (5966):BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:41.745 [server] | Setting device attributes 0x0BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:41.745 [server] + backupjob.cpp (5971):BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:41.745 [server] | BackupJob: trying to mount appending volume.BENGINE: [07/19/12 11:40:35] [1520] 2012-07-19T11:39:41.745 [server] + storagedevice.cpp (332):
07-18-2012 08:59 PM
Funny, when I edit this backup to use all drives in VTL, I find that BE2012 only use the 6th or 7th drives only in the VTL!
I'm going crazy now...
07-18-2012 10:42 PM
I tried to remove, unfortunately, BE2012 prompted "Product licenses may not be deleted"
07-18-2012 10:46 PM
Follow the procedure mentioned in this TN to do so: http://www.symantec.com/docs/TECH181692
07-18-2012 11:41 PM
I tried to unselect the license and reboot the server.
The result still the same, the BE2012 only use the 6th and 7th drive in the VTL.
07-18-2012 11:44 PM
Have you tried the same with LEO license?
07-18-2012 11:49 PM
I downloaded the license file again from the https://licensing.symantec.com/
07-19-2012 07:25 PM
Yesterday, the Symantec support called me and tried to figure out what was happening.
As troubleshoot with Symantec support, we finally deleted the VTL in data domain, restarted the services and tried to backup again. Sadly, the result still the same. Then he gathered the log for further investigation.
After disconnected the line with Symantec support, I found that he did not cancel the testing backup jobs before deleting the VTL in data domain. Hence, BE2012 reported an error message for some active jobs in the tape drives which were not existing.
And none of backup jobs can be started, all in "Queued" states. (I guessed the last 2 available tapes drives were occupied by the testing backup job.)
Unfortunately, that was after office hour!!!! I stayed in office over time for troubleshoot by myself...
And finally I had to logon to Database level to delete the orphan entries in "Device" table of BEDB database, and restarted the services.
Finally BE2012 now can use all the tape drives in VTL now. I hope this is the last time I call for Symantec....